[WIP] use fork+execvp instead of fork

[unstabler]

DESCRIPTION

• prevents connection loss to XPC services after fork(), which leads to VideoToolbox-related function breaks

TODO

• merge Implement VideoToolbox-powered H.264 encoder for macOS neutrinolabs/xrdp#2263 to xrdp-tumod after gfx pipeline has implemented

[unstabler]

Hi Gyuhwan,

You wrote:

DESCRIPTION OF PROBLEM
Hello, we are implementing macOS support for an open source remote desktop project called xrdp.

We are implementing a feature to compress image data with h.264 codec to save bandwidth, but when the daemon process fork(), VTCompressionSessionCreate() fails to initialize and returns -12903( kVTInvalidSessionErr).

If VTCompressionSessionCreate() is called without fork(), initialization succeeds normally.

Is there something we are doing wrong?

(+ I heard before that there is a problem that fork() breaks the Objective-C runtime library (Foundation?), is the VideoToolbox code written in plain C also affected?)

STEPS TO REPRODUCE

• call fork()
• call VTCompressionSessionCreate() from child process

It seems that you’re calling fork without calling exec*. Is that right?

If so, that’s not supported on macOS. Many Apple frameworks use Mach messaging to communicate with helper processes, for example, launchd daemons and agents. Mach messaging and fork don’t play well together. The child inherits the parent’s memory but it does not inherit the parent’s Mach port namespace. So the child thinks it has access to a bunch of Mach services but it doesn’t.

Many of the most-commonly used frameworks, like Foundation, have asserts that detect this situation. It seems that VideoToolbox is reporting an error in this case.

In short, if you call fork without calling exec* you will run into problems. If you’re porting Unix-y code that relies on this then you may get away with this, but not if you use any of Apple’s frameworks.

If you're spinning up a new process for some VideoToolbox work, most folks use threads for this sort thing.

Apple Developer Technical Support

[matt335672]

Couple of things to think about for you!