[Feature] Make APIs non-blocking

[zcbenz]

In MLX to get the results of computation, the main thread has to wait for results from GPU threads with blocking APIs like mx::eval and mx::synchronize.

This causes a few problems:

1. For scripting languages like Node.js, all operations are supposed to be non-blocking, when calling blocking APIs like mx.eval, the whole process "freezes" and you can not even terminate it with Ctrl-C.
2. It is impractical to integrate MLX into a GUI app: you are not supposed to wait for an event synchronously in the GUI thread, doing so results in UI lags.
3. It is very hard to stop a generation: when the main thread is blocked at mx.eval, it would be challenge to tell the script to stop the generation after mx.eval is done when the script itself also runs in the main thread.

So it would be necessary to be able to wait for evaluation asynchronously, for example await mx.asyncEval(array) in Node.js. To be able to do so we would need a few changes in MLX:

1. Make the C++ mx::async_eval API return the synchronizer event.
2. Make the Event movable.
3. Make it safe to wait for the synchronizer event in any thread. (I believe it is already so but I want confirmation from MLX team.)

Implementing the asynchronously waitable async_eval API in language bindings would be:

1. Return a Promise in the API.
2. Wait for the synchronizer event in a new thread.
3. Resolve the Promise in the main thread after the event is signaled.

[awni]

Could you spawn a thread to call MLX eval and have the calling thread get a promise from the spawned thread?

I guess it's not clear to me what we gain by doing this inside MLX vs just handle it in the calling code which is more flexible and doesn't require us to add a synchronization type.

[zcbenz]

I didn't realize it is possible to call eval in a separate thread, that should solve my problem, thanks!

[angeloskath]

Although it is possible, I would advise caution as some parts of MLX core are not thread safe. Basically if you are doing function transformations while evaluating on two different threads that may cause problems. I can't think of anything other than the tracing counter right now but we weren't going for thread safety throughout core which means there could be more thread unsafe things.

Creating new arrays should be fine and of course simply waiting/checking for the result or rendering the previous one is also 100% fine.