Support (or don't proclude) the Gradle Worker API

[nhoughto]

One inherent limitation of Gradle projects is only a single task can run at any given time, with the Worker API (https://docs.gradle.org/current/userguide/worker_api.html#converting_to_the_worker_api) to allow many jobs to be run within a single task, using the Worker API requires specifically coding for it though.

The current 3.x by accident (i assume?) has a number of the right abstractions to make using the worker API possible, but similar to #190 having classes not be open makes extending the various innards of the library hard/impossible. For example the ProjectApiHelper abstraction solves for a large part of the normal problems implementors have with using the Worker API, but closed classes and NodeExtension relying on Project are two other blockers.

Since the use of the Worker API is pretty specific (for ex run N jest test suites in parallel) and the intent of this library is quite broad (run npm/yarn stuff) I don't expect you would do the implementation to support this, just maybe make it more possible for this library be extended to support it? Open to PRs?

[deepy]

Opening up the classes makes sense, but I'm curious about what ProjectApiHelper would help you solve, the way we use it is to get compatibility with Gradle 5 and 6+ at the same time. It's pretty specifically tailored to just help us call the right thing depending on Gradle version.

If you're looking at implementing something I'm guessing the abstraction is less useful and you probably just want to use things like ObjectFactory, ExecOperations and FileSystemOperations directly.

The closed classes are going to get opened, but I haven't had enough time to get working on it yet, though I'm curious on why the NodeExtension relying on Project is a blocker.
Do you intend to use it as a general data class?

[nhoughto]

So current plan is basically to reimplement YarnTask as a WorkAction and WorkParameters, to do that you have to work within extra constraints: https://docs.gradle.org/current/userguide/custom_tasks.html#worker_api Basically you can only deal with managed objects so that gradle can reason about all the dependencies etc, no more just passing arbitrary stuff around. It’s a bit tricky to POC with closed classes but my guess is getting the NodeExecConfiguration and configuring it won’t be possible as it requires the Project, and accessing Project is a no no (I’m pretty sure) they really want you to describe all inputs to the task in the work parameters and that basically needs to be serialisable, so no project, NodeExtensions etc. That’s why that abstraction ProjectHelperApi is great as it can be satisfied by things available in the worker api. Open classes will def help 👍🏼 Thanks

…

On 30 Aug 2021, 8:50 PM +1000, Alex Nordlund ***@***.***>, wrote: Opening up the classes makes sense, but I'm curious about what ProjectApiHelper would help you solve, the way we use it is to get compatibility with Gradle 5 and 6+ at the same time. It's pretty specifically tailored to just help us call the right thing depending on Gradle version. If you're looking at implementing something I'm guessing the abstraction is less useful and you probably just want to use things like ObjectFactory, ExecOperations and FileSystemOperations directly. The closed classes are going to get opened, but I haven't had enough time to get working on it yet, though I'm curious on why the NodeExtension relying on Project is a blocker. Do you intend to use it as a general data class? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[deepy]

NodeExecConfiguration doesn't necessarily need Project though :-)

In your case I think you might want to have Gradle inject the services that are in ProjectHelperApi instead of depending on our internal helper, the link provided shows how to get FileSystemOperations but you could also use ProjectHelperApi as a reference

[nhoughto]

Yep that’s what I’m doing, I hit a wall with NodeExecConfig tho and closed classes, so I’ll have another look when they get opened 👍🏼

…

On 30 Aug 2021, 10:41 PM +1000, Alex Nordlund ***@***.***>, wrote: NodeExecConfiguration doesn't necessarily need Project though :-) In your case I think you might want to have Gradle inject the services that are in ProjectHelperApi instead of depending on our internal helper, the link provided shows how to get FileSystemOperations but you could also use ProjectHelperApi as a reference — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[deepy]

I've created #202 where I've started opening classes, it doesn't have all of them yet but I believe it should be the major ones necessary, let me know if there's any specifics you feel might be missing
(except NpmExecRunner/YarnExecRunner, I haven't gotten around to those yet)

[deepy]

I've had some success with getting node detection (and downloading) working and while node detection should work great with the Worker API I'm unfortunately not quite sure how to expose this to a library, unless the library is created and used in the same build I guess

[XhstormR]

ping

[deepy]

@XhstormR bad news is that the initial idea has class loader issues
The good news is that the abstractions for workers are getting better
If you have a concrete use-case for workers I can see if I can create an example