Consolidation of multiple similar libraries

[keithamus]

Hi @mysticatea and @k88hudson! Thanks for your hard work with npm-run-all!

I maintain a similar library - the functionality is slightly different but it shares a lot of common functionality with npm-run-all. My library is called parallelshell - which I maintain with @paulpflug. I am also aware of a few other libraries that attempt to solve the problem of properly parallelising npm (or shell) tasks:

• concurrently which is written by @kimmobrunfeldt and has had contributions from @luv2code.
• shell-executor which is written by @royriojas.
• npm-parallel which is written by @spion
• pkgrun which is written by @reggi

We're all trying to reach the same goal - run npm tasks quickly and easily in parallel (and sometimes serially), and have them log & exit in ways you'd expect. For the most part, each of us (I presume) has convergently developed these solutions. While it's great that we're all solving this problem, its bad that we're all solving it...

The point

I think we should combine forces, to make one canonical parallelisation tool. Personally, I see no point in continuing development of both parallelshell and npm-run-all - we may as well just have one. Having said that, I'm still using parallelshell because it solves particular bugs that I (and @paulpflug) have solved (with great pain) which npm-run-all is yet to solve.

The way I see it - I'm happy (as long as @paulpflug is) to port the fixes we have developed in parallelshell, over to npm-run-all rather than expend effort developing the features I want from npm-run-all inside of parallelshell. I dont want to speak for all other developers (@paulpflug, @kimmobrunfeldt, @luv2code, @royriojas, @spion, and @reggi) - but I for one would certainly like for all of these projects to consolidate under one label. If and when these projects converge, I'd like everyone of the original authors/contributors to be a contributor of the new project - to ensure that it remains well maintained. How does this sound to you?

FYI, I chose to file this issue here - rather than any of the other projects - because I think this project has the best name. By and large they are pretty much all interchangeable (or certainly could be with a bit of development).

Summary

• npm-run-all and parallelshell are very similar (as are other projects)
• We should develop one
• (IMO) All of the maintainers of the other projects should all be core contributors of this project, and deprecate their respective projects.
• Everyone is happy and users have one well developed tool to chose from.

[luv2code]

I'm okay with unifying into this project.

[paulpflug]

👍

[spion]

Yeah, I just figured out #6 :(

I was thinking of writing an alternate implementation that uses the data from package.json but doesn't rely on npm run at all - especially given that npm run seems to be so slow (or is it just slow for me?)

In any case, I agree it would be best if the effort was concentrated, but personally I'm not sure what the direction should be...

[mysticatea]

Wow, really interesting.
I am grateful for your proposal.
I'd like to cooperate to make helpful tool for developers!

@spion I feel npm run is slow too. So I have been thinking that directly-spawn content of npm tasks. But it's very hard....

[royriojas]

👍 happy to merge, help or document :), I guess #6 is an issue that most of us have faced at some point, probably the most important thing a library that runs several things in parallel should have.

[keithamus]

I would definitely say spawning from the package.json, rather than npm run is a good idea. parallelshell has been working on solving the problem of properly killing subshells and the processes running in them, and we have darkguy2008/parallelshell#30 which is going pretty well thanks to @paulpflug. I think we could take this and turn it into a PR for npm-run-all.

Not depending on npm run as an executable also opens us up to the ability to have npm-run-all run arbitrary scripts, not just npm ones - perhaps with a -e flag (e.g. npm-run-all build:* --sequential -e 'echo "done!"').

Also I think it'd be a good idea to see how npm run itself works. Here's the run-script command which mostly builds up which scripts to run (e.g. pre and post prefixes) - L159-167 build up the commands to run, and L166 is what each command runs (i.e. it calls the lifecycle command with the package json, command, working directory and true). lifecycle.js is basically all of the execution magic, including making all of the env vars, building the shell string and actually running the command. When it all comes down to it, all npm run is essentially doing export ${lots_of_env_vars}; sh -c ${npm_script_string}.

I would be interested in @othiym23 and @isaacs opinions about possibly modularising some of this - so we can all re-use the same piece of code for execution logic. They may well also have some insight for us as they wrote most of npm's lifecycle.js 😄 (although most of that code is 5 years old and so I'd understand if they've forgotten).

[kimmobrunfeldt]

I too think that merging these projects makes sense. One comment about the naming though. I chose a general name because concurrently is not tied to npm run scripts, you can also use the tool for any shell command.

[paulpflug]

How about we make 2 packages. the first focusing on sequential and parallel shell execution (name maybe 'parallelshell') and the second focusing on launching npm tasks (grep from project.json or similar)) while depending on the first for execution (name maybe 'npm-run')

I think that would be super transparent for the users and limits the necessary flag count.

[keithamus]

I agree, that sounds like a good idea.

[reggi]

Thanks @keithamus for putting this together. I love this idea. I too think that this should have a more general name and not be npm specific. Just to play devils advicate, I'd love this all to be within one project with special flags for npm specific functionality (e.x. --npm-run or --pkg-glob) , just a thought. I embrace a clean / clear UNIX based design, but keeping in mind that we're all node.js developers and that whatever this thing ends up being is going to be very tied to that ecosystem is important. However two commands as @paulpflug describes also makes sense. 👍. With all that said I'm looking forward to seeing this through and would love to help in any capacity.

[reggi]

Just found these projects that touch on this stuff.

• https://github.com/benoror/better-npm-run
• https://github.com/bahmutov/with-package

[k88hudson]

Hey everyone, awesome to see this coming together 👍 npm scripts for the win!

Personally I see two separate but related goals here;

1. To run tasks in parallel in a way that works on all platforms
2. To provide a better syntax for organizing and running npm scripts or groups of npm scripts

Obviously Windows compatibility is a goal for both 1 and 2, so some code/ideas could be shared, but I would prefer to see them remain separately focused on their respective goals.

If anyone is interested, I know some people on npm/who work on the cli and I would be happy to reach out if they have suggestions, or if any of you want to provide some feedback.

Thanks so much @keithamus for opening this issue and for linking to everyone's projects, you are all awesome 💃 I'm giving another talk on npm scripts at View Source conf in November, and I can tell you more and more people are getting excited about doing front end automation this way thanks to your work and blog posts 😄

[k88hudson]

In fact, I will definitely bring this up with the folks at npm. Thanks also @mysticatea for writing this :)

[keithamus]

Hey @k88hudson! Thanks for the kind words. I definitely think feedback from the peeps at npm would help here.

Aside; @k88hudson I watched your (absolutely excellent) npm talk at Nordic.js which led me to npm-run-all and prompted this issue. Small world 🌏!