[CI] Avoid installing unnecessary dependencies

[dyladan]

          nit: This `npm install semver` installs all dependencies anyway (npm issue: https://github.com/npm/cli/issues/3023).  This could consider just running `npm ci` OR also installing semver globally (untested): `npm install -g semver`. The latter, if it works, would avoid installing ~800MB of deps.

Originally posted by @trentm in #4238 (comment)

I'm working on it