🐛 Dependency-Pinning: only score detected ecosystems

[gabibguti]

What kind of change does this PR introduce?

This is a bug fix. Pinned-Dependencies score used to score a 10 for ecossystems not used by the project. Now, it scores -1 when the ecossystem is not used, and this score is not included in the final score for Pinned-Dependencies.

• PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

For example, if project A uses only Python and has only Python or pip dependencies, the project would score 10 for Dockerfiles, npm, go and other ecossystems dependencies it does not use.

What is the new behavior (if this is a feature change)?**

In this PR, if no dependencies are found for an ecossystem, this ecossystem score is not included in the final score.

• Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

Fixes #3254
Fixes #2679

Special notes for your reviewer

This change will likely decrease the Pinned-Dependencies score for all projects.

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

Pinned-Dependencies score includes only used ecossystems (npm, go, pip, etc.)

[codecov]

Codecov Report

Merging #3436 (2e3e9a5) into main (8752511) will increase coverage by 0.27%.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3436      +/-   ##
==========================================
+ Coverage   64.40%   64.68%   +0.27%     
==========================================
  Files         188      188              
  Lines       13425    13412      -13     
==========================================
+ Hits         8647     8676      +29     
+ Misses       4304     4273      -31     
+ Partials      474      463      -11     

[spencerschrock]

Left a partial review, first thoughts:

1. Touching asPointer in unrelated files makes the change bigger than it needs to be
2. The scoring approach is different than discussed.

[gabibguti]

Ok, ready to take a look again!

[spencerschrock]

Looks good, just some small optional things.

[spencerschrock]

I plan on running a quick A/B of scoring changes to make sure everything looks good, and following up with my earlier request for feedback from the other maintainers.

[spencerschrock]

I plan on running a quick A/B of scoring changes to make sure everything looks good

In general, I think this is working as intended. Across a random selection of 10 popular repos, we see 6 of them drop to a score of 0 as none of their dependencies are pinned:

score: # of repos before -> after
-1: 0 -> 0
 0: 0 -> 6
 1: 1 -> 0
 2: 1 -> 1
 3: 0 -> 0
 4: 1 -> 1
 5: 1 -> 0
 6: 0 -> 0
 7: 2 -> 0
 8: 2 -> 0
 9: 0 -> 0
10: 2 -> 2

For the repos that still score non-zero, should there be any Info logging for ecosystems we do find? For example

go run main.go --repo github.com/apache/commons-codec --checks Pinned-Dependencies --format json --show-details | jq
{
  "date": "2023-09-18T13:41:25-07:00",
  "repo": {
    "name": "github.com/apache/commons-codec",
    "commit": "585497f09b026f6602daf986723a554e051bdfe6"
  },
  "scorecard": {
    "version": "",
    "commit": "unknown"
  },
  "score": 10,
  "checks": [
    {
      "details": null,
      "score": 10,
      "reason": "all dependencies are pinned",
      "name": "Pinned-Dependencies",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies",
        "short": "Determines if the project has declared and pinned the dependencies of its build process."
      }
    }
  ],
  "metadata": null
}

and following up with my earlier request for feedback from the other maintainers.

There was no objection from other maintainers.

[gabibguti]

For the repos that still score non-zero, should there be any Info logging for ecosystems we do find? For example

Yes. I will add an Info log for "2 out of 10 go install dependencies pinned" for ecosystems we do find. So we have the information if all dependencies of an ecossystem are pinned, and how much work was that. I will address this tomorrow.

[spencerschrock]

I think this looks good thanks.

go run main.go --repo ossf/scorecard  --checks Pinned-Dependencies --format json --show-details | jq
{
  "date": "2023-09-24T20:13:47-07:00",
  "repo": {
    "name": "github.com/ossf/scorecard",
    "commit": "5a5a6561d65067678990e69dc3d16a711c98c048"
  },
  "scorecard": {
    "version": "",
    "commit": "unknown"
  },
  "score": 9,
  "checks": [
    {
      "details": [
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/goreleaser.yaml:89: update your workflow using https://app.stepsecurity.io/secureworkflow/ossf/scorecard/goreleaser.yaml/main?enable=pin",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/slsa-goreleaser.yml:47: update your workflow using https://app.stepsecurity.io/secureworkflow/ossf/scorecard/slsa-goreleaser.yml/main?enable=pin",
        "Info: 51 out of 51 GitHub-owned GitHubAction dependencies pinned",
        "Info: 46 out of 48 third-party GitHubAction dependencies pinned",
        "Info: 25 out of 25 containerImage dependencies pinned",
        "Info: 1 out of 1 goCommand dependencies pinned"
      ],

[spencerschrock]

By the way, I did the A/B comparison, and I think these info statements help. Repos in general had a score drop as expected.

More repos had a -1 score than I anticipated but mainly from our lack of detecting certain ecosystems (e.g. yarn, which would be handled in other PRs). I'll get to merging this today, but trying to get some higher priority fixes in first.

[gabibguti]

Good to know, thanks for the feedback!