[ENH] Add BM25 embedding function to JS

[jairad26]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • This PR adds a JS embedding function to implement BM25, referencing the existing rust bm25 impl. It also has tests to validate the ef, and ensure it matches the rust tests 1:1.
• New functionality
  • ...

Test plan

How are these changes tested?
tested manually + added unit tests

• [ x] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Migration plan

Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?

Observability plan

What is the plan to instrument and monitor this change?

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[jairad26]

• [CHORE] Bump js ai-embeddings versions #5759
• [ENH] Add BM25 embedding function to JS #5756 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Introduce JS‐side BM25 sparse embedding function package

A new TypeScript implementation of BM25 is added and wired into the JS client build. The PR creates a standalone package @chroma-core/chroma-bm25 that mirrors the existing Rust BM25 logic, exposes a schema-validated configuration API, and extends the all meta-package. The change also brings unit-tests, build scripts, stopword lists, and workspace dependency updates.

Key Changes

• Created new package clients/new-js/packages/ai-embeddings/chroma-bm25 with source, tests, build, and typing configs
• Implemented ChromaBm25EmbeddingFunction in src/index.ts (tokenizer, Murmur3 hashing, BM25 weighting, config helpers, registration via registerSparseEmbeddingFunction)
• Added large English stop-word list in src/stopwords.ts and exported DEFAULT_CHROMA_BM25_STOPWORDS
• Provided comprehensive Jest test‐suite src/index.test.ts that parity-checks against Rust reference
• Added JSON schema schemas/embedding_functions/chroma_bm25.json and wired it into common schema-utils.ts
• Updated workspace manifests: pnpm-lock.yaml, all bundle package.json and export barrel, plus new dev/runtime deps (e.g., snowball-stemmers)
• Added build tooling (tsup.config.ts, jest.config.ts, tsconfig.json) for the new package

Affected Areas

• clients/new-js/packages/ai-embeddings/chroma-bm25 (new)
• ai-embeddings/common schema utilities
• ai-embeddings/all meta-package
• pnpm-lock.yaml workspace graph

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

[BestPractice]

This is a great addition! The code is well-structured and the tests are comprehensive.

One minor suggestion to improve the public API of this new package is to export DEFAULT_CHROMA_BM25_STOPWORDS as a readonly array. This prevents consumers of the package from accidentally modifying the default list, which is generally safer for exported constants.

Suggested change

	export const DEFAULT_CHROMA_BM25_STOPWORDS = [...DEFAULT_ENGLISH_STOPWORDS];
	export const DEFAULT_CHROMA_BM25_STOPWORDS: readonly string[] = DEFAULT_ENGLISH_STOPWORDS;

This follows TypeScript best practices for immutable exports. The readonly modifier ensures that consumers cannot use mutating methods like push(), pop(), or direct index assignment, preventing accidental modification of shared constants across different parts of an application.

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Context for Agents

[**BestPractice**]

This is a great addition! The code is well-structured and the tests are comprehensive.

One minor suggestion to improve the public API of this new package is to export `DEFAULT_CHROMA_BM25_STOPWORDS` as a `readonly` array. This prevents consumers of the package from accidentally modifying the default list, which is generally safer for exported constants.

```suggestion
export const DEFAULT_CHROMA_BM25_STOPWORDS: readonly string[] = DEFAULT_ENGLISH_STOPWORDS;
```

This follows TypeScript best practices for immutable exports. The `readonly` modifier ensures that consumers cannot use mutating methods like `push()`, `pop()`, or direct index assignment, preventing accidental modification of shared constants across different parts of an application.

⚡ **Committable suggestion**

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

File: clients/new-js/packages/ai-embeddings/chroma-bm25/src/index.ts
Line: 198

[propel-code-bot]

[BestPractice]

This switch statement uses intentional fall-throughs, which is correct for the Murmur3 algorithm. To improve readability and prevent future maintainers from mistakenly "fixing" this, it's good practice to make the fall-throughs explicit with comments.

Suggested Change

Suggested change

	case 3:
	k1 ^= (key.charCodeAt(i + 2) & 0xff) << 16;
	case 2:
	k1 ^= (key.charCodeAt(i + 1) & 0xff) << 8;
	case 1:
	case 3:
	k1 ^= (key.charCodeAt(i + 2) & 0xff) << 16;
	// falls through
	case 2:
	k1 ^= (key.charCodeAt(i + 1) & 0xff) << 8;
	// falls through
	case 1:

⚡ Committable suggestion

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Context for Agents

[**BestPractice**]

This `switch` statement uses intentional fall-throughs, which is correct for the Murmur3 algorithm. To improve readability and prevent future maintainers from mistakenly "fixing" this, it's good practice to make the fall-throughs explicit with comments.

<details>
<summary>Suggested Change</summary>

```suggestion
            case 3:
                k1 ^= (key.charCodeAt(i + 2) & 0xff) << 16;
            // falls through
            case 2:
                k1 ^= (key.charCodeAt(i + 1) & 0xff) << 8;
            // falls through
            case 1:
```

⚡ **Committable suggestion**

Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

</details>

File: clients/new-js/packages/ai-embeddings/chroma-bm25/src/index.ts
Line: 77