fix(evals): update save_memory evals and simplify tool description

[NTaylorMullen]

Summary

This PR fixes the failing save_memory behavioral evaluations by aligning them with the tool's core mandate: storing global user context, not workspace-local facts.

Details

The original evaluations were 'testing for failure' by asking the agent to store project-specific details (DB schemas, entry points) in long-term global memory. This caused significant regressions, especially with Gemini 3 Pro, which correctly identified these as project-local facts and refused to call the tool.

Key changes:

• Eval Updates: Converted project-specific memory tests into negative evaluations that assert save_memory is NOT called.
• Term Standardization: Replaced 'project' with 'workspace' to clarify that context local to the current environment (including sub-folders) should remain local.
• Tool Description: Simplified and strengthened the save_memory description to explicitly forbid workspace-local facts while minimizing token usage.
• Environment Stability: Added standard file tools to evaluations to prevent hallucinations when models try to verify workspace context.

Verified stable passage across both Gemini 2.5 Pro and Gemini 3 Pro.

Related Issues

Part of the ongoing behavioral evaluation stabilization effort.

[gemini-code-assist]

Summary of Changes

Hello @NTaylorMullen, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses issues with the save_memory tool's behavioral evaluations by refining its intended use. The core change ensures that the tool is exclusively used for storing global user context, such as personal preferences, and explicitly prevents its invocation for workspace-specific details like file paths or project commands. This clarification is reinforced through updated evaluation tests, standardized terminology, and a more precise tool description, leading to more stable and accurate model behavior.

Highlights

• Evaluation Updates: Refactored save_memory evaluations to correctly test that the tool is not called for workspace-specific information, addressing previous failures where models incorrectly attempted to save local facts globally.
• Tool Description Clarity: The save_memory tool's description has been significantly simplified and strengthened to explicitly state it's for global user context only, strictly forbidding the storage of workspace-local facts.
• Terminology Alignment: Standardized the use of 'workspace' instead of 'project' in evaluations to better reflect the scope of local context and avoid ambiguity.
• Evaluation Environment Enhancement: Enhanced evaluation settings by including standard file tools (list_directory, read_file, run_shell_command) to prevent model hallucinations and provide a more stable testing environment.

Changelog

• evals/save_memory.eval.ts
  • Changed several evalTest cases from expecting save_memory to be called for "project's database schema location", "specific project test command", and "project's main entry point" to expecting it not to be called for "workspace's database schema location", "workspace build artifact location", and "workspace's main entry point".
  • Updated the assert logic in these negative tests to check rig.readToolLogs().some(...) and expect(...).toBe(false).
  • Added list_directory, read_file, run_shell_command to the tools.core settings for these evaluations.
  • Replaced the "Agent remembers project's main entry point" test with "Agent remembers user's birthday", asserting save_memory is called for this global personal fact.
  • Removed checkModelOutputContent calls from the negative evaluations.
• packages/core/src/tools/memoryTool.ts
  • Removed the explicit FunctionDeclaration import and memoryToolSchemaData constant, inlining the schema definition.
  • Rewrote the memoryToolDescription to be much more concise and emphatic about its purpose: "Saves concise global user context (preferences, facts) for use across ALL workspaces." It now includes a "CRITICAL: GLOBAL CONTEXT ONLY" section explicitly forbidding workspace-specific context.
  • Appended examples like "Always lint after building" to the description passed to the super constructor.

Activity

• No specific activity (comments, reviews, or progress updates) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Size Change: -1.03 kB (0%)

Total Size: 23.9 MB

Filename	Size	Change
./bundle/gemini.js	23.8 MB	-1.03 kB (0%)

ℹ️ View Unchanged

Filename	Size
./bundle/sandbox-macos-permissive-closed.sb	1.03 kB
./bundle/sandbox-macos-permissive-open.sb	890 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB
./bundle/sandbox-macos-restrictive-closed.sb	3.29 kB
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB

_{compressed-size-action}

[gemini-code-assist]

Code Review

This pull request effectively addresses the failing save_memory evaluations by correctly aligning them with the tool's intended purpose of storing global, not workspace-local, context. The simplification and strengthening of the save_memory tool description is a significant improvement that makes the instructions for the model much clearer and more forceful. The changes in the evaluation tests are also well-aligned with this new, stricter definition of the tool's function. The suggestion to further improve the formatting of the tool description prompt is valid and has been retained.