Skip to content

test(evals): mark all save_memory evals as USUALLY_PASSES due to unreliability #18786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jerop merged 1 commit into from
Feb 11, 2026
Merged

test(evals): mark all save_memory evals as USUALLY_PASSES due to unreliability #18786

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions evals/save_memory.eval.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import {
describe('save_memory', () => {
const TEST_PREFIX = 'Save memory test: ';
const rememberingFavoriteColor = "Agent remembers user's favorite color";
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: rememberingFavoriteColor,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand All @@ -36,7 +36,7 @@ describe('save_memory', () => {
},
});
const rememberingCommandRestrictions = 'Agent remembers command restrictions';
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: rememberingCommandRestrictions,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand All @@ -57,7 +57,7 @@ describe('save_memory', () => {
});

const rememberingWorkflow = 'Agent remembers workflow preferences';
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: rememberingWorkflow,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand All @@ -79,7 +79,7 @@ describe('save_memory', () => {

const ignoringTemporaryInformation =
'Agent ignores temporary conversation details';
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: ignoringTemporaryInformation,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand All @@ -104,7 +104,7 @@ describe('save_memory', () => {
});

const rememberingPetName = "Agent remembers user's pet's name";
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: rememberingPetName,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand All @@ -125,7 +125,7 @@ describe('save_memory', () => {
});

const rememberingCommandAlias = 'Agent remembers custom command aliases';
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: rememberingCommandAlias,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand All @@ -147,7 +147,7 @@ describe('save_memory', () => {

const ignoringDbSchemaLocation =
"Agent ignores workspace's database schema location";
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: ignoringDbSchemaLocation,
params: {
settings: {
Expand Down Expand Up @@ -178,7 +178,7 @@ describe('save_memory', () => {

const rememberingCodingStyle =
"Agent remembers user's coding style preference";
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: rememberingCodingStyle,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand All @@ -200,7 +200,7 @@ describe('save_memory', () => {

const ignoringBuildArtifactLocation =
'Agent ignores workspace build artifact location';
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: ignoringBuildArtifactLocation,
params: {
settings: {
Expand Down Expand Up @@ -230,7 +230,7 @@ describe('save_memory', () => {
});

const ignoringMainEntryPoint = "Agent ignores workspace's main entry point";
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: ignoringMainEntryPoint,
params: {
settings: {
Expand Down Expand Up @@ -260,7 +260,7 @@ describe('save_memory', () => {
});

const rememberingBirthday = "Agent remembers user's birthday";
evalTest('ALWAYS_PASSES', {
evalTest('USUALLY_PASSES', {
name: rememberingBirthday,
params: {
settings: { tools: { core: ['save_memory'] } },
Expand Down
Loading