Skip to content

fix: updating a few metrics to take into account computation at the agent level#93

Open
Castorche wants to merge 3 commits intomainfrom
fix/compute-metrics-for-agents
Open

fix: updating a few metrics to take into account computation at the agent level#93
Castorche wants to merge 3 commits intomainfrom
fix/compute-metrics-for-agents

Conversation

@Castorche
Copy link
Contributor

@Castorche Castorche commented Oct 23, 2025

Description

This PR updates a few of the metrics to compute them at the agent level.

Type of Change

  • Bugfix
  • New Feature
  • Breaking Change
  • Refactor
  • Documentation
  • Other (please describe)

Checklist

  • I have read the contributing guidelines
  • Existing issues have been referenced (where applicable)
  • I have verified this change is not present in other open pull requests
  • Functionality is documented
  • All code style checks pass
  • New code contribution is covered by automated tests
  • All new and existing tests pass

…gent level

Signed-off-by: Jacques Samain <jsamain@cisco.com>
@Castorche Castorche requested a review from a team as a code owner October 23, 2025 09:51
Signed-off-by: Jacques Samain <jsamain@cisco.com>
Copy link
Contributor

@k-payumo k-payumo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The processor.py should be updated imo to iterate through the relevant agent_names, then apply these metrics to spans belonging to that respective agent. Similar to the changes that are proposed in the deepeval adapter, but should be abstracted higher level to how the MCE iterates through entities.

We may need to spend to time to determine which metrics we should consider as a complete E2E evaluation of as MAS, vs independent evaluations of individual agents.

Copy link
Contributor

@k-payumo k-payumo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comment on redacting of images. Doesn't need to be removed here but just a heads up, will have a more holistic filter that removes unused metadata fields such as

PAYLOAD_FILTER_KEYS: Set[str] = {
"embedding",
"image_url",
"url",
"class",
"module",
"config_specs",
"lc_attributes",
"lc_secrets",
"model_computed_fields",
"model_config",
"optional_variables",
"input_types",
"cache",
"verbose",
"callbacks",
"tags",
}

Signed-off-by: Jacques Samain <jsamain@cisco.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants