fix: updating a few metrics to take into account computation at the agent level#93
fix: updating a few metrics to take into account computation at the agent level#93
Conversation
…gent level Signed-off-by: Jacques Samain <jsamain@cisco.com>
Signed-off-by: Jacques Samain <jsamain@cisco.com>
k-payumo
left a comment
There was a problem hiding this comment.
The processor.py should be updated imo to iterate through the relevant agent_names, then apply these metrics to spans belonging to that respective agent. Similar to the changes that are proposed in the deepeval adapter, but should be abstracted higher level to how the MCE iterates through entities.
We may need to spend to time to determine which metrics we should consider as a complete E2E evaluation of as MAS, vs independent evaluations of individual agents.
k-payumo
left a comment
There was a problem hiding this comment.
Minor comment on redacting of images. Doesn't need to be removed here but just a heads up, will have a more holistic filter that removes unused metadata fields such as
PAYLOAD_FILTER_KEYS: Set[str] = {
"embedding",
"image_url",
"url",
"class",
"module",
"config_specs",
"lc_attributes",
"lc_secrets",
"model_computed_fields",
"model_config",
"optional_variables",
"input_types",
"cache",
"verbose",
"callbacks",
"tags",
}
Signed-off-by: Jacques Samain <jsamain@cisco.com>
Description
This PR updates a few of the metrics to compute them at the agent level.
Type of Change
Checklist