feat(plan): create a metric for execution of plans generated in plan mode#18236
feat(plan): create a metric for execution of plans generated in plan mode#18236
Conversation
Summary of ChangesHello @Adib234, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enhances the application's telemetry capabilities by adding a new metric to quantify how often generated plans are actually executed. This will provide valuable insights into user behavior within 'Plan Mode' and the effectiveness of the plan generation process, allowing for data-driven improvements. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new telemetry metric, PLAN_EXECUTION_COUNT, to track the execution of plans generated in Plan Mode. The changes are well-structured and consistently follow existing patterns for adding new telemetry events and metrics. I've identified one critical issue in the implementation of the PlanExecutionEvent class where it fails to initialize required properties, leading to data loss in telemetry. This also includes ensuring telemetry keys accurately reflect semantic meaning, as per repository guidelines. My review includes a code suggestion to fix this.
I am having trouble creating individual review comments. Click here to see my feedback.
packages/core/src/telemetry/types.ts (1927-1939)
The PlanExecutionEvent class doesn't initialize the event.name and event.timestamp properties required by the BaseTelemetryEvent interface. This leads to missing timestamp information in the telemetry data for OpenTelemetry. Additionally, the attribute key for the event name should be 'event.name' for consistency with OpenTelemetry semantic conventions and other events in the codebase, rather than event_name.
This change initializes the properties in the constructor and corrects the attribute in toOpenTelemetryAttributes to ensure complete and consistent event logging.
constructor(approvalMode: ApprovalMode) {
this['event.name'] = this.eventName;
this['event.timestamp'] = new Date().toISOString();
this.approval_mode = approvalMode;
}
'event.name': string;
'event.timestamp': string;
toOpenTelemetryAttributes(config: Config): LogAttributes {
return {
...getCommonAttributes(config),
'event.name': this.eventName,
'event.timestamp': this['event.timestamp'],
approval_mode: this.approval_mode,
};
}References
- When logging events, ensure that the keys used for telemetry data accurately reflect the semantic meaning of the data being logged to avoid misinterpretation. Introduce new, more specific keys when existing ones are semantically confusing in a given context.
Adib234
force-pushed
the
adibakm/execution-metric-plan-mode
branch
from
ab8ae74 to
122cbdd
Compare
|
Size Change: +2.69 kB (+0.01%) Total Size: 23.7 MB
ℹ️ View Unchanged
|
gemini-cli
bot
added
area/core
Adib234
changed the title
jerop
left a comment
jerop
left a comment
There was a problem hiding this comment.
please update the data reference: https://github.com/google-gemini/gemini-cli/blob/main/docs/cli/telemetry.md#logs-and-metrics
3 participants
Summary
This PR instruments a new telemetry metric,
PLAN_EXECUTION_COUNT, to track the execution of "Plan Mode". Specifically, it records an event whenever a user successfully exits Plan Mode via the ExitPlanMode tool with an approved plan, capturing the subsequent approval mode (e.g., default, autoEdit). This data will help quantify how often generated plans lead to execution.Details
Related Issues
Closes #16862
How to Validate
Enable plan mode and approve a plan, once the exit plan mode tool gets triggered a log should be emitted. I've verified that the log appears here https://cloudlogging.app.goo.gl/5omQxcqXPFzhFSih9
Pre-Merge Checklist