[Feature Request]: Token Count

[Jarvx]

Feature Description

I really enjoy using llama, but is there a better way to track how many tokens we have used in agent workflow?
So far I cannot find any doc about this and the count for cached token is also limited.

I'd say, it's definitely not difficult to implement since each LLM raw response contains this information.

Reason

No response

Value of Feature

No response

[logan-markewich]

You can make your own pretty easily
https://colab.research.google.com/drive/1QV01kCEncYZ0Ym6o6reHPcffizSVxsQg?usp=sharing

[Jarvx]

You can make your own pretty easily https://colab.research.google.com/drive/1QV01kCEncYZ0Ym6o6reHPcffizSVxsQg?usp=sharing

Thanks, I am using this way as the makeshift for my task. But it does not contain cached token count?

[logan-markewich]

@Jarvx you can update the code to include it -- you can see that it's just accessing the raw openai response, so all the info there (i made this a while back as an example for someone else)

[Kowalskiexe]

Thanks @logan-markewich

I bump this feature request. It'd be useful to have it out-of-the-box

[logan-markewich]

The issue is, token counts reported by LLMs vary greatly across providers, especially for caching (i.e. openai just reports "cached tokens", anthropic reports "read" and "write" cached tokens). So now, as maintainers, we end up with endless if/else statements and spaghetti code trying to support everyone, when really, its quite simple to support yourself.

The above example I linked really is the easiest way to do this. Imo I need to put that example in the docs :)

There is an aging builtin, but tbh it needs to be migrated to the newer instrumentation system (and even then, might not be worth it, due to the reasons listed above).
https://docs.llamaindex.ai/en/stable/examples/observability/TokenCountingHandler/

Maybe the instrumentation example I linked above can be integrated in some way thats still flexible (i.e. the user needs to provide some callback to parse the token counts?)

[Jarvx]

The issue is, token counts reported by LLMs vary greatly across providers, especially for caching (i.e. openai just reports "cached tokens", anthropic reports "read" and "write" cached tokens). So now, as maintainers, we end up with endless if/else statements and spaghetti code trying to support everyone, when really, its quite simple to support yourself.

The above example I linked really is the easiest way to do this. Imo I need to put that example in the docs :)

There is an aging builtin, but tbh it needs to be migrated to the newer instrumentation system (and even then, might not be worth it, due to the reasons listed above). https://docs.llamaindex.ai/en/stable/examples/observability/TokenCountingHandler/

Maybe the instrumentation example I linked above can be integrated in some way thats still flexible (i.e. the user needs to provide some callback to parse the token counts?)

Thanks. I agree, such fragementation of these LLM vendors are disappointing. Can you make your example to the official docs? I think the example will help more people.

It's really a nice project but I cannot find useful information quickly.

[dosubot]

Hi, @Jarvx. I'm Dosu, and I'm helping the LlamaIndex team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You requested a feature to track token usage in the llama agent workflow, highlighting insufficient documentation.
• Logan-markewich provided a workaround via a Google Colab link, which you are using but noted it lacks cached token count details.
• Logan-markewich explained the complexity due to varying token count reports from different LLM providers.
• Kowalskiexe supported your feature request, and you suggested including Logan-markewich's example in the official documentation.

Next Steps:

• Please let me know if this issue is still relevant to the latest version of the LlamaIndex repository by commenting here.
• If there is no further activity, I will automatically close this issue in 7 days.

Thank you for your understanding and contribution!