Warn when individual files have an especially high token count

[foresturquhart]

Implemented by Copilot in Agent mode. Resolves #11.

[Copilot]

Pull Request Overview

This PR adds functionality to warn when individual files exceed a configurable token-count threshold during serialization.

• Introduces CountFileTokens for per-file token analysis.
• Extends the Serializer interface and implementations (XML, text, markdown) to accept and act on a highTokenCountThreshold.
• Adds CLI flag and config support (HighTokenCountThreshold) and updates the runner to pass the new threshold through.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
internal/tokens/counter.go	Added CountFileTokens helper for per-file token counting.
internal/serializer/xml.go	Extended Serialize and readAndNormalizeContent to warn on high tokens.
internal/serializer/text.go	Same threshold logic added to plain-text serializer.
internal/serializer/markdown.go	Same threshold logic added to markdown serializer.
internal/serializer/serializer.go	Updated Serializer interface signature with new threshold param.
internal/core/runner.go	Passed HighTokenCountThreshold into Serialize call.
internal/config/defaults.go	Added DefaultHighTokenCountThreshold.
internal/config/config.go	Wire up HighTokenCountThreshold in Config and CLI parsing.
cmd/grimoire/main.go	Introduced --high-token-threshold CLI flag.
go.mod	Added new dependencies (tiktoken-go, updated other modules).

Comments suppressed due to low confidence (1)

internal/tokens/counter.go:88

• Missing unit tests for CountFileTokens to validate its behavior (including empty content and error paths). Consider adding relevant tests.

func CountFileTokens(filePath, content string) (int, error) {

[Copilot]

[nitpick] The token count warning logic is duplicated across serializer implementations (xml, text, markdown). Consider extracting this into a common helper to reduce duplication.

[Copilot]

CountTokens creates a new encoder for each invocation, which could be costly when processing many files. Consider reusing a shared encoder instance to improve performance.