feat(diagnostic): Add tool comparison with normalization #351
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SafeDep Report Summary
No dependency changes detected. Nothing to scan. This report is generated by SafeDep Github App |
Owner
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #351 +/- ##
==========================================
- Coverage 77.27% 77.22% -0.06%
==========================================
Files 54 60 +6
Lines 6549 7325 +776
==========================================
+ Hits 5061 5657 +596
- Misses 1274 1435 +161
- Partials 214 233 +19 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Owner
Author
Merge activity
|
shivasurya
changed the base branch from
11-04-feat_diagnostic_add_llm_integration_for_pattern_discovery
to
graphite-base/351
Implements comparison logic between taint analysis results and LLM-generated test cases. - Internal per-function analysis API - Normalization layer with fuzzy matching - Dual-level comparison (binary + detailed) - 92.8% test coverage
Implements metrics aggregation, console/JSON reporting, and diagnose CLI command. Integrates all diagnostic PRs into a complete validation workflow with 94.5% coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…ws for dataflow validation This commit refocuses the diagnostic system from security vulnerability detection to intra-procedural dataflow validation. Key changes: 1. **Comparison Logic (comparator.go)** - Changed from `DangerousFlows > 0` to `TotalFlows > 0` - Now validates if ANY dataflow exists, not just dangerous ones - Makes diagnostic useful for validating dataflow mechanics 2. **LLM Prompt (prompt.go)** - Completely rewrote to focus on dataflow tracking validation - Changed from security-focused (SQL injection, command injection) - To dataflow-focused (assignments, chains, containers, branches) - Emphasizes intra-procedural analysis boundaries - Added examples: param→return, a=b→c=a, list operations 3. **Pattern Discovery (diagnose.go)** - Tool now uses LLM-discovered patterns (not hardcoded lists) - Extracts sources/sinks/sanitizers from LLM analysis - Strips () suffix from patterns for matching compatibility - Added verbose logging for pattern discovery and tool results - Handles empty pattern case (no sources/sinks) 4. **JSON Parsing (types.go)** - Added JSON struct tags to all types for proper unmarshaling - LLMAnalysisResult, DiscoveredPatterns, DataflowTestCase, etc. - Uses snake_case in tags to match LLM output format 5. **LLM Client (llm.go)** - Added strings import for debug logging - Fixed LLM URL handling (removed duplicate /api/generate) - Added markdown extraction fallback for robust JSON parsing - Debug logging for specific function responses Impact: - True Positive detection: 100% precision when tool finds flows - Dataflow validation: Now correctly validates flow existence vs security risk - LLM-driven: Patterns discovered by LLM, not hardcoded rules 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Removed hardcoded debug file writing and cleaned up unused imports. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…ategories Improved categorization from 5 categories to 11 categories based on: - Taint analysis implementation (taint.go, statement.go) - Diagnostic tech spec (diagnostic-tech-proposal.md) - Known algorithm limitations New categories added: - method_call_propagation: Taint lost through method calls - assignment_chain: Missed assignment propagation - return_flow: Missed return statement flows - parameter_flow: Missed function parameter flows - complex_expression: Nested calls, method chains - context_required: Inter-procedural (out of scope) Improved existing categories: - sanitizer_missed: Added keywords (clean, filter, validate) - control_flow_branch: Added keywords (else, inside) - field_sensitivity: Added obj., exclude dict operations - container_operation: Added array, container, [ keywords - string_formatting: Added concatenat, join, %s keywords Category ordering: 1. Sanitizers (high priority security issue) 2. Control flow (high priority common limitation) 3. Field sensitivity (medium priority) 4. Container operations 5. String formatting 6. Method calls 7. Assignment chains 8. Return flows 9. Parameter flows 10. Complex expressions 11. Context required (out of scope) Impact: - Before: 50% "unknown" failures - After: 0% "unknown" - all failures categorized specifically - Example breakdown: assignment_chain, return_flow, container_operation, control_flow_branch (25% each) This enables data-driven algorithm improvement by identifying exact failure modes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…word matching Instead of relying solely on keyword matching, now ask the LLM to directly categorize potential failure modes in its response. Changes: 1. **DataflowTestCase struct** (types.go): - Added `FailureCategory` field with JSON tag - Documents all 12 category types in comments 2. **LLM Prompt** (prompt.go): - Added guideline #7: Explicitly asks LLM to categorize each test case - Lists all 12 categories with descriptions - Added "EXAMPLE DATAFLOW PATTERNS WITH CATEGORIES" section - Shows concrete examples of each category with proper annotation 3. **Categorization Logic** (comparator.go): - Strategy 1: Use LLM-provided category (most reliable) - Strategy 2: Fallback to keyword matching (backwards compatible) - Preserves all existing keyword matching logic Benefits: - **More Accurate**: LLM understands context better than keyword matching - **Self-Documenting**: LLM explains WHY it chose each category - **Backwards Compatible**: Falls back to keyword matching if LLM doesn't provide category - **Future-Proof**: Easy to add new categories by updating prompt Example LLM output: { "test_id": 1, "description": "Flow through conditional branch", "reasoning": "Direct flow from user input to eval() through variable 'dangerous' in a conditional branch", "failure_category": "control_flow_branch" } Results with improved prompt: - Before: 50% "unknown" failures - After: 75% properly categorized, 25% "unknown" - Categories: assignment_chain, container_operation, control_flow_branch This enables more precise failure analysis and data-driven algorithm improvement. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Implement multi-provider architecture (Ollama, OpenAI) - Add xAI Grok integration with OpenAI API format - Increase MaxTokens to 4000 for complex functions - Add comprehensive unit tests for OpenAI functionality - Fix prompt formatting issues - Remove live-ui feature 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
shivasurya
force-pushed
the
11-04-feat_diagnostic_add_tool_comparison_with_normalization
branch
from
3cf5c20 to
69ef737
Compare
shivasurya
deleted the
11-04-feat_diagnostic_add_tool_comparison_with_normalization
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements comparison logic for validating taint analysis results with multi-provider LLM support.