Support grouping for cases in metacoder config

[valerie-autumn-skye]

Requested by @justaddcoffee on Berkeley BOP Slack #contextualizer channel (Sept 4):

Maybe we should label each test with one of (say) 5 types so we can break out plots by how they performed for (say):

• Bibliographic metadata - title, authors, affiliation, doi, publisher
• Publication text extraction - getting stuff from specific sections
• Figure / Table extraction - getting figures and tables
• Supplementary material - getting supplemental data
• Retraction status - was paper extracted, etc.

@cmungall and @ct-parker supported the idea.

@justaddcoffee committed changes to mcp_literature_eval config including a case_category for each case.

[github-actions]

Issue Summary & Analysis

This feature request addresses the need for case categorization in metacoder's evaluation framework. The request originated from @justaddcoffee in the Berkeley BOP Slack #contextualizer channel and has support from @cmungall and @ct-parker.

Current Status

• Issue Type: Enhancement/Feature Request
• Assigned to: @ct-parker
• Labels: enhancement
• Status: Open, recently created (Sept 4, 2025)

Core Requirements

The proposed system would categorize evaluation test cases into 5 main types for improved analysis and reporting:

1. Bibliographic metadata - title, authors, affiliation, doi, publisher
2. Publication text extraction - getting content from specific sections
3. Figure/Table extraction - extracting figures and tables
4. Supplementary material - accessing supplemental data
5. Retraction status - determining if paper was retracted

Technical Implementation Context

Based on my analysis of the codebase:

Current Configuration Structure (src/metacoder/evals/eval_model.py:6):

• The EvalCase class already has a tags field for categorization
• Cases support additional_metadata for custom fields
• No case_category field currently exists

Proposed Enhancement Options:

1. Option A: Extend existing tags field usage

   cases:
     - name: "PMC123_title"
       tags: ["bibliographic_metadata"]
       metrics: [CorrectnessMetric]

2. Option B: Add dedicated case_category field

   cases:
     - name: "PMC123_title"
       case_category: "bibliographic_metadata"
       metrics: [CorrectnessMetric]

3. Option C: Use additional_metadata approach

   cases:
     - name: "PMC123_title"
       additional_metadata:
         category: "bibliographic_metadata"

Related Infrastructure

Potential Impact Areas:

• Evaluation Runner (src/metacoder/evals/runner.py) - needs category-aware reporting
• Results Analysis - plotting and metrics breakdown by category
• Configuration Validation - ensuring valid category values

Literature Context

This categorization aligns with standard bibliometric and scientometric analysis frameworks:

• Metadata Quality Assessment: Well-established field focusing on bibliographic data accuracy
• Content Extraction Evaluation: Common in digital library and text mining research
• Supplementary Data Access: Increasingly important for research reproducibility
• Retraction Detection: Critical for research integrity and evidence synthesis

Recommendations

1. Implementation Priority: High - this enables more granular evaluation insights
2. Approach: Recommend Option B (dedicated field) for clarity and explicit validation
3. Validation: Add enum validation for the 5 category types
4. Backwards Compatibility: Make field optional to support existing configs
5. Documentation: Update config specification with category guidelines

Connection to Broader Metacoder Goals

This enhancement directly supports metacoder's mission as a unified evaluation framework for AI coding assistants by:

• Enabling domain-specific performance analysis
• Supporting targeted benchmarking for literature processing tasks
• Facilitating comparative studies across different task types
• Improving evaluation reporting and visualization capabilities

The categorization will be particularly valuable for scientific literature processing evaluations, which appear to be a key use case based on the issue context and existing MCP integrations (artl, pubmed, etc.).

---

Note: @justaddcoffee has already begun implementation work on the mcp_literature_eval config with case_category field additions.

[valerie-autumn-skye]

I'm going to change case_category to group in my PR for this issue. So, as with the name key in the config becoming case_name in the results, group will become case_group. This should be more intuitive for the user.