Skip to content

Audit and standardize exception handling in instructor library#1897

Merged
jxnl merged 4 commits intomainfrom
claude/audit-exception-handling-011CUr4StJP53nzCaifM9KtE
Nov 6, 2025
Merged

Audit and standardize exception handling in instructor library#1897
jxnl merged 4 commits intomainfrom
claude/audit-exception-handling-011CUr4StJP53nzCaifM9KtE

Conversation

@jxnl
Copy link
Copy Markdown
Collaborator

@jxnl jxnl commented Nov 6, 2025

Important

Enhances exception handling in the instructor library by adding new exception types and ensuring backward compatibility, with tests to verify these changes.

  • Behavior:
    • Introduces ResponseParsingError, MultimodalError, and AsyncValidationError in exceptions.py for specific error scenarios.
    • Ensures backward compatibility by inheriting from both ValueError and InstructorError.
  • Exception Handling:
    • Updates function_calls.py and multimodal.py to raise new exceptions for specific error cases.
    • Adds detailed docstrings and attributes to new exceptions for better debugging.
  • Testing:
    • Adds test_exception_backwards_compat.py to verify backward compatibility and correct inheritance of new exceptions.
    • Tests ensure new exceptions can be caught as ValueError and InstructorError.

This description was created by Ellipsis for 35ae7e8. You can customize this summary. It will automatically update as commits are pushed.

This commit significantly enhances the exception handling system across
the instructor library with improved documentation, context, and consistency.

## Exception Class Enhancements

- Added comprehensive docstrings to all exception classes with:
  - Detailed descriptions of when exceptions are raised
  - Common causes and solutions
  - Practical usage examples
  - Attribute documentation

- Enhanced InstructorError base class with detailed usage patterns
- Documented FailedAttempt, IncompleteOutputException, InstructorRetryException
- Added usage guidance for ValidationError, ProviderError, ConfigurationError,
  ModeError, ClientError, and AsyncValidationError

## New Exception Types

- **ResponseParsingError**: For LLM response parsing failures
  - Includes mode and raw_response attributes for better debugging
  - Used when response format is unexpected or malformed

- **MultimodalError**: For multimodal content processing errors
  - Includes content_type (image/audio/pdf) and file_path attributes
  - Provides clear context for file and format issues

## Replaced Generic Exceptions

### processing/function_calls.py
- Replaced ValueError with ResponseParsingError for:
  - Missing completion choices
  - Provider errors
  - Cohere response parsing failures
  - Bedrock response parsing failures
  - Gemini JSON extraction failures
  - Response tool call failures

- Replaced ValueError with ConfigurationError for:
  - Invalid patch modes
  - Multiple tool call restrictions
  - Invalid response_model types

### processing/multimodal.py
- Replaced ValueError with MultimodalError for:
  - File not found errors (with file_path context)
  - Unsupported image/audio/PDF formats (with content_type)
  - File reading failures

### processing/response.py
- Replaced ValueError with ConfigurationError for:
  - Invalid/unsupported modes (with list of available modes)

## Improved Error Context

All new exceptions include contextual information:
- mode: The mode being used when the error occurred
- raw_response: The raw LLM response for debugging
- content_type: The type of multimodal content (image/audio/pdf)
- file_path: The file path when file operations fail

This addresses the need for better monitoring and diagnostics by providing
access to raw responses and additional context in validation/parsing errors.

## Backwards Compatibility

- All existing exception types maintained
- Deprecated import path (instructor.exceptions) still works
- All exception hierarchy preserved
- New exceptions exported in both core and main modules

## Testing

- All existing tests pass (43/43 tests passed)
- Exception hierarchy tests verify proper inheritance
- FailedAttempt tracking tests verify context preservation

BREAKING CHANGES: None - all changes are backwards compatible
This commit ensures full backwards compatibility for the new exception
types and updates documentation to reflect the enhanced exception system.

## Backwards Compatibility Enhancements

### ResponseParsingError
- Now inherits from both ValueError and InstructorError
- Existing code catching ValueError will still work
- Provides additional context (mode, raw_response) when needed

### MultimodalError
- Now inherits from both ValueError and InstructorError
- Existing code catching ValueError will still work
- Provides additional context (content_type, file_path) when needed

This follows the same pattern as AsyncValidationError which already
inherits from both ValueError and InstructorError.

## Testing

Added comprehensive backwards compatibility tests:
- test_response_parsing_error_is_value_error
- test_multimodal_error_is_value_error
- test_async_validation_error_is_value_error
- test_exception_inheritance_chain
- test_mixed_exception_catching
- test_exception_attributes_preserved

All 49 tests pass (26 exception tests + 6 compat tests + 17 other tests)

## Documentation Updates

Updated /docs/concepts/error_handling.md with:

1. New exception type documentation:
   - ResponseParsingError with mode and raw_response attributes
   - MultimodalError with content_type and file_path attributes
   - AsyncValidationError documentation

2. Backwards Compatibility section:
   - Shows how old code (catching ValueError) still works
   - Shows how new code can access additional context
   - Clear migration path for users

3. Diagnostic Context section:
   - Examples of logging ResponseParsingError with full context
   - Examples of logging MultimodalError with file information
   - Examples of logging InstructorRetryException with attempt history
   - Demonstrates monitoring and debugging capabilities

## Breaking Changes

NONE - All changes are fully backwards compatible:
- Old code catching ValueError continues to work
- New exceptions provide additional context when needed
- Exception hierarchy preserved
- Existing exception types unchanged

This ensures a smooth upgrade path for users moving to version 1.14.
This summary document provides a complete overview of all exception
handling improvements made for the 1.14 release, including:

- All changes made to exception classes
- Backwards compatibility strategy and verification
- Test results (49/49 tests passing)
- Benefits for users (monitoring, debugging, context)
- Migration guide (no changes required)
- File-by-file change details

This serves as both documentation and a reference for the changes.
@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
instructor 35ae7e8 Nov 06 2025, 05:42 AM

@github-actions github-actions Bot added documentation Improvements or additions to documentation enhancement New feature or request python Pull requests that update python code size:M This PR changes 30-99 lines, ignoring generated files. labels Nov 6, 2025
Copy link
Copy Markdown
Contributor

@ellipsis-dev ellipsis-dev Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

Looks good to me! 👍

Reviewed everything up to 35ae7e8 in 1 minute and 2 seconds. Click for details.
  • Reviewed 1181 lines of code in 8 files
  • Skipped 0 files when reviewing.
  • Skipped posting 7 draft comments. View those below.
  • Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. docs/concepts/error_handling.md:119
  • Draft comment:
    The documentation for new exceptions (ResponseParsingError, MultimodalError, AsyncValidationError) is thorough and clear. Consider explicitly noting in each section that these exceptions remain backwards compatible by inheriting from ValueError and InstructorError.
  • Reason this comment was not posted:
    Comment was not on a location in the diff, so it can't be submitted as a review comment.
2. instructor/processing/function_calls.py:290
  • Draft comment:
    Replace assert statements (e.g. checking function_call.name) with explicit exception handling (such as raising a ResponseParsingError) for production reliability, since assert may be disabled in optimized runtimes.
  • Reason this comment was not posted:
    Comment was not on a location in the diff, so it can't be submitted as a review comment.
3. instructor/core/exceptions.py:520
  • Draft comment:
    The new exception classes (ResponseParsingError and MultimodalError) are well documented with clear attributes and example usage. Ensure that the error messages remain consistent across the library.
  • Reason this comment was not posted:
    Comment did not seem useful. Confidence is useful = 30% <= threshold 85% This comment is asking the author to ensure consistency in error messages across the library. While it is important to maintain consistency, the comment is not specific about what needs to be changed or improved. It doesn't provide a specific suggestion or point out a specific issue. Therefore, it seems to be more of a general reminder rather than a specific actionable comment.
4. instructor/exceptions.py:11
  • Draft comment:
    The deprecation warning in this backward compatibility module is clear. Confirm that the chosen stacklevel (2) correctly points users to the caller for proper guidance.
  • Reason this comment was not posted:
    Comment was not on a location in the diff, so it can't be submitted as a review comment.
5. tests/test_exception_backwards_compat.py:12
  • Draft comment:
    The tests verifying exception inheritance and catching are comprehensive and ensure backward compatibility. They effectively confirm that the new exceptions can be caught as ValueError, InstructorError, and their specific types.
  • Reason this comment was not posted:
    Comment did not seem useful. Confidence is useful = 0% <= threshold 85% This comment is purely informative and does not provide any actionable feedback or suggestions for improvement. It simply praises the existing tests without offering any specific guidance or identifying any issues.
6. instructor/processing/multimodal.py:115
  • Draft comment:
    Using @lru_cache on methods like from_url and from_path can improve performance, but be cautious of potential staleness if the underlying file or URL content changes. Consider whether cache invalidation is needed in some scenarios.
  • Reason this comment was not posted:
    Confidence changes required: 80% <= threshold 85% None
7. instructor/core/__init__.py:41
  • Draft comment:
    Typo found: 'apatch' in line 41 looks like it might be an accidental entry or typo. Please verify if this should be corrected.
  • Reason this comment was not posted:
    Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_yr4SNtodm3hiIVYk

You can customize Ellipsis by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.

@jxnl jxnl merged commit d14fc54 into main Nov 6, 2025
8 of 13 checks passed
@jxnl jxnl deleted the claude/audit-exception-handling-011CUr4StJP53nzCaifM9KtE branch November 6, 2025 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request python Pull requests that update python code size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants