feat: Improve error handling with comprehensive exception hierarchy

docs/concepts/error_handling.md

@@ -0,0 +1,293 @@
+---
+title: Error Handling in Instructor
+description: Learn how to effectively handle errors and exceptions when using Instructor for structured outputs.
+---
+
+# Error Handling
+
+Instructor provides a comprehensive exception hierarchy to help you handle errors gracefully and debug issues effectively. This guide covers the various exception types and best practices for error handling.
+
+## Exception Hierarchy
+
+All Instructor-specific exceptions inherit from `InstructorError`, making it easy to catch all Instructor-related errors:
+
+```python
+from instructor.exceptions import (
+    InstructorError,
+    IncompleteOutputException,
+    InstructorRetryException,
+    ValidationError,
+    ProviderError,
+    ConfigurationError,
+    ModeError,
+    ClientError
+)
+```
+
+### Base Exception
+
+- **`InstructorError`**: Base exception for all Instructor-specific errors. Catch this to handle any Instructor error.
+
+### Specific Exception Types
+
+#### `IncompleteOutputException`
+Raised when the LLM output is incomplete due to reaching the maximum token limit.
+
+```python
+try:
+    response = client.chat.completions.create(
+        model="gpt-3.5-turbo",
+        response_model=DetailedReport,
+        messages=[{"role": "user", "content": "Write a very long report..."}],
+        max_tokens=50  # Very low limit
+    )
+except IncompleteOutputException as e:
+    print(f"Output was truncated: {e}")
+    print(f"Last completion: {e.last_completion}")
+```
+
+#### `InstructorRetryException`
+Raised when all retry attempts have been exhausted.
+
+```python
+try:
+    response = client.chat.completions.create(
+        model="gpt-3.5-turbo",
+        response_model=UserDetail,
+        messages=[{"role": "user", "content": "Extract user info..."}],
+        max_retries=3
+    )
+except InstructorRetryException as e:
+    print(f"Failed after {e.n_attempts} attempts")
+    print(f"Last error: {e}")
+    print(f"Last completion: {e.last_completion}")
+    print(f"Total usage: {e.total_usage}")
+```
+
+#### `ValidationError`
+Raised when response validation fails. This is different from Pydantic's ValidationError and provides additional context.
+
+```python
+try:
+    response = client.chat.completions.create(
+        model="gpt-3.5-turbo",
+        response_model=StrictModel,
+        messages=[{"role": "user", "content": "Extract data..."}]
+    )
+except ValidationError as e:
+    print(f"Validation failed: {e}")
+```
+
+#### `ProviderError`
+Raised for provider-specific errors, includes the provider name for context.
+
+```python
+try:
+    client = instructor.from_anthropic(invalid_client)
+except ProviderError as e:
+    print(f"Provider {e.provider} error: {e}")
+```
+
+#### `ConfigurationError`
+Raised for configuration-related issues like invalid parameters or missing dependencies.
+
+```python
+try:
+    client = instructor.from_provider("invalid/model")
+except ConfigurationError as e:
+    print(f"Configuration error: {e}")
+```
+
+#### `ModeError`
+Raised when an invalid mode is used for a specific provider.
+
+```python
+try:
+    client = instructor.from_anthropic(
+        anthropic.Anthropic(),
+        mode=instructor.Mode.TOOLS  # Wrong mode for Anthropic
+    )
+except ModeError as e:
+    print(f"Invalid mode '{e.mode}' for provider '{e.provider}'")
+    print(f"Valid modes: {', '.join(e.valid_modes)}")
+```
+
+#### `ClientError`
+Raised for client initialization or usage errors.
+
+```python
+try:
+    client = instructor.from_anthropic("not_a_client")
+except ClientError as e:
+    print(f"Client error: {e}")
+```
+
+## Best Practices
+
+### 1. Catch Specific Exceptions When Possible
+
+```python
+from instructor.exceptions import (
+    IncompleteOutputException,
+    InstructorRetryException,
+    ValidationError
+)
+
+try:
+    response = client.chat.completions.create(...)
+except IncompleteOutputException as e:
+    # Handle truncated output - maybe increase max_tokens
+    logger.warning(f"Output truncated, retrying with more tokens")
+    response = client.chat.completions.create(..., max_tokens=2000)
+except InstructorRetryException as e:
+    # Handle retry exhaustion - maybe fallback logic
+    logger.error(f"Failed after {e.n_attempts} attempts")
+    return None
+except ValidationError as e:
+    # Handle validation errors - maybe relax constraints
+    logger.error(f"Validation failed: {e}")
+    raise
+```
+
+### 2. Use the Base Exception for General Error Handling
+
+```python
+from instructor.exceptions import InstructorError
+
+try:
+    response = client.chat.completions.create(...)
+except InstructorError as e:
+    # Catches any Instructor-specific error
+    logger.error(f"Instructor error: {type(e).__name__}: {e}")
+    raise
+```
+
+### 3. Handle Provider Setup Errors
+
+```python
+from instructor.exceptions import ConfigurationError, ClientError, ModeError
+
+def create_client(provider: str, mode: str = None):
+    try:
+        client = instructor.from_provider(provider)
+        return client
+    except ConfigurationError as e:
+        print(f"Configuration issue: {e}")
+        # Maybe guide user to install missing package
+    except ModeError as e:
+        print(f"Invalid mode. Valid modes for {e.provider}: {e.valid_modes}")
+        # Retry with a valid mode
+    except ClientError as e:
+        print(f"Client initialization failed: {e}")
+        # Check client setup
+```
+
+### 4. Logging and Monitoring
+
+```python
+import logging
+from instructor.exceptions import InstructorError
+
+logger = logging.getLogger(__name__)
+
+def extract_data(content: str):
+    try:
+        return client.chat.completions.create(
+            model="gpt-3.5-turbo",
+            response_model=DataModel,
+            messages=[{"role": "user", "content": content}]
+        )
+    except InstructorError as e:
+        logger.exception(
+            "Failed to extract data",
+            extra={
+                "error_type": type(e).__name__,
+                "provider": getattr(e, 'provider', None),
+                "attempts": getattr(e, 'n_attempts', None),
+            }
+        )
+        raise
+```
+
+### 5. Graceful Degradation
+
+```python
+from instructor.exceptions import ValidationError, InstructorRetryException
+
+def extract_with_fallback(content: str):
+    # Try with strict model first
+    try:
+        return client.chat.completions.create(
+            model="gpt-4",
+            response_model=StrictDataModel,
+            messages=[{"role": "user", "content": content}]
+        )
+    except ValidationError:
+        # Fall back to less strict model
+        logger.warning("Strict validation failed, trying relaxed model")
+        return client.chat.completions.create(
+            model="gpt-3.5-turbo",
+            response_model=RelaxedDataModel,
+            messages=[{"role": "user", "content": content}]
+        )
+    except InstructorRetryException:
+        # Final fallback
+        logger.error("All retries exhausted, returning None")
+        return None
+```
+
+## Integration with Hooks
+
+Instructor's hooks system can be used to monitor and handle errors programmatically:
+
+```python
+from instructor import Instructor
+from instructor.exceptions import ValidationError
+
+def on_parse_error(error: Exception):
+    if isinstance(error, ValidationError):
+        # Log validation errors to monitoring service
+        monitoring.log_validation_error(str(error))
+
+client = Instructor(...)
+client.hooks.on("parse:error", on_parse_error)
+```
+
+## Common Error Scenarios
+
+### Missing Dependencies
+
+```python
+try:
+    client = instructor.from_provider("anthropic/claude-3")
+except ConfigurationError as e:
+    if "package is required" in str(e):
+        print("Please install the anthropic package: pip install anthropic")
+```
+
+### Invalid Provider Format
+
+```python
+try:
+    client = instructor.from_provider("invalid-format")
+except ConfigurationError as e:
+    print(e)  # Model string must be in format "provider/model-name"
+```
+
+### Unsupported Mode
+
+```python
+try:
+    client = instructor.from_openai(
+        openai.OpenAI(),
+        mode=instructor.Mode.ANTHROPIC_TOOLS  # Wrong mode
+    )
+except ModeError as e:
+    print(f"Use one of these modes instead: {e.valid_modes}")
+```
+
+## See Also
+
+- [Retrying](./retrying.md) - Learn about retry strategies
+- [Validation](./validation.md) - Understanding validation in Instructor
+- [Hooks](./hooks.md) - Using hooks for error monitoring

docs/concepts/hooks.md

@@ -295,12 +295,33 @@ def handle_completion_error(error: Exception) -> None:
     print(f"## Completion error: {error}")
     print(f"Type: {type(error).__name__}")
     print(f"Message: {str(error)}")
+
+    # Handle specific Instructor exceptions
+    from instructor.exceptions import (
+        IncompleteOutputException,
+        ValidationError,
+        ProviderError
+    )
+
+    if isinstance(error, IncompleteOutputException):
+        print(f"Output was truncated. Last completion: {error.last_completion}")
+    elif isinstance(error, ValidationError):
+        print("Validation failed - check your model schema")
+    elif isinstance(error, ProviderError):
+        print(f"Provider {error.provider} had an issue")
 
 
 def log_parse_error(error: Exception) -> None:
     print(f"## Parse error: {error}")
     print(f"Type: {type(error).__name__}")
     print(f"Message: {str(error)}")
+
+    # You can also check for Pydantic validation errors
+    from pydantic import ValidationError as PydanticValidationError
+    if isinstance(error, PydanticValidationError):
+        print("Pydantic validation errors:")
+        for err in error.errors():
+            print(f"  - {err['loc']}: {err['msg']}")
 
 
 # Handler for a custom logger that records how many errors have occurred
@@ -409,6 +430,66 @@ handler: CompletionErrorHandler = my_error_handler
 client.on("completion:error", handler)
 ```
 
+## Error Handling with Hooks
+
+Hooks provide an excellent way to monitor and handle errors consistently across your application. You can use them with Instructor's exception hierarchy for sophisticated error handling:
+
+```python
+from instructor.exceptions import (
+    InstructorError,
+    IncompleteOutputException,
+    InstructorRetryException,
+    ValidationError,
+    ProviderError,
+    ConfigurationError
+)
+import logging
+
+logger = logging.getLogger(__name__)
+
+class ErrorMonitor:
+    def __init__(self):
+        self.error_counts = {
+            "incomplete": 0,
+            "validation": 0,
+            "provider": 0,
+            "retry_exhausted": 0,
+            "other": 0
+        }
+
+    def handle_error(self, error: Exception):
+        # Log the error with appropriate level
+        if isinstance(error, IncompleteOutputException):
+            self.error_counts["incomplete"] += 1
+            logger.warning(f"Incomplete output: {error}")
+        elif isinstance(error, ValidationError):
+            self.error_counts["validation"] += 1
+            logger.error(f"Validation failed: {error}")
+        elif isinstance(error, ProviderError):
+            self.error_counts["provider"] += 1
+            logger.error(f"Provider error ({error.provider}): {error}")
+        elif isinstance(error, InstructorRetryException):
+            self.error_counts["retry_exhausted"] += 1
+            logger.critical(f"All retries exhausted after {error.n_attempts} attempts")
+        else:
+            self.error_counts["other"] += 1
+            logger.error(f"Unexpected error: {type(error).__name__}: {error}")
+
+    def get_stats(self):
+        return self.error_counts
+
+# Usage
+monitor = ErrorMonitor()
+client = instructor.from_openai(openai.OpenAI())
+
+client.on("completion:error", monitor.handle_error)
+client.on("parse:error", monitor.handle_error)
+client.on("completion:last_attempt", monitor.handle_error)
+
+# After running your application
+print(f"Error statistics: {monitor.get_stats()}")
+```
+
 ## Hooks in Testing
 
 Hooks are particularly useful for testing, as they allow you to inspect the arguments and responses without modifying your application code: