[Feature Request] Support for Third-Party Plugin Block Names in Core Schema

Hey, I am looking into your library and was wondering whether you would be open for this feature request. I generated the following with AI:

### TL;DR

Tavern currently hardcodes block names like `mqtt_publish` and `grpc_request` in its core schema, preventing third-party plugins from defining their own semantic blocks (e.g., `sql_query`, `cli_command`, `api_call`).

This proposal adds a dynamic plugin API that lets plugins register **custom block names and schemas** at runtime — without requiring core changes or `--tavern-http-backend` flags.
It improves developer experience, plugin modularity, and feature parity between official and community plugins.

### Problem

At present:

* Official plugins (HTTP, MQTT, gRPC) have block names hardcoded in `tavern/_core/schema/tests.jsonschema.yaml`.
* Third-party plugins cannot register custom blocks without forking Tavern or using workarounds.
* Plugin users must remember to pass `--tavern-http-backend=<plugin>` on every pytest run.

This makes third-party plugin development cumbersome and limits Tavern’s extensibility.

### Real-World Use Cases

Developers are extending Tavern beyond HTTP and MQTT:

* **Direct Python API testing** — invoke internal SDK methods without HTTP overhead. I am interested in this one.
* **Database testing** — run and validate SQL queries (`sql_query`, `sql_result`).
* **GraphQL** — query and verify mutations with semantic blocks (`graphql_query`, `graphql_data`).
* **CLI / WebSocket / IoT protocols** — use descriptive block names like `cli_command` or `ws_send`.

Without native support, all these require CLI flags or schema hacks.

## Current Behavior

### Official Plugins (Built into Tavern)
Official plugins like gRPC and MQTT have their block names hardcoded in Tavern's core schema:

```yaml
# From tavern/_core/schema/tests.jsonschema.yaml
properties:
  mqtt_publish:
    $ref: "#/definitions/mqtt_publish"
  mqtt_response:
    oneOf: [...]
  grpc_request:
    $ref: "#/definitions/grpc_request"
  grpc_response:
    $ref: "#/definitions/grpc_response"
```

This enables auto-detection - users can write:

```yaml
stages:
  - name: Test MQTT
    mqtt_publish:  # ← Auto-detects MQTT plugin
      topic: "test/topic"
    mqtt_response:
      topic: "test/response"
```

### Third-Party Plugins (External)

Third-party plugins **cannot** add custom block names without forking Tavern and modifying the core schema. They must:

1. Use standard `request`/`response` blocks
2. Add `type: <discriminator>` to differentiate from HTTP
3. Require backend selection via CLI flag

```yaml
stages:
  - name: Test API
    request:
      type: api_direct_call  # ← Required discriminator
      api_name: myapi
      method: fetch_data
    response:
      status_code: 200
```

**Command line:**
```bash
# Users must remember to add the flag every time
pytest tests/ --tavern-http-backend=my_plugin -v
```

## Desired Behavior

Third-party plugins should be able to register custom block names that get automatically merged into Tavern's core schema at runtime.

### Proposed API

The API should work the same way for both official and third-party plugins.

#### How Official Plugins Currently Work (MQTT Example)

**Plugin Structure:**
```python
# tavern/_plugins/mqtt/tavernhook.py

from .client import MQTTClient
from .request import MQTTRequest
from .response import MQTTResponse

session_type = MQTTClient
request_type = MQTTRequest
request_block_name = "mqtt_publish"  # ← Block name
verifier_type = MQTTResponse
response_block_name = "mqtt_response"  # ← Block name

# Currently NOT exported - schema is hardcoded in core
schema = load_schema_from_file("jsonschema.yaml")
```

**Current Problem:** The `mqtt_publish` and `mqtt_response` block names are **hardcoded** in `tavern/_core/schema/tests.jsonschema.yaml`:

```yaml
# HARDCODED in Tavern core - third-party plugins can't do this!
properties:
  mqtt_publish:
    $ref: "#/definitions/mqtt_publish"
  mqtt_response:
    oneOf:
      - $ref: "#/definitions/mqtt_response"
      - type: array
        items:
          $ref: "#/definitions/mqtt_response"
```

#### How It Should Work (Universal Plugin API)

**Step 1: Plugin exports its schema definitions**

```python
# myproject/tavernhook.py (third-party plugin)
# OR tavern/_plugins/mqtt/tavernhook.py (official plugin)

session_type = MyClient
request_type = MyRequest
request_block_name = "api_call"  # ← Custom block name
verifier_type = MyVerifier
response_block_name = "api_response"  # ← Custom block name

# NEW: Plugin exports its block schemas
block_schemas = {
    "api_call": {
        "type": "object",
        "required": ["api_name"],
        "properties": {
            "api_name": {"type": "string"},
            "method": {"type": "string"},
            "params": {"type": "object"}
        }
    },
    "api_response": {
        "type": "object",
        "properties": {
            "status_code": {"type": "integer"},
            "verify_response_with": {"type": "array"}
        }
    }
}
```

**Step 2: Tavern discovers and merges schemas automatically**

```python
# tavern/_core/schema/files.py (modification)

def load_schema_file(schema_filename, with_plugins=None):
    """Load base schema and merge plugin block schemas"""
    base_schema = _load_yaml(schema_filename)

    # Discover all plugins via stevedore
    import stevedore

    for namespace in ['tavern_http', 'tavern_mqtt', 'tavern_grpc']:
        mgr = stevedore.ExtensionManager(
            namespace=namespace,
            invoke_on_load=False,
        )

        for plugin in mgr:
            plugin_module = plugin.plugin

            # Check if plugin exports block_schemas
            if hasattr(plugin_module, 'block_schemas'):
                # Merge into stage properties
                base_schema['definitions']['stage']['properties'].update(
                    plugin_module.block_schemas
                )

    return base_schema
```

**Step 3: Entry point registration (no change)**
```python
# setup.py or pyproject.toml (same as before)
entry_points={
    "tavern_http": [
        "my_plugin = myproject.tavernhook"
    ]
}
```

#### Migration Path for Official Plugins

Official plugins like MQTT would be refactored to use the same API:

**Before (hardcoded in core schema):**
```yaml
# tavern/_core/schema/tests.jsonschema.yaml
definitions:
  stage:
    properties:
      mqtt_publish:  # ← Hardcoded here
        $ref: "#/definitions/mqtt_publish"
      mqtt_response:  # ← Hardcoded here
        oneOf: [...]
```

**After (plugin exports schema):**
```python
# tavern/_plugins/mqtt/tavernhook.py

session_type = MQTTClient
request_type = MQTTRequest
request_block_name = "mqtt_publish"
verifier_type = MQTTResponse
response_block_name = "mqtt_response"

# NEW: Export block schemas (previously only in core)
block_schemas = {
    "mqtt_publish": {
        "type": "object",
        "required": ["topic"],
        "properties": {
            "topic": {"type": "string"},
            "payload": {"type": "string"},
            "qos": {"type": "integer", "minimum": 0, "maximum": 2},
            "retain": {"type": "boolean"}
        }
    },
    "mqtt_response": {
        "type": "object",
        "required": ["topic"],
        "properties": {
            "topic": {"type": "string"},
            "payload": {"type": "string"},
            "timeout": {"type": "number"}
        }
    }
}
```

**Benefits:**
- ✅ Schema lives with the plugin code (better organization)
- ✅ Same API for official and third-party plugins
- ✅ Core schema can be simpler (less hardcoding)
- ✅ Plugins are truly modular

#### User Test File
```yaml
stages:
  - name: Test API
    api_call:  # ← Auto-detected! No backend flag needed
      api_name: myapi
      method: fetch_data
      params:
        id: 123
    api_response:
      status_code: 200
```

**Command line:**
```bash
# No backend flag needed - auto-detected from block names!
pytest tests/ -v
```

## How Official Plugins Would Look With This API

To show backward compatibility and universal design:

### MQTT Plugin (Official)

**Current Structure:**
```python
# tavern/_plugins/mqtt/tavernhook.py
session_type = MQTTClient
request_type = MQTTRequest
request_block_name = "mqtt_publish"
verifier_type = MQTTResponse
response_block_name = "mqtt_response"
schema = load_from_file("jsonschema.yaml")  # Connection config only
```

```yaml
# tavern/_core/schema/tests.jsonschema.yaml (HARDCODED)
definitions:
  stage:
    properties:
      mqtt_publish:
        $ref: "#/definitions/mqtt_publish"
```

**With Proposed API:**
```python
# tavern/_plugins/mqtt/tavernhook.py
session_type = MQTTClient
request_type = MQTTRequest
request_block_name = "mqtt_publish"
verifier_type = MQTTResponse
response_block_name = "mqtt_response"

# NEW: Plugin defines its own block schemas
block_schemas = {
    "mqtt_publish": {
        "$ref": "#/definitions/mqtt_publish"
    },
    "mqtt_response": {
        "oneOf": [
            {"$ref": "#/definitions/mqtt_response"},
            {"type": "array", "items": {"$ref": "#/definitions/mqtt_response"}}
        ]
    }
}

# Connection config (unchanged)
schema = load_from_file("jsonschema.yaml")
```

### gRPC Plugin (Official)

**With Proposed API:**
```python
# tavern/_plugins/grpc/tavernhook.py
session_type = GRPCClient
request_type = GRPCRequest
request_block_name = "grpc_request"
verifier_type = GRPCResponse
response_block_name = "grpc_response"

# NEW: Plugin defines its own block schemas
block_schemas = {
    "grpc_request": {
        "$ref": "#/definitions/grpc_request"
    },
    "grpc_response": {
        "$ref": "#/definitions/grpc_response"
    }
}
```

### HTTP/REST Plugin (Official)

**With Proposed API:**
```python
# tavern/_plugins/rest/tavernhook.py
class TavernRestPlugin:
    session_type = requests.Session
    request_type = RestRequest
    request_block_name = "request"  # Standard HTTP blocks
    verifier_type = RestResponse
    response_block_name = "response"

    # NEW: Plugin defines its own block schemas
    block_schemas = {
        "request": {
            "$ref": "#/definitions/http_request"
        },
        "response": {
            "$ref": "#/definitions/http_response"
        }
    }
```

### Third-Party Plugin (Example: Direct API)

**With Proposed API:**
```python
# myproject/tavernhook.py
session_type = DirectAPIClient
request_type = DirectAPIRequest
request_block_name = "api_call"  # Custom blocks
verifier_type = DirectAPIVerifier
response_block_name = "api_response"

# Plugin defines its own block schemas (same as official!)
block_schemas = {
    "api_call": {
        "type": "object",
        "required": ["api_name"],
        "properties": {
            "api_name": {"type": "string"},
            "method": {"type": "string"},
            "params": {"type": "object"}
        }
    },
    "api_response": {
        "type": "object",
        "properties": {
            "status_code": {"type": "integer"},
            "verify_response_with": {"type": "array"}
        }
    }
}
```

**Key Point:** Official and third-party plugins use the **exact same API** - only difference is distribution (built-in vs external package).

## Implementation Suggestions

### Option 1: Dynamic Schema Extension

Modify `tavern/_core/schema/files.py` to dynamically merge plugin schemas:

```python
def load_schema_file(schema_filename, with_plugins=None):
    """Load schema and merge plugin schemas dynamically"""
    schema = _load_base_schema(schema_filename)

    if with_plugins:
        for plugin_name in with_plugins:
            plugin_schema = _load_plugin_schema(plugin_name)
            if plugin_schema:
                # Merge plugin's block definitions into stage properties
                schema['definitions']['stage']['properties'].update(
                    plugin_schema
                )

    return schema
```

### Option 2: Plugin Schema Discovery

Add a new entry point group `tavern_schema` for schema registration:

```python
# pyproject.toml
[project.entry-points.tavern_schema]
my_plugin = "myproject.tavernhook:schema"
```

Then in Tavern's schema loading:

```python
import stevedore

def load_all_plugin_schemas():
    """Discover and merge all plugin schemas"""
    mgr = stevedore.ExtensionManager(
        namespace='tavern_schema',
        invoke_on_load=False,
    )

    merged_schemas = {}
    for ext in mgr:
        plugin_schema = ext.plugin  # The 'schema' dict from tavernhook
        merged_schemas.update(plugin_schema)

    return merged_schemas
```

### Option 3: Convention-Based Schema Files

Allow plugins to provide `schema.yaml` files that Tavern automatically discovers and merges:

```
myproject/
├── tavernhook.py
├── schema.yaml  # ← Auto-discovered and merged
```

```yaml
# myproject/schema.yaml
api_call:
  type: object
  required: [api_name]
  properties:
    api_name:
      type: string
    method:
      type: string

api_response:
  type: object
  properties:
    status_code:
      type: integer
```

## Benefits

1. **Better DX for Plugin Authors**: Third-party plugins get the same capabilities as official plugins
2. **Better UX for Users**: No need to remember backend flags for each plugin
3. **Cleaner Test Files**: No `type: discriminator` needed
4. **Auto-Detection**: Tavern automatically routes to the correct plugin based on block names
5. **Extensibility**: Opens Tavern to a richer plugin ecosystem

## Current Workarounds

Third-party plugin authors currently must:

### Workaround 1: pytest.ini Configuration
```ini
[pytest]
addopts = --tavern-http-backend=my_plugin
```
- ❌ Requires configuration file
- ❌ Not portable
- ❌ Affects all tests

### Workaround 2: pytest Plugin Auto-Injection
```python
# pytest_plugin.py
def pytest_load_initial_conftests(early_config, parser, args):
    args.append('--tavern-http-backend=my_plugin')
```

**Entry point:**
```python
entry_points={
    "pytest11": ["my_plugin = myproject.pytest_plugin"]
}
```

- ✅ Works without config files
- ✅ Transparent to users
- ⚠️ Feels like a hack
- ⚠️ Not a Tavern-native solution

### Workaround 3: Schema Monkey-Patching
```python
# Patch Tavern's schema validation (fragile)
from tavern._core.schema import jsonschema

_original_verify = jsonschema.verify_jsonschema

def patched_verify(to_verify, schema):
    # Bypass validation for our custom types
    # ...
    return _original_verify(to_verify, schema)

jsonschema.verify_jsonschema = patched_verify
```

- ❌ Fragile
- ❌ Breaks on Tavern updates
- ❌ Bad practice

## Comparison with Other Test Frameworks

### pytest (Python)
Plugins can extend pytest's configuration and behavior through entry points without modifying core.

### Jest (JavaScript)
Plugins can add custom matchers and transformers that are auto-discovered.

### Playwright (JavaScript/Python)
Fixtures and plugins are auto-discovered through configuration.

**Tavern should support similar extensibility for third-party plugins.**

## Related Issues

- None found (please link if there are existing discussions)

## Example Use Case

A third-party plugin that enables direct Python function calls instead of HTTP requests:

**Current (with workaround):**
```bash
pytest tests/ --tavern-http-backend=my_plugin -v
```

**Desired:**
```bash
pytest tests/ -v  # Just works!
```

**Test file:**
```yaml
test_name: Test Direct API Call
stages:
  - name: Fetch data
    api_call:  # ← Would be auto-detected
      api_name: my_api
      method: fetch_data
      params:
        id: 12345
    api_response:
      status_code: 200
      verify_response_with:
        - function: validate_schema
```

## Questions

1. Is there interest in supporting dynamic schema extension for third-party plugins?
2. Which implementation approach would be preferred?



Uh oh!

[Feature Request] Support for Third-Party Plugin Block Names in Core Schema #978

Description

TL;DR

Problem

Real-World Use Cases

Current Behavior

Official Plugins (Built into Tavern)

Third-Party Plugins (External)

Desired Behavior

Proposed API

How Official Plugins Currently Work (MQTT Example)

How It Should Work (Universal Plugin API)

Migration Path for Official Plugins

User Test File

How Official Plugins Would Look With This API

MQTT Plugin (Official)

gRPC Plugin (Official)

HTTP/REST Plugin (Official)

Third-Party Plugin (Example: Direct API)

Implementation Suggestions

Option 1: Dynamic Schema Extension

Option 2: Plugin Schema Discovery

Option 3: Convention-Based Schema Files

Benefits

Current Workarounds

Workaround 1: pytest.ini Configuration

Workaround 2: pytest Plugin Auto-Injection

Workaround 3: Schema Monkey-Patching

Comparison with Other Test Frameworks

pytest (Python)

Jest (JavaScript)

Playwright (JavaScript/Python)

Related Issues

Example Use Case

Questions

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions