[Polymarket] Add opt-in pruning for inactive Polymarket instrument state

[fedoraiver]

Summary

Add an opt-in prune_inactive_instruments setting to PolymarketDataClientConfig.

When enabled, the Polymarket data client prunes per-instrument market data state after the last active market data subscription for an instrument has been removed.

The intended use case is long-running processes which dynamically subscribe to many short-lived Polymarket markets, for example recurring prediction markets where instruments are opened, streamed briefly, then closed and never used again.

Problem

PolymarketDataClient keeps several per-instrument caches while streaming live market data:

• order_books
• last_quotes
• instruments
• token_meta

For static instrument universes this is fine. For dynamic Polymarket workloads, especially short-duration markets, the client can discover and stream a large number of instruments over time. After market data subscriptions are removed, the instrument may no longer be useful to the process, but its cached state remains in memory.

In a 24/7 service, this means memory can grow with the number of historical instruments seen by the client instead of the number of currently active instruments.

Proposal

Add a new config field:

pub struct PolymarketDataClientConfig {
    // ...
    /// Whether to prune per-instrument market data caches after all market data
    /// subscriptions are removed.
    #[builder(default)]
    pub prune_inactive_instruments: bool,
    // ...
}

Default: false.

This preserves current behavior unless the user explicitly opts in.

Expose the same option through the Python config constructor:

PolymarketDataClientConfig(
    # ...
    prune_inactive_instruments=True,
)

Behavior

After each market data unsubscribe path:

• unsubscribe_book_deltas
• unsubscribe_quotes
• unsubscribe_trades

the client checks whether the instrument still has any active market data subscriptions:

active_quote_subs.contains(&instrument_id)
    || active_delta_subs.contains(&instrument_id)
    || active_trade_subs.contains(&instrument_id)

If prune_inactive_instruments is false, cleanup is skipped entirely.

If the option is enabled and no active market data subscription remains, the client removes:

order_books.remove(&instrument_id);
last_quotes.remove(&instrument_id);
instruments.remove(&instrument_id);
token_meta.remove(&token_id);

The implementation resolves the token id from the live instrument cache when possible, and falls back to scanning token_meta by instrument_id when needed.

ws_open_tokens is intentionally not pruned by this cleanup path. WebSocket subscribe/unsubscribe state is reconciled separately by the existing websocket subscription logic.

[cjdsellers]

Hi @fedoraiver,

Thanks for writing this up and for putting together #3947.

The memory concern is valid, but we're not going to merge that PR because the abstraction is the wrong way around:

• A venue adapter should not own cache retention or pruning policy.
• An adapter-level prune_inactive_instruments switch would push bespoke cache mutation logic into each adapter.
• Cache mutation should stay owned by the platform/sync core, with users deciding what policy they want before calling it.
• I have a patch landing shortly which introduces CacheView, so normal adapters receive a read-only cache view and cannot mutate platform cache state directly.

For the pruning use case, the right follow-up is a user-facing/platform-level prune API, not Polymarket-specific adapter logic (this will be landing later today).

Please keep the reports coming if you run into more usability issues like this 👌.

[cjdsellers]

Hi @fedoraiver,

Quick follow-up: the platform-level prune API I mentioned is now on develop as Cache.purge_instrument(instrument_id) (both Rust and Python): e78da9e.

For your Polymarket use case you can call it from a strategy or actor when a market resolves; the adapter docs include a worked example. The guard refuses while any associated order is non-terminal or any position is open. Active subscriptions stay owned by the data engine, so unsubscribe before purging if you no longer want updates.

Docs here: https://github.com/nautechsystems/nautilus_trader/blob/develop/docs/concepts/cache.md#purging-cached-data

I'll follow up later today in the #polymarket Discord with the usability improvements that went in today.