Added janitor task for automatic deposit recovery with Prometheus metrics

[sigurpol]

Implements automatic cleanup of old discarded submissions to reclaim deposits.
Close #980.

Features:

• Triggers when the election round increments (runs once per new round)
• Scans the last 5 rounds for old submissions at most (the usual process is to simply clear the previous round).
• Calls clear_old_round_data() to recover deposits and clean storage
• Non-blocking integration with existing mining operations
• Comprehensive error handling (critical vs recoverable errors)

Architecture

• Introduce JanitorMessage enum and janitor_task for deposit recovery
• Use dedicated bounded channel for janitor communication
• Update listener to send janitor ticks to janitor task
• Ensure mining and janitor operations are independent and non-blocking

┌──────────────────────────────────────────────────────────────────────────┐
│   ┌─────────────┐                      ┌─────────────┐            ┌─────────────┐
└──▶│ Listener    │                      │   Miner     │            │ Blockchain  │
    │             │  Snapshot/Signed     │             │            │             │
    │ ┌─────────┐ │ ────────────────────▶│ ┌─────────┐ │ (solutions)│             │
    │ │ Stream  │ │  (mining work)       │ │ Mining  │ │───────────▶│             │
    │ └─────────┘ │                      │ └─────────┘ │            │             │
    │      │      │  Round++             │ ┌─────────┐ │            │             │
    │      ▼      │ ────────────────────▶│ │ Clear   │ │            │             │
    │ ┌─────────┐ │                      │ │ Snapshot│ │            │             │
    │ │ Phase   │ │                      │ └─────────┘ │            │             │
    │ │ Check   │ │  Round++             └─────────────┘            │             │
    │ └─────────┘ │ ────────────────────▶┌─────────────┐            │             │
    │             │  (deposit cleanup)   │  Janitor    │ (cleanup)  │             │
    │             │                      │ ┌─────────┐ │───────────▶│             │
    │             │                      │ │ Cleanup │ │            │             │
    │             │                      │ └─────────┘ │            │             │
    └─────────────┘                      └─────────────┘            └─────────────┘

Prometheus Metrics Added:

Counters:

• staking_miner_janitor_cleanup_success_total: Successful cleanup operations
• staking_miner_janitor_cleanup_failures_total: Failed cleanup operations

Gauges:

• staking_miner_janitor_cleanup_duration_ms: Time taken for last cleanup
• staking_miner_janitor_old_submissions_found: Old submissions discovered
• staking_miner_janitor_old_submissions_cleared: Submissions successfully cleared

Key Metrics:

• Success rate: success_total / (success_total + failures_total)
• Performance: avg(cleanup_duration_ms)
• Activity: old_submissions_cleared shows actual deposit recovery

Example Prometheus queries:

staking_miner_janitor_cleanup_success_total / (staking_miner_janitor_cleanup_success_total + staking_miner_janitor_cleanup_failures_total)

increase(staking_miner_janitor_cleanup_success_total[24h])

staking_miner_janitor_old_submissions_cleared / staking_miner_janitor_old_submissions_found

Logs

An example of logs below when a previous submission is cleared

2025-06-20T20:09:03.221683Z DEBUG polkadot-staking-miner: Detected round increment 9 -> 10    
2025-06-20T20:09:03.221707Z TRACE polkadot-staking-miner: Sent janitor tick for round 10    
2025-06-20T20:09:03.221711Z DEBUG polkadot-staking-miner: Round increment in Off phase, signaling snapshot cleanup    
2025-06-20T20:09:03.221718Z TRACE polkadot-staking-miner: Block #792, Phase Off - nothing to do    
2025-06-20T20:09:03.221731Z TRACE polkadot-staking-miner: Running janitor cleanup for round 10    
2025-06-20T20:09:03.221745Z TRACE polkadot-staking-miner: Clearing snapshots    
2025-06-20T20:09:03.221749Z TRACE polkadot-staking-miner: Scanning round 9 for old submissions (current round: 10, scanning rounds from 9)    
2025-06-20T20:09:03.222313Z DEBUG polkadot-staking-miner: Found old submission in round 9 with 4 pages, attempting cleanup    
2025-06-20T20:09:03.222324Z DEBUG polkadot-staking-miner: Clearing old round data for round 9 with 4 witness pages    
2025-06-20T20:09:11.232603Z TRACE polkadot-staking-miner: Block #793, Phase Off - nothing to do    
2025-06-20T20:09:15.243377Z TRACE polkadot-staking-miner: Block #794, Phase Off - nothing to do    
2025-06-20T20:09:23.253658Z TRACE polkadot-staking-miner: Block #795, Phase Off - nothing to do    
2025-06-20T20:09:27.257928Z TRACE polkadot-staking-miner: Block #796, Phase Off - nothing to do    
2025-06-20T20:09:35.275623Z TRACE polkadot-staking-miner: Block #797, Phase Off - nothing to do    
2025-06-20T20:09:39.278849Z DEBUG polkadot-staking-miner: Successfully submitted clear_old_round_data for round 9    
2025-06-20T20:09:39.278892Z  INFO polkadot-staking-miner: Successfully cleaned up old submission from round 9 (4 witness pages)    
2025-06-20T20:09:39.278901Z  INFO polkadot-staking-miner: Janitor cleaned up 1 old submissions in 36057ms    

Integration test

An integration test now verifies the scenario where two miners submit identical solutions. Both solutions are successfully submitted; one is rewarded while the other is discarded after clear_old_round() has been explicitly called by the miner with the non-winning solution.

2025-06-20T13:13:08.930566Z  INFO monitor: Bob solution discarded!    
2025-06-20T13:13:08.998134Z  INFO monitor: 🤑 Successfully completed two-miner test: both submitted solutions, one rewarded, one discarded! Duration: 1179.356788208s 🤑    
test submit_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1179.38s

[sigurpol]

My usual suspects as reviewers are all OOO these days. This PR is needed for Kusama AH migration.
@jsdw , @Overkillus or @tdimitrov : if you have time to spare / waste, I would appreciate a review 🙏 🙇

[jsdw]

For somebody who doesn't know this stuff so well: if this didn't exist, would the old deposits be locked up forever without such a "cleanup" phase to claim them back?

[sigurpol]

correct, there is no automatic reclaim, the election pallet does not automatically return deposits for valid not best solutions.

[jsdw]

Nice job on the README rewrite/tidyup!

[jsdw]

I am not an expert in the specifics of staking, but the code looks clean and the approach makes sense to me, modulo a couple of tiny comments; nice one!