Delete keys matching pattern

[satheeshaGowda]

Problem Statement

Valkey currently lacks a built-in, safe method for deleting keys matching a pattern. This fundamental gap forces developers into complex, error-prone workflows that compromise both performance and reliability.

Current Limitations

Developers must resort to suboptimal solutions that introduce significant complexity:

1. Multi-Step Client-Side Operations

# Fragile shell pipeline approach
valkey-cli --scan --pattern "*:foo:bar:*" | xargs -L 100 valkey-cli unlink

Issues: Not atomic, prone to partial failures, requires external tooling

2. Custom Lua Scripting

-- Complex multi-iteration script for safe atomic operations
local function scan_and_unlink(pattern)
    local cursor = "0"
    local total_deleted = 0
    local keys_to_delete = {}
    
    repeat
        local result = redis.call('SCAN', cursor, 'MATCH', pattern, 'COUNT', 100)
        cursor = result[1]
        local keys = result[2]
        
        for i = 1, #keys do
            keys_to_delete[#keys_to_delete + 1] = keys[i]
        end
    until cursor == "0"
    
    if #keys_to_delete > 0 then
        total_deleted = redis.call('UNLINK', unpack(keys_to_delete))
    end
    
    return total_deleted
end

-- Usage: EVAL "..." 0 "prefix:foo:bar:*"
return scan_and_unlink(ARGV[1])

Issues: Complex cursor management, memory pressure from collecting all keys, requires advanced Lua expertise, potential script timeout on large datasets

3. Manual Cursor Management

Developers must implement complex logic handling:

• Cursor state management across iterations
• Error recovery and retry mechanisms
• Partial failure scenarios
• Memory pressure from large result sets

These workarounds lead to increased development time, maintenance burden, and potential data inconsistencies.

Proposed Solution

Enhance the SCAN command with an optional UNLINK parameter that enables safe, atomic, non-blocking deletion of matching keys during iteration.

Enhanced Command Syntax

SCAN cursor [MATCH pattern] [COUNT count] [TYPE type] [UNLINK]

Usage Examples

Basic Pattern Deletion

# Clean up expired session keys
valkey-cli --scan --pattern 'session:*' --unlink

# Remove user cache with custom batch size
valkey-cli --scan --pattern 'cache:user:*' --count 1000 --unlink

Server Command Examples

# Delete test data after integration tests
127.0.0.1:6379> SCAN 0 MATCH test:* UNLINK
1) "0"                    # Cursor (scan complete)
2) 1) "test:user:1001"    # Keys that were unlinked
   2) "test:user:1002"
   3) "test:session:abc"
3) (integer) 3            # Number of keys unlinked

# Large-scale cleanup with controlled batching
127.0.0.1:6379> SCAN 0 MATCH temp:* COUNT 500 UNLINK
1) "1847"                 # More keys to scan
2) 1) "temp:data:1"       # First batch unlinked
   2) "temp:data:2"
   ...
   498) "temp:cache:500"
3) (integer) 500          # Batch size reached

# Continue scanning from cursor
127.0.0.1:6379> SCAN 1847 MATCH temp:* COUNT 500 UNLINK
1) "0"                    # Scan complete
2) 1) "temp:final:1"      # Remaining keys unlinked
3) (integer) 1847          # Final batch size

Alternatives you've considered:

• Create a new command, something like SCANUNLINK or SCANDEL, which could lead to API fragmentation and duplicate/fragmented logic.
• Extend existing UNLINK command to take pattern as an argument ex: UNLINK session:*, which could lead to blocks on large datasets, breaks UNLINK semantics

Future Scope

If this PR is accepted , we have plan to add UNLINK options to following commands as well

• SSCAN
• ZSCAN
• HSCAN

[zuiderkwast]

If we add an UNLINK option to SCAN, it turns SCAN into a write command, to it needs the WRITE command flag. It means SCAN can't be executed on readonly replicas. This would be a breaking change.

I think it would be better to introduce a new command like SCANDEL with the same options as SCAN.

We may also want to think about this feature can be combined with a cluster-wide scan as discussed in #33.

If you want to delete many keys in this way, the replies will be large. Maybe some users would just want to returns the number of deleted keys instead of the list of keys?

Regarding your custom Lua script: If you iterate until cursor == 0 in the Lua script, it will hang the server. Slow scripts can cause cluster failovers and other unintended effects. It's better to let the script behave like your suggested scan command, i.e. return a new cursor each time. The loop should be on the client side, possible with some small sleep between each call if you don't want to keep the server too busy.

We could add a correct and well-tested Lua script to our docs, so users don't need to invent this by themselves. We could also add this feature to valkey-cli even without a new server command. valkey-cli can use Lua.

[satheeshaGowda]

If we add an UNLINK option to SCAN, it turns SCAN into a write command, to it needs the WRITE command flag. It means SCAN can't be executed on readonly replicas. This would be a breaking change.

I totally agree, and it was my concern as well .

We may also want to think about this feature can be combined with a cluster-wide scan as discussed in #33.

This is an interesting ask, we would be interested to look into it.

I think it would be better to introduce a new command like SCANDEL with the same options as SCAN.

we concur with this approach. If we have consensus, will re-purpose this PR to introduce new command SCANDEL

[madolson]

Decision: Wait to see if there is more traction. We'll discuss this topic in Amsterdam as well.

[hpatro]

Will add my thoughts from the weekly meeting:

I think people started using restricted namespace / keyspace for their application as we shaped that as the future with cluster mode (single db) and ACL rules supporting keyspace restriction. However, we never provided any support to clean up the keyspace similar to FLUSHDB to delete all the keys atomically.

Instead of adding a new command for SCANDEL, I would prefer us to provide a better primitive for long term.

• Short term solution could be to provide users with well crafted LUA script which users can confidently run it on the server and avoid higher tail latency for other clients.
• Long term, I think we should nudge people to use databases. We should build named databases ([NEW] Introduce database logical names #1601) if it helps with ease of usage and support FLUSHDB <dbname> on top of it.

[zuiderkwast]

@hpatro I agree, we can nudge people to using databases, but there may be other reasons to delete keys matching a pattern. Now, we have optimizations for scanning a pattern that implies a single slot. We can scan only one slot in this case.

I believe SCAN + delete is a very common use case. As @stockholmux mentioned in a meeting, people want to avoid Lua scripts because they're error prone. (Example: In the script posted above, in cluster mode, the UNLINK call with multiple keys can return a CROSSSLOT error, if SCAN returned keys from multiple slots.)

I'm leaning towards accepting a SCANDEL command. I've created an issue for it, to decouple it from this implementation PR:

• [NEW] SCANDEL = SCAN + delete #2623

[madolson]

@madolson Add a status on the Valkey Roadmap for things that need a new design or more thought. @ranshid suggested "No Motivation"