[NEW] Support for Empty SET

[nmvk]

The problem/use-case that the feature addresses
Currently, a key is deleted when the set is empty. Distinguishing between an empty set and a key that does not exist offers many advantages. For example, if we store SQL results in a set, an empty set can represent that a query was executed but the result was empty, whereas null might imply that we need to make a database call to store in the set.

Description of the feature
Without breaking any backward compatibilities we scan support the Empty set with set of new commands and arguments

Write commands
SADD key
Existing SADD would be modified to make members as optional, invoking SADD with only key will create an empty set.

SPOP key [count] MT → Additional argument
Add new optional argument to retain the SET if no element exist.

SREMMT key member [member ...] → [New Command]
Same as SREM would keep the empty set once all members are removed.

SMOVEMT , SINTERSTOREMT, SDIFFSTOREMT, and SUNIONSTOREMT would be implemented as followups if needed. One can delete the empty set using DEL command.

Read commands
SCARD would still return 0 for empty set. To distinguish empty set from a non existing key one can use EXISTS followed by SCARD. Same can be approach for the SISMEMBER and SMISMEMBER.

SMEMBERSMT key → [New Command]
Similar to SMEMBERS but will return null when key does not exist. This command can be achieved by using EXISTS + SMEMBERS but would be supported for ease of use.

Alternatives you've considered
Provide a new param to change the default behavior.

[hpatro]

This is another feature which had quite a strong demand redis/redis#6048 . I think we should make a decision to distinguish between empty vs NULL for data type like set.

@valkey-io/core-team

[sungming2]

I am looking into this issue

[zuiderkwast]

I'm skeptical. If it had been like that from the beginning that commands like SPOP leaves an empty set, then that would have been much better. Then SPOP, HDEL, ZREM and other commands would just not delete a key. Only DEL would delete a key.

But for historical reasons, we have what we have. Adding empty sets now and at the same keeping the default behavior of deleting the key when it becomes empty has problems:

• Complexity. Duplicated commands and new arguments to existing commands.
• Risk of confusion. Although it may be useful to some, this can cause confusions to users. I believe the simplicity is a key success factor.

Regarding alternatives, maybe a new config to make commands not automatically delete empty keys is a good idea? At first sight it seems to be a very big semantical change, but in practice maybe it doesn't actually break many applications? I'm just speculating, but if we allow users to test it and evaluate it, perhaps we can make a more informed decision later.

Are there any other alternatives?

[sungming2]

I have discussed with @hpatro yesterday, I think there could be two approaches.

1. Provide a config like "allow-empty-set(object)"

• Straight forward, avoiding the need for extensive API modification
• Supports backward compatibility
• Managing both features simultaneously can complicate client-side usage, as this toggling the feature requires applications to explicitly handle deletions (e.g., using DEL to remove a set).

2. Introduce new commands: SREMMT(essential)

• There is strong demand for new commands supporting empty sets within the developer community.
• Simplifies migration, as developers can directly replace existing commands with their corresponding SREMMT, minimizing extra workload.
• Additional commands like SPOPMT, SMOVEMT, and SUNIONMT may need to be developed to fully support this feature, increasing the implementation complexity.
• Potential misuse or misunderstanding could be leading to bugs in application / extra memory usage.

[sungming2]

In terms of SMEMBERS behavior, We probably don't want to modify SMEMBERS to return null for a non-existent key and an empty array for an empty set, as each command should have a single, clear purpose. Its role is to retrieve the members of a set, not to determine whether the key exists.

[murphyjacob4]

Does it make sense to start by adding one new command SEMPTY (or similarly named) that creates an empty set? I think keeping the delete-on-empty behavior for the existing commands makes sense for backwards compatibility, and having the commands change behavior based on a server config is not very friendly to users who might have multiple applications using the same server.

I think SEMPTY would give us all the required behavior to unblock those who want to use empty sets, without making us change the behavior of the existing set-mutation commands. Obviously - there will be a difference in behavior for set-access related commands (I.e. SCARD returns 0 on these sets, SMEMBERS can return empty or NULL, etc), but use of SEMPTY would act as an opt-in method for those who want to use empty sets so I think it is okay.

I think SEMPTY will be needed either way, or we end up with clunky mechanics like "add an element and immediately remove that element to get an empty set". And for the primary use case I have seen (query result caching) it should solve the problem completely (if your query has 0 results, call SEMPTY <key>). For those wanting to do "remove-one-element-but-do-not-delete-the-set" - it doesn't solve it completely (unless you do something with MULTI/EXEC), but I am not sure this is the primary use case

For RDB - using SEMPTY would be equivalent to using a new data type - empty sets would not be able to be loaded on previous versions.

[sungming2]

Thanks @murphyjacob4
I think introducing SEMPTY is also good idea

I considered introducing new commands, but IMO, there are potential downsides:

1. New commands could increase the complexity of command management. This might lead to further requests to implement similar commands for other data types, such as lists and sorted sets, adding more maintenance overhead.
2. Using a configuration setting is more flexible and could be extended to support other data types in the future like allow-empty-object, making it a more scalable solution.
3. Each data type has a clear set of commands with consistent behavior. Adding new commands selectively would introduce exceptions to this simplicity which could lead to unintended behavior/result in the application

[PingXie]

The proposal makes sense to me, but I share the skepticism raised earlier by @zuiderkwast.

The SEMPTY approach introduces a few concerns:

• It'd require every delete option to be explicitly followed by SEMPTY and wrapped in MULTI/EXEC to meet the expectation described at https://www.bennadel.com/blog/2965-redis-doesn-t-store-empty-sets-or-hashes-and-will-delete-empty-sets-and-hashes.htm.
• It'd necessitate introducing additional *EMPTY commands for other "container" data structures, such as LIST

I have a doubt about the usability of *EMPTY.

"Client capa" also doesn’t seem quite right here, as it involves global/server-wide behavior changes. What happens if one client expects empty sets while another does not? This feels problematic.

Server configurations might fare a bit better than "client capa," but even then, altering existing behavior in this way feels too drastic and potentially messy, because the change proposed shouldn't be limited to SET but applicable to pretty much all data structures (other than strings).

[soloestoy]

I often hear similar requests, not only for set but also for empty zset, hash, and list. This request is reasonable, but I share the same skepticism as @zuiderkwast , it could increase complexity and lead to confusion in usage.

I am wondering if there could be an alternative solution. For instance, we could set attributes for the key, allowing the user to specify that the key can be empty by using a command like attribute set key allow-empty. This way, when executing commands like spop, hdel, etc., even if the last element is removed, the key itself does not get deleted.

[sungming2]

@soloestoy Thanks for the suggestion,
From my perspective, this approach would likely require maintaining a separate attribute within the object to track whether it is emptiable or not. This could increase memory usage and result in larger RDB files.

[soloestoy]

I don’t think this will lead to memory and RDB expansion. It only requires 1 bit to indicate whether empty is allowed. There is enough space in the serverObject:

struct serverObject {
    unsigned type : 4;
    unsigned encoding : 4;
    unsigned lru : LRU_BITS; /* LRU time (relative to global lru_clock) or
                              * LFU data (least significant 8 bits frequency
                              * and most significant 16 bits access time). */
    int refcount;
    void *ptr;
};

for example, encoding can be compressed to 3 bits (provided that we need to utilize the type and categorize the encoding, allowing for 8 encodings (3 bits) under one type, instead of the current setup with only 16 encodings (4 bits) without distinguishing types). This way, the extra 1 bit can indicate whether empty is allowed.

[hpatro]

I am wondering if there could be an alternative solution. For instance, we could set attributes for the key, allowing the user to specify that the key can be empty by using a command like attribute set key allow-empty. This way, when executing commands like spop, hdel, etc., even if the last element is removed, the key itself does not get deleted.

@soloestoy Do we want to do this at a key level or provide a config at server level to flip when the application is ready to deal with empty object(s)?

My initial suggestion to @sungming2 was to explore allow-empty-object type of config with that we wouldn't need to introduce whole new set of commands and adoption could be quicker based on the devs adopting their application.

[PingXie]

This is too big a change to be gated by a server config IMO. Also I feel that it is a bit odd to require the clients inquire a server config in order to reason about, essentially, the protocol level behavior.

I like @soloestoy's per object attribute idea. I think it is extensible too for other similar changes. We would need an attribute get <key> too. However, discovery and compatibility are not completely solved - an uneducated client when encountering such an container object would not realize the behavior change and might not act correctly.