Wishlist

[mattsta]

It would be nice to finally de-crapify and de-egoify the project fully. Just make a new primary version and go backwards incompatible.

I'm fairly certain the blanket "license change" is also illegal since they didn't get permission from every copyright holder to rip their code apart from the license (also why linux can never change from GPL2), but if they don't care, then we don't have to care about un-licensing all their code either. Evil goes both ways I guess (that article has some timeline inaccuracies and rationale inaccuracies and leaves out multiple other key players, but it's mostly correct in how redis was "stolen" by motivated exploiters over the years).

Wishlist

• finally remove all master/slave terminology (See Don't use master/slave terminology (non-breaking changes only) #36)
• remove sentinel and cluster (they will never work reliably because they are fundamentally flawed in their design. notice how nothing else in the world uses them or their design patterns? at least clickhouse liked zookeeper so they rewrote it in C++ for their own usage)
  • overall it's more stable going with separation-of-concerns and using consistent-hash-replication sharding proxies in front of these things anyway, though maybe some CRDTs will need to get involved
• port all tcl scripts to something better (it's 2024 not 1994)
  • new code robots can greatly help with this I imagine
• use clang-format for all the code (it's 2024 not 1994)
  • clang-format -style="{BasedOnStyle: llvm, IndentWidth: 4, AllowShortFunctionsOnASingleLine: None, KeepEmptyLinesAtTheStartOfBlocks: false, InsertBraces: true; InsertNewlineAtEOF: true}"
• stop supporting weird platforms because it's 2024 and nobody cares about big endian or 32 bit systems anymore (it causes unnecessary maintenance overhead for refactoring and building new features)
• stop using CRC everywhere. again, it's 2024 not 1994. Nobody should be using CRC anymore because it's designed for 1970s serial interfaces. We have much better systems now.
  • like, this thing is complete amateur garbage what are people even doing anymore: https://github.com/placeholderkv/placeholderkv/blob/unstable/src/crc16_slottable.h
  • attempting to use designed-to-be-slow CRC running at 300 MB/s (or 1.5 GB/s with my improvements I spent a year coming up with 10 years ago) to blockingly validate multi-gigabyte files is just computational malpractice when things like xxh3 or xxh128 exists running at 30 GB/s.
• fix the hand written, hand parsed, self-rewriting config file system abomination (nothing else in the world does this, and it caused a wormable exploit in the past so should never be trusted. it only exists because "lol its a cool trick!!!" unprofessional insecure programming)
• stop using hand written Makefiles and use CMake instead (again, it's 2024 not 1994)
  • I actually spent a few hundred hours making jemalloc compatible with CMake in 2017-2019 then when I aksed FB if they could pay for my contributions they said "uh, we can send you a t-shirt?" then they never even sent the t-shirt (a month later, FB paid a $5 billion government fine for their typical FB-style business practices, but they sure don't have any money to pay developers... go figure?)
• convert the info/status output to regular readable JSON instead of hand-written output formats requiring hand-written parsers to read
• continue to refactor and improve a lot of the data structures from their unrefined CS101 origins

there's probably a couple other dozen things too, but that's my long-term gripe list I'm really surprised most people haven't been motivated to see the problems and fix them yet for long term project reliability and continuity.

Conclusion

I fixed all these problems (and more!) in my own rewrite a couple years ago (just imagine you put luajit, sqlite, erlang, redis, and memcached in a blender then a new modern high-performance multi-core secure in-memory cache system popped out), but I tried to sell it instead of giving away thousands of hours of work across 5+ years for free, so it never got any traction: https://carrierdb.cloud/

Provenance

i used to do stuff here but the project preferred to remain computationally and culturally conservative instead of embracing the future (except for selling out for profit at the expense of community and collaboration and user experience, of course) :(

Work I created which the project sometimes fought against but eventually included anyway:

• entire module system was my own design and implementation, but the project fought against it until later they decided it could be used for profit
• geo commands (though they aren't complete and should really use h3 instead of geohash, but nobody seems to understand enough to maintain it over the past 10 years)
• JSON experiments (not sure what happened with those, but my redis replacement server can switch between output formats of redis, memcached, JSON, and even python (only format with actual set syntax in the output), and it also supports unlimited nesting of any data type each with their own nested expiration values too)
• massive memory improvements to the lists structures (further efficiency improvements are in my redis replacement server too along with improving all the other data structures as well) — this was inspired by the custom changes to redis Twitter was running internally over thousands of servers totaling terabytes of RAM (which was a lot in 2015), but the twitter devs refused to release their improvements because "we can't get legal to clear it," so I re-built their ideas in an even better version from scratch (since then, millions of servers have benefited from these efficiency improvements for free, so thanks to me i guess)
• crc speed improvements which took a couple hundred hours of on-and-off experiments over a year to finally get the logic working because redis uses a random weird non-standard CRC64 implementation for some reason so no other libraries are compatible with it unless you do extensive custom workarounds all over the place.

placeholderkv$ rg "Matt Stancliff"                                                                                       on unstable
src/quicklist.c
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>

src/geo.c
2: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/geohash_helper.c
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/crcspeed.h
1:/* Copyright (c) 2014, Matt Stancliff <matt@genges.com>

src/quicklist.h
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>

src/geohash_helper.h
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/geohash.h
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/zmalloc.c
865: * 3) Was modified for Redis by Matt Stancliff.

src/crcspeed.c
4: * Modifications by Matt Stancliff <matt@genges.com>:

src/geohash.c
3: * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.

src/crc64.c
1:/* Copyright (c) 2014, Matt Stancliff <matt@genges.com>

deps/hiredis/hiredis.c
4: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/net.c
5: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/net.h
5: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/fuzzing/format_command_fuzzer.c
4: * Copyright (c) 2020, Matt Stancliff <matt at genges dot com>,

deps/hiredis/hiredis.h
4: * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,

deps/hiredis/CHANGELOG.md
511:* Fix tests when assert() undefined (Keith Bennett, Matt Stancliff)

[PingXie]

+1 on all, @mattsta!

These are all great points and suggestions! These changes would have little impact on the existing users but would go a long way to support future innovations. I would love to work with you to make these happen.

[madolson]

there's probably a couple other dozen things too, but that's my long-term gripe list I'm really surprised most people haven't been motivated to see the problems and fix them yet for long term project reliability and continuity.

Honestly, your comment of "culture conservatism" resonated strong with me. I spent a bunch of time trying to move to python tests, and the resistance from the former Redis guys (Oran and Yossi) was insurmountable. I'm perhaps a bit more conservative in the fact that I think we should be deeply fearful of breaking changes for Redis, given it's place in the stack, but definitely believe we need to be moving faster.

[mattsta]

I think we should be deeply fearful of breaking changes for Redis, given it's place in the stack, but definitely believe we need to be moving faster.

Exactly. There is a balance between continuity of existing systems while also not remaining stuck with 15 year old designs for the next 15+ years. I raised a lot of these standard project maintenance issues 10 years ago and they never got better, so now it looks even more archaic in a lot of places.

Honestly, your comment of "culture conservatism" resonated strong with me. I spent a bunch of time trying to move to python tests, and the resistance from the former Redis guys (Oran and Yossi) was insurmountable.

I'm sorry I couldn't have prevented the current situation. I tried to advocate for both not giving the project away to a corrupt company and also for more consistent full time project management and architecture improvements (which was misunderstood as trying to "take over the project" then everything blew up), but I failed to generate the change I wanted to see in the world. Remember this one? ah, memories: http://antirez.com/news/87

But as for moving tests, it would obviously be great to move them to python. I bet with some careful work we could paste tcl files into Claude and ask for pytest formatted results. It would also be nice to stand up more concurrent testing if we can isolate tests to not step on each other.

[zuiderkwast]

Matt, your perspective and experience is very valuable. I'm glad you came to this fork. Like Madelyn, I think we need to be both radical and conservative.

We'll need to create issues for each of these points to discuss them one by one, and have some kind of categorization and decision making process.... We'll come to that.

[wenerme]

If we are live in 2024, can we have some http(ws,http1,http2,http3) based protocol support builtin ?

[zuiderkwast]

Yeah, if we drop the idea of vendoring all dependencies (see #15), we can definitely have optional support for those. I'd like to have RESP over QUIC (multiplexed streams of commands over one connection) and optional compilation with liburing (io_uring).

[mattsta]

I still have drafts of a redis binary network protocol from 2014-2015 based on http/2 I was planning (including things like multiplexing using per-command client command ids matched to reply ids for non-head-of-line blocking concurrency, etc).

I think it's interesting to note the difference between the network protocol and the data protocol though. Currently redis operates a hybrid network+data protocol, but we can easily split the network protocol into a more streamlined binary format with multiple addressable streams allowing different data output formats too.

For data output, it turns out of all formats possible, JSON is the most efficient format if your data isn't majority binary blobs.

What does JSON solve?

• you don't need extra bytes for length prefixes on strings since all strings are contained within two quotes as the start/end string signal: "string"
• you don't need to pre-specify the length or depth or type of all your data structures either since JSON data structures are also start/end delimited
  • though, JSON isn't an amazing format for streaming without workarounds like ndjson (and it doesn't have a set type or other extensible types without creating custom tagged union wrappers)
• numbers written in ascii decimal are optimal because shorter numbers require fewer bytes, and shorter numbers are more common
• implicit type lengths defined by {} and "" and [] and split by , are more efficient than requiring up-front lengths and type markers delimited everywhere like $5\r\nHELLO\r\n instead of just "hello" for user data
  • *2\r\n$5\r\nhello\r\n$5\r\nworld\r\n is just ["hello","world"] instead, etc
• escaping JSON strings can be checked for bad values and converted quickly using SIMD operations on intel and ARM in 16 byte chunks (or potentially even larger)
• PLUS, you don't have to write yet another conversion layer from some rando server protocol to JSON, you can just yeet your command results directly back at users (so your app servers just act as literal data proxies instead of structured transformation nodes if your data is scoped properly with no other intermediate transformation steps).

[zuiderkwast]

We can add opt-in JSON along as HELLO json. It has everything that RESP2 has and parts of what RESP3 has (maps) but lacks some features: no difference between arrays and sets (not a big deal), push messages (can be done at protocol lever if we use JSON over HTTP/2 though).

We don't have a JSON dependency right now though. Can we split this out into a separate issue please?

[Edit] One huge disadvantage of JSON it that it can't store binary data. Strings must be valid Unicode. To store binary data in JSON strings, people use tricks like Base64.

[stockholmux]

finally remove all master/slave terminology

+1 this is the right time start this.

[wenerme]

Speaking from my personal experience, my preference for NATS over Redis stems from NATS's support for WebSocket-based transport. This choice does not imply that I will use NATS directly in a browser environment. However, it allows me to leverage the existing infrastructure while bypassing the constraints associated with port requirements.

[mattsta]

Just a note looking through more recent issues people are adding:

• Allow using AOF (the collection of commands) instead of RDB for full synchronization #59
• Should functions be synchronized among cluster nodes? #58
• [NEW] Send cluster topology changes as push messages #57
• [NEW] Enable using a connection while waiting for a blocking command #54

These are all literally things I brought up as major design flaws 10 years ago (with plans to fix!). I'm really surprised there's been no progress on so many of these basic architecture and design issues.

Even simple things like "the script isn't replicated everywhere so you get random failures" was happening 10 years ago ya'll and nobody decided to improve the system in the meantime? I'm curious what's missing. Initiative? Permission? Ability? Project management prioritization? Lack of curiosity? Only following "profitable" improvements instead of overall stability? (another interpretation may be the project has intentionally preferred to remain less extensible and flexible to retain lock-in so people don't "grow the project" in "unapproved" directions where profit can't be captured)

Cluster topology still is causing client problems? AOF and RDB formats and differences are still causing problems and requires a full version revision for all changes instead of having extensible metadata built-in? It's almost like the project has been afraid of addressing any of the original architecture and design inconsistencies?

It's worth remembering a lot of these architecture and design decisions weren't made by some expert committee having a combined 100 years of experience in distributed systems and distributed consistency protocols and flexible persistence and reliability and storage formats... it's mostly just "the ideas of some guy having fun building a personal database on a macbook air in 2010." Somehow, "because redis has always done it this way" became codified as a reason to never change or improve any of these core faults of the design? The core design of redis has always been treated as almost sacred and unquestionable and unchangeable even though high visibility deficiencies are scattered throughout. It's all just software and not some immutable laws of the universe.

wut.

[bitnom]

Things I want:

• JSON/REJSON
• Graph
• Search
• Timeseries
• Vectors
• Websockets (And possibly gRPC, additionally but not instead of)
• Clustering
• A fully compatible (And clusterable) WASM build.

We should probably do these via polls in the Discussion section though.