|
| 1 | +# Cluster Maintenance Notifications Example |
| 2 | + |
| 3 | +This example demonstrates how to use maintenance notifications with Redis Cluster to handle slot migrations seamlessly. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +When Redis Cluster performs slot migrations (e.g., during resharding or node rebalancing), the client can receive push notifications to: |
| 8 | + |
| 9 | +1. **SMIGRATING** - Indicates slots are being migrated (relaxes timeouts) |
| 10 | +2. **SMIGRATED** - Indicates slot migration completed (triggers cluster state reload) |
| 11 | + |
| 12 | +This example shows how to: |
| 13 | +- Enable maintnotifications for ClusterClient |
| 14 | +- Track SMIGRATING and SMIGRATED notifications |
| 15 | +- Monitor cluster state reloads |
| 16 | +- Continue operations seamlessly during migrations |
| 17 | + |
| 18 | +## Prerequisites |
| 19 | + |
| 20 | +You need a running Redis Cluster with at least 3 master nodes. |
| 21 | + |
| 22 | +## Running the Example |
| 23 | + |
| 24 | +### Basic Usage |
| 25 | + |
| 26 | +```bash |
| 27 | +# Run with default cluster addresses (localhost:7000, 7001, 7002) |
| 28 | +go run main.go |
| 29 | + |
| 30 | +# Or specify custom cluster addresses |
| 31 | +REDIS_CLUSTER_ADDRS="localhost:7000,localhost:7001,localhost:7002" go run main.go |
| 32 | +``` |
| 33 | + |
| 34 | +### Expected Output |
| 35 | + |
| 36 | +``` |
| 37 | +2024/12/02 10:00:00 Connected to Redis cluster with maintnotifications enabled |
| 38 | +2024/12/02 10:00:00 The client will automatically handle SMIGRATING and SMIGRATED notifications |
| 39 | +2024/12/02 10:00:00 Press Ctrl+C to exit |
| 40 | +2024/12/02 10:00:10 Completed 10 operations |
| 41 | +2024/12/02 10:00:20 Completed 20 operations |
| 42 | +``` |
| 43 | + |
| 44 | +## Testing Slot Migration |
| 45 | + |
| 46 | +To see maintnotifications in action, trigger a slot migration: |
| 47 | + |
| 48 | +### Using cae-resp-proxy (Recommended for Testing) |
| 49 | + |
| 50 | +```bash |
| 51 | +# Terminal 1: Start the proxy |
| 52 | +docker run -d \ |
| 53 | + -p 7000:6379 -p 8000:3000 \ |
| 54 | + -e TARGET_HOST=host.docker.internal \ |
| 55 | + -e TARGET_PORT=6379 \ |
| 56 | + redislabs/client-resp-proxy:latest |
| 57 | + |
| 58 | +# Terminal 2: Run the example (connecting through proxy) |
| 59 | +REDIS_CLUSTER_ADDRS="localhost:7000" go run main.go |
| 60 | + |
| 61 | +# Terminal 3: Inject SMIGRATING notification |
| 62 | +curl -X POST "http://localhost:8000/send-to-all-clients?encoding=raw" \ |
| 63 | + --data-binary ">3\r\n\$10\r\nSMIGRATING\r\n:12345\r\n\$4\r\n1000\r\n" |
| 64 | + |
| 65 | +# Inject SMIGRATED notification (new format) |
| 66 | +curl -X POST "http://localhost:8000/send-to-all-clients?encoding=raw" \ |
| 67 | + --data-binary ">4\r\n\$9\r\nSMIGRATED\r\n:12346\r\n:1\r\n*1\r\n\$20\r\n127.0.0.1:6380 1000\r\n" |
| 68 | +``` |
| 69 | + |
| 70 | +Expected output after injection: |
| 71 | +``` |
| 72 | +2024/12/02 10:01:00 SMIGRATING notification received: SeqID=12345, Slots=[1000] |
| 73 | +2024/12/02 10:01:05 SMIGRATED notification received (reload #1): SeqID=12346, Endpoints=[127.0.0.1:6380 1000] |
| 74 | +``` |
| 75 | + |
| 76 | +### Using Real Cluster Migration |
| 77 | + |
| 78 | +```bash |
| 79 | +# Reshard slots from one node to another |
| 80 | +redis-cli --cluster reshard 127.0.0.1:7000 \ |
| 81 | + --cluster-from <source-node-id> \ |
| 82 | + --cluster-to <target-node-id> \ |
| 83 | + --cluster-slots 100 \ |
| 84 | + --cluster-yes |
| 85 | +``` |
| 86 | + |
| 87 | +During resharding, you should see SMIGRATING and SMIGRATED notifications in the example output. |
| 88 | + |
| 89 | +## Code Walkthrough |
| 90 | + |
| 91 | +### 1. Enable Maintnotifications |
| 92 | + |
| 93 | +```go |
| 94 | +client := redis.NewClusterClient(&redis.ClusterOptions{ |
| 95 | + Addrs: []string{"localhost:7000", "localhost:7001", "localhost:7002"}, |
| 96 | + Protocol: 3, // RESP3 required for push notifications |
| 97 | + |
| 98 | + MaintNotificationsConfig: &maintnotifications.Config{ |
| 99 | + Mode: maintnotifications.ModeEnabled, |
| 100 | + RelaxedTimeout: 30 * time.Second, |
| 101 | + }, |
| 102 | +}) |
| 103 | +``` |
| 104 | + |
| 105 | +### 2. Track Notifications with Custom Hook |
| 106 | + |
| 107 | +```go |
| 108 | +client.OnNewNode(func(nodeClient *redis.Client) { |
| 109 | + manager := nodeClient.GetMaintNotificationsManager() |
| 110 | + if manager != nil { |
| 111 | + hook := ¬ificationTracker{ |
| 112 | + onSMigrated: func(seqID int64, endpoints []string) { |
| 113 | + log.Printf("SMIGRATED: SeqID=%d, Endpoints=%v", |
| 114 | + seqID, endpoints) |
| 115 | + }, |
| 116 | + onSMigrating: func(seqID int64, slots []string) { |
| 117 | + log.Printf("SMIGRATING: SeqID=%d, Slots=%v", seqID, slots) |
| 118 | + }, |
| 119 | + } |
| 120 | + manager.AddNotificationHook(hook) |
| 121 | + } |
| 122 | +}) |
| 123 | +``` |
| 124 | + |
| 125 | +### 3. Perform Operations |
| 126 | + |
| 127 | +The client automatically handles notifications in the background. Your application code continues to work normally: |
| 128 | + |
| 129 | +```go |
| 130 | +client.Set(ctx, "key", "value", 0) |
| 131 | +client.Get(ctx, "key") |
| 132 | +``` |
| 133 | + |
| 134 | +## Key Features Demonstrated |
| 135 | + |
| 136 | +### Automatic Timeout Relaxation |
| 137 | + |
| 138 | +When SMIGRATING is received: |
| 139 | +- Read/write timeouts are automatically relaxed to `RelaxedTimeout` |
| 140 | +- Prevents false failures during slot migration |
| 141 | +- Automatically restored when migration completes |
| 142 | + |
| 143 | +### Automatic Cluster State Reload |
| 144 | + |
| 145 | +When SMIGRATED is received: |
| 146 | +- Cluster state is automatically reloaded |
| 147 | +- Client learns about new slot ownership |
| 148 | +- Subsequent commands are routed to correct nodes |
| 149 | +- Deduplication ensures reload happens only once per SeqID |
| 150 | + |
| 151 | +### Seamless Operation |
| 152 | + |
| 153 | +- No application code changes needed during migration |
| 154 | +- Operations continue without interruption |
| 155 | +- Automatic retry and redirection handled by client |
| 156 | + |
| 157 | +## Configuration Options |
| 158 | + |
| 159 | +```go |
| 160 | +MaintNotificationsConfig: &maintnotifications.Config{ |
| 161 | + // Mode: auto (default), enabled, or disabled |
| 162 | + Mode: maintnotifications.ModeEnabled, |
| 163 | + |
| 164 | + // Timeout to use during migration (default: 10s) |
| 165 | + RelaxedTimeout: 30 * time.Second, |
| 166 | + |
| 167 | + // Duration to keep relaxed timeout after handoff (default: 5s) |
| 168 | + PostHandoffRelaxedDuration: 5 * time.Second, |
| 169 | + |
| 170 | + // Circuit breaker settings |
| 171 | + CircuitBreakerFailureThreshold: 5, |
| 172 | + CircuitBreakerResetTimeout: 60 * time.Second, |
| 173 | +} |
| 174 | +``` |
| 175 | + |
| 176 | +## Monitoring |
| 177 | + |
| 178 | +The example tracks: |
| 179 | +- Number of SMIGRATING notifications received |
| 180 | +- Number of SMIGRATED notifications received |
| 181 | +- Number of cluster state reloads triggered |
| 182 | +- Operation success/failure rates |
| 183 | + |
| 184 | +You can extend this to: |
| 185 | +- Export metrics to Prometheus |
| 186 | +- Log to structured logging system |
| 187 | +- Alert on excessive migration activity |
| 188 | + |
| 189 | +## Troubleshooting |
| 190 | + |
| 191 | +### No Notifications Received |
| 192 | + |
| 193 | +1. Verify RESP3 is enabled (`Protocol: 3`) |
| 194 | +2. Check Redis version supports push notifications (Redis 6.0+) |
| 195 | +3. Verify maintnotifications mode is not disabled |
| 196 | +4. Check Redis cluster is actually performing migrations |
| 197 | + |
| 198 | +### Operations Failing During Migration |
| 199 | + |
| 200 | +1. Increase `RelaxedTimeout` value |
| 201 | +2. Check network connectivity |
| 202 | +3. Verify cluster has sufficient resources |
| 203 | +4. Review Redis logs for errors |
| 204 | + |
| 205 | +### Excessive Cluster State Reloads |
| 206 | + |
| 207 | +1. Check for rapid slot migrations |
| 208 | +2. Verify deduplication is working (same SeqID should reload once) |
| 209 | +3. Consider adding cooldown period in custom hook |
| 210 | + |
| 211 | +## Related Examples |
| 212 | + |
| 213 | +- [Disable Maintnotifications](../disable-maintnotifications/) - How to disable the feature |
| 214 | +- [Maintnotifications PubSub](../maintnotifiations-pubsub/) - Using with Pub/Sub |
| 215 | + |
| 216 | +## References |
| 217 | + |
| 218 | +- [Maintenance Notifications Documentation](../../maintnotifications/README.md) |
| 219 | +- [Redis Cluster Specification](https://redis.io/topics/cluster-spec) |
| 220 | +- [RESP3 Protocol](https://github.com/redis/redis-specifications/blob/master/protocol/RESP3.md) |
| 221 | + |
0 commit comments