RI:7799: Stream bulk delete report #5278

pawelangelow · 2025-12-01T11:36:17Z

What

Bulk delete can include a huge number of keys, and our current implementation keeps them in memory. So the robust solution in this case is to stream all the data to a report (successes and failures).

So these changes on a high level look like this:

User initiates bulk delete with "Download report" checkbox enabled => sends generateReport: true
Backend creates bulk action and returns downloadUrl in the overview, but waits before processing (via waitForStreamIfNeeded())
Frontend immediately triggers HTTP GET to /bulk-actions/:id/report (automatic, no button click)
Backend attaches HTTP response stream to BulkAction via setStreamingResponse(res):
- Writes report header directly to HTTP stream
- Resolves the promise => unblocks processing
Processing runs with direct HTTP streaming:
- Runner deletes keys via Redis
- Calls bulkAction.writeToReport() => writes directly to HTTP response
- Only tracks counts in memory (addFailed(1), addSuccess(1))
When complete, finalizeReport() writes summary and calls res.end()
Browser receives the complete file and saves it automatically

Update

After the copilot suggestion - if the HTTP request that starts the actual delete process, for some reason, hangs, the user receives an error that the bulk delete didn't work. If they disable the report, the delete process starts immediately.

Screen.Recording.2025-12-01.at.14.21.11.mov

Testing

Set up a database with a large enough set (that depends on the RAM, but let's say > 1M keys)
Bulk delete everything (*)

Expected result: All good
Actual result: it fails at some point, like in the following recording:

Screen.Recording.2025-11-27.at.15.33.02.mov

After

Screen.Recording.2025-12-01.at.13.31.40.mov

The result log format

Bulk Delete Report
Command Executed for each key: UNLINK key_name
A summary is provided at the end of this file.
==================
string:926189__53.... - OK.  <--- when delete was successful
string:941__807a7..... - Error: NOPERM No permissions to access a key <--- when there is an error
...
=============
Summary:
=============
Status: completed
Processed: 1000 keys
Succeeded: 2 keys
Failed: 998 keys

Note

Adds streaming bulk delete reports with a new download endpoint, updates processing to write directly to the stream, and introduces a UI checkbox that auto-starts the download.

Backend:
- Streaming reports: Add GET /bulk-actions/:id/report/download in bulk-actions.controller.ts and streamReport(id, res) in BulkActionsService to set headers and attach the response stream.
- BulkAction streaming: Implement report flow in models/bulk-action.ts (waitForStreamIfNeeded, setStreamingResponse, writeToReport, finalizeReport, timeout handling) and include downloadUrl and error in getOverview.
- DTO/Provider: Add generateReport?: boolean to CreateBulkActionDto; pass through in BulkActionsProvider.create.
- Runners: Update simple runner to stop accumulating errors/keys in memory and call bulkAction.writeToReport(...) while maintaining counters.
- Interfaces: Extend IBulkActionOverview with downloadUrl and error; extend IBulkAction with writeToReport.
Frontend:
- UI toggle: Add "Download report" checkbox (Redux bulkDelete.generateReport) to control report generation.
- Auto-download: On create, include generateReport; when downloadUrl arrives, trigger download via triggerDownloadFromUrl using getBaseUrl().
- Status/UX: Show failure banner with server error; remove legacy summary download button.
- Utils: Add triggerDownloadFromUrl helper.
Tests:
- Add/expand unit tests for streaming flow (service, BulkAction, runner) and UI checkbox behavior; remove obsolete summary button tests.

^{Written by Cursor Bugbot for commit 9ce17ac. This will update automatically on new commits. Configure here.}

github-actions · 2025-12-01T11:53:23Z

Code Coverage - Integration Tests

Status	Category	Percentage	Covered / Total
🟢	Statements	81.49%	16309/20012
🟡	Branches	64.53%	7362/11407
🟡	Functions	70.38%	2286/3248
🟢	Lines	81.13%	15343/18911

github-actions · 2025-12-01T11:53:47Z

Code Coverage - Backend unit tests

St.^❔	Category	Percentage	Covered / Total
🟢	Statements	92.33%	13936/15093
🟡	Branches	74.07%	4201/5672
🟢	Functions	85.87%	2145/2498
🟢	Lines	92.14%	13323/14459

Test suite run success

2988 tests passing in 287 suites.

Report generated by 🧪jest coverage report action from 9ce17ac

redisinsight/api/src/modules/bulk-actions/models/bulk-action.ts

github-actions · 2025-12-01T12:04:07Z

Code Coverage - Frontend unit tests

St.^❔	Category	Percentage	Covered / Total
🟢	Statements	82.85%	20952/25289
🟡	Branches	68.06%	8845/12996
🟡	Functions	77.89%	5720/7344
🟢	Lines	83.26%	20518/24644

Test suite run success

5440 tests passing in 701 suites.

Report generated by 🧪jest coverage report action from 9ce17ac

ArtemHoruzhenko · 2025-12-02T07:24:50Z

redisinsight/api/src/modules/bulk-actions/interfaces/bulk-action.interface.ts

  getFilter(): BulkActionFilter;
  changeState(): void;
  getSocket(): Socket;
+  writeToReport(keyName: string, success: boolean, error?: string): void;


key name shouldn't be a string. We must use Buffers

ArtemHoruzhenko · 2025-12-02T07:25:25Z

...ght/api/src/modules/bulk-actions/models/runners/simple/abstract.bulk-action.simple.runner.ts

-    const errors = [];

    res.forEach(([err], i) => {
+      const keyName = keys[i].toString();


We must use Buffers

redisinsight/api/src/modules/bulk-actions/models/bulk-action-summary.spec.ts

ArtemHoruzhenko · 2025-12-02T07:36:19Z

redisinsight/api/src/modules/bulk-actions/bulk-actions.controller.ts

+  @Get(':id/report/download')
+  async downloadReport(
+    @Param() { id }: BulkActionIdDto,
+    @Res() res: Response,
+  ): Promise<void> {
+    await this.service.streamReport(id, res);
+  }


This solution does not support multi-tenancy we need to rework it in the future

ArtemHoruzhenko

We can go as is with assumption that we will back to it soon and enhance

redisinsight/ui/src/components/bulk-actions-config/BulkActionsConfig.tsx

cursor · 2025-12-02T08:48:24Z

redisinsight/api/src/modules/bulk-actions/models/bulk-action.ts

+  }
+
+  private getDownloadUrl(): string {
+    return `databases/${this.databaseId}/bulk-actions/${this.id}/report/download`;


Bug: URL path mismatch prevents report downloads

The getDownloadUrl() method generates a URL with a databases/${databaseId}/ prefix, but the BulkActionsController is registered at just /bulk-actions without that prefix. The generated URL databases/{databaseId}/bulk-actions/{id}/report/download won't match the actual controller endpoint /bulk-actions/{id}/report/download, causing 404 errors when the frontend attempts to download reports.

Additional Locations (1)

redisinsight/api/src/modules/bulk-actions/bulk-actions.controller.ts#L16-L30

redisinsight/api/src/modules/bulk-actions/bulk-actions.service.ts

pawelangelow added 12 commits November 28, 2025 15:43

ui: delete download report button

572effc

be: remove keys from summary

69d6fd0

this works

2a1fefc

unit tests

77e9b5b

ui: add download report toggle

4bc0e7a

ui: stop showing errors

a0f0191

be: stop reporting error messages, just count

8fa1f91

use timestamp in the report name

e5bb9d6

fe: fix tests

54a67de

lint issues

4431ce2

fix test

a66892f

cleanup

6231d39

pawelangelow self-assigned this Dec 1, 2025

pawelangelow added 2 commits December 1, 2025 13:38

fix tests

b9cc9c4

cleanup: leftovers

5c566c0

pawelangelow marked this pull request as ready for review December 1, 2025 11:50

pawelangelow requested review from ArtemHoruzhenko, DimoHG, KrumTy, dantovska, pd-redis, tptodorov and valkirilov as code owners December 1, 2025 11:50

cursor bot reviewed Dec 1, 2025

View reviewed changes

redisinsight/api/src/modules/bulk-actions/models/bulk-action.ts Show resolved Hide resolved

valkirilov previously approved these changes Dec 1, 2025

View reviewed changes

handle missing HTTP request to start actual delete process

fa2ef44

pawelangelow dismissed valkirilov’s stale review via fa2ef44 December 1, 2025 12:25

ArtemHoruzhenko reviewed Dec 2, 2025

View reviewed changes

ArtemHoruzhenko previously approved these changes Dec 2, 2025

View reviewed changes

KrumTy reviewed Dec 2, 2025

View reviewed changes

redisinsight/ui/src/components/bulk-actions-config/BulkActionsConfig.tsx Outdated Show resolved Hide resolved

redisinsight/ui/src/components/bulk-actions-config/BulkActionsConfig.tsx Outdated Show resolved Hide resolved

pawelangelow added 3 commits December 2, 2025 10:25

use buffers

56a3d5f

extract downloadFile util

6ad83e0

address PR comment

b68adec

pawelangelow dismissed ArtemHoruzhenko’s stale review via b68adec December 2, 2025 08:42

cursor bot reviewed Dec 2, 2025

View reviewed changes

move download util to a separate file

c1c74d7

cursor bot reviewed Dec 2, 2025

View reviewed changes

redisinsight/api/src/modules/bulk-actions/bulk-actions.service.ts Show resolved Hide resolved

handle late or failed stream

9ce17ac

KrumTy approved these changes Dec 2, 2025

View reviewed changes

dantovska approved these changes Dec 2, 2025

View reviewed changes

pawelangelow merged commit 9ef8847 into main Dec 2, 2025
51 of 52 checks passed

pawelangelow deleted the feature/RI-7799/http-streaming-download branch December 2, 2025 11:09

valkirilov pushed a commit that referenced this pull request Dec 3, 2025

RI:7799: Stream bulk delete report (#5278)

d89e35b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

RI:7799: Stream bulk delete report #5278

RI:7799: Stream bulk delete report #5278

Uh oh!

pawelangelow commented Dec 1, 2025 •

edited by cursor bot

Loading

Uh oh!

github-actions bot commented Dec 1, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

github-actions bot commented Dec 1, 2025 •

edited

Loading

Uh oh!

ArtemHoruzhenko Dec 2, 2025

Uh oh!

ArtemHoruzhenko Dec 2, 2025

Uh oh!

Uh oh!

ArtemHoruzhenko Dec 2, 2025

Uh oh!

ArtemHoruzhenko left a comment

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 2, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

RI:7799: Stream bulk delete report #5278

RI:7799: Stream bulk delete report #5278

Uh oh!

Conversation

pawelangelow commented Dec 1, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Update

Testing

After

The result log format

Uh oh!

github-actions bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage - Integration Tests

Uh oh!

github-actions bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage - Backend unit tests

Test suite run success

Uh oh!

Uh oh!

github-actions bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage - Frontend unit tests

Test suite run success

Uh oh!

ArtemHoruzhenko Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ArtemHoruzhenko Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ArtemHoruzhenko Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ArtemHoruzhenko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 2, 2025

Choose a reason for hiding this comment

Bug: URL path mismatch prevents report downloads

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pawelangelow commented Dec 1, 2025 •

edited by cursor bot

Loading

github-actions bot commented Dec 1, 2025 •

edited

Loading

github-actions bot commented Dec 1, 2025 •

edited

Loading

github-actions bot commented Dec 1, 2025 •

edited

Loading