Skip to content

feat: audit log improvements — CSV/JSON export and real-time WebSocket stream#215

Merged
BillChirico merged 3 commits intomainfrom
feat/issue-136
Mar 2, 2026
Merged

feat: audit log improvements — CSV/JSON export and real-time WebSocket stream#215
BillChirico merged 3 commits intomainfrom
feat/issue-136

Conversation

@BillChirico
Copy link
Collaborator

Summary

Enhances the audit log system with two missing features from the issue:

✅ Export to CSV/JSON

  • New endpoint: GET /guilds/:id/audit-log/export
  • Supports format=csv or format=json (default)
  • Up to 10k rows per export (vs 100 for paginated view)
  • All existing filters apply (action, userId, startDate, endDate)
  • CSV uses proper RFC 4180 quoting (escapeCsvValue)
  • JSON wraps entries with metadata (guildId, exportedAt, count)
  • Triggers file download via Content-Disposition: attachment

✅ Real-time WebSocket updates

  • New WebSocket server on /ws/audit-log (same HMAC ticket auth as /ws/logs)
  • Clients authenticate, then receive live entry events as mutations are committed
  • Optional per-client filter: { type: 'filter', guildId, action, userId }
  • Broadcast hooks into insertAuditEntry via RETURNING clause (no extra query)
  • Lifecycle wired into startServer/stopServer
  • Cap of 10 concurrent clients; 30s heartbeat

Already present (no changes needed)

  • ✅ Filter by action type, user, date range — existed in the original route
  • ✅ Pagination — existed in the original route

Tests

  • 30 route tests (existing 14 + 16 new export tests)
  • 17 WebSocket stream tests (connection, auth, filtering, broadcast, edge cases)
  • All 47 new tests pass

Diff stat

src/api/middleware/auditLog.js    |  24 ++-
src/api/routes/auditLog.js        | 201 ++++++++++-----
src/api/server.js                 |  12 ++
src/api/ws/auditStream.js         | 322 +++++++++++++++++++++++++++
tests/api/routes/auditLog.test.js | 234 ++++++++++++++++++++
tests/api/ws/auditStream.test.js  | 349 +++++++++++++++++++++++++++++++
6 files changed, 1102 insertions(+), 40 deletions(-)

Closes #136

…tream

- Add GET /:id/audit-log/export endpoint (CSV and JSON, up to 10k rows)
- Add /ws/audit-log WebSocket server for real-time audit entry broadcast
- Refactor buildFilters() shared helper to eliminate duplication
- Hook broadcastAuditEntry() into insertAuditEntry (RETURNING id+created_at)
- Wire setupAuditStream/stopAuditStream into startServer/stopServer lifecycle
- Add escapeCsvValue/rowsToCsv helpers with full test coverage
- 30 route tests + 17 WebSocket stream tests, all green

Closes #136
Copilot AI review requested due to automatic review settings March 2, 2026 04:36
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 2, 2026

Warning

Rate limit exceeded

@BillChirico has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 25 minutes and 49 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 4c49a09 and 7bbcade.

📒 Files selected for processing (6)
  • src/api/middleware/auditLog.js
  • src/api/routes/auditLog.js
  • src/api/server.js
  • src/api/ws/auditStream.js
  • tests/api/routes/auditLog.test.js
  • tests/api/ws/auditStream.test.js
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/issue-136

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds audit log export (CSV/JSON) and a real-time WebSocket audit stream to improve dashboard audit log consumption.

Changes:

  • Added GET /guilds/:id/audit-log/export with CSV/JSON output, larger limits, and export-specific rate limiting.
  • Introduced /ws/audit-log WebSocket stream with HMAC ticket auth, optional filtering, heartbeat, and client cap.
  • Updated audit log insertion to RETURNING the inserted row and broadcast it to connected WebSocket clients.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/api/middleware/auditLog.js Broadcasts newly inserted audit entries to the audit WebSocket stream using RETURNING row data.
src/api/routes/auditLog.js Adds export endpoint, CSV helpers, and refactors shared filter-building logic.
src/api/server.js Wires audit stream lifecycle into server start/stop.
src/api/ws/auditStream.js Implements WebSocket audit log streaming server (auth, filter, heartbeat, broadcast).
tests/api/routes/auditLog.test.js Adds unit tests for CSV helpers and integration tests for the export endpoint.
tests/api/ws/auditStream.test.js Adds coverage for audit stream connection/auth/filter/broadcast behaviors.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@greptile-apps
Copy link

greptile-apps bot commented Mar 2, 2026

Greptile Summary

This PR successfully implements the two missing features from issue #136: CSV/JSON export and real-time WebSocket streaming for audit logs.

Key Changes:

  • Export endpoint (GET /guilds/:id/audit-log/export) supports CSV and JSON formats with up to 10k rows, proper RFC 4180 CSV escaping, and all existing filters
  • WebSocket server (/ws/audit-log) provides real-time updates using HMAC ticket authentication (matching the existing log stream pattern), client-side filtering, and 30-second heartbeat
  • Broadcast integration uses PostgreSQL RETURNING clause to broadcast new entries without additional queries
  • Code refactoring extracted filter logic into reusable buildFilters() helper

Implementation Quality:

  • Follows all project conventions: ESM modules, Winston logging, parameterized queries, proper error handling
  • Comprehensive test coverage with 47 total tests (30 route + 17 WebSocket tests)
  • Defensive programming with try-catch blocks preventing audit failures from affecting streaming
  • Enhanced WebSocket heartbeat with readyState checks and error handling (improvement over existing log stream)
  • Configurable auth timeout via AUDIT_STREAM_AUTH_TIMEOUT_MS environment variable

Architecture:

  • Non-blocking audit middleware ensures streaming failures don't impact audit integrity
  • Proper lifecycle management integrated into server startup/shutdown
  • Rate limiting: 30 req/min for standard queries, 10 req/min for exports
  • Max 10 concurrent WebSocket clients with automatic cleanup

Confidence Score: 5/5

  • This PR is safe to merge with high confidence
  • Code follows all project security and style conventions, has comprehensive test coverage, uses parameterized queries throughout, implements proper authentication (HMAC tickets for WebSocket, guild admin middleware for HTTP), includes defensive error handling, and introduces no breaking changes or security vulnerabilities
  • No files require special attention

Important Files Changed

Filename Overview
src/api/middleware/auditLog.js Modified to broadcast audit entries to WebSocket clients via RETURNING clause integration
src/api/routes/auditLog.js Added export endpoint with CSV/JSON formats and refactored filter logic into reusable buildFilters helper
src/api/server.js Integrated audit stream lifecycle (setup/teardown) into server startup and shutdown
src/api/ws/auditStream.js New WebSocket server for real-time audit log streaming with HMAC auth, filtering, and heartbeat

Sequence Diagram

sequenceDiagram
    participant Client as Dashboard Client
    participant HTTP as Express Server
    participant WS as WebSocket Server
    participant Middleware as Audit Middleware
    participant DB as PostgreSQL
    
    Note over Client,DB: Real-time Audit Stream Flow
    Client->>WS: Connect to /ws/audit-log
    WS->>Client: Connection established
    Client->>WS: {type: 'auth', ticket: 'hmac'}
    WS->>WS: Validate HMAC ticket
    WS->>Client: {type: 'auth_ok'}
    Client->>WS: {type: 'filter', guildId, action}
    WS->>Client: {type: 'filter_ok'}
    
    Note over Client,DB: Mutation triggers broadcast
    Client->>HTTP: PUT /guilds/:id/config
    HTTP->>Middleware: Audit middleware intercepts
    Middleware->>DB: INSERT ... RETURNING *
    DB->>Middleware: Return new entry with id
    Middleware->>WS: broadcastAuditEntry(entry)
    WS->>WS: Match against client filters
    WS->>Client: {type: 'entry', entry: {...}}
    
    Note over Client,DB: Export Flow
    Client->>HTTP: GET /guilds/:id/audit-log/export?format=csv
    HTTP->>HTTP: Validate auth & guild admin
    HTTP->>DB: SELECT * WHERE ... LIMIT 10000
    DB->>HTTP: Return matching entries
    HTTP->>Client: CSV file download
Loading

Last reviewed commit: 7bbcade

coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 2, 2026
- ws.ping() crash: guard with readyState check + try/catch to avoid
  crashing heartbeat interval when socket not OPEN
- stopAuditStream race: make setupAuditStream async and await
  stopAuditStream() to prevent concurrent WebSocketServer creation
- Query param array coercion: add typeof === 'string' checks for
  startDate/endDate to handle Express string|string[]|undefined
- CSV CRLF quoting: add \r to RFC 4180 special-char check for proper
  Windows line ending handling
- Test timeouts: make AUTH_TIMEOUT_MS configurable via
  AUDIT_STREAM_AUTH_TIMEOUT_MS env var, use 100ms in tests
Copilot AI review requested due to automatic review settings March 2, 2026 09:53
@BillChirico BillChirico merged commit abfb03c into main Mar 2, 2026
11 of 13 checks passed
@BillChirico BillChirico deleted the feat/issue-136 branch March 2, 2026 09:53
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

BillChirico added a commit that referenced this pull request Mar 2, 2026
…t stream (#215)

* feat: audit log improvements — CSV/JSON export, real-time WebSocket stream

- Add GET /:id/audit-log/export endpoint (CSV and JSON, up to 10k rows)
- Add /ws/audit-log WebSocket server for real-time audit entry broadcast
- Refactor buildFilters() shared helper to eliminate duplication
- Hook broadcastAuditEntry() into insertAuditEntry (RETURNING id+created_at)
- Wire setupAuditStream/stopAuditStream into startServer/stopServer lifecycle
- Add escapeCsvValue/rowsToCsv helpers with full test coverage
- 30 route tests + 17 WebSocket stream tests, all green

Closes #136

* fix: PR #215 review feedback - audit stream fixes

- ws.ping() crash: guard with readyState check + try/catch to avoid
  crashing heartbeat interval when socket not OPEN
- stopAuditStream race: make setupAuditStream async and await
  stopAuditStream() to prevent concurrent WebSocketServer creation
- Query param array coercion: add typeof === 'string' checks for
  startDate/endDate to handle Express string|string[]|undefined
- CSV CRLF quoting: add \r to RFC 4180 special-char check for proper
  Windows line ending handling
- Test timeouts: make AUTH_TIMEOUT_MS configurable via
  AUDIT_STREAM_AUTH_TIMEOUT_MS env var, use 100ms in tests
BillChirico added a commit that referenced this pull request Mar 2, 2026
* security: escape user content in triage prompt delimiters (#164)

Add escapePromptDelimiters() to HTML-encode < and > in user-supplied
message content before it is inserted between XML-style section tags
in the LLM prompt.

Without escaping, a crafted message containing the literal text
`</messages-to-evaluate>` could break out of the user-content section
and inject attacker-controlled instructions into the prompt structure.

Changes:
- Add escapePromptDelimiters(text) utility exported from triage-prompt.js
- Apply escape to m.content and m.replyTo.content in buildConversationText()
- Add 13 new tests covering the escape function and injection scenarios

Closes #164

* security: escape & chars and author fields in prompt delimiters

* fix(security): escape & in prompt delimiters and escape author fields

- Add & → &amp; escape first in escapePromptDelimiters() to prevent
  HTML entity bypass attacks (e.g. &lt;/messages-to-evaluate&gt;)
- Also escape m.author and m.replyTo.author since Discord display
  names are user-controlled and can contain < / > characters

Addresses review feedback on PR #204.

* fix: guard replyTo.content before .slice() to handle null/undefined

* perf: SQL-based conversation pagination + missing DB indexes (#221)

Fixes three performance bottlenecks identified in code review of
recently merged features (PR #121 conversations viewer, PR #190 AI feedback).

## Changes

### migrations/004_performance_indexes.cjs (new)
Four new indexes targeting hot query paths:

- idx_ai_feedback_guild_created (guild_id, created_at DESC)
  getFeedbackTrend() and getRecentFeedback() filtered by guild_id
  AND created_at but only had a single-column guild_id index, forcing
  a full guild scan + sort on every trend/recent call.

- idx_conversations_content_trgm (GIN, pg_trgm)
  content ILIKE '%...%' search was a sequential scan. GIN/trgm index
  reduces this from O(n) to O(log n * trigram matches).
  Requires pg_trgm extension (added idempotently).

- idx_conversations_guild_created (guild_id, created_at DESC)
  Default 30-day listing query filters guild_id + created_at. The
  existing 3-column (guild_id, channel_id, created_at) composite is
  suboptimal when channel_id is not in the predicate.

- idx_flagged_messages_guild_message (guild_id, message_id)
  Conversation detail + flag endpoints query flagged_messages by
  guild_id AND message_id = ANY(...). Existing index only covers
  (guild_id, status).

### src/api/routes/conversations.js
**GET / — Replace in-memory pagination with SQL CTE grouping**

Before: fetched up to 10,000 message rows into Node memory, grouped
them in JavaScript (O(n) time + memory), then sliced for pagination.
Every page request loaded the full 10k row dataset.

After: single SQL query using window functions (LAG + SUM OVER) to
identify conversation boundaries and aggregate summaries directly.
COUNT(*) OVER() provides total count without a second query.
Pagination happens at the DB with LIMIT/OFFSET on summary rows.
Memory overhead is now proportional to page size (default 25), not
total conversation volume.

Removed now-unused buildConversationSummary() helper (logic inlined
into the SQL-side aggregation).

**POST /:conversationId/flag — Parallel verification queries**

Before: msgCheck and anchorCheck ran sequentially (~2× RTT).
After: both run in parallel via Promise.all (1× RTT for verification).

### tests/api/routes/conversations.test.js
Updated 'should return paginated conversations' test to mock the new
SQL CTE response shape (pre-aggregated summary rows) instead of raw
message rows. All 41 conversation tests pass.

* feat: channel-level quiet mode via bot mention (#173) (#213)

* feat: quiet mode per-channel via bot mention (#173)

- Add quietMode.js module with Redis+memory storage
- Parse duration from natural language (30m, 1 hour, etc.)
- Permission gated via config.quietMode.allowedRoles
- Commands: quiet, unquiet, status
- Suppress AI responses during quiet mode in events.js
- Add quietMode section to config.json (disabled by default)
- Add quietMode to configAllowlist.js for dashboard editing

* test: add quiet mode tests (41 tests, all passing)

* style: fix biome formatting in quietMode.js, events.js, and test

* fix(web): fix ai-feedback-stats TypeScript and formatting errors

* fix: gate quiet mode checks on enabled flag, validate TTL, honor maxDurationMinutes config

- events.js: Wrap isQuietMode() calls in guildConfig.quietMode?.enabled check
  to avoid unnecessary Redis lookups and prevent stale records from suppressing
  AI responses when the feature is disabled (PRRT_kwDORICdSM5xdbmp, PRRT_kwDORICdSM5xdbmx)

- quietMode.js: Add TTL validation in setQuiet() to guard against 0, negative,
  or NaN values that would error in Redis (PRRT_kwDORICdSM5xdbm3)

- quietMode.js: Update parseDurationFromContent() to accept config parameter
  and honor guildConfig.quietMode.maxDurationMinutes. Also clamp defaultSeconds
  to the effective max (PRRT_kwDORICdSM5xdbm_)

- configValidation.js: Add quietMode schema entry with enabled, maxDurationMinutes,
  and allowedRoles properties (PRRT_kwDORICdSM5xdbnH)

* style: fix biome formatting in quietMode.js and ai-feedback-stats.tsx

* feat: audit log improvements — CSV/JSON export and real-time WebSocket stream (#215)

* feat: audit log improvements — CSV/JSON export, real-time WebSocket stream

- Add GET /:id/audit-log/export endpoint (CSV and JSON, up to 10k rows)
- Add /ws/audit-log WebSocket server for real-time audit entry broadcast
- Refactor buildFilters() shared helper to eliminate duplication
- Hook broadcastAuditEntry() into insertAuditEntry (RETURNING id+created_at)
- Wire setupAuditStream/stopAuditStream into startServer/stopServer lifecycle
- Add escapeCsvValue/rowsToCsv helpers with full test coverage
- 30 route tests + 17 WebSocket stream tests, all green

Closes #136

* fix: PR #215 review feedback - audit stream fixes

- ws.ping() crash: guard with readyState check + try/catch to avoid
  crashing heartbeat interval when socket not OPEN
- stopAuditStream race: make setupAuditStream async and await
  stopAuditStream() to prevent concurrent WebSocketServer creation
- Query param array coercion: add typeof === 'string' checks for
  startDate/endDate to handle Express string|string[]|undefined
- CSV CRLF quoting: add \r to RFC 4180 special-char check for proper
  Windows line ending handling
- Test timeouts: make AUTH_TIMEOUT_MS configurable via
  AUDIT_STREAM_AUTH_TIMEOUT_MS env var, use 100ms in tests

* feat: voice channel activity tracking — join/leave/move, leaderboard, export (#212)

* feat: add voice_sessions migration (#135)

* feat: add voice tracking module — join/leave/move/flush/leaderboard (#135)

* feat: wire voiceStateUpdate handler into event registration (#135)

* feat: add /voice command — leaderboard, stats, export subcommands (#135)

* feat: add voice config defaults to config.json (#135)

* feat: wire voice flush start/stop into bot lifecycle (#135)

* feat: add voice to config API allowlist (#135)

* fix: SQL UPDATE subquery for closeSession, fix import order (#135)

* fix(voice): resolve race conditions and missing config schema

- Fix openSession: update in-memory state only AFTER DB INSERT succeeds
- Fix closeSession: delete from in-memory state only AFTER DB UPDATE succeeds
- Fix: allow closeSession on leave/move even when feature is disabled
- Fix migration: add UNIQUE constraint to partial index to prevent duplicates
- Fix: move 'Voice join' log to after openSession succeeds
- Add voice config to CONFIG_SCHEMA for validation

---------

Co-authored-by: Bill <bill@example.com>

* feat(dashboard): auto-save config with 500ms debounce (#199)

* feat(dashboard): replace manual save with auto-save (500ms debounce)

- Remove 'Save Changes' button; saving now fires automatically 500ms
  after the last config change (no changes → no network call)
- Add saveStatus state ('idle' | 'saving' | 'saved' | 'error') with
  AutoSaveStatus component showing spinner, check, or error+retry
- Add isLoadingConfigRef guard so the initial fetchConfig load never
  triggers a spurious PATCH
- Ctrl+S still works: clears debounce timer and saves immediately
- Keep 'beforeunload' warning for validation errors that block save
- Replace yellow unsaved-changes banner with a destructive validation
  error banner (only shown when save is actually blocked)
- Error state shows 'Save failed' + 'Retry' button for user recovery

Closes #189

* test(dashboard): add auto-save tests for ConfigEditor

- No PATCH on initial config load
- Validation error banner suppresses auto-save
- 'Saving...' spinner visible while PATCH in-flight
- 'Save failed' + Retry button on PATCH error

* fix(dashboard): prevent fetchConfig from overwriting saveStatus after successful save

Add skipSaveStatusReset parameter to fetchConfig so that post-save reloads
preserve the 'saved' status indicator instead of immediately resetting to 'idle'.

* test(dashboard): use fake timers, restore vi.stubGlobal, fix assertions, add idle/saved coverage

- Replace real setTimeout delays with vi.useFakeTimers() + vi.advanceTimersByTimeAsync()
  for deterministic, fast debounce tests
- Add afterEach cleanup: vi.unstubAllGlobals() + vi.useRealTimers()
- Replace toBeTruthy() with toBeInTheDocument() for Testing Library queries
- Add idle state test (no status indicator shown after load)
- Add saved state test (shows 'Saved' after successful save)
- Update file-level comment to list all four states

---------

Co-authored-by: Bill Chirico <bill@volvox.gg>

* feat: Reaction role menus (#162) (#205)

* feat: reaction role menus - core module, command, event hooks, migration

Implements issue #162: reaction role menus.

- Add migration 004 creating reaction_role_menus and reaction_role_entries tables
- Add src/modules/reactionRoles.js with DB helpers, embed builder, event handlers
- Add src/commands/reactionrole.js with /reactionrole create|add|remove|delete|list
- Wire handleReactionRoleAdd/Remove into registerReactionHandlers in events.js

Roles are granted on reaction add and revoked on reaction remove.
All mappings persist in PostgreSQL across bot restarts.

* test: reaction role menus - 40 tests covering module and command

- tests/modules/reactionRoles.test.js: resolveEmojiString, buildReactionRoleEmbed,
  all DB helpers, handleReactionRoleAdd, handleReactionRoleRemove
- tests/commands/reactionrole.test.js: all 5 subcommands (create, add, remove,
  delete, list) including error paths and guild ownership checks
- Fix biome lint: import sort order + unused import removal

* fix: remove unused import in reactionrole command

---------

Co-authored-by: Bill Chirico <bill@volvox.gg>

* fix(security): validate GitHub owner/repo format before gh CLI call (#198)

* fix(security): validate GitHub owner/repo format before gh CLI call

Prevents API path traversal by validating owner/repo segments against
a strict allowlist regex before interpolating them into the gh CLI
invocation.

Adds:
- VALID_GH_NAME regex (/^[a-zA-Z0-9._-]+$/)
- isValidGhRepo() helper (exported for testing)
- Guard in fetchRepoEvents() — returns [] and warns on invalid input
- Strengthened guard in pollGuildFeed() split logic

Fixes #160

* test(security): add validation tests for GitHub owner/repo format

Covers isValidGhRepo(), VALID_GH_NAME regex, and fetchRepoEvents()
validation guard introduced in fix for #160.

19 new tests verify:
- Valid alphanumeric/dot/hyphen/underscore names pass
- Path traversal (../../etc/passwd) is rejected at both entry points
- Slashes, empty strings, non-strings, spaces all rejected
- Shell metacharacters (; && $()) blocked
- gh CLI is NOT invoked when validation fails
- warn() fires with the invalid values (observable audit trail)
- Valid owner/repo still reach gh CLI unchanged

* fix(security): reject pure-dot owner/repo names to prevent path traversal

* test(githubFeed): add tests for pure-dot path traversal bypass

---------

Co-authored-by: Bill Chirico <bill@volvox.gg>

---------

Co-authored-by: Bill <bill@example.com>
Co-authored-by: Bill Chirico <bill@volvox.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: custom commands and auto-responders — configurable via dashboard

2 participants