Skip to content

fix: logs cleanup#41275

Merged
wyattwalter merged 2 commits intoreleasefrom
ww-logging-cleanup
Oct 3, 2025
Merged

fix: logs cleanup#41275
wyattwalter merged 2 commits intoreleasefrom
ww-logging-cleanup

Conversation

@wyattwalter
Copy link
Contributor

@wyattwalter wyattwalter commented Oct 2, 2025

Description

Tip

Add a TL;DR when the description is longer than 500 words or extremely technical (helps the content, marketing, and DevRel team).

Please also include relevant motivation and context. List any dependencies that are required for this change. Add links to Notion, Figma or any other documents that might be relevant to the PR.

Addresses at least a couple of issues in a number of support tickets about logs volume:

  • we were double-logging all messages via Supervisor's eventlistener:stdout configuration. Once to the sub-process's logs, and once to another file in the logs/supervisor directory. The purpose of this listener is to send logs to stdout/stderr so they can be picked up by log aggregation services, no need to write again.
  • we had debug logs enabled for Caddy which was creating quite a bit of log volume in logs/editor/<hostname>-stderr.log
  • bonus fix: in a multi-container deployment, all containers were trying to write to logs/supervisor/supervisord.log making trying to troubleshoot those deployments more difficult.

Warning

If no issue exists, please create an issue first, and check with the maintainers if the issue is valid.

Automation

/ok-to-test tags="@tag.Sanity"

🔍 Cypress test results

Tip

🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/18222964844
Commit: 54b5a1a
Cypress dashboard.
Tags: @tag.Sanity
Spec:


Fri, 03 Oct 2025 13:38:52 UTC

Communication

Should the DevRel and Marketing teams inform users about this change?

  • Yes
  • No

Summary by CodeRabbit

  • Chores
    • Simplified logging to route process output to standard output with hostname tagging, reducing per-file logs and disk usage.
    • Improved reliability of log capture with a dedicated stdout event handler.
    • Reduced log noise by disabling debug logging in the web server configuration.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 2, 2025

Walkthrough

Updates supervisor logging: switches main logfile path, null-routes stdout/stderr files, and configures an event listener with a Python result handler for PROCESS_LOG events. Removes Caddy debug directive from the generated Caddyfile. No public API or exported entity changes.

Changes

Cohort / File(s) Summary
Supervisor logging and event listener
deploy/docker/fs/etc/supervisor/supervisord.conf
Changed supervisor logfile to %(ENV_HOSTNAME)s-stdout.log under supervisor log dir; set stdout/stderr logfiles to /dev/null and removed related rotation options; added/clarified eventlistener block with command and comments; retained buffer_size, events, and result_handler.
Caddy config tweak
deploy/docker/fs/opt/appsmith/caddy-reconfigure.mjs
Removed debug directive from generated Caddyfile server block.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Proc as Managed Process
  participant Sup as supervisord
  participant EL as stdout EventListener (Python handler)
  note over Sup: stdout/stderr files -> /dev/null
  Proc->>Sup: Emits PROCESS_LOG events
  Sup->>EL: Dispatch PROCESS_LOG events
  EL-->>Sup: result_handler responses (success/ack)
  note right of EL: Handles log events per new listener command
  Sup->>Sup: Writes main log to %(ENV_HOSTNAME)s-stdout.log
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

Logs now whisper into the void, serene,
A listener wakes where lines convene.
Caddy sheds its debug cloak at night,
Configs align—concise and tight.
Hosts inscribe their names in streams,
Quiet pipes, well-ordered dreams.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Description Check ⚠️ Warning The PR description includes the required Description, Automation, Cypress test results, and Communication sections but omits the mandatory “Fixes” line that links this change to an issue number or URL, which is required by the repository template for traceability. Without this reference, it is unclear which issue the PR addresses. As a result, the description does not fully conform to the template. Please add a “Fixes #” or “Fixes ” line under the Description section to reference the issue this PR addresses and satisfy the repository’s template requirements.
✅ Passed checks (2 passed)
Check name Status Explanation
Title Check ✅ Passed The title uses a clear conventional-commit prefix and concisely summarizes the primary change—cleaning up excessive logging—making it immediately clear to reviewers what the PR addresses.
Docstring Coverage ✅ Passed No functions found in the changes. Docstring coverage check skipped.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ww-logging-cleanup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions bot added the Bug Something isn't working label Oct 2, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4df6b92 and 54b5a1a.

📒 Files selected for processing (2)
  • deploy/docker/fs/etc/supervisor/supervisord.conf (2 hunks)
  • deploy/docker/fs/opt/appsmith/caddy-reconfigure.mjs (0 hunks)
💤 Files with no reviewable changes (1)
  • deploy/docker/fs/opt/appsmith/caddy-reconfigure.mjs
🔇 Additional comments (2)
deploy/docker/fs/etc/supervisor/supervisord.conf (2)

13-13: No further action needed: %(ENV_HOSTNAME)s references the container’s HOSTNAME, which is always set in Docker, Kubernetes, Cloud Run, and Heroku.


38-40: Handler module exists as referenced
The file deploy/docker/fs/usr/lib/python3/dist-packages/supervisor/appsmith_supervisor_stdout.py is present and correctly referenced in supervisord.conf.

Comment on lines +45 to +46
stdout_logfile=/dev/null
stderr_logfile=/dev/null
Copy link
Contributor

@coderabbitai coderabbitai bot Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Ensure error handling in the Python handler to prevent log loss.

Routing stdout/stderr to /dev/null prevents double-logging as intended. However, if the Python handler in line 41 fails or crashes, logs would be permanently lost. Ensure the handler has robust error handling and consider adding monitoring/alerting for handler failures.

Run the following script to check for error handling in the Python handler:


🏁 Script executed:

#!/bin/bash
# Description: Check error handling in the appsmith_supervisor_stdout module

# Find and examine the Python handler for error handling patterns
fd -t f 'appsmith_supervisor_stdout.py' --exec cat {} \; | rg -C5 'try:|except:|logging.error|raise'

# Check for any supervisor event handler error logs or monitoring
rg -n 'event_handler|PROCESS_LOG.*error|supervisor.*error' --type=py -g 'deploy/**'

Length of output: 400


🏁 Script executed:

#!/bin/bash
rg -n -C10 'def event_handler' deploy/docker/fs/usr/lib/python3/dist-packages/supervisor/appsmith_supervisor_stdout.py

Length of output: 768


Add error handling to event_handler
Wrap the logic in deploy/docker/fs/usr/lib/python3/dist-packages/supervisor/appsmith_supervisor_stdout.py lines 24–28 in a try/except and emit any exceptions to stderr or a fallback logger to prevent silent failures and lost logs.

🤖 Prompt for AI Agents
deploy/docker/fs/usr/lib/python3/dist-packages/supervisor/appsmith_supervisor_stdout.py
around lines 24–28: the event_handler logic must be wrapped in a try/except to
avoid silent failures; surround the existing body with a broad try block, catch
Exception as e, emit the exception details to stderr (or use a fallback logger
via logging.exception) and ensure the handler returns or continues gracefully so
logs are not lost.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not completely wrong.. but this is also the case if we were to continue persisting these to disk in the way we were before this proposed change.

Whatever exception is thrown would go to stderr of the container, so it's not really lost. And we don't gain a lot from continuing to write it to disk either.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wyattwalter, you're absolutely right—this failure mode was already present before the change, and routing to /dev/null doesn't introduce new risk. Since exceptions from the handler would still appear in the container's stderr, observability is maintained without the double-logging overhead. Thank you for the clarification!

@wyattwalter
Copy link
Contributor Author

/build-deploy-preview skipTests=True

@github-actions
Copy link

github-actions bot commented Oct 3, 2025

Deploying Your Preview: https://github.com/appsmithorg/appsmith/actions/runs/18222774846.
Workflow: On demand build Docker image and deploy preview.
skip-tests: . env: .
PR: 41275.
recreate: .

@pratapaprasanna pratapaprasanna added the ok-to-test Required label for CI label Oct 3, 2025
@github-actions
Copy link

github-actions bot commented Oct 3, 2025

Deploy-Preview-URL: https://ce-41275.dp.appsmith.com

@wyattwalter wyattwalter merged commit 336d318 into release Oct 3, 2025
49 checks passed
@wyattwalter wyattwalter deleted the ww-logging-cleanup branch October 3, 2025 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Something isn't working ok-to-test Required label for CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants