Unified agent logging and fixes queue processor

README.md

@@ -141,7 +141,7 @@ TinyClaw includes `tinyoffice/`, a Next.js web portal for operating TinyClaw fro
 - **Chat Console** - Send messages to default agent, `@agent`, or `@team`
 - **Agents & Teams** - Create, edit, and remove agents/teams
 - **Tasks (Kanban)** - Create tasks, drag across stages, assign to agent/team
-- **Logs & Events** - Inspect queue logs and streaming events
+- **Logs & Events** - Inspect structured runtime logs and streaming events
 - **Settings** - Edit TinyClaw configuration (`settings.json`) via UI
 - **Office View** - Visual simulation of agent interactions
 
@@ -180,6 +180,13 @@ Commands work with `tinyclaw` (if CLI installed) or `./tinyclaw.sh` (direct scri
 | `logs [type]` | View logs (discord/telegram/whatsapp/queue/heartbeat/all) | `tinyclaw logs queue` |
 | `attach`      | Attach to tmux session                                    | `tinyclaw attach`     |
 
+### Logging
+
+- TinyClaw uses structured JSON logs backed by `pino` for Node runtimes.
+- Set `LOG_LEVEL=debug|info|warn|error` before starting TinyClaw to control verbosity.
+- `/api/logs` returns merged historical entries across `queue`, `api`, `telegram`, `discord`, `whatsapp`, `daemon`, and `heartbeat`.
+- Log files rotate at `10 MB` with `5` retained archives per source.
+
 ### Agent Commands
 
 | Command                               | Description                     | Example                                                      |
@@ -423,7 +430,7 @@ tinyclaw/
 │   │   ├── incoming/
 │   │   ├── processing/
 │   │   └── outgoing/
-│   ├── logs/             # All logs
+│   ├── logs/             # Structured NDJSON logs + rotated archives
 │   ├── channels/         # Channel state
 │   ├── files/            # Uploaded files
 │   ├── pairing.json      # Sender allowlist state (pending + approved)

docs/QUEUE.md

@@ -251,6 +251,7 @@ The API server runs on port 3777 (configurable via `TINYCLAW_API_PORT`):
 | `POST /api/message` | Enqueue a message |
 | `GET /api/queue/status` | Queue depth (pending, processing, dead) |
 | `GET /api/responses` | Recent responses |
+| `GET /api/logs` | Unified structured log history across queue, API, channels, daemon, and heartbeat |
 | `GET /api/queue/dead` | Dead messages |
 | `POST /api/queue/dead/:id/retry` | Retry a dead message |
 | `DELETE /api/queue/dead/:id` | Delete a dead message |
@@ -264,6 +265,13 @@ Periodic cleanup tasks run automatically:
 - **Acked response pruning**: Every hour (responses acked > 24h ago)
 - **Conversation TTL**: Every 30 minutes (team conversations older than 30 min)
 
+## Logging
+
+- Node runtimes write structured NDJSON logs with `pino`.
+- `LOG_LEVEL=debug|info|warn|error` controls verbosity for queue, API, and channel clients.
+- `/api/logs` merges current and rotated files for `queue`, `api`, `telegram`, `discord`, `whatsapp`, `daemon`, and `heartbeat`.
+- Log files rotate at `10 MB` and retain the previous `5` archives per source.
+
 ## Debugging
 
 ### Check Queue Status

docs/RESPONSE-DELIVERY-OPTIONS.md

@@ -0,0 +1,116 @@
+# Response Delivery Options for `heartbeat`, `web`, and `cli`
+
+## Problem
+
+TinyClaw currently writes outgoing responses for all channels into the `responses` table, but only the Telegram, Discord, and WhatsApp channel clients poll `/api/responses/pending` and call `/api/responses/:id/ack`.
+
+That leaves these channels with no delivery consumer:
+
+- `heartbeat`
+- `web`
+- `cli`
+
+As a result, their rows remain `status = 'pending'` indefinitely even when the response was already useful for logging or operator visibility.
+
+## Current State
+
+- `heartbeat` responses are generated by the queue processor and only observed indirectly by `lib/heartbeat-cron.sh` via `GET /api/responses?limit=20`.
+- `web` responses are visible in logs and TinyOffice data, but there is no response delivery worker or ack path for them.
+- `cli` responses can be inspected through the API and DB, but there is no terminal-side consumer that acks them.
+
+## Option 1: Auto-ack non-delivery channels after enqueue
+
+### Design
+
+Treat `heartbeat`, `web`, and `cli` as non-delivery channels. Continue writing their responses to the `responses` table for audit/history, but immediately mark them `acked` after enqueue.
+
+### Pros
+
+- Smallest code change
+- `responsesPending` becomes a real signal for only deliverable channels
+- No long-term buildup from pseudo-channels
+- Keeps historical response rows
+
+### Cons
+
+- You lose visibility into "not yet viewed" versus "already viewed"
+- `acked` stops meaning "delivered to an external user" and becomes "no further delivery required"
+
+### Best fit
+
+Use this if `heartbeat`, `web`, and `cli` are primarily observability/operator channels, not real outbound transport queues.
+
+## Option 2: Add explicit consumers and ack on read
+
+### Design
+
+Keep all responses pending until a channel-specific consumer reads them:
+
+- TinyOffice web UI fetches pending `web` responses and acks after display
+- CLI tooling fetches pending `cli` responses and acks after printing
+- heartbeat runner fetches pending `heartbeat` responses and acks after logging
+
+### Pros
+
+- Preserves strict queue semantics
+- `pending` accurately means "not yet consumed"
+- Best model if those channels should behave like real delivery targets
+
+### Cons
+
+- Most code and coordination work
+- Web/CLI delivery semantics need to be defined precisely
+- UI refreshes and polling create ambiguity around what counts as "consumed"
+
+### Best fit
+
+Use this if you want `web`, `cli`, and `heartbeat` to behave as first-class delivery channels with lifecycle guarantees.
+
+## Option 3: Split delivery queues from audit records
+
+### Design
+
+Only enqueue into `responses` for channels that have delivery workers. For `heartbeat`, `web`, and `cli`, store final outputs somewhere else:
+
+- structured logs
+- chat history files
+- a new `response_history` table
+
+### Pros
+
+- Cleanest semantics long term
+- `responses` remains a true delivery queue
+- Avoids overloading `acked`
+
+### Cons
+
+- Larger schema and API change
+- Requires migration of any UI that currently expects all outputs in `responses`
+- More implementation effort than Option 1
+
+### Best fit
+
+Use this if you want a clean architectural separation between "deliver this" and "record that it happened."
+
+## Recommendation
+
+Recommend **Option 1** first.
+
+It solves the operational problem quickly:
+
+- `responsesPending` becomes meaningful again
+- backlog metrics stop being polluted by non-delivery channels
+- response history is still preserved in SQLite
+
+If later you want inbox-like behavior for `web` or `cli`, you can move to Option 2 or Option 3 with clearer semantics.
+
+## Suggested Decision
+
+Short term:
+
+- implement Option 1 for `heartbeat`, `web`, and `cli`
+
+Medium term:
+
+- decide whether `web` should graduate to a true consumer flow
+- if yes, move `web` to Option 2 while leaving `heartbeat` on Option 1

lib/common.sh

@@ -86,9 +86,121 @@ get_channel_token() {
     done
 }
 
-# Logging function
+# Structured log helpers
+rotate_log_file() {
+    local file="$1"
+    local max_bytes=$((10 * 1024 * 1024))
+    local max_files=5
+
+    [ -f "$file" ] || return 0
+
+    local size
+    size=$(wc -c < "$file" | tr -d ' ')
+    if [ "$size" -lt "$max_bytes" ]; then
+        return 0
+    fi
+
+    local ext="${file##*.}"
+    local base="${file%.*}"
+    local i
+    for ((i=max_files; i>=1; i--)); do
+        local current="${base}.${i}.${ext}"
+        local previous
+        if [ "$i" -eq 1 ]; then
+            previous="$file"
+        else
+            previous="${base}.$((i-1)).${ext}"
+        fi
+
+        [ -f "$previous" ] || continue
+        [ ! -f "$current" ] || rm -f "$current"
+        mv "$previous" "$current"
+    done
+}
+
+write_structured_log() {
+    local source="$1"
+    local component="$2"
+    local level="$3"
+    shift 3
+
+    local msg="$*"
+    local file="$LOG_DIR/${source}.log"
+    local timestamp
+    timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
+
+    mkdir -p "$LOG_DIR"
+    rotate_log_file "$file"
+
+    if command -v jq >/dev/null 2>&1; then
+        jq -nc \
+            --arg time "$timestamp" \
+            --arg level "$level" \
+            --arg source "$source" \
+            --arg component "$component" \
+            --arg msg "$msg" \
+            '{time:$time,level:$level,source:$source,component:$component,msg:$msg}' >> "$file"
+    else
+        node -e 'const [time, level, source, component, msg] = process.argv.slice(1); console.log(JSON.stringify({ time, level, source, component, msg }));' \
+            "$timestamp" "$level" "$source" "$component" "$msg" >> "$file"
+    fi
+}
+
+normalize_log_level() {
+    local raw
+    raw=$(printf '%s' "${1:-info}" | tr '[:upper:]' '[:lower:]')
+    case "$raw" in
+        trace|verbose) echo "debug" ;;
+        debug) echo "debug" ;;
+        info|"") echo "info" ;;
+        warn|warning) echo "warn" ;;
+        error|err|fatal) echo "error" ;;
+        *) echo "info" ;;
+    esac
+}
+
+log_level_priority() {
+    case "$(normalize_log_level "$1")" in
+        debug) echo 0 ;;
+        info) echo 1 ;;
+        warn) echo 2 ;;
+        error) echo 3 ;;
+        *) echo 1 ;;
+    esac
+}
+
 log() {
-    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_DIR/daemon.log"
+    local candidate_level="${1:-}"
+    local level="info"
+    local threshold
+    local msg
+
+    case "$(normalize_log_level "$candidate_level")" in
+        debug|info|warn|error)
+            if [ "$candidate_level" = "$(normalize_log_level "$candidate_level")" ] || \
+               [ "$candidate_level" = "DEBUG" ] || [ "$candidate_level" = "INFO" ] || \
+               [ "$candidate_level" = "WARN" ] || [ "$candidate_level" = "WARNING" ] || \
+               [ "$candidate_level" = "ERROR" ] || [ "$candidate_level" = "verbose" ] || \
+               [ "$candidate_level" = "VERBOSE" ] || [ "$candidate_level" = "trace" ] || \
+               [ "$candidate_level" = "TRACE" ] || [ "$candidate_level" = "fatal" ] || \
+               [ "$candidate_level" = "FATAL" ] || [ "$candidate_level" = "err" ] || \
+               [ "$candidate_level" = "ERR" ]; then
+                level="$(normalize_log_level "$candidate_level")"
+                shift
+            fi
+            ;;
+    esac
+
+    msg="$*"
+    [ -n "$msg" ] || return 0
+
+    threshold="$(normalize_log_level "${LOG_LEVEL:-info}")"
+    if [ "$(log_level_priority "$level")" -lt "$(log_level_priority "$threshold")" ]; then
+        return 0
+    fi
+
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $msg"
+    write_structured_log "daemon" "daemon" "$level" "$msg"
 }
 
 # Load settings from JSON

lib/daemon.sh

@@ -18,18 +18,23 @@ start_daemon() {
         PUPPETEER_SKIP_DOWNLOAD=true npm install
     fi
 
-    # Build TypeScript if any src file is newer than its dist counterpart
+    # Build TypeScript if any source file is newer than its dist counterpart
     local needs_build=false
     if [ ! -d "$SCRIPT_DIR/dist" ]; then
         needs_build=true
     else
-        for ts_file in "$SCRIPT_DIR"/src/*.ts; do
-            local js_file="$SCRIPT_DIR/dist/$(basename "${ts_file%.ts}.js")"
+        while IFS= read -r ts_file; do
+            local relative_path="${ts_file#"$SCRIPT_DIR/src/"}"
+            local js_file="$SCRIPT_DIR/dist/${relative_path%.ts}.js"
             if [ ! -f "$js_file" ] || [ "$ts_file" -nt "$js_file" ]; then
                 needs_build=true
                 break
             fi
-        done
+        done < <(find "$SCRIPT_DIR/src" -type f -name '*.ts' ! -path "$SCRIPT_DIR/src/visualizer/*")
+
+        if [ "$needs_build" = false ] && { [ "$SCRIPT_DIR/tsconfig.json" -nt "$SCRIPT_DIR/dist/queue-processor.js" ] || [ "$SCRIPT_DIR/package.json" -nt "$SCRIPT_DIR/dist/queue-processor.js" ]; }; then
+            needs_build=true
+        fi
     fi
     if [ "$needs_build" = true ]; then
         echo -e "${YELLOW}Building TypeScript...${NC}"
@@ -102,6 +107,11 @@ start_daemon() {
     # Write tokens to .env for the Node.js clients
     local env_file="$SCRIPT_DIR/.env"
     : > "$env_file"
+    if [ -n "${LOG_LEVEL:-}" ]; then
+        local normalized_log_level
+        normalized_log_level="$(normalize_log_level "$LOG_LEVEL")"
+        echo "LOG_LEVEL=${normalized_log_level}" >> "$env_file"
+    fi
     for ch in "${ACTIVE_CHANNELS[@]}"; do
         local env_var
         env_var="$(channel_token_env "$ch")"