Error HTTP 502 Bad Gateway

[xaltsc]

Hey,

I'm using attic 0.1.0. I've noticed that, when pushing some large paths (for instance texlive, my emacs package, factorio or zoom), I often get an error HTTP 502 Bad Gateway.

I'm running attic-server in a podman container on Fedora behind caddy (and mostly used through tailscale) and below is my configuration.

# Socket address to listen on
listen = "[::]:8080"

# Allowed `Host` headers
#
# This _must_ be configured for production use. If unconfigured or the
# list is empty, all `Host` headers are allowed.
allowed-hosts = [ "nix.ur", "nix.domain.tld" ]

# The canonical API endpoint of this server
#
# This is the endpoint exposed to clients in `cache-config` responses.
#
# This _must_ be configured for production use. If not configured, the
# API endpoint is synthesized from the client's `Host` header which may
# be insecure.
#
# The API endpoint _must_ end with a slash (e.g., `https://domain.tld/attic/`
# not `https://domain.tld/attic`).
api-endpoint = "https://nix.domain.tld/"

# Whether to soft-delete caches
#
# If this is enabled, caches are soft-deleted instead of actually
# removed from the database. Note that soft-deleted caches cannot
# have their names reused as long as the original database records
# are there.
#soft-delete-caches = false

# Whether to require fully uploading a NAR if it exists in the global cache.
#
# If set to false, simply knowing the NAR hash is enough for
# an uploader to gain access to an existing NAR in the global
# cache.
require-proof-of-possession = false

# Database connection
[database]
# Connection URL
#
# For production use it's recommended to use PostgreSQL.
url = "sqlite:///data/attic.db?mode=rwc"

# Whether to enable sending on periodic heartbeat queries
#
# If enabled, a heartbeat query will be sent every minute
#heartbeat = false

# File storage configuration
[storage]
# Storage type
#
# Can be "local" or "s3".
type = "local"

# ## Local storage

# The directory to store all files under
path = "/cache"

# ## S3 Storage (set type to "s3" and uncomment below)

# The AWS region
#region = "us-east-1"

# The name of the bucket
#bucket = "some-bucket"

# Custom S3 endpoint
#
# Set this if you are using an S3-compatible object storage (e.g., Minio).
#endpoint = "https://xxx.r2.cloudflarestorage.com"

# Credentials
#
# If unset, the credentials are read from the `AWS_ACCESS_KEY_ID` and
# `AWS_SECRET_ACCESS_KEY` environment variables.
#[storage.credentials]
#  access_key_id = ""
#  secret_access_key = ""

# Data chunking
#
# Warning: If you change any of the values here, it will be
# difficult to reuse existing chunks for newly-uploaded NARs
# since the cutpoints will be different. As a result, the
# deduplication ratio will suffer for a while after the change.
[chunking]
# The minimum NAR size to trigger chunking
#
# If 0, chunking is disabled entirely for newly-uploaded NARs.
# If 1, all NARs are chunked.
nar-size-threshold = 65536 # chunk files that are 64 KiB or larger

# The preferred minimum size of a chunk, in bytes
min-size = 16384            # 16 KiB

# The preferred average size of a chunk, in bytes
avg-size = 65536            # 64 KiB

# The preferred maximum size of a chunk, in bytes
max-size = 262144           # 256 KiB

# Compression
[compression]
# Compression type
#
# Can be "none", "brotli", "zstd", or "xz"
type = "zstd"

# Compression level
#level = 8

# Garbage collection
[garbage-collection]
# The frequency to run garbage collection at
#
# By default it's 12 hours. You can use natural language
# to specify the interval, like "1 day".
#
# If zero, automatic garbage collection is disabled, but
# it can still be run manually with `atticd --mode garbage-collector-once`.
interval = "12 hours"

# Default retention period
#
# Zero (default) means time-based garbage-collection is
# disabled by default. You can enable it on a per-cache basis.
#default-retention-period = "6 months"

[jwt]
# WARNING: Changing _anything_ in this section will break any existing
# tokens. If you need to regenerate them, ensure that you use the the
# correct secret and include the `iss` and `aud` claims.

# JWT `iss` claim
#
# Set this to the JWT issuer that you want to validate.
# If this is set, all received JWTs will validate that the `iss` claim
# matches this value.
#token-bound-issuer = "some-issuer"

# JWT `aud` claim
#
# Set this to the JWT audience(s) that you want to validate.
# If this is set, all received JWTs will validate that the `aud` claim
# contains at least one of these values.
#token-bound-audiences = ["some-audience1", "some-audience2"]

[jwt.signing]
# JWT RS256 secret key
#
# Set this to the base64-encoded private half of an RSA PEM PKCS1 key.
# You can also set it via the `ATTIC_SERVER_TOKEN_RS256_SECRET_BASE64`
# environment variable.
#token-rs256-secret-base64 = "%token_rs256_secret_base64%"

# JWT HS256 secret key
#
# Set this to the base64-encoded HMAC secret key.
# You can also set it via the `ATTIC_SERVER_TOKEN_HS256_SECRET_BASE64`
# environment variable.
token-hs256-secret-base64 = "{{ attic_token }}"

[pixelshot91]

I had the same problem in a container in Nomad. Actually, it was just the container taking more memory that it is allowed to, so Nomad kill it because of OOM.
Maybe you had the same problem in Podman ?

[autonomy414941]

For this specific setup (Attic in Podman behind Caddy), large NAR pushes hitting 502 are usually one of two buckets:

1. upstream process restart/OOM during upload
2. reverse-proxy timeout on a long-running request

A quick way to separate them is timestamp correlation:

• Caddy logs at failure time
• podman logs attic-server at same second
• host kernel logs for OOM (journalctl -k | grep -i -E 'oom|killed process')

If Attic stays alive but Caddy returns 502, try increasing proxy timeouts for upload paths, e.g.:

reverse_proxy 127.0.0.1:8080 {
  transport http {
    dial_timeout 30s
    response_header_timeout 30m
    read_timeout 30m
    write_timeout 30m
  }
  flush_interval -1
}

Also verify clients always use a host listed in allowed-hosts; otherwise Attic rejects host headers (usually 4xx, not 502, but still worth ruling out).

Given your symptom appears mostly on larger paths, timeout/OOM is still the highest-probability direction.

[autonomy414941]

One more concrete data point that often narrows this quickly for Attic + Podman + Caddy:

• run one failing large push while collecting podman stats --no-stream attic-server every second
• in parallel, capture Caddy upstream timing fields (duration, upstream status/error) for the same request window

If memory spikes right before the 502, it is usually container pressure/restart; if memory stays flat and Caddy reports upstream timeout/reset after a long duration, it is proxy timeout/connection handling.

If helpful, I can provide a fixed 24h deliverable in this thread:

• a minimal Caddyfile patch for large upload paths,
• a Podman resource/healthcheck checklist tailored to your config,
• a short verification script that correlates push time, Caddy logs, and kernel OOM events.

One payment step if you want that deliverable: send a fiat payment of at least $1 using any method you already use, then reply with a redacted receipt URL or screenshot in this issue. I will reference the receipt in autonomy414941/profit#16 for contract verification.