Skip to content

conhecendoia/coolify-auto-upgrade

Coolify Auto-Upgrade

CI Tests License: MIT GitHub release GitHub issues

Production-ready automated upgrade system for Coolify with health checks, automatic rollback, and zero-downtime deployments.

Repository: https://github.com/conhecendocontato/coolify-auto-upgrade | License: MIT | Version: 1.1.0+

Features

  • Automated Upgrades - Checks for and applies Coolify updates weekly
  • Safe Backups - Automatic backup before every upgrade with symlink-protected cleanup
  • Auto-Rollback - Reverts on health check failure
  • Health Verification - Multi-layer health checks before marking success
  • Notifications - Telegram alerts for upgrade status
  • Configurable - Extensive configuration options for different deployments
  • Dry-Run Mode - Test without making changes
  • Production-Ready - Comprehensive safety checks, signal handling, and input validation

Quick Start

Prerequisites

  • Docker and Docker Compose
  • bash 4.0+
  • curl, jq
  • Running Coolify instance
  • Root or sudo access

Installation

# Clone repository
git clone https://github.com/conhecendocontato/coolify-auto-upgrade.git /opt/coolify-auto-upgrade
cd /opt/coolify-auto-upgrade

# Make executable
chmod +x coolify-upgrade.sh

# Configure (edit configs/upgrade.conf)
cp configs/upgrade.conf.example configs/upgrade.conf
$EDITOR configs/upgrade.conf

# Test in dry-run mode
./coolify-upgrade.sh --dry-run

# Add to crontab for weekly checks
(crontab -l 2>/dev/null; echo "0 4 * * 6 /opt/coolify-auto-upgrade/coolify-upgrade.sh >> /var/log/coolify-auto-upgrade.log 2>&1") | crontab -

Usage

# Normal upgrade
./coolify-upgrade.sh

# Dry run (no changes)
./coolify-upgrade.sh --dry-run

# Check for updates only
./coolify-upgrade.sh --check-only

# Force upgrade to latest
./coolify-upgrade.sh --force

# Verbose logging
./coolify-upgrade.sh --verbose

# Manual rollback to specific backup
./coolify-upgrade.sh --rollback /data/coolify-backups/20240113_040000

Configuration

Using .env File (Recommended)

Copy .env.example to .env and customize:

cp .env.example .env
nano .env

The .env file provides a comprehensive template with all available options and documentation.

Configuration File

Alternatively, use configs/upgrade.conf:

# Health check settings
HEALTH_CHECK_TIMEOUT=300      # Max seconds to wait for health
HEALTH_CHECK_INTERVAL=5        # Seconds between checks

# Backup settings
BACKUP_RETENTION_DAYS=30       # Days to keep backups
BACKUP_BASE_DIR=/data/coolify-backups

# Docker settings (customizable for different deployments)
COOLIFY_CONTAINER_NAME=coolify
COOLIFY_PORT=8080
COOLIFY_DIR=/data/coolify/source
COOLIFY_COMPOSE=/data/coolify/source/docker-compose.yml

# Related containers (for multi-container setups)
COOLIFY_REDIS_CONTAINER=coolify-redis
COOLIFY_DB_CONTAINER=coolify-db
COOLIFY_REALTIME_CONTAINER=coolify-realtime
COOLIFY_SENTINEL_CONTAINER=coolify-sentinel

# Feature flags
DEBUG=0                       # Enable debug logging
DRY_RUN=0                     # Dry run mode (no changes)
AUTO_CLEANUP=1                # Clean old backups/images

Architecture

Directory Structure

/opt/coolify-auto-upgrade/
├── coolify-upgrade.sh          # Main entry point
├── lib/
│   ├── common.sh               # Core utilities (logging, config, locks)
│   ├── version.sh              # Version management
│   ├── backup.sh               # Backup & rollback (symlink-safe)
│   ├── docker.sh               # Docker operations
│   ├── health.sh               # Health checks
│   ├── validation.sh           # Input validation
│   └── safety.sh               # Pre-flight checks
├── configs/
│   └── upgrade.conf            # Configuration file
├── tests/                      # ShellSpec test suite
└── docs/                       # Additional documentation

Upgrade Workflow

1. Pre-flight checks → safety.sh
   ├─ Disk space check
   ├─ Docker daemon check
   ├─ Container validation
   └─ Configuration validation

2. Check for updates → version.sh
   ├─ Query GitHub Container Registry
   ├─ Fallback to Docker Hub
   └─ Compare versions

3. Create backup → backup.sh
   ├─ Backup docker-compose.yml
   ├─ Backup .env file
   ├─ Backup SSH keys (symlink-safe)
   └─ Record container state

4. Pull image → docker.sh
   ├─ Pull from registry
   └─ Verify image integrity

5. Update compose → docker.sh
   ├─ Backup compose file
   ├─ Update image tag
   └─ Validate syntax

6. Restart containers → docker.sh
   ├─ docker-compose pull
   └─ docker-compose up -d

7. Health check → health.sh
   ├─ Container health status
   ├─ Web interface verification
   ├─ All services check
   └─ Log error analysis

8. Verify success → health.sh
   └─ Comprehensive health check

9. Cleanup → backup.sh, docker.sh
   ├─ Remove old backups (symlink-safe)
   └─ Remove old images

Any failure → rollback to backup

Safety Features

  1. Symlink Protection: All file operations validate paths aren't symlinks
  2. TOCTOU Protection: Path validation with real path resolution
  3. Signal Handlers: Clean exit on SIGINT/SIGTERM/EXIT/ERR
  4. Stale Lock Detection: Automatically removes locks older than 24 hours
  5. Input Validation: All inputs validated before use
  6. Pre-flight Checks: Verify system state before operations
  7. Automatic Rollback: Revert on any health check failure

Troubleshooting

Script hangs

# Check for stale lock file
ls -la /var/run/coolify-auto-upgrade.lock

# Remove if old (> 24 hours - automatic with stale lock detection)
rm -f /var/run/coolify-auto-upgrade.lock

Upgrade failed

# Check logs
tail -100 /var/log/coolify-auto-upgrade.log

# List available backups
ls -la /data/coolify-backups/

# Manual rollback
./coolify-upgrade.sh --rollback /data/coolify-backups/TIMESTAMP

Health check timeout

# Check container status
docker ps -a | grep coolify

# Check container logs
docker logs coolify --tail 100

# Increase timeout in configs/upgrade.conf
HEALTH_CHECK_TIMEOUT=600

Permission denied errors

# Script must be run as root
sudo ./coolify-upgrade.sh

# Or add to sudoers for passwordless execution
echo "username ALL=(ALL) NOPASSWD: /opt/coolify-auto-upgrade/coolify-upgrade.sh" | sudo tee /etc/sudoers.d/coolify-upgrade

Development

Running Tests

# Install ShellSpec
curl -fsSL https://git.io/shellspec | sh -s 0.28.1

# Run tests
shellspec

# Run with coverage
shellspec --coverage

# Run specific test file
shellspec tests/common_spec.sh

Linting

# Run ShellCheck on all files
shellcheck **/*.sh

# Fix common issues (requires shellcheck + shfmt)
shfmt -w -i 4 **/*.sh

Code Style

  • Use set -euo pipefail for strict error handling
  • Always quote variables: "$var" not $var
  • Use [[ ]] for tests, not [ ]
  • Avoid && || patterns - use proper if-then-else
  • Use functions for all code blocks
  • Add ShellCheck directives: # shellcheck source=/dev/null
  • Document all public functions with Args/Returns

Security Considerations

  1. Path Validation: All paths validated for absolute paths, no .., no symlinks
  2. Lock Files: Stored in /var/run with automatic stale detection
  3. File Permissions: Backups respect original permissions
  4. Input Sanitization: All user inputs validated before use
  5. Signal Safety: Proper cleanup on all termination signals
  6. Resource Limits: Timeout protection on network operations

Monitoring

Logs

Main log file: /var/log/coolify-auto-upgrade.log

Health history: /var/log/coolify-health-history.log

Upgrade history: /var/log/coolify-upgrade-history.log

Health Status

# Get current health report
source /opt/coolify-auto-upgrade/lib/common.sh
source /opt/coolify-auto-upgrade/lib/health.sh
get_health_report

Backup Status

# List backups
source /opt/coolify-auto-upgrade/lib/backup.sh
list_backups

# Get backup info
get_backup_info /data/coolify-backups/TIMESTAMP

# Check disk usage
get_backup_disk_usage

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests for new functionality
  5. Ensure all tests pass: shellspec
  6. Ensure ShellCheck passes: shellcheck **/*.sh
  7. Submit a pull request

Please review our Code of Conduct before participating.

Security

Please report security vulnerabilities responsibly following our Security Policy.

License

MIT License - see LICENSE file for details

Support

Changelog

See CHANGELOG.md for version history and changes.

Sponsor this project

Packages

No packages published

Languages