DJLIB - DJ Library Management System

A comprehensive Python tool for managing, organizing, and maintaining DJ music libraries with advanced duplicate detection, metadata normalization, and Serato integration.

Project Board: https://github.com/users/E7H31234L/projects/3

🎯 Overview

DJLIB provides a complete pipeline for managing large music libraries with features including:

• Audio file scanning and probing with FFmpeg/FFprobe integration
• Metadata extraction and normalization using Mutagen and ExifTool
• Advanced duplicate detection with audio fingerprinting
• Key analysis via keyfinder-cli and BPM/energy analysis via Essentia (optional)
• Serato-optimized tagging with Camelot key detection
• Safe file operations with automatic backup and quarantine
• Progress tracking and resumable workflows

🚀 Quick Start

Prerequisites

# Required tools
sudo apt install ffmpeg ffprobe sqlite3

# Optional but recommended
sudo apt install fpcalc essentia-streaming-extractor keyfinder-cli
pip install mutagen tqdm

# Web UI (optional)
pip install fastapi uvicorn rq redis pydantic

Installation

# Clone and install
git clone https://github.com/E7H31234L/djlib_cleanup.git
cd djlib-cleanup
chmod +x scripts/djlib.py scripts/preFlight_djlib_check.py

# Create symlink for system-wide access
mkdir -p ~/.local/bin
ln -s "$(pwd)/scripts/djlib.py" ~/.local/bin/djlib
ln -s "$(pwd)/scripts/preFlight_djlib_check.py" ~/.local/bin/djlib-preflight

First Run

# Validate environment
djlib-preflight --music-root "/path/to/your/music"

# Run analysis pipeline (safe, no changes)
djlib --music-root "/path/to/your/music" run --resume

# Skip analysis step (faster runs)
djlib --music-root "/path/to/your/music" run --no-analysis

# Review what would be changed
djlib --music-root "/path/to/your/music" summary --json

# Apply repairs safely (with automatic backups)
djlib --music-root "/path/to/your/music" --apply repair remux --resume

📋 Commands

Core Pipeline Commands

# Complete end-to-end workflow
djlib --music-root "/path/to/music" run

# Resume interrupted workflow
djlib --music-root "/path/to/music" resume

# Individual pipeline stages
djlib --music-root "/path/to/music" scan          # Discover files
djlib --music-root "/path/to/music" probe         # Analyze audio properties
djlib --music-root "/path/to/music" tags          # Extract metadata
djlib --music-root "/path/to/music" analysis      # BPM/energy/key analysis
djlib --music-root "/path/to/music" dedupe        # Find duplicates

Management Commands

# Duplicate removal (safe - moves to quarantine)
djlib --music-root "/path/to/music" --apply dedupe-filenames
djlib --music-root "/path/to/music" --apply dedupe-titles

# Metadata normalization
djlib --music-root "/path/to/music" --apply normalize-tags
djlib --music-root "/path/to/music" --apply normalize-filenames

# Quarantine management
djlib --music-root "/path/to/music" --apply quarantine

# Library statistics and reporting
djlib --music-root "/path/to/music" summary
djlib --music-root "/path/to/music" logs
djlib --music-root "/path/to/music" progress

Analysis and Debugging

# File-level investigation
djlib --music-root "/path/to/music" explain 12345

# Artist alias management
djlib --music-root "/path/to/music" artist-aliases --search "Daft Punk"

# Interactive troubleshooting
djlib --music-root "/path/to/music" runbook

# Compliance checking
djlib --music-root "/path/to/music" compliance-check

🗂️ Workspace Structure

DJLIB creates a structured workspace in ~/.local/share/djlib/:

~/.local/share/djlib/
|-- manifest.sqlite                 # Main database
|-- workspaces/
|   `-- run_20240124_143000/         # Timestamped run folders
|       |-- probes/                 # Audio analysis results
|       |-- repairs/                # File repair operations
|       |-- backups/                # Original file backups
|       |-- logs/                   # Detailed operation logs
|       `-- checkpoints/            # Progress state files
`-- quarantine/                     # Duplicate/junk files

🔧 Configuration

Environment Variables

export MUSIC_ROOT="/path/to/your/music"          # Default music directory
export XDG_DATA_HOME="$HOME/.local/share"       # Database location
export XDG_STATE_HOME="$HOME/.local/state"      # Workspace location
export DJLIB_SOURCE_DIR="/path/to/normalized"    # File organizer source override
export DJLIB_MUSIC_ROOT="/path/to/music"         # File organizer target override
export DJLIB_MUSIC_ROOT_NAMES="Music,music"      # Auto-detect target folders
export DJLIB_OPERATOR="DJ Name"                   # Operator name for run logs
export REDIS_URL="redis://localhost:6379/0"       # Web UI job queue
export DJLIB_WEBUI_AUTO=true                       # Auto-launch Web UI on runs
export DJLIB_WEBUI_URL="http://127.0.0.1:8000/healthz"  # Web UI health check

Store secrets in environment variables or .env (ignored by git).

Command Line Options

# Global options
djlib --music-root "/path/to/music" \
      --workspace "/custom/workspace" \
      --db "/custom/db.sqlite" \
      --label "MyLibrary" \
      --json \
      --apply \
      --verbose [COMMAND]

File Organizer Config

scripts/file_organizer/config.toml is auto-generated on first run. You can copy and customize the sample in scripts/file_organizer/config.sample.toml or edit the generated file directly:

[file_organizer]
source_dir = "/path/to/normalized"
target_root = "/path/to/music"
quarantine_root = "/path/to/music/quarantine"
backup_root = "/path/to/music/backups"
batch_size = 200
source_recursive = true
backup_overwrites = true
create_archive = true
dry_run = false

# Optional tag preservation (list or map)
preserve_tags = ["artist", "track", "title"]

🛡️ Safety Features

Pre-flight Validation

# Comprehensive environment check
djlib-preflight --music-root "/path/to/music" --strict --json

Exit codes:

• 0: Safe to proceed
• 1: Warnings present (proceed with caution)
• 2-5: Critical errors (do not proceed)

Backup and Recovery

• Automatic backups: Original files are backed up before any modification
• Quarantine system: Duplicates are moved, not deleted
• Resumable workflows: Interrupted operations can be safely resumed
• Dry-run mode: Preview changes before applying

📊 Database Schema

The system uses SQLite with the following key tables:

• roots: Registered music library roots
• files: File metadata and paths
• audio_probe: Technical audio properties
• tags_read: Extracted ID3/metadata tags
• fingerprints: Audio signatures for duplicate detection
• duplicates: Duplicate group relationships

See scripts/djlib_sql_schema_v2 for complete schema.

🎵 Serato Integration

DJLIB optimizes tags for Serato compatibility:

• ID3v2.3 + UTF-16 encoding
• Essential frames only: Title, Artist, Album, Genre, Year, Track, BPM, Key
• Camelot key notation: Automatic musical key detection and conversion
• BPM normalization: Consistent tempo tagging

🔍 Duplicate Detection

Multi-layered duplicate detection:

1. Duration similarity (+/- 0.5 seconds)
2. Partial audio signatures (first/last 64KB)
3. Acoustic fingerprinting (chromaprint/fpcalc)
4. SHA256 verification (within duplicate groups)
5. Filename similarity (optional)

📝 Examples

Basic Library Analysis

# Analyze library safely
djlib --music-root "/media/USB/Music" run

# Get JSON summary
djlib --music-root "/media/USB/Music" summary --json > library_report.json

# Check specific file
djlib --music-root "/media/USB/Music" explain 12345

Duplicate Management

# Find filename duplicates (high similarity)
djlib --music-root "/media/USB/Music" --apply dedupe-filenames --threshold 0.95

# Find title-based duplicates
djlib --music-root "/media/USB/Music" --apply dedupe-titles

# Manual review before quarantine
djlib --music-root "/media/USB/Music" dedupe

Metadata Normalization

# Normalize all tags to Serato standards
djlib --music-root "/media/USB/Music" --apply normalize-tags --all

# Normalize filenames to canonical format
djlib --music-root "/media/USB/Music" --apply normalize-filenames

Advanced Workflows

# Resume from specific phase
djlib --music-root "/media/USB/Music" resume --resume-from dedupe

# Limit processing for testing
djlib --music-root "/media/USB/Music" run --limit 100

# Verbose logging
djlib --music-root "/media/USB/Music" run --verbose --progress-every 50

🔧 Troubleshooting

Common Issues

1. Missing dependencies: Run djlib-preflight --install-missing
2. Permission errors: Ensure read/write access to music directory
3. Interrupted operations: Use resume command to continue
4. Storage space: Check workspace has room for backups

Debug Mode

# Enable verbose logging
djlib --music-root "/path/to/music" run --verbose

# Follow progress in real-time
djlib --music-root "/path/to/music" progress --follow

# Interactive troubleshooting
djlib --music-root "/path/to/music" runbook

🧰 Additional Tools

• File organizer: python -m scripts.file_organizer.main --dry-run
• File organizer preflight: python -m scripts.file_organizer.main --preflight
• File organizer web UI: uvicorn scripts.file_organizer.web_ui:app --reload --host 0.0.0.0 --port 8000
• File organizer web UI launcher: python -m scripts.file_organizer.webui_launcher
• Web UI guide: scripts/file_organizer/README_WEBUI.md
• Command references: docs/COMMAND_CHEATSHEET_ALL.md

Note: The file organizer CLI auto-launches the Web UI on runs/resume unless disabled with --no-webui or DJLIB_WEBUI_AUTO=false.

📄 License

This project is open source. See the repository for licensing details.

🤝 Contributing

Contributions welcome. Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

📞 Support

For issues and questions:

• Check the troubleshooting runbook: djlib --music-root "/path" runbook
• Review logs: djlib --music-root "/path" logs
• File issues on GitHub

Important: Always run with --dry-run first when possible, and ensure you have backups of critical music files before using --apply options.