jq-forensics

A collection of specialized jq functions for digital forensics and incident response to convert proprietary formats, parse artifacts, and streamline DFIR analysis without custom parsers.

Why Not in jq Core?

These functions were originally proposed as PRs to add as built-in functions to jq itself. However, they are better suited as an external module because:

1. Domain-specific functionality - These functions are primarily useful for digital forensics and incident response, not general JSON processing
2. No overhead for others - Users who don't need forensic analysis don't carry these functions in their jq installation
3. Faster iteration - External modules can be updated and improved independently without waiting for jq releases
4. jq's module system is designed for this - The ~/.jq auto-include mechanism allows custom functions to work exactly like built-ins
5. Follows jq philosophy - Keep the core lean, extend through modules

The jq maintainers suggested this approach, and it turned out to be the right call, it's easier to maintain, distribute, and use.

Functions

Timestamps

• fromwebkit - Converts WebKit/Chrome timestamps (microseconds since 1601-01-01) into ISO 8601
• fromcocoa - Converts macOS/iOS Cocoa timestamps (seconds since 2001-01-01) into ISO 8601
• fromunix - Converts Unix timestamps (seconds or milliseconds since 1970-01-01) into ISO 8601
• toreadable - Takes any timestamp and makes it human-friendly (YYYY-MM-DD HH:MM:SS)

Transformations

• todefang - Makes malicious URLs/IPs/emails safe to share in docs (defangs http/https, @, and dots)
• fromdefang - Converts them back when you need the original

Installation

Option 1: Automatic Installation

Linux / macOS:

git clone https://github.com/calilkhalil/jq-forensics.git
cd jq-forensics
chmod +x scripts/install.sh
./scripts/install.sh

Windows:

git clone https://github.com/calilkhalil/jq-forensics.git
cd jq-forensics
powershell -ExecutionPolicy Bypass -File scripts/install.ps1

Note: On Windows, creating symlinks requires either:

• Running PowerShell as Administrator, OR
• Enabling Developer Mode in Windows Settings

Option 2: Manual Installation

Linux / macOS:

# Clone the repository
git clone https://github.com/calilkhalil/jq-forensics.git

# Copy to user directory
mkdir -p ~/.jq-forensics
cp -r jq-forensics/src ~/.jq-forensics/
cp jq-forensics/forensics.jq ~/.jq-forensics/

# Create symlink
ln -sf ~/.jq-forensics/forensics.jq ~/.jq

Windows:

# Clone the repository
git clone https://github.com/calilkhalil/jq-forensics.git

# Copy to user directory
mkdir $env:USERPROFILE\.jq-forensics
cp -r jq-forensics\src $env:USERPROFILE\.jq-forensics\
cp jq-forensics\forensics.jq $env:USERPROFILE\.jq-forensics\

# Create symlink (requires admin)
New-Item -ItemType SymbolicLink -Path "$env:USERPROFILE\.jq" -Target "$env:USERPROFILE\.jq-forensics\forensics.jq"

Quick Start

After installation, the functions work like built-in jq functions:

Timestamp Conversion Examples

# Analyze Chrome history with readable timestamps
jq '.[] | {
  url: .url,
  title: .title,
  last_visit: .last_visit_time | fromwebkit | toreadable
}' History.json

# Convert Unix boot timestamps from macOS system logs
echo '{"boot_time": 1741420298}' | jq '.boot_time | fromunix | toreadable'
# "2025-03-08 09:51:38"

# Chain multiple conversions for readability
echo '13318523932000000' | jq 'fromwebkit | toreadable'
# "2012-03-15 10:12:12"

IOC Defanging Examples

# Defang IOCs from incident report
jq '.indicators | map(todefang)' incident_iocs.json

# Process mixed IOCs in threat intel feed
echo '["https://evil.com", "http://malware.net", "192.168.1.1", "[email protected]"]' | 
  jq 'map(todefang)'
# ["hxxps://evil[.]com", "hxxp://malware[.]net", "192[.]168[.]1[.]1", "attacker[@]phish[.]org"]

# Restore defanged IOCs for analysis
echo '"hxxps://evil[.]com"' | jq 'fromdefang'
# "https://evil.com"

Real-World Forensic Analysis Examples

# Analyze Chrome browsing history with timestamps
jq '.[] | select(.url | test("malicious")) | {
  url: .url,
  visit_time: .last_visit_time | fromwebkit | toreadable,
  visit_count: .visit_count
}' ~/.config/google-chrome/Default/History.json

# Parse macOS system boot times from logs
grep "BOOT_TIME" system.log | jq -R 'split(" ") | {
  date: .[0:3] | join(" "),
  boot_time: .[4] | tonumber | fromunix | toreadable
}'

# Defang IOCs before sharing in incident reports
jq '{
  title: .title,
  indicators: .indicators | map(todefang),
  timestamp: .timestamp | fromunix | toreadable
}' threat_intel.json > safe_report.json

Testing

Run the test suite to verify all functions work correctly:

chmod +x tests/runner.sh
./tests/runner.sh

The test suite uses jq pipelines to validate all timestamp conversions and transformations. Tests are defined in tests/tests.jq as pipelines that should evaluate to true when they pass.

Architecture

This project follows the Single Responsibility Principle:

• Each function lives in its own module under src/
• forensics.jq orchestrates all modules
• Easy to extend: add a new module in src/ and include it in forensics.jq

Uninstall

Linux / macOS

rm -rf ~/.jq-forensics
rm ~/.jq

Windows

Remove-Item -Recurse -Force $env:USERPROFILE\.jq-forensics
Remove-Item $env:USERPROFILE\.jq

License

Public Domain / Unlicense

This software is released into the public domain. You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.