Enhanced GitHub Token Integration for Docker Builds

[tokk-nv]

Problem Statement

This PR addresses the persistent and frustrating GitHub API rate limiting issue that plagues Docker builds:

ADD failed: failed to GET https://api.github.com/repos/dusty-nv/sudonim/git/refs/heads/main with status 403 Forbidden: {"message":"API rate limit exceeded for 216.228.112.22. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

Current Impact:

• Builds fail randomly due to rate limiting (60 requests/hour for unauthenticated)
  • This happens more often in a corporate environment
  • Ref: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28
• Developers must manually use --no-github-api flag as a workaround
• Cache invalidation functionality is lost when using the workaround
• Poor user experience with cryptic error messages

Solution

Implement intelligent GitHub token integration that automatically handles API rate limiting while maintaining all existing functionality.

Key Features

🔑 Enhanced GitHub Token Detection

• Support for multiple environment variables: GITHUB_TOKEN, GITHUB_PAT, GH_TOKEN
• Intelligent fallback with helpful warnings when tokens are missing
• Clear guidance on rate limit implications

🎯 Automatic Dockerfile Pre-processing

• Detects ADD https://api.github.com/... lines in Dockerfiles
• Pre-fetches commit data using authenticated API calls (5000 requests/hour vs 60)
• Transforms ADD instructions to COPY with pre-fetched data
• Injects commit hashes as build arguments for version tracking

🔄 Seamless Fallback System

• Success path: Uses enhanced approach with pre-fetched data
• Fallback path: Automatically falls back to existing --no-github-api workaround
• Explicit bypass: --no-github-api flag still works exactly as before
• No build failures due to GitHub API issues

♻️ Automatic Cleanup

• Removes temporary files after builds
• Maintains a clean package directory state
• No file accumulation over time

User Experience

With Valid Token (Recommended)

export GITHUB_TOKEN=ghp_your_token_here
./build.sh sudonim  # Works automatically with enhanced caching

Without Token (Graceful Fallback)

./build.sh sudonim  # Automatically falls back with helpful warnings

Explicit Fallback (Always Works)

./build.sh --no-github-api sudonim  # Original behavior unchanged

Technical Implementation

Dockerfile Transformation

Before:

ADD https://api.github.com/repos/dusty-nv/sudonim/git/refs/heads/main /tmp/sudonim_version.json

After:

COPY .github-api-temp/dusty-nv_sudonim_main.json /tmp/sudonim_version.json

Build Arguments

Automatically adds commit hashes as build arguments:

[tokk-nv]

Tested following on test-github-token-integration branch (merged on the current upstream dev).

# Make sure no GitHub token is set
unset GITHUB_TOKEN GITHUB_PAT GH_TOKEN
   
# Try building a package that uses GitHub API calls
./build.sh pytorch
   
# Set your GitHub token (replace with your actual token)
export GITHUB_TOKEN=ghp_your_actual_token_here
   
# Build the same package
./build.sh pytorch

[Copilot]

Pull Request Overview

This PR implements enhanced GitHub token integration for Docker builds to address persistent GitHub API rate limiting issues. The solution automatically detects GitHub API calls in Dockerfiles, pre-fetches data using authenticated requests, and provides graceful fallback behavior.

• Adds automatic GitHub token detection from multiple environment variables (GITHUB_TOKEN, GITHUB_PAT, GH_TOKEN)
• Implements Dockerfile preprocessing to replace ADD instructions with COPY using pre-fetched GitHub data
• Provides seamless fallback to existing --no-github-api workaround when token is unavailable or preprocessing fails

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test_github_token_integration.py	Comprehensive test script for validating GitHub token detection and Dockerfile preprocessing functionality
jetson_containers/network.py	Core implementation of GitHub token detection and Dockerfile preprocessing logic
jetson_containers/container.py	Integration of GitHub API preprocessing into the container build pipeline with cleanup
jetson_containers/build.py	Updated help text for --no-github-api flag to reflect enhanced functionality
docs/github-token-integration.md	Complete documentation for the new GitHub token integration feature

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The file is being opened and read multiple times unnecessarily. The content should be read once and stored in a variable to avoid repeated I/O operations.

[Copilot]

The regular expression pattern is complex and could benefit from documentation explaining what each capture group represents (owner/repo, branch, destination path).

[Copilot]

The cleanup loop calls find_package() for each package, which may be inefficient if this function performs expensive operations. Consider reusing the package information already loaded during the build process.

Suggested change

	try:
	for pkg in packages:
	pkg_info = find_package(pkg)
	# Cache package info for all packages to avoid repeated find_package() calls
	package_infos = {pkg: find_package(pkg) for pkg in packages}
	try:
	for pkg in packages:
	pkg_info = package_infos[pkg]