Fix ci and benchmarks (#581)

This PR addresses several critical areas focusing on CI reliability,
infrastructure improvements, and code quality enhancements. The changes
fall into four main categories:

**Original CI Failures:**

- Tests were randomly failing due to port conflicts between concurrent
database instances
- Multiple test suites trying to use port 27017 for MongoDB
simultaneously
- MySQL and PostgreSQL tests colliding on their default ports
- These conflicts caused connection timeouts and "address already in
use" errors
- The CI builds were failing due to a combination of static analysis
errors and inconsistent code formatting, which also impacted overall
code maintainability.
- Previous: Manual documentation led to code/doc drift

**Root Cause Analysis:**

- IPDX workflows were starting database services without configurable
ports
- Our custom tests were also starting databases on default ports
- When both ran together, or when parallel tests ran, they would
conflict
- No proper cleanup between test runs meant ports could stay occupied

**Problem Impact:**

- CI builds were becoming unreliable and flaky
- PRs required multiple retries and still failed
- Test failures weren't actual code issues but infrastructure problems


**Implementation** 
**CI/Testing Infrastructure**

- Improved CI workflow configuration with better caching strategies
- Added MongoDB integration test support and documentation
- Implemented cache cleanup workflow
- Enhanced test stability and reliability

**Code Quality & Architecture**

- Refactored client code structure and API organization
- Improved thread safety in analytics package
- Enhanced error handling and type safety
- Fixed duration handling in client configuration

**Documentation**

- Added integration test requirements documentation
- Improved API documentation
- Reorganized documentation generation
- Restored original docgen.sh functionality and regenerated complete CLI
documentation with proper environment variable handling

**Dependencies**

- Updated multiple dependencies to newer versions
- Removed unused dependencies

**Note**: I will address the risks and Mitigations in this
[issue](#582)
. Kindly add anything missing on the issue as a comment to the PR for
reference. I will lift them to the issue when creating a new pr for
them.

Author: Sankara-Jefferson

.github/actions/go-check-setup/action.yml

@@ -9,8 +9,10 @@ runs:
         path: |
           ~/.cache/go-build
           ~/go/pkg/mod
-        key: ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
+        key: ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}-${{ github.ref }}-${{ github.sha }}
         restore-keys: |
+          ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}-${{ github.ref }}-
+          ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}-
           ${{ matrix.os }}-golang-${{ matrix.go }}-
 
     - name: Setup Go

.github/actions/go-test-setup/action.yml

@@ -3,14 +3,21 @@ description: Setup Cache
 runs:
   using: "composite"
   steps:
+    - name: Clean Go cache directories
+      shell: bash
+      run: |
+        rm -rf ~/.cache/go-build
+        rm -rf ~/go/pkg/mod
     - name: Setup Golang caches
       uses: actions/cache@v4
       with:
         path: |
           ~/.cache/go-build
           ~/go/pkg/mod
-        key: ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}
+        key: ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}-${{ github.ref }}-${{ github.sha }}
         restore-keys: |
+          ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}-${{ github.ref }}-
+          ${{ matrix.os }}-golang-${{ matrix.go }}-${{ hashFiles('**/go.sum') }}-
           ${{ matrix.os }}-golang-${{ matrix.go }}-
     - name: Setup PostgreSQL database
       uses: ikalnytskyi/action-setup-postgres@v6

.github/workflows/cache-cleanup.yml

@@ -0,0 +1,31 @@
+name: Cache Cleanup
+on:
+  workflow_dispatch:  # Manual trigger
+  schedule:
+    - cron: '0 0 * * 0'  # Run weekly on Sunday at midnight
+
+jobs:
+  cleanup:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Cleanup
+        run: |
+          # Get all cache keys
+          CACHE_KEYS=$(gh cache list -L 1000 | awk '{print $1}')
+          
+          # Calculate total size
+          TOTAL_SIZE=$(gh cache list -L 1000 | awk '{sum += $2} END {print sum}')
+          
+          # If total size > 8GB (keeping buffer from 10GB limit)
+          if [ "$TOTAL_SIZE" -gt 8000000000 ]; then
+            # Delete older caches until we're under 8GB
+            echo "$CACHE_KEYS" | while read key; do
+              gh cache delete "$key" -f
+              TOTAL_SIZE=$(gh cache list -L 1000 | awk '{sum += $2} END {print sum}')
+              if [ "$TOTAL_SIZE" -lt 8000000000 ]; then
+                break
+              fi
+            done
+          fi
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/docgen.yml

@@ -0,0 +1,27 @@
+name: Documentation Generation
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  docgen:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: 'go.mod'
+      - name: Initialize database for doc generation
+        run: |
+          cd singularity
+          ./singularity admin init
+      - name: Generate documentation
+        run: |
+          cd singularity
+          sh docgen.sh

.github/workflows/go-check.yml

@@ -2,8 +2,9 @@ name: Go Checks
 
 on:
   pull_request:
+    branches: [main, develop]
   push:
-    branches: ["main"]
+    branches: [main, develop]
   workflow_dispatch:
 
 permissions:
@@ -14,19 +15,88 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  go-check:
-    uses: ipdxco/unified-github-workflows/.github/workflows/[email protected]
+  go-check-all:
+    name: go-check / All
+    runs-on: ubuntu-latest
+    services:
+      mysql:
+        image: mysql:8.0
+        env:
+          MYSQL_ROOT_PASSWORD: root
+          MYSQL_DATABASE: singularity
+          MYSQL_USER: singularity
+          MYSQL_PASSWORD: singularity
+        ports:
+          - 3306:3306
+        options: >-
+          --health-cmd="mysqladmin ping"
+          --health-interval=10s
+          --health-timeout=5s
+          --health-retries=3
+      
+      postgres:
+        image: postgres:15
+        env:
+          POSTGRES_USER: singularity
+          POSTGRES_PASSWORD: singularity
+          POSTGRES_DB: singularity
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    
+    steps:
+      - uses: actions/checkout@v3
+      
+      - name: Set up Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: "1.21.x"
+          
+      - name: Wait for PostgreSQL
+        run: |
+          echo "Waiting for PostgreSQL..."
+          for i in {1..10}; do
+            if PGPASSWORD=singularity psql -h localhost -U singularity -d singularity -c "SELECT 1" > /dev/null 2>&1; then
+              echo "Postgres is ready!"
+              break
+            fi
+            sleep 3
+          done
+
+      - name: Verify MySQL connection
+        run: mysql -h127.0.0.1 -P3306 -usingularity -psingularity -e "SELECT VERSION();"
+
+      - name: Verify PostgreSQL connection
+        run: PGPASSWORD=singularity psql -h localhost -U singularity -d singularity -c "SELECT version();"
+
+      - name: Ensure swagger directories exist
+        run: mkdir -p client/swagger/client
+
+      - name: Install swagger tools
+        run: go install github.com/go-swagger/go-swagger/cmd/[email protected]
+          
+      - name: Generate code
+        run: go generate ./client/swagger/...
+
+      - name: Build
+        run: go build ./...
+
+      - name: Run tests
+        run: go test -v ./...
 
   staticcheck:
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout code
-        uses: actions/checkout@v3
+      - uses: actions/checkout@v3
 
       - name: Set up Go
         uses: actions/setup-go@v4
         with:
-          go-version: "1.21"
+          go-version: "1.21.x"
 
       - name: Install staticcheck
         run: go install honnef.co/go/tools/cmd/staticcheck@latest

.github/workflows/go-test.yml

@@ -2,8 +2,9 @@ name: Go Test
 
 on:
   pull_request:
+    branches: [main, develop]
   push:
-    branches: ["main"]
+    branches: [main, develop]
   workflow_dispatch:
 
 permissions:
@@ -14,5 +15,50 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  go-test:
-    uses: ipdxco/unified-github-workflows/.github/workflows/[email protected]
+  go-test-this:
+    name: go-test / ${{ matrix.os }} (go this)
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Start MongoDB
+        if: runner.os == 'Linux'
+        uses: supercharge/[email protected]
+        with:
+          mongodb-version: '6.0'
+          mongodb-port: 27018
+
+      - name: Set up Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: '1.21'
+
+      - name: Run Go Tests
+        run: go test ./...
+
+  go-test-next:
+    name: go-test / ${{ matrix.os }} (go next)
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Start MongoDB
+        if: runner.os == 'Linux'
+        uses: supercharge/[email protected]
+        with:
+          mongodb-version: '6.0'
+          mongodb-port: 27018
+
+      - name: Set up Go
+        uses: actions/setup-go@v4
+        with:
+          go-version: '1.21'
+
+      - name: Run Go Tests
+        run: go test ./...

README.md

@@ -427,3 +427,16 @@ The internal tool used by `js-singularity` to regenerate the CAR that captures t
 
 ## License
 Dual-licensed under [MIT](https://github.com/filecoin-project/lotus/blob/master/LICENSE-MIT) + [Apache 2.0](https://github.com/filecoin-project/lotus/blob/master/LICENSE-APACHE)
+
+## Integration Tests & MongoDB
+
+Some integration tests require a MongoDB instance running on `localhost:27018`.
+
+- **CI:** MongoDB is automatically started on port 27018 in GitHub Actions workflows.
+- **Local Development:** You must start MongoDB locally on port 27018 before running tests:
+
+```bash
+mongod --port 27018
+```
+
+If MongoDB is not available, related tests will be skipped or fail with a connection error.

analytics/analytics.go

@@ -21,7 +21,10 @@ import (
 
 const flushInterval = time.Hour
 
-var Enabled = true
+var (
+	mu      sync.RWMutex
+	Enabled = true
+)
 
 var logger = log.Logger("analytics")
 
@@ -37,6 +40,8 @@ var logger = log.Logger("analytics")
 // Returns:
 //   - An error if there are issues fetching the instance id from the database or if the database appears empty.
 func Init(ctx context.Context, db *gorm.DB) error {
+	mu.Lock()
+	defer mu.Unlock()
 	if Instance != "" {
 		return nil
 	}
@@ -68,6 +73,27 @@ var (
 	Identity string
 )
 
+// GetInstance safely returns the Instance value
+func GetInstance() string {
+	mu.RLock()
+	defer mu.RUnlock()
+	return Instance
+}
+
+// GetIdentity safely returns the Identity value
+func GetIdentity() string {
+	mu.RLock()
+	defer mu.RUnlock()
+	return Identity
+}
+
+// IsEnabled safely returns the Enabled value
+func IsEnabled() bool {
+	mu.RLock()
+	defer mu.RUnlock()
+	return Enabled
+}
+
 type Collector struct {
 	mu            sync.Mutex
 	packJobEvents []PackJobEvent