Refactored header-based auth scans not to normalize the header names.

[halcyondream]

Proposed changes

Description here: https://github.com/orgs/projectdiscovery/discussions/6478

Re: checklist, please let me know if there is anything else I can do to support with test cases, etc.

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• Bug Fixes

  • Preserve exact header casing when forwarding headers, preventing unintended canonicalization and improving compatibility with case-sensitive APIs.

• Documentation

  • Clarified in README and reference docs that headers from secret files retain their original casing.
  • Added notes to configuration reference explaining preserved header casing for authentication secrets.

[coderabbitai]

Walkthrough

Replaced header-setting logic to assign headers directly into the request header map using the provided key casing, and added documentation notes across README, SYNTAX-REFERENCE, and secret-related structs clarifying that secret-file headers preserve exact casing.

Changes

Cohort / File(s)	Summary
Header assignment change pkg/authprovider/authx/headers_auth.go	Replaced req.Header.Set(header.Key, header.Value) with direct map assignment req.Header[header.Key] = []string{header.Value} in Apply and ApplyOnRR; added NOTE comments documenting preservation of exact header key casing.
Documentation updates README.md, SYNTAX-REFERENCE.md	Inserted notes under CLOUD -> AUTHENTICATION and relevant syntax reference sections stating that headers loaded from secrets files preserve exact casing (useful for case-sensitive APIs).
Secret struct comments pkg/authprovider/authx/file.go	Added trailing comments to exported fields (Secret.Headers, KV.Key) noting that header keys preserve exact casing when loaded from secrets files.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant AuthX as authx.Apply / ApplyOnRR
    participant Req as http.Request.Header

    Client->>AuthX: call Apply / ApplyOnRR with headers to add
    Note over AuthX: iterate headers from auth secret/template
    AuthX->>Req: write directly to header map\nreq.Header[header.Key] = [header.Value]
    Note over Req: header key preserved as provided (no canonicalization)
    AuthX-->>Client: return modified request / roundtrip

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

I hop through headers, keys kept true,
No Canonical dance — just the casing you knew.
I tuck each value in the map with care,
A quiet note that the headers stay fair.
Little rabbit, neat and aware. 🐇✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title "Refactored header-based auth scans not to normalize the header names." is a concise, single-sentence summary that directly reflects the primary change (preserving header name casing rather than canonicalizing it) and matches the modifications in pkg/authprovider/authx/headers_auth.go and related documentation; it is specific and not misleading.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (3)

pkg/authprovider/authx/headers_auth.go (3)

26-26: Preserving case via direct map writes is fine, but dedupe and validate header names before insert.

• Without dedupe you can emit duplicate fields (e.g., "Authorization" + "authorization") if earlier code set a canonicalized key.
• Also guard against illegal header-name bytes; your nonCanonicalMIMEHeaderKey is a no-op (see below), so invalid names will slip through and be rejected by net/http at send time.

Apply this diff here:

-        req.Header[nonCanonicalMIMEHeaderKey(header.Key)] = []string{header.Value}
+        setRawHeader(req.Header, header.Key, header.Value)

Add this helper (outside this hunk):

// setRawHeader sets a header using the original casing, after basic validation,
// and removes case-insensitive duplicates to avoid multiple fields on the wire.
// Special-case Host: set req.Host elsewhere if needed.
func setRawHeader(h http.Header, key, value string) {
    if key == "" {
        return
    }
    // RFC7230 tchar validation (reuse your validHeaderFieldByte)
    for i := 0; i < len(key); i++ {
        if !validHeaderFieldByte(key[i]) {
            return // skip invalid header names rather than sending a broken request
        }
    }
    // De-duplicate existing entries with different casing
    for k := range h {
        if strings.EqualFold(k, key) && k != key {
            delete(h, k)
        }
    }
    h[key] = []string{value}
}

And update imports:

 import (
     "net/http"
+    "strings"

Questions to verify:

• Do we intentionally allow duplicate header fields with different casing? If yes, we can drop the dedupe loop.
• Should "Host" be supported? If so, wire it via req.Host instead of h["Host"].

---

33-33: Mirror the same fix for retryablehttp.Request.

Use the same helper to dedupe/validate.

Apply this diff:

-        req.Header[nonCanonicalMIMEHeaderKey(header.Key)] = []string{header.Value}
+        setRawHeader(req.Header, header.Key, header.Value)

---

47-69: Avoid hand-rolled token-byte bitsets; prefer a well-tested helper or add tests.

Reimplementing RFC7230 tchar logic is brittle. Either:

• Use golang.org/x/net/http/httpguts.IsTokenByte, or
• Keep this function but add unit tests covering all allowed/disallowed bytes.

Option A (recommended):

-func validHeaderFieldByte(c byte) bool {
-    const mask = 0 |
-        (1<<(10)-1)<<'0' |
-        (1<<(26)-1)<<'a' |
-        (1<<(26)-1)<<'A' |
-        1<<'!' | 1<<'#' | 1<<'$' | 1<<'%' | 1<<'&' | 1<<'\'' |
-        1<<'*' | 1<<'+' | 1<<'-' | 1<<'.' | 1<<'^' | 1<<'_' |
-        1<<'`' | 1<<'|' | 1<<'~'
-    return ((uint64(1)<<c)&(mask&(1<<64-1)) |
-        (uint64(1)<<(c-64))&(mask>>64)) != 0
-}
+import "golang.org/x/net/http/httpguts"
+
+func validHeaderFieldByte(c byte) bool { return httpguts.IsTokenByte(c) }

If adding x/net is undesirable, I can generate a tight table-driven test to lock this down.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 876974f and 08e527a.

📒 Files selected for processing (1)

• pkg/authprovider/authx/headers_auth.go (1 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt
Run static analysis with go vet

Files:

• pkg/authprovider/authx/headers_auth.go

🧠 Learnings (1)

📚 Learning: 2024-12-13T08:42:57.506Z

Learnt from: dwisiswant0
PR: projectdiscovery/nuclei#5772
File: pkg/authprovider/authx/cookies_auth.go:36-38
Timestamp: 2024-12-13T08:42:57.506Z
Learning: In the `ApplyOnRR` method of `CookiesAuthStrategy` in `pkg/authprovider/authx/cookies_auth.go`, when setting cookies, only remove the cookies that are about to be set to avoid removing essential session cookies.

Applied to files:

• pkg/authprovider/authx/headers_auth.go

[coderabbitai]

Actionable comments posted: 0

♻️ Duplicate comments (1)

pkg/authprovider/authx/headers_auth.go (1)

27-29: Replace direct req.Header assignments with a safe setter that validates names and removes case-insensitive dupes

Direct map writes bypass canonicalization, allow case-only duplicates and illegal header bytes. Implement a small helper and use it (also guard nils and ensure req.Header != nil).

• Change pkg/authprovider/authx/headers_auth.go — Apply (lines ~27–29) and ApplyOnRR (lines ~36–38): add nil checks, ensure req.Header is non-nil, and call setHeaderRaw instead of assigning req.Header[key] directly.
• Also change pkg/protocols/http/build_request.go — replace the direct assignments at lines 411 and 449 (req.Header[key] = []string{value}) with the same safe logic (guard req.Header, then setHeaderRaw).
• Add helper (either as a shared util or per-package) and imports. Example (copy as-is):

func (s *HeadersAuthStrategy) Apply(req *http.Request) {
-	for _, header := range s.Data.Headers {
-		req.Header[header.Key] = []string{header.Value}
-	}
+	if s == nil || s.Data == nil {
+		return
+	}
+	if req.Header == nil {
+		req.Header = make(http.Header)
+	}
+	for _, header := range s.Data.Headers {
+		setHeaderRaw(req.Header, header.Key, header.Value)
+	}
}

func (s *HeadersAuthStrategy) ApplyOnRR(req *retryablehttp.Request) {
-	for _, header := range s.Data.Headers {
-		req.Header[header.Key] = []string{header.Value}
-	}
+	if s == nil || s.Data == nil {
+		return
+	}
+	if req.Header == nil {
+		req.Header = make(http.Header)
+	}
+	for _, header := range s.Data.Headers {
+		setHeaderRaw(req.Header, header.Key, header.Value)
+	}
}

Helper and imports (add net/textproto and strings where you add the helper):

import (
    "net/http"
    "net/textproto"
    "strings"

    "github.com/projectdiscovery/retryablehttp-go"
)

func setHeaderRaw(h http.Header, key, value string) {
    if h == nil || key == "" {
        return
    }
    // Fallback to canonical form if any invalid byte is present.
    for i := 0; i < len(key); i++ {
        c := key[i]
        if c <= 0x20 || c >= 0x7f || strings.ContainsRune("()<>@,;:\\\"/[]?={} \t", rune(c)) {
            key = textproto.CanonicalMIMEHeaderKey(key)
            break
        }
    }
    // Remove existing keys equal by case-insensitive comparison to avoid duplicates.
    for k := range h {
        if strings.EqualFold(k, key) {
            delete(h, k)
        }
    }
    h[key] = []string{value}
}

• Mirror the cookie-setting learning: only remove case-insensitive keys that collide with the key you are about to set.
• Run gofmt and go vet locally after changes.

🧹 Nitpick comments (1)

pkg/authprovider/authx/headers_auth.go (1)

24-26: Clarify case‑preservation note (HTTP/2/3 lowercases headers).

Header names are case‑insensitive by spec; HTTP/2 and HTTP/3 force lowercase on the wire. Please avoid promising exact casing across all transports.

Apply this wording tweak:

-// NOTE: This preserves exact header casing (e.g., barAuthToken stays as barAuthToken)
-// This is useful for APIs that require case-sensitive header names
+// NOTE: Best‑effort: preserves header key bytes for HTTP/1.x. HTTP/2/3 lower‑case names on the wire.
+// Useful for HTTP/1.x backends or signing schemes that include header-name bytes.

Also applies to: 33-35

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e63d937 and 5b18ee8.

⛔ Files ignored due to path filters (1)

• pkg/authprovider/authx/testData/example-auth.yaml is excluded by !**/*.yaml

📒 Files selected for processing (4)

• README.md (1 hunks)
• SYNTAX-REFERENCE.md (2 hunks)
• pkg/authprovider/authx/file.go (2 hunks)
• pkg/authprovider/authx/headers_auth.go (1 hunks)

✅ Files skipped from review due to trivial changes (3)

• SYNTAX-REFERENCE.md
• README.md
• pkg/authprovider/authx/file.go

🧰 Additional context used

📓 Path-based instructions (1)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt
Run static analysis with go vet

Files:

• pkg/authprovider/authx/headers_auth.go

🧠 Learnings (1)

📚 Learning: 2024-12-13T08:42:57.506Z

Learnt from: dwisiswant0
PR: projectdiscovery/nuclei#5772
File: pkg/authprovider/authx/cookies_auth.go:36-38
Timestamp: 2024-12-13T08:42:57.506Z
Learning: In the `ApplyOnRR` method of `CookiesAuthStrategy` in `pkg/authprovider/authx/cookies_auth.go`, when setting cookies, only remove the cookies that are about to be set to avoid removing essential session cookies.

Applied to files:

• pkg/authprovider/authx/headers_auth.go

🧬 Code graph analysis (1)

pkg/authprovider/authx/headers_auth.go (1)

pkg/fuzz/component/headers.go (1)

• Header (12-16)