Add raw HTTP request support to pipe mode #773
Conversation
hahwul
commented
Jul 5, 2025
hahwul
commented
- Add --rawdata and --http flags to pipe command - Implement runRawDataPipeMode to parse raw HTTP requests from stdin - Support base64-encoded input for raw requests - Update documentation for scanning from raw HTTP requests via file and pipe modes
- Add --rawdata and --http flags to pipe command - Implement runRawDataPipeMode to parse raw HTTP requests from stdin - Support base64-encoded input for raw requests - Update documentation for scanning from raw HTTP requests via file and pipe modes
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @hahwul, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the pipe command's capabilities by introducing support for scanning targets directly from raw HTTP request data provided via standard input. This feature, along with new flags and automatic base64 decoding, streamlines the workflow for security professionals integrating Dalfox with other tools that output raw HTTP requests.
Highlights
- New Functionality: Raw HTTP Request Support in Pipe Mode: The
pipecommand now supports processing raw HTTP requests directly from standard input. This allows users to pipe raw request data (e.g., from Burp Suite or ZAP) into Dalfox for scanning. - New Command-Line Flags: Two new flags,
--rawdataand--http, have been added to thepipecommand.--rawdataenables the raw data parsing mode, and--httpforces the target URL to use HTTP instead of the default HTTPS when constructing the URL from raw request components. - Base64 Encoded Input Support: The new raw data pipe mode automatically detects and decodes base64-encoded input, providing flexibility for how raw requests can be supplied.
- Documentation Updates: The
scan-from-rawreq.mddocumentation has been updated to include detailed instructions and examples for using bothfileand the newly addedpipemodes with raw HTTP requests, including base64 examples.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
| var path, body, host, target string | ||
| bodyswitch := false | ||
| for index, line := range ff { | ||
| if index == 0 { | ||
| parse := strings.Split(line, " ") | ||
| if len(parse) > 1 { | ||
| options.Method = parse[0] | ||
| path = parse[1] | ||
| } else { | ||
| printing.DalLog("ERROR", "HTTP Raw Request Format Error", options) | ||
| os.Exit(1) | ||
| } | ||
| } else { | ||
| if strings.Contains(line, "Host: ") || strings.Contains(line, "host: ") { | ||
| host = line[6:] | ||
| } else { | ||
| parse := strings.Split(line, ":") | ||
| if len(parse) > 1 { | ||
| options.Header = append(options.Header, line) | ||
| } | ||
| } | ||
| if bodyswitch { | ||
| body = body + line | ||
| } | ||
| if len(line) == 0 { | ||
| bodyswitch = true | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
The HTTP request parsing loop has potential issues:
- The request body is reconstructed by concatenating lines without considering newlines, which can corrupt multi-line bodies.
- Using
body = body + linefor string concatenation in a loop is inefficient for large bodies. - Parsing the request line with
strings.Split(line, " ")is not robust against multiple spaces. - The
Hostheader check is case-sensitive and brittle to extra whitespace.
Consider refactoring the parsing logic to improve correctness and robustness.
var path, host, target string
var body strings.Builder
// Normalize line endings and split headers from body
rawReq = strings.ReplaceAll(rawReq, "\r\n", "\n")
parts := strings.SplitN(rawReq, "\n\n", 2)
headerLines := strings.Split(parts[0], "\n")
if len(parts) > 1 {
body.WriteString(parts[1])
}
if len(headerLines) == 0 || headerLines[0] == "" {
printing.DalLog("ERROR", "HTTP Raw Request Format Error: Empty request", options)
os.Exit(1)
}
// Parse first line (request line)
firstLine := headerLines[0]
parse := strings.Fields(firstLine)
if len(parse) > 1 {
options.Method = parse[0]
path = parse[1]
} else {
printing.DalLog("ERROR", "HTTP Raw Request Format Error", options)
os.Exit(1)
}
// Parse headers
for _, line := range headerLines[1:] {
if strings.HasPrefix(strings.ToLower(line), "host:") {
host = strings.TrimSpace(line[len("host:"):])
} else if strings.Contains(line, ":") {
options.Header = append(options.Header, line)
}
}| target = "https://" + host + path | ||
| } | ||
| } | ||
| _, _ = scanning.Scan(target, options, "single") |
There was a problem hiding this comment.
The error returned by scanning.Scan is ignored. Handle the error to prevent silent failures.
_, err = scanning.Scan(target, options, "single")
if err != nil {
printing.DalLog("ERROR", "Scanning failed for target "+target+": "+err.Error(), options)
}| } | ||
|
|
||
| // runRawDataPipeMode processes a file containing raw HTTP request data | ||
| func runRawDataPipeMode(cmd *cobra.Command) { |
There was a problem hiding this comment.
The function runRawDataPipeMode directly exits the program using os.Exit(1) when encountering errors. This abrupt termination can hinder testability and reusability. It's better to return an error to allow the caller function to handle the error gracefully.
| func runRawDataPipeMode(cmd *cobra.Command) { | |
| func runRawDataPipeMode(cmd *cobra.Command) error { |
| // runRawDataPipeMode processes a file containing raw HTTP request data | ||
| func runRawDataPipeMode(cmd *cobra.Command) { | ||
| printing.DalLog("SYSTEM", "Using pipe mode with raw data format", options) | ||
| bytes, err := ioutil.ReadAll(os.Stdin) |
There was a problem hiding this comment.
The io/ioutil package is deprecated. Use io.ReadAll instead.
| bytes, err := ioutil.ReadAll(os.Stdin) | |
| bytes, err := io.ReadAll(os.Stdin) |
|
|
||
| ```http | ||
| POST https://www.hahwul.com/?q=xspear HTTP/1.1 | ||
| POST https://www.hahwul.com/?q=dalfox HTTP/1.1 |
There was a problem hiding this comment.
The query parameter value in the sample raw request has been changed from xspear to dalfox. Ensure this change aligns with the intended purpose of the documentation.
| POST https://www.hahwul.com/?q=dalfox HTTP/1.1 | |
| POST https://www.hahwul.com/?q=xspear HTTP/1.1 |
