Describe the bug:
The Sites Tree File Format export produced by the exim add-on treats all request with bodies as if they are application/x-www-form-urlencoded POST requests using the format key1=value1&key2=value2. It attempts to parse them, strip values, and produce an output like key1=&key2=.
This produces inconsistencies when the request body uses other formats, such as JSON or XML:
- Multi-line bodies are truncated and only the first line is preserved in the
data field.
- Bodies with
= characters are misinterpreted. For example, an XML declaration like <?xml version="1.0"?> becomes <?xml version= after parsing.
- The bodies of non-POST requests with bodies (e.g.
PUT, PATCH) are not included, since only POST requests go through the body-extraction logic.
Steps to reproduce the behavior:
Run the following automation plan, which uses a requestor job to send different types of HTTP requests, and an export job to export the site tree.
env:
contexts:
- name: Default Context
urls:
- https://httpbingo.org
jobs:
- type: requestor
requests:
- name: GET Request
url: https://httpbingo.org/get?key1=value1&key2=value2
method: GET
- name: POST Request With Form Data
url: https://httpbingo.org/post
method: POST
data: 'key1=value1&key2=value2'
headers:
- "Content-Type: application/x-www-form-urlencoded"
- name: PUT Request
url: https://httpbingo.org/put
method: PUT
data: '{"key1": "value1", "key2": "value2"}'
headers:
- 'Content-Type: application/json'
- name: POST Request With Multipart Form Data
url: https://httpbingo.org/post
method: POST
data: |
--boundary
Content-Disposition: form-data; name="key1"
value1
--boundary
Content-Disposition: form-data; name="key2"
value2
--boundary--
headers:
- "Content-Type: multipart/form-data; boundary=boundary"
- name: PATCH Request
url: https://httpbingo.org/patch
method: PATCH
data: '{"key1": "value1", "key2": "value2"}'
headers:
- 'Content-Type: application/json'
- name: POST Request With JSON Data
url: https://httpbingo.org/post
method: POST
data: '{"key1": "value1", "key2": "value2"}'
headers:
- 'Content-Type: application/json'
- name: POST Request With XML Data
url: https://httpbingo.org/post
method: POST
data: |
<?xml version="1.0"?>
<key1>value1</key1>
<key2>value2</key2>
headers:
- "User-Agent: test"
- 'Content-Type: application/xml'
- type: export
parameters:
type: yaml
source: sitestree
fileName: sitestree.yamlNote: httpbingo.org requires a User-Agent header to respond correctly, but it's omitted here for brevity since the response is not relevant to this report.
The resulting sitestree.yaml file should look like this:
- node: Sites
children:
- node: https://httpbingo.org
url: https://httpbingo.org
method: GET
children:
- node: "GET:get(key1,key2)"
url: https://httpbingo.org/get?key1=value1&key2=value2
method: GET
responseLength: 930
statusCode: 200
- node: PATCH:patch
url: https://httpbingo.org/patch
method: PATCH
responseLength: 1080
statusCode: 200
- node: POST:post()(<?xml version)
url: https://httpbingo.org/post
method: POST
responseLength: 1100
statusCode: 200
data: '%3C%3Fxml+version='
- node: "POST:post()(key1,key2)"
url: https://httpbingo.org/post
method: POST
responseLength: 1103
statusCode: 200
data: key1=&key2=
- node: POST:post()(multipart/form-data)
url: https://httpbingo.org/post
method: POST
responseLength: 1238
statusCode: 200
- node: "POST:post()({\"key1\": \"value1\", \"key2\": \"value2\"})"
url: https://httpbingo.org/post
method: POST
responseLength: 1078
statusCode: 200
data: '%7B%22key1%22%3A+%22value1%22%2C++%22key2%22%3A+%22value2%22%7D='
- node: PUT:put
url: https://httpbingo.org/put
method: PUT
responseLength: 1076
statusCode: 200Note the following behavior:
Expected
GET Request: no body, no data field.POST Request with Form Data: keys are kept, values are stripped.POST Request with Multipart Form Data: the body is present but intentionally left out in the export.
Unexpected
PUT Request and PATCH Request are missing the data field, even though they have bodies.POST Request with JSON Data is mis-parsed (ends with an = character.)POST Request with XML Data is mis-parsed (only the first line is kept, and everything after the first = is removed.)
Expected behavior:
- PUT and PATCH requests include a
data field as long as they have bodies.
- Request bodies are exported with minimal parsing to support non-form requests.
Software versions:
ZAP
Version: 2.16.1
Installed Add-ons: [...[id=exim, version=0.14.0]...]
Screenshots:
No response
Errors from the zap.log file:
No response
Additional context:
Omitting the values of each key in x-www-form-urlencoded bodies seems reasonable, but this approach could lead to inconsistencies with JSON, XML, and possibly other formats. These would need to either:
- Contain the full body
- Or require parsing and sanitizing based on their formats, which (IMHO) adds unnecessary complexity
It might be worth considering keeping the entire body with minimal parsing, unless there's a strong reason to do otherwise.
Would you like to help fix this issue?
Describe the bug:
The Sites Tree File Format export produced by the exim add-on treats all request with bodies as if they are
application/x-www-form-urlencodedPOST requests using the formatkey1=value1&key2=value2. It attempts to parse them, strip values, and produce an output likekey1=&key2=.This produces inconsistencies when the request body uses other formats, such as JSON or XML:
datafield.=characters are misinterpreted. For example, an XML declaration like<?xml version="1.0"?>becomes<?xml version=after parsing.PUT,PATCH) are not included, since onlyPOSTrequests go through the body-extraction logic.Steps to reproduce the behavior:
Run the following automation plan, which uses a
requestorjob to send different types of HTTP requests, and anexportjob to export the site tree.Note:
httpbingo.orgrequires aUser-Agentheader to respond correctly, but it's omitted here for brevity since the response is not relevant to this report.The resulting
sitestree.yamlfile should look like this:Note the following behavior:
Expected
GET Request: no body, nodatafield.POST Request with Form Data: keys are kept, values are stripped.POST Request with Multipart Form Data: the body is present but intentionally left out in the export.Unexpected
PUT RequestandPATCH Requestare missing thedatafield, even though they have bodies.POST Request with JSON Datais mis-parsed (ends with an=character.)POST Request with XML Datais mis-parsed (only the first line is kept, and everything after the first=is removed.)Expected behavior:
datafield as long as they have bodies.Software versions:
Screenshots:
No response
Errors from the zap.log file:
No response
Additional context:
Omitting the values of each key in
x-www-form-urlencodedbodies seems reasonable, but this approach could lead to inconsistencies with JSON, XML, and possibly other formats. These would need to either:It might be worth considering keeping the entire body with minimal parsing, unless there's a strong reason to do otherwise.
Would you like to help fix this issue?