Grafana agent http proxy not fully supported

[martinjaegerdk]

Hi all,
First time contributor here.

We are in need of the grafana-agent supporting being run behind a HTTP_PROXY.
It is somewhat supported today but not fully.

I have done testing on metrics, traces and logs (with oauth2authentication), and here is what i found.

Stream	Data	Oauth2	Comment
logs	✅	❌	oauth2 works with v0.33.0 of prometheus/common
metrics	✅	❌	oauth2 works with v0.33.0 of prometheus/common
traces	✅	✅	Config file does not support proxy_url, but does pickup ENV HTTP_PROXY/HTTPS_PROXY when set

Getting logs and metrics to work is easy, as it simply involves upgrading of a pkg.

Upgrading to v0.33.0 of prometheus/common, allows to specify proxy_url on oauth2: (both for logs and metrics)
(verified that it works)

metrics:
  wal_directory: /tmp/grafana-agent/wal
  global:
    scrape_interval: 15s
    remote_write:
      - url: myurl.com
        proxy_url: http://localhost:8888
        oauth2:
          client_id: ${CLIENT_ID} 
          client_secret: ${CLIENT_SECRET}
          token_url: myurl.com
          scopes: ["myscope"]
          proxy_url: http://localhost:8888  <---- 

Now with traces, i have tried specifying proxy_url on remote_write and on remote_write/oauth2 which makes the agent blow up on startup (due to YAML parse error). Neither are supported it seems.

I also tried specifying the HTTP_PROXY as mentioned in the OTEL docs, but this doesn't work either.

All the above brings me to my points.

1. Can anyone confirm that this is true
2. If true, i would like to contribute to grafana-agent, making HTTP_PROXY support better. How would i get started?

[martinjaegerdk]

Here is how i have tested on my laptop:

Testing

A local nginx-proxy can be run in docker, using this repo: https://github.com/reiz/nginx_proxy

docker run --name proxy -d -p 8888:8888 -v ${PWD}/nginx_blacklist.conf:/usr/local/nginx/conf/nginx.conf reiz/nginx_proxy:0.0.3
See connects through the proxy by doing:

docker exec -it proxy /bin/bash 
# when inside container
tail -f /var/log/nginx_access.log

To verify the actual connections from the agent, you need the agent running (below works on MacOS)

# grab the PID of the agent
ps aux

# then use lsof (list open files)
lsof -Pi | grep {{PID}}

This is what a proxy connection looks like: (given the proxy is on port 8888)

Here is an example, where both a proxy connection and a non-proxy connection is established:

[rfratto]

Hey there 👋 Sorry for the late reply, all of the maintainers were out of the office last week.

We should definitely try to get proxy_url support working consistently.

For oauth2 and logs/metrics: is the suggestion that the fix is likely to just update prometheus/common to the latest version? That'd be a pretty easy change we'd be welcome to include ASAP.

HTTP_PROXY not working with traces will take some cross-collaboration with OpenTelemtry-collector. Before we start discussing whether we should open something upstream, can you confirm that HTTP_PROXY is set in the Grafana Agent process?

IMO, ideally, people work with OpenTelemetry-collector to allow an HTTP proxy to be configured via the config file and not just the environment variable. That would make things more consistent for us downstream.

[martinjaegerdk]

For oauth2 and logs/metrics: is the suggestion that the fix is likely to just update prometheus/common to the latest version? That'd be a pretty easy change we'd be welcome to include ASAP.

Yes. Not even latest version is needed, "just" v0.33.0. (latest version is v0.34.0)

HTTP_PROXY not working with traces will take some cross-collaboration with OpenTelemtry-collector. Before we start discussing whether we should open something upstream, can you confirm that HTTP_PROXY is set in the Grafana Agent process?

I have an update on this. If i set HTTPS_PROXY (with an S) in env it is picked up by the grafana-agent.
I had misunderstood what the difference between the two env vars was.
My understanding, was that HTTP/HTTPS denoted the connection to the proxy server. And since the proxy server i use, does not use TLS, i thought that i should just specify HTTP_PROXY.

But it actually works like this:
Based on the destination URL "scheme" (e.g. oauth2 endpoint or tempo), either HTTP_PROXY or HTTPS_PROXY is chosen.
In my case, the destination is always HTTPS, meaning that it would always take the value in HTTPS_PROXY, which i had not set.
Go docs states this as default behaviour when using http.DefaultTransport in net/http, which is the case for the otel http client.

To summarise:

• Metrics/logs needs prometheus/common v.0.33.0 to have proxy_url support for oauth2 (i have tested locally, and it works with this version)
• Traces works if i specify HTTPS_PROXY in env (as my destination is HTTPS, not the proxy server itself)

Actions:

• Upgrade to prometheus/common v.0.33.0 (@rfratto , would you like a PR from me on the this?)

[martinjaegerdk]

@rfratto - PR ready #1711