Add scrape target update endpoint to the Collector's Prometheus receiver

[Aneurysm9]

In order to support distribution of Prometheus scrape targets among a set of Collector instances as required for #6 we will need to create a mechanism for updating the set of scrape targets used by any given Collector instance. The Prometheus receiver should be extended with a server that can process requests to update the scrape targets for a given job. It should receive a list of Prometheus static_config entries that are written to a file the receiver is watching with a file_sd_config. This service should only be active when enabled by a new configuration option for the Prometheus receiver that is disabled by default. The configuration should also allow the user to specify the port the service will listen on along with any other service configuration items that may be appropriate.

This issue should be used to track the design of this service and to aggregate any other issues or PRs created during implementation.

[dashpole]

According to #32, the targets are being written to a configmap, but the above says we want a server that processes requests. Which are we planning to do?

Edit: based on a comment in the doc, it looks like we are writing the static config to a configmap, and using a server to process requests to update scrape targets.

[Aneurysm9]

The Collector configuration is written to a configmap. It would be updated to include a file_sd_config entry for each job with target discovery and assignment managed by the operator. The update mechanism contemplated here would be to update the files referenced by the file_sd_config entries in the Collector config.

[dashpole]

Could the update mechanism live outside the prometheus receiver (e.g. an extension that writes to files)?

If it is a separate component, marked experimental (at least initially), that may address some of the concerns over adding temporary functionality to help meet near-term deadlines.

[dashpole]

From the discussion today:

• The advantage of including it in the prometheus receiver is that the receiver can manage files, and hide the complexity from the user-facing config.
• The disadvantage is potentially around velocity and stability: Adding to the prometheus receiver means we won't be able to change this design later, or may delay marking the receiver stable if we aren't confident in this design. Having a separate component might allow us to try out other ideas in parallel (such as configuration using OTLP).

[alolita]

An alternative design was discussed in the Prometheus wg today. @dashpole @rakyll looking forward to your feedback on this.

[alolita]

@alexperez52 @Raul9595 can you verify this now works with Prometheus 2.28?