Implement a target allocator to support scrape target update and collector instance replicas in a StatefulSet #60
Description
Is your feature request related to a problem? Please describe.
With the support of StatefulSet resources now implemented, there is the use case where there are more than 1 stable collector instances in the StatefulSet. Currently there is no way to delegate workload to these instances.
The load balancer design document was reviewed on the 06/23 Prometheus WG meeting.
https://docs.google.com/document/d/1GdpCqdkz72291l_8S-jio7EhVltminobvqshsFih6vA/edit
Describe the solution you'd like
An Operator managed load balancer component needs to be added to handle the delegation of exposing scrape targets to multiple collector instances. Statefulset resources are now supported by the Operator and as mentioned in #33, we will need to create a mechanism for updating the set of scrape targets used by any given Collector instance. This would utilize http based service discovery which is now available after the Prometheus 2.28 release.
Describe alternatives you've considered
Another design that was considered is an Operator-Integrated Load balancer but it was decided to go with the Operator-Managed Load balancer deployment. More information on the pros and cons can be found in the follow document.
https://docs.google.com/document/d/1YRFIZl-N0Igr1uQQ_MVm10JX2-9hy9OItNDU2PJ-eiE/edit#heading=h.w6m58i6u3fzh
Additional context
Older design proposals that were considered will be added here for context
1st design - https://docs.google.com/document/d/1Q8ZzjpJ4jO9TJX_HPCn3MxbXHkgKeOgdZuT1HDSsGVc/edit#heading=h.69vohvepyb6x
2nd design - https://docs.google.com/document/d/1Q8ZzjpJ4jO9TJX_HPCn3MxbXHkgKeOgdZuT1HDSsGVc/edit#heading=h.69vohvepyb6x
3rd design - https://docs.google.com/document/d/1Q8ZzjpJ4jO9TJX_HPCn3MxbXHkgKeOgdZuT1HDSsGVc/edit#heading=h.69vohvepyb6x
cc: @alolita