Add support for gloo

[yuval-k]

This PR adds gloo\flagger integration.

It uses the gloo UpstreamGroup object. The advantage of using this is that the routes need only be updated in one place, and it will apply to all routes that point to that upstream group.

[stefanprodan]

@yuval-k I'm having a hard time testing this, the Gloo gateway proxy is buggy, most of the time the /metrics endpoint returns 503.

Prometheus fails to scrape Envoy, at most one call in ten will make it:

Same issue from inside the pod:

kubectl -n gloo-system exec -it gateway-proxy-5fc9c76b84-tgfst sh

/ # wget -q -O - localhost:8081/metrics
wget: server returned error: HTTP/1.1 503 Service Unavailable

I've reproduce this on several clusters, my conclusion is that the metrics endpoint returns 503 on 90% of the request. Both v0.13.27 and v0.13.28 have this issue.

[yuval-k]

Weird! I'll look in to the failure in the /metrics - thank you for letting me know.

[yuval-k]

metrics issue should be solved (it was access to admin port in general)

[yuval-k]

not sure why the travis job failed - maybe because lyft/protoc-gen-validate is now envoyproxy/protoc-gen-validate?

[stefanprodan]

I see that you’ve changed the metrics and the tests are failing. Run make test locally to see what’s failing.

[yuval-k]

good point!

[stefanprodan]

@yuval-k there is one more test failing:

--- FAIL: TestGlooRouter_Sync (0.00s)
    gloo_test.go:39: Got RoutingRule Destinations 2 wanted 1

[yuval-k]

thanks for the pointer!

[stefanprodan]

Is it possible to add support for A/B testing with headers and cookies filters? I've seen that Gloo has this capability but not with UpstreamGroup.

[yuval-k]

that's a good question.. here's the trade-off:
the goal of an UpstreamGroup is to separation of concerns; flagger owns the UpstreamGroup, and the operator owns the virtual service. so the operator only needs to route to the upstream group and flagger will take care of the rest. That way flagger doesn't need to edit the virtual service, and every route that points to the upstream group will be routed to the canary automatically.

So if we want to keep this separation of concerns, we would have to add header matching to upstream group. This will take time to implement in gloo, as now we'll have one route translating to multiple envoy routes, which is not something gloo currently supports.

An alternative would be to edit the VirtualService, but then there's a potential for error in understanding which route are own by flagger and which by the operator.

[stefanprodan]

Right! I think it makes sense to use delegation as in allow Flagger to own an object that's referenced in the main VirtualService and not operate on the VirtualService at all.

[yuval-k]

I agree - I think upstream groups are the better way to go. the problem is that adding the matching to an UpstreamGroup is non-trivial, as it requires two routes on the envoy level; while the current gloo implementation maps one gloo route to one envoy route.

[stefanprodan]

LGTM

I'll write the install docs and a short tutorial