cgrpc: step `headers:` not applied to reflection RPC, fails on auth-required servers

[kimoto]

Summary

runn's cgrpc runner sends the server reflection RPC without any auth headers, so it fails against gRPC servers whose auth interceptor requires auth on every RPC (including reflection). There is currently no configuration to attach headers to the reflection RPC.

Mechanism

Reflection client is set up in connectAndResolve using a bare ctx, which propagates from run(). Step-level headers: are only applied to the final method call, not to the reflection RPC. When the server requires auth on reflection (Athenz, Cloud Run, Istio, Envoy with JWT, ...), ListServices() fails with UNAUTHENTICATED and runn returns the error before reaching the method call.

Compared to other clients

client	auth on reflection
grpcurl	automatic via -H
buf curl	--reflect-header
runn v1.9.2	none — headers: is method-only

Proof-of-concept fix (5 lines)

Threading step headers: into the reflection ctx works:

 func (rnr *grpcRunner) run(ctx context.Context, r *grpcRequest, s *step) error {
   o := s.parent
-  if err := rnr.connectAndResolve(ctx, o); err != nil {
+  reflCtx := setHeaders(ctx, r.headers)
+  if err := rnr.connectAndResolve(reflCtx, o); err != nil {
     return err
   }

Caveat: this PoC ties reflection auth to the first step's headers: (rnr.mds is cached after the first call) — a single header reused for both, like grpcurl.

Related

#1475 — feature request for runner-level metadata:. One possible API shape that would address the behavior described here.

Self-contained repro (server + patched runn build + 3-tool comparison)

#!/usr/bin/env bash
# requires: go, runn, grpcurl, buf
set -euo pipefail
T=$(mktemp -d) && cd "$T"

# auth-required gRPC server (rejects all RPCs incl. reflection without `Bearer demo123`)
cat > server.go <<'GO'
package main
import (
	"context"; "log"; "net"
	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/examples/helloworld/helloworld"
	"google.golang.org/grpc/metadata"
	"google.golang.org/grpc/reflection"
	"google.golang.org/grpc/status"
)
func authed(ctx context.Context) bool {
	md, _ := metadata.FromIncomingContext(ctx); a := md.Get("authorization")
	return len(a) > 0 && a[0] == "Bearer demo123"
}
type s struct{ helloworld.UnimplementedGreeterServer }
func (s) SayHello(_ context.Context, r *helloworld.HelloRequest) (*helloworld.HelloReply, error) {
	return &helloworld.HelloReply{Message: "hi " + r.GetName()}, nil
}
func main() {
	l, _ := net.Listen("tcp", "127.0.0.1:50051")
	d := status.Error(codes.Unauthenticated, "auth required")
	g := grpc.NewServer(
		grpc.UnaryInterceptor(func(c context.Context, r any, _ *grpc.UnaryServerInfo, h grpc.UnaryHandler) (any, error) {
			if !authed(c) { return nil, d }; return h(c, r)
		}),
		grpc.StreamInterceptor(func(srv any, ss grpc.ServerStream, _ *grpc.StreamServerInfo, h grpc.StreamHandler) error {
			if !authed(ss.Context()) { return d }; return h(srv, ss)
		}),
	)
	helloworld.RegisterGreeterServer(g, s{}); reflection.Register(g)
	log.Fatal(g.Serve(l))
}
GO
cat > repro.yml <<'YML'
runners: { cgrpc: { addr: "127.0.0.1:50051", tls: false } }
steps:
  RPC_SayHello:
    cgrpc:
      helloworld.Greeter/SayHello:
        headers: { authorization: "Bearer demo123" }
        message: { name: "w" }
    test: current.res.status == 0
YML
go mod init r >/dev/null 2>&1; go mod tidy >/dev/null 2>&1
go run server.go >/dev/null 2>&1 & SRV=$!; trap 'kill $SRV 2>/dev/null' EXIT; sleep 2

# clone runn v1.9.2 and apply the 5-line fix
git clone --depth 1 --branch v1.9.2 https://github.com/k1LoW/runn.git ./runn-src >/dev/null 2>&1
cat > p.patch <<'PATCH'
--- a/grpc.go
+++ b/grpc.go
@@ -151,3 +151,4 @@
 	o := s.parent
-	if err := rnr.connectAndResolve(ctx, o); err != nil {
+	reflCtx := setHeaders(ctx, r.headers)
+	if err := rnr.connectAndResolve(reflCtx, o); err != nil {
 		return err
PATCH
(cd ./runn-src && git apply ../p.patch && go build -o ../runn-patched ./cmd/runn) >/dev/null 2>&1

go version
echo "$(runn --version)"
echo "patched $(./runn-patched --version) (built from $(cd ./runn-src && git rev-parse --short HEAD))"
grpcurl -version 2>&1 | head -1
echo "buf curl $(buf --version)"
echo
echo "== runn         (reflection) → FAIL ==" ; runn run repro.yml || true
echo "== patched runn (reflection) → OK   ==" ; ./runn-patched run repro.yml
echo "== grpcurl -H                → OK   ==" ; grpcurl -plaintext -H "authorization: Bearer demo123" -d '{"name":"w"}' localhost:50051 helloworld.Greeter/SayHello
echo "== buf curl --reflect-header → OK   ==" ; buf curl --protocol grpc --http2-prior-knowledge --reflect-header "authorization: Bearer demo123" -H "authorization: Bearer demo123" -d '{"name":"w"}' http://127.0.0.1:50051/helloworld.Greeter/SayHello

Output (Mac, go1.26.4, runn v1.9.2, grpcurl 1.9.3, buf 1.71.0; run-local values like the runbook SHA and tmpdir path stripped for readability):

go version go1.26.4 darwin/arm64
runn version 1.9.2
patched runn version 1.9.2 (built from 67fa684)
grpcurl 1.9.3
buf curl 1.71.0

== runn         (reflection) → FAIL ==
F

1) repro.yml
  Failure/Error: gRPC request failed on "[No Description]".steps.RPC_SayHello: rpc error: code = Unauthenticated desc = auth required
  Failure step (repro.yml):
  3   RPC_SayHello:
  4     cgrpc:
  5       helloworld.Greeter/SayHello:
  6         headers: { authorization: "Bearer demo123" }
  7         message: { name: "w" }
  8     test: current.res.status == 0


1 scenario, 0 skipped, 1 failure
== patched runn (reflection) → OK   ==
.

1 scenario, 0 skipped, 0 failures
== grpcurl -H                → OK   ==
{
  "message": "hi w"
}
== buf curl --reflect-header → OK   ==
{
  "message": "hi w"
}