sql/pgwire: TestAuthenticationAndHBARules failed

[cockroach-teamcity]

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.2 @ 88018946832e96e689c8f180f9f4621ae2f61834:

        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence/root (0.02s)
=== RUN   TestAuthenticationAndHBARules/insecure=false
    --- FAIL: TestAuthenticationAndHBARules/insecure=false (33.95s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence1877545467
    test_server_shim.go:93: cluster virtualization disabled due to issue: #107310 (expected label: C-bug)
    conditional_wrap.go:190: 
        pkg/sql/pgwire_test/pkg/sql/pgwire/auth_test.go:236: (hbaRunTest.func1)
        	NOTICE: .GetAdminHTTPClient() called via implicit interface ApplicationLayerInterface;
        HINT: consider using .ApplicationLayer().GetAdminHTTPClient() instead.
    conditional_wrap.go:194: TIP: consider replacing the test server initialization from:
            ts, ... := serverutils.StartServer(t, ...)
            defer ts.Stopper().Stop(...)
        to:
            srv, ... := serverutils.StartServer(t, ...)
            defer srv.Stopper().Stop(...)
            ts := srv.ApplicationLayer()
        
        See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/13395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:9:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/13395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/hba_default_equivalence:12:
        set_hba [0 args]
        host  all root all cert-password
        host  all all  all cert-password
        local all all      password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password
        # host  all all  all cert-password
        # local all all      password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS METHOD        OPTIONS
        loopback all      all  all     trust
        host     all      root all     cert-password
        host     all      all  all     cert-password
        local    all      all          password
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_hba_default_equivalence1877545467
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/hba_default_equivalence (1.59s)

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• sql/pgwire: TestAuthenticationAndHBARules failed #126063 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-sql-foundations branch-release-23.2.7-rc]
• sql/pgwire: TestAuthenticationAndHBARules failed #124629 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1]

/cc @cockroachdb/sql-foundations @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-39809

[rimadeodhar]

Would this be prod-security?

[rafiss]

There is overlap with what this test covers. Some of it is related to the pgwire protocol, which SQL Foundations does own. But the behaviors around 3rd party authentication are under prod-sec.

In this case, the error is even more mysterious. Even though CRDB is running with insecure=false, the test hit the error: pq: SSL is not enabled on the server.

This message comes from the lib/pq driver: https://github.com/lib/pq/blob/3d613208bca2e74f2a20e04126ed30bcb5c4cc27/conn.go#L1128-L1130

It should only hit that error if the server fails to send back an S response as part of the connection handshake. However, I don't see how it's possible for that to happen:

cockroach/pkg/sql/pgwire/server.go

Lines 212 to 215 in efd3306

	var (
	sslSupported = []byte{'S'}
	sslUnsupported = []byte{'N'}
	)

cockroach/pkg/sql/pgwire/pre_serve.go

Lines 475 to 487 in efd3306

	if tlsConfig == nil {
	// We don't have a TLS configuration available, so we can't honor
	// the client's request.
	n, serverErr = conn.Write(sslUnsupported)
	if serverErr != nil {
	return
	}
	} else {
	// We have a TLS configuration. Upgrade the connection.
	n, serverErr = conn.Write(sslSupported)
	if serverErr != nil {
	return
	}

[cockroach-teamcity]

sql/pgwire.TestAuthenticationAndHBARules failed with artifacts on release-23.2 @ 1d0b703cfb44b065483eba85d1df620e0af32e78:

        HINT: consider using .ApplicationLayer().GetAdminHTTPClient() instead.
    conditional_wrap.go:194: TIP: consider replacing the test server initialization from:
            ts, ... := serverutils.StartServer(t, ...)
            defer ts.Stopper().Stop(...)
        to:
            srv, ... := serverutils.StartServer(t, ...)
            defer srv.Stopper().Stop(...)
            ts := srv.ApplicationLayer()
        
        See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/488/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:3:
        config [1 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/488/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:6:
        sql [0 args]
        CREATE USER passworduser WITH PASSWORD 'pass'
        ----
        ok
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/28616e575ae934b92eba220441c7ca42/logTestAuthenticationAndHBARules_insecure=false_special_cases1375380984
        --- FAIL: TestAuthenticationAndHBARules/insecure=false/special_cases (1.31s)
=== RUN   TestAuthenticationAndHBARules/insecure=false/special_cases/root_user_cannot_use_password
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/488/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:17:
        set_hba [0 args]
        host all root 0.0.0.0/0 password
        ----
        # Active authentication configuration on this node:
        # Original configuration:
        # loopback all all all trust       # built-in CockroachDB default
        # host  all root all cert-password # CockroachDB mandatory rule
        # host all root 0.0.0.0/0 password
        #
        # Interpreted configuration:
        # TYPE   DATABASE USER ADDRESS   METHOD        OPTIONS
        loopback all      all  all       trust
        host     all      root all       cert-password
        host     all      root 0.0.0.0/0 password
    datadriven.go:259: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/488/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/pgwire/pgwire_test_/pgwire_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/pgwire/testdata/auth/special_cases:32:
         
        expected:
        ERROR: password authentication failed for user root (SQLSTATE 28P01)
        
        found:
        ERROR: pq: SSL is not enabled on the server
            --- FAIL: TestAuthenticationAndHBARules/insecure=false/special_cases/root_user_cannot_use_password (0.01s)

Parameters:

• TAGS=bazel,gss
• stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• sql/pgwire: TestAuthenticationAndHBARules failed #127745 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot T-sql-foundations branch-release-23.1 release-blocker]
• sql/pgwire: TestAuthenticationAndHBARules failed #126758 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.1.24-rc]
• sql/pgwire: TestAuthenticationAndHBARules failed #126063 sql/pgwire: TestAuthenticationAndHBARules failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-23.2.7-rc]

This test on roachdash | Improve this report!