roachtest: acceptance/gossip/peerings failed

[cockroach-teamcity]

roachtest.acceptance/gossip/peerings failed with artifacts on master @ 2b0cff2d653829dce61fefff7f69d76013c43e63:

The test failed on branch=master, cloud=local:
test artifacts and logs in: artifacts/acceptance/gossip/peerings/run_1
	gossip.go:302,acceptance.go:104,test_runner.go:779: failed to get gossip status from node 4: Get "http://127.0.0.1:26264/_status/gossip/local": dial tcp 127.0.0.1:26264: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*gossipUtil).check.func1
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:199
		  | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*gossipUtil).check
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:194
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runGossipPeerings
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:301
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:104
		  | main.(*testRunner).runTest.func2
		  | 	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (2) failed to get gossip status from node 4
		Wraps: (3) Get "http://127.0.0.1:26264/_status/gossip/local"
		Wraps: (4) dial tcp 127.0.0.1:26264
		Wraps: (5) connect
		Wraps: (6) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *url.Error (4) *net.OpError (5) *os.SyscallError (6) syscall.Errno

	cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 4: dead (exit status unknown)

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

[stevendanna]

This also failed on a PR for me recently. In both cases, one of the nodes has gone down with the following fatal log message:

a52188f4f75f> F211204 00:40:09.814203 704 kv/kvserver/store_raft.go:348 ⋮ [n4,s4,r19/2:{-}] 85 while applying snapshot: while applying snapshot: error rate limiting bulk io write: context canceled
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:621
  | [...repeated from below...]
Wraps: (2) while applying snapshot
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.limitBulkIOWrite
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/syncing_write.go:54
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*SSTSnapshotStorageFile).Write
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_sst_snapshot_storage.go:183
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*SSTSnapshotStorageScratch).WriteSST
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_sst_snapshot_storage.go:118
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raftstorage.go:867
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:619
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:340
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:218
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:278
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_snapshot.go:663
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:82
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:72
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1.1
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:433
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:434
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:491
  | runtime.goexit
  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (4) error rate limiting bulk io write

[tbg]

Cc @nvanbenschoten this is probably fallout from being reactive to ctx cancellation now (sent from mobile, please excuse my brevity)

…

On Sat, Dec 4, 2021, 11:27 Steven Danna ***@***.***> wrote: This also failed on a PR for me recently. In both cases, one of the nodes has gone down with the following fatal log message: a52188f4f75f> F211204 00:40:09.814203 704 kv/kvserver/store_raft.go:348 ⋮ [n4,s4,r19/2:{-}] 85 while applying snapshot: while applying snapshot: error rate limiting bulk io write: context canceled (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:621 | [...repeated from below...] Wraps: (2) while applying snapshot Wraps: (3) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver.limitBulkIOWrite | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/syncing_write.go:54 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*SSTSnapshotStorageFile).Write | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_sst_snapshot_storage.go:183 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*SSTSnapshotStorageScratch).WriteSST | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_sst_snapshot_storage.go:118 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raftstorage.go:867 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:619 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:340 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:218 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:278 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_snapshot.go:663 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:82 | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr | /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:72 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1.1 | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:433 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1 | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:434 | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 | /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:491 | runtime.goexit | /usr/local/go/src/runtime/asm_amd64.s:1581 Wraps: (4) error rate limiting bulk io write — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#73465 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGXPZDINWCPEELGYAPXLF3UPHUKPANCNFSM5JK5NARA> .

[nvb]

Yes, this will be fixed by @erikgrinaker's #73484.