Backup fails with "404 page not found"

[overvenus]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

bin/br backup full --pd pd0:2379 --storage "local:///data/backup/full"

2. What did you expect to see?

Backup success.

3. What did you see instead?

Backup fails with an error in details.

Details

[2020/11/23 11:34:37.628 +00:00] [INFO] [ddl.go:308] ["[ddl] stop DDL"] [ID=1a25cf50-eb60-42cd-ad4b-424fe3235882]
[2020/11/23 11:34:37.629 +00:00] [INFO] [domain.go:1466] ["serverIDKeeper exited."]
[2020/11/23 11:34:37.629 +00:00] [INFO] [domain.go:654] ["domain closed"] ["take time"=1.575145ms]
[2020/11/23 11:34:37.629 +00:00] [ERROR] [tidb.go:83] ["[ddl] init domain failed"] [error="404 page not found\n"]

errorVerbose="404 page not found

github.com/pingcap/tidb/domain/infosync.doRequest
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/domain/infosync/info.go:339
github.com/pingcap/tidb/domain/infosync.GetAllRuleBundles
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/domain/infosync/info.go:371
github.com/pingcap/tidb/domain.(*Domain).loadInfoSchema
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/domain/domain.go:152
github.com/pingcap/tidb/domain.(*Domain).Reload
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/domain/domain.go:389
github.com/pingcap/tidb/domain.(*Domain).Init
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/domain/domain.go:794
github.com/pingcap/tidb/session.(*domainMap).Get.func1
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/session/tidb.go:79
github.com/pingcap/tidb/util.RunWithRetry
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/util/misc.go:62
github.com/pingcap/tidb/session.(*domainMap).Get
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/session/tidb.go:70
github.com/pingcap/tidb/session.GetDomain
	github.com/pingcap/tidb@v0.0.0-20201110015039-8d35f17c17f3/session/session.go:1981
github.com/pingcap/br/pkg/gluetidb.Glue.GetDomain
	github.com/pingcap/br@/pkg/gluetidb/glue.go:52
github.com/pingcap/br/pkg/conn.NewMgr
	github.com/pingcap/br@/pkg/conn/conn.go:151
github.com/pingcap/br/pkg/task.NewMgr
	github.com/pingcap/br@/pkg/task/common.go:363
github.com/pingcap/br/pkg/task.RunBackup
	github.com/pingcap/br@/pkg/task/backup.go:181
github.com/pingcap/br/cmd.runBackupCommand
	github.com/pingcap/br@/cmd/backup.go:24
github.com/pingcap/br/cmd.newFullBackupCommand.func1
	github.com/pingcap/br@/cmd/backup.go:87
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.0.0/command.go:842
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.0.0/command.go:950
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.0.0/command.go:887
main.main
	github.com/pingcap/br@/main.go:58
runtime.main
	runtime/proc.go:203
runtime.goexit
	runtime/asm_amd64.s:1357

4. What version of BR and TiDB/TiKV/PD are you using?

br -V

tidb-server -V
Release Version: v4.0.0-beta-282-g4af313c2f
Git Commit Hash: 4af313c2f5fd40a07c59e7cf94f312c3896cc2e9
Git Branch: master
UTC Build Time: 2020-03-03 10:01:26
GoVersion: go1.13
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

tikv-server -V
TiKV
Release Version: 4.0.0-rc.2
Edition: Community
Git Commit Hash: afaaf1287e4d7bdd1f5be532f3c9a5d543881d66
Git Commit Branch: release-4.0-defaultnotfound
UTC Build Time: 2020-05-25 03:27:01
Rust Version: rustc 1.42.0-nightly (0de96d37f 2019-12-19)
Enable Features: jemalloc portable sse protobuf-codec
Profile: dist_release

pd-server -V
Release Version: v4.0.0-beta.1-12-g8ccbcd36
Git Commit Hash: 8ccbcd36d6e4210af69152672797ac716ba947d1
Git Branch: master
UTC Build Time: 2020-03-02 05:38:37

[kennytm]

Some customized version of PD does not have the "placement-rule" API.

IMO TiDB's GetAllRuleBundles() should fallback to return an empty slice if PD returned 404.

[rleungx]

I'm going to take a look at this issue.