Merge TiKV(bce01cf) into Proxy by hongyunyan · Pull Request #249 · pingcap/tidb-engine-ext

hongyunyan · 2023-01-04T04:59:24Z

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ref tikv#12901 Add `put_recover_from_raft_db` and `recover_from_raft_db` to raft engine for checking if store should be recovered from states in raftdb. Signed-off-by: 5kbpers <[email protected]>

ref tikv#12849 Support new engine trait can_apply_snapshot Signed-off-by: CalvinNeo <[email protected]>

ref tikv#13433 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…c keys (tikv#13426) close tikv#13425, ref pingcap/tidb#35525 Don't treat newer Rollback records as write conflicts for non-pessimistic keys in pessimistic transactions. They can cause false positive errors because they can be written even if the pessimistic lock of the corresponding row key exists. Rollback records are only used to prevent retried prewrite from succeeding. Even if the Rollback record of the current transaction is collapsed by a newer record, it is safe to prewrite this non-pessimistic key because either the primary key is rolled back or it's protected because it's written by CheckSecondaryLocks. Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13435 Add hook to observe the update of safe ts to calculate the TiFlash synchronization progress, including leader safe ts from check_leader RPC, and the update of self safe ts when receive check_leader RPC or apply. Signed-off-by: hehechen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13095 Update rust-rocksdb Signed-off-by: tabokie <[email protected]>

close tikv#13442 Signed-off-by: 3pointer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13232 Signed-off-by: Leavrth <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12849 Support self-defined addr/status_addr Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13452 Signed-off-by: haojinming <[email protected]>

ref tikv#13353 Signed-off-by: hehechen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12849 Support coprocessor observer pre_commit Signed-off-by: CalvinNeo <[email protected]>

close tikv#13417 Signed-off-by: YangKeao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…types (tikv#13387) close tikv#13386, ref pingcap/tidb#37481 Signed-off-by: YangKeao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13058 Some useless methods are not ported. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842 It's a super set of batch channel. It can be used as a batch channel or just a future channel. This is the first PR to use unified thread pool in apply system for v2. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…shared_db() (tikv#13465) close tikv#13463 Signed-off-by: BornChanger <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#11119 Signed-off-by: tabokie <[email protected]>

…s different with TiDB. (tikv#13472) close tikv#13458 Signed-off-by: Meng Xin <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12901 Set sequence number to `ApplyRes` Signed-off-by: 5kbpers <[email protected]> Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Jay <[email protected]>

ref tikv#13319, ref tikv#13322 Signed-off-by: SpadeA-Tang <[email protected]> Co-authored-by: Xinye Tao <[email protected]>

close tikv#13436 Signed-off-by: haojinming <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

) ref tikv#13406 Signed-off-by: SpadeA-Tang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…ult value update (tikv#13457) close tikv#13456 Signed-off-by: Leavrth <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13058 Everything is almost the same as engine_rocks except engine_tirocks put snapshot into an `Arc`. The old implementation is not safe as the snapshot can be released while the iterator is still being used. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12020, ref tikv#12842, ref tikv#13334 This PR implements write proposal and proposal batch. Compared to v1, a custom codec is introduced to make batch more efficient. Epoch checker will be added in next PR. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13483 Signed-off-by: gengliqi <[email protected]>

…store (tikv#13374) close tikv#13376 Signed-off-by: fengou1 <[email protected]> Signed-off-by: 3pointer <[email protected]> Signed-off-by: fengou1 <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: qupeng <[email protected]>

ref tikv#13303, close tikv#13448 Since deleting the old MVCC keys directly will cause the GC Compaction Filter to mishandle the deletion entry, this PR refactors the `FlashbackToVersionReadPhase` and `FlashbackToVersion` commands to use the MVCC overwriting to achieve the flashback. Signed-off-by: JmPotato <[email protected]>

…re (tikv#13028) close tikv#12344 Signed-off-by: fengou1 <[email protected]> Signed-off-by: 3pointer <[email protected]> Signed-off-by: fengou1 <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: qupeng <[email protected]>

ref tikv#12842 This PR fixes several issues: - raftstore CPU usage missing in grafana - restart fails because incorrect path asserts - restart fails because of missing flush records - get snapshots fails occasionally because of lease not renew Signed-off-by: Jay Lee <[email protected]>

ref tikv#12842 This PR makes check leader works by introducing a trait `ExternRegionInfo`. Signed-off-by: Jay Lee <[email protected]>

ref tikv#12842 Signed-off-by: bufferflies <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842 Signed-off-by: tabokie <[email protected]>

close tikv#13941, ref pingcap/tidb#39620 - If failed to get initial snapshot, remove the subscription as soon as possible. - Added a cache of getting checkpoint. This cache is lease-based -- the lease time is simply the tick interval of the coordinator. - Make the channel size huger for don't blocking the main loop when many regions migrating. Signed-off-by: hillium <[email protected]> Signed-off-by: hillium <[email protected]> Signed-off-by: 山岚 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

…v#13984) ref tikv#12842 Signed-off-by: tabokie <[email protected]>

ref tikv#12842 Signed-off-by: tabokie <[email protected]> Signed-off-by: Xinye Tao <[email protected]>

ref tikv#12842 Implement periodical purge in v2. Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842 If operations like snapshot, split, are aborted by restart, they needs to be either resumed or cleanup. This PR checks for garbage after restart and resume committed operations. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

CLAassistant · 2023-01-04T04:59:41Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
7 out of 14 committers have signed the CLA.

✅ dveeden
✅ lijie0123
✅ SpadeA-Tang
✅ HuSharp
✅ fengou1
✅ hongyunyan
✅ lance6716
❌ v01dstar
❌ BusyJay
❌ YuJuncen
❌ MyonKeminta
❌ tabokie
❌ LykxSassinator
❌ bufferflies
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Signed-off-by: hongyunyan <[email protected]>

CalvinNeo

lgtm

hongyunyan · 2023-01-05T06:35:31Z

/run-all-tests

hongyunyan · 2023-01-05T06:36:04Z

/run-build-check-old (pull_request)

Signed-off-by: hongyunyan <[email protected]>

5kbpers and others added 30 commits September 8, 2022 17:48

raft_engine: set recover_state (tikv#13272)

aebdada

ref tikv#12901 Add `put_recover_from_raft_db` and `recover_from_raft_db` to raft engine for checking if store should be recovered from states in raftdb. Signed-off-by: 5kbpers <[email protected]>

raftstore: Implement engine trait can_apply_snapshot (tikv#12924)

079a069

ref tikv#12849 Support new engine trait can_apply_snapshot Signed-off-by: CalvinNeo <[email protected]>

*: move ioload to tikv_util (tikv#13421)

3e86307

ref tikv#13433 Signed-off-by: Ryan Leung <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

engine: update rust-rocksdb (tikv#13393)

cc127a0

close tikv#13095 Update rust-rocksdb Signed-off-by: tabokie <[email protected]>

cloud: support backup to s3 when object lock enabled. (tikv#13350)

49223a7

close tikv#13442 Signed-off-by: 3pointer <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

log-backup: merge small files in each flush (tikv#13233)

63465fa

close tikv#13232 Signed-off-by: Leavrth <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

server: support customized addr/status_addr (tikv#13234)

8c93b91

ref tikv#12849 Support self-defined addr/status_addr Signed-off-by: CalvinNeo <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

test: Separate TestPdClient from test_raftstore component (tikv#13453)

2563311

close tikv#13452 Signed-off-by: haojinming <[email protected]>

resolved_ts: track ingest sst (tikv#13454)

b6b9638

ref tikv#13353 Signed-off-by: hehechen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

raftstore: Implement coprocessor observer pre_persist (tikv#12957)

592d423

ref tikv#12849 Support coprocessor observer pre_commit Signed-off-by: CalvinNeo <[email protected]>

copr: implement several mysql time related types json (tikv#13418)

d762bd8

close tikv#13417 Signed-off-by: YangKeao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

copr: don't convert bool to int while comparing json bool with other …

8e311cd

…types (tikv#13387) close tikv#13386, ref pingcap/tidb#37481 Signed-off-by: YangKeao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

engine_tirocks: port util (tikv#13462)

1179131

close tikv#13058 Some useless methods are not ported. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

multi-rocksdb: avoid mutex deadlock between open_tablet() and create_…

32f93fc

…shared_db() (tikv#13465) close tikv#13463 Signed-off-by: BornChanger <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

engine: update raft-engine (tikv#13464)

bdfde51

ref tikv#11119 Signed-off-by: tabokie <[email protected]>

tidb_query_expr: fix the problem that the result of castFloatAsTime i…

7348756

…s different with TiDB. (tikv#13472) close tikv#13458 Signed-off-by: Meng Xin <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

raftstore: set seqno in ApplyRes (tikv#12978)

fd84225

ref tikv#12901 Set sequence number to `ApplyRes` Signed-off-by: 5kbpers <[email protected]> Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Co-authored-by: Jay <[email protected]>

gc_worker: revert some draft changes in PR#13322 (tikv#13471)

2a67c47

ref tikv#13319, ref tikv#13322 Signed-off-by: SpadeA-Tang <[email protected]> Co-authored-by: Xinye Tao <[email protected]>

backup: Add rawkv ttl expired data metric (tikv#13438)

846b2bd

close tikv#13436 Signed-off-by: haojinming <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

gc_worker: make background gc compatible with multi-rocks db (tikv#13402

6fb3637

) ref tikv#13406 Signed-off-by: SpadeA-Tang <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

log-backup: update for compression type compatibility and config defa…

c168aba

…ult value update (tikv#13457) close tikv#13456 Signed-off-by: Leavrth <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

copr: support more regexp functions (tikv#13480)

bcfbd56

close tikv#13483 Signed-off-by: gengliqi <[email protected]>

BusyJay and others added 10 commits December 27, 2022 10:00

support check leader (tikv#13987)

5806cd1

ref tikv#12842 This PR makes check leader works by introducing a trait `ExternRegionInfo`. Signed-off-by: Jay Lee <[email protected]>

fix the bug that send mistake peer snapshot (tikv#13915)

929b329

ref tikv#12842 Signed-off-by: bufferflies <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

raftstore-v2: compact and gc raft logs (tikv#13846)

f21361d

ref tikv#12842 Signed-off-by: tabokie <[email protected]>

server: improve the ergonomics of sharing things between tablets (tik…

c3903b8

…v#13984) ref tikv#12842 Signed-off-by: tabokie <[email protected]>

raftstore-v2: gc tablets (tikv#13974)

77c2199

ref tikv#12842 Signed-off-by: tabokie <[email protected]> Signed-off-by: Xinye Tao <[email protected]>

raftstore-v2: purge raft engine (tikv#13993)

0404a7c

ref tikv#12842 Implement periodical purge in v2. Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

change by tikv Signed-off-by: hongyunyan <[email protected]>

5b5da3a

hongyunyan added 9 commits January 4, 2023 13:30

fix format

1e889fc

Signed-off-by: hongyunyan <[email protected]>

fix compile

99bda73

Signed-off-by: hongyunyan <[email protected]>

fix compile

028c154

Signed-off-by: hongyunyan <[email protected]>

fix test

eab3e0c

Signed-off-by: hongyunyan <[email protected]>

fix test

89ee249

Signed-off-by: hongyunyan <[email protected]>

fix test

b210b3f

Signed-off-by: hongyunyan <[email protected]>

fix test

444b419

Signed-off-by: hongyunyan <[email protected]>

fix test

3457613

Signed-off-by: hongyunyan <[email protected]>

fix test

7525c88

Signed-off-by: hongyunyan <[email protected]>

hongyunyan changed the title ~~merge tikv into proxy~~ merge tikv(bce01cf) into proxy Jan 5, 2023

hongyunyan changed the title ~~merge tikv(bce01cf) into proxy~~ Merge TiKV(bce01cf) into Proxy Jan 5, 2023

CalvinNeo approved these changes Jan 5, 2023

View reviewed changes

hongyunyan added 2 commits January 6, 2023 12:31

fix test

c23455e

Signed-off-by: hongyunyan <[email protected]>

for clone

a4e4358

hongyunyan merged commit 6ce5c72 into pingcap:raftstore-proxy Jan 7, 2023

LittleFall mentioned this pull request Jan 12, 2023

server info get wrong memory capacity pingcap/tiflash#6632

Closed

CalvinNeo added a commit that referenced this pull request Jan 28, 2023

[Cloud] Merge raftstore-proxy master(PR #249) (#259)

fcaa14a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge TiKV(bce01cf) into Proxy#249

Merge TiKV(bce01cf) into Proxy#249
hongyunyan merged 474 commits intopingcap:raftstore-proxyfrom
hongyunyan:hongyunyan-merge-tikv-master

hongyunyan commented Jan 4, 2023

Uh oh!

CLAassistant commented Jan 4, 2023 •

edited

Loading

Uh oh!

CalvinNeo left a comment

Uh oh!

hongyunyan commented Jan 5, 2023

Uh oh!

hongyunyan commented Jan 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

hongyunyan commented Jan 4, 2023

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

CLAassistant commented Jan 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CalvinNeo left a comment

Choose a reason for hiding this comment

Uh oh!

hongyunyan commented Jan 5, 2023

Uh oh!

hongyunyan commented Jan 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

CLAassistant commented Jan 4, 2023 •

edited

Loading