Skip to content

log-backup: merge small files in each flush#13233

Merged
ti-chi-bot merged 35 commits intotikv:masterfrom
Leavrth:stream_reduce_small_files
Sep 13, 2022
Merged

log-backup: merge small files in each flush#13233
ti-chi-bot merged 35 commits intotikv:masterfrom
Leavrth:stream_reduce_small_files

Conversation

@Leavrth
Copy link
Contributor

@Leavrth Leavrth commented Aug 5, 2022 •
edited
Loading

Signed-off-by: Leavrth [email protected]

What is changed and how it works?

Issue Number: Close #13232

What's Changed:

  1. merge small files in each flush.
  2. compress the files before upload the merged file.
  3. download the merged file in part.

Related changes

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)
  • No code
    test-plan

Release note 

None.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 5, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • YuJuncen
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. contribution This PR is from a community contributor. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 5, 2022
@Leavrth Leavrth changed the title merge small files in each flush log-backup: merge small files in each flush Aug 5, 2022
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2022
3pointer
3pointer reviewed Aug 5, 2022
View reviewed changes
YuJuncen
YuJuncen reviewed Aug 5, 2022
View reviewed changes
Leavrth and others added 12 commits August 5, 2022 17:26
@Leavrth
resolve conflicts
06dd80e 
Signed-off-by: Leavrth <[email protected]>
@Leavrth
commit some suggestions
88f1973 
Signed-off-by: Leavrth <[email protected]>
@Leavrth
skip the empty metadata
cead5e6 
Signed-off-by: Leavrth <[email protected]>
@Leavrth
support for azure blob
33dd28c 
Signed-off-by: Leavrth <[email protected]>
@Leavrth
reback kvproto
804ca6f 
Signed-off-by: Leavrth <[email protected]>
@Leavrth
Merge branch 'master' into stream_reduce_small_files
cd3d89c
@Leavrth
tidy cargo.toml
9b8e9d5 
Signed-off-by: Leavrth <[email protected]>
@Leavrth
Merge branch 'stream_reduce_small_files' of github.com:Leavrth/tikv i…
989d69f 
…nto stream_reduce_small_files
@Leavrth
Merge branch 'master' into stream_reduce_small_files
9d4e611
@Leavrth
add unit test
cfde070 
Signed-off-by: Leavrth <[email protected]>
@Leavrth
Merge branch 'stream_reduce_small_files' of github.com:Leavrth/tikv i…
e3c044d 
…nto stream_reduce_small_files
@Leavrth
fmt & clippy
5bd0d83 
Signed-off-by: Leavrth <[email protected]>
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 25, 2022
joccau
joccau reviewed Aug 25, 2022
View reviewed changes
joccau
joccau reviewed Aug 25, 2022
View reviewed changes
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 26, 2022
@Leavrth Leavrth mentioned this pull request Sep 2, 2022
Open
@Leavrth Leavrth mentioned this pull request Sep 2, 2022
Merged
13 tasks
@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Sep 8, 2022
YuJuncen
YuJuncen approved these changes Sep 8, 2022
View reviewed changes
Copy link
Contributor

@YuJuncen YuJuncen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 8, 2022
joccau
joccau approved these changes Sep 13, 2022
View reviewed changes
Copy link
Contributor

@joccau joccau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 13, 2022

@joccau: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

Details

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

overvenus
overvenus reviewed Sep 13, 2022
View reviewed changes
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 13, 2022
@overvenus
Copy link
Member

overvenus commented Sep 13, 2022

/merge

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 13, 2022

@overvenus: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 13, 2022

This pull request has been accepted and is ready to merge.

DetailsCommit hash: 03a4fab

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 13, 2022
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 13, 2022

@Leavrth: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 63465fa into tikv:master Sep 13, 2022
@ti-chi-bot ti-chi-bot added this to the Pool milestone Sep 13, 2022
lizhenhuan pushed a commit to lizhenhuan/tikv that referenced this pull request Sep 21, 2022
@Leavrth @ti-chi-bot @lizhenhuan
log-backup: merge small files in each flush (tikv#13233)
471b1da 
close tikv#13232

Signed-off-by: Leavrth <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
Signed-off-by: lizhenhuan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@overvenus overvenus overvenus approved these changes

@YuJuncen YuJuncen YuJuncen approved these changes

+2 more reviewers

@3pointer 3pointer 3pointer left review comments

@joccau joccau joccau approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

contribution This PR is from a community contributor. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

log-backup: merge small files in each flush

6 participants

@Leavrth @ti-chi-bot @overvenus @3pointer @YuJuncen @joccau