Implement clear parquet metadata cache command #713

arthurpassos · 2025-04-04T12:45:22Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

SYSTEM DROP PARQUET METADATA CACHE command

Documentation entry for user-facing changes

altinity-robot · 2025-04-04T13:35:39Z

This is an automated comment for commit 6971379 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ error
Regression aarch64 Tiered Storage s3amazon	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ failure
Regression aarch64 Tiered Storage s3gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ failure
Sign aarch64	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ error
Sign release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ error
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	❌ failure

Successful checks

Check name	Description	Status
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Ready for release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Alter move partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Benchmark aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Benchmark gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Benchmark minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 LDAP authentication	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 LDAP external_user_directory	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 LDAP role_mapping	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Parquet aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Parquet minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Parquet	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 aes_encryption	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 atomic_insert	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 base_58	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 datetime64_extended_range	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 dns	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 engines	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 example	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 extended_precision_data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 kafka	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 kerberos	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 key_value	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 memory	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 part_moves_between_shards	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 selects	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 session_timezone	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 window_functions	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Alter move partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Benchmark aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Benchmark gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Benchmark minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success

arthurpassos · 2025-04-06T15:36:17Z

I have added some tests, but haven't executed them locally yet. When I get home, with better internet connection to download dependencies, I'll run it

ianton-ru · 2025-04-07T14:55:11Z

src/Interpreters/InterpreterSystemQuery.cpp

+            getContext()->checkAccess(AccessType::SYSTEM_DROP_PARQUET_METADATA_CACHE);
+            ParquetFileMetaDataCache::instance()->clear();
+#endif
+            break;


NIT: May be throw exception when USE_PARQUET not defined, as for USE_EMBEDDED_COMPILER below?
(I don't think than sometimes someone assemble clickhouse without parquet support)

Enmk

LGTM

Implement clear parquet metadata cache command

it was introduced in #713

implement clear parquet metadata cache command, no tests for cluster yet

44faec6

arthurpassos changed the title ~~implement clear parquet metadata cache command, no tests for cluster yet~~ Implement clear parquet metadata cache command Apr 4, 2025

ianton-ru previously approved these changes Apr 5, 2025

View reviewed changes

add cluster tests

6971379

arthurpassos dismissed ianton-ru’s stale review via 6971379 April 6, 2025 15:35

arthurpassos added 6 commits April 7, 2025 09:32

Merge branch 'antalya' into clear_parquet_metadata_cache

e53b670

Update test.py

58f3cb1

Update test.py

0baf53d

move config dir

7f60fc2

fix minioendpoint and withminio

f612bd5

fix integ tests

32f5d58

ianton-ru previously approved these changes Apr 7, 2025

View reviewed changes

arthurpassos dismissed ianton-ru’s stale review via f0e46ab April 7, 2025 14:59

throw in case support is disabled and command is used

97ca59e

arthurpassos force-pushed the clear_parquet_metadata_cache branch from f0e46ab to 97ca59e Compare April 7, 2025 15:02

Enmk approved these changes Apr 7, 2025

View reviewed changes

Enmk merged commit 6b8543b into antalya Apr 7, 2025
169 of 264 checks passed

MyroTk added the antalya-25.2.2 Planned for 25.2.2 release label Apr 7, 2025

vzakaznikov added antalya antalya-25.2 labels May 22, 2025

arthurpassos pushed a commit that referenced this pull request May 22, 2025

Merge pull request #713 from Altinity/clear_parquet_metadata_cache

3445959

Implement clear parquet metadata cache command

Enmk mentioned this pull request May 27, 2025

25.3 Antalya port - Parquet metadata caching #795

Merged

13 tasks

Enmk added a commit that referenced this pull request Aug 7, 2025

Removed test_parquet_drop_metadata_cache

9105bb0

it was introduced in #713

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement clear parquet metadata cache command #713

Implement clear parquet metadata cache command #713

Uh oh!

arthurpassos commented Apr 4, 2025

Uh oh!

altinity-robot commented Apr 4, 2025 •

edited

Loading

Uh oh!

arthurpassos commented Apr 6, 2025

Uh oh!

ianton-ru Apr 7, 2025

Uh oh!

Enmk left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Implement clear parquet metadata cache command #713

Implement clear parquet metadata cache command #713

Uh oh!

Conversation

arthurpassos commented Apr 4, 2025

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

altinity-robot commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arthurpassos commented Apr 6, 2025

Uh oh!

ianton-ru Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

Enmk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

altinity-robot commented Apr 4, 2025 •

edited

Loading