[HUDI-4937] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers #6815

alexeykudinkin · 2022-09-28T05:47:57Z

Change Logs

Currently, HoodieTable is holding HoodieBackedTableMetadata that is setup not to reuse actual LogScanner and HFileReader used to read MT itself.

This is proving to be wasteful on a number of occasions already, including (not an exhaustive list):

[SUPPORT] Incremental cleaning never used during insert #6373: Cleaner leveraging file-listing takes 5min, vs the MT based one taking 4h

Changes

Eliminating reuse flag from the HoodieBackedTableMetadata ctor to make sure readers are always reused
Holding HoodieBackedTableMetadata as Broadcast in HoodieTable to make sure it's being deserialized once per executor (instead of currently being once per task)
Made HoodieBackedTableMetadata serializable object (still omitting the readers though)
Restored HoodieSparkKryoRegistrar to be able to register serializer for SerializableConfiguration

Impact

Risk level: Low

Documentation Update

Documentation update is required now specifying that --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar property will be mandatory

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

nsivabalan · 2022-09-28T07:08:04Z

hey Alexey. I am not sure if we can completely remove the reuse flag.
for eg, even within the lifespan of a commit, when the writeClient started, the tableMetadata is instantiated ... and then index lookup, partitioner and then write to actual write (by executors). in preCommit, we do conflict resolution. So, even if we have not refreshed the hoodie timeline so far(implicitly file readers from metadata table ), we need to refresh it here to get the latest data.

Vinoth was involved while designing the reuse flag.

CC @prasannarajaperumal @vinothchandar

alexeykudinkin · 2022-09-29T20:57:29Z

@hudi-bot run azure

nsivabalan

This needs to be thoroughly vetted. We have some code paths, where we disabled reuse explicitly.

Can we list out all callers where re-use is false and we can confirm that enabling re-use is the right fix.

From what I glean, I see two places.

Metadata table writer initalized w/ reuse = false, which makes sense.

  static HoodieTableMetadata create(HoodieEngineContext engineContext, HoodieMetadataConfig metadataConfig, String datasetBasePath,
                                    String spillableMapPath) {
    return create(engineContext, metadataConfig, datasetBasePath, spillableMapPath, false);
  }

But this method has quite a few callers.

let's sync up sometime and go over the list to ensure we don't cause any regression w/ unintentional revert.

danny0405 · 2023-01-18T07:23:00Z

hudi-common/src/main/java/org/apache/hudi/common/fs/LeakTrackingFSDataInputStream.java

+
+  private static final Logger LOG = Logger.getLogger(LeakTrackingFSDataInputStream.class.getName());
+
+  private final StackTraceElement[] callSite;


This is only for debug purpose, right?

- Broadcasted instance of MT for Spark - Transient instance of MT for Flink This would allow to make sure that wherever `HoodieTable` is used MT will be broadcasted no more than once per executor

…streams potentially holding on to a S3 connections

…kedTableMetadata`

…ces held, as well as destroy held objects

…t` for ser/de to make sure clean up sequence doesn't re-fetch the `Broadcast` unnecessarily

… of holding the `Broadcast`

alexeykudinkin · 2023-01-24T05:23:00Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java

+    }
+  }
+
+  public static <T> HoodieSparkTable<T> create(HoodieWriteConfig config, HoodieEngineContext context) {


These ones were moved as is

alexeykudinkin · 2023-01-24T17:35:11Z

hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java

    super(config, context, metaClient);
  }

+  @Override


This just been moved; no changes (there were other changes, but these were reverted back)

alexeykudinkin · 2023-01-24T17:35:23Z

hudi-client/hudi-java-client/src/main/java/org/apache/hudi/table/HoodieJavaTable.java

  }

+  @Override
+  protected HoodieIndex getIndex(HoodieWriteConfig config, HoodieEngineContext context) {


hudi-bot · 2023-01-24T20:55:49Z

CI report:

e3b6af9 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

nsivabalan

LGTM except for 1 comment.

nsivabalan · 2023-01-24T23:02:34Z

hudi-common/src/main/java/org/apache/hudi/metadata/BaseTableMetadata.java

+  protected static final long MAX_MEMORY_SIZE_IN_BYTES = 1024 * 1024 * 1024;
+  // NOTE: Buffer-size is deliberately set pretty low, since MT internally is relying
+  //       on HFile (serving as persisted binary key-value mapping) to do caching
+  protected static final int BUFFER_SIZE = 10 * 1024; // 10Kb


did you get a chance to run any benchmarks w/ this fix?
even w. Hfile, if the inputStream size is set to 10Kb, while original hfile block size is 64Kb, we might make more remote calls just to fetch 1 hfile block right? instead of just 1.

alexeykudinkin · 2023-02-06T20:48:09Z

This change is currently blocked on HUDI-3397: now that HoodieTable is going to become a stateful object, we need to make sure that resources are cleaned up appropriately. This in turn requires to make sure that all of the actual write operations (except being committed to the timeline) be completed by the time WriteClient returns from its operations (which is currently not the case)

parisni · 2023-06-28T14:57:19Z

hi @nsivabalan ! is this supposed to be integrated in 0.14 ? As of 0.13.1, cleaning w/ MDT still is incredibly slow in this context:

huge number of partitions >= 100k
huge number of affected partitions >=10k OR incremental cleaning not kicks in b/c no files to clean at all

We recently had a cleaning job w/ MDT running for 24h while dropping the MDT the job ended in few minutes.

Thanks

yihua

Problems addressed by this PR are already resolved on the latest master. Closing this PR.

alexeykudinkin force-pushed the ak/mt-repars-fix branch from 2e3420c to 12160b8 Compare September 29, 2022 05:48

nsivabalan reviewed Oct 8, 2022

View reviewed changes

xushiyan added priority:high Significant impact; potential bugs metadata labels Oct 31, 2022

xushiyan assigned nsivabalan Oct 31, 2022

alexeykudinkin mentioned this pull request Nov 15, 2022

[HUDI-5012][HUDI-4921] Batch clean delete files retry #6890

Closed

4 tasks

alexeykudinkin added priority:critical Production degraded; pipelines stalled and removed priority:high Significant impact; potential bugs labels Nov 30, 2022

nsivabalan added the release-0.12.2 Patches targetted for 0.12.2 label Dec 6, 2022

alexeykudinkin removed the release-0.12.2 Patches targetted for 0.12.2 label Dec 6, 2022

alexeykudinkin force-pushed the ak/mt-repars-fix branch from 12160b8 to 0025243 Compare January 11, 2023 04:05

alexeykudinkin mentioned this pull request Jan 11, 2023

[HUDI-5534] Optimizing Bloom Index lookup when using Bloom Filters from Metadata Table #7642

Merged

4 tasks

alexeykudinkin force-pushed the ak/mt-repars-fix branch 2 times, most recently from 13fb788 to 3d90e88 Compare January 18, 2023 02:10

alexeykudinkin added priority:blocker Production down; release blocker and removed priority:critical Production degraded; pipelines stalled labels Jan 18, 2023

danny0405 reviewed Jan 18, 2023

View reviewed changes

alexeykudinkin force-pushed the ak/mt-repars-fix branch 3 times, most recently from f8826f0 to 987bcb8 Compare January 19, 2023 01:25

alexeykudinkin changed the title ~~[HUDI-4937] Fix HoodieTable injecting non-reusable HoodieBackedTableMetadata aggressively flushing MT readers~~ [HUDI-4937][Stacked on 7702] Fix HoodieTable injecting non-reusable HoodieBackedTableMetadata aggressively flushing MT readers Jan 19, 2023

alexeykudinkin force-pushed the ak/mt-repars-fix branch 7 times, most recently from 031dc62 to 97af245 Compare January 20, 2023 06:16

Alexey Kudinkin added 14 commits January 24, 2023 09:37

Revisited HoodieTable to hold

822e1dc

- Broadcasted instance of MT for Spark - Transient instance of MT for Flink This would allow to make sure that wherever `HoodieTable` is used MT will be broadcasted no more than once per executor

Make HoodieBackedTableMetadata serializable

df71694

Added LeakTrackingFSDataInputStream to track "leaked" (non-closed) …

c76049a

…streams potentially holding on to a S3 connections

Fixing compilation

10516b1

Reduced MT buffer size from 10Mb to 10Kb

9ba06fe

Fixed HoodieTable close to re-throw unchecked

e560c75

Avoid wiring HoodieSparkSessionExtension in "hudi-spark-client"

47d2bcf

Killed superfluous Hadoop's Configuration

058a991

Reverted HoodieTableFileSystemView to be serializable in `HoodieBac…

c8f0524

…kedTableMetadata`

Make sure ByteBuffer is being duplicated prior to being read out

8c7ffe0

Added destroy method to Transient allowing to free up any referen…

9709b1b

…ces held, as well as destroy held objects

Revisited HoodieSparkTable to hold Transient relying on `Broadcas…

64f35dc

…t` for ser/de to make sure clean up sequence doesn't re-fetch the `Broadcast` unnecessarily

Avoid holding additional SerializableConfiguration refs

e63c45c

Reverted HoodieTable to hold HoodieTableMetadata directly instead…

e3b6af9

… of holding the `Broadcast`

alexeykudinkin force-pushed the ak/mt-repars-fix branch from 2879baf to e3b6af9 Compare January 24, 2023 17:38

alexeykudinkin commented Jan 24, 2023

View reviewed changes

nsivabalan reviewed Jan 24, 2023

View reviewed changes

alexeykudinkin added priority:critical Production degraded; pipelines stalled and removed priority:blocker Production down; release blocker labels Jan 24, 2023

vinothchandar added the release-0.14.0 label Apr 25, 2023

nsivabalan removed the release-0.14.0 label Jul 3, 2023

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Feb 26, 2024

xushiyan removed the metadata label Dec 2, 2025

hudi-bot mentioned this pull request Dec 9, 2025

Fix HoodieTable injecting HoodieBackedTableMetadata not reusing underlying MT readers #15461

Open

yihua reviewed Dec 13, 2025

View reviewed changes

yihua closed this Dec 13, 2025

github-project-automation bot moved this from 🏗 Under discussion to ✅ Done in Hudi PR Support Dec 13, 2025


		private static final Logger LOG = Logger.getLogger(LeakTrackingFSDataInputStream.class.getName());

		private final StackTraceElement[] callSite;

[HUDI-4937] Fix HoodieTable injecting non-reusable HoodieBackedTableMetadata aggressively flushing MT readers #6815

[HUDI-4937] Fix HoodieTable injecting non-reusable HoodieBackedTableMetadata aggressively flushing MT readers #6815

Uh oh!

Conversation

alexeykudinkin commented Sep 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Documentation Update

Contributor's checklist

Uh oh!

nsivabalan commented Sep 28, 2022

Uh oh!

alexeykudinkin commented Sep 29, 2022

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

danny0405 Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 24, 2023

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 24, 2023

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 24, 2023

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Jan 24, 2023

CI report:

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

nsivabalan Jan 24, 2023

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin commented Feb 6, 2023

Uh oh!

parisni commented Jun 28, 2023

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

[HUDI-4937] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers #6815

[HUDI-4937] Fix `HoodieTable` injecting non-reusable `HoodieBackedTableMetadata` aggressively flushing MT readers #6815

alexeykudinkin commented Sep 28, 2022 •

edited

Loading