[SUPPORT] data loss in new base file.

[wqwl611]

Describe the problem you faced
I find some data loss in the new base file: [00000000-9e95-4471-bba0-5604a282aa34-0_0-12-4_20230208003459996.parquet].
I doubt that compaction plan may miss some delta log.
How can I check the archive compaction plan?

Environment Description

• Hudi version : 0.11.1

• Spark version : 3.2.0

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) : hdfs

• Running on Docker? (yes/no) :no

[yihua]

Hi @wqwl611 thanks for raising this. Could you clarify what kind of data loss do you observe (missing records, updates not applied, missing columns, etc.)? Also, are there any failures or other table services running before the compaction happened, based on the Hudi timeline? Which Hudi release are you using? Does the data loss happen on the read side?

To inspect the compaction plan, you may use Hudi CLI compaction commands or directly check the requested compaction instant under .hoodie/(avrocat <instant_time>.compaction.requested). If the compaction commit is archived, you may only look at the archived file for now to understand the compaction plan.

[wqwl611]

@yihua Thanks for reply. the loss is missing records. I find no service failures. I use hudi0.11.1.
I can't read the loss data by snapshot read, but I can read the loss data by timetravel read with specified instant time.
Is there any tool to help inspect the archived file?

[wqwl611]

@yihua I can find all other delta file in arvhive file except the log: .00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.6_0-5545-224979

[yihua]

Hi @wqwl611 could you check the Hudi timeline and see which commits wrote the log files with the prefix .00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703? and how's the sequence of the commits in the Hudi timeline? That'll help us understand the sequence of events and see if there's any bug in MOR snapshot read or compaction.

[yihua]

@wqwl611 do you use Slack or Wechat? Feel free to DM me for faster iteration.

[wqwl611]

Hi @wqwl611 could you check the Hudi timeline and see which commits wrote the log files with the prefix .00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703? and how's the sequence of the commits in the Hudi timeline? That'll help us understand the sequence of events and see if there's any bug in MOR snapshot read or compaction.

@yihua I check the active timeline，and can't not find which commit contains this log, it should be archived, but I can't find this delta log in the archived file as well.

[wqwl611]

the following is relate compaction meta in archive file, and it don't contain the delta log[.00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.6_0-5545-224979], and I don't find the same log in other meta eithger.
@yihua

{"hoodieCommitMetadata": null, "hoodieCleanMetadata": null, "hoodieCompactionMetadata": null, "hoodieRollbackMetadata": null, "hoodieSavePointMetadata": null, "commitTime": "20230208003459996", "actionType": "compaction", "version": null, "hoodieCompactionPlan": {"operations": [{"baseInstantTime": "20230208001523703", "deltaFilePaths": [".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.1_0-26568-763035", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.1_0-26748-768915", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.2_0-26568-763035", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.3_0-26748-768915", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.4_0-9952-310687", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.5_1-0-1"], "dataFilePath": "00000000-9e95-4471-bba0-5604a282aa34-0_1-8918-275143_20230208001523703.parquet", "fileId": "00000000-9e95-4471-bba0-5604a282aa34-0", "partitionPath": "_db=xx/_tbl=hh/_id_mod=12", "metrics": {"TOTAL_LOG_FILES": 6.0, "TOTAL_IO_READ_MB": 5930.0, "TOTAL_LOG_FILES_SIZE": 6.149091603E9, "TOTAL_IO_WRITE_MB": 66.0, "TOTAL_IO_MB": 5996.0}, "bootstrapFilePath": null}, {"baseInstantTime": "20230208000815127", "deltaFilePaths": [".00000000-9e25-4c39-b05c-657a82bbcf63-0_20230208000815127.log.1_0-8899-274704", ".00000000-9e25-4c39-b05c-657a82bbcf63-0_20230208000815127.log.1_0-8918-275142"], "dataFilePath": "00000000-9e25-4c39-b05c-657a82bbcf63-0_0-3-3_20230208000815127.parquet", "fileId": "00000000-9e25-4c39-b05c-657a82bbcf63-0", "partitionPath": "_db=xx/_tbl=xx/_id_mod=11", "metrics": {"TOTAL_LOG_FILES": 2.0, "TOTAL_IO_READ_MB": 2371.0, "TOTAL_LOG_FILES_SIZE": 2.045651049E9, "TOTAL_IO_WRITE_MB": 420.0, "TOTAL_IO_MB": 2791.0}, "bootstrapFilePath": null}], "extraMetadata": {}, "version": 2}, "hoodieCleanerPlan": null, "actionState": "REQUESTED", "hoodieReplaceCommitMetadata": null, "hoodieRequestedReplaceMetadata": null, "HoodieInflightReplaceMetadata": null, "hoodieIndexCommitMetadata": null}
{"hoodieCommitMetadata": null, "hoodieCleanMetadata": null, "hoodieCompactionMetadata": null, "hoodieRollbackMetadata": null, "hoodieSavePointMetadata": null, "commitTime": "20230208003459996", "actionType": "compaction", "version": null, "hoodieCompactionPlan": {"operations": [{"baseInstantTime": "20230208001523703", "deltaFilePaths": [".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.1_0-26568-763035", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.1_0-26748-768915", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.2_0-26568-763035", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.3_0-26748-768915", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.4_0-9952-310687", ".00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.5_1-0-1"], "dataFilePath": "00000000-9e95-4471-bba0-5604a282aa34-0_1-8918-275143_20230208001523703.parquet", "fileId": "00000000-9e95-4471-bba0-5604a282aa34-0", "partitionPath": "_db=xx/_tbl=hh/_id_mod=12", "metrics": {"TOTAL_LOG_FILES": 6.0, "TOTAL_IO_READ_MB": 5930.0, "TOTAL_LOG_FILES_SIZE": 6.149091603E9, "TOTAL_IO_WRITE_MB": 66.0, "TOTAL_IO_MB": 5996.0}, "bootstrapFilePath": null}, {"baseInstantTime": "20230208000815127", "deltaFilePaths": [".00000000-9e25-4c39-b05c-657a82bbcf63-0_20230208000815127.log.1_0-8899-274704", ".00000000-9e25-4c39-b05c-657a82bbcf63-0_20230208000815127.log.1_0-8918-275142"], "dataFilePath": "00000000-9e25-4c39-b05c-657a82bbcf63-0_0-3-3_20230208000815127.parquet", "fileId": "00000000-9e25-4c39-b05c-657a82bbcf63-0", "partitionPath": "_db=xx/_tbl=xx/_id_mod=11", "metrics": {"TOTAL_LOG_FILES": 2.0, "TOTAL_IO_READ_MB": 2371.0, "TOTAL_LOG_FILES_SIZE": 2.045651049E9, "TOTAL_IO_WRITE_MB": 420.0, "TOTAL_IO_MB": 2791.0}, "bootstrapFilePath": null}], "extraMetadata": {}, "version": 2}, "hoodieCleanerPlan": null, "actionState": "INFLIGHT", "hoodieReplaceCommitMetadata": null, "hoodieRequestedReplaceMetadata": null, "HoodieInflightReplaceMetadata": null, "hoodieIndexCommitMetadata": null}

[yihua]

@wqwl611 if you cannot find .00000000-9e95-4471-bba0-5604a282aa34-0_20230208001523703.log.6_0-5545-224979 in any commit metadata, it's likely that the log block(s) in this log file is written by a failed commit, so it is not included in the compaction plan, which is expected.

[danny0405]

Hi, @wqwl611 , guess you encountered the bug introduced in release 0.11.x: #6179