Skip to content

[Understanding CLP IR] How are Fluenbit generated IRs turned to Archives #13

@shadowshot-x

Description

@shadowshot-x

Hi CLP Team.

I have been exploring CLP and trying to understand more about the technology. I am able to generate zstd compressed IRs and upload to S3.

The piece which feels missing is how to convert these IRs to archives so I can Query them using Presto.

I am using CLP 0.7.0 and latest Fluentbit clp plugin

Command: ./clp-s c /mnt/logs/archives/ --timestamp-key '@time' /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir.zst
Error

2026-01-08T16:54:54.040+00:00 [error] Encountered curl error while ingesting /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir.zst - Code: 3 - Message: 
2026-01-08T16:54:54.041+00:00 [error] Could not deduce content type for input /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir.zst
2026-01-08T16:54:54.050+00:00 [error] Encountered error while parsing input.

Also tried decompressing the ZST
Command: ./clp-s c /mnt/logs/archives/ --timestamp-key '@time' /mnt/logs/test_0_2026-01-08T11_40_27+05_30_139bce15-095c-4ea3-b8ce-8a11ecf77aa6.ir
Error

2026-01-08T16:56:55.684+00:00 [error] Encountered error while creating kv-ir deserializer: (93) - Protocol not supported
2026-01-08T16:56:55.694+00:00 [error] Encountered error while parsing input.

If we can convert fluentbit generated IRs to archives, that will be a huge win for the platform!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions