out_azure_kusto: Add changes to defer close old resource handles in subsequent cycles

[ag-ramachandran]

This pull request improves the safety and stability of resource management in the Azure Kusto output plugin by introducing deferred destruction of old resources. This change helps prevent use-after-free errors during high-volume operations where resources might still be in use by other threads when a refresh occurs.

Resource lifecycle management improvements:

• Added fields (old_blob_ha, old_queue_ha, old_identity_token) to struct flb_azure_kusto_resources to track old resources that are pending cleanup, enabling deferred destruction.
• Updated the resource loading logic in azure_kusto_load_ingestion_resources to first destroy resources from two refresh cycles ago, then move current resources to the "old" fields before assigning new ones, ensuring safe resource transitions.
• Modified flb_azure_kusto_resources_clear to clean up any old resources pending destruction, ensuring proper resource deallocation and preventing memory leaks.

Summary by CodeRabbit

• Bug Fixes
  • Improved Azure Kusto output stability under high-volume concurrent ingestion by adding deferred cleanup and two-stage rotation of connection and token resources to prevent use-after-free and race conditions.
  • Ensured safe cleanup of pending-old resources and added contextual debug/info logging for rotation and destruction to aid reliability and troubleshooting.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds three deferred-cleanup fields and implements a two-stage resource rotation in the Azure Kusto plugin to delay destruction of previous upstreams and identity tokens while new resources are installed, preventing use-after-free during high-volume concurrent operations.

Changes

Cohort / File(s)	Summary
Header Definitions plugins/out_azure_kusto/azure_kusto.h	Added three public fields to struct flb_azure_kusto_resources: old_blob_ha, old_queue_ha, and old_identity_token, plus a comment explaining deferred destruction to avoid use-after-free.
Resource Management plugins/out_azure_kusto/azure_kusto_conf.c	Added cleanup of old_* members in flb_azure_kusto_resources_clear. Implemented two-stage resource rotation in azure_kusto_load_ingestion_resources: destroy prior old_*, move current active resources to old_* slots, assign newly created blob/queue upstreams and identity token to active slots, update load_time, and emit debug/info logs for rotation and cleanup.

Sequence Diagram(s)

sequenceDiagram
    participant Loader as azure_kusto_load_ingestion_resources
    participant Parser as storage_parser
    participant Identity as IdentityService
    participant Active as ActiveResources
    participant Old as OldResources

    Loader->>Parser: parse storage config
    Parser-->>Loader: storage parameters
    Loader->>Identity: request new identity token
    Identity-->>Loader: return new token
    Loader->>Old: destroy existing old_blob_ha/old_queue_ha/old_identity_token (if any)
    Loader->>Old: move Active.blob_ha -> Old.old_blob_ha\nActive.queue_ha -> Old.old_queue_ha\nActive.identity_token -> Old.old_identity_token
    Loader->>Active: assign new blob_ha, queue_ha, identity_token
    Loader->>Loader: update load_time

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐇 I nibble threads and stash the old,

I swap the keys before they're cold.
Two stages now, I tuck and keep,
Let old ones rest while new ones leap.
Soft hops, safe swaps—no rush to fold.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title clearly summarizes the main change: deferring closure of old resource handles in the Azure Kusto output plugin to manage resource cleanup across refresh cycles.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cosmo0920]

The patch looks good but we need to add a Signed-off-by: line in each of commit.
And also, we need to use out_azure_kusto: prefix in this patch.

❌ Commit 33d6e3bd95 failed:
Missing prefix in commit subject: 'Add debug message'

❌ Commit 0c9b1b2b58 failed:
Missing prefix in commit subject: '* Add changes to defer close old resource handles in subsequent cycles'

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@plugins/out_azure_kusto/azure_kusto_conf.c`:
- Around line 616-638: The rotation logic in
azure_kusto_load_ingestion_resources moves and destroys HA and token objects
while worker functions azure_kusto_create_blob and azure_kusto_enqueue_ingestion
may access them without holding resources_mutex, leading to use-after-free; fix
by ensuring callers grab stable references under the mutex (or an equivalent
refcount) before using them: in azure_kusto_create_blob and
azure_kusto_enqueue_ingestion acquire resources_mutex, copy
ctx->resources->blob_ha/queue_ha and ctx->resources->identity_token into local
variables, call flb_upstream_ha_node_get (or increment the object's refcount)
while still holding the lock, then release the lock and use the obtained
node/token, and update azure_kusto_load_ingestion_resources to only destroy
old_* objects after they are no longer referenced (i.e., after refcount reaches
zero) or ensure destruction happens under the same mutex that prevents new gets.

[ag-ramachandran]

The patch looks good but we need to add a Signed-off-by: line in each of commit. And also, we need to use out_azure_kusto: prefix in this patch.

❌ Commit 33d6e3bd95 failed:
Missing prefix in commit subject: 'Add debug message'

❌ Commit 0c9b1b2b58 failed:
Missing prefix in commit subject: '* Add changes to defer close old resource handles in subsequent cycles'

Hello @cosmo0920, Fixed the comments. Tested as well and so far it looks okay. If it looks okay, kindly review/approve the same.