azure_ai_foundry.py is working a treat, but I am getting too many sources

[ThoSt81]

Bug title

Only 4 or 5 sources are shown in the response but many more are added

Describe the bug

In this example I had 3 sources added (the ones mentioned in the response) but after 1 second, more got added.
The first 3 also have a high score (80%) and the rest is 50% or lower.

I tried turing off followup generation but this did not have an effect.

I tried analyzing this with Copilot but could not apply a fix:

What you’re seeing

During streaming, your function emits only the citations that are actually referenced in the content (e.g., [doc1], [doc2]). That’s by design in __emit_openwebui_citation_events which filters by the indices it finds in the text.
Shortly after the stream finishes, additional citations appear that are not referenced in the generated content.

Why this happens
In the streaming path, you do not remove or filter the raw context.citations included in the final message JSON that comes from Azure. Your code only emits filtered citation events, but it leaves the provider’s full context.citations list intact in the SSE JSON that the UI also receives. The UI then renders those extra citations when it processes the final message, which looks like “more citations arriving a bit later.”
Concretely:

_emit_openwebui_citation_events(...) filters to only the referenced [docX] indices. If none are found, it falls back to all citations (for backward compatibility).
stream_processor_with_citations(...) updates content (turning [docX] into clickable links) but does not prune choices[].message.context.citations (or choices[].delta.context...) from the SSE JSON lines it forwards.
In the non‑stream path, enhance_azure_search_response(...) converts [docX] to links but likewise does not filter the citations array in the returned response.

As a result, you get two sources of citations in the UI:

Your filtered OpenWebUI citation events (correct, only referenced), and
The provider’s unfiltered context.citations included in the final message (causing extra, unreferenced cards to appear afterwards).

Steps to reproduce

1. Create new funtion
2. Connect with azure ai and azure ai data source
3. Chat

Environment

OWUI latest version
Latest azure_ai_foundry.py

Additional context

No response

[owndev]

Your errors are strange, because for me, the documents that aren't mentioned are filtered out correctly.

I had Copilot implement changes.

Could you test the following versions and give feedback on which one is better or whether the problem has been resolved:

• 7d6778116f8092a55a6da88fcbf6dfb74cbfc998
• f5d80825fa4b78d3951429653113337c06b68719

[ThoSt81]

Neither worked for me. I did some more troubleshooting with Copilot and found that a regex is not matching and thus all citations are emitted:

_extract_referenced_citations(content)
Uses DOC_REF_PATTERN = re.compile(r"\[doc(\d+)\]") to find markers like [doc1], [doc2], … in the content and returns a set of those indices. If the model outputs a slightly different format (e.g., [doc 1], [doc(1)], [[doc1]], or uppercase), the current regex won’t match, causing referenced_indices to be empty and triggering the “emit all citations” fallback.

As a workaround, I have now changed

# If we couldn't find any references, include all citations (backward compatibility)
if not referenced_indices:
    referenced_indices = set(range(1, len(citations) + 1))
    log.debug(
        f"No [docX] references found in content, including all {len(citations)} citations"
)

to:

# If we couldn't find any references, DO NOT emit unreferenced citations
if not referenced_indices:
    referenced_indices = set()
    log.info(
        "No [docX] references found in content; suppressing emission of unreferenced citations"
    )

This appears to work, I will now check why the regex is not matching and let you know. Maybe this is something special to my environment, not sure yet.