[QUESTION] No Access to InternalResponsesAssistantMessage?

[andyjmorgan]

How can we help?

The internal nature of this content is restrictive if you're trying to harvest annotations, citations. etc. The webcitations BinaryData is not deserialized so cannot be caught in that event. I can create a stub class for this, but it also means I need to concat previous results to get the content.

The ContentTextOuput, contains the citation data, but is internal and inaccessible.

I'm attempting to get access to it, from StreamingResponseCompletedUpdate if that helps.

[ShivangiReja]

Hi @andyjmorgan, Thanks for reaching out! The annotations and citations are actually accessible through the public API. Here's how you can get them from StreamingResponseCompletedUpdate:

foreach (StreamingResponseUpdate update in streamingResponseUpdate)
{
    if (update is StreamingResponseCompletedUpdate completedUpdate)
    {
        // Access the completed response
        ResponseResult response = completedUpdate.Response;
        
        foreach (ResponseItem item in response.OutputItems)
        {
            if (item is MessageResponseItem messageItem)
            {
                foreach (ResponseContentPart contentPart in messageItem.Content)
                {
                    if (contentPart.Kind == ResponseContentPartKind.OutputText)
                    {
                        // Get the text
                        string text = contentPart.Text;
                        
                        // Get annotations/citations - these are publicly accessible!
                        IReadOnlyList<ResponseMessageAnnotation> annotations = contentPart.OutputTextAnnotations;
                        
                        foreach (ResponseMessageAnnotation annotation in annotations)
                        {
                            // For web search citations:
                            if (annotation is UriCitationMessageAnnotation uriCitation)
                            {
                                Console.WriteLine($"URL: {uriCitation.Uri}");
                                Console.WriteLine($"Title: {uriCitation.Title}");
                                Console.WriteLine($"Range: [{uriCitation.StartIndex}-{uriCitation.EndIndex}]");
                            }
                            // For file search citations:
                            else if (annotation is FileCitationMessageAnnotation fileCitation)
                            {
                                Console.WriteLine($"File ID: {fileCitation.FileId}");
                                Console.WriteLine($"Filename: {fileCitation.Filename}");
                                Console.WriteLine($"Index: {fileCitation.Index}");
                            }
                            // For container file citations:
                            else if (annotation is ContainerFileCitationMessageAnnotation containerCitation)
                            {
                                Console.WriteLine($"Container ID: {containerCitation.ContainerId}");
                                Console.WriteLine($"File ID: {containerCitation.FileId}");
                            }
                        }
                    }
                }
            }
        }
    }
}

While InternalResponsesAssistantMessage is internal, the MessageResponseItem base class exposes the Content property publicly, and ResponseContentPart exposes OutputTextAnnotations publicly, so you have full access to all citation data without needing the internal types.

[andyjmorgan]

Amazing thank you! Any thoughts on the BinaryData on the actual streamed citation result? It doesn't have a concrete class.

[ShivangiReja]

@andyjmorgan Just to make sure I understand correctly, are you referring to StreamingResponseTextAnnotationAddedUpdate.Annotation being exposed as BinaryData instead of a strongly-typed annotation?

If so, this is a known issue (#630) where the spec defines the annotation field as unknown, which causes the generator to emit BinaryData.

Until we get that corrected in the spec and regenerate the SDK, you can work around it by deserializing the BinaryData payload using ModelReaderWriter:

foreach (StreamingResponseUpdate update in streamingResponseUpdate)
{
    if (update is StreamingResponseTextAnnotationAddedUpdate annotationUpdate)
    {
        // Deserialize the BinaryData to the concrete annotation type
        ResponseMessageAnnotation annotation = ModelReaderWriter.Read<ResponseMessageAnnotation>(annotationUpdate.Annotation);
        
        // Now you can cast to specific types
        if (annotation is UriCitationMessageAnnotation uriCitation)
        {
            Console.WriteLine($"[Streaming] URL: {uriCitation.Uri}");
            Console.WriteLine($"[Streaming] Title: {uriCitation.Title}");
        }
        else if (annotation is FileCitationMessageAnnotation fileCitation)
        {
            Console.WriteLine($"[Streaming] File: {fileCitation.FileId}");
        }
        // ... etc
    }
}

[andyjmorgan]

Thank you so much for this, it's incredibly useful.