Transcript logging and call summaries

[BenGWeeks]

Summary

Save meeting transcripts to Azure Blob Storage so they:

1. Persist beyond meeting end (solves Transcript pane is cleared when switching panes #41 - transcript lost when switching panes)
2. Enable post-meeting summary generation

Design Decisions

• Storage: Azure Blob Storage (JSON files)
• Retention: 30 days (lifecycle policy)
• Summary timing: Auto-generate at meeting end + on-demand via /summary command

Architecture

During Meeting:
  Audio → SpeechTranscriptionService → Async write to Blob

Meeting End:
  1. Save final transcript to Blob
  2. Generate summary via Pennie
  3. Post summary to Teams chat

On-Demand:
  /summary command → Fetch from Blob → Generate via Pennie → Post to Teams

Blob Structure

transcripts/
  {meetingId}/
    transcript.json     # Array of TranscriptionResult
    summary.txt         # Generated summary
    metadata.json       # Meeting start/end times, participants

Implementation Tasks

Infrastructure

• Add transcripts container to existing storage account (infra/modules/monitoring.bicep)
• Add 30-day lifecycle policy for automatic cleanup

Bot Services

• Create ITranscriptStorageService interface
• Create TranscriptStorageService (Azure.Storage.Blobs)
• Modify SpeechTranscriptionService to persist on each recognition
• Add GenerateMeetingSummaryAsync() to PennieAgentClient

Bot Commands

• Add auto-summary generation on meeting end (MediaBot.OnMembersRemovedAsync)
• Add /summary command handler

Configuration

• Add storage config to appsettings.json
• Register ITranscriptStorageService in DI

Files to Modify

File	Change
infra/modules/monitoring.bicep	Add transcripts container + lifecycle policy
bot/Services/TranscriptStorageService.cs	NEW - Blob storage service
bot/Services/ITranscriptStorageService.cs	NEW - Interface
bot/Services/SpeechTranscriptionService.cs	Add blob persistence
bot/Services/PennieAgentClient.cs	Add GenerateMeetingSummaryAsync()
bot/Bots/MediaBot.cs	Auto-summary + /summary command
bot/Program.cs	Register new service
bot/appsettings.json	Add storage config

Related Issues

• Closes Transcript pane is cleared when switching panes #41 (Transcript pane cleared when switching panes)

Future Enhancements (not in scope)

• Export transcript as PDF/Word
• Link work items to specific transcript lines
• Search across all meeting transcripts
• Email summary to participants

[BenGWeeks]

Full Implementation Plan

Key Design Decision: Single Source of Truth

Remove in-memory caching and session storage. Blob Storage is the only source of truth.

• Write: Speech recognition → Blob Storage (immediately)
• Read: Control Panel / API → Blob Storage (on demand)

This eliminates:

• ❌ In-memory _transcripts dictionary in SpeechTranscriptionService
• ❌ Session/cache synchronization bugs
• ❌ Transcript lost when switching panes (Transcript pane is cleared when switching panes #41)

Additional Requirements

1. "Summarize Meeting" button in Control Panel UI
2. Include meeting chat (Teams chat messages) in summary, not just transcript

---

Architecture

During Meeting:
  Audio → SpeechTranscriptionService → Write to Blob (immediate)
  Teams Chat → Graph API → Write to Blob (immediate)
  Control Panel → GET /api/meeting/{id}/transcripts → Read from Blob

Meeting End:
  1. Generate summary via Pennie (transcript + chat)
  2. Save summary to Blob
  3. Post summary to Teams chat

On-Demand (3 ways):
  - /summary command in Teams chat
  - "Summarize Meeting" button in Control Panel
  - POST /api/meeting/{meetingId}/summarize

---

Blob Structure

transcripts/
  {meetingId}/
    transcript.jsonl    # JSON Lines - append each recognition
    chat.jsonl          # JSON Lines - append each chat message
    summary.txt         # Generated summary
    metadata.json       # Meeting start/end times, participants

Using JSON Lines (.jsonl) for efficient append-only writes during meeting.

---

Implementation Details

1. Infrastructure: Add Transcripts Container

File: infra/modules/monitoring.bicep

// Blob container for meeting transcripts
resource transcriptsContainer 'Microsoft.Storage/storageAccounts/blobServices/containers@2023-01-01' = {
  name: '${storageAccount.name}/default/transcripts'
  properties: {
    publicAccess: 'None'
  }
}

// Lifecycle policy for 30-day retention
resource lifecyclePolicy 'Microsoft.Storage/storageAccounts/managementPolicies@2023-01-01' = {
  name: '${storageAccount.name}/default'
  properties: {
    policy: {
      rules: [{
        name: 'delete-old-transcripts'
        enabled: true
        type: 'Lifecycle'
        definition: {
          filters: {
            blobTypes: ['blockBlob']
            prefixMatch: ['transcripts/']
          }
          actions: {
            baseBlob: {
              delete: { daysAfterModificationGreaterThan: 30 }
            }
          }
        }
      }]
    }
  }
}

2. New Service: TranscriptStorageService

File: bot/Services/TranscriptStorageService.cs (NEW)

public interface ITranscriptStorageService
{
    // Write (append to blob)
    Task AppendTranscriptLineAsync(string meetingId, TranscriptionResult result);
    Task AppendChatMessageAsync(string meetingId, ChatMessage message);
    
    // Read (from blob)
    Task<List<TranscriptionResult>> GetTranscriptAsync(string meetingId);
    Task<List<ChatMessage>> GetChatMessagesAsync(string meetingId);
    
    // Summary
    Task SaveMeetingSummaryAsync(string meetingId, string summary);
    Task<string?> GetMeetingSummaryAsync(string meetingId);
}

Implementation details:

• Use Azure.Storage.Blobs NuGet package
• Authenticate via DefaultAzureCredential (managed identity)
• Use AppendBlobClient for efficient append-only writes
• Store as JSON Lines (one JSON object per line)

3. Modify SpeechTranscriptionService - Remove In-Memory Storage

File: bot/Services/SpeechTranscriptionService.cs

Remove:

• _transcripts dictionary
• GetTranscripts() method (replaced by blob storage)

Change Recognized event handler:

// OLD: _transcripts[meetingId].Add(result);
// NEW: Write directly to blob
await _storageService.AppendTranscriptLineAsync(meetingId, result);

4. Update Transcript API Endpoint

File: bot/Controllers/MeetingController.cs

[HttpGet("{meetingId}/transcripts")]
public async Task<IActionResult> GetTranscripts(string meetingId, [FromQuery] int? sinceIndex = null)
{
    var transcripts = await _storageService.GetTranscriptAsync(meetingId);
    
    if (sinceIndex.HasValue)
        transcripts = transcripts.Skip(sinceIndex.Value).ToList();
    
    return Ok(new { transcripts, totalCount = transcripts.Count });
}

5. Add Chat Message Capture

File: bot/Bots/MediaBot.cs

// On chat message received:
await _storageService.AppendChatMessageAsync(meetingId, new ChatMessage {
    Sender = message.From.DisplayName,
    Text = message.Body.Content,
    Timestamp = message.CreatedDateTime
});

6. Add Summary Generation to PennieAgentClient

File: bot/Services/PennieAgentClient.cs

public async Task<string> GenerateMeetingSummaryAsync(
    string meetingId,
    List<TranscriptionResult> transcript,
    List<ChatMessage> chatMessages,
    CancellationToken cancellationToken = default)
{
    var transcriptText = string.Join("\n",
        transcript.Select(t => $"[{t.Timestamp:HH:mm:ss}] {t.Speaker}: {t.Text}"));
    
    var chatText = string.Join("\n",
        chatMessages.Select(c => $"[{c.Timestamp:HH:mm:ss}] {c.Sender} (chat): {c.Text}"));

    var prompt = $"""
        Please provide a summary of this meeting including:
        1. Key discussion points
        2. Decisions made
        3. Action items identified
        4. Work items created (with IDs if any)
        5. Outstanding questions

        Transcript (spoken):
        {transcriptText}

        Chat Messages:
        {chatText}
        """;

    // Send to Pennie and get response
}

7. Auto-Summary on Meeting End

File: bot/Bots/MediaBot.cs

// In OnMembersRemovedAsync():
var transcript = await _storageService.GetTranscriptAsync(meetingId);
var chatMessages = await _storageService.GetChatMessagesAsync(meetingId);
var summary = await _agentClient.GenerateMeetingSummaryAsync(meetingId, transcript, chatMessages);
await _storageService.SaveMeetingSummaryAsync(meetingId, summary);
await turnContext.SendActivityAsync(MessageFactory.Text($"**Meeting Summary**\n\n{summary}"));

8. /summary Command Handler

File: bot/Bots/MediaBot.cs

if (text.StartsWith("/summary", StringComparison.OrdinalIgnoreCase))
{
    var transcript = await _storageService.GetTranscriptAsync(meetingId);
    var chatMessages = await _storageService.GetChatMessagesAsync(meetingId);
    var summary = await _agentClient.GenerateMeetingSummaryAsync(meetingId, transcript, chatMessages);
    await turnContext.SendActivityAsync(MessageFactory.Text($"**Meeting Summary**\n\n{summary}"));
}

9. API Endpoint for Control Panel Button

File: bot/Controllers/MeetingController.cs

[HttpPost("{meetingId}/summarize")]
public async Task<IActionResult> SummarizeMeeting(string meetingId)
{
    var transcript = await _storageService.GetTranscriptAsync(meetingId);
    var chatMessages = await _storageService.GetChatMessagesAsync(meetingId);
    
    if (transcript.Count == 0 && chatMessages.Count == 0)
        return NotFound("No transcript or chat found for this meeting.");
    
    var summary = await _agentClient.GenerateMeetingSummaryAsync(meetingId, transcript, chatMessages);
    await _storageService.SaveMeetingSummaryAsync(meetingId, summary);
    
    // Post to Teams chat
    await PostToTeamsChat(meetingId, $"**Meeting Summary**\n\n{summary}");
    
    return Ok(new { summary });
}

10. Control Panel UI - "Summarize Meeting" Button

File: Control Panel frontend

<button onclick="summarizeMeeting()">Summarize Meeting</button>

11. Configuration

File: bot/appsettings.json

{
  "TranscriptStorage": {
    "StorageAccountName": "",
    "ContainerName": "transcripts"
  }
}

12. DI Registration

File: bot/Program.cs

builder.Services.AddSingleton<ITranscriptStorageService, TranscriptStorageService>();

---

Files to Modify

File	Change
infra/modules/monitoring.bicep	Add transcripts container + 30-day lifecycle policy
bot/Services/TranscriptStorageService.cs	NEW - Blob storage service
bot/Services/ITranscriptStorageService.cs	NEW - Interface
bot/Services/SpeechTranscriptionService.cs	Remove in-memory storage, write to blob
bot/Services/PennieAgentClient.cs	Add GenerateMeetingSummaryAsync()
bot/Bots/MediaBot.cs	Auto-summary + /summary command + chat capture
bot/Controllers/MeetingController.cs	Update transcript endpoint + add summarize endpoint
bot/Program.cs	Register new service
bot/appsettings.json	Add storage config
Control Panel UI	Add "Summarize Meeting" button

---

Code to Remove

File	Remove
bot/Services/SpeechTranscriptionService.cs	_transcripts dictionary, GetTranscripts() method
Any session/cache code	Client-side transcript caching

---

Testing

1. Unit tests: Mock ITranscriptStorageService
2. Integration test: Create meeting, speak, send chat, switch panes, verify transcript persists
3. Manual test: Join real Teams meeting, use /summary command and button

Related Issues

• Closes Transcript pane is cleared when switching panes #41 (Transcript pane cleared when switching panes)