Conversation
…erarchy Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed
JinLee794
added a commit
that referenced
this pull request
Jan 15, 2026
* Delete samples/labs/dev/leadership_phrases.txt
* Update version and SKU name in staging params
* Change version for text-embedding-3-large model
Updated the version of the text-embedding-3-large model.
* Update main.tfvars.staging.json
* Update communication.tf
* feat: Enhance status envelope with optional label and update frontend to derive WS URL
- Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages.
- Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets.
- Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`.
- Enhanced `App.jsx` to format event type labels and summarize event data for better user experience.
- Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing.
- Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events.
- Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`.
- Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`.
* refactor: Comment out unused email communication service domain resource
* refactor: Comment out unused Azure email communication service resources
* feat: Enhance event handling and UI components
- Added new utility functions for formatting event types and summarizing event data in App.jsx.
- Improved ChatBubble component to display event messages with formatted labels and timestamps.
- Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags.
- Introduced websocket URL derivation in postprovision.sh for better backend integration.
- Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting.
- Updated package.json to include js-yaml and upgraded vite version.
* add value
* feat: Enhance distributed session handling and improve PayPal agent interactions
- Implement distributed session bus using Redis for cross-replica session routing in connection manager.
- Add methods for publishing session envelopes to Redis channels.
- Introduce confirmation context for call center transfers to ensure explicit user consent.
- Update PayPal agent templates to clarify authentication and routing guidelines.
- Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively.
- Improve error handling and logging for distributed session delivery and Redis interactions.
- Refactor session envelope handling in frontend to accommodate new event types and improve user experience.
* feat: Enhance status tone metadata and improve chat bubble styling
* feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes
* feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios
* feat: Implement retry mechanism for browser session ID resolution in media streaming
* feat: Enhance session management and greeting handling across various components
* fixing session mapping for acs calls
* add value
* add value
* adding test file
* Adding agents and templates for credit card recommendation and fee dispute agents
* add value
* Enhance audio transcription settings across agents and adjust logging levels for better debugging
* Enhance audio transcription settings across agents and adjust logging levels for better debugging
* add value
* add value
* Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments
* add value
* Add Azure Voice Live model configuration and outputs
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* remove sensitive contact information and unused transfer agency client data
* feat: Introduce Agent Consolidation Plan with YAML-driven architecture
- Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`.
- Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry.
- Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries.
- Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry.
- Detailed implementation roadmap divided into phases for gradual migration and integration.
- Included backward compatibility strategy to ensure existing agents function without modification.
- Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist.
* Refactor speech cascade handler and routing for browser communication
- Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events.
- Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input.
- Changed routing from `/realtime` to `/browser` for browser communication endpoints.
- Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap.
- Introduced WebSocket helper functions for better organization and clarity in messaging.
- Enhanced connection manager to handle Redis pubsub reconnections on credential expiration.
- Updated frontend components to reflect routing changes for browser communication.
- Adjusted tests to align with the new browser routing and functionality.
- Commented out live metrics enabling condition in telemetry configuration for future consideration.
* feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls
- Introduced , , , and decorators for OpenTelemetry instrumentation.
- Implemented context manager for tracking conversation turns with detailed metrics.
- Added helper functions for recording GenAI and speech metrics.
- Enhanced span attributes for Azure Application Insights visualization.
* Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality.
* feat: Enhance telemetry and tracing for CosmosDB and latency tool
- Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking.
- Integrated tracing spans in the LatencyTool for better observability in Application Insights.
- Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics.
- Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence.
- Removed outdated endpoints review document.
* feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints
test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications
* WARNING!!!! MAJOR REFACTOR COMMIT
- Removed the VoiceLive SDK integration module from the backend.
- Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections.
- Integrated the AgentTopologyPanel into the main application layout.
- Updated the BackendIndicator to include agent count and selection functionality.
- Enhanced the ConversationControls with a fixed view switcher for better accessibility.
- Improved the useBackendHealth hook to handle various agent data structures.
- Updated styles for better responsiveness and visual consistency across components.
- Modified utility functions to format agent inventory data correctly.
- Adjusted import paths in orchestrators and tests to reflect the new backend structure.
* feat: Enhance agent handoff process and response handling; refactor UI components for improved usability
* feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents
* refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization
* feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements
- Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically.
- Updated panel styling for improved aesthetics, including a gradient background and adjusted borders.
- Enhanced scrollbar visibility and overflow handling for better user experience.
refactor: Simplify GraphListView filter logic
- Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied.
- Cleaned up useEffect dependencies for better performance and clarity.
docs: Introduce Backend Voice & Agents Architecture documentation
- Added comprehensive documentation outlining the architecture of backend voice and agent modules.
- Detailed separation of concerns between voice transport and agent business logic.
- Included data flow diagrams and module responsibilities for clarity.
docs: Create Handoff Logic Inventory for better understanding of handoff processes
- Documented the handoff logic across backend voice and agent modules.
- Established a single source of truth for handoff mappings and protocols.
- Summarized cleanup phases and their impact on the codebase.
fix: Update logging to safely handle span names
- Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan.
fix: Adjust telemetry configuration to capture all loggers
- Changed logger_name default to an empty string in TelemetryConfig to capture all loggers.
* feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging
* feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration
* feat: Enhance Agent Details Panel and Session Management
- Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display.
- Implemented logic to show agent name, description, tools, and model/voice details based on session configuration.
- Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview.
- Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating.
- Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use.
- Enhanced on_demand_pool.py to validate cached resources and remove invalid ones.
- Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check.
* Refactor code structure for improved readability and maintainability
* feat: Enhance MemoManager with background persistence and lifecycle management
- Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis.
- Implemented task deduplication to cancel previous persistence tasks when a new one is initiated.
- Removed unused auto-refresh functionality and related attributes from MemoManager.
- Updated tests to verify new persistence behavior and ensure proper task management.
- Enhanced error handling and logging for background persistence operations.
* feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization
* feat(session): enhance session ID management and URL parameter support
- Added `pickSessionIdFromUrl` function to extract session ID from URL parameters.
- Updated `getOrCreateSessionId` to allow session ID restoration from URL.
- Refactored `setSessionId` for better logging and session management.
- Improved `createNewSessionId` to utilize `setSessionId`.
docs(api): restructure API documentation for clarity and completeness
- Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, Agent Builder, Demo Environment, and TTS Health.
- Added detailed descriptions and examples for each endpoint.
- Included new sections for interactive API documentation and WebSocket endpoints.
docs(api-reference): update WebSocket message types and endpoint details
- Clarified message types for incoming audio data and control messages.
- Updated WebSocket endpoint URLs and query parameters for browser conversations and dashboard relay.
docs(architecture): refine agent architecture diagrams for clarity
- Adjusted diagrams to improve readability and understanding of the agent framework and orchestration.
fix(architecture): correct orchestration mode comparison table
- Updated ratings for Azure Speech voices and simplicity of setup in the orchestration comparison table.
docs(getting-started): add demo guide and enhance onboarding experience
- Introduced a new demo guide to facilitate user onboarding and provide structured paths for different user levels.
- Enhanced the getting started guide with tips and recommended paths for new users.
feat(aoai): implement OpenAI connection warmup to reduce latency
- Added `warm_openai_connection` function to pre-establish OpenAI connection and reduce cold-start latency on first call.
feat(speech): implement token warmup for Speech API to minimize latency
- Added `warm_token` method in `SpeechTokenManager` to pre-fetch tokens during startup, reducing latency on first API call.
* feat(healthcare): Implement Nurse Triage Agent with symptom assessment and routing capabilities
- Introduced a comprehensive voice agent for healthcare triage.
- Added agent configuration and prompt templates for patient interaction.
- Developed healthcare tools for patient verification, clinical knowledge search, and symptom urgency assessment.
- Integrated routing logic for scheduling appointments and emergency transfers.
- Enhanced documentation with demo scenarios and testing instructions.
* feat: Implement logging utility and session management
- Added a logger utility to manage console logging levels and filtering.
- Created session management functions to handle session IDs, including retrieval from URL and session creation.
- Developed styles for the frontend components to ensure consistent UI design.
- Configured Vite for the frontend build process with proper asset handling and environment variable support.
- Introduced scripts for starting the backend and frontend development servers, including Azure Dev Tunnel hosting.
* feat: Simplify agent handoff process by refining context management and removing redundant data collection
* feat: Enhance agent handoff process by managing conversation history and user context
* feat: Enhance message handling by persisting tool calls and results as JSON for conversation continuity
* feat: Implement silent handoff protocol across agents to enhance user experience and streamline transitions
* feat: Add Azure App Configuration module with RBAC and Key Vault integration
- Implemented main resource for Azure App Configuration in Terraform.
- Added outputs for App Configuration details including ID, name, and endpoint.
- Defined variables for App Configuration module, including identity and Key Vault integration.
- Updated main Terraform outputs to include App Configuration details.
- Enhanced error handling in Azure OpenAI client for missing endpoint configuration.
- Improved Redis manager to handle port configuration with better error messaging.
- Updated requirements to include Azure App Configuration SDKs.
* first code clean up
* enabling oidc
* Refactor code structure and remove redundant sections for improved readability and maintainability
* add value
* add value
* feat: Add managed certificate and domain registration modules
- Introduced `managed-cert-example.bicep` for example usage of managed certificate deployment.
- Created `managed-cert.bicep` to handle App Service Domain registration and managed SSL certificate generation.
- Implemented `role-assignment.bicep` for managing role assignments with support for built-in and custom roles.
- Added `windows-vm.bicep` for deploying a Windows VM as a jumphost with necessary networking components.
- Developed `peer-virtual-networks.bicep` for establishing peering between virtual networks.
- Implemented `private-dns-zone.bicep` for creating and linking private DNS zones to virtual networks.
- Created `private-endpoint.bicep` for deploying private endpoints with DNS zone integration.
- Added `vnet.bicep` for creating virtual networks with associated subnets and network security groups.
- Updated `types.bicep` with new types for model deployment, role assignments, and network configurations.
- Developed `secret.bicep` for managing secrets in Azure Key Vault.
- Created `network.bicep` for orchestrating network resources including virtual networks and subnets.
* fix: Update default location parameter in create_storage function for clarity
* feat: Extract AZURE_LOCATION from environment-specific tfvars file if not set
* feat: Implement location resolution with fallback chain in preprovision script
* fix: Update Dockerfile to install runtime dependencies and mitigate vulnerabilities
* chore: Update CHANGELOG for version 1.5.0 release and remove changenotes.md; enable remote builds in azure.yaml; enhance terraform initialization script with location prompts
* feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity
* further deployment cleanup, docs update/tweaks, adding more todos
* removing unused dependency in src/herlpers.py
* refactor: Update architecture diagram in README for clarity and consistency in orchestration modes
* add value
* Refactor Terraform configuration:
- Update main.tf to adjust foundry account and project naming conventions.
- Remove feature flags and keys from appconfig module as they are now managed externally.
- Clean up variables.tf by removing unused variables and updating descriptions.
- Delete provider configuration file as it is no longer needed.
- Change default application name from "rtaudioagent" to "artagent" and adjust related settings.
- Modify connection settings and pool sizes for improved performance.
* feat: Enhance Azure Voice Live integration and refactor configuration management
* last changes
* feat: Add app configuration bootstrap to initialize environment variables
* Enhance configuration loading with .env.local support and update documentation
* fix voicelive output attributes
* add
* Refactor agent paths and update documentation for agent discovery and configuration
* Add Insurance Voice Agent Scenario documentation and update navigation
- Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management.
- Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section.
* Add integration proposal for Spec-Driven Development methodology in ARTVoice
* add value
* Enhance Terraform configuration and scripts for Voice Live integration
- Update Dockerfile to install dependencies and set up virtual environment.
- Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling.
- Refactor sync-appconfig.sh to streamline key-value imports and feature flag management.
- Add provider.conf.json generation for remote state backend configuration.
- Update main.tf and outputs.tf to support new Voice Live model deployments.
- Introduce voice_live_location and voice_live_model_deployments variables in variables.tf.
* feat: Add Concierge agent configuration and prompts for banking scenarios
- Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations.
- Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes.
- Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios.
- Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading.
- Updated default start agent to BankingConcierge and added validation for agent existence at startup.
* feat: Enhance scenario loading to support orchestration.yaml naming convention
* feat: Implement scenario-based handoff map resolution for orchestrator configuration
* cicd test for azd deploy
* feat: Update audio handling and documentation dependencies for improved installation and error handling
* feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management
* feat: Revise documentation deployment workflow to enhance dependency management and streamline build process
* modified docs workflow
* feat: Add site_dir configuration to mkdocs.yml for improved site structure
* feat: Allow mkdocs build to proceed with warnings by removing --strict flag
* fix: Update health check endpoint in postprovision script to use correct API path
* refactor: Remove outdated AZD deployment workflow and update documentation links for clarity
* fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly
* refactor: Simplify state key handling in provider configuration by using environment name
* fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults
* fix: Use coalesce function for location assignment in storage account resource
* refactor: Remove unused backend API public URL variable and related validation
* refactor: Remove unused backend API public URL and source phone number from environment parameter files
* improvements flow
* fix: Implement auto-selection and timeout for user input in setup scripts
* add value
* fix: Update naming conventions for foundry account and project variables in locals
* fix: Update name from rtaudioagent to artaudioagent in environment parameter files
* fix: Update name from rtaudioagent to artaudioagent in environment parameter files
* fix: Update documentation URLs to reflect new repository location
* feat: Enhance API documentation and tagging for better clarity and organization
* docs: Update documentation links and improve clarity across various guides
* refactor: replace deploy-azd workflow with reusable template and remove redundant summary job
- Updated the deployment workflow name to "Deploy to Azure".
- Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml.
- Removed the deployment summary job and its associated steps to streamline the workflow.
* fix: Add run-name to the Azure deployment workflow for better clarity
* fix: Update condition for output extraction in deployment workflow
* fix: Update GitHub token to use secrets for enhanced security
* feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment
* adding rg as env var set at the gh env level
* fix: Add emoji to workflow names for better visibility
* feat: Update documentation workflow name and enhance README with deployment badges
* fix: Update README layout and enhance navigation links for better user experience
* fix: Restore header for ARTVoice Accelerator Framework in README
* add value
* fix: Update README layout for improved clarity and navigation
* Enhance provisioning scripts and documentation
- Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal.
- Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning.
- Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms.
- Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration.
- Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts.
- Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions.
* feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements
* fix(ci): simplify test-azd-hooks workflow tests and run in parallel
- Remove fragile grep-based function extraction that caused syntax errors
- Run lint, linux, macos, windows tests in parallel (no dependencies)
- Trigger on all pushes to main/staging (remove path filters for push)
- Simplify backend configuration test to avoid function sourcing issues
* feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration
* feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow
* feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup
* feat: Update test job names for clarity and enhance preflight checks for CI mode
* feat: Update preflight checks to conditionally include Docker in CI mode and log its status
* feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting
* feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks
* feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock
* feat: Update Azure deployment workflows and normalize container memory formats
* feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options
* feat: Remove outdated troubleshooting documentation for deployment issues
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/test-azd-hooks.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: Implement TTS Streaming Latency Analysis and Optimization Plan
- Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture.
- Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming.
- Proposed a multi-phase optimization strategy to address identified issues, including:
- Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task.
- Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch.
- Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming.
- Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing.
- Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming.
- Created a detailed test implementation plan with metrics and success criteria to validate improvements.
test: Add unit tests for HandoffService
- Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods.
- Implemented tests for greeting selection and context building to ensure proper functionality.
- Added tests for the HandoffResolution dataclass to verify properties and default values.
* feat: Add Scenario Builder component and integrate with RealTimeVoiceApp
- Introduced ScenarioBuilder component for visual orchestration of agent flows.
- Implemented drag-and-drop functionality for agents and handoff configuration.
- Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders.
- Enhanced state management for agent scenarios, including creation and updates.
- Integrated new handoff editor for configuring agent interactions.
* Refactor code structure for improved readability and maintainability
* Add error handling for Redis connection issues and implement unit tests for HandoffService
- Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts.
- Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building.
- Added tests for HandoffResolution dataclass to ensure correct property behavior and default values.
* Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts
* Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates.
* Refactor environment variable assignment in deploy workflow for clarity
* Refactor tests and dependencies following module renaming and API changes
- Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock.
- Updated conftest.py to mock configuration and Azure OpenAI client for tests.
- Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules.
- Adjusted imports in test_artagent_wshelpers.py for orchestrator path change.
- Skipped tests in test_call_transfer_service.py due to API changes in toolstore.
- Updated datetime usage in test_demo_env_phrase_bias.py to use UTC.
- Modified websocket endpoint assertions in test_realtime.py to reflect new paths.
- Added new test file test_voice_handler_components.py for voice handler components.
* Add comprehensive tests for VoiceLive handler and orchestrator memory management
- Implement tests to verify cleanup functionality in LiveOrchestrator.
- Ensure proper registration and unregistration of orchestrators in the registry.
- Test background task tracking and cleanup mechanisms.
- Validate greeting task cancellation during orchestrator cleanup.
- Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry.
- Verify user message history deque is properly bounded and cleared on cleanup.
- Add scenario update tests to ensure correct agent management during updates.
- Optimize hot path functions to ensure non-blocking behavior during network calls.
* feat: Enhance AgentBuilder with consistent field names and improved UI elements
* Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues.
* feat: Add predefined handoff condition patterns to enhance scenario orchestration
* add value
* feat(metrics): Introduce shared metrics factory for lazy initialization
- Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics.
- Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics.
- Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization.
- Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling.
- Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls.
- Introduced utility functions for building common metric attributes, ensuring consistency across metrics.
* feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation
* fix: Simplify environment determination logic in deployment workflow
* add value
* feat: Add user flow screenshots and enhance documentation for guided agent setup
* feat: Enhance scenario testing instructions for clarity and user guidance
* fix: Correct image paths in quickstart guide for accurate rendering
* feat: Add initial agent builder and template selection screenshots to quickstart guide
* feat: Add demo profile creation steps and related images to quickstart guide
* feat: Implement EasyAuth configuration script and integrate into post-provisioning process
* refactor: Remove backend IP restrictions configuration and related outputs
* Added non qualifying rush response to ensure clear model behavior
* updated order so confirmation statement is in the correct spot
* add value
* add value
* chore: Remove unused workflow images for demo profiles
* fix: Update demo profile creation images in quickstart guide
* fix: Update home screen image in quickstart guide
* fix: Update home screen and scenario images in quickstart guide
* add value
* add value
* add value
* add value
* add value
* add
* add value
* art
* add opentelemetry import for tracing support in TTS module
* refactor: update LiveOrchestrator to enhance user message history management and improve handoff context
* Refactor TTS Playback and Voice Handling
- Consolidated TTS playback logic into a unified class for speech cascade.
- Removed deprecated VoiceSessionContext and related compatibility shims.
- Enhanced error handling during tool initialization and event handler registration.
- Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings.
- Improved logging for TTS synthesis and streaming processes.
- Added new handoff tool registration for dynamic routing.
* refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts
* refactor: enhance EasyAuth interactive prompts and streamline user choices
* refactor: enhance run-name logic for Azure deployment workflow
* fix: update environment logic for pull_request events in Azure deployment workflow
* refactor: update preprovision hook execution and streamline backend configuration
* feat: add context variable support for handoffs and enhance UI for variable mapping
* feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Responses API Infrastructure & Dual Model Configuration (#16)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* fix: update project version to 2.0.0-beta in pyproject.toml
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Orchestrator Integration + Optimizations (#17)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: integrate Responses API in orchestrators and add optimizations
**Cascade Orchestrator:**
- Update model selection to use agent.get_model_for_mode('cascade')
- Integrate Responses API routing based on endpoint_preference
- Add error handling for unsupported parameters
- Extract TTS processing into separate tts_processor module
**VoiceLive Orchestrator:**
- Update to use agent.get_model_for_mode('voicelive')
- Add registry cleanup to prevent unbounded growth
- Improve memory management and stale orchestrator cleanup
- Extract DTMF processing into separate dtmf_processor module
**Tests:**
- Add test_cascade_orchestrator_entry_points
- Add test_cascade_llm_processing
- Add test_dtmf_processor
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Evaluation Framework + Frontend UI (#18)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: add evaluation framework and frontend UI for Responses API
**Evaluation Framework:**
- Add EventRecorder with git commit SHA tracking
- Add API-aware scoring with budget adjustments for verbosity
- Add scenario runner for automated testing
- Add CLI for running evaluations
- Add validate_phases.py for phase-based validation
- Add wrappers for endpoint detection
**Frontend UI:**
- Add cascade_model and voicelive_model selectors in Agent Builder
- Add Responses API endpoint preference dropdown
- Add conditional fields for verbosity, reasoning_effort, etc.
- Update ScenarioBuilder with model configuration options
- Display API version fields
**Documentation:**
- Add docs/testing/model-evaluation.md
- Add evaluation playground Jupyter notebook
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: voice handler refactoring and MediaHandler migration
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
* Enhance logging and user prompts in preflight and pre-provisioning scripts (#20)
- Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting.
- Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh.
- Added color-coded success, warning, and error messages for better visibility.
- Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values.
- Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: voice handler refactoring and MediaHandler migration (#21)
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* enhanced the scenariobuilder with flowy (#22)
* docs: add comprehensive voice processing architecture documentation
Add complete documentation for the voice processing architecture:
New Documentation:
- docs/architecture/voice/README.md - Comprehensive voice architecture guide
* VoiceHandler overview and usage patterns
* TTS playback and text processing
* Speech cascade pipeline documentation
* Audio specifications for browser and ACS transports
* Testing guidelines with actual test file references
* Troubleshooting guide for common issues
- apps/artagent/backend/voice/README.md - Developer quick reference
* Directory structure and module organization
* Quick start examples
* Common tasks and patterns
* File location guide
* Testing commands
Documentation Updates:
- docs/mkdocs.yml - Add voice architecture to navigation
- docs/operations/troubleshooting.md - Add voice-specific troubleshooting
Key Improvements:
- Fixed mkdocs formatting for proper list rendering
- Updated all test references to match actual test files:
* test_voice_handler_components.py
* test_voice_handler_compat.py
* test_cascade_orchestrator_entry_points.py
* test_cascade_llm_processing.py
- Verified all script references (quick_test.sh, test_orchestrator.py)
- Added prerequisites for running tests with dev dependencies
- Included both basic and advanced testing examples
All file paths and examples have been verified against the actual codebase.
Related to #[TBD]
* Add custom styles for Flowy flowchart integration with agent blocks
* feat: Enhance output port visibility logic in ScenarioGraphCanvas
* feat: Add expandable full prompt view for source agent in HandoffEditorDialog
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Refactor ACS logging and add default orchestration scenario
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
* Refactor ACS logging and add default orchestration scenario (#23)
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Enhance logging functions to use log_plain for consistency and clarity in local development setup script
* Disable view toggle buttons for chat/graph/timeline in ConversationControls
* Add panning functionality to ScenarioGraphCanvas and reset button
* Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes
* feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts
- Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation.
- Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements.
- Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints.
- Improved metrics naming conventions for clarity in load testing results.
* feat: Update Voice Live readiness status to use event envelope format
---------
Co-authored-by: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Co-authored-by: Anna Quincy <annaquincy@microsoft.com>
Co-authored-by: Jin Lee <jinle@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
JinLee794
added a commit
that referenced
this pull request
Jan 15, 2026
* Update version and SKU name in staging params
* Change version for text-embedding-3-large model
Updated the version of the text-embedding-3-large model.
* Update main.tfvars.staging.json
* Update communication.tf
* feat: Enhance status envelope with optional label and update frontend to derive WS URL
- Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages.
- Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets.
- Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`.
- Enhanced `App.jsx` to format event type labels and summarize event data for better user experience.
- Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing.
- Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events.
- Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`.
- Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`.
* refactor: Comment out unused email communication service domain resource
* refactor: Comment out unused Azure email communication service resources
* feat: Enhance event handling and UI components
- Added new utility functions for formatting event types and summarizing event data in App.jsx.
- Improved ChatBubble component to display event messages with formatted labels and timestamps.
- Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags.
- Introduced websocket URL derivation in postprovision.sh for better backend integration.
- Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting.
- Updated package.json to include js-yaml and upgraded vite version.
* add value
* feat: Enhance distributed session handling and improve PayPal agent interactions
- Implement distributed session bus using Redis for cross-replica session routing in connection manager.
- Add methods for publishing session envelopes to Redis channels.
- Introduce confirmation context for call center transfers to ensure explicit user consent.
- Update PayPal agent templates to clarify authentication and routing guidelines.
- Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively.
- Improve error handling and logging for distributed session delivery and Redis interactions.
- Refactor session envelope handling in frontend to accommodate new event types and improve user experience.
* feat: Enhance status tone metadata and improve chat bubble styling
* feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes
* feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios
* feat: Implement retry mechanism for browser session ID resolution in media streaming
* feat: Enhance session management and greeting handling across various components
* fixing session mapping for acs calls
* add value
* add value
* adding test file
* Adding agents and templates for credit card recommendation and fee dispute agents
* add value
* Enhance audio transcription settings across agents and adjust logging levels for better debugging
* Enhance audio transcription settings across agents and adjust logging levels for better debugging
* add value
* add value
* Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments
* add value
* Add Azure Voice Live model configuration and outputs
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* remove sensitive contact information and unused transfer agency client data
* feat: Introduce Agent Consolidation Plan with YAML-driven architecture
- Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`.
- Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry.
- Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries.
- Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry.
- Detailed implementation roadmap divided into phases for gradual migration and integration.
- Included backward compatibility strategy to ensure existing agents function without modification.
- Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist.
* Refactor speech cascade handler and routing for browser communication
- Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events.
- Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input.
- Changed routing from `/realtime` to `/browser` for browser communication endpoints.
- Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap.
- Introduced WebSocket helper functions for better organization and clarity in messaging.
- Enhanced connection manager to handle Redis pubsub reconnections on credential expiration.
- Updated frontend components to reflect routing changes for browser communication.
- Adjusted tests to align with the new browser routing and functionality.
- Commented out live metrics enabling condition in telemetry configuration for future consideration.
* feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls
- Introduced , , , and decorators for OpenTelemetry instrumentation.
- Implemented context manager for tracking conversation turns with detailed metrics.
- Added helper functions for recording GenAI and speech metrics.
- Enhanced span attributes for Azure Application Insights visualization.
* Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality.
* feat: Enhance telemetry and tracing for CosmosDB and latency tool
- Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking.
- Integrated tracing spans in the LatencyTool for better observability in Application Insights.
- Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics.
- Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence.
- Removed outdated endpoints review document.
* feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints
test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications
* WARNING!!!! MAJOR REFACTOR COMMIT
- Removed the VoiceLive SDK integration module from the backend.
- Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections.
- Integrated the AgentTopologyPanel into the main application layout.
- Updated the BackendIndicator to include agent count and selection functionality.
- Enhanced the ConversationControls with a fixed view switcher for better accessibility.
- Improved the useBackendHealth hook to handle various agent data structures.
- Updated styles for better responsiveness and visual consistency across components.
- Modified utility functions to format agent inventory data correctly.
- Adjusted import paths in orchestrators and tests to reflect the new backend structure.
* feat: Enhance agent handoff process and response handling; refactor UI components for improved usability
* feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents
* refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization
* feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements
- Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically.
- Updated panel styling for improved aesthetics, including a gradient background and adjusted borders.
- Enhanced scrollbar visibility and overflow handling for better user experience.
refactor: Simplify GraphListView filter logic
- Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied.
- Cleaned up useEffect dependencies for better performance and clarity.
docs: Introduce Backend Voice & Agents Architecture documentation
- Added comprehensive documentation outlining the architecture of backend voice and agent modules.
- Detailed separation of concerns between voice transport and agent business logic.
- Included data flow diagrams and module responsibilities for clarity.
docs: Create Handoff Logic Inventory for better understanding of handoff processes
- Documented the handoff logic across backend voice and agent modules.
- Established a single source of truth for handoff mappings and protocols.
- Summarized cleanup phases and their impact on the codebase.
fix: Update logging to safely handle span names
- Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan.
fix: Adjust telemetry configuration to capture all loggers
- Changed logger_name default to an empty string in TelemetryConfig to capture all loggers.
* feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging
* feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration
* feat: Enhance Agent Details Panel and Session Management
- Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display.
- Implemented logic to show agent name, description, tools, and model/voice details based on session configuration.
- Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview.
- Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating.
- Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use.
- Enhanced on_demand_pool.py to validate cached resources and remove invalid ones.
- Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check.
* Refactor code structure for improved readability and maintainability
* feat: Enhance MemoManager with background persistence and lifecycle management
- Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis.
- Implemented task deduplication to cancel previous persistence tasks when a new one is initiated.
- Removed unused auto-refresh functionality and related attributes from MemoManager.
- Updated tests to verify new persistence behavior and ensure proper task management.
- Enhanced error handling and logging for background persistence operations.
* feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization
* feat(session): enhance session ID management and URL parameter support
- Added `pickSessionIdFromUrl` function to extract session ID from URL parameters.
- Updated `getOrCreateSessionId` to allow session ID restoration from URL.
- Refactored `setSessionId` for better logging and session management.
- Improved `createNewSessionId` to utilize `setSessionId`.
docs(api): restructure API documentation for clarity and completeness
- Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, Agent Builder, Demo Environment, and TTS Health.
- Added detailed descriptions and examples for each endpoint.
- Included new sections for interactive API documentation and WebSocket endpoints.
docs(api-reference): update WebSocket message types and endpoint details
- Clarified message types for incoming audio data and control messages.
- Updated WebSocket endpoint URLs and query parameters for browser conversations and dashboard relay.
docs(architecture): refine agent architecture diagrams for clarity
- Adjusted diagrams to improve readability and understanding of the agent framework and orchestration.
fix(architecture): correct orchestration mode comparison table
- Updated ratings for Azure Speech voices and simplicity of setup in the orchestration comparison table.
docs(getting-started): add demo guide and enhance onboarding experience
- Introduced a new demo guide to facilitate user onboarding and provide structured paths for different user levels.
- Enhanced the getting started guide with tips and recommended paths for new users.
feat(aoai): implement OpenAI connection warmup to reduce latency
- Added `warm_openai_connection` function to pre-establish OpenAI connection and reduce cold-start latency on first call.
feat(speech): implement token warmup for Speech API to minimize latency
- Added `warm_token` method in `SpeechTokenManager` to pre-fetch tokens during startup, reducing latency on first API call.
* feat(healthcare): Implement Nurse Triage Agent with symptom assessment and routing capabilities
- Introduced a comprehensive voice agent for healthcare triage.
- Added agent configuration and prompt templates for patient interaction.
- Developed healthcare tools for patient verification, clinical knowledge search, and symptom urgency assessment.
- Integrated routing logic for scheduling appointments and emergency transfers.
- Enhanced documentation with demo scenarios and testing instructions.
* feat: Implement logging utility and session management
- Added a logger utility to manage console logging levels and filtering.
- Created session management functions to handle session IDs, including retrieval from URL and session creation.
- Developed styles for the frontend components to ensure consistent UI design.
- Configured Vite for the frontend build process with proper asset handling and environment variable support.
- Introduced scripts for starting the backend and frontend development servers, including Azure Dev Tunnel hosting.
* feat: Simplify agent handoff process by refining context management and removing redundant data collection
* feat: Enhance agent handoff process by managing conversation history and user context
* feat: Enhance message handling by persisting tool calls and results as JSON for conversation continuity
* feat: Implement silent handoff protocol across agents to enhance user experience and streamline transitions
* feat: Add Azure App Configuration module with RBAC and Key Vault integration
- Implemented main resource for Azure App Configuration in Terraform.
- Added outputs for App Configuration details including ID, name, and endpoint.
- Defined variables for App Configuration module, including identity and Key Vault integration.
- Updated main Terraform outputs to include App Configuration details.
- Enhanced error handling in Azure OpenAI client for missing endpoint configuration.
- Improved Redis manager to handle port configuration with better error messaging.
- Updated requirements to include Azure App Configuration SDKs.
* first code clean up
* enabling oidc
* Refactor code structure and remove redundant sections for improved readability and maintainability
* add value
* add value
* feat: Add managed certificate and domain registration modules
- Introduced `managed-cert-example.bicep` for example usage of managed certificate deployment.
- Created `managed-cert.bicep` to handle App Service Domain registration and managed SSL certificate generation.
- Implemented `role-assignment.bicep` for managing role assignments with support for built-in and custom roles.
- Added `windows-vm.bicep` for deploying a Windows VM as a jumphost with necessary networking components.
- Developed `peer-virtual-networks.bicep` for establishing peering between virtual networks.
- Implemented `private-dns-zone.bicep` for creating and linking private DNS zones to virtual networks.
- Created `private-endpoint.bicep` for deploying private endpoints with DNS zone integration.
- Added `vnet.bicep` for creating virtual networks with associated subnets and network security groups.
- Updated `types.bicep` with new types for model deployment, role assignments, and network configurations.
- Developed `secret.bicep` for managing secrets in Azure Key Vault.
- Created `network.bicep` for orchestrating network resources including virtual networks and subnets.
* fix: Update default location parameter in create_storage function for clarity
* feat: Extract AZURE_LOCATION from environment-specific tfvars file if not set
* feat: Implement location resolution with fallback chain in preprovision script
* fix: Update Dockerfile to install runtime dependencies and mitigate vulnerabilities
* chore: Update CHANGELOG for version 1.5.0 release and remove changenotes.md; enable remote builds in azure.yaml; enhance terraform initialization script with location prompts
* feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity
* further deployment cleanup, docs update/tweaks, adding more todos
* removing unused dependency in src/herlpers.py
* refactor: Update architecture diagram in README for clarity and consistency in orchestration modes
* add value
* Refactor Terraform configuration:
- Update main.tf to adjust foundry account and project naming conventions.
- Remove feature flags and keys from appconfig module as they are now managed externally.
- Clean up variables.tf by removing unused variables and updating descriptions.
- Delete provider configuration file as it is no longer needed.
- Change default application name from "rtaudioagent" to "artagent" and adjust related settings.
- Modify connection settings and pool sizes for improved performance.
* feat: Enhance Azure Voice Live integration and refactor configuration management
* last changes
* feat: Add app configuration bootstrap to initialize environment variables
* Enhance configuration loading with .env.local support and update documentation
* fix voicelive output attributes
* add
* Refactor agent paths and update documentation for agent discovery and configuration
* Add Insurance Voice Agent Scenario documentation and update navigation
- Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management.
- Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section.
* Add integration proposal for Spec-Driven Development methodology in ARTVoice
* add value
* Enhance Terraform configuration and scripts for Voice Live integration
- Update Dockerfile to install dependencies and set up virtual environment.
- Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling.
- Refactor sync-appconfig.sh to streamline key-value imports and feature flag management.
- Add provider.conf.json generation for remote state backend configuration.
- Update main.tf and outputs.tf to support new Voice Live model deployments.
- Introduce voice_live_location and voice_live_model_deployments variables in variables.tf.
* feat: Add Concierge agent configuration and prompts for banking scenarios
- Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations.
- Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes.
- Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios.
- Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading.
- Updated default start agent to BankingConcierge and added validation for agent existence at startup.
* feat: Enhance scenario loading to support orchestration.yaml naming convention
* feat: Implement scenario-based handoff map resolution for orchestrator configuration
* cicd test for azd deploy
* feat: Update audio handling and documentation dependencies for improved installation and error handling
* feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management
* feat: Revise documentation deployment workflow to enhance dependency management and streamline build process
* modified docs workflow
* feat: Add site_dir configuration to mkdocs.yml for improved site structure
* feat: Allow mkdocs build to proceed with warnings by removing --strict flag
* fix: Update health check endpoint in postprovision script to use correct API path
* refactor: Remove outdated AZD deployment workflow and update documentation links for clarity
* fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly
* refactor: Simplify state key handling in provider configuration by using environment name
* fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults
* fix: Use coalesce function for location assignment in storage account resource
* refactor: Remove unused backend API public URL variable and related validation
* refactor: Remove unused backend API public URL and source phone number from environment parameter files
* improvements flow
* fix: Implement auto-selection and timeout for user input in setup scripts
* add value
* fix: Update naming conventions for foundry account and project variables in locals
* fix: Update name from rtaudioagent to artaudioagent in environment parameter files
* fix: Update name from rtaudioagent to artaudioagent in environment parameter files
* fix: Update documentation URLs to reflect new repository location
* feat: Enhance API documentation and tagging for better clarity and organization
* docs: Update documentation links and improve clarity across various guides
* refactor: replace deploy-azd workflow with reusable template and remove redundant summary job
- Updated the deployment workflow name to "Deploy to Azure".
- Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml.
- Removed the deployment summary job and its associated steps to streamline the workflow.
* fix: Add run-name to the Azure deployment workflow for better clarity
* fix: Update condition for output extraction in deployment workflow
* fix: Update GitHub token to use secrets for enhanced security
* feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment
* adding rg as env var set at the gh env level
* fix: Add emoji to workflow names for better visibility
* feat: Update documentation workflow name and enhance README with deployment badges
* fix: Update README layout and enhance navigation links for better user experience
* fix: Restore header for ARTVoice Accelerator Framework in README
* add value
* fix: Update README layout for improved clarity and navigation
* Enhance provisioning scripts and documentation
- Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal.
- Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning.
- Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms.
- Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration.
- Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts.
- Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions.
* feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements
* fix(ci): simplify test-azd-hooks workflow tests and run in parallel
- Remove fragile grep-based function extraction that caused syntax errors
- Run lint, linux, macos, windows tests in parallel (no dependencies)
- Trigger on all pushes to main/staging (remove path filters for push)
- Simplify backend configuration test to avoid function sourcing issues
* feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration
* feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow
* feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup
* feat: Update test job names for clarity and enhance preflight checks for CI mode
* feat: Update preflight checks to conditionally include Docker in CI mode and log its status
* feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting
* feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks
* feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock
* feat: Update Azure deployment workflows and normalize container memory formats
* feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options
* feat: Remove outdated troubleshooting documentation for deployment issues
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/test-azd-hooks.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: Implement TTS Streaming Latency Analysis and Optimization Plan
- Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture.
- Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming.
- Proposed a multi-phase optimization strategy to address identified issues, including:
- Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task.
- Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch.
- Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming.
- Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing.
- Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming.
- Created a detailed test implementation plan with metrics and success criteria to validate improvements.
test: Add unit tests for HandoffService
- Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods.
- Implemented tests for greeting selection and context building to ensure proper functionality.
- Added tests for the HandoffResolution dataclass to verify properties and default values.
* feat: Add Scenario Builder component and integrate with RealTimeVoiceApp
- Introduced ScenarioBuilder component for visual orchestration of agent flows.
- Implemented drag-and-drop functionality for agents and handoff configuration.
- Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders.
- Enhanced state management for agent scenarios, including creation and updates.
- Integrated new handoff editor for configuring agent interactions.
* Refactor code structure for improved readability and maintainability
* Add error handling for Redis connection issues and implement unit tests for HandoffService
- Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts.
- Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building.
- Added tests for HandoffResolution dataclass to ensure correct property behavior and default values.
* Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts
* Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates.
* Refactor environment variable assignment in deploy workflow for clarity
* Refactor tests and dependencies following module renaming and API changes
- Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock.
- Updated conftest.py to mock configuration and Azure OpenAI client for tests.
- Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules.
- Adjusted imports in test_artagent_wshelpers.py for orchestrator path change.
- Skipped tests in test_call_transfer_service.py due to API changes in toolstore.
- Updated datetime usage in test_demo_env_phrase_bias.py to use UTC.
- Modified websocket endpoint assertions in test_realtime.py to reflect new paths.
- Added new test file test_voice_handler_components.py for voice handler components.
* Add comprehensive tests for VoiceLive handler and orchestrator memory management
- Implement tests to verify cleanup functionality in LiveOrchestrator.
- Ensure proper registration and unregistration of orchestrators in the registry.
- Test background task tracking and cleanup mechanisms.
- Validate greeting task cancellation during orchestrator cleanup.
- Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry.
- Verify user message history deque is properly bounded and cleared on cleanup.
- Add scenario update tests to ensure correct agent management during updates.
- Optimize hot path functions to ensure non-blocking behavior during network calls.
* feat: Enhance AgentBuilder with consistent field names and improved UI elements
* Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues.
* feat: Add predefined handoff condition patterns to enhance scenario orchestration
* add value
* feat(metrics): Introduce shared metrics factory for lazy initialization
- Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics.
- Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics.
- Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization.
- Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling.
- Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls.
- Introduced utility functions for building common metric attributes, ensuring consistency across metrics.
* feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation
* fix: Simplify environment determination logic in deployment workflow
* add value
* feat: Add user flow screenshots and enhance documentation for guided agent setup
* feat: Enhance scenario testing instructions for clarity and user guidance
* fix: Correct image paths in quickstart guide for accurate rendering
* feat: Add initial agent builder and template selection screenshots to quickstart guide
* feat: Add demo profile creation steps and related images to quickstart guide
* feat: Implement EasyAuth configuration script and integrate into post-provisioning process
* refactor: Remove backend IP restrictions configuration and related outputs
* Added non qualifying rush response to ensure clear model behavior
* updated order so confirmation statement is in the correct spot
* add value
* add value
* chore: Remove unused workflow images for demo profiles
* fix: Update demo profile creation images in quickstart guide
* fix: Update home screen image in quickstart guide
* fix: Update home screen and scenario images in quickstart guide
* add value
* add value
* add value
* add value
* add value
* add
* add value
* art
* add opentelemetry import for tracing support in TTS module
* refactor: update LiveOrchestrator to enhance user message history management and improve handoff context
* Refactor TTS Playback and Voice Handling
- Consolidated TTS playback logic into a unified class for speech cascade.
- Removed deprecated VoiceSessionContext and related compatibility shims.
- Enhanced error handling during tool initialization and event handler registration.
- Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings.
- Improved logging for TTS synthesis and streaming processes.
- Added new handoff tool registration for dynamic routing.
* refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts
* refactor: enhance EasyAuth interactive prompts and streamline user choices
* refactor: enhance run-name logic for Azure deployment workflow
* fix: update environment logic for pull_request events in Azure deployment workflow
* refactor: update preprovision hook execution and streamline backend configuration
* feat: add context variable support for handoffs and enhance UI for variable mapping
* feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Responses API Infrastructure & Dual Model Configuration (#16)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* fix: update project version to 2.0.0-beta in pyproject.toml
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Orchestrator Integration + Optimizations (#17)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: integrate Responses API in orchestrators and add optimizations
**Cascade Orchestrator:**
- Update model selection to use agent.get_model_for_mode('cascade')
- Integrate Responses API routing based on endpoint_preference
- Add error handling for unsupported parameters
- Extract TTS processing into separate tts_processor module
**VoiceLive Orchestrator:**
- Update to use agent.get_model_for_mode('voicelive')
- Add registry cleanup to prevent unbounded growth
- Improve memory management and stale orchestrator cleanup
- Extract DTMF processing into separate dtmf_processor module
**Tests:**
- Add test_cascade_orchestrator_entry_points
- Add test_cascade_llm_processing
- Add test_dtmf_processor
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Evaluation Framework + Frontend UI (#18)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: add evaluation framework and frontend UI for Responses API
**Evaluation Framework:**
- Add EventRecorder with git commit SHA tracking
- Add API-aware scoring with budget adjustments for verbosity
- Add scenario runner for automated testing
- Add CLI for running evaluations
- Add validate_phases.py for phase-based validation
- Add wrappers for endpoint detection
**Frontend UI:**
- Add cascade_model and voicelive_model selectors in Agent Builder
- Add Responses API endpoint preference dropdown
- Add conditional fields for verbosity, reasoning_effort, etc.
- Update ScenarioBuilder with model configuration options
- Display API version fields
**Documentation:**
- Add docs/testing/model-evaluation.md
- Add evaluation playground Jupyter notebook
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: voice handler refactoring and MediaHandler migration
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
* Enhance logging and user prompts in preflight and pre-provisioning scripts (#20)
- Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting.
- Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh.
- Added color-coded success, warning, and error messages for better visibility.
- Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values.
- Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: voice handler refactoring and MediaHandler migration (#21)
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* enhanced the scenariobuilder with flowy (#22)
* docs: add comprehensive voice processing architecture documentation
Add complete documentation for the voice processing architecture:
New Documentation:
- docs/architecture/voice/README.md - Comprehensive voice architecture guide
* VoiceHandler overview and usage patterns
* TTS playback and text processing
* Speech cascade pipeline documentation
* Audio specifications for browser and ACS transports
* Testing guidelines with actual test file references
* Troubleshooting guide for common issues
- apps/artagent/backend/voice/README.md - Developer quick reference
* Directory structure and module organization
* Quick start examples
* Common tasks and patterns
* File location guide
* Testing commands
Documentation Updates:
- docs/mkdocs.yml - Add voice architecture to navigation
- docs/operations/troubleshooting.md - Add voice-specific troubleshooting
Key Improvements:
- Fixed mkdocs formatting for proper list rendering
- Updated all test references to match actual test files:
* test_voice_handler_components.py
* test_voice_handler_compat.py
* test_cascade_orchestrator_entry_points.py
* test_cascade_llm_processing.py
- Verified all script references (quick_test.sh, test_orchestrator.py)
- Added prerequisites for running tests with dev dependencies
- Included both basic and advanced testing examples
All file paths and examples have been verified against the actual codebase.
Related to #[TBD]
* Add custom styles for Flowy flowchart integration with agent blocks
* feat: Enhance output port visibility logic in ScenarioGraphCanvas
* feat: Add expandable full prompt view for source agent in HandoffEditorDialog
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Refactor ACS logging and add default orchestration scenario
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
* Refactor ACS logging and add default orchestration scenario (#23)
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Enhance logging functions to use log_plain for consistency and clarity in local development setup script
* Disable view toggle buttons for chat/graph/timeline in ConversationControls
* Add panning functionality to ScenarioGraphCanvas and reset button
* Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes
* feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts
- Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation.
- Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements.
- Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints.
- Improved metrics naming conventions for clarity in load testing results.
* feat: Update Voice Live readiness status to use event envelope format
* Consolidate infrastructure resource documentation into infra/README.md (#26)
* Initial plan
* Add comprehensive infrastructure resources documentation with private networking links
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Consolidate infrastructure documentation into infra/README.md
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Co-authored-by: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Co-authored-by: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Co-authored-by: Anna Quincy <annaquincy@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
JinLee794
added a commit
that referenced
this pull request
Jan 20, 2026
#29) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * Syncinc to Azure Samples (#95) * Delete samples/labs/dev/leadership_phrases.txt * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources and remove invalid ones. - Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check. * Refactor code structure for improved readability and maintainability * feat: Enhance MemoManager with background persistence and lifecycle management - Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis. - Implemented task deduplication to cancel previous persistence tasks when a new one is initiated. - Removed unused auto-refresh functionality and related attributes from MemoManager. - Updated tests to verify new persistence behavior and ensure proper task management. - Enhanced error handling and logging for background persistence operations. * feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization * feat(session): enhance session ID management and URL parameter support - Added `pickSessionIdFromUrl` function to extract session ID from URL parameters. - Updated `getOrCreateSessionId` to allow session ID restoration from URL. - Refactored `setSessionId` for better logging and session management. - Improved `createNewSessionId` to utilize `setSessionId`. docs(api): restructure API documentation for clarity and completeness - Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, Agent Builder, Demo Environment, and TTS Health. - Added detailed descriptions and examples for each endpoint. - Included new sections for interactive API documentation and WebSocket endpoints. docs(api-reference): update WebSocket message types and endpoint details - Clarified message types for incoming audio data and control messages. - Updated WebSocket endpoint URLs and query parameters for browser conversations and dashboard relay. docs(architecture): refine agent architecture diagrams for clarity - Adjusted diagrams to improve readability and understanding of the agent framework and orchestration. fix(architecture): correct orchestration mode comparison table - Updated ratings for Azure Speech voices and simplicity of setup in the orchestration comparison table. docs(getting-started): add demo guide and enhance onboarding experience - Introduced a new demo guide to facilitate user onboarding and provide structured paths for different user levels. - Enhanced the getting started guide with tips and recommended paths for new users. feat(aoai): implement OpenAI connection warmup to reduce latency - Added `warm_openai_connection` function to pre-establish OpenAI connection and reduce cold-start latency on first call. feat(speech): implement token warmup for Speech API to minimize latency - Added `warm_token` method in `SpeechTokenManager` to pre-fetch tokens during startup, reducing latency on first API call. * feat(healthcare): Implement Nurse Triage Agent with symptom assessment and routing capabilities - Introduced a comprehensive voice agent for healthcare triage. - Added agent configuration and prompt templates for patient interaction. - Developed healthcare tools for patient verification, clinical knowledge search, and symptom urgency assessment. - Integrated routing logic for scheduling appointments and emergency transfers. - Enhanced documentation with demo scenarios and testing instructions. * feat: Implement logging utility and session management - Added a logger utility to manage console logging levels and filtering. - Created session management functions to handle session IDs, including retrieval from URL and session creation. - Developed styles for the frontend components to ensure consistent UI design. - Configured Vite for the frontend build process with proper asset handling and environment variable support. - Introduced scripts for starting the backend and frontend development servers, including Azure Dev Tunnel hosting. * feat: Simplify agent handoff process by refining context management and removing redundant data collection * feat: Enhance agent handoff process by managing conversation history and user context * feat: Enhance message handling by persisting tool calls and results as JSON for conversation continuity * feat: Implement silent handoff protocol across agents to enhance user experience and streamline transitions * feat: Add Azure App Configuration module with RBAC and Key Vault integration - Implemented main resource for Azure App Configuration in Terraform. - Added outputs for App Configuration details including ID, name, and endpoint. - Defined variables for App Configuration module, including identity and Key Vault integration. - Updated main Terraform outputs to include App Configuration details. - Enhanced error handling in Azure OpenAI client for missing endpoint configuration. - Improved Redis manager to handle port configuration with better error messaging. - Updated requirements to include Azure App Configuration SDKs. * first code clean up * enabling oidc * Refactor code structure and remove redundant sections for improved readability and maintainability * add value * add value * feat: Add managed certificate and domain registration modules - Introduced `managed-cert-example.bicep` for example usage of managed certificate deployment. - Created `managed-cert.bicep` to handle App Service Domain registration and managed SSL certificate generation. - Implemented `role-assignment.bicep` for managing role assignments with support for built-in and custom roles. - Added `windows-vm.bicep` for deploying a Windows VM as a jumphost with necessary networking components. - Developed `peer-virtual-networks.bicep` for establishing peering between virtual networks. - Implemented `private-dns-zone.bicep` for creating and linking private DNS zones to virtual networks. - Created `private-endpoint.bicep` for deploying private endpoints with DNS zone integration. - Added `vnet.bicep` for creating virtual networks with associated subnets and network security groups. - Updated `types.bicep` with new types for model deployment, role assignments, and network configurations. - Developed `secret.bicep` for managing secrets in Azure Key Vault. - Created `network.bicep` for orchestrating network resources including virtual networks and subnets. * fix: Update default location parameter in create_storage function for clarity * feat: Extract AZURE_LOCATION from environment-specific tfvars file if not set * feat: Implement location resolution with fallback chain in preprovision script * fix: Update Dockerfile to install runtime dependencies and mitigate vulnerabilities * chore: Update CHANGELOG for version 1.5.0 release and remove changenotes.md; enable remote builds in azure.yaml; enhance terraform initialization script with location prompts * feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity * further deployment cleanup, docs update/tweaks, adding more todos * removing unused dependency in src/herlpers.py * refactor: Update architecture diagram in README for clarity and consistency in orchestration modes * add value * Refactor Terraform configuration: - Update main.tf to adjust foundry account and project naming conventions. - Remove feature flags and keys from appconfig module as they are now managed externally. - Clean up variables.tf by removing unused variables and updating descriptions. - Delete provider configuration file as it is no longer needed. - Change default application name from "rtaudioagent" to "artagent" and adjust related settings. - Modify connection settings and pool sizes for improved performance. * feat: Enhance Azure Voice Live integration and refactor configuration management * last changes * feat: Add app configuration bootstrap to initialize environment variables * Enhance configuration loading with .env.local support and update documentation * fix voicelive output attributes * add * Refactor agent paths and update documentation for agent discovery and configuration * Add Insurance Voice Agent Scenario documentation and update navigation - Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management. - Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section. * Add integration proposal for Spec-Driven Development methodology in ARTVoice * add value * Enhance Terraform configuration and scripts for Voice Live integration - Update Dockerfile to install dependencies and set up virtual environment. - Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling. - Refactor sync-appconfig.sh to streamline key-value imports and feature flag management. - Add provider.conf.json generation for remote state backend configuration. - Update main.tf and outputs.tf to support new Voice Live model deployments. - Introduce voice_live_location and voice_live_model_deployments variables in variables.tf. * feat: Add Concierge agent configuration and prompts for banking scenarios - Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations. - Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes. - Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios. - Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading. - Updated default start agent to BankingConcierge and added validation for agent existence at startup. * feat: Enhance scenario loading to support orchestration.yaml naming convention * feat: Implement scenario-based handoff map resolution for orchestrator configuration * cicd test for azd deploy * feat: Update audio handling and documentation dependencies for improved installation and error handling * feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management * feat: Revise documentation deployment workflow to enhance dependency management and streamline build process * modified docs workflow * feat: Add site_dir configuration to mkdocs.yml for improved site structure * feat: Allow mkdocs build to proceed with warnings by removing --strict flag * fix: Update health check endpoint in postprovision script to use correct API path * refactor: Remove outdated AZD deployment workflow and update documentation links for clarity * fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly * refactor: Simplify state key handling in provider configuration by using environment name * fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults * fix: Use coalesce function for location assignment in storage account resource * refactor: Remove unused backend API public URL variable and related validation * refactor: Remove unused backend API public URL and source phone number from environment parameter files * improvements flow * fix: Implement auto-selection and timeout for user input in setup scripts * add value * fix: Update naming conventions for foundry account and project variables in locals * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update documentation URLs to reflect new repository location * feat: Enhance API documentation and tagging for better clarity and organization * docs: Update documentation links and improve clarity across various guides * refactor: replace deploy-azd workflow with reusable template and remove redundant summary job - Updated the deployment workflow name to "Deploy to Azure". - Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml. - Removed the deployment summary job and its associated steps to streamline the workflow. * fix: Add run-name to the Azure deployment workflow for better clarity * fix: Update condition for output extraction in deployment workflow * fix: Update GitHub token to use secrets for enhanced security * feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment * adding rg as env var set at the gh env level * fix: Add emoji to workflow names for better visibility * feat: Update documentation workflow name and enhance README with deployment badges * fix: Update README layout and enhance navigation links for better user experience * fix: Restore header for ARTVoice Accelerator Framework in README * add value * fix: Update README layout for improved clarity and navigation * Enhance provisioning scripts and documentation - Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal. - Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning. - Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms. - Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration. - Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts. - Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions. * feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements * fix(ci): simplify test-azd-hooks workflow tests and run in parallel - Remove fragile grep-based function extraction that caused syntax errors - Run lint, linux, macos, windows tests in parallel (no dependencies) - Trigger on all pushes to main/staging (remove path filters for push) - Simplify backend configuration test to avoid function sourcing issues * feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration * feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow * feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup * feat: Update test job names for clarity and enhance preflight checks for CI mode * feat: Update preflight checks to conditionally include Docker in CI mode and log its status * feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting * feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks * feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock * feat: Update Azure deployment workflows and normalize container memory formats * feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options * feat: Remove outdated troubleshooting documentation for deployment issues * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/workflows/test-azd-hooks.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: Implement TTS Streaming Latency Analysis and Optimization Plan - Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture. - Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming. - Proposed a multi-phase optimization strategy to address identified issues, including: - Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task. - Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch. - Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming. - Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing. - Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming. - Created a detailed test implementation plan with metrics and success criteria to validate improvements. test: Add unit tests for HandoffService - Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods. - Implemented tests for greeting selection and context building to ensure proper functionality. - Added tests for the HandoffResolution dataclass to verify properties and default values. * feat: Add Scenario Builder component and integrate with RealTimeVoiceApp - Introduced ScenarioBuilder component for visual orchestration of agent flows. - Implemented drag-and-drop functionality for agents and handoff configuration. - Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders. - Enhanced state management for agent scenarios, including creation and updates. - Integrated new handoff editor for configuring agent interactions. * Refactor code structure for improved readability and maintainability * Add error handling for Redis connection issues and implement unit tests for HandoffService - Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts. - Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building. - Added tests for HandoffResolution dataclass to ensure correct property behavior and default values. * Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts * Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates. * Refactor environment variable assignment in deploy workflow for clarity * Refactor tests and dependencies following module renaming and API changes - Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock. - Updated conftest.py to mock configuration and Azure OpenAI client for tests. - Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules. - Adjusted imports in test_artagent_wshelpers.py for orchestrator path change. - Skipped tests in test_call_transfer_service.py due to API changes in toolstore. - Updated datetime usage in test_demo_env_phrase_bias.py to use UTC. - Modified websocket endpoint assertions in test_realtime.py to reflect new paths. - Added new test file test_voice_handler_components.py for voice handler components. * Add comprehensive tests for VoiceLive handler and orchestrator memory management - Implement tests to verify cleanup functionality in LiveOrchestrator. - Ensure proper registration and unregistration of orchestrators in the registry. - Test background task tracking and cleanup mechanisms. - Validate greeting task cancellation during orchestrator cleanup. - Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry. - Verify user message history deque is properly bounded and cleared on cleanup. - Add scenario update tests to ensure correct agent management during updates. - Optimize hot path functions to ensure non-blocking behavior during network calls. * feat: Enhance AgentBuilder with consistent field names and improved UI elements * Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues. * feat: Add predefined handoff condition patterns to enhance scenario orchestration * add value * feat(metrics): Introduce shared metrics factory for lazy initialization - Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics. - Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics. - Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization. - Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling. - Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls. - Introduced utility functions for building common metric attributes, ensuring consistency across metrics. * feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation * fix: Simplify environment determination logic in deployment workflow * add value * feat: Add user flow screenshots and enhance documentation for guided agent setup * feat: Enhance scenario testing instructions for clarity and user guidance * fix: Correct image paths in quickstart guide for accurate rendering * feat: Add initial agent builder and template selection screenshots to quickstart guide * feat: Add demo profile creation steps and related images to quickstart guide * feat: Implement EasyAuth configuration script and integrate into post-provisioning process * refactor: Remove backend IP restrictions configuration and related outputs * Added non qualifying rush response to ensure clear model behavior * updated order so confirmation statement is in the correct spot * add value * add value * chore: Remove unused workflow images for demo profiles * fix: Update demo profile creation images in quickstart guide * fix: Update home screen image in quickstart guide * fix: Update home screen and scenario images in quickstart guide * add value * add value * add value * add value * add value * add * add value * art * add opentelemetry import for tracing support in TTS module * refactor: update LiveOrchestrator to enhance user message history management and improve handoff context * Refactor TTS Playback and Voice Handling - Consolidated TTS playback logic into a unified class for speech cascade. - Removed deprecated VoiceSessionContext and related compatibility shims. - Enhanced error handling during tool initialization and event handler registration. - Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings. - Improved logging for TTS synthesis and streaming processes. - Added new handoff tool registration for dynamic routing. * refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts * refactor: enhance EasyAuth interactive prompts and streamline user choices * refactor: enhance run-name logic for Azure deployment workflow * fix: update environment logic for pull_request events in Azure deployment workflow * refactor: update preprovision hook execution and streamline backend configuration * feat: add context variable support for handoffs and enhance UI for variable mapping * feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Responses API Infrastructure & Dual Model Configuration (#16) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * fix: update project version to 2.0.0-beta in pyproject.toml --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Orchestrator Integration + Optimizations (#17) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: integrate Responses API in orchestrators and add optimizations **Cascade Orchestrator:** - Update model selection to use agent.get_model_for_mode('cascade') - Integrate Responses API routing based on endpoint_preference - Add error handling for unsupported parameters - Extract TTS processing into separate tts_processor module **VoiceLive Orchestrator:** - Update to use agent.get_model_for_mode('voicelive') - Add registry cleanup to prevent unbounded growth - Improve memory management and stale orchestrator cleanup - Extract DTMF processing into separate dtmf_processor module **Tests:** - Add test_cascade_orchestrator_entry_points - Add test_cascade_llm_processing - Add test_dtmf_processor Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Evaluation Framework + Frontend UI (#18) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: add evaluation framework and frontend UI for Responses API **Evaluation Framework:** - Add EventRecorder with git commit SHA tracking - Add API-aware scoring with budget adjustments for verbosity - Add scenario runner for automated testing - Add CLI for running evaluations - Add validate_phases.py for phase-based validation - Add wrappers for endpoint detection **Frontend UI:** - Add cascade_model and voicelive_model selectors in Agent Builder - Add Responses API endpoint preference dropdown - Add conditional fields for verbosity, reasoning_effort, etc. - Update ScenarioBuilder with model configuration options - Display API version fields **Documentation:** - Add docs/testing/model-evaluation.md - Add evaluation playground Jupyter notebook Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] * Enhance logging and user prompts in preflight and pre-provisioning scripts (#20) - Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting. - Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh. - Added color-coded success, warning, and error messages for better visibility. - Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values. - Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration (#21) Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * enhanced the scenariobuilder with flowy (#22) * docs: add comprehensive voice processing architecture documentation Add complete documentation for the voice processing architecture: New Documentation: - docs/architecture/voice/README.md - Comprehensive voice architecture guide * VoiceHandler overview and usage patterns * TTS playback and text processing * Speech cascade pipeline documentation * Audio specifications for browser and ACS transports * Testing guidelines with actual test file references * Troubleshooting guide for common issues - apps/artagent/backend/voice/README.md - Developer quick reference * Directory structure and module organization * Quick start examples * Common tasks and patterns * File location guide * Testing commands Documentation Updates: - docs/mkdocs.yml - Add voice architecture to navigation - docs/operations/troubleshooting.md - Add voice-specific troubleshooting Key Improvements: - Fixed mkdocs formatting for proper list rendering - Updated all test references to match actual test files: * test_voice_handler_components.py * test_voice_handler_compat.py * test_cascade_orchestrator_entry_points.py * test_cascade_llm_processing.py - Verified all script references (quick_test.sh, test_orchestrator.py) - Added prerequisites for running tests with dev dependencies - Included both basic and advanced testing examples All file paths and examples have been verified against the actual codebase. Related to #[TBD] * Add custom styles for Flowy flowchart integration with agent blocks * feat: Enhance output port visibility logic in ScenarioGraphCanvas * feat: Add expandable full prompt view for source agent in HandoffEditorDialog --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Refactor ACS logging and add default orchestration scenario - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. * Refactor ACS logging and add default orchestration scenario (#23) - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Enhance logging functions to use log_plain for consistency and clarity in local development setup script * Disable view toggle buttons for chat/graph/timeline in ConversationControls * Add panning functionality to ScenarioGraphCanvas and reset button * Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes * feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts - Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation. - Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements. - Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints. - Improved metrics naming conventions for clarity in load testing results. * feat: Update Voice Live readiness status to use event envelope format --------- Co-authored-by: Jin Lee <94473824+JinLee794@users.noreply.github.com> Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> Co-authored-by: Anna Quincy <annaquincy@microsoft.com> Co-authored-by: Jin Lee <jinle@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Consolidate infrastructure resource documentation into infra/README.md (#26) * Initial plan * Add comprehensive infrastructure resources documentation with private networking links Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> * Consolidate infrastructure documentation into infra/README.md Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> * enhancement: infra docs readme update (#100) * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources and remove invalid ones. - Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check. * Refactor code structure for improved readability and maintainability * feat: Enhance MemoManager with background persistence and lifecycle management - Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis. - Implemented task deduplication to cancel previous persistence tasks when a new one is initiated. - Removed unused auto-refresh functionality and related attributes from MemoManager. - Updated tests to verify new persistence behavior and ensure proper task management. - Enhanced error handling and logging for background persistence operations. * feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization * feat(session): enhance sess…
JinLee794
added a commit
that referenced
this pull request
Jan 20, 2026
* feat: Enhance status envelope with optional label and update frontend to derive WS URL
- Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages.
- Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets.
- Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`.
- Enhanced `App.jsx` to format event type labels and summarize event data for better user experience.
- Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing.
- Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events.
- Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`.
- Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`.
* refactor: Comment out unused email communication service domain resource
* refactor: Comment out unused Azure email communication service resources
* feat: Enhance event handling and UI components
- Added new utility functions for formatting event types and summarizing event data in App.jsx.
- Improved ChatBubble component to display event messages with formatted labels and timestamps.
- Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags.
- Introduced websocket URL derivation in postprovision.sh for better backend integration.
- Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting.
- Updated package.json to include js-yaml and upgraded vite version.
* add value
* feat: Enhance distributed session handling and improve PayPal agent interactions
- Implement distributed session bus using Redis for cross-replica session routing in connection manager.
- Add methods for publishing session envelopes to Redis channels.
- Introduce confirmation context for call center transfers to ensure explicit user consent.
- Update PayPal agent templates to clarify authentication and routing guidelines.
- Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively.
- Improve error handling and logging for distributed session delivery and Redis interactions.
- Refactor session envelope handling in frontend to accommodate new event types and improve user experience.
* feat: Enhance status tone metadata and improve chat bubble styling
* feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes
* feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios
* feat: Implement retry mechanism for browser session ID resolution in media streaming
* feat: Enhance session management and greeting handling across various components
* fixing session mapping for acs calls
* add value
* add value
* adding test file
* Adding agents and templates for credit card recommendation and fee dispute agents
* add value
* Enhance audio transcription settings across agents and adjust logging levels for better debugging
* Enhance audio transcription settings across agents and adjust logging levels for better debugging
* add value
* add value
* Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments
* add value
* Add Azure Voice Live model configuration and outputs
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* fixing voicelive chat sequence on the ui
* remove sensitive contact information and unused transfer agency client data
* feat: Introduce Agent Consolidation Plan with YAML-driven architecture
- Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`.
- Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry.
- Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries.
- Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry.
- Detailed implementation roadmap divided into phases for gradual migration and integration.
- Included backward compatibility strategy to ensure existing agents function without modification.
- Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist.
* Refactor speech cascade handler and routing for browser communication
- Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events.
- Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input.
- Changed routing from `/realtime` to `/browser` for browser communication endpoints.
- Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap.
- Introduced WebSocket helper functions for better organization and clarity in messaging.
- Enhanced connection manager to handle Redis pubsub reconnections on credential expiration.
- Updated frontend components to reflect routing changes for browser communication.
- Adjusted tests to align with the new browser routing and functionality.
- Commented out live metrics enabling condition in telemetry configuration for future consideration.
* feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls
- Introduced , , , and decorators for OpenTelemetry instrumentation.
- Implemented context manager for tracking conversation turns with detailed metrics.
- Added helper functions for recording GenAI and speech metrics.
- Enhanced span attributes for Azure Application Insights visualization.
* Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality.
* feat: Enhance telemetry and tracing for CosmosDB and latency tool
- Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking.
- Integrated tracing spans in the LatencyTool for better observability in Application Insights.
- Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics.
- Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence.
- Removed outdated endpoints review document.
* feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints
test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications
* WARNING!!!! MAJOR REFACTOR COMMIT
- Removed the VoiceLive SDK integration module from the backend.
- Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections.
- Integrated the AgentTopologyPanel into the main application layout.
- Updated the BackendIndicator to include agent count and selection functionality.
- Enhanced the ConversationControls with a fixed view switcher for better accessibility.
- Improved the useBackendHealth hook to handle various agent data structures.
- Updated styles for better responsiveness and visual consistency across components.
- Modified utility functions to format agent inventory data correctly.
- Adjusted import paths in orchestrators and tests to reflect the new backend structure.
* feat: Enhance agent handoff process and response handling; refactor UI components for improved usability
* feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents
* refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization
* feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements
- Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically.
- Updated panel styling for improved aesthetics, including a gradient background and adjusted borders.
- Enhanced scrollbar visibility and overflow handling for better user experience.
refactor: Simplify GraphListView filter logic
- Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied.
- Cleaned up useEffect dependencies for better performance and clarity.
docs: Introduce Backend Voice & Agents Architecture documentation
- Added comprehensive documentation outlining the architecture of backend voice and agent modules.
- Detailed separation of concerns between voice transport and agent business logic.
- Included data flow diagrams and module responsibilities for clarity.
docs: Create Handoff Logic Inventory for better understanding of handoff processes
- Documented the handoff logic across backend voice and agent modules.
- Established a single source of truth for handoff mappings and protocols.
- Summarized cleanup phases and their impact on the codebase.
fix: Update logging to safely handle span names
- Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan.
fix: Adjust telemetry configuration to capture all loggers
- Changed logger_name default to an empty string in TelemetryConfig to capture all loggers.
* feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging
* feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration
* feat: Enhance Agent Details Panel and Session Management
- Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display.
- Implemented logic to show agent name, description, tools, and model/voice details based on session configuration.
- Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview.
- Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating.
- Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use.
- Enhanced on_demand_pool.py to validate cached resources and remove invalid ones.
- Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check.
* Refactor code structure for improved readability and maintainability
* feat: Enhance MemoManager with background persistence and lifecycle management
- Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis.
- Implemented task deduplication to cancel previous persistence tasks when a new one is initiated.
- Removed unused auto-refresh functionality and related attributes from MemoManager.
- Updated tests to verify new persistence behavior and ensure proper task management.
- Enhanced error handling and logging for background persistence operations.
* feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization
* feat(session): enhance session ID management and URL parameter support
- Added `pickSessionIdFromUrl` function to extract session ID from URL parameters.
- Updated `getOrCreateSessionId` to allow session ID restoration from URL.
- Refactored `setSessionId` for better logging and session management.
- Improved `createNewSessionId` to utilize `setSessionId`.
docs(api): restructure API documentation for clarity and completeness
- Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, Agent Builder, Demo Environment, and TTS Health.
- Added detailed descriptions and examples for each endpoint.
- Included new sections for interactive API documentation and WebSocket endpoints.
docs(api-reference): update WebSocket message types and endpoint details
- Clarified message types for incoming audio data and control messages.
- Updated WebSocket endpoint URLs and query parameters for browser conversations and dashboard relay.
docs(architecture): refine agent architecture diagrams for clarity
- Adjusted diagrams to improve readability and understanding of the agent framework and orchestration.
fix(architecture): correct orchestration mode comparison table
- Updated ratings for Azure Speech voices and simplicity of setup in the orchestration comparison table.
docs(getting-started): add demo guide and enhance onboarding experience
- Introduced a new demo guide to facilitate user onboarding and provide structured paths for different user levels.
- Enhanced the getting started guide with tips and recommended paths for new users.
feat(aoai): implement OpenAI connection warmup to reduce latency
- Added `warm_openai_connection` function to pre-establish OpenAI connection and reduce cold-start latency on first call.
feat(speech): implement token warmup for Speech API to minimize latency
- Added `warm_token` method in `SpeechTokenManager` to pre-fetch tokens during startup, reducing latency on first API call.
* feat(healthcare): Implement Nurse Triage Agent with symptom assessment and routing capabilities
- Introduced a comprehensive voice agent for healthcare triage.
- Added agent configuration and prompt templates for patient interaction.
- Developed healthcare tools for patient verification, clinical knowledge search, and symptom urgency assessment.
- Integrated routing logic for scheduling appointments and emergency transfers.
- Enhanced documentation with demo scenarios and testing instructions.
* feat: Implement logging utility and session management
- Added a logger utility to manage console logging levels and filtering.
- Created session management functions to handle session IDs, including retrieval from URL and session creation.
- Developed styles for the frontend components to ensure consistent UI design.
- Configured Vite for the frontend build process with proper asset handling and environment variable support.
- Introduced scripts for starting the backend and frontend development servers, including Azure Dev Tunnel hosting.
* feat: Simplify agent handoff process by refining context management and removing redundant data collection
* feat: Enhance agent handoff process by managing conversation history and user context
* feat: Enhance message handling by persisting tool calls and results as JSON for conversation continuity
* feat: Implement silent handoff protocol across agents to enhance user experience and streamline transitions
* feat: Add Azure App Configuration module with RBAC and Key Vault integration
- Implemented main resource for Azure App Configuration in Terraform.
- Added outputs for App Configuration details including ID, name, and endpoint.
- Defined variables for App Configuration module, including identity and Key Vault integration.
- Updated main Terraform outputs to include App Configuration details.
- Enhanced error handling in Azure OpenAI client for missing endpoint configuration.
- Improved Redis manager to handle port configuration with better error messaging.
- Updated requirements to include Azure App Configuration SDKs.
* first code clean up
* enabling oidc
* Refactor code structure and remove redundant sections for improved readability and maintainability
* add value
* add value
* feat: Add managed certificate and domain registration modules
- Introduced `managed-cert-example.bicep` for example usage of managed certificate deployment.
- Created `managed-cert.bicep` to handle App Service Domain registration and managed SSL certificate generation.
- Implemented `role-assignment.bicep` for managing role assignments with support for built-in and custom roles.
- Added `windows-vm.bicep` for deploying a Windows VM as a jumphost with necessary networking components.
- Developed `peer-virtual-networks.bicep` for establishing peering between virtual networks.
- Implemented `private-dns-zone.bicep` for creating and linking private DNS zones to virtual networks.
- Created `private-endpoint.bicep` for deploying private endpoints with DNS zone integration.
- Added `vnet.bicep` for creating virtual networks with associated subnets and network security groups.
- Updated `types.bicep` with new types for model deployment, role assignments, and network configurations.
- Developed `secret.bicep` for managing secrets in Azure Key Vault.
- Created `network.bicep` for orchestrating network resources including virtual networks and subnets.
* fix: Update default location parameter in create_storage function for clarity
* feat: Extract AZURE_LOCATION from environment-specific tfvars file if not set
* feat: Implement location resolution with fallback chain in preprovision script
* fix: Update Dockerfile to install runtime dependencies and mitigate vulnerabilities
* chore: Update CHANGELOG for version 1.5.0 release and remove changenotes.md; enable remote builds in azure.yaml; enhance terraform initialization script with location prompts
* feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity
* further deployment cleanup, docs update/tweaks, adding more todos
* removing unused dependency in src/herlpers.py
* refactor: Update architecture diagram in README for clarity and consistency in orchestration modes
* add value
* Refactor Terraform configuration:
- Update main.tf to adjust foundry account and project naming conventions.
- Remove feature flags and keys from appconfig module as they are now managed externally.
- Clean up variables.tf by removing unused variables and updating descriptions.
- Delete provider configuration file as it is no longer needed.
- Change default application name from "rtaudioagent" to "artagent" and adjust related settings.
- Modify connection settings and pool sizes for improved performance.
* feat: Enhance Azure Voice Live integration and refactor configuration management
* last changes
* feat: Add app configuration bootstrap to initialize environment variables
* Enhance configuration loading with .env.local support and update documentation
* fix voicelive output attributes
* add
* Refactor agent paths and update documentation for agent discovery and configuration
* Add Insurance Voice Agent Scenario documentation and update navigation
- Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management.
- Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section.
* Add integration proposal for Spec-Driven Development methodology in ARTVoice
* add value
* Enhance Terraform configuration and scripts for Voice Live integration
- Update Dockerfile to install dependencies and set up virtual environment.
- Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling.
- Refactor sync-appconfig.sh to streamline key-value imports and feature flag management.
- Add provider.conf.json generation for remote state backend configuration.
- Update main.tf and outputs.tf to support new Voice Live model deployments.
- Introduce voice_live_location and voice_live_model_deployments variables in variables.tf.
* feat: Add Concierge agent configuration and prompts for banking scenarios
- Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations.
- Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes.
- Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios.
- Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading.
- Updated default start agent to BankingConcierge and added validation for agent existence at startup.
* feat: Enhance scenario loading to support orchestration.yaml naming convention
* feat: Implement scenario-based handoff map resolution for orchestrator configuration
* cicd test for azd deploy
* feat: Update audio handling and documentation dependencies for improved installation and error handling
* feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management
* feat: Revise documentation deployment workflow to enhance dependency management and streamline build process
* modified docs workflow
* feat: Add site_dir configuration to mkdocs.yml for improved site structure
* feat: Allow mkdocs build to proceed with warnings by removing --strict flag
* fix: Update health check endpoint in postprovision script to use correct API path
* refactor: Remove outdated AZD deployment workflow and update documentation links for clarity
* fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly
* refactor: Simplify state key handling in provider configuration by using environment name
* fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults
* fix: Use coalesce function for location assignment in storage account resource
* refactor: Remove unused backend API public URL variable and related validation
* refactor: Remove unused backend API public URL and source phone number from environment parameter files
* improvements flow
* fix: Implement auto-selection and timeout for user input in setup scripts
* add value
* fix: Update naming conventions for foundry account and project variables in locals
* fix: Update name from rtaudioagent to artaudioagent in environment parameter files
* fix: Update name from rtaudioagent to artaudioagent in environment parameter files
* fix: Update documentation URLs to reflect new repository location
* feat: Enhance API documentation and tagging for better clarity and organization
* docs: Update documentation links and improve clarity across various guides
* refactor: replace deploy-azd workflow with reusable template and remove redundant summary job
- Updated the deployment workflow name to "Deploy to Azure".
- Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml.
- Removed the deployment summary job and its associated steps to streamline the workflow.
* fix: Add run-name to the Azure deployment workflow for better clarity
* fix: Update condition for output extraction in deployment workflow
* fix: Update GitHub token to use secrets for enhanced security
* feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment
* adding rg as env var set at the gh env level
* fix: Add emoji to workflow names for better visibility
* feat: Update documentation workflow name and enhance README with deployment badges
* fix: Update README layout and enhance navigation links for better user experience
* fix: Restore header for ARTVoice Accelerator Framework in README
* add value
* fix: Update README layout for improved clarity and navigation
* Enhance provisioning scripts and documentation
- Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal.
- Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning.
- Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms.
- Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration.
- Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts.
- Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions.
* feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements
* fix(ci): simplify test-azd-hooks workflow tests and run in parallel
- Remove fragile grep-based function extraction that caused syntax errors
- Run lint, linux, macos, windows tests in parallel (no dependencies)
- Trigger on all pushes to main/staging (remove path filters for push)
- Simplify backend configuration test to avoid function sourcing issues
* feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration
* feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow
* feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup
* feat: Update test job names for clarity and enhance preflight checks for CI mode
* feat: Update preflight checks to conditionally include Docker in CI mode and log its status
* feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting
* feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks
* feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock
* feat: Update Azure deployment workflows and normalize container memory formats
* feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options
* feat: Remove outdated troubleshooting documentation for deployment issues
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/test-azd-hooks.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: Implement TTS Streaming Latency Analysis and Optimization Plan
- Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture.
- Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming.
- Proposed a multi-phase optimization strategy to address identified issues, including:
- Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task.
- Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch.
- Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming.
- Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing.
- Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming.
- Created a detailed test implementation plan with metrics and success criteria to validate improvements.
test: Add unit tests for HandoffService
- Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods.
- Implemented tests for greeting selection and context building to ensure proper functionality.
- Added tests for the HandoffResolution dataclass to verify properties and default values.
* feat: Add Scenario Builder component and integrate with RealTimeVoiceApp
- Introduced ScenarioBuilder component for visual orchestration of agent flows.
- Implemented drag-and-drop functionality for agents and handoff configuration.
- Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders.
- Enhanced state management for agent scenarios, including creation and updates.
- Integrated new handoff editor for configuring agent interactions.
* Refactor code structure for improved readability and maintainability
* Add error handling for Redis connection issues and implement unit tests for HandoffService
- Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts.
- Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building.
- Added tests for HandoffResolution dataclass to ensure correct property behavior and default values.
* Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts
* Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates.
* Refactor environment variable assignment in deploy workflow for clarity
* Refactor tests and dependencies following module renaming and API changes
- Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock.
- Updated conftest.py to mock configuration and Azure OpenAI client for tests.
- Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules.
- Adjusted imports in test_artagent_wshelpers.py for orchestrator path change.
- Skipped tests in test_call_transfer_service.py due to API changes in toolstore.
- Updated datetime usage in test_demo_env_phrase_bias.py to use UTC.
- Modified websocket endpoint assertions in test_realtime.py to reflect new paths.
- Added new test file test_voice_handler_components.py for voice handler components.
* Add comprehensive tests for VoiceLive handler and orchestrator memory management
- Implement tests to verify cleanup functionality in LiveOrchestrator.
- Ensure proper registration and unregistration of orchestrators in the registry.
- Test background task tracking and cleanup mechanisms.
- Validate greeting task cancellation during orchestrator cleanup.
- Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry.
- Verify user message history deque is properly bounded and cleared on cleanup.
- Add scenario update tests to ensure correct agent management during updates.
- Optimize hot path functions to ensure non-blocking behavior during network calls.
* feat: Enhance AgentBuilder with consistent field names and improved UI elements
* Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues.
* feat: Add predefined handoff condition patterns to enhance scenario orchestration
* add value
* feat(metrics): Introduce shared metrics factory for lazy initialization
- Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics.
- Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics.
- Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization.
- Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling.
- Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls.
- Introduced utility functions for building common metric attributes, ensuring consistency across metrics.
* feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation
* fix: Simplify environment determination logic in deployment workflow
* add value
* feat: Add user flow screenshots and enhance documentation for guided agent setup
* feat: Enhance scenario testing instructions for clarity and user guidance
* fix: Correct image paths in quickstart guide for accurate rendering
* feat: Add initial agent builder and template selection screenshots to quickstart guide
* feat: Add demo profile creation steps and related images to quickstart guide
* feat: Implement EasyAuth configuration script and integrate into post-provisioning process
* refactor: Remove backend IP restrictions configuration and related outputs
* Added non qualifying rush response to ensure clear model behavior
* updated order so confirmation statement is in the correct spot
* add value
* add value
* chore: Remove unused workflow images for demo profiles
* fix: Update demo profile creation images in quickstart guide
* fix: Update home screen image in quickstart guide
* fix: Update home screen and scenario images in quickstart guide
* add value
* add value
* add value
* add value
* add value
* add
* add value
* art
* add opentelemetry import for tracing support in TTS module
* refactor: update LiveOrchestrator to enhance user message history management and improve handoff context
* Refactor TTS Playback and Voice Handling
- Consolidated TTS playback logic into a unified class for speech cascade.
- Removed deprecated VoiceSessionContext and related compatibility shims.
- Enhanced error handling during tool initialization and event handler registration.
- Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings.
- Improved logging for TTS synthesis and streaming processes.
- Added new handoff tool registration for dynamic routing.
* refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts
* refactor: enhance EasyAuth interactive prompts and streamline user choices
* refactor: enhance run-name logic for Azure deployment workflow
* fix: update environment logic for pull_request events in Azure deployment workflow
* refactor: update preprovision hook execution and streamline backend configuration
* feat: add context variable support for handoffs and enhance UI for variable mapping
* feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Responses API Infrastructure & Dual Model Configuration (#16)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* fix: update project version to 2.0.0-beta in pyproject.toml
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Orchestrator Integration + Optimizations (#17)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: integrate Responses API in orchestrators and add optimizations
**Cascade Orchestrator:**
- Update model selection to use agent.get_model_for_mode('cascade')
- Integrate Responses API routing based on endpoint_preference
- Add error handling for unsupported parameters
- Extract TTS processing into separate tts_processor module
**VoiceLive Orchestrator:**
- Update to use agent.get_model_for_mode('voicelive')
- Add registry cleanup to prevent unbounded growth
- Improve memory management and stale orchestrator cleanup
- Extract DTMF processing into separate dtmf_processor module
**Tests:**
- Add test_cascade_orchestrator_entry_points
- Add test_cascade_llm_processing
- Add test_dtmf_processor
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: Evaluation Framework + Frontend UI (#18)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: add evaluation framework and frontend UI for Responses API
**Evaluation Framework:**
- Add EventRecorder with git commit SHA tracking
- Add API-aware scoring with budget adjustments for verbosity
- Add scenario runner for automated testing
- Add CLI for running evaluations
- Add validate_phases.py for phase-based validation
- Add wrappers for endpoint detection
**Frontend UI:**
- Add cascade_model and voicelive_model selectors in Agent Builder
- Add Responses API endpoint preference dropdown
- Add conditional fields for verbosity, reasoning_effort, etc.
- Update ScenarioBuilder with model configuration options
- Display API version fields
**Documentation:**
- Add docs/testing/model-evaluation.md
- Add evaluation playground Jupyter notebook
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: voice handler refactoring and MediaHandler migration
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
* Enhance logging and user prompts in preflight and pre-provisioning scripts (#20)
- Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting.
- Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh.
- Added color-coded success, warning, and error messages for better visibility.
- Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values.
- Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* feat: voice handler refactoring and MediaHandler migration (#21)
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* enhanced the scenariobuilder with flowy (#22)
* docs: add comprehensive voice processing architecture documentation
Add complete documentation for the voice processing architecture:
New Documentation:
- docs/architecture/voice/README.md - Comprehensive voice architecture guide
* VoiceHandler overview and usage patterns
* TTS playback and text processing
* Speech cascade pipeline documentation
* Audio specifications for browser and ACS transports
* Testing guidelines with actual test file references
* Troubleshooting guide for common issues
- apps/artagent/backend/voice/README.md - Developer quick reference
* Directory structure and module organization
* Quick start examples
* Common tasks and patterns
* File location guide
* Testing commands
Documentation Updates:
- docs/mkdocs.yml - Add voice architecture to navigation
- docs/operations/troubleshooting.md - Add voice-specific troubleshooting
Key Improvements:
- Fixed mkdocs formatting for proper list rendering
- Updated all test references to match actual test files:
* test_voice_handler_components.py
* test_voice_handler_compat.py
* test_cascade_orchestrator_entry_points.py
* test_cascade_llm_processing.py
- Verified all script references (quick_test.sh, test_orchestrator.py)
- Added prerequisites for running tests with dev dependencies
- Included both basic and advanced testing examples
All file paths and examples have been verified against the actual codebase.
Related to #[TBD]
* Add custom styles for Flowy flowchart integration with agent blocks
* feat: Enhance output port visibility logic in ScenarioGraphCanvas
* feat: Add expandable full prompt view for source agent in HandoffEditorDialog
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Refactor ACS logging and add default orchestration scenario
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
* Refactor ACS logging and add default orchestration scenario (#23)
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
* Enhance logging functions to use log_plain for consistency and clarity in local development setup script
* Disable view toggle buttons for chat/graph/timeline in ConversationControls
* Add panning functionality to ScenarioGraphCanvas and reset button
* Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes
* feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts
- Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation.
- Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements.
- Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints.
- Improved metrics naming conventions for clarity in load testing results.
* feat: Update Voice Live readiness status to use event envelope format
* Consolidate infrastructure resource documentation into infra/README.md (#26)
* Initial plan
* Add comprehensive infrastructure resources documentation with private networking links
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Consolidate infrastructure documentation into infra/README.md
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Replace customer-specific names with generic tokens in agentstore and toolstore (#28)
* Initial plan
* Replace customer-specific names (Erica, Merrill) with generic tokens in agentstore and toolstore
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Update docstring examples and complete customer name replacement
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Update test fixtures to use generic assistant names instead of Erica
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Revert data field name change to preserve backward compatibility with data model
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Revert docstring examples to use actual agent identifier for accuracy
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com>
* Update apps/artagent/backend/registries/toolstore/handoffs.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Co-authored-by: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Co-authored-by: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Co-authored-by: Anna Quincy <annaquincy@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
JinLee794
added a commit
that referenced
this pull request
Jan 22, 2026
…oyment fixes/enhancements (#30) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * Syncinc to Azure Samples (#95) * Delete samples/labs/dev/leadership_phrases.txt * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources and remove invalid ones. - Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check. * Refactor code structure for improved readability and maintainability * feat: Enhance MemoManager with background persistence and lifecycle management - Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis. - Implemented task deduplication to cancel previous persistence tasks when a new one is initiated. - Removed unused auto-refresh functionality and related attributes from MemoManager. - Updated tests to verify new persistence behavior and ensure proper task management. - Enhanced error handling and logging for background persistence operations. * feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization * feat(session): enhance session ID management and URL parameter support - Added `pickSessionIdFromUrl` function to extract session ID from URL parameters. - Updated `getOrCreateSessionId` to allow session ID restoration from URL. - Refactored `setSessionId` for better logging and session management. - Improved `createNewSessionId` to utilize `setSessionId`. docs(api): restructure API documentation for clarity and completeness - Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, Agent Builder, Demo Environment, and TTS Health. - Added detailed descriptions and examples for each endpoint. - Included new sections for interactive API documentation and WebSocket endpoints. docs(api-reference): update WebSocket message types and endpoint details - Clarified message types for incoming audio data and control messages. - Updated WebSocket endpoint URLs and query parameters for browser conversations and dashboard relay. docs(architecture): refine agent architecture diagrams for clarity - Adjusted diagrams to improve readability and understanding of the agent framework and orchestration. fix(architecture): correct orchestration mode comparison table - Updated ratings for Azure Speech voices and simplicity of setup in the orchestration comparison table. docs(getting-started): add demo guide and enhance onboarding experience - Introduced a new demo guide to facilitate user onboarding and provide structured paths for different user levels. - Enhanced the getting started guide with tips and recommended paths for new users. feat(aoai): implement OpenAI connection warmup to reduce latency - Added `warm_openai_connection` function to pre-establish OpenAI connection and reduce cold-start latency on first call. feat(speech): implement token warmup for Speech API to minimize latency - Added `warm_token` method in `SpeechTokenManager` to pre-fetch tokens during startup, reducing latency on first API call. * feat(healthcare): Implement Nurse Triage Agent with symptom assessment and routing capabilities - Introduced a comprehensive voice agent for healthcare triage. - Added agent configuration and prompt templates for patient interaction. - Developed healthcare tools for patient verification, clinical knowledge search, and symptom urgency assessment. - Integrated routing logic for scheduling appointments and emergency transfers. - Enhanced documentation with demo scenarios and testing instructions. * feat: Implement logging utility and session management - Added a logger utility to manage console logging levels and filtering. - Created session management functions to handle session IDs, including retrieval from URL and session creation. - Developed styles for the frontend components to ensure consistent UI design. - Configured Vite for the frontend build process with proper asset handling and environment variable support. - Introduced scripts for starting the backend and frontend development servers, including Azure Dev Tunnel hosting. * feat: Simplify agent handoff process by refining context management and removing redundant data collection * feat: Enhance agent handoff process by managing conversation history and user context * feat: Enhance message handling by persisting tool calls and results as JSON for conversation continuity * feat: Implement silent handoff protocol across agents to enhance user experience and streamline transitions * feat: Add Azure App Configuration module with RBAC and Key Vault integration - Implemented main resource for Azure App Configuration in Terraform. - Added outputs for App Configuration details including ID, name, and endpoint. - Defined variables for App Configuration module, including identity and Key Vault integration. - Updated main Terraform outputs to include App Configuration details. - Enhanced error handling in Azure OpenAI client for missing endpoint configuration. - Improved Redis manager to handle port configuration with better error messaging. - Updated requirements to include Azure App Configuration SDKs. * first code clean up * enabling oidc * Refactor code structure and remove redundant sections for improved readability and maintainability * add value * add value * feat: Add managed certificate and domain registration modules - Introduced `managed-cert-example.bicep` for example usage of managed certificate deployment. - Created `managed-cert.bicep` to handle App Service Domain registration and managed SSL certificate generation. - Implemented `role-assignment.bicep` for managing role assignments with support for built-in and custom roles. - Added `windows-vm.bicep` for deploying a Windows VM as a jumphost with necessary networking components. - Developed `peer-virtual-networks.bicep` for establishing peering between virtual networks. - Implemented `private-dns-zone.bicep` for creating and linking private DNS zones to virtual networks. - Created `private-endpoint.bicep` for deploying private endpoints with DNS zone integration. - Added `vnet.bicep` for creating virtual networks with associated subnets and network security groups. - Updated `types.bicep` with new types for model deployment, role assignments, and network configurations. - Developed `secret.bicep` for managing secrets in Azure Key Vault. - Created `network.bicep` for orchestrating network resources including virtual networks and subnets. * fix: Update default location parameter in create_storage function for clarity * feat: Extract AZURE_LOCATION from environment-specific tfvars file if not set * feat: Implement location resolution with fallback chain in preprovision script * fix: Update Dockerfile to install runtime dependencies and mitigate vulnerabilities * chore: Update CHANGELOG for version 1.5.0 release and remove changenotes.md; enable remote builds in azure.yaml; enhance terraform initialization script with location prompts * feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity * further deployment cleanup, docs update/tweaks, adding more todos * removing unused dependency in src/herlpers.py * refactor: Update architecture diagram in README for clarity and consistency in orchestration modes * add value * Refactor Terraform configuration: - Update main.tf to adjust foundry account and project naming conventions. - Remove feature flags and keys from appconfig module as they are now managed externally. - Clean up variables.tf by removing unused variables and updating descriptions. - Delete provider configuration file as it is no longer needed. - Change default application name from "rtaudioagent" to "artagent" and adjust related settings. - Modify connection settings and pool sizes for improved performance. * feat: Enhance Azure Voice Live integration and refactor configuration management * last changes * feat: Add app configuration bootstrap to initialize environment variables * Enhance configuration loading with .env.local support and update documentation * fix voicelive output attributes * add * Refactor agent paths and update documentation for agent discovery and configuration * Add Insurance Voice Agent Scenario documentation and update navigation - Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management. - Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section. * Add integration proposal for Spec-Driven Development methodology in ARTVoice * add value * Enhance Terraform configuration and scripts for Voice Live integration - Update Dockerfile to install dependencies and set up virtual environment. - Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling. - Refactor sync-appconfig.sh to streamline key-value imports and feature flag management. - Add provider.conf.json generation for remote state backend configuration. - Update main.tf and outputs.tf to support new Voice Live model deployments. - Introduce voice_live_location and voice_live_model_deployments variables in variables.tf. * feat: Add Concierge agent configuration and prompts for banking scenarios - Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations. - Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes. - Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios. - Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading. - Updated default start agent to BankingConcierge and added validation for agent existence at startup. * feat: Enhance scenario loading to support orchestration.yaml naming convention * feat: Implement scenario-based handoff map resolution for orchestrator configuration * cicd test for azd deploy * feat: Update audio handling and documentation dependencies for improved installation and error handling * feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management * feat: Revise documentation deployment workflow to enhance dependency management and streamline build process * modified docs workflow * feat: Add site_dir configuration to mkdocs.yml for improved site structure * feat: Allow mkdocs build to proceed with warnings by removing --strict flag * fix: Update health check endpoint in postprovision script to use correct API path * refactor: Remove outdated AZD deployment workflow and update documentation links for clarity * fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly * refactor: Simplify state key handling in provider configuration by using environment name * fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults * fix: Use coalesce function for location assignment in storage account resource * refactor: Remove unused backend API public URL variable and related validation * refactor: Remove unused backend API public URL and source phone number from environment parameter files * improvements flow * fix: Implement auto-selection and timeout for user input in setup scripts * add value * fix: Update naming conventions for foundry account and project variables in locals * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update documentation URLs to reflect new repository location * feat: Enhance API documentation and tagging for better clarity and organization * docs: Update documentation links and improve clarity across various guides * refactor: replace deploy-azd workflow with reusable template and remove redundant summary job - Updated the deployment workflow name to "Deploy to Azure". - Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml. - Removed the deployment summary job and its associated steps to streamline the workflow. * fix: Add run-name to the Azure deployment workflow for better clarity * fix: Update condition for output extraction in deployment workflow * fix: Update GitHub token to use secrets for enhanced security * feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment * adding rg as env var set at the gh env level * fix: Add emoji to workflow names for better visibility * feat: Update documentation workflow name and enhance README with deployment badges * fix: Update README layout and enhance navigation links for better user experience * fix: Restore header for ARTVoice Accelerator Framework in README * add value * fix: Update README layout for improved clarity and navigation * Enhance provisioning scripts and documentation - Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal. - Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning. - Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms. - Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration. - Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts. - Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions. * feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements * fix(ci): simplify test-azd-hooks workflow tests and run in parallel - Remove fragile grep-based function extraction that caused syntax errors - Run lint, linux, macos, windows tests in parallel (no dependencies) - Trigger on all pushes to main/staging (remove path filters for push) - Simplify backend configuration test to avoid function sourcing issues * feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration * feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow * feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup * feat: Update test job names for clarity and enhance preflight checks for CI mode * feat: Update preflight checks to conditionally include Docker in CI mode and log its status * feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting * feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks * feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock * feat: Update Azure deployment workflows and normalize container memory formats * feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options * feat: Remove outdated troubleshooting documentation for deployment issues * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/workflows/test-azd-hooks.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: Implement TTS Streaming Latency Analysis and Optimization Plan - Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture. - Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming. - Proposed a multi-phase optimization strategy to address identified issues, including: - Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task. - Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch. - Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming. - Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing. - Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming. - Created a detailed test implementation plan with metrics and success criteria to validate improvements. test: Add unit tests for HandoffService - Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods. - Implemented tests for greeting selection and context building to ensure proper functionality. - Added tests for the HandoffResolution dataclass to verify properties and default values. * feat: Add Scenario Builder component and integrate with RealTimeVoiceApp - Introduced ScenarioBuilder component for visual orchestration of agent flows. - Implemented drag-and-drop functionality for agents and handoff configuration. - Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders. - Enhanced state management for agent scenarios, including creation and updates. - Integrated new handoff editor for configuring agent interactions. * Refactor code structure for improved readability and maintainability * Add error handling for Redis connection issues and implement unit tests for HandoffService - Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts. - Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building. - Added tests for HandoffResolution dataclass to ensure correct property behavior and default values. * Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts * Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates. * Refactor environment variable assignment in deploy workflow for clarity * Refactor tests and dependencies following module renaming and API changes - Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock. - Updated conftest.py to mock configuration and Azure OpenAI client for tests. - Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules. - Adjusted imports in test_artagent_wshelpers.py for orchestrator path change. - Skipped tests in test_call_transfer_service.py due to API changes in toolstore. - Updated datetime usage in test_demo_env_phrase_bias.py to use UTC. - Modified websocket endpoint assertions in test_realtime.py to reflect new paths. - Added new test file test_voice_handler_components.py for voice handler components. * Add comprehensive tests for VoiceLive handler and orchestrator memory management - Implement tests to verify cleanup functionality in LiveOrchestrator. - Ensure proper registration and unregistration of orchestrators in the registry. - Test background task tracking and cleanup mechanisms. - Validate greeting task cancellation during orchestrator cleanup. - Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry. - Verify user message history deque is properly bounded and cleared on cleanup. - Add scenario update tests to ensure correct agent management during updates. - Optimize hot path functions to ensure non-blocking behavior during network calls. * feat: Enhance AgentBuilder with consistent field names and improved UI elements * Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues. * feat: Add predefined handoff condition patterns to enhance scenario orchestration * add value * feat(metrics): Introduce shared metrics factory for lazy initialization - Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics. - Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics. - Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization. - Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling. - Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls. - Introduced utility functions for building common metric attributes, ensuring consistency across metrics. * feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation * fix: Simplify environment determination logic in deployment workflow * add value * feat: Add user flow screenshots and enhance documentation for guided agent setup * feat: Enhance scenario testing instructions for clarity and user guidance * fix: Correct image paths in quickstart guide for accurate rendering * feat: Add initial agent builder and template selection screenshots to quickstart guide * feat: Add demo profile creation steps and related images to quickstart guide * feat: Implement EasyAuth configuration script and integrate into post-provisioning process * refactor: Remove backend IP restrictions configuration and related outputs * Added non qualifying rush response to ensure clear model behavior * updated order so confirmation statement is in the correct spot * add value * add value * chore: Remove unused workflow images for demo profiles * fix: Update demo profile creation images in quickstart guide * fix: Update home screen image in quickstart guide * fix: Update home screen and scenario images in quickstart guide * add value * add value * add value * add value * add value * add * add value * art * add opentelemetry import for tracing support in TTS module * refactor: update LiveOrchestrator to enhance user message history management and improve handoff context * Refactor TTS Playback and Voice Handling - Consolidated TTS playback logic into a unified class for speech cascade. - Removed deprecated VoiceSessionContext and related compatibility shims. - Enhanced error handling during tool initialization and event handler registration. - Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings. - Improved logging for TTS synthesis and streaming processes. - Added new handoff tool registration for dynamic routing. * refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts * refactor: enhance EasyAuth interactive prompts and streamline user choices * refactor: enhance run-name logic for Azure deployment workflow * fix: update environment logic for pull_request events in Azure deployment workflow * refactor: update preprovision hook execution and streamline backend configuration * feat: add context variable support for handoffs and enhance UI for variable mapping * feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Responses API Infrastructure & Dual Model Configuration (#16) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * fix: update project version to 2.0.0-beta in pyproject.toml --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Orchestrator Integration + Optimizations (#17) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: integrate Responses API in orchestrators and add optimizations **Cascade Orchestrator:** - Update model selection to use agent.get_model_for_mode('cascade') - Integrate Responses API routing based on endpoint_preference - Add error handling for unsupported parameters - Extract TTS processing into separate tts_processor module **VoiceLive Orchestrator:** - Update to use agent.get_model_for_mode('voicelive') - Add registry cleanup to prevent unbounded growth - Improve memory management and stale orchestrator cleanup - Extract DTMF processing into separate dtmf_processor module **Tests:** - Add test_cascade_orchestrator_entry_points - Add test_cascade_llm_processing - Add test_dtmf_processor Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Evaluation Framework + Frontend UI (#18) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: add evaluation framework and frontend UI for Responses API **Evaluation Framework:** - Add EventRecorder with git commit SHA tracking - Add API-aware scoring with budget adjustments for verbosity - Add scenario runner for automated testing - Add CLI for running evaluations - Add validate_phases.py for phase-based validation - Add wrappers for endpoint detection **Frontend UI:** - Add cascade_model and voicelive_model selectors in Agent Builder - Add Responses API endpoint preference dropdown - Add conditional fields for verbosity, reasoning_effort, etc. - Update ScenarioBuilder with model configuration options - Display API version fields **Documentation:** - Add docs/testing/model-evaluation.md - Add evaluation playground Jupyter notebook Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] * Enhance logging and user prompts in preflight and pre-provisioning scripts (#20) - Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting. - Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh. - Added color-coded success, warning, and error messages for better visibility. - Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values. - Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration (#21) Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * enhanced the scenariobuilder with flowy (#22) * docs: add comprehensive voice processing architecture documentation Add complete documentation for the voice processing architecture: New Documentation: - docs/architecture/voice/README.md - Comprehensive voice architecture guide * VoiceHandler overview and usage patterns * TTS playback and text processing * Speech cascade pipeline documentation * Audio specifications for browser and ACS transports * Testing guidelines with actual test file references * Troubleshooting guide for common issues - apps/artagent/backend/voice/README.md - Developer quick reference * Directory structure and module organization * Quick start examples * Common tasks and patterns * File location guide * Testing commands Documentation Updates: - docs/mkdocs.yml - Add voice architecture to navigation - docs/operations/troubleshooting.md - Add voice-specific troubleshooting Key Improvements: - Fixed mkdocs formatting for proper list rendering - Updated all test references to match actual test files: * test_voice_handler_components.py * test_voice_handler_compat.py * test_cascade_orchestrator_entry_points.py * test_cascade_llm_processing.py - Verified all script references (quick_test.sh, test_orchestrator.py) - Added prerequisites for running tests with dev dependencies - Included both basic and advanced testing examples All file paths and examples have been verified against the actual codebase. Related to #[TBD] * Add custom styles for Flowy flowchart integration with agent blocks * feat: Enhance output port visibility logic in ScenarioGraphCanvas * feat: Add expandable full prompt view for source agent in HandoffEditorDialog --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Refactor ACS logging and add default orchestration scenario - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. * Refactor ACS logging and add default orchestration scenario (#23) - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Enhance logging functions to use log_plain for consistency and clarity in local development setup script * Disable view toggle buttons for chat/graph/timeline in ConversationControls * Add panning functionality to ScenarioGraphCanvas and reset button * Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes * feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts - Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation. - Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements. - Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints. - Improved metrics naming conventions for clarity in load testing results. * feat: Update Voice Live readiness status to use event envelope format --------- Co-authored-by: Jin Lee <94473824+JinLee794@users.noreply.github.com> Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> Co-authored-by: Anna Quincy <annaquincy@microsoft.com> Co-authored-by: Jin Lee <jinle@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * enhancement: infra docs readme update (#100) * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources and remove invalid ones. - Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check. * Refactor code structure for improved readability and maintainability * feat: Enhance MemoManager with background persistence and lifecycle management - Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis. - Implemented task deduplication to cancel previous persistence tasks when a new one is initiated. - Removed unused auto-refresh functionality and related attributes from MemoManager. - Updated tests to verify new persistence behavior and ensure proper task management. - Enhanced error handling and logging for background persistence operations. * feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization * feat(session): enhance session ID management and URL parameter support - Added `pickSessionIdFromUrl` function to extract session ID from URL parameters. - Updated `getOrCreateSessionId` to allow session ID restoration from URL. - Refactored `setSessionId` for better logging and session management. - Improved `createNewSessionId` to utilize `setSessionId`. docs(api): restructure API documentation for clarity and completeness - Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, …
JinLee794
added a commit
that referenced
this pull request
Jan 26, 2026
…t overview (#40) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * Syncinc to Azure Samples (#95) * Delete samples/labs/dev/leadership_phrases.txt * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources and remove invalid ones. - Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check. * Refactor code structure for improved readability and maintainability * feat: Enhance MemoManager with background persistence and lifecycle management - Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis. - Implemented task deduplication to cancel previous persistence tasks when a new one is initiated. - Removed unused auto-refresh functionality and related attributes from MemoManager. - Updated tests to verify new persistence behavior and ensure proper task management. - Enhanced error handling and logging for background persistence operations. * feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization * feat(session): enhance session ID management and URL parameter support - Added `pickSessionIdFromUrl` function to extract session ID from URL parameters. - Updated `getOrCreateSessionId` to allow session ID restoration from URL. - Refactored `setSessionId` for better logging and session management. - Improved `createNewSessionId` to utilize `setSessionId`. docs(api): restructure API documentation for clarity and completeness - Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, Agent Builder, Demo Environment, and TTS Health. - Added detailed descriptions and examples for each endpoint. - Included new sections for interactive API documentation and WebSocket endpoints. docs(api-reference): update WebSocket message types and endpoint details - Clarified message types for incoming audio data and control messages. - Updated WebSocket endpoint URLs and query parameters for browser conversations and dashboard relay. docs(architecture): refine agent architecture diagrams for clarity - Adjusted diagrams to improve readability and understanding of the agent framework and orchestration. fix(architecture): correct orchestration mode comparison table - Updated ratings for Azure Speech voices and simplicity of setup in the orchestration comparison table. docs(getting-started): add demo guide and enhance onboarding experience - Introduced a new demo guide to facilitate user onboarding and provide structured paths for different user levels. - Enhanced the getting started guide with tips and recommended paths for new users. feat(aoai): implement OpenAI connection warmup to reduce latency - Added `warm_openai_connection` function to pre-establish OpenAI connection and reduce cold-start latency on first call. feat(speech): implement token warmup for Speech API to minimize latency - Added `warm_token` method in `SpeechTokenManager` to pre-fetch tokens during startup, reducing latency on first API call. * feat(healthcare): Implement Nurse Triage Agent with symptom assessment and routing capabilities - Introduced a comprehensive voice agent for healthcare triage. - Added agent configuration and prompt templates for patient interaction. - Developed healthcare tools for patient verification, clinical knowledge search, and symptom urgency assessment. - Integrated routing logic for scheduling appointments and emergency transfers. - Enhanced documentation with demo scenarios and testing instructions. * feat: Implement logging utility and session management - Added a logger utility to manage console logging levels and filtering. - Created session management functions to handle session IDs, including retrieval from URL and session creation. - Developed styles for the frontend components to ensure consistent UI design. - Configured Vite for the frontend build process with proper asset handling and environment variable support. - Introduced scripts for starting the backend and frontend development servers, including Azure Dev Tunnel hosting. * feat: Simplify agent handoff process by refining context management and removing redundant data collection * feat: Enhance agent handoff process by managing conversation history and user context * feat: Enhance message handling by persisting tool calls and results as JSON for conversation continuity * feat: Implement silent handoff protocol across agents to enhance user experience and streamline transitions * feat: Add Azure App Configuration module with RBAC and Key Vault integration - Implemented main resource for Azure App Configuration in Terraform. - Added outputs for App Configuration details including ID, name, and endpoint. - Defined variables for App Configuration module, including identity and Key Vault integration. - Updated main Terraform outputs to include App Configuration details. - Enhanced error handling in Azure OpenAI client for missing endpoint configuration. - Improved Redis manager to handle port configuration with better error messaging. - Updated requirements to include Azure App Configuration SDKs. * first code clean up * enabling oidc * Refactor code structure and remove redundant sections for improved readability and maintainability * add value * add value * feat: Add managed certificate and domain registration modules - Introduced `managed-cert-example.bicep` for example usage of managed certificate deployment. - Created `managed-cert.bicep` to handle App Service Domain registration and managed SSL certificate generation. - Implemented `role-assignment.bicep` for managing role assignments with support for built-in and custom roles. - Added `windows-vm.bicep` for deploying a Windows VM as a jumphost with necessary networking components. - Developed `peer-virtual-networks.bicep` for establishing peering between virtual networks. - Implemented `private-dns-zone.bicep` for creating and linking private DNS zones to virtual networks. - Created `private-endpoint.bicep` for deploying private endpoints with DNS zone integration. - Added `vnet.bicep` for creating virtual networks with associated subnets and network security groups. - Updated `types.bicep` with new types for model deployment, role assignments, and network configurations. - Developed `secret.bicep` for managing secrets in Azure Key Vault. - Created `network.bicep` for orchestrating network resources including virtual networks and subnets. * fix: Update default location parameter in create_storage function for clarity * feat: Extract AZURE_LOCATION from environment-specific tfvars file if not set * feat: Implement location resolution with fallback chain in preprovision script * fix: Update Dockerfile to install runtime dependencies and mitigate vulnerabilities * chore: Update CHANGELOG for version 1.5.0 release and remove changenotes.md; enable remote builds in azure.yaml; enhance terraform initialization script with location prompts * feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity * further deployment cleanup, docs update/tweaks, adding more todos * removing unused dependency in src/herlpers.py * refactor: Update architecture diagram in README for clarity and consistency in orchestration modes * add value * Refactor Terraform configuration: - Update main.tf to adjust foundry account and project naming conventions. - Remove feature flags and keys from appconfig module as they are now managed externally. - Clean up variables.tf by removing unused variables and updating descriptions. - Delete provider configuration file as it is no longer needed. - Change default application name from "rtaudioagent" to "artagent" and adjust related settings. - Modify connection settings and pool sizes for improved performance. * feat: Enhance Azure Voice Live integration and refactor configuration management * last changes * feat: Add app configuration bootstrap to initialize environment variables * Enhance configuration loading with .env.local support and update documentation * fix voicelive output attributes * add * Refactor agent paths and update documentation for agent discovery and configuration * Add Insurance Voice Agent Scenario documentation and update navigation - Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management. - Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section. * Add integration proposal for Spec-Driven Development methodology in ARTVoice * add value * Enhance Terraform configuration and scripts for Voice Live integration - Update Dockerfile to install dependencies and set up virtual environment. - Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling. - Refactor sync-appconfig.sh to streamline key-value imports and feature flag management. - Add provider.conf.json generation for remote state backend configuration. - Update main.tf and outputs.tf to support new Voice Live model deployments. - Introduce voice_live_location and voice_live_model_deployments variables in variables.tf. * feat: Add Concierge agent configuration and prompts for banking scenarios - Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations. - Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes. - Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios. - Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading. - Updated default start agent to BankingConcierge and added validation for agent existence at startup. * feat: Enhance scenario loading to support orchestration.yaml naming convention * feat: Implement scenario-based handoff map resolution for orchestrator configuration * cicd test for azd deploy * feat: Update audio handling and documentation dependencies for improved installation and error handling * feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management * feat: Revise documentation deployment workflow to enhance dependency management and streamline build process * modified docs workflow * feat: Add site_dir configuration to mkdocs.yml for improved site structure * feat: Allow mkdocs build to proceed with warnings by removing --strict flag * fix: Update health check endpoint in postprovision script to use correct API path * refactor: Remove outdated AZD deployment workflow and update documentation links for clarity * fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly * refactor: Simplify state key handling in provider configuration by using environment name * fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults * fix: Use coalesce function for location assignment in storage account resource * refactor: Remove unused backend API public URL variable and related validation * refactor: Remove unused backend API public URL and source phone number from environment parameter files * improvements flow * fix: Implement auto-selection and timeout for user input in setup scripts * add value * fix: Update naming conventions for foundry account and project variables in locals * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update documentation URLs to reflect new repository location * feat: Enhance API documentation and tagging for better clarity and organization * docs: Update documentation links and improve clarity across various guides * refactor: replace deploy-azd workflow with reusable template and remove redundant summary job - Updated the deployment workflow name to "Deploy to Azure". - Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml. - Removed the deployment summary job and its associated steps to streamline the workflow. * fix: Add run-name to the Azure deployment workflow for better clarity * fix: Update condition for output extraction in deployment workflow * fix: Update GitHub token to use secrets for enhanced security * feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment * adding rg as env var set at the gh env level * fix: Add emoji to workflow names for better visibility * feat: Update documentation workflow name and enhance README with deployment badges * fix: Update README layout and enhance navigation links for better user experience * fix: Restore header for ARTVoice Accelerator Framework in README * add value * fix: Update README layout for improved clarity and navigation * Enhance provisioning scripts and documentation - Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal. - Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning. - Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms. - Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration. - Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts. - Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions. * feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements * fix(ci): simplify test-azd-hooks workflow tests and run in parallel - Remove fragile grep-based function extraction that caused syntax errors - Run lint, linux, macos, windows tests in parallel (no dependencies) - Trigger on all pushes to main/staging (remove path filters for push) - Simplify backend configuration test to avoid function sourcing issues * feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration * feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow * feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup * feat: Update test job names for clarity and enhance preflight checks for CI mode * feat: Update preflight checks to conditionally include Docker in CI mode and log its status * feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting * feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks * feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock * feat: Update Azure deployment workflows and normalize container memory formats * feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options * feat: Remove outdated troubleshooting documentation for deployment issues * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/workflows/test-azd-hooks.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: Implement TTS Streaming Latency Analysis and Optimization Plan - Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture. - Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming. - Proposed a multi-phase optimization strategy to address identified issues, including: - Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task. - Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch. - Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming. - Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing. - Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming. - Created a detailed test implementation plan with metrics and success criteria to validate improvements. test: Add unit tests for HandoffService - Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods. - Implemented tests for greeting selection and context building to ensure proper functionality. - Added tests for the HandoffResolution dataclass to verify properties and default values. * feat: Add Scenario Builder component and integrate with RealTimeVoiceApp - Introduced ScenarioBuilder component for visual orchestration of agent flows. - Implemented drag-and-drop functionality for agents and handoff configuration. - Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders. - Enhanced state management for agent scenarios, including creation and updates. - Integrated new handoff editor for configuring agent interactions. * Refactor code structure for improved readability and maintainability * Add error handling for Redis connection issues and implement unit tests for HandoffService - Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts. - Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building. - Added tests for HandoffResolution dataclass to ensure correct property behavior and default values. * Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts * Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates. * Refactor environment variable assignment in deploy workflow for clarity * Refactor tests and dependencies following module renaming and API changes - Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock. - Updated conftest.py to mock configuration and Azure OpenAI client for tests. - Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules. - Adjusted imports in test_artagent_wshelpers.py for orchestrator path change. - Skipped tests in test_call_transfer_service.py due to API changes in toolstore. - Updated datetime usage in test_demo_env_phrase_bias.py to use UTC. - Modified websocket endpoint assertions in test_realtime.py to reflect new paths. - Added new test file test_voice_handler_components.py for voice handler components. * Add comprehensive tests for VoiceLive handler and orchestrator memory management - Implement tests to verify cleanup functionality in LiveOrchestrator. - Ensure proper registration and unregistration of orchestrators in the registry. - Test background task tracking and cleanup mechanisms. - Validate greeting task cancellation during orchestrator cleanup. - Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry. - Verify user message history deque is properly bounded and cleared on cleanup. - Add scenario update tests to ensure correct agent management during updates. - Optimize hot path functions to ensure non-blocking behavior during network calls. * feat: Enhance AgentBuilder with consistent field names and improved UI elements * Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues. * feat: Add predefined handoff condition patterns to enhance scenario orchestration * add value * feat(metrics): Introduce shared metrics factory for lazy initialization - Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics. - Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics. - Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization. - Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling. - Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls. - Introduced utility functions for building common metric attributes, ensuring consistency across metrics. * feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation * fix: Simplify environment determination logic in deployment workflow * add value * feat: Add user flow screenshots and enhance documentation for guided agent setup * feat: Enhance scenario testing instructions for clarity and user guidance * fix: Correct image paths in quickstart guide for accurate rendering * feat: Add initial agent builder and template selection screenshots to quickstart guide * feat: Add demo profile creation steps and related images to quickstart guide * feat: Implement EasyAuth configuration script and integrate into post-provisioning process * refactor: Remove backend IP restrictions configuration and related outputs * Added non qualifying rush response to ensure clear model behavior * updated order so confirmation statement is in the correct spot * add value * add value * chore: Remove unused workflow images for demo profiles * fix: Update demo profile creation images in quickstart guide * fix: Update home screen image in quickstart guide * fix: Update home screen and scenario images in quickstart guide * add value * add value * add value * add value * add value * add * add value * art * add opentelemetry import for tracing support in TTS module * refactor: update LiveOrchestrator to enhance user message history management and improve handoff context * Refactor TTS Playback and Voice Handling - Consolidated TTS playback logic into a unified class for speech cascade. - Removed deprecated VoiceSessionContext and related compatibility shims. - Enhanced error handling during tool initialization and event handler registration. - Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings. - Improved logging for TTS synthesis and streaming processes. - Added new handoff tool registration for dynamic routing. * refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts * refactor: enhance EasyAuth interactive prompts and streamline user choices * refactor: enhance run-name logic for Azure deployment workflow * fix: update environment logic for pull_request events in Azure deployment workflow * refactor: update preprovision hook execution and streamline backend configuration * feat: add context variable support for handoffs and enhance UI for variable mapping * feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Responses API Infrastructure & Dual Model Configuration (#16) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * fix: update project version to 2.0.0-beta in pyproject.toml --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Orchestrator Integration + Optimizations (#17) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: integrate Responses API in orchestrators and add optimizations **Cascade Orchestrator:** - Update model selection to use agent.get_model_for_mode('cascade') - Integrate Responses API routing based on endpoint_preference - Add error handling for unsupported parameters - Extract TTS processing into separate tts_processor module **VoiceLive Orchestrator:** - Update to use agent.get_model_for_mode('voicelive') - Add registry cleanup to prevent unbounded growth - Improve memory management and stale orchestrator cleanup - Extract DTMF processing into separate dtmf_processor module **Tests:** - Add test_cascade_orchestrator_entry_points - Add test_cascade_llm_processing - Add test_dtmf_processor Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Evaluation Framework + Frontend UI (#18) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: add evaluation framework and frontend UI for Responses API **Evaluation Framework:** - Add EventRecorder with git commit SHA tracking - Add API-aware scoring with budget adjustments for verbosity - Add scenario runner for automated testing - Add CLI for running evaluations - Add validate_phases.py for phase-based validation - Add wrappers for endpoint detection **Frontend UI:** - Add cascade_model and voicelive_model selectors in Agent Builder - Add Responses API endpoint preference dropdown - Add conditional fields for verbosity, reasoning_effort, etc. - Update ScenarioBuilder with model configuration options - Display API version fields **Documentation:** - Add docs/testing/model-evaluation.md - Add evaluation playground Jupyter notebook Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] * Enhance logging and user prompts in preflight and pre-provisioning scripts (#20) - Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting. - Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh. - Added color-coded success, warning, and error messages for better visibility. - Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values. - Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration (#21) Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * enhanced the scenariobuilder with flowy (#22) * docs: add comprehensive voice processing architecture documentation Add complete documentation for the voice processing architecture: New Documentation: - docs/architecture/voice/README.md - Comprehensive voice architecture guide * VoiceHandler overview and usage patterns * TTS playback and text processing * Speech cascade pipeline documentation * Audio specifications for browser and ACS transports * Testing guidelines with actual test file references * Troubleshooting guide for common issues - apps/artagent/backend/voice/README.md - Developer quick reference * Directory structure and module organization * Quick start examples * Common tasks and patterns * File location guide * Testing commands Documentation Updates: - docs/mkdocs.yml - Add voice architecture to navigation - docs/operations/troubleshooting.md - Add voice-specific troubleshooting Key Improvements: - Fixed mkdocs formatting for proper list rendering - Updated all test references to match actual test files: * test_voice_handler_components.py * test_voice_handler_compat.py * test_cascade_orchestrator_entry_points.py * test_cascade_llm_processing.py - Verified all script references (quick_test.sh, test_orchestrator.py) - Added prerequisites for running tests with dev dependencies - Included both basic and advanced testing examples All file paths and examples have been verified against the actual codebase. Related to #[TBD] * Add custom styles for Flowy flowchart integration with agent blocks * feat: Enhance output port visibility logic in ScenarioGraphCanvas * feat: Add expandable full prompt view for source agent in HandoffEditorDialog --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Refactor ACS logging and add default orchestration scenario - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. * Refactor ACS logging and add default orchestration scenario (#23) - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Enhance logging functions to use log_plain for consistency and clarity in local development setup script * Disable view toggle buttons for chat/graph/timeline in ConversationControls * Add panning functionality to ScenarioGraphCanvas and reset button * Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes * feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts - Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation. - Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements. - Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints. - Improved metrics naming conventions for clarity in load testing results. * feat: Update Voice Live readiness status to use event envelope format --------- Co-authored-by: Jin Lee <94473824+JinLee794@users.noreply.github.com> Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> Co-authored-by: Anna Quincy <annaquincy@microsoft.com> Co-authored-by: Jin Lee <jinle@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Consolidate infrastructure resource documentation into infra/README.md (#26) * Initial plan * Add comprehensive infrastructure resources documentation with private networking links Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> * Consolidate infrastructure documentation into infra/README.md Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> * enhancement: infra docs readme update (#100) * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources and remove invalid ones. - Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check. * Refactor code structure for improved readability and maintainability * feat: Enhance MemoManager with background persistence and lifecycle management - Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis. - Implemented task deduplication to cancel previous persistence tasks when a new one is initiated. - Removed unused auto-refresh functionality and related attributes from MemoManager. - Updated tests to verify new persistence behavior and ensure proper task management. - Enhanced error handling and logging for background persistence operations. * feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization * feat(session): enhan…
JinLee794
added a commit
that referenced
this pull request
Jan 26, 2026
…, evaluation framework (#41) * cleanup legacy media handler, minor test and deploy cleanup (#25) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * refactor: remove legacy Redis management classes and related files - Deleted AzureRedisManager, AsyncAzureRedisManager, RedisKeyManager, and associated models. - Removed unused Redis interaction logic to streamline the codebase. - Updated tests to reflect changes in the VoiceHandler module, removing deprecated MediaHandler alias. - Ensured compatibility with the new voice module structure. --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> Co-authored-by: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com> * feat: enhance azd environment variable handling with error checks and… (#29) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * Syncinc to Azure Samples (#95) * Delete samples/labs/dev/leadership_phrases.txt * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources and remove invalid ones. - Improved error logging in text_to_speech.py to include detailed initialization failure information and added is_ready property for synthesizer readiness check. * Refactor code structure for improved readability and maintainability * feat: Enhance MemoManager with background persistence and lifecycle management - Added support for background persistence in MemoManager, allowing non-blocking state saving to Redis. - Implemented task deduplication to cancel previous persistence tasks when a new one is initiated. - Removed unused auto-refresh functionality and related attributes from MemoManager. - Updated tests to verify new persistence behavior and ensure proper task management. - Enhanced error handling and logging for background persistence operations. * feat: Add Connection Warmup Analysis document for Azure Speech & OpenAI optimization * feat(session): enhance session ID management and URL parameter support - Added `pickSessionIdFromUrl` function to extract session ID from URL parameters. - Updated `getOrCreateSessionId` to allow session ID restoration from URL. - Refactored `setSessionId` for better logging and session management. - Improved `createNewSessionId` to utilize `setSessionId`. docs(api): restructure API documentation for clarity and completeness - Organized API endpoints into categories: Health & Monitoring, Call Management, Media Streaming, Browser Conversations, Session Metrics, Agent Builder, Demo Environment, and TTS Health. - Added detailed descriptions and examples for each endpoint. - Included new sections for interactive API documentation and WebSocket endpoints. docs(api-reference): update WebSocket message types and endpoint details - Clarified message types for incoming audio data and control messages. - Updated WebSocket endpoint URLs and query parameters for browser conversations and dashboard relay. docs(architecture): refine agent architecture diagrams for clarity - Adjusted diagrams to improve readability and understanding of the agent framework and orchestration. fix(architecture): correct orchestration mode comparison table - Updated ratings for Azure Speech voices and simplicity of setup in the orchestration comparison table. docs(getting-started): add demo guide and enhance onboarding experience - Introduced a new demo guide to facilitate user onboarding and provide structured paths for different user levels. - Enhanced the getting started guide with tips and recommended paths for new users. feat(aoai): implement OpenAI connection warmup to reduce latency - Added `warm_openai_connection` function to pre-establish OpenAI connection and reduce cold-start latency on first call. feat(speech): implement token warmup for Speech API to minimize latency - Added `warm_token` method in `SpeechTokenManager` to pre-fetch tokens during startup, reducing latency on first API call. * feat(healthcare): Implement Nurse Triage Agent with symptom assessment and routing capabilities - Introduced a comprehensive voice agent for healthcare triage. - Added agent configuration and prompt templates for patient interaction. - Developed healthcare tools for patient verification, clinical knowledge search, and symptom urgency assessment. - Integrated routing logic for scheduling appointments and emergency transfers. - Enhanced documentation with demo scenarios and testing instructions. * feat: Implement logging utility and session management - Added a logger utility to manage console logging levels and filtering. - Created session management functions to handle session IDs, including retrieval from URL and session creation. - Developed styles for the frontend components to ensure consistent UI design. - Configured Vite for the frontend build process with proper asset handling and environment variable support. - Introduced scripts for starting the backend and frontend development servers, including Azure Dev Tunnel hosting. * feat: Simplify agent handoff process by refining context management and removing redundant data collection * feat: Enhance agent handoff process by managing conversation history and user context * feat: Enhance message handling by persisting tool calls and results as JSON for conversation continuity * feat: Implement silent handoff protocol across agents to enhance user experience and streamline transitions * feat: Add Azure App Configuration module with RBAC and Key Vault integration - Implemented main resource for Azure App Configuration in Terraform. - Added outputs for App Configuration details including ID, name, and endpoint. - Defined variables for App Configuration module, including identity and Key Vault integration. - Updated main Terraform outputs to include App Configuration details. - Enhanced error handling in Azure OpenAI client for missing endpoint configuration. - Improved Redis manager to handle port configuration with better error messaging. - Updated requirements to include Azure App Configuration SDKs. * first code clean up * enabling oidc * Refactor code structure and remove redundant sections for improved readability and maintainability * add value * add value * feat: Add managed certificate and domain registration modules - Introduced `managed-cert-example.bicep` for example usage of managed certificate deployment. - Created `managed-cert.bicep` to handle App Service Domain registration and managed SSL certificate generation. - Implemented `role-assignment.bicep` for managing role assignments with support for built-in and custom roles. - Added `windows-vm.bicep` for deploying a Windows VM as a jumphost with necessary networking components. - Developed `peer-virtual-networks.bicep` for establishing peering between virtual networks. - Implemented `private-dns-zone.bicep` for creating and linking private DNS zones to virtual networks. - Created `private-endpoint.bicep` for deploying private endpoints with DNS zone integration. - Added `vnet.bicep` for creating virtual networks with associated subnets and network security groups. - Updated `types.bicep` with new types for model deployment, role assignments, and network configurations. - Developed `secret.bicep` for managing secrets in Azure Key Vault. - Created `network.bicep` for orchestrating network resources including virtual networks and subnets. * fix: Update default location parameter in create_storage function for clarity * feat: Extract AZURE_LOCATION from environment-specific tfvars file if not set * feat: Implement location resolution with fallback chain in preprovision script * fix: Update Dockerfile to install runtime dependencies and mitigate vulnerabilities * chore: Update CHANGELOG for version 1.5.0 release and remove changenotes.md; enable remote builds in azure.yaml; enhance terraform initialization script with location prompts * feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity * further deployment cleanup, docs update/tweaks, adding more todos * removing unused dependency in src/herlpers.py * refactor: Update architecture diagram in README for clarity and consistency in orchestration modes * add value * Refactor Terraform configuration: - Update main.tf to adjust foundry account and project naming conventions. - Remove feature flags and keys from appconfig module as they are now managed externally. - Clean up variables.tf by removing unused variables and updating descriptions. - Delete provider configuration file as it is no longer needed. - Change default application name from "rtaudioagent" to "artagent" and adjust related settings. - Modify connection settings and pool sizes for improved performance. * feat: Enhance Azure Voice Live integration and refactor configuration management * last changes * feat: Add app configuration bootstrap to initialize environment variables * Enhance configuration loading with .env.local support and update documentation * fix voicelive output attributes * add * Refactor agent paths and update documentation for agent discovery and configuration * Add Insurance Voice Agent Scenario documentation and update navigation - Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management. - Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section. * Add integration proposal for Spec-Driven Development methodology in ARTVoice * add value * Enhance Terraform configuration and scripts for Voice Live integration - Update Dockerfile to install dependencies and set up virtual environment. - Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling. - Refactor sync-appconfig.sh to streamline key-value imports and feature flag management. - Add provider.conf.json generation for remote state backend configuration. - Update main.tf and outputs.tf to support new Voice Live model deployments. - Introduce voice_live_location and voice_live_model_deployments variables in variables.tf. * feat: Add Concierge agent configuration and prompts for banking scenarios - Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations. - Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes. - Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios. - Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading. - Updated default start agent to BankingConcierge and added validation for agent existence at startup. * feat: Enhance scenario loading to support orchestration.yaml naming convention * feat: Implement scenario-based handoff map resolution for orchestrator configuration * cicd test for azd deploy * feat: Update audio handling and documentation dependencies for improved installation and error handling * feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management * feat: Revise documentation deployment workflow to enhance dependency management and streamline build process * modified docs workflow * feat: Add site_dir configuration to mkdocs.yml for improved site structure * feat: Allow mkdocs build to proceed with warnings by removing --strict flag * fix: Update health check endpoint in postprovision script to use correct API path * refactor: Remove outdated AZD deployment workflow and update documentation links for clarity * fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly * refactor: Simplify state key handling in provider configuration by using environment name * fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults * fix: Use coalesce function for location assignment in storage account resource * refactor: Remove unused backend API public URL variable and related validation * refactor: Remove unused backend API public URL and source phone number from environment parameter files * improvements flow * fix: Implement auto-selection and timeout for user input in setup scripts * add value * fix: Update naming conventions for foundry account and project variables in locals * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update name from rtaudioagent to artaudioagent in environment parameter files * fix: Update documentation URLs to reflect new repository location * feat: Enhance API documentation and tagging for better clarity and organization * docs: Update documentation links and improve clarity across various guides * refactor: replace deploy-azd workflow with reusable template and remove redundant summary job - Updated the deployment workflow name to "Deploy to Azure". - Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml. - Removed the deployment summary job and its associated steps to streamline the workflow. * fix: Add run-name to the Azure deployment workflow for better clarity * fix: Update condition for output extraction in deployment workflow * fix: Update GitHub token to use secrets for enhanced security * feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment * adding rg as env var set at the gh env level * fix: Add emoji to workflow names for better visibility * feat: Update documentation workflow name and enhance README with deployment badges * fix: Update README layout and enhance navigation links for better user experience * fix: Restore header for ARTVoice Accelerator Framework in README * add value * fix: Update README layout for improved clarity and navigation * Enhance provisioning scripts and documentation - Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal. - Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning. - Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms. - Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration. - Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts. - Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions. * feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements * fix(ci): simplify test-azd-hooks workflow tests and run in parallel - Remove fragile grep-based function extraction that caused syntax errors - Run lint, linux, macos, windows tests in parallel (no dependencies) - Trigger on all pushes to main/staging (remove path filters for push) - Simplify backend configuration test to avoid function sourcing issues * feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration * feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow * feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup * feat: Update test job names for clarity and enhance preflight checks for CI mode * feat: Update preflight checks to conditionally include Docker in CI mode and log its status * feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting * feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks * feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock * feat: Update Azure deployment workflows and normalize container memory formats * feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options * feat: Remove outdated troubleshooting documentation for deployment issues * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/workflows/test-azd-hooks.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: Implement TTS Streaming Latency Analysis and Optimization Plan - Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture. - Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming. - Proposed a multi-phase optimization strategy to address identified issues, including: - Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task. - Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch. - Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming. - Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing. - Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming. - Created a detailed test implementation plan with metrics and success criteria to validate improvements. test: Add unit tests for HandoffService - Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods. - Implemented tests for greeting selection and context building to ensure proper functionality. - Added tests for the HandoffResolution dataclass to verify properties and default values. * feat: Add Scenario Builder component and integrate with RealTimeVoiceApp - Introduced ScenarioBuilder component for visual orchestration of agent flows. - Implemented drag-and-drop functionality for agents and handoff configuration. - Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders. - Enhanced state management for agent scenarios, including creation and updates. - Integrated new handoff editor for configuring agent interactions. * Refactor code structure for improved readability and maintainability * Add error handling for Redis connection issues and implement unit tests for HandoffService - Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts. - Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building. - Added tests for HandoffResolution dataclass to ensure correct property behavior and default values. * Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts * Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates. * Refactor environment variable assignment in deploy workflow for clarity * Refactor tests and dependencies following module renaming and API changes - Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock. - Updated conftest.py to mock configuration and Azure OpenAI client for tests. - Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules. - Adjusted imports in test_artagent_wshelpers.py for orchestrator path change. - Skipped tests in test_call_transfer_service.py due to API changes in toolstore. - Updated datetime usage in test_demo_env_phrase_bias.py to use UTC. - Modified websocket endpoint assertions in test_realtime.py to reflect new paths. - Added new test file test_voice_handler_components.py for voice handler components. * Add comprehensive tests for VoiceLive handler and orchestrator memory management - Implement tests to verify cleanup functionality in LiveOrchestrator. - Ensure proper registration and unregistration of orchestrators in the registry. - Test background task tracking and cleanup mechanisms. - Validate greeting task cancellation during orchestrator cleanup. - Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry. - Verify user message history deque is properly bounded and cleared on cleanup. - Add scenario update tests to ensure correct agent management during updates. - Optimize hot path functions to ensure non-blocking behavior during network calls. * feat: Enhance AgentBuilder with consistent field names and improved UI elements * Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues. * feat: Add predefined handoff condition patterns to enhance scenario orchestration * add value * feat(metrics): Introduce shared metrics factory for lazy initialization - Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics. - Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics. - Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization. - Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling. - Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls. - Introduced utility functions for building common metric attributes, ensuring consistency across metrics. * feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation * fix: Simplify environment determination logic in deployment workflow * add value * feat: Add user flow screenshots and enhance documentation for guided agent setup * feat: Enhance scenario testing instructions for clarity and user guidance * fix: Correct image paths in quickstart guide for accurate rendering * feat: Add initial agent builder and template selection screenshots to quickstart guide * feat: Add demo profile creation steps and related images to quickstart guide * feat: Implement EasyAuth configuration script and integrate into post-provisioning process * refactor: Remove backend IP restrictions configuration and related outputs * Added non qualifying rush response to ensure clear model behavior * updated order so confirmation statement is in the correct spot * add value * add value * chore: Remove unused workflow images for demo profiles * fix: Update demo profile creation images in quickstart guide * fix: Update home screen image in quickstart guide * fix: Update home screen and scenario images in quickstart guide * add value * add value * add value * add value * add value * add * add value * art * add opentelemetry import for tracing support in TTS module * refactor: update LiveOrchestrator to enhance user message history management and improve handoff context * Refactor TTS Playback and Voice Handling - Consolidated TTS playback logic into a unified class for speech cascade. - Removed deprecated VoiceSessionContext and related compatibility shims. - Enhanced error handling during tool initialization and event handler registration. - Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings. - Improved logging for TTS synthesis and streaming processes. - Added new handoff tool registration for dynamic routing. * refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts * refactor: enhance EasyAuth interactive prompts and streamline user choices * refactor: enhance run-name logic for Azure deployment workflow * fix: update environment logic for pull_request events in Azure deployment workflow * refactor: update preprovision hook execution and streamline backend configuration * feat: add context variable support for handoffs and enhance UI for variable mapping * feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12) Infrastructure Changes: - Delete 6 obsolete latency_tool implementations (~2200 lines) - Install SessionContextSpanProcessor for automatic session correlation - Replace LatencyTool with @trace_speech decorators in legacy paths - Remove latency_tool field from VoiceSessionContext Speech Services & Dependencies: - Add @trace_speech for STT partial/final transcripts with attributes - Add TTS attributes: voice, output_format, language, audio_size_bytes - Standardize ACS and Redis span attributes with OTel conventions - Add voice_session root SERVER span in media/browser endpoints Orchestrator & Token Tracking: - Add tool execution and agent handoff observability spans - Fix token tracking to use actual API usage data (not estimates) - Update Azure OpenAI API to 2024-10-01-preview - Add session metadata timestamps to MemoManager Benefits: - Single source of truth (ConversationTurnSpan + OTel) - Complete E2E traces in Application Insights - Accurate cost tracking and token visibility - ~2300 lines of dead code removed Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Responses API Infrastructure & Dual Model Configuration (#16) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * fix: update project version to 2.0.0-beta in pyproject.toml --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Orchestrator Integration + Optimizations (#17) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: integrate Responses API in orchestrators and add optimizations **Cascade Orchestrator:** - Update model selection to use agent.get_model_for_mode('cascade') - Integrate Responses API routing based on endpoint_preference - Add error handling for unsupported parameters - Extract TTS processing into separate tts_processor module **VoiceLive Orchestrator:** - Update to use agent.get_model_for_mode('voicelive') - Add registry cleanup to prevent unbounded growth - Improve memory management and stale orchestrator cleanup - Extract DTMF processing into separate dtmf_processor module **Tests:** - Add test_cascade_orchestrator_entry_points - Add test_cascade_llm_processing - Add test_dtmf_processor Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: Evaluation Framework + Frontend UI (#18) * feat: enhance azd environment variable handling with error checks and local state support * fix: update foundry account and project naming conventions for consistency * feat: add Responses API infrastructure and dual model configuration **Infrastructure Changes:** - Add UnifiedResponse dataclass for dual endpoint support - Implement _should_use_responses_endpoint() routing logic - Add _prepare_responses_params() and _prepare_chat_params() methods - Update generate_response() to route between /chat/completions and /responses **Model Configuration:** - Add cascade_model and voicelive_model fields to AgentConfig - Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases - Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens - Update ModelConfigSchema in agent_builder API **Tests:** - Add test_generate_response_respects_responses_config - Add test_generate_response_respects_chat_config - Add TestUnifiedAgentGetModelForMode test suite This PR provides the foundation for Responses API support without changing orchestrator behavior. * feat: add evaluation framework and frontend UI for Responses API **Evaluation Framework:** - Add EventRecorder with git commit SHA tracking - Add API-aware scoring with budget adjustments for verbosity - Add scenario runner for automated testing - Add CLI for running evaluations - Add validate_phases.py for phase-based validation - Add wrappers for endpoint detection **Frontend UI:** - Add cascade_model and voicelive_model selectors in Agent Builder - Add Responses API endpoint preference dropdown - Add conditional fields for verbosity, reasoning_effort, etc. - Update ScenarioBuilder with model configuration options - Display API version fields **Documentation:** - Add docs/testing/model-evaluation.md - Add evaluation playground Jupyter notebook Depends on: PR #1 (Responses API Infrastructure) --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19) Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] * Enhance logging and user prompts in preflight and pre-provisioning scripts (#20) - Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting. - Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh. - Added color-coded success, warning, and error messages for better visibility. - Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values. - Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * feat: voice handler refactoring and MediaHandler migration (#21) Major refactoring of voice processing architecture: Core Voice Changes: - Implement new VoiceHandler as primary entry point for voice sessions - Delete deprecated speech_cascade/tts.py (652 lines removed) - Consolidate TTS functionality into voice/tts/playback.py - Enhance CascadeOrchestrator with improved turn management - Add VoiceSessionContext for clean dependency injection API & Integration: - Migrate /api/v1/browser/conversation to VoiceHandler - Migrate /api/v1/media/stream to VoiceHandler - Create MediaHandler→VoiceHandler compatibility alias - Update media_handler.py for backward compatibility Infrastructure: - Improve telemetry with Azure-style span naming - Enhance ACS helpers with better session management - Update session terminator for lifecycle management - Add orchestration improvements for unified agents Configuration & Samples: - Update auth agent and insurance scenario configs - Add handoff tool enhancements with context variables - Update gpt_flow sample for new patterns Frontend: - Refactor App.jsx for improved voice handling UI Testing & Documentation: - Add test_voice_handler_compat.py for backward compatibility - Add MEDIAHANDLER_MIGRATION.md tracking document This change maintains full backward compatibility while establishing the foundation for cleaner voice processing patterns going forward. Closes #[TBD] Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * enhanced the scenariobuilder with flowy (#22) * docs: add comprehensive voice processing architecture documentation Add complete documentation for the voice processing architecture: New Documentation: - docs/architecture/voice/README.md - Comprehensive voice architecture guide * VoiceHandler overview and usage patterns * TTS playback and text processing * Speech cascade pipeline documentation * Audio specifications for browser and ACS transports * Testing guidelines with actual test file references * Troubleshooting guide for common issues - apps/artagent/backend/voice/README.md - Developer quick reference * Directory structure and module organization * Quick start examples * Common tasks and patterns * File location guide * Testing commands Documentation Updates: - docs/mkdocs.yml - Add voice architecture to navigation - docs/operations/troubleshooting.md - Add voice-specific troubleshooting Key Improvements: - Fixed mkdocs formatting for proper list rendering - Updated all test references to match actual test files: * test_voice_handler_components.py * test_voice_handler_compat.py * test_cascade_orchestrator_entry_points.py * test_cascade_llm_processing.py - Verified all script references (quick_test.sh, test_orchestrator.py) - Added prerequisites for running tests with dev dependencies - Included both basic and advanced testing examples All file paths and examples have been verified against the actual codebase. Related to #[TBD] * Add custom styles for Flowy flowchart integration with agent blocks * feat: Enhance output port visibility logic in ScenarioGraphCanvas * feat: Add expandable full prompt view for source agent in HandoffEditorDialog --------- Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Refactor ACS logging and add default orchestration scenario - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. * Refactor ACS logging and add default orchestration scenario (#23) - Removed info-level logging for ACS configuration details to reduce verbosity. - Changed some logging statements to debug level for better log management. - Updated peer.service attribute in telemetry to use "azure-communication-services". - Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations. Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> * Enhance logging functions to use log_plain for consistency and clarity in local development setup script * Disable view toggle buttons for chat/graph/timeline in ConversationControls * Add panning functionality to ScenarioGraphCanvas and reset button * Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes * feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts - Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation. - Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements. - Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints. - Improved metrics naming conventions for clarity in load testing results. * feat: Update Voice Live readiness status to use event envelope format --------- Co-authored-by: Jin Lee <94473824+JinLee794@users.noreply.github.com> Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com> Co-authored-by: Anna Quincy <annaquincy@microsoft.com> Co-authored-by: Jin Lee <jinle@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Consolidate infrastructure resource documentation into infra/README.md (#26) * Initial plan * Add comprehensive infrastructure resources documentation with private networking links Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> * Consolidate infrastructure documentation into infra/README.md Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JinLee794 <94473824+JinLee794@users.noreply.github.com> * enhancement: infra docs readme update (#100) * Update version and SKU name in staging params * Change version for text-embedding-3-large model Updated the version of the text-embedding-3-large model. * Update main.tfvars.staging.json * Update communication.tf * feat: Enhance status envelope with optional label and update frontend to derive WS URL - Added optional `label` parameter to `make_status_envelope` function in `envelopes.py` to allow custom labels in status messages. - Updated `entrypoint.sh` to derive WebSocket URL from `BACKEND_URL` or use `WS_URL` if provided, replacing placeholders in frontend assets. - Upgraded `js-yaml` and `vite` dependencies in `package.json` and `package-lock.json`. - Enhanced `App.jsx` to format event type labels and summarize event data for better user experience. - Introduced new demo scenarios in `DemoScenariosWidget.jsx` to showcase Microsoft Copilot Studio integration and ACS call routing. - Added tests for call transfer events in `test_acs_events_handlers.py` to ensure correct envelope broadcasting for transfer accepted and failed events. - Created a new Jupyter notebook for custom speech model demonstration in `12-custom-speech-model.ipynb`. - Updated Terraform parameters to include a new text embedding model in `main.tfvars.dev.json`. * refactor: Comment out unused email communication service domain resource * refactor: Comment out unused Azure email communication service resources * feat: Enhance event handling and UI components - Added new utility functions for formatting event types and summarizing event data in App.jsx. - Improved ChatBubble component to display event messages with formatted labels and timestamps. - Updated DemoScenariosWidget to include new scenarios and enhanced filtering options based on tags. - Introduced websocket URL derivation in postprovision.sh for better backend integration. - Added tests for call transfer events in test_acs_events_handlers.py to ensure proper envelope broadcasting. - Updated package.json to include js-yaml and upgraded vite version. * add value * feat: Enhance distributed session handling and improve PayPal agent interactions - Implement distributed session bus using Redis for cross-replica session routing in connection manager. - Add methods for publishing session envelopes to Redis channels. - Introduce confirmation context for call center transfers to ensure explicit user consent. - Update PayPal agent templates to clarify authentication and routing guidelines. - Enhance real-time voice app to manage relay WebSocket connections and handle session updates more effectively. - Improve error handling and logging for distributed session delivery and Redis interactions. - Refactor session envelope handling in frontend to accommodate new event types and improve user experience. * feat: Enhance status tone metadata and improve chat bubble styling * feat: Implement background task handling for MFA delivery and improve greeting messages in handoff processes * feat: Enhance call escalation process with detailed transfer context and improve PayPal agent handoff scenarios * feat: Implement retry mechanism for browser session ID resolution in media streaming * feat: Enhance session management and greeting handling across various components * fixing session mapping for acs calls * add value * add value * adding test file * Adding agents and templates for credit card recommendation and fee dispute agents * add value * Enhance audio transcription settings across agents and adjust logging levels for better debugging * Enhance audio transcription settings across agents and adjust logging levels for better debugging * add value * add value * Implement Azure Voice Live service integration and enhance Terraform configurations for voice model deployments * add value * Add Azure Voice Live model configuration and outputs * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * fixing voicelive chat sequence on the ui * remove sensitive contact information and unused transfer agency client data * feat: Introduce Agent Consolidation Plan with YAML-driven architecture - Added a comprehensive proposal for consolidating agent architecture in `apps/rtagent/backend/src/agents/`. - Established key goals including single source of truth for agent definitions, auto-discovery, and unified tool registry. - Analyzed current architecture and identified pain points such as manual handoff registration and duplicate tool registries. - Proposed a new solution architecture featuring enhanced YAML schema, auto-discovery engine, and unified tool registry. - Detailed implementation roadmap divided into phases for gradual migration and integration. - Included backward compatibility strategy to ensure existing agents function without modification. - Provided extensive documentation on YAML schema, CLI tool usage, and migration checklist. * Refactor speech cascade handler and routing for browser communication - Updated speech cascade handler to prioritize `on_greeting` callback over `on_tts_request` for greeting events. - Added `queue_user_text` method to `SpeechCascadeHandler` for queuing user text input. - Changed routing from `/realtime` to `/browser` for browser communication endpoints. - Modified orchestration logic to ensure TTS responses are sent with blocking behavior to prevent overlap. - Introduced WebSocket helper functions for better organization and clarity in messaging. - Enhanced connection manager to handle Redis pubsub reconnections on credential expiration. - Updated frontend components to reflect routing changes for browser communication. - Adjusted tests to align with the new browser routing and functionality. - Commented out live metrics enabling condition in telemetry configuration for future consideration. * feat(telemetry): add decorators for tracing LLM, dependency, speech, and ACS calls - Introduced , , , and decorators for OpenTelemetry instrumentation. - Implemented context manager for tracking conversation turns with detailed metrics. - Added helper functions for recording GenAI and speech metrics. - Enhanced span attributes for Azure Application Insights visualization. * Remove telemetry configuration module (telemetry_config_v2.py) to streamline codebase and eliminate unused functionality. * feat: Enhance telemetry and tracing for CosmosDB and latency tool - Added OpenTelemetry tracing to CosmosDB operations with a decorator for latency tracking. - Integrated tracing spans in the LatencyTool for better observability in Application Insights. - Updated telemetry configuration to suppress noisy logs and added new attributes for speech cascade metrics. - Created unit tests for SessionAgentManager, covering configuration management, override resolution, handoff management, and persistence. - Removed outdated endpoints review document. * feat: Add useBackendHealth hook for backend health checks and integrate with readiness, agents, and health endpoints test: Implement integration tests for VoiceLive Session Agent Manager, covering agent resolution, handoff mapping, and runtime modifications * WARNING!!!! MAJOR REFACTOR COMMIT - Removed the VoiceLive SDK integration module from the backend. - Added a new AgentTopologyPanel component to the frontend for displaying agent inventory and connections. - Integrated the AgentTopologyPanel into the main application layout. - Updated the BackendIndicator to include agent count and selection functionality. - Enhanced the ConversationControls with a fixed view switcher for better accessibility. - Improved the useBackendHealth hook to handle various agent data structures. - Updated styles for better responsiveness and visual consistency across components. - Modified utility functions to format agent inventory data correctly. - Adjusted import paths in orchestrators and tests to reflect the new backend structure. * feat: Enhance agent handoff process and response handling; refactor UI components for improved usability * feat: Update change notes for v2/speech-orchestration-and-monitoring branch; highlight major features, improvements, and new agents * refactor: Remove Unified Agent Configuration Module; streamline agent management and improve code organization * feat: Enhance ProfileDetailsPanel with resizable functionality and UI improvements - Added resizable panel feature to ProfileDetailsPanel, allowing users to adjust width dynamically. - Updated panel styling for improved aesthetics, including a gradient background and adjusted borders. - Enhanced scrollbar visibility and overflow handling for better user experience. refactor: Simplify GraphListView filter logic - Removed default selection logic for filters in GraphListView, allowing users to start with no filters applied. - Cleaned up useEffect dependencies for better performance and clarity. docs: Introduce Backend Voice & Agents Architecture documentation - Added comprehensive documentation outlining the architecture of backend voice and agent modules. - Detailed separation of concerns between voice transport and agent business logic. - Included data flow diagrams and module responsibilities for clarity. docs: Create Handoff Logic Inventory for better understanding of handoff processes - Documented the handoff logic across backend voice and agent modules. - Established a single source of truth for handoff mappings and protocols. - Summarized cleanup phases and their impact on the codebase. fix: Update logging to safely handle span names - Modified TraceLogFilter to safely retrieve span names, preventing attribute errors with NonRecordingSpan. fix: Adjust telemetry configuration to capture all loggers - Changed logger_name default to an empty string in TelemetryConfig to capture all loggers. * feat: Implement context-aware greeting rendering in VoiceLive agent; enhance session management and logging * feat: Refactor agent configuration and voice handling; streamline agent switching and TTS integration * feat: Enhance Agent Details Panel and Session Management - Added sessionAgentConfig prop to AgentDetailsPanel for dynamic agent configuration display. - Implemented logic to show agent name, description, tools, and model/voice details based on session configuration. - Introduced a new PanelCard in AgentDetailsPanel to display session agent configuration, including model, voice, and prompt preview. - Updated App component to fetch session agent configuration on agent panel visibility and manage agent creation/updating. - Added validation for TTS client initialization in dedicated_tts_pool.py to ensure clients are ready before use. - Enhanced on_demand_pool.py to validate cached resources an…
JinLee794
pushed a commit
that referenced
this pull request
Jan 26, 2026
commit ab8c62bd00ba036d03836dd76c4d8a56de5a7f78
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Jan 26 14:29:37 2026 -0600
Refactor code structure for improved readability and maintainability
commit 2fa85353436f03d1ca7cca191c5d8f007bcf8417
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Jan 26 14:03:37 2026 -0600
feat: Add original_name field to AgentInfo and enhance scenario builder for custom agent visibility
commit 039f84839b7a85e9ffb1b7966c246b94cab210c4
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Wed Jan 7 12:15:31 2026 -0600
refactor: remove legacy Redis management classes and related files
- Deleted AzureRedisManager, AsyncAzureRedisManager, RedisKeyManager, and associated models.
- Removed unused Redis interaction logic to streamline the codebase.
- Updated tests to reflect changes in the VoiceHandler module, removing deprecated MediaHandler alias.
- Ensured compatibility with the new voice module structure.
commit 496947d0d7f309f6317bd31b39dfebf7a222620d
Merge: a7ccd8cb 1900437c
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Mon Jan 5 09:55:05 2026 +0000
Merge pull request #24 from AIappsGBBFactory/staging
commit 1900437cd11ba91bdb23d13623362610e755fbf2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 16:59:08 2026 -0600
feat: Update Voice Live readiness status to use event envelope format
commit ce643c293e774ac0290e7a5f6a68735043440989
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 16:54:03 2026 -0600
feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts
- Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation.
- Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements.
- Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints.
- Improved metrics naming conventions for clarity in load testing results.
commit 34997aa57606aa9783bceb49fbb71aaa94150546
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 12:33:47 2026 -0600
Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes
commit 5a9e0544e15752a5c47a7eb0b963a079b2d7a3b6
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:39:48 2026 -0600
Add panning functionality to ScenarioGraphCanvas and reset button
commit 10fb31d6b451201b80489bb6f31256007f80c6df
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:15:58 2026 -0600
Disable view toggle buttons for chat/graph/timeline in ConversationControls
commit a719fd6696b59caac66334aea0e50e74cff34a21
Merge: 31cf499d f07fdb90
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:13:22 2026 -0600
Merge branch 'feat/voice-handler-refactoring' into staging
commit 31cf499dc690adc4fb23e66b87835d569fe6d725
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:54:48 2026 -0600
Enhance logging functions to use log_plain for consistency and clarity in local development setup script
commit b75fbb7ac70198d142d2d0431e96f5f02b79f5b0
Merge: aaa6fa03 10a9cca9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:54:38 2026 -0600
Merge branch 'staging' of github.com:aiappsgbbfactory/art-voice-agent-accelerator into staging
commit 10a9cca90b11fcdcc776c8a2de24eb58ab99e1f7
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 10:45:16 2026 -0600
Refactor ACS logging and add default orchestration scenario (#23)
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit aaa6fa0363a047ee438fed7aedd03169ee56e586
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:43:31 2026 -0600
Refactor ACS logging and add default orchestration scenario
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
commit 08fca7270cd2ca0afc89c27501c15187b6bc9b9f
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:41 2026 -0600
enhanced the scenariobuilder with flowy (#22)
* docs: add comprehensive voice processing architecture documentation
Add complete documentation for the voice processing architecture:
New Documentation:
- docs/architecture/voice/README.md - Comprehensive voice architecture guide
* VoiceHandler overview and usage patterns
* TTS playback and text processing
* Speech cascade pipeline documentation
* Audio specifications for browser and ACS transports
* Testing guidelines with actual test file references
* Troubleshooting guide for common issues
- apps/artagent/backend/voice/README.md - Developer quick reference
* Directory structure and module organization
* Quick start examples
* Common tasks and patterns
* File location guide
* Testing commands
Documentation Updates:
- docs/mkdocs.yml - Add voice architecture to navigation
- docs/operations/troubleshooting.md - Add voice-specific troubleshooting
Key Improvements:
- Fixed mkdocs formatting for proper list rendering
- Updated all test references to match actual test files:
* test_voice_handler_components.py
* test_voice_handler_compat.py
* test_cascade_orchestrator_entry_points.py
* test_cascade_llm_processing.py
- Verified all script references (quick_test.sh, test_orchestrator.py)
- Added prerequisites for running tests with dev dependencies
- Included both basic and advanced testing examples
All file paths and examples have been verified against the actual codebase.
Related to #[TBD]
* Add custom styles for Flowy flowchart integration with agent blocks
* feat: Enhance output port visibility logic in ScenarioGraphCanvas
* feat: Add expandable full prompt view for source agent in HandoffEditorDialog
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit ed1ee049af18f5b724e0a274140a60fcb8aa82be
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:24 2026 -0600
feat: voice handler refactoring and MediaHandler migration (#21)
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cb4fbc19bb6e2536590d9f55cd261aac39042343
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:10 2026 -0600
Enhance logging and user prompts in preflight and pre-provisioning scripts (#20)
- Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting.
- Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh.
- Added color-coded success, warning, and error messages for better visibility.
- Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values.
- Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit f07fdb9042b2ef67eb8ec390aeda4c01a0666098
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sat Jan 3 22:48:03 2026 -0600
feat: voice handler refactoring and MediaHandler migration
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
commit 52ee0196c5a07cb7ee9a7b3635c0dcbaf3967fdb
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 13:50:35 2026 -0600
Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cf5fd497c19603f44d7272c494e1709d0a63a6a4
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:58:18 2026 -0600
feat: Evaluation Framework + Frontend UI (#18)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: add evaluation framework and frontend UI for Responses API
**Evaluation Framework:**
- Add EventRecorder with git commit SHA tracking
- Add API-aware scoring with budget adjustments for verbosity
- Add scenario runner for automated testing
- Add CLI for running evaluations
- Add validate_phases.py for phase-based validation
- Add wrappers for endpoint detection
**Frontend UI:**
- Add cascade_model and voicelive_model selectors in Agent Builder
- Add Responses API endpoint preference dropdown
- Add conditional fields for verbosity, reasoning_effort, etc.
- Update ScenarioBuilder with model configuration options
- Display API version fields
**Documentation:**
- Add docs/testing/model-evaluation.md
- Add evaluation playground Jupyter notebook
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 72de37f3d441ebd50a7bc81480c8a1e9ec683e61
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:58:07 2026 -0600
feat: Orchestrator Integration + Optimizations (#17)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: integrate Responses API in orchestrators and add optimizations
**Cascade Orchestrator:**
- Update model selection to use agent.get_model_for_mode('cascade')
- Integrate Responses API routing based on endpoint_preference
- Add error handling for unsupported parameters
- Extract TTS processing into separate tts_processor module
**VoiceLive Orchestrator:**
- Update to use agent.get_model_for_mode('voicelive')
- Add registry cleanup to prevent unbounded growth
- Improve memory management and stale orchestrator cleanup
- Extract DTMF processing into separate dtmf_processor module
**Tests:**
- Add test_cascade_orchestrator_entry_points
- Add test_cascade_llm_processing
- Add test_dtmf_processor
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit f9fabce297901ddae5258f99e4abbaa59e7993c8
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:53:36 2026 -0600
feat: Responses API Infrastructure & Dual Model Configuration (#16)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* fix: update project version to 2.0.0-beta in pyproject.toml
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit b19697b7d1aa4e0a24e70d3ebc6b027a664e549e
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:51 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cac95f0098fd2573e5ae6d0b461df9168f791710
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:38 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 2e27b282d6477a6e2d373646751f2c10b1a48e2b
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:25 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 3e00e817e8849a0378a556e9ccc7fa25dc558862
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:15 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit d68f36d7811c3954c2b49ffc9edcee127834a2ad
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 13:20:03 2025 -0600
feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 281267b1a2c2d5bf977e89b89c5b818104e0c2be
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 22 16:49:45 2025 -0600
feat: add context variable support for handoffs and enhance UI for variable mapping
commit d7d5be7b338e3d3491c8af75ee8108d262efdc67
Merge: 8b06eca3 4af93f3b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 22 13:13:33 2025 -0600
Merge branch 'main' into staging
commit 8b06eca322154c670bc23d71e1236e60f16033ed
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:24:11 2025 -0600
refactor: update preprovision hook execution and streamline backend configuration
commit 8b76c48ed45c8c075d914078c67a59b77b2f0e37
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:11:48 2025 -0600
fix: update environment logic for pull_request events in Azure deployment workflow
commit 05b8ad122b4c1fed643925da3af47acdf63ec806
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:10:03 2025 -0600
refactor: enhance run-name logic for Azure deployment workflow
commit 2ce3535278aa697a28da9b3d20c3f7f65358da81
Merge: 614c4e8e 7fea92c7
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 09:53:19 2025 -0600
fix: add peer dependency flag to multiple packages in package-lock.json
commit 614c4e8ec8904a0ea1f52e39474cee1faa15ec77
Merge: 57380059 5b174970
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Fri Dec 19 09:14:13 2025 -0600
Merge branch 'Azure-Samples:staging' into staging
commit 57380059b6e4e52337857c95dd20aae42ca72d32
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 09:07:12 2025 -0600
refactor: enhance EasyAuth interactive prompts and streamline user choices
commit 5b174970bde1dbc3906ac5e65c45e0f7a7d3ca87
Merge: c328369c 36a3de9f
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 18 18:26:27 2025 -0600
Merge pull request #81 from AIappsGBBFactory/staging
commit 36a3de9f2cd8764ea8f17fc8f9c42f65eb6bc478
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 18:20:48 2025 -0600
refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts
commit 1ce8584bb77255efde28f533d703439ed025d714
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 18:12:41 2025 -0600
Refactor TTS Playback and Voice Handling
- Consolidated TTS playback logic into a unified class for speech cascade.
- Removed deprecated VoiceSessionContext and related compatibility shims.
- Enhanced error handling during tool initialization and event handler registration.
- Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings.
- Improved logging for TTS synthesis and streaming processes.
- Added new handoff tool registration for dynamic routing.
commit 0d529bc066458d606a06d1aff2012ec6668c713c
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 17:24:58 2025 -0600
refactor: update LiveOrchestrator to enhance user message history management and improve handoff context
commit 41bf0fb431b27063671616ecc3a49d8c80db487d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 17:01:59 2025 -0600
add opentelemetry import for tracing support in TTS module
commit b51e7f5474e7a4b7f88f3047f45de6903ddf79cb
Merge: 5c2aac51 c328369c
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 18 16:35:25 2025 -0600
Merge branch 'staging' into staging
commit 5c2aac51f5d371f8ac568b9b271dd6909268f118
Merge: d0b279c7 85a8cf76
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 18 16:28:31 2025 -0600
Merge pull request #8 from AIappsGBBFactory/docs/user-flows
Docs/user flows - and adding easyauth as part of postprovision
commit c328369c3945c3ad9150c2cb493e42b8a5f13d8a
Merge: 4af52a50 9671ae49
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 18 16:27:33 2025 -0600
Merge pull request #80 from Azure-Samples/usecases/claims
Usecases/claims
commit 9671ae49e90a40c9ca6cc6fbf00d5386d12c2914
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 16:19:07 2025 -0600
art
commit 7678afad82ec5d4a926ad6dcbd005a12da9eeb0d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:53:19 2025 -0600
add value
commit 8fdd4443d25007421c8618c9f5f95ede84b23f5c
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:47:40 2025 -0600
add
commit 5716f3aa0c822ca0c34f038f485251b3688a242d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:43:02 2025 -0600
add value
commit 540f3f053b279fc2af04f188d54e54017b04bd71
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:39:44 2025 -0600
add value
commit 79ad2f19fe7180bee3777aab44cd3d76658637c8
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:25:33 2025 -0600
add value
commit f929399392f223d27c46cae28199bc3724a9973d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 14:48:50 2025 -0600
add value
commit 313e07f4b6d49560fedc730af0aa08c6f8cf3a55
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 10:59:26 2025 -0600
add value
commit 85a8cf76f2f462e21696d673acd3bbdcc27905e4
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:58:29 2025 -0600
fix: Update home screen and scenario images in quickstart guide
commit 24da693c035b38a6d598dfa04144990f801c32b8
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:55:18 2025 -0600
fix: Update home screen image in quickstart guide
commit 181035146300409b2447a2105dcaded125f3c0ba
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:52:40 2025 -0600
fix: Update demo profile creation images in quickstart guide
commit 524412c307e5b2ee1400067d492a7928efc83645
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:45:28 2025 -0600
chore: Remove unused workflow images for demo profiles
commit 4b99db67087f1185c5f21da1490292baf143dd06
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 00:03:57 2025 -0600
add value
commit b69106aed84f2cb5967827617aac1ffd1d154ad2
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 17 22:21:59 2025 -0600
add value
commit 90793e67edc4af12d55e7143912739561a64e39b
Author: Anna Quincy <annaquincy@microsoft.com>
Date: Wed Dec 17 14:32:34 2025 -0800
updated order so confirmation statement is in the correct spot
commit 9226954539efdf742109140cff8b7b399e680ad8
Author: Anna Quincy <annaquincy@microsoft.com>
Date: Wed Dec 17 14:22:24 2025 -0800
Added non qualifying rush response to ensure clear model behavior
commit c561d6702b8c58698c6a2e1fdb15aa1e1b47d001
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 18:14:56 2025 -0600
refactor: Remove backend IP restrictions configuration and related outputs
commit d0214091d970610479461d80d0bdbb52089c00ff
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 17:49:35 2025 -0600
feat: Implement EasyAuth configuration script and integrate into post-provisioning process
commit 6a02169c371f4bc49c0e2c82190c3a16772c28a2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 16:21:00 2025 -0600
feat: Add demo profile creation steps and related images to quickstart guide
commit 902f0f2fe3db1e0f144860b40da638b3c7388d11
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 16:09:11 2025 -0600
feat: Add initial agent builder and template selection screenshots to quickstart guide
commit a490de102fc16430835181095b2581de64337bf3
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:40:55 2025 -0600
fix: Correct image paths in quickstart guide for accurate rendering
commit 35cb6b91c227425bf63ea27db8125c8ad710e3a9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:35:13 2025 -0600
feat: Enhance scenario testing instructions for clarity and user guidance
commit cbdf736580e466b655c88fcdcb4ae3ac3239e594
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:13:33 2025 -0600
feat: Add user flow screenshots and enhance documentation for guided agent setup
commit 7407971c65741bcbf0b79dfeb527aad4176afa3f
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Tue Dec 16 12:04:38 2025 -0600
add value
commit d0b279c77c6a2db2efdd3404d8f02c94640957b0
Merge: 39cf1e74 ae97dd04
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Tue Dec 16 09:22:33 2025 -0600
Merge pull request #7 from AIappsGBBFactory/feat/scenario-orch
feat: Implement TTS Streaming Latency Analysis and Optimization Plan
commit ae97dd041d6fa5ea346f4b22c19bcd4b52c37e77
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 22:13:06 2025 -0600
fix: Simplify environment determination logic in deployment workflow
commit 83c6c50ffe916bf83549fca563326e19c647bb1a
Merge: ef62ef1c 39cf1e74
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 22:04:24 2025 -0600
Merge branch 'staging' into feat/scenario-orch
commit ef62ef1c75b1bc4c25d58ddfe1701abaa7961a57
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 21:57:58 2025 -0600
feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation
commit 29e7f82828871d57cc6b61fcd0dae2285e850cba
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 20:44:45 2025 -0600
feat(metrics): Introduce shared metrics factory for lazy initialization
- Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics.
- Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics.
- Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization.
- Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling.
- Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls.
- Introduced utility functions for building common metric attributes, ensuring consistency across metrics.
commit 09a1eb40f52a4bfd3c534dff57adccc0b9ad85ef
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Mon Dec 15 20:28:06 2025 -0600
add value
commit 1bdeddfa91755a154f149771e4e8de1cb1740285
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 20:19:51 2025 -0600
feat: Add predefined handoff condition patterns to enhance scenario orchestration
commit 10edd38fb2491878dd25500a1f7448be3ae2b031
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 19:43:11 2025 -0600
Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues.
commit 04db80cb036397d07c184c128b2815ed839ae71e
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 22:48:16 2025 -0600
feat: Enhance AgentBuilder with consistent field names and improved UI elements
commit dfda340c92f544d37f0ce1ec625cf59454132c40
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 22:38:42 2025 -0600
Add comprehensive tests for VoiceLive handler and orchestrator memory management
- Implement tests to verify cleanup functionality in LiveOrchestrator.
- Ensure proper registration and unregistration of orchestrators in the registry.
- Test background task tracking and cleanup mechanisms.
- Validate greeting task cancellation during orchestrator cleanup.
- Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry.
- Verify user message history deque is properly bounded and cleared on cleanup.
- Add scenario update tests to ensure correct agent management during updates.
- Optimize hot path functions to ensure non-blocking behavior during network calls.
commit 4af52a50b2c2af562cca78a53f3fa8109cd97571
Merge: 6b01d803 b8b0c279
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Sun Dec 14 19:06:46 2025 -0600
Merge pull request #79 from AIappsGBBFactory/hotfix/agent-context-and-handoffs
Hotfix/agent context and handoffs
commit b8b0c27948ab46474de405ff826b2caf2480c7d4
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 14:56:55 2025 -0600
Refactor tests and dependencies following module renaming and API changes
- Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock.
- Updated conftest.py to mock configuration and Azure OpenAI client for tests.
- Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules.
- Adjusted imports in test_artagent_wshelpers.py for orchestrator path change.
- Skipped tests in test_call_transfer_service.py due to API changes in toolstore.
- Updated datetime usage in test_demo_env_phrase_bias.py to use UTC.
- Modified websocket endpoint assertions in test_realtime.py to reflect new paths.
- Added new test file test_voice_handler_components.py for voice handler components.
commit 39cf1e74bc7894f3d86a43e847083269b093077e
Merge: 6b01d803 e837e43a
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Dec 14 13:02:22 2025 -0600
Merge pull request #6 from AIappsGBBFactory/hotfix/agent-context-and-handoffs
Hotfix/agent context and handoffs
commit e837e43a3e8720573878da492da293b854c271de
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:46:17 2025 -0600
Refactor environment variable assignment in deploy workflow for clarity
commit 79cb83af0b74817d826c13497d49eeddff79d49c
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:41:49 2025 -0600
Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates.
commit de6557a573bf35ee743433202bba6b71bab8279f
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:30:00 2025 -0600
Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts
commit 6666a9a9f44e2597486b77a5e3a2bc83f655c341
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:29:47 2025 -0600
Add error handling for Redis connection issues and implement unit tests for HandoffService
- Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts.
- Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building.
- Added tests for HandoffResolution dataclass to ensure correct property behavior and default values.
commit 94e39aedabf146d7312d16a4dc2a8f95da92df70
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 11:59:57 2025 -0600
Refactor code structure for improved readability and maintainability
commit 33b8c7d9a6cb2f56bdeb354d1f29c8f49e13177b
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 00:47:20 2025 -0600
feat: Add Scenario Builder component and integrate with RealTimeVoiceApp
- Introduced ScenarioBuilder component for visual orchestration of agent flows.
- Implemented drag-and-drop functionality for agents and handoff configuration.
- Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders.
- Enhanced state management for agent scenarios, including creation and updates.
- Integrated new handoff editor for configuring agent interactions.
commit 1b68215a7db47af4cd82efc873d8a236db8942a5
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 23:55:20 2025 -0600
feat: Implement TTS Streaming Latency Analysis and Optimization Plan
- Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture.
- Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming.
- Proposed a multi-phase optimization strategy to address identified issues, including:
- Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task.
- Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch.
- Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming.
- Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing.
- Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming.
- Created a detailed test implementation plan with metrics and success criteria to validate improvements.
test: Add unit tests for HandoffService
- Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods.
- Implemented tests for greeting selection and context building to ensure proper functionality.
- Added tests for the HandoffResolution dataclass to verify properties and default values.
commit 6b01d8031600614788302ef3003fcb6ab8497a2e
Merge: bb3826f4 a5a1d73e
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Sat Dec 13 14:27:08 2025 -0600
Merge pull request #78 from AIappsGBBFactory/feat/troubleshooting-enhancements
commit a5a1d73e2c833eb253ad72f3ab44353edd1b2e84
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:57:18 2025 -0600
Update .github/workflows/test-azd-hooks.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit a23e18744f594477fc014cf2f73c30215754b654
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:52:27 2025 -0600
Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit aaeae928f9da2ae145cc11f6d58f989931d4c52b
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:51:29 2025 -0600
Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit da6ead6f5b53fca1f32ab8fb339c8d993eb43155
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 13:50:27 2025 -0600
feat: Remove outdated troubleshooting documentation for deployment issues
commit b8a16dc733ba8de4ec424ea8786ebebb2de68472
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 13:41:13 2025 -0600
feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options
commit 7f9f889585584fa3f46be4b7a9e49d301c74e2cf
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 21:31:48 2025 -0600
feat: Update Azure deployment workflows and normalize container memory formats
commit cb6354e7462607c8398acdcef687d21c2b92786f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 20:47:45 2025 -0600
feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock
commit a4a19a33ebef5ad9f85ede7d4fa4486869285863
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 20:27:52 2025 -0600
feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks
commit 5b7a0979166b514eff9109a7e72a04b734851103
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:39:53 2025 -0600
feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting
commit 4d71fa7d39e1825799a3066a821567aa7a916645
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:35:10 2025 -0600
feat: Update preflight checks to conditionally include Docker in CI mode and log its status
commit e1a840e1a3b4cd0d2d825c82d552f8c5abd2674b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:31:06 2025 -0600
feat: Update test job names for clarity and enhance preflight checks for CI mode
commit 40e33ef7f6ba9c9a61ac035f906b3ad8fa8aa815
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:27:24 2025 -0600
feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup
commit 17da7f785413157274c2e8c33e41c559a4eca098
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:21:40 2025 -0600
feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow
commit d54b9ce2e1d0d39f1a8e5bd4e793c78187e12140
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:16:05 2025 -0600
feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration
commit a6d737c73a8e66e2750e0fbec357d7a3445d9bd0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:58:57 2025 -0600
fix(ci): simplify test-azd-hooks workflow tests and run in parallel
- Remove fragile grep-based function extraction that caused syntax errors
- Run lint, linux, macos, windows tests in parallel (no dependencies)
- Trigger on all pushes to main/staging (remove path filters for push)
- Simplify backend configuration test to avoid function sourcing issues
commit 46acb1415e5010a6a422df52470e944f29e4d223
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:56:01 2025 -0600
feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements
commit 819dd90f21444df83f93ad16faf55d5c7f3b972c
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:51:26 2025 -0600
Enhance provisioning scripts and documentation
- Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal.
- Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning.
- Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms.
- Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration.
- Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts.
- Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions.
commit bb3826f445e296cb6960cfdb8937275b2a145e55
Merge: 4b10e18d 516b6b6c
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Fri Dec 12 12:41:50 2025 -0600
Merge pull request #77 from AIappsGBBFactory/staging
docs updates and more pipeline related fixes/enhancements
commit 516b6b6c196ea779dde679e19ae79ec2e1d2f1d0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:39:05 2025 -0600
fix: Update README layout for improved clarity and navigation
commit 4b10e18dc77699a66c821f03005a7db3e7b2a337
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Fri Dec 12 12:37:50 2025 -0600
add value
commit fd2ca5acb094ef361bc19f63f716ae15b80e96b7
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:36:45 2025 -0600
fix: Restore header for ARTVoice Accelerator Framework in README
commit 6c33df1eebd99ce4275fc4528e5ecd9c1414220e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:21:26 2025 -0600
fix: Update README layout and enhance navigation links for better user experience
commit 9167d13735bb18ea38f1468b79cac3f1ebd2096a
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:09:40 2025 -0600
feat: Update documentation workflow name and enhance README with deployment badges
commit e170dd435528384a40a8af269130fd0aeb0f18c1
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:51:53 2025 -0600
fix: Add emoji to workflow names for better visibility
commit c913abefa7149407c4fe18f69733189a2c3f9ba9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:31:13 2025 -0600
adding rg as env var set at the gh env level
commit 7251f57308443dfc0392e74198ee640eef539b05
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:26:23 2025 -0600
feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment
commit 1b077d15d652d1cc3453f1440d8e6917b8a73bb3
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:00:16 2025 -0600
fix: Update GitHub token to use secrets for enhanced security
commit 5391566295a7d967946bc0875a2bc90bd8ecbbaa
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 10:42:42 2025 -0600
fix: Update condition for output extraction in deployment workflow
commit f9cf3c83a35643b2325a23e3091355270a761f04
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 10:12:23 2025 -0600
fix: Add run-name to the Azure deployment workflow for better clarity
commit f051b1387b6daacb84ccbfbb8a54519f5458a99d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 09:51:54 2025 -0600
refactor: replace deploy-azd workflow with reusable template and remove redundant summary job
- Updated the deployment workflow name to "Deploy to Azure".
- Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml.
- Removed the deployment summary job and its associated steps to streamline the workflow.
commit b99084b98a6dba3b4b7a03b897e4bd9950e8b01f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 22:53:40 2025 -0600
docs: Update documentation links and improve clarity across various guides
commit 2c88c9acb23a3c8563f482db7a8ce323a48b11fd
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 22:25:29 2025 -0600
feat: Enhance API documentation and tagging for better clarity and organization
commit db05f3aa81db340548234d88d32a10512fa3b1eb
Merge: 0787f03f 598604c0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:53:03 2025 -0600
Merge branch 'staging' of github.com:aiappsgbbfactory/art-voice-agent-accelerator into staging
commit 0787f03fb9e60f53cc15d88f14233dd900152c9b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:52:32 2025 -0600
fix: Update documentation URLs to reflect new repository location
commit 598604c0de3c1933108d117c0475920ca6a1f886
Merge: b36476ca f8904934
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 11 19:27:17 2025 -0800
Merge pull request #2 from AIappsGBBFactory/v2/deploy-fixes
improvements flow
commit 0ed67a24dfe7aef67b7ad9c937ae5db8a70999f7
Merge: 38d9ca97 f8904934
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 11 21:25:16 2025 -0600
Merge pull request #76 from AIappsGBBFactory/v2/deploy-fixes
commit 38d9ca979e21ea73311806a83ddcd136be4095cd
Merge: b36476ca 6dc26a3e
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 11 19:22:18 2025 -0800
Merge pull request #75 from Azure-Samples/v2-pablosal
V2 pablosal
commit f89049349f864ff2aa6f361f643d6fb54e5d7fce
Merge: 3a2eee5c 6dc26a3e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:18:47 2025 -0600
Merge branch 'v2-pablosal' into v2/deploy-fixes
commit 3a2eee5cbf555628ede657fce81847dd539a1ad2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:18:40 2025 -0600
fix: Update name from rtaudioagent to artaudioagent in environment parameter files
commit 5abe9e52786d9f9bed4b0ba855e4bd8a3785d434
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:49:22 2025 -0600
fix: Update name from rtaudioagent to artaudioagent in environment parameter files
commit 4f7796e2553c8af09ecad3f556b1098e3ee549af
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:42:19 2025 -0600
fix: Update naming conventions for foundry account and project variables in locals
commit 6dc26a3e61f67ab76fac21e7e20ad6cb91f1d27f
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 20:38:23 2025 -0600
add value
commit 56cb1d267d5287519d77b120371b2f829fa378a5
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:08:42 2025 -0600
fix: Implement auto-selection and timeout for user input in setup scripts
commit cc597f3867bced6279a242818d016ad2362aa853
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 19:14:50 2025 -0600
improvements flow
commit b1e0ad34fcedcfcdf6a4224a1e81ca8e35287074
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 19:12:49 2025 -0600
refactor: Remove unused backend API public URL and source phone number from environment parameter files
commit 33602eb894cd1c95fa8f04dee54af0942e374af0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 19:12:44 2025 -0600
refactor: Remove unused backend API public URL variable and related validation
commit 07ac39faee035c1bc77e2e81379e2100ca866023
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 18:56:58 2025 -0600
fix: Use coalesce function for location assignment in storage account resource
commit f1f827a742b8a78ade67de786e6d78394278be04
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 18:43:04 2025 -0600
fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults
commit 82c8bf46df1948c2d5fe05db75d2af19dbddba6f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:32:12 2025 -0600
refactor: Simplify state key handling in provider configuration by using environment name
commit 7b9cd66693b4ae86c53f6039c0b36d9fe0fac00e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:24:29 2025 -0600
fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly
commit ed31e932febda5b8c94809965ffa1158d7198d73
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:20:43 2025 -0600
refactor: Remove outdated AZD deployment workflow and update documentation links for clarity
commit 29925e9ef2e2dd44b94169682c899621e4c2ee1f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:51:58 2025 -0600
fix: Update health check endpoint in postprovision script to use correct API path
commit 0f4a0f99ccf7366d4e473cb18c16fca9052cd597
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:45:03 2025 -0600
feat: Allow mkdocs build to proceed with warnings by removing --strict flag
commit 76c89462941e9f4a5b051736790433fcb05117af
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:42:33 2025 -0600
feat: Add site_dir configuration to mkdocs.yml for improved site structure
commit 34936afc98757908a0deae9d2710bc1a6457703d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:38:46 2025 -0600
modified docs workflow
commit da10e6e258a015bcdba373ae713aa9a8b5377fd6
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:36:06 2025 -0600
feat: Revise documentation deployment workflow to enhance dependency management and streamline build process
commit a64e7fa4ab94ccbeac0fd9a4afb0cf82e77fb8ff
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:30:41 2025 -0600
feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management
commit 200c14026da0deb56bef3813304eea6075091b69
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:15:37 2025 -0600
feat: Update audio handling and documentation dependencies for improved installation and error handling
commit e83c84f23790b3387d5da440e67d84d98ca1960b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:07:12 2025 -0600
cicd test for azd deploy
commit 4eb33fa287cb8bc664118439fdd9902c0658d0d8
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 14:55:59 2025 -0600
feat: Implement scenario-based handoff map resolution for orchestrator configuration
commit f97621578bfe54ab06a5fd50c2772f99acd63834
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 14:29:14 2025 -0600
feat: Enhance scenario loading to support orchestration.yaml naming convention
commit 3203b58f12850d20b06c0b116385d424dd157641
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 13:54:57 2025 -0600
feat: Add Concierge agent configuration and prompts for banking scenarios
- Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations.
- Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes.
- Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios.
- Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading.
- Updated default start agent to BankingConcierge and added validation for agent existence at startup.
commit 3c402dcbcbf36ad2ffcde6799b613d88610a9fd0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 11:34:05 2025 -0600
Enhance Terraform configuration and scripts for Voice Live integration
- Update Dockerfile to install dependencies and set up virtual environment.
- Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling.
- Refactor sync-appconfig.sh to streamline key-value imports and feature flag management.
- Add provider.conf.json generation for remote state backend configuration.
- Update main.tf and outputs.tf to support new Voice Live model deployments.
- Introduce voice_live_location and voice_live_model_deployments variables in variables.tf.
commit 3365e8b4836dbc77b48c5c207c0f093c4518eeb4
Merge: d8a10ddc 313a39d4
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 11 00:38:58 2025 -0600
Merge pull request #73 from AIappsGBBFactory/v2/jinle-more-fixes
Updated documentation based on new scenario-based structure, added proposal for speckit integration
commit d8a10ddc9f00198cff3b03a1d28286c235f519b8
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 00:37:50 2025 -0600
add value
commit 313a39d42078245ee9b0fe983a53066c355b5f23
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 20:53:29 2025 -0600
Add integration proposal for Spec-Driven Development methodology in ARTVoice
commit 9b9b7a4841cc52a0a9f62e01ac7ee731b8804abd
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 20:32:28 2025 -0600
Add Insurance Voice Agent Scenario documentation and update navigation
- Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management.
- Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section.
commit 872732a1b8cb6935d27c50dd750654a8677dc972
Merge: 05160950 7e466729
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 19:30:09 2025 -0600
Merge branch 'v2-pablosal' of github.com:azure-samples/art-voice-agent-accelerator into v2/jinle-more-fixes
commit 051609509bf07e863b199fcf7090a0049ad17667
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 19:26:01 2025 -0600
Refactor agent paths and update documentation for agent discovery and configuration
commit 7e4667298ca253a9bce3472c584d9443b67b244e
Merge: ae5bb836 51c850c6
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Wed Dec 10 18:09:25 2025 -0600
Merge pull request #72 from AIappsGBBFactory/v2-pablosal-merge-deployment-changes
V2 pablosal merge deployment optimizations branch
commit ae5bb83671e92ef2dddb28325016fdeee6bfc5f5
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 10 18:08:56 2025 -0600
add
commit 51c850c69e644a7a06cc62312bb4dd0259a6c9f6
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 17:57:04 2025 -0600
fix voicelive output attributes
commit a641e4227262ec1f5412dba8e4c7e04b0611dfb6
Merge: 0a2ced23 b6416043
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 13:26:52 2025 -0800
Merge branch 'merge/v2/deployment-optimizations' into v2-pablosal-merge-deployment-changes
commit 0a2ced2376934167731b5765e5b03e306e349d4d
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 12:57:18 2025 -0800
Enhance configuration loading with .env.local support and update documentation
commit b64160439878128a5c45b8d6f4cbbb56b6dbf8a5
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 11:39:59 2025 -0800
feat: Add app configuration bootstrap to initialize environment variables
commit 5d7135fc424713dc7db1544858cdebd279c41ea1
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 10 13:24:43 2025 -0600
last changes
commit c540c2141b46393f6a7e168f69402e3426a866a0
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 10:12:41 2025 -0800
feat: Enhance Azure Voice Live integration and refactor configuration management
commit f6ec1bbd5d4c11c8f2d8f4e26a5e872cc64a6489
Author: Jin Lee <jinle@microsoft.com>
Date: Tue Dec 9 23:21:11 2025 -0800
Refactor Terraform configuration:
- Update main.tf to adjust foundry account and project naming conventions.
- Remove feature flags and keys from appconfig module as they are now managed externally.
- Clean up variables.tf by removing unused variables and updating descriptions.
- Delete provider configuration file as it is no longer needed.
- Change default application name from "rtaudioagent" to "artagent" and adjust related settings.
- Modify connection settings and pool sizes for improved performance.
commit fc1815100982a4e81d88fd9ef87357ba7a1fbde2
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Tue Dec 9 23:58:52 2025 -0600
add value
commit eaef115e37f0474e85ea7d9143547f81f1cdeebc
Author: Jin Lee <jinle@microsoft.com>
Date: Mon Dec 8 01:29:33 2025 -0800
refactor: Update architecture diagram in README for clarity and consistency in orchestration modes
commit 3197e0cb64b8e90a688380cd412f19fea0085bff
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 23:28:56 2025 -0700
removing unused dependency in src/herlpers.py
commit 7ad2546503ccb6cf203335de754102bfe67d21f3
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 23:28:13 2025 -0700
further deployment cleanup, docs update/tweaks, adding more todos
commit 7893afd6542da8aa742e9217ca2914beb3a54443
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 15:47:36 2025 -0600
feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity
commit a2e370fccdd0ed4c9a6a1d94ec5e082c0bcee1c3
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 15:16:07 2025 -0600
chore: Update CH…
JinLee794
added a commit
that referenced
this pull request
Jan 26, 2026
* Squashed commit of the following:
commit ab8c62bd00ba036d03836dd76c4d8a56de5a7f78
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Jan 26 14:29:37 2026 -0600
Refactor code structure for improved readability and maintainability
commit 2fa85353436f03d1ca7cca191c5d8f007bcf8417
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Jan 26 14:03:37 2026 -0600
feat: Add original_name field to AgentInfo and enhance scenario builder for custom agent visibility
commit 039f84839b7a85e9ffb1b7966c246b94cab210c4
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Wed Jan 7 12:15:31 2026 -0600
refactor: remove legacy Redis management classes and related files
- Deleted AzureRedisManager, AsyncAzureRedisManager, RedisKeyManager, and associated models.
- Removed unused Redis interaction logic to streamline the codebase.
- Updated tests to reflect changes in the VoiceHandler module, removing deprecated MediaHandler alias.
- Ensured compatibility with the new voice module structure.
commit 496947d0d7f309f6317bd31b39dfebf7a222620d
Merge: a7ccd8cb 1900437c
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Mon Jan 5 09:55:05 2026 +0000
Merge pull request #24 from AIappsGBBFactory/staging
commit 1900437cd11ba91bdb23d13623362610e755fbf2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 16:59:08 2026 -0600
feat: Update Voice Live readiness status to use event envelope format
commit ce643c293e774ac0290e7a5f6a68735043440989
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 16:54:03 2026 -0600
feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts
- Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation.
- Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements.
- Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints.
- Improved metrics naming conventions for clarity in load testing results.
commit 34997aa57606aa9783bceb49fbb71aaa94150546
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 12:33:47 2026 -0600
Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes
commit 5a9e0544e15752a5c47a7eb0b963a079b2d7a3b6
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:39:48 2026 -0600
Add panning functionality to ScenarioGraphCanvas and reset button
commit 10fb31d6b451201b80489bb6f31256007f80c6df
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:15:58 2026 -0600
Disable view toggle buttons for chat/graph/timeline in ConversationControls
commit a719fd6696b59caac66334aea0e50e74cff34a21
Merge: 31cf499d f07fdb90
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:13:22 2026 -0600
Merge branch 'feat/voice-handler-refactoring' into staging
commit 31cf499dc690adc4fb23e66b87835d569fe6d725
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:54:48 2026 -0600
Enhance logging functions to use log_plain for consistency and clarity in local development setup script
commit b75fbb7ac70198d142d2d0431e96f5f02b79f5b0
Merge: aaa6fa03 10a9cca9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:54:38 2026 -0600
Merge branch 'staging' of github.com:aiappsgbbfactory/art-voice-agent-accelerator into staging
commit 10a9cca90b11fcdcc776c8a2de24eb58ab99e1f7
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 10:45:16 2026 -0600
Refactor ACS logging and add default orchestration scenario (#23)
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit aaa6fa0363a047ee438fed7aedd03169ee56e586
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:43:31 2026 -0600
Refactor ACS logging and add default orchestration scenario
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
commit 08fca7270cd2ca0afc89c27501c15187b6bc9b9f
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:41 2026 -0600
enhanced the scenariobuilder with flowy (#22)
* docs: add comprehensive voice processing architecture documentation
Add complete documentation for the voice processing architecture:
New Documentation:
- docs/architecture/voice/README.md - Comprehensive voice architecture guide
* VoiceHandler overview and usage patterns
* TTS playback and text processing
* Speech cascade pipeline documentation
* Audio specifications for browser and ACS transports
* Testing guidelines with actual test file references
* Troubleshooting guide for common issues
- apps/artagent/backend/voice/README.md - Developer quick reference
* Directory structure and module organization
* Quick start examples
* Common tasks and patterns
* File location guide
* Testing commands
Documentation Updates:
- docs/mkdocs.yml - Add voice architecture to navigation
- docs/operations/troubleshooting.md - Add voice-specific troubleshooting
Key Improvements:
- Fixed mkdocs formatting for proper list rendering
- Updated all test references to match actual test files:
* test_voice_handler_components.py
* test_voice_handler_compat.py
* test_cascade_orchestrator_entry_points.py
* test_cascade_llm_processing.py
- Verified all script references (quick_test.sh, test_orchestrator.py)
- Added prerequisites for running tests with dev dependencies
- Included both basic and advanced testing examples
All file paths and examples have been verified against the actual codebase.
Related to #[TBD]
* Add custom styles for Flowy flowchart integration with agent blocks
* feat: Enhance output port visibility logic in ScenarioGraphCanvas
* feat: Add expandable full prompt view for source agent in HandoffEditorDialog
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit ed1ee049af18f5b724e0a274140a60fcb8aa82be
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:24 2026 -0600
feat: voice handler refactoring and MediaHandler migration (#21)
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cb4fbc19bb6e2536590d9f55cd261aac39042343
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:10 2026 -0600
Enhance logging and user prompts in preflight and pre-provisioning scripts (#20)
- Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting.
- Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh.
- Added color-coded success, warning, and error messages for better visibility.
- Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values.
- Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit f07fdb9042b2ef67eb8ec390aeda4c01a0666098
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sat Jan 3 22:48:03 2026 -0600
feat: voice handler refactoring and MediaHandler migration
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
commit 52ee0196c5a07cb7ee9a7b3635c0dcbaf3967fdb
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 13:50:35 2026 -0600
Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cf5fd497c19603f44d7272c494e1709d0a63a6a4
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:58:18 2026 -0600
feat: Evaluation Framework + Frontend UI (#18)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: add evaluation framework and frontend UI for Responses API
**Evaluation Framework:**
- Add EventRecorder with git commit SHA tracking
- Add API-aware scoring with budget adjustments for verbosity
- Add scenario runner for automated testing
- Add CLI for running evaluations
- Add validate_phases.py for phase-based validation
- Add wrappers for endpoint detection
**Frontend UI:**
- Add cascade_model and voicelive_model selectors in Agent Builder
- Add Responses API endpoint preference dropdown
- Add conditional fields for verbosity, reasoning_effort, etc.
- Update ScenarioBuilder with model configuration options
- Display API version fields
**Documentation:**
- Add docs/testing/model-evaluation.md
- Add evaluation playground Jupyter notebook
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 72de37f3d441ebd50a7bc81480c8a1e9ec683e61
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:58:07 2026 -0600
feat: Orchestrator Integration + Optimizations (#17)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: integrate Responses API in orchestrators and add optimizations
**Cascade Orchestrator:**
- Update model selection to use agent.get_model_for_mode('cascade')
- Integrate Responses API routing based on endpoint_preference
- Add error handling for unsupported parameters
- Extract TTS processing into separate tts_processor module
**VoiceLive Orchestrator:**
- Update to use agent.get_model_for_mode('voicelive')
- Add registry cleanup to prevent unbounded growth
- Improve memory management and stale orchestrator cleanup
- Extract DTMF processing into separate dtmf_processor module
**Tests:**
- Add test_cascade_orchestrator_entry_points
- Add test_cascade_llm_processing
- Add test_dtmf_processor
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit f9fabce297901ddae5258f99e4abbaa59e7993c8
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:53:36 2026 -0600
feat: Responses API Infrastructure & Dual Model Configuration (#16)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* fix: update project version to 2.0.0-beta in pyproject.toml
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit b19697b7d1aa4e0a24e70d3ebc6b027a664e549e
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:51 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cac95f0098fd2573e5ae6d0b461df9168f791710
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:38 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 2e27b282d6477a6e2d373646751f2c10b1a48e2b
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:25 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 3e00e817e8849a0378a556e9ccc7fa25dc558862
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:15 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit d68f36d7811c3954c2b49ffc9edcee127834a2ad
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 13:20:03 2025 -0600
feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 281267b1a2c2d5bf977e89b89c5b818104e0c2be
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 22 16:49:45 2025 -0600
feat: add context variable support for handoffs and enhance UI for variable mapping
commit d7d5be7b338e3d3491c8af75ee8108d262efdc67
Merge: 8b06eca3 4af93f3b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 22 13:13:33 2025 -0600
Merge branch 'main' into staging
commit 8b06eca322154c670bc23d71e1236e60f16033ed
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:24:11 2025 -0600
refactor: update preprovision hook execution and streamline backend configuration
commit 8b76c48ed45c8c075d914078c67a59b77b2f0e37
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:11:48 2025 -0600
fix: update environment logic for pull_request events in Azure deployment workflow
commit 05b8ad122b4c1fed643925da3af47acdf63ec806
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:10:03 2025 -0600
refactor: enhance run-name logic for Azure deployment workflow
commit 2ce3535278aa697a28da9b3d20c3f7f65358da81
Merge: 614c4e8e 7fea92c7
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 09:53:19 2025 -0600
fix: add peer dependency flag to multiple packages in package-lock.json
commit 614c4e8ec8904a0ea1f52e39474cee1faa15ec77
Merge: 57380059 5b174970
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Fri Dec 19 09:14:13 2025 -0600
Merge branch 'Azure-Samples:staging' into staging
commit 57380059b6e4e52337857c95dd20aae42ca72d32
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 09:07:12 2025 -0600
refactor: enhance EasyAuth interactive prompts and streamline user choices
commit 5b174970bde1dbc3906ac5e65c45e0f7a7d3ca87
Merge: c328369c 36a3de9f
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 18 18:26:27 2025 -0600
Merge pull request #81 from AIappsGBBFactory/staging
commit 36a3de9f2cd8764ea8f17fc8f9c42f65eb6bc478
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 18:20:48 2025 -0600
refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts
commit 1ce8584bb77255efde28f533d703439ed025d714
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 18:12:41 2025 -0600
Refactor TTS Playback and Voice Handling
- Consolidated TTS playback logic into a unified class for speech cascade.
- Removed deprecated VoiceSessionContext and related compatibility shims.
- Enhanced error handling during tool initialization and event handler registration.
- Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings.
- Improved logging for TTS synthesis and streaming processes.
- Added new handoff tool registration for dynamic routing.
commit 0d529bc066458d606a06d1aff2012ec6668c713c
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 17:24:58 2025 -0600
refactor: update LiveOrchestrator to enhance user message history management and improve handoff context
commit 41bf0fb431b27063671616ecc3a49d8c80db487d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 17:01:59 2025 -0600
add opentelemetry import for tracing support in TTS module
commit b51e7f5474e7a4b7f88f3047f45de6903ddf79cb
Merge: 5c2aac51 c328369c
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 18 16:35:25 2025 -0600
Merge branch 'staging' into staging
commit 5c2aac51f5d371f8ac568b9b271dd6909268f118
Merge: d0b279c7 85a8cf76
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 18 16:28:31 2025 -0600
Merge pull request #8 from AIappsGBBFactory/docs/user-flows
Docs/user flows - and adding easyauth as part of postprovision
commit c328369c3945c3ad9150c2cb493e42b8a5f13d8a
Merge: 4af52a50 9671ae49
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 18 16:27:33 2025 -0600
Merge pull request #80 from Azure-Samples/usecases/claims
Usecases/claims
commit 9671ae49e90a40c9ca6cc6fbf00d5386d12c2914
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 16:19:07 2025 -0600
art
commit 7678afad82ec5d4a926ad6dcbd005a12da9eeb0d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:53:19 2025 -0600
add value
commit 8fdd4443d25007421c8618c9f5f95ede84b23f5c
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:47:40 2025 -0600
add
commit 5716f3aa0c822ca0c34f038f485251b3688a242d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:43:02 2025 -0600
add value
commit 540f3f053b279fc2af04f188d54e54017b04bd71
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:39:44 2025 -0600
add value
commit 79ad2f19fe7180bee3777aab44cd3d76658637c8
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:25:33 2025 -0600
add value
commit f929399392f223d27c46cae28199bc3724a9973d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 14:48:50 2025 -0600
add value
commit 313e07f4b6d49560fedc730af0aa08c6f8cf3a55
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 10:59:26 2025 -0600
add value
commit 85a8cf76f2f462e21696d673acd3bbdcc27905e4
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:58:29 2025 -0600
fix: Update home screen and scenario images in quickstart guide
commit 24da693c035b38a6d598dfa04144990f801c32b8
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:55:18 2025 -0600
fix: Update home screen image in quickstart guide
commit 181035146300409b2447a2105dcaded125f3c0ba
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:52:40 2025 -0600
fix: Update demo profile creation images in quickstart guide
commit 524412c307e5b2ee1400067d492a7928efc83645
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:45:28 2025 -0600
chore: Remove unused workflow images for demo profiles
commit 4b99db67087f1185c5f21da1490292baf143dd06
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 00:03:57 2025 -0600
add value
commit b69106aed84f2cb5967827617aac1ffd1d154ad2
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 17 22:21:59 2025 -0600
add value
commit 90793e67edc4af12d55e7143912739561a64e39b
Author: Anna Quincy <annaquincy@microsoft.com>
Date: Wed Dec 17 14:32:34 2025 -0800
updated order so confirmation statement is in the correct spot
commit 9226954539efdf742109140cff8b7b399e680ad8
Author: Anna Quincy <annaquincy@microsoft.com>
Date: Wed Dec 17 14:22:24 2025 -0800
Added non qualifying rush response to ensure clear model behavior
commit c561d6702b8c58698c6a2e1fdb15aa1e1b47d001
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 18:14:56 2025 -0600
refactor: Remove backend IP restrictions configuration and related outputs
commit d0214091d970610479461d80d0bdbb52089c00ff
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 17:49:35 2025 -0600
feat: Implement EasyAuth configuration script and integrate into post-provisioning process
commit 6a02169c371f4bc49c0e2c82190c3a16772c28a2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 16:21:00 2025 -0600
feat: Add demo profile creation steps and related images to quickstart guide
commit 902f0f2fe3db1e0f144860b40da638b3c7388d11
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 16:09:11 2025 -0600
feat: Add initial agent builder and template selection screenshots to quickstart guide
commit a490de102fc16430835181095b2581de64337bf3
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:40:55 2025 -0600
fix: Correct image paths in quickstart guide for accurate rendering
commit 35cb6b91c227425bf63ea27db8125c8ad710e3a9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:35:13 2025 -0600
feat: Enhance scenario testing instructions for clarity and user guidance
commit cbdf736580e466b655c88fcdcb4ae3ac3239e594
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:13:33 2025 -0600
feat: Add user flow screenshots and enhance documentation for guided agent setup
commit 7407971c65741bcbf0b79dfeb527aad4176afa3f
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Tue Dec 16 12:04:38 2025 -0600
add value
commit d0b279c77c6a2db2efdd3404d8f02c94640957b0
Merge: 39cf1e74 ae97dd04
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Tue Dec 16 09:22:33 2025 -0600
Merge pull request #7 from AIappsGBBFactory/feat/scenario-orch
feat: Implement TTS Streaming Latency Analysis and Optimization Plan
commit ae97dd041d6fa5ea346f4b22c19bcd4b52c37e77
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 22:13:06 2025 -0600
fix: Simplify environment determination logic in deployment workflow
commit 83c6c50ffe916bf83549fca563326e19c647bb1a
Merge: ef62ef1c 39cf1e74
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 22:04:24 2025 -0600
Merge branch 'staging' into feat/scenario-orch
commit ef62ef1c75b1bc4c25d58ddfe1701abaa7961a57
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 21:57:58 2025 -0600
feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation
commit 29e7f82828871d57cc6b61fcd0dae2285e850cba
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 20:44:45 2025 -0600
feat(metrics): Introduce shared metrics factory for lazy initialization
- Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics.
- Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics.
- Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization.
- Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling.
- Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls.
- Introduced utility functions for building common metric attributes, ensuring consistency across metrics.
commit 09a1eb40f52a4bfd3c534dff57adccc0b9ad85ef
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Mon Dec 15 20:28:06 2025 -0600
add value
commit 1bdeddfa91755a154f149771e4e8de1cb1740285
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 20:19:51 2025 -0600
feat: Add predefined handoff condition patterns to enhance scenario orchestration
commit 10edd38fb2491878dd25500a1f7448be3ae2b031
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 19:43:11 2025 -0600
Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues.
commit 04db80cb036397d07c184c128b2815ed839ae71e
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 22:48:16 2025 -0600
feat: Enhance AgentBuilder with consistent field names and improved UI elements
commit dfda340c92f544d37f0ce1ec625cf59454132c40
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 22:38:42 2025 -0600
Add comprehensive tests for VoiceLive handler and orchestrator memory management
- Implement tests to verify cleanup functionality in LiveOrchestrator.
- Ensure proper registration and unregistration of orchestrators in the registry.
- Test background task tracking and cleanup mechanisms.
- Validate greeting task cancellation during orchestrator cleanup.
- Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry.
- Verify user message history deque is properly bounded and cleared on cleanup.
- Add scenario update tests to ensure correct agent management during updates.
- Optimize hot path functions to ensure non-blocking behavior during network calls.
commit 4af52a50b2c2af562cca78a53f3fa8109cd97571
Merge: 6b01d803 b8b0c279
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Sun Dec 14 19:06:46 2025 -0600
Merge pull request #79 from AIappsGBBFactory/hotfix/agent-context-and-handoffs
Hotfix/agent context and handoffs
commit b8b0c27948ab46474de405ff826b2caf2480c7d4
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 14:56:55 2025 -0600
Refactor tests and dependencies following module renaming and API changes
- Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock.
- Updated conftest.py to mock configuration and Azure OpenAI client for tests.
- Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules.
- Adjusted imports in test_artagent_wshelpers.py for orchestrator path change.
- Skipped tests in test_call_transfer_service.py due to API changes in toolstore.
- Updated datetime usage in test_demo_env_phrase_bias.py to use UTC.
- Modified websocket endpoint assertions in test_realtime.py to reflect new paths.
- Added new test file test_voice_handler_components.py for voice handler components.
commit 39cf1e74bc7894f3d86a43e847083269b093077e
Merge: 6b01d803 e837e43a
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Dec 14 13:02:22 2025 -0600
Merge pull request #6 from AIappsGBBFactory/hotfix/agent-context-and-handoffs
Hotfix/agent context and handoffs
commit e837e43a3e8720573878da492da293b854c271de
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:46:17 2025 -0600
Refactor environment variable assignment in deploy workflow for clarity
commit 79cb83af0b74817d826c13497d49eeddff79d49c
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:41:49 2025 -0600
Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates.
commit de6557a573bf35ee743433202bba6b71bab8279f
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:30:00 2025 -0600
Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts
commit 6666a9a9f44e2597486b77a5e3a2bc83f655c341
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:29:47 2025 -0600
Add error handling for Redis connection issues and implement unit tests for HandoffService
- Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts.
- Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building.
- Added tests for HandoffResolution dataclass to ensure correct property behavior and default values.
commit 94e39aedabf146d7312d16a4dc2a8f95da92df70
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 11:59:57 2025 -0600
Refactor code structure for improved readability and maintainability
commit 33b8c7d9a6cb2f56bdeb354d1f29c8f49e13177b
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 00:47:20 2025 -0600
feat: Add Scenario Builder component and integrate with RealTimeVoiceApp
- Introduced ScenarioBuilder component for visual orchestration of agent flows.
- Implemented drag-and-drop functionality for agents and handoff configuration.
- Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders.
- Enhanced state management for agent scenarios, including creation and updates.
- Integrated new handoff editor for configuring agent interactions.
commit 1b68215a7db47af4cd82efc873d8a236db8942a5
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 23:55:20 2025 -0600
feat: Implement TTS Streaming Latency Analysis and Optimization Plan
- Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture.
- Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming.
- Proposed a multi-phase optimization strategy to address identified issues, including:
- Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task.
- Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch.
- Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming.
- Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing.
- Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming.
- Created a detailed test implementation plan with metrics and success criteria to validate improvements.
test: Add unit tests for HandoffService
- Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods.
- Implemented tests for greeting selection and context building to ensure proper functionality.
- Added tests for the HandoffResolution dataclass to verify properties and default values.
commit 6b01d8031600614788302ef3003fcb6ab8497a2e
Merge: bb3826f4 a5a1d73e
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Sat Dec 13 14:27:08 2025 -0600
Merge pull request #78 from AIappsGBBFactory/feat/troubleshooting-enhancements
commit a5a1d73e2c833eb253ad72f3ab44353edd1b2e84
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:57:18 2025 -0600
Update .github/workflows/test-azd-hooks.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit a23e18744f594477fc014cf2f73c30215754b654
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:52:27 2025 -0600
Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit aaeae928f9da2ae145cc11f6d58f989931d4c52b
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:51:29 2025 -0600
Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit da6ead6f5b53fca1f32ab8fb339c8d993eb43155
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 13:50:27 2025 -0600
feat: Remove outdated troubleshooting documentation for deployment issues
commit b8a16dc733ba8de4ec424ea8786ebebb2de68472
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 13:41:13 2025 -0600
feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options
commit 7f9f889585584fa3f46be4b7a9e49d301c74e2cf
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 21:31:48 2025 -0600
feat: Update Azure deployment workflows and normalize container memory formats
commit cb6354e7462607c8398acdcef687d21c2b92786f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 20:47:45 2025 -0600
feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock
commit a4a19a33ebef5ad9f85ede7d4fa4486869285863
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 20:27:52 2025 -0600
feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks
commit 5b7a0979166b514eff9109a7e72a04b734851103
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:39:53 2025 -0600
feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting
commit 4d71fa7d39e1825799a3066a821567aa7a916645
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:35:10 2025 -0600
feat: Update preflight checks to conditionally include Docker in CI mode and log its status
commit e1a840e1a3b4cd0d2d825c82d552f8c5abd2674b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:31:06 2025 -0600
feat: Update test job names for clarity and enhance preflight checks for CI mode
commit 40e33ef7f6ba9c9a61ac035f906b3ad8fa8aa815
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:27:24 2025 -0600
feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup
commit 17da7f785413157274c2e8c33e41c559a4eca098
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:21:40 2025 -0600
feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow
commit d54b9ce2e1d0d39f1a8e5bd4e793c78187e12140
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:16:05 2025 -0600
feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration
commit a6d737c73a8e66e2750e0fbec357d7a3445d9bd0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:58:57 2025 -0600
fix(ci): simplify test-azd-hooks workflow tests and run in parallel
- Remove fragile grep-based function extraction that caused syntax errors
- Run lint, linux, macos, windows tests in parallel (no dependencies)
- Trigger on all pushes to main/staging (remove path filters for push)
- Simplify backend configuration test to avoid function sourcing issues
commit 46acb1415e5010a6a422df52470e944f29e4d223
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:56:01 2025 -0600
feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements
commit 819dd90f21444df83f93ad16faf55d5c7f3b972c
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:51:26 2025 -0600
Enhance provisioning scripts and documentation
- Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal.
- Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning.
- Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms.
- Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration.
- Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts.
- Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions.
commit bb3826f445e296cb6960cfdb8937275b2a145e55
Merge: 4b10e18d 516b6b6c
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Fri Dec 12 12:41:50 2025 -0600
Merge pull request #77 from AIappsGBBFactory/staging
docs updates and more pipeline related fixes/enhancements
commit 516b6b6c196ea779dde679e19ae79ec2e1d2f1d0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:39:05 2025 -0600
fix: Update README layout for improved clarity and navigation
commit 4b10e18dc77699a66c821f03005a7db3e7b2a337
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Fri Dec 12 12:37:50 2025 -0600
add value
commit fd2ca5acb094ef361bc19f63f716ae15b80e96b7
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:36:45 2025 -0600
fix: Restore header for ARTVoice Accelerator Framework in README
commit 6c33df1eebd99ce4275fc4528e5ecd9c1414220e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:21:26 2025 -0600
fix: Update README layout and enhance navigation links for better user experience
commit 9167d13735bb18ea38f1468b79cac3f1ebd2096a
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:09:40 2025 -0600
feat: Update documentation workflow name and enhance README with deployment badges
commit e170dd435528384a40a8af269130fd0aeb0f18c1
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:51:53 2025 -0600
fix: Add emoji to workflow names for better visibility
commit c913abefa7149407c4fe18f69733189a2c3f9ba9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:31:13 2025 -0600
adding rg as env var set at the gh env level
commit 7251f57308443dfc0392e74198ee640eef539b05
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:26:23 2025 -0600
feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment
commit 1b077d15d652d1cc3453f1440d8e6917b8a73bb3
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:00:16 2025 -0600
fix: Update GitHub token to use secrets for enhanced security
commit 5391566295a7d967946bc0875a2bc90bd8ecbbaa
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 10:42:42 2025 -0600
fix: Update condition for output extraction in deployment workflow
commit f9cf3c83a35643b2325a23e3091355270a761f04
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 10:12:23 2025 -0600
fix: Add run-name to the Azure deployment workflow for better clarity
commit f051b1387b6daacb84ccbfbb8a54519f5458a99d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 09:51:54 2025 -0600
refactor: replace deploy-azd workflow with reusable template and remove redundant summary job
- Updated the deployment workflow name to "Deploy to Azure".
- Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml.
- Removed the deployment summary job and its associated steps to streamline the workflow.
commit b99084b98a6dba3b4b7a03b897e4bd9950e8b01f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 22:53:40 2025 -0600
docs: Update documentation links and improve clarity across various guides
commit 2c88c9acb23a3c8563f482db7a8ce323a48b11fd
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 22:25:29 2025 -0600
feat: Enhance API documentation and tagging for better clarity and organization
commit db05f3aa81db340548234d88d32a10512fa3b1eb
Merge: 0787f03f 598604c0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:53:03 2025 -0600
Merge branch 'staging' of github.com:aiappsgbbfactory/art-voice-agent-accelerator into staging
commit 0787f03fb9e60f53cc15d88f14233dd900152c9b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:52:32 2025 -0600
fix: Update documentation URLs to reflect new repository location
commit 598604c0de3c1933108d117c0475920ca6a1f886
Merge: b36476ca f8904934
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 11 19:27:17 2025 -0800
Merge pull request #2 from AIappsGBBFactory/v2/deploy-fixes
improvements flow
commit 0ed67a24dfe7aef67b7ad9c937ae5db8a70999f7
Merge: 38d9ca97 f8904934
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 11 21:25:16 2025 -0600
Merge pull request #76 from AIappsGBBFactory/v2/deploy-fixes
commit 38d9ca979e21ea73311806a83ddcd136be4095cd
Merge: b36476ca 6dc26a3e
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 11 19:22:18 2025 -0800
Merge pull request #75 from Azure-Samples/v2-pablosal
V2 pablosal
commit f89049349f864ff2aa6f361f643d6fb54e5d7fce
Merge: 3a2eee5c 6dc26a3e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:18:47 2025 -0600
Merge branch 'v2-pablosal' into v2/deploy-fixes
commit 3a2eee5cbf555628ede657fce81847dd539a1ad2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:18:40 2025 -0600
fix: Update name from rtaudioagent to artaudioagent in environment parameter files
commit 5abe9e52786d9f9bed4b0ba855e4bd8a3785d434
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:49:22 2025 -0600
fix: Update name from rtaudioagent to artaudioagent in environment parameter files
commit 4f7796e2553c8af09ecad3f556b1098e3ee549af
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:42:19 2025 -0600
fix: Update naming conventions for foundry account and project variables in locals
commit 6dc26a3e61f67ab76fac21e7e20ad6cb91f1d27f
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 20:38:23 2025 -0600
add value
commit 56cb1d267d5287519d77b120371b2f829fa378a5
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:08:42 2025 -0600
fix: Implement auto-selection and timeout for user input in setup scripts
commit cc597f3867bced6279a242818d016ad2362aa853
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 19:14:50 2025 -0600
improvements flow
commit b1e0ad34fcedcfcdf6a4224a1e81ca8e35287074
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 19:12:49 2025 -0600
refactor: Remove unused backend API public URL and source phone number from environment parameter files
commit 33602eb894cd1c95fa8f04dee54af0942e374af0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 19:12:44 2025 -0600
refactor: Remove unused backend API public URL variable and related validation
commit 07ac39faee035c1bc77e2e81379e2100ca866023
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 18:56:58 2025 -0600
fix: Use coalesce function for location assignment in storage account resource
commit f1f827a742b8a78ade67de786e6d78394278be04
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 18:43:04 2025 -0600
fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults
commit 82c8bf46df1948c2d5fe05db75d2af19dbddba6f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:32:12 2025 -0600
refactor: Simplify state key handling in provider configuration by using environment name
commit 7b9cd66693b4ae86c53f6039c0b36d9fe0fac00e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:24:29 2025 -0600
fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly
commit ed31e932febda5b8c94809965ffa1158d7198d73
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:20:43 2025 -0600
refactor: Remove outdated AZD deployment workflow and update documentation links for clarity
commit 29925e9ef2e2dd44b94169682c899621e4c2ee1f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:51:58 2025 -0600
fix: Update health check endpoint in postprovision script to use correct API path
commit 0f4a0f99ccf7366d4e473cb18c16fca9052cd597
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:45:03 2025 -0600
feat: Allow mkdocs build to proceed with warnings by removing --strict flag
commit 76c89462941e9f4a5b051736790433fcb05117af
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:42:33 2025 -0600
feat: Add site_dir configuration to mkdocs.yml for improved site structure
commit 34936afc98757908a0deae9d2710bc1a6457703d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:38:46 2025 -0600
modified docs workflow
commit da10e6e258a015bcdba373ae713aa9a8b5377fd6
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:36:06 2025 -0600
feat: Revise documentation deployment workflow to enhance dependency management and streamline build process
commit a64e7fa4ab94ccbeac0fd9a4afb0cf82e77fb8ff
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:30:41 2025 -0600
feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management
commit 200c14026da0deb56bef3813304eea6075091b69
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:15:37 2025 -0600
feat: Update audio handling and documentation dependencies for improved installation and error handling
commit e83c84f23790b3387d5da440e67d84d98ca1960b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:07:12 2025 -0600
cicd test for azd deploy
commit 4eb33fa287cb8bc664118439fdd9902c0658d0d8
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 14:55:59 2025 -0600
feat: Implement scenario-based handoff map resolution for orchestrator configuration
commit f97621578bfe54ab06a5fd50c2772f99acd63834
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 14:29:14 2025 -0600
feat: Enhance scenario loading to support orchestration.yaml naming convention
commit 3203b58f12850d20b06c0b116385d424dd157641
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 13:54:57 2025 -0600
feat: Add Concierge agent configuration and prompts for banking scenarios
- Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations.
- Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes.
- Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios.
- Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading.
- Updated default start agent to BankingConcierge and added validation for agent existence at startup.
commit 3c402dcbcbf36ad2ffcde6799b613d88610a9fd0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 11:34:05 2025 -0600
Enhance Terraform configuration and scripts for Voice Live integration
- Update Dockerfile to install dependencies and set up virtual environment.
- Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling.
- Refactor sync-appconfig.sh to streamline key-value imports and feature flag management.
- Add provider.conf.json generation for remote state backend configuration.
- Update main.tf and outputs.tf to support new Voice Live model deployments.
- Introduce voice_live_location and voice_live_model_deployments variables in variables.tf.
commit 3365e8b4836dbc77b48c5c207c0f093c4518eeb4
Merge: d8a10ddc 313a39d4
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 11 00:38:58 2025 -0600
Merge pull request #73 from AIappsGBBFactory/v2/jinle-more-fixes
Updated documentation based on new scenario-based structure, added proposal for speckit integration
commit d8a10ddc9f00198cff3b03a1d28286c235f519b8
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 00:37:50 2025 -0600
add value
commit 313a39d42078245ee9b0fe983a53066c355b5f23
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 20:53:29 2025 -0600
Add integration proposal for Spec-Driven Development methodology in ARTVoice
commit 9b9b7a4841cc52a0a9f62e01ac7ee731b8804abd
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 20:32:28 2025 -0600
Add Insurance Voice Agent Scenario documentation and update navigation
- Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management.
- Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section.
commit 872732a1b8cb6935d27c50dd750654a8677dc972
Merge: 05160950 7e466729
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 19:30:09 2025 -0600
Merge branch 'v2-pablosal' of github.com:azure-samples/art-voice-agent-accelerator into v2/jinle-more-fixes
commit 051609509bf07e863b199fcf7090a0049ad17667
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 19:26:01 2025 -0600
Refactor agent paths and update documentation for agent discovery and configuration
commit 7e4667298ca253a9bce3472c584d9443b67b244e
Merge: ae5bb836 51c850c6
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Wed Dec 10 18:09:25 2025 -0600
Merge pull request #72 from AIappsGBBFactory/v2-pablosal-merge-deployment-changes
V2 pablosal merge deployment optimizations branch
commit ae5bb83671e92ef2dddb28325016fdeee6bfc5f5
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 10 18:08:56 2025 -0600
add
commit 51c850c69e644a7a06cc62312bb4dd0259a6c9f6
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 17:57:04 2025 -0600
fix voicelive output attributes
commit a641e4227262ec1f5412dba8e4c7e04b0611dfb6
Merge: 0a2ced23 b6416043
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 13:26:52 2025 -0800
Merge branch 'merge/v2/deployment-optimizations' into v2-pablosal-merge-deployment-changes
commit 0a2ced2376934167731b5765e5b03e306e349d4d
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 12:57:18 2025 -0800
Enhance configuration loading with .env.local support and update documentation
commit b64160439878128a5c45b8d6f4cbbb56b6dbf8a5
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 11:39:59 2025 -0800
feat: Add app configuration bootstrap to initialize environment variables
commit 5d7135fc424713dc7db1544858cdebd279c41ea1
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 10 13:24:43 2025 -0600
last changes
commit c540c2141b46393f6a7e168f69402e3426a866a0
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 10:12:41 2025 -0800
feat: Enhance Azure Voice Live integration and refactor configuration management
commit f6ec1bbd5d4c11c8f2d8f4e26a5e872cc64a6489
Author: Jin Lee <jinle@microsoft.com>
Date: Tue Dec 9 23:21:11 2025 -0800
Refactor Terraform configuration:
- Update main.tf to adjust foundry account and project naming conventions.
- Remove feature flags and keys from appconfig module as they are now managed externally.
- Clean up variables.tf by removing unused variables and updating descriptions.
- Delete provider configuration file as it is no longer needed.
- Change default application name from "rtaudioagent" to "artagent" and adjust related settings.
- Modify connection settings and pool sizes for improved performance.
commit fc1815100982a4e81d88fd9ef87357ba7a1fbde2
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Tue Dec 9 23:58:52 2025 -0600
add value
commit eaef115e37f0474e85ea7d9143547f81f1cdeebc
Author: Jin Lee <jinle@microsoft.com>
Date: Mon Dec 8 01:29:33 2025 -0800
refactor: Update architecture diagram in README for clarity and consistency in orchestration modes
commit 3197e0cb64b8e90a688380cd412f19fea0085bff
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 23:28:56 2025 -0700
removing unused dependency in src/herlpers.py
commit 7ad2546503ccb6cf203335de754102bfe67d21f3
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 23:28:13 2025 -0700
further deployment cleanup, docs update/tweaks, adding more todos
commit 7893afd6542da8aa742e9217ca2914beb3a54443
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 15:47:36 2025 -0600
feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity
commit a2e370fccdd0ed4c9a6a1d94ec5e082c0bcee1c3
Author: Jin Lee <jinle@microsoft.com>
Date: …
JinLee794
added a commit
that referenced
this pull request
Jan 27, 2026
* Squashed commit of the following:
commit ab8c62bd00ba036d03836dd76c4d8a56de5a7f78
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Jan 26 14:29:37 2026 -0600
Refactor code structure for improved readability and maintainability
commit 2fa85353436f03d1ca7cca191c5d8f007bcf8417
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Jan 26 14:03:37 2026 -0600
feat: Add original_name field to AgentInfo and enhance scenario builder for custom agent visibility
commit 039f84839b7a85e9ffb1b7966c246b94cab210c4
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Wed Jan 7 12:15:31 2026 -0600
refactor: remove legacy Redis management classes and related files
- Deleted AzureRedisManager, AsyncAzureRedisManager, RedisKeyManager, and associated models.
- Removed unused Redis interaction logic to streamline the codebase.
- Updated tests to reflect changes in the VoiceHandler module, removing deprecated MediaHandler alias.
- Ensured compatibility with the new voice module structure.
commit 496947d0d7f309f6317bd31b39dfebf7a222620d
Merge: a7ccd8cb 1900437c
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Mon Jan 5 09:55:05 2026 +0000
Merge pull request #24 from AIappsGBBFactory/staging
commit 1900437cd11ba91bdb23d13623362610e755fbf2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 16:59:08 2026 -0600
feat: Update Voice Live readiness status to use event envelope format
commit ce643c293e774ac0290e7a5f6a68735043440989
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 16:54:03 2026 -0600
feat: Add mkdocs-mermaid-zoom dependency and update locust load test scripts
- Added mkdocs-mermaid-zoom to pyproject.toml and uv.lock for enhanced diagram support in documentation.
- Enhanced locustfile.acs_media.py with rate limit detection and error handling improvements.
- Introduced locustfile.browser_conversation.py for testing browser-based voice conversation endpoints.
- Improved metrics naming conventions for clarity in load testing results.
commit 34997aa57606aa9783bceb49fbb71aaa94150546
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 12:33:47 2026 -0600
Update CHANGELOG.md for 2.0.0-beta.1 release: add new features, enhancements, fixes, and infrastructure changes
commit 5a9e0544e15752a5c47a7eb0b963a079b2d7a3b6
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:39:48 2026 -0600
Add panning functionality to ScenarioGraphCanvas and reset button
commit 10fb31d6b451201b80489bb6f31256007f80c6df
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:15:58 2026 -0600
Disable view toggle buttons for chat/graph/timeline in ConversationControls
commit a719fd6696b59caac66334aea0e50e74cff34a21
Merge: 31cf499d f07fdb90
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 11:13:22 2026 -0600
Merge branch 'feat/voice-handler-refactoring' into staging
commit 31cf499dc690adc4fb23e66b87835d569fe6d725
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:54:48 2026 -0600
Enhance logging functions to use log_plain for consistency and clarity in local development setup script
commit b75fbb7ac70198d142d2d0431e96f5f02b79f5b0
Merge: aaa6fa03 10a9cca9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:54:38 2026 -0600
Merge branch 'staging' of github.com:aiappsgbbfactory/art-voice-agent-accelerator into staging
commit 10a9cca90b11fcdcc776c8a2de24eb58ab99e1f7
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 10:45:16 2026 -0600
Refactor ACS logging and add default orchestration scenario (#23)
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit aaa6fa0363a047ee438fed7aedd03169ee56e586
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sun Jan 4 10:43:31 2026 -0600
Refactor ACS logging and add default orchestration scenario
- Removed info-level logging for ACS configuration details to reduce verbosity.
- Changed some logging statements to debug level for better log management.
- Updated peer.service attribute in telemetry to use "azure-communication-services".
- Introduced a new orchestration.yaml file defining a default customer service scenario with multiple agents and handoff configurations.
commit 08fca7270cd2ca0afc89c27501c15187b6bc9b9f
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:41 2026 -0600
enhanced the scenariobuilder with flowy (#22)
* docs: add comprehensive voice processing architecture documentation
Add complete documentation for the voice processing architecture:
New Documentation:
- docs/architecture/voice/README.md - Comprehensive voice architecture guide
* VoiceHandler overview and usage patterns
* TTS playback and text processing
* Speech cascade pipeline documentation
* Audio specifications for browser and ACS transports
* Testing guidelines with actual test file references
* Troubleshooting guide for common issues
- apps/artagent/backend/voice/README.md - Developer quick reference
* Directory structure and module organization
* Quick start examples
* Common tasks and patterns
* File location guide
* Testing commands
Documentation Updates:
- docs/mkdocs.yml - Add voice architecture to navigation
- docs/operations/troubleshooting.md - Add voice-specific troubleshooting
Key Improvements:
- Fixed mkdocs formatting for proper list rendering
- Updated all test references to match actual test files:
* test_voice_handler_components.py
* test_voice_handler_compat.py
* test_cascade_orchestrator_entry_points.py
* test_cascade_llm_processing.py
- Verified all script references (quick_test.sh, test_orchestrator.py)
- Added prerequisites for running tests with dev dependencies
- Included both basic and advanced testing examples
All file paths and examples have been verified against the actual codebase.
Related to #[TBD]
* Add custom styles for Flowy flowchart integration with agent blocks
* feat: Enhance output port visibility logic in ScenarioGraphCanvas
* feat: Add expandable full prompt view for source agent in HandoffEditorDialog
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit ed1ee049af18f5b724e0a274140a60fcb8aa82be
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:24 2026 -0600
feat: voice handler refactoring and MediaHandler migration (#21)
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cb4fbc19bb6e2536590d9f55cd261aac39042343
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Jan 4 00:50:10 2026 -0600
Enhance logging and user prompts in preflight and pre-provisioning scripts (#20)
- Updated logging functions in preflight-checks.sh, ssl-preprovision.sh, sync-appconfig.sh, postprovision.sh, and preprovision.sh for consistent output formatting.
- Improved user prompts for SSL certificate configuration and Azure Entra group creation in ssl-preprovision.sh and postprovision.sh.
- Added color-coded success, warning, and error messages for better visibility.
- Modified the handling of environment variables in postprovision.sh to ensure updates are made without overwriting existing values.
- Updated Terraform configurations to manage app configuration and cognitive account settings with soft delete options.
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit f07fdb9042b2ef67eb8ec390aeda4c01a0666098
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Sat Jan 3 22:48:03 2026 -0600
feat: voice handler refactoring and MediaHandler migration
Major refactoring of voice processing architecture:
Core Voice Changes:
- Implement new VoiceHandler as primary entry point for voice sessions
- Delete deprecated speech_cascade/tts.py (652 lines removed)
- Consolidate TTS functionality into voice/tts/playback.py
- Enhance CascadeOrchestrator with improved turn management
- Add VoiceSessionContext for clean dependency injection
API & Integration:
- Migrate /api/v1/browser/conversation to VoiceHandler
- Migrate /api/v1/media/stream to VoiceHandler
- Create MediaHandler→VoiceHandler compatibility alias
- Update media_handler.py for backward compatibility
Infrastructure:
- Improve telemetry with Azure-style span naming
- Enhance ACS helpers with better session management
- Update session terminator for lifecycle management
- Add orchestration improvements for unified agents
Configuration & Samples:
- Update auth agent and insurance scenario configs
- Add handoff tool enhancements with context variables
- Update gpt_flow sample for new patterns
Frontend:
- Refactor App.jsx for improved voice handling UI
Testing & Documentation:
- Add test_voice_handler_compat.py for backward compatibility
- Add MEDIAHANDLER_MIGRATION.md tracking document
This change maintains full backward compatibility while establishing
the foundation for cleaner voice processing patterns going forward.
Closes #[TBD]
commit 52ee0196c5a07cb7ee9a7b3635c0dcbaf3967fdb
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 13:50:35 2026 -0600
Cleaning up lifecycle management logic into dedicated structure, keep main.py clean (#19)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cf5fd497c19603f44d7272c494e1709d0a63a6a4
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:58:18 2026 -0600
feat: Evaluation Framework + Frontend UI (#18)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: add evaluation framework and frontend UI for Responses API
**Evaluation Framework:**
- Add EventRecorder with git commit SHA tracking
- Add API-aware scoring with budget adjustments for verbosity
- Add scenario runner for automated testing
- Add CLI for running evaluations
- Add validate_phases.py for phase-based validation
- Add wrappers for endpoint detection
**Frontend UI:**
- Add cascade_model and voicelive_model selectors in Agent Builder
- Add Responses API endpoint preference dropdown
- Add conditional fields for verbosity, reasoning_effort, etc.
- Update ScenarioBuilder with model configuration options
- Display API version fields
**Documentation:**
- Add docs/testing/model-evaluation.md
- Add evaluation playground Jupyter notebook
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 72de37f3d441ebd50a7bc81480c8a1e9ec683e61
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:58:07 2026 -0600
feat: Orchestrator Integration + Optimizations (#17)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* feat: integrate Responses API in orchestrators and add optimizations
**Cascade Orchestrator:**
- Update model selection to use agent.get_model_for_mode('cascade')
- Integrate Responses API routing based on endpoint_preference
- Add error handling for unsupported parameters
- Extract TTS processing into separate tts_processor module
**VoiceLive Orchestrator:**
- Update to use agent.get_model_for_mode('voicelive')
- Add registry cleanup to prevent unbounded growth
- Improve memory management and stale orchestrator cleanup
- Extract DTMF processing into separate dtmf_processor module
**Tests:**
- Add test_cascade_orchestrator_entry_points
- Add test_cascade_llm_processing
- Add test_dtmf_processor
Depends on: PR #1 (Responses API Infrastructure)
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit f9fabce297901ddae5258f99e4abbaa59e7993c8
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Jan 3 10:53:36 2026 -0600
feat: Responses API Infrastructure & Dual Model Configuration (#16)
* feat: enhance azd environment variable handling with error checks and local state support
* fix: update foundry account and project naming conventions for consistency
* feat: add Responses API infrastructure and dual model configuration
**Infrastructure Changes:**
- Add UnifiedResponse dataclass for dual endpoint support
- Implement _should_use_responses_endpoint() routing logic
- Add _prepare_responses_params() and _prepare_chat_params() methods
- Update generate_response() to route between /chat/completions and /responses
**Model Configuration:**
- Add cascade_model and voicelive_model fields to AgentConfig
- Add get_model_for_mode() with support for 'cascade', 'media', 'voicelive', 'realtime' aliases
- Add Responses API fields: endpoint_preference, verbosity, min_p, typical_p, reasoning_effort, include_reasoning, max_completion_tokens
- Update ModelConfigSchema in agent_builder API
**Tests:**
- Add test_generate_response_respects_responses_config
- Add test_generate_response_respects_chat_config
- Add TestUnifiedAgentGetModelForMode test suite
This PR provides the foundation for Responses API support without changing orchestrator behavior.
* fix: update project version to 2.0.0-beta in pyproject.toml
---------
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit b19697b7d1aa4e0a24e70d3ebc6b027a664e549e
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:51 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#12)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit cac95f0098fd2573e5ae6d0b461df9168f791710
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:38 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#13)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 2e27b282d6477a6e2d373646751f2c10b1a48e2b
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:25 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#15)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 3e00e817e8849a0378a556e9ccc7fa25dc558862
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 14:33:15 2025 -0600
feat(telemetry): consolidate to OpenTelemetry and establish proper hierarchy (#14)
Infrastructure Changes:
- Delete 6 obsolete latency_tool implementations (~2200 lines)
- Install SessionContextSpanProcessor for automatic session correlation
- Replace LatencyTool with @trace_speech decorators in legacy paths
- Remove latency_tool field from VoiceSessionContext
Speech Services & Dependencies:
- Add @trace_speech for STT partial/final transcripts with attributes
- Add TTS attributes: voice, output_format, language, audio_size_bytes
- Standardize ACS and Redis span attributes with OTel conventions
- Add voice_session root SERVER span in media/browser endpoints
Orchestrator & Token Tracking:
- Add tool execution and agent handoff observability spans
- Fix token tracking to use actual API usage data (not estimates)
- Update Azure OpenAI API to 2024-10-01-preview
- Add session metadata timestamps to MemoManager
Benefits:
- Single source of truth (ConversationTurnSpan + OTel)
- Complete E2E traces in Application Insights
- Accurate cost tracking and token visibility
- ~2300 lines of dead code removed
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit d68f36d7811c3954c2b49ffc9edcee127834a2ad
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 25 13:20:03 2025 -0600
feat: enhance TTS processing by adding text sanitization and sentence boundary detection (#11)
Co-authored-by: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
commit 281267b1a2c2d5bf977e89b89c5b818104e0c2be
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 22 16:49:45 2025 -0600
feat: add context variable support for handoffs and enhance UI for variable mapping
commit d7d5be7b338e3d3491c8af75ee8108d262efdc67
Merge: 8b06eca3 4af93f3b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 22 13:13:33 2025 -0600
Merge branch 'main' into staging
commit 8b06eca322154c670bc23d71e1236e60f16033ed
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:24:11 2025 -0600
refactor: update preprovision hook execution and streamline backend configuration
commit 8b76c48ed45c8c075d914078c67a59b77b2f0e37
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:11:48 2025 -0600
fix: update environment logic for pull_request events in Azure deployment workflow
commit 05b8ad122b4c1fed643925da3af47acdf63ec806
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 10:10:03 2025 -0600
refactor: enhance run-name logic for Azure deployment workflow
commit 2ce3535278aa697a28da9b3d20c3f7f65358da81
Merge: 614c4e8e 7fea92c7
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 09:53:19 2025 -0600
fix: add peer dependency flag to multiple packages in package-lock.json
commit 614c4e8ec8904a0ea1f52e39474cee1faa15ec77
Merge: 57380059 5b174970
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Fri Dec 19 09:14:13 2025 -0600
Merge branch 'Azure-Samples:staging' into staging
commit 57380059b6e4e52337857c95dd20aae42ca72d32
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 19 09:07:12 2025 -0600
refactor: enhance EasyAuth interactive prompts and streamline user choices
commit 5b174970bde1dbc3906ac5e65c45e0f7a7d3ca87
Merge: c328369c 36a3de9f
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 18 18:26:27 2025 -0600
Merge pull request #81 from AIappsGBBFactory/staging
commit 36a3de9f2cd8764ea8f17fc8f9c42f65eb6bc478
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 18:20:48 2025 -0600
refactor: streamline EasyAuth enabling process in CI mode and improve interactive prompts
commit 1ce8584bb77255efde28f533d703439ed025d714
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 18:12:41 2025 -0600
Refactor TTS Playback and Voice Handling
- Consolidated TTS playback logic into a unified class for speech cascade.
- Removed deprecated VoiceSessionContext and related compatibility shims.
- Enhanced error handling during tool initialization and event handler registration.
- Updated model configuration handling in UnifiedAgent to prioritize mode-specific settings.
- Improved logging for TTS synthesis and streaming processes.
- Added new handoff tool registration for dynamic routing.
commit 0d529bc066458d606a06d1aff2012ec6668c713c
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 17:24:58 2025 -0600
refactor: update LiveOrchestrator to enhance user message history management and improve handoff context
commit 41bf0fb431b27063671616ecc3a49d8c80db487d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 17:01:59 2025 -0600
add opentelemetry import for tracing support in TTS module
commit b51e7f5474e7a4b7f88f3047f45de6903ddf79cb
Merge: 5c2aac51 c328369c
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 18 16:35:25 2025 -0600
Merge branch 'staging' into staging
commit 5c2aac51f5d371f8ac568b9b271dd6909268f118
Merge: d0b279c7 85a8cf76
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 18 16:28:31 2025 -0600
Merge pull request #8 from AIappsGBBFactory/docs/user-flows
Docs/user flows - and adding easyauth as part of postprovision
commit c328369c3945c3ad9150c2cb493e42b8a5f13d8a
Merge: 4af52a50 9671ae49
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 18 16:27:33 2025 -0600
Merge pull request #80 from Azure-Samples/usecases/claims
Usecases/claims
commit 9671ae49e90a40c9ca6cc6fbf00d5386d12c2914
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 16:19:07 2025 -0600
art
commit 7678afad82ec5d4a926ad6dcbd005a12da9eeb0d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:53:19 2025 -0600
add value
commit 8fdd4443d25007421c8618c9f5f95ede84b23f5c
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:47:40 2025 -0600
add
commit 5716f3aa0c822ca0c34f038f485251b3688a242d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:43:02 2025 -0600
add value
commit 540f3f053b279fc2af04f188d54e54017b04bd71
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:39:44 2025 -0600
add value
commit 79ad2f19fe7180bee3777aab44cd3d76658637c8
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 15:25:33 2025 -0600
add value
commit f929399392f223d27c46cae28199bc3724a9973d
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 14:48:50 2025 -0600
add value
commit 313e07f4b6d49560fedc730af0aa08c6f8cf3a55
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 10:59:26 2025 -0600
add value
commit 85a8cf76f2f462e21696d673acd3bbdcc27905e4
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:58:29 2025 -0600
fix: Update home screen and scenario images in quickstart guide
commit 24da693c035b38a6d598dfa04144990f801c32b8
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:55:18 2025 -0600
fix: Update home screen image in quickstart guide
commit 181035146300409b2447a2105dcaded125f3c0ba
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:52:40 2025 -0600
fix: Update demo profile creation images in quickstart guide
commit 524412c307e5b2ee1400067d492a7928efc83645
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 18 10:45:28 2025 -0600
chore: Remove unused workflow images for demo profiles
commit 4b99db67087f1185c5f21da1490292baf143dd06
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 18 00:03:57 2025 -0600
add value
commit b69106aed84f2cb5967827617aac1ffd1d154ad2
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 17 22:21:59 2025 -0600
add value
commit 90793e67edc4af12d55e7143912739561a64e39b
Author: Anna Quincy <annaquincy@microsoft.com>
Date: Wed Dec 17 14:32:34 2025 -0800
updated order so confirmation statement is in the correct spot
commit 9226954539efdf742109140cff8b7b399e680ad8
Author: Anna Quincy <annaquincy@microsoft.com>
Date: Wed Dec 17 14:22:24 2025 -0800
Added non qualifying rush response to ensure clear model behavior
commit c561d6702b8c58698c6a2e1fdb15aa1e1b47d001
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 18:14:56 2025 -0600
refactor: Remove backend IP restrictions configuration and related outputs
commit d0214091d970610479461d80d0bdbb52089c00ff
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 17:49:35 2025 -0600
feat: Implement EasyAuth configuration script and integrate into post-provisioning process
commit 6a02169c371f4bc49c0e2c82190c3a16772c28a2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 16:21:00 2025 -0600
feat: Add demo profile creation steps and related images to quickstart guide
commit 902f0f2fe3db1e0f144860b40da638b3c7388d11
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 16:09:11 2025 -0600
feat: Add initial agent builder and template selection screenshots to quickstart guide
commit a490de102fc16430835181095b2581de64337bf3
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:40:55 2025 -0600
fix: Correct image paths in quickstart guide for accurate rendering
commit 35cb6b91c227425bf63ea27db8125c8ad710e3a9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:35:13 2025 -0600
feat: Enhance scenario testing instructions for clarity and user guidance
commit cbdf736580e466b655c88fcdcb4ae3ac3239e594
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Tue Dec 16 14:13:33 2025 -0600
feat: Add user flow screenshots and enhance documentation for guided agent setup
commit 7407971c65741bcbf0b79dfeb527aad4176afa3f
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Tue Dec 16 12:04:38 2025 -0600
add value
commit d0b279c77c6a2db2efdd3404d8f02c94640957b0
Merge: 39cf1e74 ae97dd04
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Tue Dec 16 09:22:33 2025 -0600
Merge pull request #7 from AIappsGBBFactory/feat/scenario-orch
feat: Implement TTS Streaming Latency Analysis and Optimization Plan
commit ae97dd041d6fa5ea346f4b22c19bcd4b52c37e77
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 22:13:06 2025 -0600
fix: Simplify environment determination logic in deployment workflow
commit 83c6c50ffe916bf83549fca563326e19c647bb1a
Merge: ef62ef1c 39cf1e74
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 22:04:24 2025 -0600
Merge branch 'staging' into feat/scenario-orch
commit ef62ef1c75b1bc4c25d58ddfe1701abaa7961a57
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 21:57:58 2025 -0600
feat: Consolidate handoff logic into a unified HandoffService for consistent behavior across orchestrators and enhance documentation
commit 29e7f82828871d57cc6b61fcd0dae2285e850cba
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 20:44:45 2025 -0600
feat(metrics): Introduce shared metrics factory for lazy initialization
- Added `metrics_factory.py` to provide a common infrastructure for OpenTelemetry metrics.
- Implemented `LazyMeter`, `LazyHistogram`, and `LazyCounter` for lazy initialization of metrics.
- Updated `speech_cascade/metrics.py` to utilize the new shared metrics factory, simplifying metric initialization.
- Refactored `voicelive/metrics.py` to use the shared factory for consistent metric handling.
- Enhanced orchestrator classes in `speech_cascade/orchestrator.py` and `voicelive/orchestrator.py` to cache orchestrator configurations, improving performance and reducing redundant calls.
- Introduced utility functions for building common metric attributes, ensuring consistency across metrics.
commit 09a1eb40f52a4bfd3c534dff57adccc0b9ad85ef
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Mon Dec 15 20:28:06 2025 -0600
add value
commit 1bdeddfa91755a154f149771e4e8de1cb1740285
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 20:19:51 2025 -0600
feat: Add predefined handoff condition patterns to enhance scenario orchestration
commit 10edd38fb2491878dd25500a1f7448be3ae2b031
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Mon Dec 15 19:43:11 2025 -0600
Refactor logging levels from info to debug in connection manager, warmable pool, Redis manager, speech auth manager, speech recognizer, and text-to-speech modules for improved log verbosity control. Remove outdated greeting context tests and add comprehensive scenario orchestration contract tests to ensure functional contracts are preserved during refactoring. Update session agent manager tests to use set comparison for agent listing to avoid dict ordering issues.
commit 04db80cb036397d07c184c128b2815ed839ae71e
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 22:48:16 2025 -0600
feat: Enhance AgentBuilder with consistent field names and improved UI elements
commit dfda340c92f544d37f0ce1ec625cf59454132c40
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 22:38:42 2025 -0600
Add comprehensive tests for VoiceLive handler and orchestrator memory management
- Implement tests to verify cleanup functionality in LiveOrchestrator.
- Ensure proper registration and unregistration of orchestrators in the registry.
- Test background task tracking and cleanup mechanisms.
- Validate greeting task cancellation during orchestrator cleanup.
- Introduce memory leak detection tests to prevent unbounded growth in orchestrator registry.
- Verify user message history deque is properly bounded and cleared on cleanup.
- Add scenario update tests to ensure correct agent management during updates.
- Optimize hot path functions to ensure non-blocking behavior during network calls.
commit 4af52a50b2c2af562cca78a53f3fa8109cd97571
Merge: 6b01d803 b8b0c279
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Sun Dec 14 19:06:46 2025 -0600
Merge pull request #79 from AIappsGBBFactory/hotfix/agent-context-and-handoffs
Hotfix/agent context and handoffs
commit b8b0c27948ab46474de405ff826b2caf2480c7d4
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 14:56:55 2025 -0600
Refactor tests and dependencies following module renaming and API changes
- Removed pytest-twisted from dev dependencies in pyproject.toml and uv.lock.
- Updated conftest.py to mock configuration and Azure OpenAI client for tests.
- Skipped tests in test_acs_media_lifecycle.py, test_acs_media_lifecycle_memory.py, and test_acs_simple.py due to dependencies on removed/renamed modules.
- Adjusted imports in test_artagent_wshelpers.py for orchestrator path change.
- Skipped tests in test_call_transfer_service.py due to API changes in toolstore.
- Updated datetime usage in test_demo_env_phrase_bias.py to use UTC.
- Modified websocket endpoint assertions in test_realtime.py to reflect new paths.
- Added new test file test_voice_handler_components.py for voice handler components.
commit 39cf1e74bc7894f3d86a43e847083269b093077e
Merge: 6b01d803 e837e43a
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sun Dec 14 13:02:22 2025 -0600
Merge pull request #6 from AIappsGBBFactory/hotfix/agent-context-and-handoffs
Hotfix/agent context and handoffs
commit e837e43a3e8720573878da492da293b854c271de
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:46:17 2025 -0600
Refactor environment variable assignment in deploy workflow for clarity
commit 79cb83af0b74817d826c13497d49eeddff79d49c
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:41:49 2025 -0600
Refactor LiveOrchestrator to prevent duplicate UI updates by omitting redundant session_updated broadcasts during context-only updates.
commit de6557a573bf35ee743433202bba6b71bab8279f
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:30:00 2025 -0600
Enhance LiveOrchestrator to handle context-only session updates without UI broadcasts
commit 6666a9a9f44e2597486b77a5e3a2bc83f655c341
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 12:29:47 2025 -0600
Add error handling for Redis connection issues and implement unit tests for HandoffService
- Enhanced AzureRedisManager to handle RedisClusterException and OSError during client connection attempts.
- Introduced comprehensive unit tests for HandoffService, covering handoff detection, target resolution, handoff resolution, greeting selection, and context building.
- Added tests for HandoffResolution dataclass to ensure correct property behavior and default values.
commit 94e39aedabf146d7312d16a4dc2a8f95da92df70
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 11:59:57 2025 -0600
Refactor code structure for improved readability and maintainability
commit 33b8c7d9a6cb2f56bdeb354d1f29c8f49e13177b
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 14 00:47:20 2025 -0600
feat: Add Scenario Builder component and integrate with RealTimeVoiceApp
- Introduced ScenarioBuilder component for visual orchestration of agent flows.
- Implemented drag-and-drop functionality for agents and handoff configuration.
- Added buttons in RealTimeVoiceApp for accessing Agent and Scenario Builders.
- Enhanced state management for agent scenarios, including creation and updates.
- Integrated new handoff editor for configuring agent interactions.
commit 1b68215a7db47af4cd82efc873d8a236db8942a5
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 23:55:20 2025 -0600
feat: Implement TTS Streaming Latency Analysis and Optimization Plan
- Added a comprehensive document outlining the critical latency issues in TTS playback within the Speech Cascade architecture.
- Identified root causes including processing loop deadlock, sentence buffering delays, queue-based event processing, and full synthesis before streaming.
- Proposed a multi-phase optimization strategy to address identified issues, including:
- Phase 0: Fix processing loop deadlock by creating a dedicated TTS processing task.
- Phase 1: Reduce sentence buffer threshold for earlier TTS chunk dispatch.
- Phase 2: Implement parallel TTS prefetching to synthesize the next sentence while streaming.
- Phase 3: Enable streaming TTS synthesis to stream audio while synthesizing.
- Phase 4: Achieve full pipeline parallelism for LLM to TTS to WebSocket streaming.
- Created a detailed test implementation plan with metrics and success criteria to validate improvements.
test: Add unit tests for HandoffService
- Created unit tests for the HandoffService, covering handoff detection, target resolution, and handoff resolution methods.
- Implemented tests for greeting selection and context building to ensure proper functionality.
- Added tests for the HandoffResolution dataclass to verify properties and default values.
commit 6b01d8031600614788302ef3003fcb6ab8497a2e
Merge: bb3826f4 a5a1d73e
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Sat Dec 13 14:27:08 2025 -0600
Merge pull request #78 from AIappsGBBFactory/feat/troubleshooting-enhancements
commit a5a1d73e2c833eb253ad72f3ab44353edd1b2e84
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:57:18 2025 -0600
Update .github/workflows/test-azd-hooks.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit a23e18744f594477fc014cf2f73c30215754b654
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:52:27 2025 -0600
Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit aaeae928f9da2ae145cc11f6d58f989931d4c52b
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Sat Dec 13 13:51:29 2025 -0600
Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
commit da6ead6f5b53fca1f32ab8fb339c8d993eb43155
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 13:50:27 2025 -0600
feat: Remove outdated troubleshooting documentation for deployment issues
commit b8a16dc733ba8de4ec424ea8786ebebb2de68472
Author: Jin Lee <jinle@microsoft.com>
Date: Sat Dec 13 13:41:13 2025 -0600
feat: Add troubleshooting guidance for Terraform state lock errors and provide remote/local fix options
commit 7f9f889585584fa3f46be4b7a9e49d301c74e2cf
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 21:31:48 2025 -0600
feat: Update Azure deployment workflows and normalize container memory formats
commit cb6354e7462607c8398acdcef687d21c2b92786f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 20:47:45 2025 -0600
feat: Add troubleshooting guidance for MkDocs module errors and update dev dependencies in uv.lock
commit a4a19a33ebef5ad9f85ede7d4fa4486869285863
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 20:27:52 2025 -0600
feat: Enhance deployment scripts with pre/post-provisioning hooks and Azure CLI extension checks
commit 5b7a0979166b514eff9109a7e72a04b734851103
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:39:53 2025 -0600
feat: Add Dev Container testing for AZD hooks with environment validation and summary reporting
commit 4d71fa7d39e1825799a3066a821567aa7a916645
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:35:10 2025 -0600
feat: Update preflight checks to conditionally include Docker in CI mode and log its status
commit e1a840e1a3b4cd0d2d825c82d552f8c5abd2674b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:31:06 2025 -0600
feat: Update test job names for clarity and enhance preflight checks for CI mode
commit 40e33ef7f6ba9c9a61ac035f906b3ad8fa8aa815
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:27:24 2025 -0600
feat: Enhance AZD hook testing with postprovision execution and Azure CLI setup
commit 17da7f785413157274c2e8c33e41c559a4eca098
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:21:40 2025 -0600
feat: Add preprovision hook execution to Linux, macOS, and Windows test jobs in CI workflow
commit d54b9ce2e1d0d39f1a8e5bd4e793c78187e12140
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 18:16:05 2025 -0600
feat: Add troubleshooting steps for "bad interpreter" errors and enhance post-provisioning instructions for phone number configuration
commit a6d737c73a8e66e2750e0fbec357d7a3445d9bd0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:58:57 2025 -0600
fix(ci): simplify test-azd-hooks workflow tests and run in parallel
- Remove fragile grep-based function extraction that caused syntax errors
- Run lint, linux, macos, windows tests in parallel (no dependencies)
- Trigger on all pushes to main/staging (remove path filters for push)
- Simplify backend configuration test to avoid function sourcing issues
commit 46acb1415e5010a6a422df52470e944f29e4d223
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:56:01 2025 -0600
feat: Update branch triggers in workflow to include feat/troubleshooting-enhancements
commit 819dd90f21444df83f93ad16faf55d5c7f3b972c
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 17:51:26 2025 -0600
Enhance provisioning scripts and documentation
- Updated postprovision.sh to clarify phone number provisioning steps and added guidance for obtaining a phone number via Azure Portal.
- Modified preprovision.sh to include preflight checks for tools, authentication, and providers before proceeding with provisioning.
- Added jq as a prerequisite in the getting-started documentation and provided installation instructions for various platforms.
- Created a new TODO-deployfixes.md file to document common issues encountered during deployment sessions, including resolutions for Docker errors, jq installation, and subscription registration.
- Expanded troubleshooting.md with detailed solutions for common deployment and provisioning issues, including authentication mismatches, Docker errors, jq command not found, and ACS phone number prompts.
- Updated variables.tf to improve the description of the voice_live_location variable, including a link to supported Azure regions.
commit bb3826f445e296cb6960cfdb8937275b2a145e55
Merge: 4b10e18d 516b6b6c
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Fri Dec 12 12:41:50 2025 -0600
Merge pull request #77 from AIappsGBBFactory/staging
docs updates and more pipeline related fixes/enhancements
commit 516b6b6c196ea779dde679e19ae79ec2e1d2f1d0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:39:05 2025 -0600
fix: Update README layout for improved clarity and navigation
commit 4b10e18dc77699a66c821f03005a7db3e7b2a337
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Fri Dec 12 12:37:50 2025 -0600
add value
commit fd2ca5acb094ef361bc19f63f716ae15b80e96b7
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:36:45 2025 -0600
fix: Restore header for ARTVoice Accelerator Framework in README
commit 6c33df1eebd99ce4275fc4528e5ecd9c1414220e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:21:26 2025 -0600
fix: Update README layout and enhance navigation links for better user experience
commit 9167d13735bb18ea38f1468b79cac3f1ebd2096a
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 12:09:40 2025 -0600
feat: Update documentation workflow name and enhance README with deployment badges
commit e170dd435528384a40a8af269130fd0aeb0f18c1
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:51:53 2025 -0600
fix: Add emoji to workflow names for better visibility
commit c913abefa7149407c4fe18f69733189a2c3f9ba9
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:31:13 2025 -0600
adding rg as env var set at the gh env level
commit 7251f57308443dfc0392e74198ee640eef539b05
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:26:23 2025 -0600
feat: Add optional GitHub PAT secret and enhance environment variable handling for Azure deployment
commit 1b077d15d652d1cc3453f1440d8e6917b8a73bb3
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 11:00:16 2025 -0600
fix: Update GitHub token to use secrets for enhanced security
commit 5391566295a7d967946bc0875a2bc90bd8ecbbaa
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 10:42:42 2025 -0600
fix: Update condition for output extraction in deployment workflow
commit f9cf3c83a35643b2325a23e3091355270a761f04
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 10:12:23 2025 -0600
fix: Add run-name to the Azure deployment workflow for better clarity
commit f051b1387b6daacb84ccbfbb8a54519f5458a99d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Fri Dec 12 09:51:54 2025 -0600
refactor: replace deploy-azd workflow with reusable template and remove redundant summary job
- Updated the deployment workflow name to "Deploy to Azure".
- Replaced the usage of the old deploy-azd.yml with a new reusable template _template-deploy-azd.yml.
- Removed the deployment summary job and its associated steps to streamline the workflow.
commit b99084b98a6dba3b4b7a03b897e4bd9950e8b01f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 22:53:40 2025 -0600
docs: Update documentation links and improve clarity across various guides
commit 2c88c9acb23a3c8563f482db7a8ce323a48b11fd
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 22:25:29 2025 -0600
feat: Enhance API documentation and tagging for better clarity and organization
commit db05f3aa81db340548234d88d32a10512fa3b1eb
Merge: 0787f03f 598604c0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:53:03 2025 -0600
Merge branch 'staging' of github.com:aiappsgbbfactory/art-voice-agent-accelerator into staging
commit 0787f03fb9e60f53cc15d88f14233dd900152c9b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:52:32 2025 -0600
fix: Update documentation URLs to reflect new repository location
commit 598604c0de3c1933108d117c0475920ca6a1f886
Merge: b36476ca f8904934
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 11 19:27:17 2025 -0800
Merge pull request #2 from AIappsGBBFactory/v2/deploy-fixes
improvements flow
commit 0ed67a24dfe7aef67b7ad9c937ae5db8a70999f7
Merge: 38d9ca97 f8904934
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 11 21:25:16 2025 -0600
Merge pull request #76 from AIappsGBBFactory/v2/deploy-fixes
commit 38d9ca979e21ea73311806a83ddcd136be4095cd
Merge: b36476ca 6dc26a3e
Author: Jin Lee <94473824+JinLee794@users.noreply.github.com>
Date: Thu Dec 11 19:22:18 2025 -0800
Merge pull request #75 from Azure-Samples/v2-pablosal
V2 pablosal
commit f89049349f864ff2aa6f361f643d6fb54e5d7fce
Merge: 3a2eee5c 6dc26a3e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:18:47 2025 -0600
Merge branch 'v2-pablosal' into v2/deploy-fixes
commit 3a2eee5cbf555628ede657fce81847dd539a1ad2
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 21:18:40 2025 -0600
fix: Update name from rtaudioagent to artaudioagent in environment parameter files
commit 5abe9e52786d9f9bed4b0ba855e4bd8a3785d434
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:49:22 2025 -0600
fix: Update name from rtaudioagent to artaudioagent in environment parameter files
commit 4f7796e2553c8af09ecad3f556b1098e3ee549af
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:42:19 2025 -0600
fix: Update naming conventions for foundry account and project variables in locals
commit 6dc26a3e61f67ab76fac21e7e20ad6cb91f1d27f
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 20:38:23 2025 -0600
add value
commit 56cb1d267d5287519d77b120371b2f829fa378a5
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 20:08:42 2025 -0600
fix: Implement auto-selection and timeout for user input in setup scripts
commit cc597f3867bced6279a242818d016ad2362aa853
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 19:14:50 2025 -0600
improvements flow
commit b1e0ad34fcedcfcdf6a4224a1e81ca8e35287074
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 19:12:49 2025 -0600
refactor: Remove unused backend API public URL and source phone number from environment parameter files
commit 33602eb894cd1c95fa8f04dee54af0942e374af0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 19:12:44 2025 -0600
refactor: Remove unused backend API public URL variable and related validation
commit 07ac39faee035c1bc77e2e81379e2100ca866023
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 18:56:58 2025 -0600
fix: Use coalesce function for location assignment in storage account resource
commit f1f827a742b8a78ade67de786e6d78394278be04
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 18:43:04 2025 -0600
fix: Skip null values when loading static parameters from tfvars file to use Terraform defaults
commit 82c8bf46df1948c2d5fe05db75d2af19dbddba6f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:32:12 2025 -0600
refactor: Simplify state key handling in provider configuration by using environment name
commit 7b9cd66693b4ae86c53f6039c0b36d9fe0fac00e
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:24:29 2025 -0600
fix: Ensure principal_id logging does not fail and handle local_state retrieval correctly
commit ed31e932febda5b8c94809965ffa1158d7198d73
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 16:20:43 2025 -0600
refactor: Remove outdated AZD deployment workflow and update documentation links for clarity
commit 29925e9ef2e2dd44b94169682c899621e4c2ee1f
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:51:58 2025 -0600
fix: Update health check endpoint in postprovision script to use correct API path
commit 0f4a0f99ccf7366d4e473cb18c16fca9052cd597
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:45:03 2025 -0600
feat: Allow mkdocs build to proceed with warnings by removing --strict flag
commit 76c89462941e9f4a5b051736790433fcb05117af
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:42:33 2025 -0600
feat: Add site_dir configuration to mkdocs.yml for improved site structure
commit 34936afc98757908a0deae9d2710bc1a6457703d
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:38:46 2025 -0600
modified docs workflow
commit da10e6e258a015bcdba373ae713aa9a8b5377fd6
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:36:06 2025 -0600
feat: Revise documentation deployment workflow to enhance dependency management and streamline build process
commit a64e7fa4ab94ccbeac0fd9a4afb0cf82e77fb8ff
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:30:41 2025 -0600
feat: Refactor app configuration handling to prioritize .env.local overrides and improve environment variable management
commit 200c14026da0deb56bef3813304eea6075091b69
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:15:37 2025 -0600
feat: Update audio handling and documentation dependencies for improved installation and error handling
commit e83c84f23790b3387d5da440e67d84d98ca1960b
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 15:07:12 2025 -0600
cicd test for azd deploy
commit 4eb33fa287cb8bc664118439fdd9902c0658d0d8
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 14:55:59 2025 -0600
feat: Implement scenario-based handoff map resolution for orchestrator configuration
commit f97621578bfe54ab06a5fd50c2772f99acd63834
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 14:29:14 2025 -0600
feat: Enhance scenario loading to support orchestration.yaml naming convention
commit 3203b58f12850d20b06c0b116385d424dd157641
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 13:54:57 2025 -0600
feat: Add Concierge agent configuration and prompts for banking scenarios
- Introduced a new YAML configuration for the Concierge agent, defining its voice, model, session, and tool configurations.
- Created a comprehensive prompt file for the Concierge agent, detailing voice and language settings, identity and trust guidelines, and operational modes.
- Implemented scenario orchestration analysis to address issues with agent initialization and fallback logic, ensuring the correct agent is set for banking scenarios.
- Renamed orchestration.yaml to scenario.yaml for consistency in scenario loading.
- Updated default start agent to BankingConcierge and added validation for agent existence at startup.
commit 3c402dcbcbf36ad2ffcde6799b613d88610a9fd0
Author: Jin Lee (HLS US SE) <jin.lee+microsoft@microsoft.com>
Date: Thu Dec 11 11:34:05 2025 -0600
Enhance Terraform configuration and scripts for Voice Live integration
- Update Dockerfile to install dependencies and set up virtual environment.
- Modify initialize-terraform.sh and local-dev-setup.sh for improved script handling.
- Refactor sync-appconfig.sh to streamline key-value imports and feature flag management.
- Add provider.conf.json generation for remote state backend configuration.
- Update main.tf and outputs.tf to support new Voice Live model deployments.
- Introduce voice_live_location and voice_live_model_deployments variables in variables.tf.
commit 3365e8b4836dbc77b48c5c207c0f093c4518eeb4
Merge: d8a10ddc 313a39d4
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Thu Dec 11 00:38:58 2025 -0600
Merge pull request #73 from AIappsGBBFactory/v2/jinle-more-fixes
Updated documentation based on new scenario-based structure, added proposal for speckit integration
commit d8a10ddc9f00198cff3b03a1d28286c235f519b8
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Thu Dec 11 00:37:50 2025 -0600
add value
commit 313a39d42078245ee9b0fe983a53066c355b5f23
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 20:53:29 2025 -0600
Add integration proposal for Spec-Driven Development methodology in ARTVoice
commit 9b9b7a4841cc52a0a9f62e01ac7ee731b8804abd
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 20:32:28 2025 -0600
Add Insurance Voice Agent Scenario documentation and update navigation
- Introduced a comprehensive guide for the Insurance Customer Service Scenario, detailing the security-focused multi-agent voice system for claims processing, fraud detection, and policy management.
- Updated mkdocs.yml to include the new Insurance documentation in the Industry Solutions section.
commit 872732a1b8cb6935d27c50dd750654a8677dc972
Merge: 05160950 7e466729
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 19:30:09 2025 -0600
Merge branch 'v2-pablosal' of github.com:azure-samples/art-voice-agent-accelerator into v2/jinle-more-fixes
commit 051609509bf07e863b199fcf7090a0049ad17667
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 19:26:01 2025 -0600
Refactor agent paths and update documentation for agent discovery and configuration
commit 7e4667298ca253a9bce3472c584d9443b67b244e
Merge: ae5bb836 51c850c6
Author: Pablo Salvador Lopez <31255154+pablosalvador10@users.noreply.github.com>
Date: Wed Dec 10 18:09:25 2025 -0600
Merge pull request #72 from AIappsGBBFactory/v2-pablosal-merge-deployment-changes
V2 pablosal merge deployment optimizations branch
commit ae5bb83671e92ef2dddb28325016fdeee6bfc5f5
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 10 18:08:56 2025 -0600
add
commit 51c850c69e644a7a06cc62312bb4dd0259a6c9f6
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 17:57:04 2025 -0600
fix voicelive output attributes
commit a641e4227262ec1f5412dba8e4c7e04b0611dfb6
Merge: 0a2ced23 b6416043
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 13:26:52 2025 -0800
Merge branch 'merge/v2/deployment-optimizations' into v2-pablosal-merge-deployment-changes
commit 0a2ced2376934167731b5765e5b03e306e349d4d
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 12:57:18 2025 -0800
Enhance configuration loading with .env.local support and update documentation
commit b64160439878128a5c45b8d6f4cbbb56b6dbf8a5
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 11:39:59 2025 -0800
feat: Add app configuration bootstrap to initialize environment variables
commit 5d7135fc424713dc7db1544858cdebd279c41ea1
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Wed Dec 10 13:24:43 2025 -0600
last changes
commit c540c2141b46393f6a7e168f69402e3426a866a0
Author: Jin Lee <jinle@microsoft.com>
Date: Wed Dec 10 10:12:41 2025 -0800
feat: Enhance Azure Voice Live integration and refactor configuration management
commit f6ec1bbd5d4c11c8f2d8f4e26a5e872cc64a6489
Author: Jin Lee <jinle@microsoft.com>
Date: Tue Dec 9 23:21:11 2025 -0800
Refactor Terraform configuration:
- Update main.tf to adjust foundry account and project naming conventions.
- Remove feature flags and keys from appconfig module as they are now managed externally.
- Clean up variables.tf by removing unused variables and updating descriptions.
- Delete provider configuration file as it is no longer needed.
- Change default application name from "rtaudioagent" to "artagent" and adjust related settings.
- Modify connection settings and pool sizes for improved performance.
commit fc1815100982a4e81d88fd9ef87357ba7a1fbde2
Author: pablosalvador10 <pablosalvadorlopez11@gmail.com>
Date: Tue Dec 9 23:58:52 2025 -0600
add value
commit eaef115e37f0474e85ea7d9143547f81f1cdeebc
Author: Jin Lee <jinle@microsoft.com>
Date: Mon Dec 8 01:29:33 2025 -0800
refactor: Update architecture diagram in README for clarity and consistency in orchestration modes
commit 3197e0cb64b8e90a688380cd412f19fea0085bff
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 23:28:56 2025 -0700
removing unused dependency in src/herlpers.py
commit 7ad2546503ccb6cf203335de754102bfe67d21f3
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 23:28:13 2025 -0700
further deployment cleanup, docs update/tweaks, adding more todos
commit 7893afd6542da8aa742e9217ca2914beb3a54443
Author: Jin Lee <jinle@microsoft.com>
Date: Sun Dec 7 15:47:36 2025 -0600
feat: Update launch configuration and scripts to use virtual environment with uv; enhance README for deployment clarity
commit a2e370fccdd0ed4c9a6a1d94ec5e082c0bcee1c3
Author: Jin Lee <jinle@microsoft.com>
…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…erarchy
Infrastructure Changes:
Speech Services & Dependencies:
Orchestrator & Token Tracking:
Benefits: