Claude/voice application setup 011 cv57 d2by dc qc yu puw fye1

[bon-zai]

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (does not change functionality, e.g. code style improvements, linting)
• Documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Please delete options that are not relevant.

• Unit Test
• Test Script (please provide)

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Maintainer Checklist

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Made sure Checks passed

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull Request Overview

This PR introduces a comprehensive dual-agent AI voice assistant application with real-time WebSocket communication, 3D animated visualization, and MCP integration. The application supports two voice AI providers: ElevenLabs (with Claude AI and Whisper STT) and Qwen 3 Omni (Alibaba's multimodal voice AI).

Key Changes:

• Full-stack voice application with Next.js frontend and Node.js backend
• WebSocket-based real-time bidirectional audio streaming
• Integration with Claude AI, Whisper, ElevenLabs, and Qwen Omni APIs
• MCP (Model Context Protocol) support for mem0 memory and Tavily search
• 3D animated orb visualization using Three.js/React Three Fiber

Reviewed Changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
voice-app/frontend/package.json	Frontend dependencies including Next.js 15, React 19, Three.js for 3D visualization
voice-app/frontend/tsconfig.json	TypeScript configuration for Next.js with bundler module resolution
voice-app/frontend/lib/types.ts	Type definitions for WebSocket messages, voice agents, and chat data
voice-app/frontend/hooks/useWebSocket.ts	Custom hook for WebSocket connection and message handling
voice-app/frontend/hooks/useAudio.ts	Audio recording and playback functionality using Web Audio API
voice-app/frontend/components/VoiceAssistant.tsx	Main component orchestrating voice interaction and UI state
voice-app/frontend/components/VoiceAgentToggle.tsx	UI component for switching between ElevenLabs and Qwen agents
voice-app/frontend/components/ChatUI.tsx	Chat interface displaying conversation history
voice-app/frontend/components/AnimatedOrb.tsx	3D visualization with particles that change color based on speaker
voice-app/backend/package.json	Backend dependencies including Anthropic SDK, OpenAI, MCP SDK
voice-app/backend/tsconfig.json	TypeScript configuration for Node.js backend
voice-app/backend/src/types/index.ts	Backend type definitions matching frontend types
voice-app/backend/src/websocket/server.ts	WebSocket server setup with session management and heartbeat
voice-app/backend/src/websocket/handlers.ts	Message routing and processing for audio chunks and agent switching
voice-app/backend/src/voice/stt.ts	OpenAI Whisper integration for speech-to-text
voice-app/backend/src/voice/tts.ts	ElevenLabs integration for text-to-speech synthesis
voice-app/backend/src/voice/qwen-omni.ts	Qwen 3 Omni real-time voice provider with WebSocket connection
voice-app/backend/src/voice/pipeline.ts	Voice processing pipeline orchestrating STT, LLM, and TTS
voice-app/backend/src/llm/claude.ts	Claude AI integration with MCP tools for memory and search
voice-app/backend/src/mcp/client.ts	MCP client manager for connecting to MCP servers
voice-app/backend/src/mcp/servers.ts	Configuration for mem0 and Tavily MCP servers
voice-app/docker-compose.yml	Docker Compose setup for running frontend and backend services
voice-app/SETUP.md	Comprehensive setup guide with environment variables and troubleshooting
voice-app/README.md	Project overview with quick start instructions
voice-app/ARCHITECTURE.md	Detailed architecture documentation with data flow diagrams

Comments suppressed due to low confidence (1)

voice-app/frontend/hooks/useWebSocket.ts:31

• Unused variable audioQueueRef.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Security concern: The WebSocket server accepts connections without any authentication or origin validation. In production, you should implement:

1. Origin validation to prevent unauthorized cross-origin connections
2. Authentication tokens or API keys
3. Rate limiting per connection
4. CORS restrictions

Consider adding a verifyClient callback to the WSServer configuration:

this.wss = new WSServer({
  port: this.port,
  maxPayload: Number(process.env.WS_MAX_PAYLOAD) || 10 * 1024 * 1024,
  verifyClient: (info) => {
    // Verify origin, authentication token, etc.
    const origin = info.origin;
    // Add your validation logic here
    return true; // or false to reject
  }
});

Suggested change

	maxPayload: Number(process.env.WS_MAX_PAYLOAD) || 10 * 1024 * 1024, // 10MB
	maxPayload: Number(process.env.WS_MAX_PAYLOAD) || 10 * 1024 * 1024, // 10MB
	verifyClient: (info, done) => {
	// Origin validation
	const allowedOrigins = (process.env.WS_ALLOWED_ORIGINS || '').split(',').map(o => o.trim()).filter(o => o);
	const origin = info.origin;
	if (allowedOrigins.length > 0 && !allowedOrigins.includes(origin)) {
	logger.warn(`WebSocket connection rejected due to invalid origin: ${origin}`);
	return done(false, 403, 'Forbidden');
	}
	// Token validation (query param)
	const url = require('url');
	const parsedUrl = url.parse(info.req.url || '', true);
	const token = parsedUrl.query && parsedUrl.query.token;
	const expectedToken = process.env.WS_AUTH_TOKEN;
	if (expectedToken && token !== expectedToken) {
	logger.warn(`WebSocket connection rejected due to invalid token: ${token}`);
	return done(false, 401, 'Unauthorized');
	}
	// Passed all checks
	return done(true);
	}

[Copilot]

Potential memory leak: Event listeners are registered on the qwenProvider using .on() for response_text and audio_chunk events, but these are never removed. If this method is called multiple times (e.g., user sends multiple messages), listeners will accumulate, causing memory leaks and potentially duplicate event handling.

Either:

1. Use .once() instead of .on() if you only expect one event per request
2. Remove the listeners after the response_complete event using .off() or .removeAllListeners()
3. Create a new provider instance per request instead of reusing the same one

Example fix:

const handlers = {
  responseText: (text: string) => { /* ... */ },
  audioChunk: (audioChunk: Buffer) => { /* ... */ }
};

qwenProvider.on('response_text', handlers.responseText);
qwenProvider.on('audio_chunk', handlers.audioChunk);

qwenProvider.once('response_complete', () => {
  // Clean up listeners
  qwenProvider.off('response_text', handlers.responseText);
  qwenProvider.off('audio_chunk', handlers.audioChunk);
  // ... rest of the code
});

Suggested change

	qwenProvider.on('response_text', (text: string) => {
	fullResponse += text;
	this.sendMessage(session.ws, WebSocketMessageType.AI_RESPONSE_TEXT, {
	text: fullResponse,
	messageId,
	});
	});
	qwenProvider.on('audio_chunk', (audioChunk: Buffer) => {
	this.sendMessage(session.ws, WebSocketMessageType.AI_RESPONSE_AUDIO, {
	audio: audioChunk.toString('base64'),
	messageId,
	isLast: false,
	});
	});
	qwenProvider.once('response_complete', () => {
	// Store handler references for cleanup
	const responseTextHandler = (text: string) => {
	fullResponse += text;
	this.sendMessage(session.ws, WebSocketMessageType.AI_RESPONSE_TEXT, {
	text: fullResponse,
	messageId,
	});
	};
	const audioChunkHandler = (audioChunk: Buffer) => {
	this.sendMessage(session.ws, WebSocketMessageType.AI_RESPONSE_AUDIO, {
	audio: audioChunk.toString('base64'),
	messageId,
	isLast: false,
	});
	};
	qwenProvider.on('response_text', responseTextHandler);
	qwenProvider.on('audio_chunk', audioChunkHandler);
	qwenProvider.once('response_complete', () => {
	// Clean up listeners to prevent memory leaks
	qwenProvider.off('response_text', responseTextHandler);
	qwenProvider.off('audio_chunk', audioChunkHandler);

[Copilot]

The cn utility function is imported from @/lib/utils but this file is missing from the repository. You need to create voice-app/frontend/lib/utils.ts with the following implementation:

import { type ClassValue, clsx } from 'clsx';
import { twMerge } from 'tailwind-merge';

export function cn(...inputs: ClassValue[]) {
  return twMerge(clsx(inputs));
}

This utility is commonly used for merging Tailwind CSS classes and is required by VoiceAssistant.tsx, VoiceAgentToggle.tsx, and ChatBubble.tsx.

[Copilot]

Inconsistent naming: The product name is "Qwen" (all lowercase except the first letter), not "QWen". Throughout the codebase and documentation, you should use the official spelling "Qwen 3 Omni" or "Qwen2.5-Omni" for consistency.

[Copilot]

Potential race condition: If the WebSocket connection fails immediately after creation but before the 'error' handler is registered, or if multiple errors occur rapidly, the promise could be rejected multiple times. Consider adding a flag to ensure the promise is only resolved/rejected once, or use a timeout to handle connection that never opens.

[Copilot]

Using require() for importing uuid is inconsistent with the ES module imports used elsewhere in the file. Change this to:

import { v4 as uuidv4 } from 'uuid';

And then use uuidv4() instead of require('uuid').v4(). Note that uuid is already imported at the top of the pipeline.ts file this way.

[Copilot]

Inconsistent naming: The product name is "Qwen" (all lowercase except the first letter), not "QWen". Throughout the codebase and documentation, you should use the official spelling "Qwen 3 Omni" or "Qwen2.5-Omni" for consistency.

Suggested change

	<span>QWen 3 Omni</span>
	<span>Qwen 3 Omni</span>

[Copilot]

The audioQueue field is declared but never used anywhere in this class. Consider removing it or implementing the intended audio queue functionality if it was meant to buffer audio chunks.

Suggested change

private audioQueue: Buffer[] = [];

[Copilot]

The voice recording button is missing accessibility attributes. Add aria-label, aria-pressed, and role attributes to improve screen reader support:

<button
  onMouseDown={handleMouseDown}
  onMouseUp={handleMouseUp}
  onTouchStart={handleMouseDown}
  onTouchEnd={handleMouseUp}
  disabled={!isConnected || status === 'processing' || status === 'speaking'}
  aria-label={isRecording ? 'Recording - Release to send' : 'Hold to record voice message'}
  aria-pressed={isRecording}
  role="button"
  className={cn(
    // ... rest of classes
  )}
>

[Copilot]

Unused import VoiceAgent.

Suggested change

VoiceAgent,