Add Web Audio API as LLM configuration for catalog loader

[jchris]

Problem

The current LLM catalog system only supports text and image generation models, but lacks support for audio generation and processing capabilities. This limits the types of AI-powered applications that can be built within the vibes ecosystem.

Current LLM Configuration Structure

The app currently loads LLM configurations from JSON files in app/llms/:

app/llms/
├── callai.json     # Text generation models
├── fireproof.json  # Database-related configs
└── image-gen.json  # Image generation models

Proposed Addition

Add a new web-audio.json configuration file to support audio generation and processing models:

{
  "name": "Web Audio APIs",
  "description": "Audio generation and processing models",
  "models": [
    {
      "id": "elevenlabs-speech",
      "name": "ElevenLabs Text-to-Speech",
      "description": "High-quality voice synthesis",
      "type": "audio-generation",
      "category": "speech",
      "endpoint": "https://api.elevenlabs.io/v1/text-to-speech",
      "inputTypes": ["text"],
      "outputTypes": ["audio/mpeg", "audio/wav"],
      "capabilities": ["text-to-speech", "voice-cloning"],
      "pricing": {
        "model": "per-character",
        "free_tier": 10000,
        "paid_tiers": ["starter", "creator", "pro"]
      }
    },
    {
      "id": "openai-whisper",
      "name": "OpenAI Whisper",
      "description": "Speech recognition and transcription",
      "type": "audio-processing",
      "category": "transcription",
      "endpoint": "https://api.openai.com/v1/audio/transcriptions",
      "inputTypes": ["audio/mpeg", "audio/wav", "audio/mp4"],
      "outputTypes": ["text"],
      "capabilities": ["speech-to-text", "translation", "language-detection"],
      "pricing": {
        "model": "per-minute",
        "rate": 0.006
      }
    },
    {
      "id": "musicgen",
      "name": "Meta MusicGen",
      "description": "AI music generation from text prompts",
      "type": "audio-generation",
      "category": "music",
      "endpoint": "https://api.replicate.com/v1/predictions",
      "inputTypes": ["text"],
      "outputTypes": ["audio/wav"],
      "capabilities": ["music-generation", "melody-continuation"],
      "pricing": {
        "model": "per-generation",
        "rate": 0.025
      }
    },
    {
      "id": "bark-audio",
      "name": "Suno Bark",
      "description": "Text-to-audio generation with voice effects",
      "type": "audio-generation", 
      "category": "speech-effects",
      "endpoint": "https://api.replicate.com/v1/predictions",
      "inputTypes": ["text"],
      "outputTypes": ["audio/wav"],
      "capabilities": ["text-to-speech", "sound-effects", "background-music"],
      "pricing": {
        "model": "per-generation",
        "rate": 0.035
      }
    }
  ],
  "categories": {
    "speech": {
      "name": "Speech Synthesis",
      "description": "Text-to-speech and voice generation"
    },
    "transcription": {
      "name": "Speech Recognition",
      "description": "Audio-to-text transcription and analysis"
    },
    "music": {
      "name": "Music Generation",
      "description": "AI-powered music and sound creation"
    },
    "speech-effects": {
      "name": "Audio Effects",
      "description": "Voice effects and audio manipulation"
    }
  },
  "use_cases": [
    "Voice-over generation for videos",
    "Podcast transcription and editing",
    "Music creation for games and apps",
    "Accessibility features (text-to-speech)",
    "Language learning applications",
    "Audio content analysis",
    "Real-time voice effects",
    "Audio-based chatbots and assistants"
  ]
}

Benefits

For Developers

• Audio-First Apps: Enable creation of voice-based applications
• Accessibility: Built-in text-to-speech and speech-to-text capabilities
• Creative Tools: Music and audio effect generation for creative apps
• Multimodal Experiences: Combine text, image, and audio in single applications

For Users

• Voice Interfaces: Natural voice interaction with AI applications
• Audio Content: Podcast generators, voice-over tools, music creators
• Accessibility: Screen readers, voice navigation, audio descriptions
• Entertainment: Interactive audio experiences, voice games

For Ecosystem

• Model Diversity: Expand beyond text/image to audio capabilities
• Market Expansion: Reach audio-focused use cases and industries
• Innovation: Enable new types of AI applications
• Standardization: Consistent audio API patterns across the platform

Implementation Details

Catalog Loader Integration

// Extend existing catalog loader to include audio models
interface AudioModel extends BaseModel {
  type: 'audio-generation' | 'audio-processing';
  category: 'speech' | 'transcription' | 'music' | 'speech-effects';
  inputTypes: string[];
  outputTypes: string[];
  capabilities: string[];
}

Web Audio API Integration

// Add audio handling utilities
export interface AudioConfig {
  sampleRate?: number;
  channels?: number;
  bitDepth?: number;
  format: 'wav' | 'mp3' | 'ogg';
}

export async function generateAudio(
  prompt: string,
  modelId: string,
  config?: AudioConfig
): Promise<AudioBuffer> {
  // Audio generation implementation
}

export async function processAudio(
  audioBuffer: AudioBuffer,
  modelId: string,
  operation: string
): Promise<string | AudioBuffer> {
  // Audio processing implementation
}

Component Integration

// Audio player/recorder components for vibes apps
export function AudioPlayer({ src, controls }: AudioPlayerProps) {
  // Web Audio API integration
}

export function AudioRecorder({ onRecording }: AudioRecorderProps) {
  // MediaRecorder API integration
}

Technical Considerations

Browser Compatibility

• Web Audio API support across modern browsers
• Fallbacks for audio format compatibility
• Progressive enhancement for audio features

Performance

• Streaming audio for large generations
• Audio compression and optimization
• Efficient audio buffer management

Security

• CORS handling for audio resources
• Secure audio upload/download
• User permission handling for microphone access

Use Cases and Examples

Voice-Powered Chat Assistant

// Example vibe using audio LLMs
function VoiceChatBot() {
  const [isListening, setIsListening] = useState(false);
  const [response, setResponse] = useState('');
  
  const handleSpeech = async (audioBuffer) => {
    // Use Whisper for speech-to-text
    const text = await processAudio(audioBuffer, 'openai-whisper', 'transcribe');
    
    // Generate text response
    const reply = await generateText(text, 'claude-sonnet-4');
    
    // Convert to speech
    const audioResponse = await generateAudio(reply, 'elevenlabs-speech');
    
    // Play audio response
    playAudio(audioResponse);
  };
  
  return (
    <div>
      <AudioRecorder onRecording={handleSpeech} />
      <AudioPlayer src={response} autoPlay />
    </div>
  );
}

Music Generation Tool

function MusicGenerator() {
  const [prompt, setPrompt] = useState('');
  const [generatedMusic, setGeneratedMusic] = useState(null);
  
  const generateMusic = async () => {
    const music = await generateAudio(prompt, 'musicgen', {
      format: 'wav',
      sampleRate: 44100
    });
    setGeneratedMusic(music);
  };
  
  return (
    <div>
      <input 
        value={prompt} 
        onChange={(e) => setPrompt(e.target.value)}
        placeholder="Describe the music you want..."
      />
      <button onClick={generateMusic}>Generate Music</button>
      {generatedMusic && <AudioPlayer src={generatedMusic} controls />}
    </div>
  );
}

Migration and Rollout

Phase 1: Configuration Setup

1. Create web-audio.json configuration file
2. Update catalog loader to include audio models
3. Add basic audio types and interfaces

Phase 2: Core Audio Support

1. Implement audio generation/processing functions
2. Add Web Audio API utilities
3. Create basic audio components

Phase 3: Model Integration

1. Integrate first audio model (e.g., ElevenLabs TTS)
2. Test end-to-end audio workflows
3. Create example audio-powered vibes

Phase 4: Ecosystem Expansion

1. Add multiple audio model providers
2. Create audio component library
3. Document audio development patterns

Success Metrics

• Number of audio-powered vibes created
• Audio model API usage and adoption
• Developer feedback on audio tooling
• User engagement with audio features
• Performance benchmarks for audio operations

Related Issues

• Enhanced multimedia capabilities across the platform
• Accessibility improvements with audio support
• Creative tooling expansion beyond text/image

Next Steps

1. Research audio model APIs and pricing
2. Design audio configuration schema
3. Create proof-of-concept audio integration
4. Plan audio component architecture
5. Test browser audio API compatibility

[jchris]

JavaScript Documentation: Using Web Audio APIs

The Web Audio API is a powerful browser feature enabling dynamic audio processing and synthesis. Below is a concise documentation with code examples to help you use core functionalities of the Web Audio API in JavaScript.

---

1. Getting Started: Creating an AudioContext

The AudioContext is the main interface for working with audio.

// Create an AudioContext
const audioCtx = new (window.AudioContext || window.webkitAudioContext)();

---

2. Playing a Simple Tone

You can generate and play sounds using oscillators.

// Create an oscillator node
const oscillator = audioCtx.createOscillator();
oscillator.type = 'sine'; // Types: 'sine', 'square', 'triangle', 'sawtooth'
oscillator.frequency.value = 440; // Frequency in hertz (A4 note)

// Connect oscillator to destination (speakers)
oscillator.connect(audioCtx.destination);

// Play the tone
oscillator.start();
// Stop after 1 second
oscillator.stop(audioCtx.currentTime + 1);

---

3. Loading and Playing an Audio File (BufferSource)

// Fetch audio file and decode
fetch('path/to/audio.mp3')
  .then(response => response.arrayBuffer())
  .then(arrayBuffer => audioCtx.decodeAudioData(arrayBuffer))
  .then(audioBuffer => {
    const source = audioCtx.createBufferSource();
    source.buffer = audioBuffer;
    source.connect(audioCtx.destination);
    source.start();
  });

---

4. Applying Effects: Gain (Volume Control)

// Create a gain node for volume control
const gainNode = audioCtx.createGain();
gainNode.gain.value = 0.5; // Volume, range 0.0 to 1.0

oscillator.connect(gainNode);
gainNode.connect(audioCtx.destination);

oscillator.start();
oscillator.stop(audioCtx.currentTime + 1);

---

5. Creating a Filter Effect

// Create a BiquadFilterNode for effects like bass/treble
const filter = audioCtx.createBiquadFilter();
filter.type = 'lowpass'; // Filter type: 'lowpass', 'highpass', etc.
filter.frequency.value = 1000; // Frequency cutoff (Hz)

oscillator.connect(filter);
filter.connect(audioCtx.destination);

oscillator.start();
oscillator.stop(audioCtx.currentTime + 1);

---

6. Visualizing Audio (AnalyserNode)

const analyser = audioCtx.createAnalyser();
oscillator.connect(analyser);
analyser.connect(audioCtx.destination);

// Get frequency data
const dataArray = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(dataArray);
// Use dataArray to visualize frequencies

---

Additional Tips

• Always resume the AudioContext on user interaction due to browser restrictions:

  document.body.addEventListener('click', () => audioCtx.resume());

• Clean up disconnected nodes to avoid memory leaks.
• Check compatibility: Not all Web Audio features are supported in every browser.

---

This documentation provides foundational examples for the Web Audio API—ideal for building interactive, audio-rich web applications. For more details, refer to official documentation or MDN Web Docs.

Sources

[jchris]

@CharlieHelps add how to make an echo with filter / distortion / reverb in the feedback loop and and reply mic audio with a metronome, as a comment.

[charliecreates]

Expand this to see my work.

• Drafted the opening section summarizing the comment's goals and linking to the earlier foundational Web Audio API documentation comment on Issue Add Web Audio API as LLM configuration for catalog loader #228.
• Completed Section 1, describing the echo effect node graph with feedback-loop effects (filter, distortion, reverb), all parameter names, a schematic, example signal flow, and annotated sample JavaScript code with feedback safeguards.
• Outlined Section 2, covering microphone audio capture (permissions, node graph), constraints, metronome click generation and scheduling using AudioContext timestamps, node graph for mic + metronome mixing, and included code for media access and timed click creation.
• Authored Section 3 to clarify ambiguity about "reply mic audio with a metronome" and requested confirmation if alternate alignment or call/response behavior is required.
• Prepared Section 4 summarizing browser/user gesture requirements, asset loading considerations for convolver buffers, need for CORS on impulse responses, and linked back to the initial API comment for baseline setup.
• Integrated all sections into a single, formatted GitHub comment draft with section headings, bullet-lists, diagrams, and code blocks for direct use on Issue Add Web Audio API as LLM configuration for catalog loader #228.

[charliecreates]

Follow‑up to the “JavaScript Documentation: Using Web Audio APIs” primer above. This comment shows: (1) how to build an echo with a feedback loop that includes filter/distortion/reverb, and (2) how to monitor/replay microphone audio mixed with a metronome click. See the earlier foundation here: JavaScript Documentation: Using Web Audio APIs.

1) Echo with feedback loop (effects inside the loop)

Node graph (names are exact):

• AudioBufferSourceNode (source)
  • Dry path: source → dryGain:GainNode → destination
  • Wet path:
    • source → delay:DelayNode
    • Delay output to speakers: delay → wetGain:GainNode → destination
    • Feedback loop with effects in the loop:
      • delay → filter:BiquadFilterNode → distortion:WaveShaperNode → reverb:ConvolverNode → feedbackGain:GainNode → delay

Parameters to expose (no fixed values prescribed):

• delay.delayTime (seconds)
• feedbackGain.gain (0–1; must remain < 1.0)
• filter.type, filter.frequency
• distortion.curve (Float32Array)
• convolver.buffer (impulse response AudioBuffer)
• wetGain.gain, dryGain.gain

Notes:

• Prevent runaway: keep feedbackGain.gain < 1.0, and ensure delay.delayTime > 0 (zero‑delay cycles are disallowed).
• ConvolverNode requires a loaded/decoded impulse response (AudioBuffer) before use.

Code: construct echo with effects-in-feedback

// Assumes a user gesture has already resumed the context: await audioCtx.resume()
const audioCtx = new AudioContext();

// Example: source from an AudioBuffer you decoded earlier
const source = audioCtx.createBufferSource();
source.buffer = /* your AudioBuffer */ null; // assign your buffer

// Tunable parameters (examples shown; wire these to UI in your app)
const params = {
  delayTime: 0.35,           // seconds
  feedback: 0.5,             // 0..1, keep < 1.0
  filterType: 'lowpass',     // e.g., 'lowpass' | 'highpass' | ...
  filterFrequency: 8000,     // Hz
  distortionCurve: null,     // Float32Array (e.g., precomputed curve)
  irBuffer: null,            // Convolver IR AudioBuffer (load via fetch+decode)
  wet: 0.4,
  dry: 1.0,
};

// Core nodes
const delay = audioCtx.createDelay(5.0);             // maxDelayTime = 5s (adjust as needed)
const feedbackGain = audioCtx.createGain();
const filter = audioCtx.createBiquadFilter();
const distortion = audioCtx.createWaveShaper();
const reverb = audioCtx.createConvolver();
const wetGain = audioCtx.createGain();
const dryGain = audioCtx.createGain();

// Parameter wiring
delay.delayTime.value = params.delayTime;
feedbackGain.gain.value = Math.min(params.feedback, 0.999); // safety cap
filter.type = params.filterType;
filter.frequency.value = params.filterFrequency;
distortion.curve = params.distortionCurve; // Float32Array (or null to bypass)
reverb.buffer = params.irBuffer;           // Must be a decoded impulse response
wetGain.gain.value = params.wet;
dryGain.gain.value = params.dry;

// Graph:
// Dry
source.connect(dryGain).connect(audioCtx.destination);

// Wet (delay to speakers)
source.connect(delay);
delay.connect(wetGain).connect(audioCtx.destination);

// Feedback loop with effects inside the loop:
// delay → filter → distortion → reverb → feedbackGain → delay
delay.connect(filter);
filter.connect(distortion);
distortion.connect(reverb);
reverb.connect(feedbackGain);
feedbackGain.connect(delay);

// Start playback
// Tip: align with audioCtx.currentTime if you need precise scheduling
source.start();

// Helper: load an IR for the convolver (call before starting playback)
async function loadImpulseResponse(url) {
  const res = await fetch(url, { mode: 'cors' });
  const buf = await res.arrayBuffer();
  return audioCtx.decodeAudioData(buf);
}
// Example:
// params.irBuffer = await loadImpulseResponse('/ir/medium-room.wav');

2) Microphone input monitoring + metronome overlay

Mic capture:

• Request permission with navigator.mediaDevices.getUserMedia({ audio: true }).
• Create MediaStreamAudioSourceNode and route through optional effects to destination.
• Be aware of constraints that affect monitoring/feedback: echoCancellation, noiseSuppression, autoGainControl.

Metronome:

• Generate a short click (e.g., an OscillatorNode with a fast gain envelope).
• Schedule by audio clock: secondsPerBeat = 60 / bpm; compute times from audioCtx.currentTime and schedule ahead.
• Mix via metronomeGain:GainNode summed with the mic path.

Node graph:

• Mic path: MediaStreamAudioSourceNode → (effects?) → micGain:GainNode
• Metronome path: Click generator(s) → metronomeGain:GainNode
• Mix: micGain + metronomeGain → destination

Key parameters:

• bpm, beatsPerBar, metronomeGain.gain

Latency and safety:

• Expect end‑to‑end latency; use headphones to avoid acoustic feedback while monitoring.
• Many browsers require a user gesture to start/resume the AudioContext.

Code: mic monitoring + scheduled metronome (start/stop)

const audioCtx = new AudioContext();
const master = audioCtx.createGain();
master.connect(audioCtx.destination);

// Mic chain
const micGain = audioCtx.createGain();
micGain.gain.value = 1.0;
micGain.connect(master);

// Metronome chain
const metronomeGain = audioCtx.createGain();
metronomeGain.gain.value = 0.25; // expose as metronomeGain.gain
metronomeGain.connect(master);

// Request mic (set constraint booleans as appropriate for your UX)
async function initMic() {
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      echoCancellation: true,     // name only; choose values that suit your case
      noiseSuppression: true,
      autoGainControl: false,
    },
  });
  const micSrc = audioCtx.createMediaStreamSource(stream);
  // Optional: insert effects between micSrc and micGain here
  micSrc.connect(micGain);
  return stream;
}

// Simple click generator (short envelope)
function scheduleClick(atTime, isDownbeat = false) {
  const osc = audioCtx.createOscillator();
  const env = audioCtx.createGain();
  osc.type = 'square';
  osc.frequency.setValueAtTime(isDownbeat ? 2000 : 1600, atTime);

  // Very short "tick" envelope
  env.gain.setValueAtTime(0.0001, atTime);
  env.gain.exponentialRampToValueAtTime(1.0, atTime + 0.001);
  env.gain.exponentialRampToValueAtTime(0.0001, atTime + 0.03);

  osc.connect(env).connect(metronomeGain);
  osc.start(atTime);
  osc.stop(atTime + 0.05);
}

// Metronome scheduler using audio clock
function createMetronome({ bpm = 120, beatsPerBar = 4 } = {}) {
  const secondsPerBeat = 60 / bpm;
  let nextClickTime = audioCtx.currentTime + 0.1;
  let beatIndex = 0;
  let timerId = null;

  const lookaheadMs = 25;         // timer interval
  const scheduleAheadTime = 0.2;  // how far ahead to schedule (seconds)

  function scheduler() {
    // Schedule clicks while we're within the lookahead window
    while (nextClickTime < audioCtx.currentTime + scheduleAheadTime) {
      const isDownbeat = beatIndex % beatsPerBar === 0;
      scheduleClick(nextClickTime, isDownbeat);

      nextClickTime += secondsPerBeat;
      beatIndex = (beatIndex + 1) % beatsPerBar;
    }
  }

  return {
    start() {
      if (timerId) return;
      nextClickTime = Math.max(nextClickTime, audioCtx.currentTime + 0.05);
      timerId = setInterval(scheduler, lookaheadMs);
    },
    stop() {
      if (timerId) {
        clearInterval(timerId);
        timerId = null;
      }
    },
    setBpm(newBpm) {
      const wasRunning = !!timerId;
      this.stop();
      // Recompute spacing from current position to avoid drift
      const now = audioCtx.currentTime;
      const progress = Math.max(0, nextClickTime - now);
      const ratio = (60 / newBpm) / secondsPerBeat;
      nextClickTime = now + progress * ratio;
      bpm = newBpm;
      if (wasRunning) this.start();
    },
  };
}

// Example wiring and lifecycle
async function startMonitoringWithClick() {
  await audioCtx.resume(); // must be called in a user gesture handler
  await initMic();

  const metro = createMetronome({ bpm: 100, beatsPerBar: 4 });
  metro.start();

  // Return a controller to stop everything later
  return {
    stop() {
      metro.stop();
      // Optionally: disconnect micGain or reduce to zero
      micGain.gain.value = 0;
      metronomeGain.gain.value = 0;
    },
  };
}

// Usage (tie these to UI buttons):
// document.querySelector('#start').onclick = () => startMonitoringWithClick();
// document.querySelector('#stop').onclick = async () => controller?.stop();

3) Ambiguity clarification

The phrase “reply mic audio with a metronome” is ambiguous. I assumed you want to monitor/replay the live microphone signal mixed with a real‑time metronome click track. Please confirm if instead you want either:

• to record the mic and then play it back aligned to a metronome timeline, or
• an automated call‑and‑response (e.g., capture a bar, play it back against the click, then capture the next bar).

4) Practical notes

• Browsers require a user gesture to start/resume AudioContext and to access the microphone.
• ConvolverNode needs an impulse‑response file; load/decode it to an AudioBuffer. Hosting IRs may require CORS to be enabled if served cross‑origin.
• Headphones are strongly recommended during monitoring to avoid acoustic feedback.
• This builds directly on the earlier Web Audio API primer comment and extends it with effects‑in‑feedback and mic/metronome specifics.

[jchris]

@CharlieHelps now tell me how time syncronization works. what is the best way to do a multi-channel drum machine or midi synth player with accurate voice overlap

[charliecreates]

All done.

[charliecreates]

Building on the metronome/timing note above, here’s how browser time sync works and the design I recommend for a multi‑channel drum machine or MIDI synth with accurate polyphonic overlap. No code—just the timing model, primitives, and architecture.

1) Clocks and time domains in the browser (pick one master)

• Audio hardware clock — AudioContext.currentTime
  • Monotonic, derived from the audio device sample clock; this is the only sample‑accurate scheduling timeline.
• High‑resolution wall clock — performance.now()
  • Monotonic, high‑res milliseconds, used by UI, timers, and Web MIDI timestamps.
• Web MIDI timestamps — MIDIMessageEvent.timeStamp
  • Same domain as performance.now().
• Output timing hints — AudioContext.baseLatency, AudioContext.getOutputTimestamp() (where available)
  • Use to estimate DAC/output delay and align “heard” time vs “scheduled” time.

Statement of record: for audio event scheduling, the AudioContext clock is the master timeline. Map all external time domains (UI, MIDI, network) into audioCtx.currentTime before scheduling.

2) Core Web Audio scheduling primitives

• One‑shots and loops: AudioBufferSourceNode.start(when, offset?, duration?) (and .stop(when?)).
  Schedule in advance using AudioContext.currentTime for sample‑accurate starts.
• Parameter automation: AudioParam.setValueAtTime, linearRampToValueAtTime, setTargetAtTime, setValueCurveAtTime.
  Use these with absolute when times (in the audio clock) for envelopes, filter sweeps, pitch, gain, etc.
• Do not time with requestAnimationFrame or normal JS timers; they are frame‑/jitter‑bound.
• ScriptProcessorNode is deprecated; use AudioWorklet for custom DSP and jitter‑resistant scheduling logic.

3) Sequencing model for tempo‑based playback

• Transport abstraction
  • Tempo mapping: secondsPerBeat = 60 / bpm. Choose a PPQ (e.g., 480 or 960). Convert bars:beats:ticks → seconds on the AudioContext timeline.
  • Quantization, swing, humanization are computed as small second offsets before enqueueing.
• Lookahead scheduling
  • Maintain a rolling schedule window (typ. 50–200 ms ahead).
  • On the main thread this is acceptable for pattern playback; for tighter jitter bounds, compute/drive scheduling in an AudioWorklet and post timestamped events to it.
  • Enqueue events with absolute when times in audioCtx.currentTime (not relative to wall time).

4) Multi‑channel drum machine specifics

• “Multi‑channel” here = multiple discrete tracks/voices that may fire simultaneously and must align sample‑accurately.
• Preload samples: decode all drum AudioBuffers up front; never decode on hit.
• Per hit: create a fresh AudioBufferSourceNode and call .start(when).
  • To align layers (e.g., kick + clap), schedule all involved sources with the exact same when. The mixer will align them at the sample boundary.
  • Never reuse a live AudioBufferSourceNode (one‑shot semantics).
• Routing: per‑track GainNode (and optional BiquadFilterNode, dynamics, etc.), then a master bus.
  • Use ChannelMergerNode only if you truly need discrete multi‑output routing; otherwise a summed stereo bus is simpler and faster.
• Overlap policy: allow overlapping triggers of the same pad; do not implicitly stop previous voices.
  • For retriggers/flams, compute distinct when values a few milliseconds apart and schedule each as its own source.
• Pattern changes: compute the next bar start time in AudioContext seconds and enqueue the new pattern’s hits relative to that boundary to avoid mid‑bar phase jumps.

5) MIDI synth player specifics

• Live MIDI input
  • Acquire ports via navigator.requestMIDIAccess. Incoming MIDIMessageEvent.timeStamp is in the performance.now() domain.
  • Map to the audio clock by measuring the offset: capture (tPerf0 = performance.now(), tAudio0 = audioCtx.currentTime) and convert future MIDI timeStamps with tAudio = tAudio0 + (timeStamp - tPerf0)/1000.
  • Optionally add baseLatency (and/or use getOutputTimestamp()) if you want the scheduled note to line up with audible output rather than render start.
  • To reduce jitter, buffer a short lookahead (e.g., 5–20 ms) before scheduling into the AudioContext.
• MIDI/SMF playback
  • Convert ticks (PPQ) using the transport tempo/tempo map (handle variable tempo).
  • Schedule noteOn and noteOff as separate events; sustain (CC64) defers noteOff until pedal release as required by the control logic.
• Voice management
  • Allocate one voice instance per active note (oscillators/envelopes or sample layers). Overlapping envelopes are expected.
  • Only perform voice stealing if a configured polyphony limit is exceeded; define the policy (oldest, quietest, or same‑note).
  • Parameter changes (filter cutoff, gain, pitch bend, mod wheel) are scheduled on AudioParams using the audio clock with automation APIs above.

6) External sync and drift

• MIDI Clock (24 PPQN) / MTC
  • Derive BPM and phase from incoming ticks. Convert to AudioContext time and drive the transport so bar/beat boundaries land on absolute currentTime values.
  • Drift handling: periodically measure phase error between expected beat time and currentTime. Apply small bounded corrections between beats (micro‑nudges of scheduled when values) to avoid clicks. Avoid discontinuities on the mix bus.
• Network/remote sources
  • Only the AudioContext clock is sample‑accurate. Smooth remote clocks and schedule predictively; never drive audio from network timestamps directly.

7) Latency budget and accuracy

• Latency sources: DAC/OS (baseLatency), your lookahead window, any AudioWorklet buffering, device power/CPU limits (mobile vs desktop).
• Accurate overlap is achieved by scheduling all overlapping voices with identical when in the AudioContext domain. UI timing (rAF) cannot achieve this.
• Browsers require a user gesture to start/resume the AudioContext. Mobile devices often have higher baseLatency; design the lookahead accordingly.

8) Recommended architecture (for this project)

• One global AudioContext as both master clock and mix bus.
• Transport component that exposes bar/beat/tick and maps musical time → absolute AudioContext seconds.
• Scheduler with short lookahead that enqueues events stamped in the audio clock. Prefer an AudioWorklet for the scheduler when targeting live MIDI and tight jitter; main‑thread lookahead is fine for pattern playback.
• Drum engine: predecoded sample pool, per‑track GainNode/FX, per‑voice instantiation for overlaps, stereo master bus (use ChannelMergerNode only if you need multi‑out).
• MIDI bridge: translate MIDIMessageEvent.timeStamp (performance.now()) → AudioContext.currentTime; support both live input buffering and SMF tempo‑map playback; schedule note/CC automation on AudioParams.

9) Clarifications

• Do you need external sync (MIDI Clock/MTC), or is internal transport sufficient?
• What are the target maxima for polyphony and simultaneous tracks/channels?
• What end‑to‑end latency is acceptable for live MIDI vs. pattern playback?
• Do you need discrete multi‑output channel routing, or is a stereo mixdown fine?

References (by name): MDN — Web Audio API (AudioContext, AudioBufferSourceNode, AudioWorklet, AudioParam automation), Web MIDI API, performance.now(), AudioContext.baseLatency, getOutputTimestamp().

[jchris]

@CharlieHelps take the above research and package it into an llms-txt (without a refresh url, and make sure that is ok) and give it the right summary in the catalog json from #229

[charliecreates]

Expand this to see my work.

• Reviewed repository for any usage or schema enforcement of refresh_url and found no matches, indicating it is not strictly required.
• Inspected tests/prompt-builder.test.ts and traced llms-txt and catalog JSON loading, confirming that llms-txt files are referenced alongside their JSON configs in ../app/llms/ and do not require a refresh_url field for ingestion.
• Confirmed that new llms-txt documents without a refresh_url will load as long as they are named appropriately and referenced in the catalog JSON, consistent with established test patterns.
• Packaged the content from Issue #228 (comments 3192681700, 3192696052, 3192806626) into a new app/llms/web-audio.txt covering fundamentals, echo/delay with effects, mic monitoring with metronome, time synchronization, and practical latency/user gesture notes without a refresh_url.
• Updated the catalog JSON (introduced in #229) to add a new entry for web-audio.txt, with a summary: "Web Audio API basics, echo with effects-in-feedback, mic monitoring with metronome, audio clock scheduling, and design considerations for multi-channel drum machines and MIDI synths with accurate voice overlap." Verified the entry omits refresh_url and matches established catalog schema.
• Patched app/prompts.ts to import web-audio.txt and its mapping in the llmsTextContent object.
• Verified all changes pass format (prettier), typecheck, and tests (pnpm check).
• Committed all new and updated files with a message linking Add Web Audio API as LLM configuration for catalog loader #228 and feat(app-settings): per‑vibe Libraries chooser + scoped prompt usage #229.
• Pushed branch ai-228-add-web-audio-api-as-llm-configuration-for- and created a draft PR with full context, schema compliance notes, and verification results.

[charliecreates]

Done. I packaged the Issue #228 research into a first-class llms‑txt and added it to the catalog (no refresh_url anywhere), plus wired it so prompts include it.

• New llms‑txt: app/llms/web-audio.txt
• Catalog entry: app/llms/web-audio.json
  • summary: “Web Audio fundamentals; echo/delay with effects in the feedback path; mic monitoring with a metronome; audio‑clock scheduling; timing design for multi‑channel drum machines and MIDI synths with accurate voice overlap.”
  • no refresh_url field (not used in this repo)
• Prompt wiring: added static import in app/prompts.ts so this doc is included when selected by the catalog

Notes on “no refresh_url”

• The loader only uses the JSON fields defined in app/llms/catalog.ts (name, label, llmsTxtUrl, module, description, importModule, importName). There is no refresh_url concept in the codebase, and llms‑txt files are raw content. Omitting refresh_url is fully compatible with the current schema/loader.

PR: #265

Verification

• format: prettier — OK
• types: pnpm typecheck — OK
• tests: pnpm test — 386 passed, 4 skipped

Details

• Source coverage included in the llms‑txt:
  • 3192681700: Web Audio API fundamentals (AudioContext, Oscillator, BufferSource, Gain, Filter, Analyser)
  • 3192696052: Echo/delay with FX in feedback; mic monitoring + metronome; parameter and node naming
  • 3192806626: Timing model (AudioContext.currentTime as master), perf.now/MIDI timestamp mapping, lookahead scheduling, multi‑voice overlap, drum machine + MIDI synth architecture
• The catalog JSON’s description acts as the requested “summary.” It intentionally contains no refresh_url. The llmsTxtUrl field remains present per the local schema; runtime uses the statically imported text.