tools: Add semantic AI server for command intent classification and validation by Copilot · Pull Request #1 · TheOriginalBytePlayer/llama.cpp

Copilot · 2025-12-29T09:32:53Z

Description

Implements a 64-bit C++ semantic AI server that processes natural language commands through intent classification and JSON schema validation. Designed for real-time command interpretation with IPC support.

Core Components

Command Schema (command-schema.h): 15 verbs across 4 action groups (CAMERA_CONTROL, ACTOR_POSE, OBJECT_MGMT, SHOT_MGMT) with parameter specifications
Intent Engine (intent-engine.h): Llama-based classification with temperature=0.1 for deterministic output, JSON extraction, early stopping on complete objects
Validator (command-validator.h): Schema validation with Levenshtein distance ≤2 for fuzzy verb matching (handles common typos)
IPC Handler (ipc-handler.h): Cross-platform Named Pipes (Windows/Unix) for sub-millisecond latency
Server (semantic-server.cpp): Dual-mode operation (interactive CLI for testing, IPC for production)

Usage

# Interactive mode for testing
./llama-semantic-server -m model.gguf --interactive
> pan left 30 degrees
{"verb":"PAN","action_group":"CAMERA_CONTROL","parameters":{"direction":"LEFT","degrees":30}}

# IPC mode for production
./llama-semantic-server -m model.gguf --pipe-name app_commands

Architecture

External audio transcription (e.g., whisper.cpp) → Named Pipe → Semantic Server → Validated JSON → Application

Testing

C++ unit tests: Schema registry, validator logic, fuzzy matching, enum conversions
Python unit tests: Schema validation, JSON serialization, error formats
CodeQL: 0 alerts

Build Integration

Added tools/semantic-server/ subdirectory to CMake. No changes to existing tools or libraries.

Original prompt

Design a 64-bit C++ Semantic AI Server using llama.cpp and whisper.cpp.

The Schema: Define a JSON-based command schema. Every command must have a Verb (e.g., PAN, LEAN, ADD, DELETE, SHOT) and each specific verbs will have some required Parameters (e.g. Direction, In/Out, etc.) and some optional parameters (e.g. Degrees, Speed, Target, PoseDescription, Subject etc.) ,

The Intent Engine: When audio is received and transcribed by Whisper, send the text to Llama-3 with a strict system prompt for Intent Classification:

'Analyze the following user input for FrameForge Studio. Map the input to one of these Action Groups: [CAMERA_CONTROL, ACTOR_POSE, OBJECT_MGMT, SHOT_MGMT].'

'Extract parameters. If the user says "PIN LEFT", match it to the nearest valid verb "PAN" with direction "LEFT".'

'If Action Group is ACTOR_POSE, generate a JSON array of joint rotations for the described pose.'

'Return ONLY a valid JSON object.'

Command Validation Logic: Implement a C++ CommandValidator class that checks the LLM's JSON against the required slots for that specific verb. If a required slot (like Direction for PAN) is missing, return an error object asking the user for clarification.

Communication: Maintain the 64-bit Sidecar as a resident process. Use a high-speed IPC (Inter-Process Communication) like Named Pipes to send the validated JSON objects to the 32-bit FrameForge Bridge.

Use JSON for Modern C++ (nlohmann/json)

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: TheOriginalBytePlayer <[email protected]>

…mprove docs Co-authored-by: TheOriginalBytePlayer <[email protected]>

Co-authored-by: TheOriginalBytePlayer <[email protected]>

Copilot

Pull request overview

This PR implements a 64-bit C++ Semantic AI Server for FrameForge Studio that processes natural language commands through intent classification and validation. The server uses llama.cpp for LLM-based command interpretation and provides IPC communication via Named Pipes for integration with external applications.

Key Changes

Command Schema System: Defines 15 verbs across 4 action groups (CAMERA_CONTROL, ACTOR_POSE, OBJECT_MGMT, SHOT_MGMT) with parameter specifications
Intent Classification Engine: Llama-based NLP processing with temperature=0.1 for deterministic output, including fuzzy matching for typo correction (Levenshtein distance ≤2)
Cross-platform IPC: Named Pipes implementation for Windows and Unix with bidirectional communication support

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
tools/semantic-server/command-schema.h	Defines command schema, verbs, action groups, and parameter specifications with enum converters
tools/semantic-server/command-validator.h	Implements validation logic with fuzzy verb matching using Levenshtein distance
tools/semantic-server/intent-engine.h	Wraps llama.cpp for intent classification with system prompt and JSON extraction
tools/semantic-server/ipc-handler.h	Cross-platform Named Pipes implementation for Windows and Unix IPC
tools/semantic-server/semantic-server.cpp	Main server application with dual-mode operation (interactive CLI and IPC)
tools/semantic-server/test-validator.cpp	C++ unit tests for schema registry, validator, fuzzy matching, and enum conversions
tools/semantic-server/test_schema.py	Python tests for command schemas, JSON validation, and error formats
tools/semantic-server/test_client.py	Test client demonstrating Named Pipe communication
tools/semantic-server/README.md	User documentation with examples and integration guide
tools/semantic-server/IMPLEMENTATION_SUMMARY.md	Architecture overview and implementation details
tools/semantic-server/CMakeLists.txt	Build configuration for the semantic server and test executable
tools/CMakeLists.txt	Integration of semantic-server subdirectory into the build system

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-29T10:04:28Z

tools/semantic-server/semantic-server.cpp

+            // Check if message is already JSON (from whisper or other source)
+            json input;
+            bool is_json_input = false;
+
+            try {
+                input = json::parse(message);
+                is_json_input = true;
+
+                // Check if it contains transcribed text
+                if (input.contains("text")) {
+                    std::string text = input["text"].get<std::string>();
+                    auto result = intent_engine.process_input(text);
+
+                    // Send validated command through pipe
+                    json output = result.to_json();
+                    std::string response = output.dump();
+                    ipc_handler.send_message(response);
+
+                    LOG_INF("Sent response: %s\n", response.c_str());
+                }
+            } catch (...) {
+                // Not JSON, treat as plain text
+                is_json_input = false;
+            }
+
+            if (!is_json_input) {
+                // Process as plain text command
+                auto result = intent_engine.process_input(message);
+
+                // Send validated command through pipe
+                json output = result.to_json();
+                std::string response = output.dump();
+                ipc_handler.send_message(response);
+
+                LOG_INF("Sent response: %s\n", response.c_str());
+            }


The nested if-else structure with is_json_input checking creates duplicated code for processing and sending responses (lines 157-166 and 173-182). This violates the DRY principle and makes maintenance harder. Consider extracting the response processing logic into a helper function to reduce duplication.

Copilot · 2025-12-29T10:04:29Z

tools/semantic-server/ipc-handler.h

+        if (pipe_handle == INVALID_HANDLE_VALUE) {
+            return false;
+        }
+
+        DWORD bytes_written;
+        BOOL success = WriteFile(
+            pipe_handle,
+            message.c_str(),
+            message.size(),
+            &bytes_written,
+            NULL
+        );
+
+        return success && bytes_written == message.size();


The pipe_handle may be invalid when send_message_windows is called if the client has disconnected between reads. This can lead to writing to an invalid handle. Consider checking if the pipe is still connected or handling the write failure more gracefully by returning false with appropriate logging.

Suggested change

if (pipe_handle == INVALID_HANDLE_VALUE) {

return false;

}

DWORD bytes_written;

BOOL success = WriteFile(

pipe_handle,

message.c_str(),

message.size(),

&bytes_written,

NULL

);

return success && bytes_written == message.size();

// If we already know the handle is invalid, fail fast.

if (pipe_handle == INVALID_HANDLE_VALUE) {

return false;

}

// Sanity-check the handle type; if it's invalid, treat as failure.

DWORD file_type = GetFileType(pipe_handle);

if (file_type == FILE_TYPE_UNKNOWN && GetLastError() == ERROR_INVALID_HANDLE) {

// Handle is no longer valid; normalize state and fail.

CloseHandle(pipe_handle);

pipe_handle = INVALID_HANDLE_VALUE;

return false;

}

DWORD bytes_written = 0;

BOOL success = WriteFile(

pipe_handle,

message.c_str(),

static_cast<DWORD>(message.size()),

&bytes_written,

NULL

);

if (!success) {

DWORD error = GetLastError();

// If the pipe is broken or the other end disconnected, close and invalidate the handle.

if (error == ERROR_BROKEN_PIPE ||

error == ERROR_NO_DATA ||

error == ERROR_PIPE_NOT_CONNECTED ||

error == ERROR_INVALID_HANDLE) {

CloseHandle(pipe_handle);

pipe_handle = INVALID_HANDLE_VALUE;

}

return false;

}

return bytes_written == static_cast<DWORD>(message.size());

Copilot · 2025-12-29T10:04:29Z

tools/semantic-server/ipc-handler.h

+    void stop() {
+        if (!running.exchange(false)) {
+            return; // Not running
+        }
+
+        // Signal the thread to stop
+        if (listener_thread.joinable()) {
+            listener_thread.join();
+        }
+
+#ifdef _WIN32
+        if (pipe_handle != INVALID_HANDLE_VALUE) {
+            CloseHandle(pipe_handle);
+            pipe_handle = INVALID_HANDLE_VALUE;
+        }
+#else
+        if (pipe_fd >= 0) {
+            close(pipe_fd);
+            pipe_fd = -1;
+        }
+        // Clean up the named pipe file
+        unlink(pipe_name.c_str());
+#endif


The order of operations in the stop() method is incorrect. The thread is joined before the pipe_handle is closed, but the thread may still be blocked on ReadFile when you try to join it. On Windows, you should either close the handle first to unblock the ReadFile (which will fail), or use overlapped I/O with a cancellation mechanism. The current implementation could cause the stop() method to hang indefinitely waiting for the thread to join.

Copilot · 2025-12-29T10:04:29Z

tools/semantic-server/intent-engine.h

+        const int n_prompt = -llama_tokenize(vocab, prompt.c_str(), prompt.size(), NULL, 0, true, true);
+        if (n_prompt <= 0) {
+            return "{}"; // Return empty object on error
+        }


When tokenization fails (returns <= 0), the function returns an empty JSON object "{}", which will later be treated as invalid by the validator. However, this error is silent and provides no information about why tokenization failed. Consider logging an error message or returning a more descriptive error through the ValidationResult.

Copilot · 2025-12-29T10:04:29Z

tools/semantic-server/semantic-server.cpp

+                }
+            } catch (...) {
+                // Not JSON, treat as plain text
+                is_json_input = false;


The variable is_json_input is set to true and then potentially set to false in the catch block, but the initial assignment to true happens before any JSON parsing attempt. This creates confusion about when the variable should be true vs false. Consider initializing to false and only setting to true after successful JSON parsing, which would also eliminate the need to set it to false in the catch block.

Suggested change

is_json_input = false;

Copilot · 2025-12-29T10:04:31Z

tools/semantic-server/intent-engine.h

+            // Check if we have a complete JSON object
+            if (response.find('}') != std::string::npos) {
+                // Try to parse to see if it's complete
+                try {
+                    json test = extract_json_from_text(response);
+                    if (!test.is_null() && test.is_object()) {
+                        break; // We have a complete JSON object
+                    }
+                } catch (...) {
+                    // Continue generating
+                }
+            }


The early stopping logic checks if a complete JSON object exists by finding the first '}' character. However, this can produce false positives if the JSON contains nested objects or strings with '}' characters. While the code then tries to parse the JSON to verify completeness, this approach may still stop prematurely in edge cases. Consider using a more robust JSON completeness check, such as tracking brace depth.

Copilot · 2025-12-29T10:04:31Z

tools/semantic-server/intent-engine.h

+                try {
+                    json test = extract_json_from_text(response);
+                    if (!test.is_null() && test.is_object()) {
+                        break; // We have a complete JSON object
+                    }
+                } catch (...) {
+                    // Continue generating


The catch-all exception handler silently continues generation when JSON parsing fails. This means the loop will continue generating tokens even if an exception occurs for a different reason (e.g., out of memory). Consider logging the exception or at least distinguishing between JSON parse errors and other exceptions.

Suggested change

try {

json test = extract_json_from_text(response);

if (!test.is_null() && test.is_object()) {

break; // We have a complete JSON object

}

} catch (...) {

// Continue generating

json test = extract_json_from_text(response);

if (!test.is_null() && test.is_object()) {

break; // We have a complete JSON object

Copilot · 2025-12-29T10:04:31Z

tools/semantic-server/intent-engine.h

+            } catch (...) {
+                return json();
+            }


The catch-all exception handler returns an empty JSON object instead of a proper error. This makes it difficult to distinguish between successful parsing that returned an empty object and actual parsing failures. Consider returning a null JSON value or a specific error indicator.

Copilot · 2025-12-29T10:04:31Z

tools/semantic-server/intent-engine.h

+            // Try to extract JSON from response if it's embedded in text
+            command = extract_json_from_text(llm_response);
+            if (command.is_null()) {
+                return ValidationResult::error("Failed to parse LLM response as JSON: " + std::string(e.what()));


The code catches json::parse_error specifically but then attempts to extract JSON from text and returns another generic error if that fails. The error message includes the original parse error details (e.what()), but if extract_json_from_text also fails, the original parse error information is lost. Consider preserving or combining both error messages for better debugging.

Suggested change

// Try to extract JSON from response if it's embedded in text

command = extract_json_from_text(llm_response);

if (command.is_null()) {

return ValidationResult::error("Failed to parse LLM response as JSON: " + std::string(e.what()));

// Preserve the original parse error message

const std::string initial_error = e.what();

// Try to extract JSON from response if it's embedded in text

try {

command = extract_json_from_text(llm_response);

} catch (const json::parse_error & e2) {

// Combine both error messages for better debugging

return ValidationResult::error(

"Failed to parse LLM response as JSON. "

"Initial parse error: " + initial_error +

" | Extracted JSON parse error: " + std::string(e2.what()));

}

if (command.is_null()) {

return ValidationResult::error(

"Failed to parse LLM response as JSON. "

"Initial parse error: " + initial_error +

" | Extracted JSON is null after fallback parsing.");

Copilot · 2025-12-29T10:04:32Z

tools/semantic-server/intent-engine.h

+    llama_context * ctx;
+    llama_model * model;
+    const llama_vocab * vocab;
+    llama_sampler * smpl;


The variable name 'smpl' is unclear and doesn't follow common naming conventions. Consider renaming to 'sampler' for better readability and maintainability.

…olved-comments Document unresolved code review comments from merged PRs #1 and #2

TheOriginalBytePlayer · 2025-12-31T07:35:00Z

@copilot apply changes based on the comments in this thread

Copilot AI self-assigned this Dec 29, 2025

Initial plan

19c1f92

Copilot AI assigned TheOriginalBytePlayer Dec 29, 2025

Copilot started work on behalf of TheOriginalBytePlayer December 29, 2025 09:33 View session

Copilot AI and others added 4 commits December 29, 2025 09:46

Implement Semantic AI Server with JSON command schema and intent engine

a10d74f

Co-authored-by: TheOriginalBytePlayer <[email protected]>

Add unit tests for semantic server validator and schema

646c2ea

Co-authored-by: TheOriginalBytePlayer <[email protected]>

Address code review feedback - add includes, validate tokenization, i…

d50d146

…mprove docs Co-authored-by: TheOriginalBytePlayer <[email protected]>

Add comprehensive implementation summary documentation

9329d34

Co-authored-by: TheOriginalBytePlayer <[email protected]>

Copilot AI changed the title ~~[WIP] Design JSON command schema for AI server~~ tools: Add semantic AI server for command intent classification and validation Dec 29, 2025

Copilot AI requested a review from TheOriginalBytePlayer December 29, 2025 09:55

Copilot finished work on behalf of TheOriginalBytePlayer December 29, 2025 09:55

TheOriginalBytePlayer approved these changes Dec 29, 2025

View reviewed changes

TheOriginalBytePlayer marked this pull request as ready for review December 29, 2025 09:57

Copilot AI review requested due to automatic review settings December 29, 2025 09:57

TheOriginalBytePlayer merged commit 57ddcaa into master Dec 29, 2025
2 checks passed

Copilot started reviewing on behalf of TheOriginalBytePlayer December 29, 2025 09:58 View session

Copilot AI reviewed Dec 29, 2025

View reviewed changes

Copilot AI mentioned this pull request Dec 29, 2025

Document unresolved code review comments from merged PRs #1 and #2 #3

Merged

10 tasks

TheOriginalBytePlayer added a commit that referenced this pull request Dec 30, 2025

Merge pull request #3 from TheOriginalBytePlayer/copilot/review-unres…

a8acd88

…olved-comments Document unresolved code review comments from merged PRs #1 and #2

TheOriginalBytePlayer deleted the copilot/define-json-command-schema branch December 31, 2025 07:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: Add semantic AI server for command intent classification and validation#1

tools: Add semantic AI server for command intent classification and validation#1
TheOriginalBytePlayer merged 5 commits intomasterfrom
copilot/define-json-command-schema

Copilot AI commented Dec 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

TheOriginalBytePlayer commented Dec 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-            // Try to extract JSON from response if it's embedded in text
-            command = extract_json_from_text(llm_response);
-            if (command.is_null()) {
-                return ValidationResult::error("Failed to parse LLM response as JSON: " + std::string(e.what()));
+            // Preserve the original parse error message
+            const std::string initial_error = e.what();
+            // Try to extract JSON from response if it's embedded in text
+            try {
+                command = extract_json_from_text(llm_response);
+            } catch (const json::parse_error & e2) {
+                // Combine both error messages for better debugging
+                return ValidationResult::error(
+                    "Failed to parse LLM response as JSON. "
+                    "Initial parse error: " + initial_error +
+                    " | Extracted JSON parse error: " + std::string(e2.what()));
+            }
+            if (command.is_null()) {
+                return ValidationResult::error(
+                    "Failed to parse LLM response as JSON. "
+                    "Initial parse error: " + initial_error +
+                    " | Extracted JSON is null after fallback parsing.");

Conversation

Copilot AI commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Core Components

Usage

Architecture

Testing

Build Integration

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

TheOriginalBytePlayer commented Dec 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Dec 29, 2025 •

edited

Loading