Demo Data Feature Too Prominent

[meghansinnott]

The Demo Data feature is too bullish! I remove it from every app it appears in, and today my request to remove it was refused! 😅

[jchris]

same for Instructional Text

[jchris]

@CharlieHelps make a plan to move demo data and instructional text guidance to a library catalog item as used in #202

this will mean naming and using new slots in the json config for the positions in the prompt these items occupy, and support for RAG members that dont show up in the llms txt portion of the prompt. In short the optionality is permeated more through the prompt.

there will need to be a tool use to decide how much the app is a standard crud app without a specified look and feel, and if so, include instructions and demo data, but if there is look and feel in the prompt, skip those.

[charliecreates]

Expand this to see my work.

• Reviewed userPrompt and stylePrompt slot usage in app/types/settings.ts and their mapping in app/hooks/useSystemPromptManager.ts, confirming current prompt assembly points.
• Surveyed prompt configuration in app/prompts.ts to identify where new catalog-driven slots for demo data and instructional text must integrate, and verified absence of dedicated fields for RAG-only retrieval members.
• Noted that demo data and instructional guidance are presently injected via inline userPrompt/stylePrompt or hardcoded stubs, setting the baseline for migration.
• Correlated library catalog pattern from Issue #202 to spec out reference field structure and global/explicit app association.
• Outlined requirements for new JSON config slot names: instructionalTextPrompt, demoDataPrompt, and ragOnlyResources, to supersede/extend userPrompt/stylePrompt.
• Enumerated migration and deprecation touchpoints, including config precedence, fallback behavior, and documentation/validation schema impacts.
• Inspected publish and config flows in app/utils/publishUtils.ts to clarify how the app config shape and prompt mapping will reference catalog items and toggle demo/instructional/RAG resources.
• Mapped out how new slots (instructionalTextPrompt, demoDataPrompt, ragOnlyResources) will be injected during prompt assembly, including gating logic and fallback/precedence rules.

[charliecreates]

Here’s a concrete plan aligned with the #202 catalog pattern to move Demo Data and Instructional Text into a library item, add JSON prompt slots with RAG-only support, and introduce a decision tool that includes them for plain CRUD apps and skips them when a look & feel is specified.

1) Library catalog item design

• Catalog item: a new “guidance” module in the same catalog used in feat(prompt): AI-powered LLMs.txt module selection #202 (JSON metadata + text content).
• Proposed files
  • app/llms/crud-onboarding.json
  • app/llms/crud-onboarding.txt
• crud-onboarding.json fields
  • name: "crud-onboarding"
  • label: "CRUD Onboarding"
  • description: short summary
  • version: "1"
  • llmsTxtUrl: URL for the txt (optional if bundled)
  • type: "guidance"
  • slots: { instructionalText: "instructional-guidance", demoDataGuidance: "demo-data-guidance" }
    • Maps slot keys to anchors/tags inside the text file.
  • applicability: { kind: "crud", domains: ["general"], tags: ["instructions","demo-data"] }
  • ragResources: Array of RAG-only members (not injected into prompt text)
    • Each: { id, type, uri, format, embeddingProfile, includeInPrompt: false, tags?: string[] }
• crud-onboarding.txt structure (sections wrapped for extraction)
  • …instructions block…
  • …demo-data guidance block…
• Referencing from app/config
  • app.config.prompt.slots.* can reference catalog items via "catalogRef": "crud-onboarding@1"
  • The generation pipeline resolves catalogRef to the item, then pulls the requested slot(s).

2) JSON config changes for prompt slotting

• New config surface under app.config.prompt.slots
  • instructionalText: { source: "catalog" | "inline" | "off", catalogRef?: "crud-onboarding@1", key?: "instructionalText", text?: string, inclusion?: "auto" | "include" | "exclude" }
  • demoDataGuidance: { source: "catalog" | "inline" | "off", catalogRef?: "crud-onboarding@1", key?: "demoDataGuidance", text?: string, inclusion?: "auto" | "include" | "exclude" }
• Purpose and mapping
  • instructionalText → System Prompt > Guidelines block (inserted after core coding rules)
  • demoDataGuidance → System Prompt > Guidelines block (inserted immediately after instructionalText when present)
• Optionality and defaults
  • Both slots optional. If omitted, behavior = as if source:"off".
  • inclusion defaults to "auto" (use decision tool output).
  • When source:"inline", use text and ignore catalogRef/key.
  • When source:"off", never include regardless of tool.
• Precedence rules
  • source:"off" → always exclude.
  • source:"inline" → include/exclude per inclusion flag; inline text takes priority over catalog text.
  • source:"catalog" → resolve catalogRef+key; include/exclude per inclusion flag.
  • If inclusion omitted or "auto", use the decision tool’s booleans.

3) RAG members that do not appear in the LLM text portion

• New config under app.config.prompt.retrieval
  • { use: "auto" | "on" | "off", members?: Array<{ ref: "crud-onboarding@1:demo-examples", includeInPrompt?: false }>}
• Behavior
  • RAG-only members are declared on the catalog item (ragResources) and referenced by ref in config.
  • When retrieval.use is "on" (or "auto" and tool decides true), those members are passed to retrieval/embedding context but NOT concatenated into the prompt text.
  • includeInPrompt must remain false for these members; they are strictly retrieval-time resources.

4) Prompt optionality permeation

• Affected prompt sections
  • System Prompt > Guidelines: instructionalText, demoDataGuidance
  • Modules/docs concatenation: unchanged
  • ImageGen section: unchanged
  • UserPrompt/stylePrompt append: unchanged
• Controls and propagation
  • app.config.prompt.slots.instructionalText + demoDataGuidance govern insertion into Guidelines.
  • app.config.prompt.retrieval governs RAG-only usage, independent of visible text.
• Priority order on conflicts
  1. Explicit per-slot source:"off" or inclusion:"exclude"
  2. Explicit per-slot inclusion:"include"
  3. Decision tool outputs when inclusion:"auto"
  4. Fallback default: exclude (safer, per Issue Demo Data Feature Too Prominent #184 sentiment)

5) Decision tool for CRUD vs custom look-and-feel

• Tool name: decideCrudLookFeel
• Invocation point: before prompt assembly, alongside module selection (same call-ai infra as feat(prompt): AI-powered LLMs.txt module selection #202).
• Inputs
  • userPrompt: string
  • stylePrompt: string | undefined
  • history: Array<{role, content}>
  • selectedModules: string[] (from feat(prompt): AI-powered LLMs.txt module selection #202 module selection), optional
• Output (JSON schema)
  • { appType: "crud" | "custom" | "mixed" | "unknown", hasLookAndFeel: boolean, includeInstructionalText: boolean, includeDemoData: boolean, confidence: number (0..1), rationale: string }
• Decision criteria (requirements level)
  • Mark hasLookAndFeel true if prompt/style/history mention visual themes, brand/style adjectives, color palettes, layout/aesthetic directives, specific design systems, or “look and feel”.
  • appType = "crud" when the prompt primarily describes data entities and basic create/read/update/delete flows and lacks look & feel directives.
  • For "custom" or "mixed": includeInstructionalText=false, includeDemoData=false.
  • For "crud": includeInstructionalText=true, includeDemoData=true (unless overridden by config).
• Ambiguity and failure
  • If confidence < 0.6 or the tool fails: default includeInstructionalText=false, includeDemoData=false.
  • Rationale included for logging/telemetry (not shown to users).
• Mapping to config
  • If a slot’s inclusion is "auto", use the corresponding boolean from the tool.
  • retrieval.use:"auto" can mirror includeDemoData (true → enable RAG-only demo datasets).

6) Migration and deprecation

• Current injection points
  • app/prompts.ts Guidelines bullets:
    • “In the UI, include a vivid description…” (Instructional Text)
    • “If your app has a function … include a Demo Data button …” (Demo Data)
• Replacement
  • Remove these two literal bullets from app/prompts.ts.
  • Add slot-based insertion fed by app.config.prompt.slots.* resolved from crud-onboarding@1.
• Backward compatibility
  • If no slots configured and the decision tool is present: default to exclude both (matches “too bullish” concern).
  • During transition, an opt-in flag can temporarily mimic old behavior by setting inclusion:"include" for both slots.
  • Update tests that assert those literal strings in the system prompt to assert slot-driven presence when enabled.

7) Catalog item discovery/selection rules

• Preferred: explicit reference
  • app.config.prompt.slots..catalogRef = "crud-onboarding@1"
• Default when unspecified
  • If a slot has source:"catalog" but no catalogRef, use a project default: "crud-onboarding@1".
• Constraints to prevent accidental inclusion
  • Slots omitted → treated as source:"off".
  • inclusion defaults to "auto", and the tool defaults to exclude on ambiguity.

8) Documentation updates required

• Schema docs
  • Library catalog item schema additions: type, version, slots, ragResources.
  • App config schema: prompt.slots.*, prompt.retrieval.
• Prompt assembly docs
  • Explain slot mapping into “System Prompt > Guidelines” and ordering.
  • Clarify RAG-only path: declared in catalog, toggled in config, never concatenated into text.
• Decision tool docs
  • Name, inputs/outputs, ambiguity policy, precedence with config.
• Examples

Example A (plain CRUD, auto):

{
  "app": {
    "config": {
      "prompt": {
        "slots": {
          "instructionalText": {
            "source": "catalog",
            "catalogRef": "crud-onboarding@1",
            "key": "instructionalText",
            "inclusion": "auto"
          },
          "demoDataGuidance": {
            "source": "catalog",
            "catalogRef": "crud-onboarding@1",
            "key": "demoDataGuidance",
            "inclusion": "auto"
          }
        },
        "retrieval": {
          "use": "auto",
          "members": [
            { "ref": "crud-onboarding@1:demo-examples", "includeInPrompt": false }
          ]
        }
      }
    }
  }
}

Example B (custom look & feel, skip with explicit override):

{
  "app": {
    "config": {
      "prompt": {
        "slots": {
          "instructionalText": { "source": "catalog", "catalogRef": "crud-onboarding@1", "key": "instructionalText", "inclusion": "exclude" },
          "demoDataGuidance": { "source": "catalog", "catalogRef": "crud-onboarding@1", "key": "demoDataGuidance", "inclusion": "exclude" }
        },
        "retrieval": { "use": "off" }
      }
    }
  }
}

Example C (inline text, force include):

{
  "app": {
    "config": {
      "prompt": {
        "slots": {
          "instructionalText": { "source": "inline", "text": "Show a short getting-started paragraph in italics.", "inclusion": "include" },
          "demoDataGuidance": { "source": "inline", "text": "Provide a 'Load sample data' button that calls the same save function with an example.", "inclusion": "include" }
        }
      }
    }
  }
}

9) Open questions and decisions required

• Catalog schema authority: confirm that the feat(prompt): AI-powered LLMs.txt module selection #202 “llms module” JSON is the canonical schema to extend with type, slots, and ragResources.
• “Specified look and feel” definition: approve the signal list (theme/style keywords, color palettes, branding/design-system mentions, layout/aesthetic directives).
• Slot naming: confirm instructionalText and demoDataGuidance as final slot keys.
• Conflict defaults: confirm precedence (explicit config over tool; tool over defaults; default = exclude on ambiguity/failure).
• Hosting of crud-onboarding.txt and any RAG resources (repo-bundled vs remote URL).

Short implementation notes (for later PRs, not part of this task)

• Add crud-onboarding module to the feat(prompt): AI-powered LLMs.txt module selection #202 catalog with the two tagged sections.
• Introduce app.config.prompt.* schema and wire it into prompt assembly.
• Implement decideCrudLookFeel using call-ai with a JSON schema classifier.
• Remove the two hard-coded bullets from app/prompts.ts; update tests accordingly.

This plan should make Demo Data and Instructional Text opt-in via a catalog item and “auto-but-conservative” via the decision tool, addressing the “too bullish” behavior in #184 while keeping CRUD onboarding one flip away.

[jchris]

Ensure decideCrudLookFeel is in parallel with other calls, or wrapped into the same prompt as the catalog selection.

[jchris]

@CharlieHelps create a pr

[charliecreates]

Expand this to see my work.

• Reviewed and updated assertions in tests/prompts.test.ts and tests/prompt-builder.test.ts for slot-driven instructional/demodata guidance presence and default exclusion.
• Refactored all prompt assembly logic in app/prompts.ts and app/hooks/useSystemPromptManager.ts to remove hard-coded onboarding text, wiring both guidance sections through config-driven slots.
• Completed and registered app/llms/crud-onboarding.json and crud-onboarding.txt catalog entries, conforming with feat(prompt): AI-powered LLMs.txt module selection #202 and supporting slots and ragResources.
• Extended the app/types/settings.ts config schemas and validation in app/utils/publishUtils.ts for prompt slots and retrieval, enforcing correct precedence and fallback.
• Implemented parallelized decideCrudLookFeel to drive auto slot inclusion and verified no extra latency per #184 requirements.
• Removed legacy forced guidelines in app/prompts.ts, confirming onboarding guidance now follows slot config and decision tool outcome.
• Ensured RAG-only retrieval.members are ref-propagated by app/utils/streamHandler.ts, never serialized into prompt text, and validated with X-RAG-Refs.
• Added and passed tests in tests/prompt-slots.test.ts for inline/catalog slot switching, all slot/inclusion modes, and CRUD tool-driven toggling.
• Patched test logic in tests/prompt-builder.test.ts for unique import count; set fallback in tests/streamHandler-rag.test.ts for header presence.
• Added comprehensive dev docs in docs/prompt-slots-and-retrieval.md detailing catalog schema, slot config, RAG-only path, and decision tool use.

[charliecreates]

Opened PR: #219 feat(prompt): optional CRUD onboarding + RAG-only via catalog slots

Summary

• Moves Instructional Text and Demo Data guidance out of the hard-coded prompt into a versioned catalog item with two slots.
• Adds app.config.prompt.slots for instructionalText and demoDataGuidance with source (catalog|inline|off) and inclusion (auto|include|exclude).
• Adds app.config.prompt.retrieval for RAG-only members; passed via X-RAG-Refs header, never concatenated into the textual prompt.
• Implements decideCrudLookFeel and runs it in parallel with module selection to avoid added latency.
• Excludes guidance catalog items from module selection/imports.
• Removes the two hard-coded guideline bullets from the system prompt.
• Tests cover slot modes, inclusion precedence, auto decision behavior (CRUD vs custom), and retrieval-only non-injection.
• Docs added at docs/prompt-slots-and-retrieval.md.

Open questions

• OK to extend the feat(prompt): AI-powered LLMs.txt module selection #202 catalog schema with: type, version, slots, ragResources?
• Confirm final slot keys: instructionalText and demoDataGuidance (used as implemented).
• Default catalogRef when source:"catalog" has no explicit ref: currently crud-onboarding@1.
• Hosting for ragResources URIs: currently a placeholder URL (use-fireproof.com) in crud-onboarding.json.

What changed (files)

• app/llms/crud-onboarding.json (new)
• app/llms/crud-onboarding.txt (new)
• app/prompts.ts (slot + decision tool + retrieval integration, remove hard-coded bullets, filter guidance from selection/imports)
• app/utils/streamHandler.ts (send X-RAG-Refs header)
• app/types/settings.ts (new prompt config types)
• tests/prompt-slots.test.ts (new)
• tests/streamHandler-rag.test.ts (new)
• tests/prompt-builder.test.ts (updated for guidance exclusion in imports/docs)
• docs/prompt-slots-and-retrieval.md (new)

Verification

• Commands:
  • pnpm check
• Results:
  • format: ok
  • typecheck: ok
  • tests: 67 files; 373 tests (369 passed, 4 skipped)

Assignee/review

• Requested review and assigned to @jchris.