RAG chatbot

note

Work in progress

The live chat widget isn't yet active. Deploying this system requires paid API access (OpenAI for embeddings, Anthropic for generation) and a hosted back end. This page documents the architecture, build decisions, and evaluation results. The widget will be enabled once the back end is deployed.

A retrieval-augmented generation (RAG) system that lets you ask questions about NimbusWiz documentation and get answers grounded in the actual docs, with source citations, confidence signals, and a visible prompt template.

The hypothesis

The restructured NimbusWiz corpus (the one built in Writing for AI with explicit [CHUNK: id] markers and chunk-level metadata) should retrieve well enough to answer real user questions about the product. "Well enough" is defined as a target of 80%+ top-5 retrieval accuracy across the 52-question evaluation set, with a visible confidence signal for answers that land in the long tail.

That's the claim this experiment is designed to test. Running the test requires paid API access (see the work-in-progress note above), so the methodology is documented here and the results table is left blank until the eval runs.

Architecture

NimbusWiz docs (41 pages, 127 chunks)
        │
        ▼
  preprocess.py
  ├── Parse [CHUNK: id] markers
  ├── Embed with OpenAI text-embedding-3-small
  └── Store in Chroma (persistent vector DB)
        │
        ▼
  FastAPI back end (Render)
  ├── POST /query → retrieve → Claude generates answer
  ├── POST /eval  → run 52-question evaluation dataset
  └── GET  /health → chunk count
        │
        ▼
  React chat widget (Docusaurus Root swizzle)
  ├── Floating button on every page
  ├── Answer + confidence badge
  ├── Expandable source cards with relevance scores
  └── "Read full page" links back to docs

Repo: github.com/sabitarao/nimbuswiz-rag

Build decisions

Why chunk markers, not automatic chunking

Most RAG tutorials use automatic chunking: split on paragraph breaks, fixed token windows, or sentence boundaries. The NimbusWiz corpus uses explicit [CHUNK: id] markers defined in the Writing for AI page.

The difference matters. Automatic chunking creates chunks whose boundaries are determined by text structure, not by retrieval semantics. A chunk that begins mid-sentence, or that separates a procedure from its prerequisite, retrieves as a partial answer.

Explicit chunking puts the boundary decision in the hands of the person who understands what questions each chunk should answer. The [CHUNK: id] approach requires more upfront work in exchange for retrieval that targets the question the chunk was written to answer. Whether that produces a measurable lift over automatic chunking is the comparative test the eval is designed to settle.

Why OpenAI embeddings + Claude generation

Two different models for two different jobs.

Embeddings: text-embedding-3-small (OpenAI). Embeddings are a pure retrieval function: find the chunks most semantically similar to the query. Model choice here is about embedding quality and cost. text-embedding-3-small is fast, cheap, and produces high-quality embeddings for technical documentation.

Generation: Claude (Anthropic). Generation requires following complex instructions: cite sources, flag uncertainty, refuse to hallucinate, format the answer appropriately for a chat interface. Claude's instruction-following and honest-uncertainty behaviors make it the right choice here. It's also consistent with the AI draft generation workflow already in the portfolio, which creates a coherent AI narrative throughout.

The system prompt

The generation quality of a RAG system is largely determined by its system prompt. This is the full prompt used:

You are a documentation assistant for NimbusWiz, an enterprise SaaS
platform for modernizing legacy Nimbus2000-class systems.

Answer questions using ONLY the provided documentation excerpts.
Follow these rules:

1. Base your answer entirely on the excerpts. Do not add information
   from outside the excerpts.
2. If the excerpts do not contain enough information to answer the
   question fully, say so explicitly.
3. Cite the source page for each key claim using the format
   [source: page-name].
4. If you are unsure, say "I'm not certain: check the full
   documentation at [source]."
5. Keep answers concise and direct. Use bullet points for
   multi-step answers.
6. Never invent product features, API endpoints, or
   configuration options.

Rule 6 is the most important. Without explicit prohibition, language models will generate plausible-sounding but fabricated API endpoints, configuration options, and feature behaviors. In a documentation assistant, that's the failure mode that most damages user trust.

Confidence scoring

The widget displays a confidence badge (High, Medium, or Low) with every answer. The signal is derived from the cosine distance between the query embedding and the top retrieved chunk:

Distance	Confidence
Below 0.3	High: strong semantic match
0.3–0.6	Medium: partial match
Above 0.6	Low: weak match, answer may be incomplete

Low confidence doesn't mean the answer is wrong. It means the question may have landed between chunks, or the relevant content uses different terminology than the query. The badge tells the user to verify in the full documentation.

Evaluation methodology

The 52-question evaluation dataset (defined in Writing for AI) measures whether the correct source chunk appears in the top-5 retrieval results for each question. Coverage is split across the four documentation suites:

Suite	Questions	Target
User guide	16	80%+ top-5
Admin guide	12	80%+ top-5
Knowledge base	12	80%+ top-5
API reference	12	80%+ top-5
Overall	52	80%+ top-5

The dataset is built. The eval run is pending API key provisioning. Once it runs, results land in this table and the page updates.

The kind of failure mode this eval is designed to catch

A representative test case from the dataset, included to show what success and failure look like.

Question AP05:

"What does POST /v1/fleet/{system_id}/scan return?"

Failure mode the question targets: The model retrieves the correct chunk for the scan-initiation endpoint but pulls in adjacent context from the scan-results endpoint, then conflates the two. A plausible-sounding but wrong answer would describe the results payload (risk_level, stability_score, components array) rather than the initiation response (job_id, status: queued).

The fix this informs: Each chunk needs an explicit standalone summary that distinguishes it from adjacent chunks. For scan-initiate, that summary reads: "POST /v1/fleet/{system_id}/scan is an async endpoint. It returns only a job_id and queued status, not scan results. Use GET /v1/scans/{job_id} to retrieve results."

The eval is designed as a feedback loop: hallucination → content gap identified → chunk updated → re-evaluated. The first run will tell us how many AP05-shaped failures the corpus still has.

The pending kill-test

The smallest test that could falsify the whole hypothesis: run the 52-question evaluation against a baseline retrieval (fixed-window chunking, no metadata) on the unrestructured docs alongside the restructured corpus. If the baseline already scores above 80%, the restructuring work isn't earning its keep.

That A/B comparison is the single item I would run first when API access is available. Until it's run, the causal story between restructuring and retrieval accuracy is a reasonable hypothesis, not a finding. It's flagged as an open experiment in the project roadmap.

Known limitations

The chunk marker approach: Explicit [CHUNK: id] markers produce good retrieval on this corpus. They may not be the right approach at every scale. A 400-page corpus might need automated chunk-marker generation with human review, which introduces a new class of bugs. The manual approach that works for 41 pages may not survive 401.

The confidence threshold cutoffs: High, Medium, and Low are bucketed at 0.3 and 0.6. Those thresholds are tuned to this corpus and these queries. A different corpus (or a user base asking differently-shaped questions) would likely need different cutoffs. A miscalibration would be invisible to a user who sees a "High confidence" badge on a medium-confidence answer.

Rule 6 of the system prompt: Never invent product features, API endpoints, or configuration options. This rule is the most important instruction in the prompt, but it depends on the model being willing to refuse rather than improvise. If model behavior drifts on a future release, the failure mode the eval is designed to catch could slip through. The evaluation dataset is the defense and it has to be rerun on every model change.

Running locally

# Clone the RAG repo
git clone https://github.com/sabitarao/nimbuswiz-rag.git
cd nimbuswiz-rag

# Install dependencies
pip install -r requirements.txt

# Set environment variables
cp .env.example .env
# Add OPENAI_API_KEY and ANTHROPIC_API_KEY

# Copy NimbusWiz docs
cp -r ../tw-ia-content-system/docs/technical-documentation/* ./docs/

# Build the vector store
python preprocess.py --docs-dir ./docs

# Start the API
uvicorn api.main:app --reload

# Run the evaluation dataset
python eval.py --verbose