Margin

Overview

Margin is a sample case study for a question-and-answer assistant over a private documentation folder. You point it at a directory of Markdown files and PDFs, ask a question in plain language, and get an answer that quotes the passages it relied on, with links back to the source.

The citations are the product. An answer you cannot check is a guess with good grammar.

The problem

Internal documentation tends to be complete and unfindable at the same time. The answer exists, somewhere, in a file whose title does not contain the words you are searching for. General chat assistants can summarise well, but they do not know your documents, and when they do not know something they rarely say so.

Goals

• Answer questions from the documents only, and say clearly when the documents do not cover a question.
• Show the exact passages behind every answer.
• Keep the documents on infrastructure you control.
• Update the index automatically when files change.

Technical approach

Documents are split into overlapping chunks that follow headings, so a passage keeps its context. Each chunk is embedded and stored with its file path and heading trail.

For each question, the API retrieves the closest chunks, sends them to the model together with the question and a strict instruction to answer only from the provided passages, and returns the answer alongside the passages it was given. If retrieval finds nothing relevant, the API says so instead of calling the model at all.

Architecture

Figure 1 shows the two paths. The ingest path watches the documents folder, chunks and embeds changed files, and upserts them into the vector store. The question path retrieves the top passages, sends them to an OpenAI-compatible model endpoint, and returns an answer with citations.

Technology stack

• Python and FastAPI for the ingest worker and the answer API.
• PostgreSQL with pgvector for embeddings and metadata in one database.
• An OpenAI-compatible model endpoint, so the model can be hosted or self-hosted.
• Docker for packaging both services.

Challenges and solutions

Chunks that lose their meaning

A paragraph that says "set it to 30 seconds" is useless without its heading. Every chunk stores the heading path above it, and that path is included both in the embedding and in the passage shown to the user.

Answers that wander off the documents

The prompt tells the model to answer only from the passages and to say when they are insufficient. The interface makes the passages impossible to miss, so a reader can spot a bad answer quickly.

Stale indexes

The ingest worker compares content hashes, so editing one file re-embeds only that file instead of the whole folder.

Results

Sample entry. Record measured outcomes here once you have them. Good candidates are the share of questions answered with a correct citation in a hand-checked sample, the time to answer, and how often the assistant correctly declines.

Lessons learned

• Retrieval quality decides answer quality. Most improvements came from better chunking, not better prompts.
• "I could not find this in the documents" is a feature. Make it a first-class answer.
• Keep the evaluation set small and real: questions people actually asked, with answers someone has checked.