AIncient Labs, home

Atelier's assistant now checks its how-to answers against the docs before you see them

In 60 test runs, it answered correctly 59 times. When it can't confirm something, it now says so and links the closest page. Here is how we got there, starting with one early user's question.

Since Atelier 0.29.0, ask the General assistant how to do something, and two things happen before it answers.

First, the docs for the version you have installed are searched. Then a second model checks every claim in the reply against those docs. Anything the docs don't back is removed. If it still can't confirm the reply, you get an honest answer instead of a confident one:

I couldn't confirm that in the Atelier docs, so I'd rather not guess. The closest page is …

In our test set of 20 questions people ask us, run three times each, it passed 59 of 60 runs.

This is how we got there. It started with one early user.

The question that started it

One of the people building with Atelier early asked the assistant how to make a component of their own. The assistant answered at once: open the Design System studio and create one there.

There is no Design System studio. They took the time to tell us.

They were right to. You can't create components in the console: developers build them as component packs, and you choose which ones your site offers. Our docs said so plainly. The assistant had never read them.

A wrong answer given with confidence costs more than no answer. You can lose ten minutes looking for a room that doesn't exist. Then you stop trusting the assistant on everything else.

We set ourselves one rule: the assistant answers from the docs, or it doesn't answer.

It took three changes. The first, the obvious one, was not enough.

Change 1: give it the docs for your version

The docs now ship inside every Atelier release. The assistant reads the docs for the version you run, not the newest ones on our website. The search works offline. The docs come from the release, not the internet, so no stranger can slip text into them.

It has two tools, both read-only. One searches the docs and returns the five best-matching sections. The other reads a whole page.

The search is ordinary PostgreSQL full-text search. We chose it over AI embeddings for a practical reason: Atelier runs on your own AI provider key, and Anthropic has no embeddings service. Keyword search works for every customer, whichever provider they use.

We also told the assistant, in so many words: search first, link the page, never invent a feature, and "I don't know" is a perfectly good answer.

The result: better, not fixed. Asked how to update from the desktop app, it found the right page, then added a menu the page never mentioned. Asked about forms, it described a Typeform workaround we have never documented. In the update case, the right page was in front of it. It still filled the gaps from memory.

Instructions ask a model to behave. We needed something that checks.

Change 2: check every answer before you see it

Every written reply from the General assistant now passes through a second, separate model call.

That checker sees only the draft reply, the docs text found for your question, and the assistant's own short map of the console's rooms. Not your conversation, not the assistant's reasoning. It goes through the reply claim by claim and asks one question of each: do the docs say this?

  • Every claim backed: the reply goes out as written.
  • Some claims not backed: they are cut, and the shortened reply is checked again, for up to two more rounds. The checker doesn't get to trust its own edit. If it still can't confirm the reply, you get the honest answer above.
  • The check can't finish: the reply doesn't go out. You get "I couldn't confirm that", with the closest page.

Two simple rules need no AI at all. A rewrite can't add a link that wasn't already in the reply or the docs. And a confirmed answer that links nothing gets its best source added.

Three things we learned building it:

  1. Checking is the harder job. Our first checker ran on the cheaper model. It let invented Stripe and Etsy integrations stand in its own rewrite. It now runs on the stronger model.
  2. A check that can be skipped is decoration. Once, a verdict was cut off at 1,000 tokens, and the invented Typeform workaround went straight through. Now an unfinished check means no reply.
  3. Don't give up too early. One leftover phrase used to throw away an otherwise good answer. Now a rewrite gets a second round before the honest fallback.

What it costs you: one extra call to the reasoning model per answer, two or three when it has to rewrite. That adds about 5 to 15 seconds. For answers about what the product can and can't do, we believe the wait is worth it. You'll find it in your provider bill, so we tell you here.

(If you follow the research: this is closest to Chain-of-Verification and RARR: verify, then revise, within the same answer.)

Change 3: one name for every thing

To measure progress, we wrote a test: 20 questions real people ask. The docs answer fourteen of them. Six they deliberately don't, such as a contact form or visitor analytics, where the right answer is "Atelier doesn't do this". An independent judge grades every answer against whole docs pages, three runs per question. Before it fails an answer, it must also fail to find a passage that backs the claim, quoted exactly: our first judge flagged sentences the docs said word for word.

The first full runs: 12, then 13, of 20 questions passed all three runs.

The failures surprised us. None were invented features any more. They were near misses: the assistant skipping the search, the checker cutting a sentence that was true, the judge misreading a right answer. Behind several of them was one cause. The product, the assistant's instructions and the docs used different names for the same thing:

  • The console's top bar said Pages. The same room's label said Content.
  • Older docs described a Globals studio. It had become Navigation & Pages.
  • The docs told desktop-app users to look under Settings. The tab is called System.

The last one turned the story around. In the update answer from Change 1, the assistant had said System. That part was right. Our docs were wrong, so nothing could back the one correct word in its reply.

So every product name now has exactly one home: a glossary that ships with the code.

  • Tests fail if a console label drifts from it.
  • A release can't ship while the docs use an old name.
  • The public Glossary page is generated from it.

We settled the words while we were at it. Your whole workspace is your studio. Each part of it is a room. You talk to an assistant. And we added What Atelier doesn't do, so that "no" has a page to link to.

Then we rewrote the docs so that every section says how to get there: "open the Identity room", never "go there".

After the glossary and the rewrite: 19 of 20 questions passed all three runs, and 59 of 60 runs overall. The one miss, a question about colours where the assistant skipped the search, now passes too. A full run costs about $4.50, cheap enough to repeat after every docs change.

Four lessons for anyone building an assistant

  1. Giving a model the right page is not enough. It will still add to the page. Only a check against the page stops that.
  2. Let a separate call do the checking, and show it only the evidence. A model reviewing its own answer, with the whole conversation in view, tends to agree with itself.
  3. "I don't know" is a good answer. One confident wrong answer costs more trust than ten honest ones.
  4. Your assistant's mistakes are an audit of your docs. Some of what looked like invention was our own naming drift, and once, the assistant was right and the docs were wrong.

Try it

Ask the assistant how to make your own component. It will tell you that components come from packs that developers build, and link the page that explains it.

Then ask it something Atelier doesn't do. It will say so.

Thank you to everyone building with Atelier this early, and above all to the person who asked that first question. Reports like theirs are how we find out where the product, and our docs, are wrong. Please keep them coming.


Atelier is an AI-first website builder built on Drupal that runs as an appliance on your own machine, with your own provider key. The source is on GitHub; install is one command.