How every number is measured.

How we measure AI answers — and what we cannot measure

Every AI answer KoreLens shows was obtained by calling a named model through its API, with live web search on, sending the tracked question word for word. This table renders from the same configuration the calls use — it cannot drift from what production sends.

• One measurement is one answer. Weekly tracking asks each prompt once per engine per week and stores the answer verbatim with the exact model id, the date, and whether live search really ran on that call. AI answers vary between runs, so a single stored answer is a dated observation — evidence of what this instrument returned that day, not “the” answer. Where a claim needs tighter precision, we run repeated samples of the same prompt and report the rate with an interval instead of a single answer.
• One answer is an observation; a rate needs a sample. Where product-level claims need more than a single dated answer, KoreLens runs repeated samples of the same buyer question and reports an appearance rate with a 95% confidence interval (Wilson), never a “ranking” from one run. Two rates are called different only when their intervals do not overlap — otherwise the honest verdict is “no verdict yet”, and that is what the product says. Sample size and interval are shown wherever a rate is shown.
• If retrieval fails, we say so. A check that could not run with live search is stored and labelled “no live web search”, and is excluded from measured mention rates — an answer from the model’s stored knowledge is a different instrument, and we do not pool instruments. A comparison whose two sides ran under different retrieval states carries a method-change warning on the comparison itself.
• What we cannot measure: ChatGPT (chatgpt.com), Gemini (gemini.google.com), Claude (claude.ai), Microsoft Copilot — the consumer apps have no supported way to be measured programmatically. You (or your agency) can record what you saw there, word for word. Recorded answers are stored labelled “entered by you”, shown with that label everywhere, and are never counted in measured rates or before/after evidence. Measured and recorded answers are never blended — this is enforced in code, not just promised.

Exactly what we send — nothing hidden

The tracked question is sent word for word as the user message. Around it we set a short system prompt that frames the model as a general-purpose assistant and tells it to search before answering and to say so rather than guess when it cannot find reliable information — the same kind of frame every consumer surface applies, made visible. Answers are capped at 600 tokens; temperature is the provider default (the consumer apps do not pin it either); no location or personalisation parameters are sent, so answers reflect the providers’ default search context, not any specific buyer’s. The frame is versioned (presence-v1); if it ever changes, results measured under different versions are never compared as if the method held.

Why we do not sell “position” — and why held-method measurement is the stronger claim

The Fix Trail: what a closed loop actually records

One data defect followed all the way through: found, fixed, confirmed by an independent re-read, and — where the fix is tied to a tracked buyer question — that question asked again, with the verdict reported either way.

1. 1 · Detected. A deterministic check found the defect — sources disagreeing, a product changed from the record you accepted, or a readiness check failing on a live page. The values and the dates that triggered it are stored, not a model’s summary of them.
2. 2 · Fixed. The correction is recorded with who authorised it and when. Where a fix is applied outside KoreLens — most are — the trail says so rather than implying we did it.
3. 3 · Confirmed by an independent re-read. We read the data back from the source and check it actually changed. A successful API response is not a confirmation, and we do not treat it as one; only the re-read can move this stage.
4. 4 · Re-measured, where there is something to re-measure. If the fix is tied to a tracked buyer question, that question is asked again under the frozen conditions of the first measurement, and the two appearance rates are compared with their intervals. If it isn’t tied to one, the trail says that plainly instead of showing a stage that can never arrive.

A stage that hasn’t happened reads as not yet reached, never as a failure. Every stage carries its date and where the fact came from. And the verdict is reported whatever it is — see below.

Prompt coverage: how many buyer questions surface you at all

Prompt coverage is the share of tracked discovery questions — realistic buyer questions that do not name the business — whose latest answered check surfaced the business. The denominator is answered questions only: a question no engine has answered yet is stated as unanswered, never counted as a miss. This is the same number the dashboard has always computed for discovery; the name is the one practitioners use for it. It measures whether the business appears at all — not position, not preference, not traffic.

Cited share: a percentage, not a ranking

For each competitor domain, cited share is the share of successful checks in the window whose stored answer cited that domain — with the sample size and a 95% interval, on the same sample gate as every other rate. The denominator is the same eligible check set the mention rate uses: API-measured, live-search answers that passed the name-collision gate. Aggregators, press pages and known name-collision domains are excluded from the table using the same classification that files them under “also cited” — and the surface says the exclusion happened. It is never presented as a ranking: being cited says an engine used the domain, nothing about position or preference, and no “#1 cited source” claim exists in the product.

Sentiment: how answers characterise you — a model judgment, labelled as one

Sentiment is the least deterministic number in this product, and it is built so that fact is visible rather than smoothed over. For each stored answer that passed the name-collision gate and actually mentions the business, a judge model classifies how the answer characterises it: favourable, neutral, unfavourable — or factually wrong, a separate outcome that is never folded into “unfavourable”, because a flattering description of the wrong company is not positive sentiment. “Wrong” means: contradicts your saved record, which is the only authority the judge is given.

• Every value names its instrument. Each judgment stores the judge model and rules version that produced it, and links to the verbatim answer it was derived from. A judgment whose supporting sentence cannot be found verbatim in the stored answer is rejected, never stored.
• Excluded answers get no value at all. Answers the name-collision gate excluded or flagged are not judged — not given a neutral score, not counted in the denominator. No characterisation of someone else can describe you.
• Same sample gate as every other rate. Sentiment is reported as a rate with its sample size and a 95% interval. Below the sample gate there is no sentiment verdict, and the product says so instead of showing one.
• Never blended. Sentiment is a fourth, separate scale. It is never merged into the readiness score or any other number, and deterministic checks are never averaged with model judgments.

What may be claimed from it: “of the N judged answers, X were classified unfavourable by the named judge model, and here is each supporting sentence.” What may not: that sentiment predicts traffic, rankings or sales; that a model’s classification is a fact about your business; or any figure at all when the sample is below the gate.

What counts as a change worth telling you about

KoreLens emails you when something it measures actually moves. Because an alert interrupts you, the bar for sending one is a published rule rather than a judgement call. An AI answer is treated as changed only when the finding changes: whether you were named, which sources were cited, or whether the question drew a usable answer at all.

• Rewording alone is never an alert. Engines rewrite the same answer constantly. If you are still named, the same sources are cited, and the viability verdict is unchanged, nothing is sent — the stored answers still differ, and you can still read both side by side.
• A week we could not measure is never a movement. If a check was refused or the page could not be read, there is no score for that week — so there is no fall to report. Gaps stay gaps.
• Answers about a different company are excluded. Where our checks indicate an engine described a similarly-named business rather than yours, that answer is left out of alerts for the same reason it is left out of your rates. It is surfaced separately, as its own finding.
• Changed conditions are not a changed answer. If one check ran with live retrieval and the other did not, the difference is our method, not their answer, and the two are never compared.
• A newly cited domain is filtered first. Marketplaces, press and reference pages, and sites whose name merely resembles yours are demoted before anything is sent. A domain we cannot categorise is still shown — refusing to guess must not become a way of hiding a real competitor. On a client’s first measured week nothing is sent at all: with no earlier week to compare against, every cited domain would be “new” only because we had not looked before.
• Small score movement is noise, and you can ask for less of it. A readiness move under five points out of 100 is check-level noise and is never recorded as a change. Above that, you can raise your own threshold so only larger moves email you — that setting decides what reaches your inbox, never what we record. The measurement and your quietness preference are kept apart on purpose, so the stream stays a complete account of what we observed.

Every alert states the change itself and links to the dated evidence for it. None of them explains why the change happened: we report that two measurements differ and show you both. Readiness movement is a measure of what AI and search systems can read on a site — never a traffic, ranking, or sales figure.