---
title: "7 Questions About AI Visibility Sampling: A Practical Guide"
url: "https://jayyseo.com/ai-visibility-sampling-7-questions/"
date: "2026-11-17T07:21:05+00:00"
updated: "2026-11-17T07:21:05+00:00"
author: "Jay"
description: "AI visibility sampling made simple: 7 questions on prompts, engines, run frequency and sample size, with a worked example and how to pick your next action."
category: "AEO"
---

# 7 Questions About AI Visibility Sampling: A Practical Guide

Ask ChatGPT the same local question twice and you can get two different shortlists. Ask it from another city, or a day later, and the list can shift again. That is normal behavior for AI assistants, and it is the reason AI visibility sampling matters so much for anyone who reports on local recommendations.

AI visibility sampling is the practice of asking a fixed set of customer questions, across chosen AI engines and locations, several times over a set period, and then measuring how often your business shows up. One answer is a snapshot. A planned sample is evidence you can act on.

This guide works through seven questions I ask before I trust any AI visibility number. I use one illustrative business from start to finish, show the exact checks I run, explain what to record from each answer, and finish with how to compare the results and pick a next action. RadarKit is the tool I use for this, so I cover it where it fits the buying decision.

## What Is AI Visibility Sampling?

Think of it like a poll. A pollster does not call one voter and announce the result. They pick a group, ask the same questions, and report a rate with a margin of error. AI visibility sampling applies the same idea to AI answers.

Your sample has four parts:

- **The question set:** the prompts a real customer would type
- **The engines:** some or all of ChatGPT, Gemini, Perplexity and Copilot, plus Google’s AI Overviews and AI Mode
- **The location:** the country or city the question is asked from, or the city named in the prompt
- **The window:** how many days you collect answers, and how often each prompt runs

The output is a rate, such as “named in 41 of 112 answers,” rather than a yes or no. Rates can be compared week to week. A single yes or no cannot.

## Why AI Answers Change From One Run to the Next

Before you plan a sample, it helps to know why answers move. There are several reasons, and none of them mean a tracker is broken.

- **Built-in randomness:** Large language models pick words with some probability, so wording and list order change between runs.
- **Live web retrieval:** Many engines search the web before answering. The pages they pull can change daily, and so can the businesses those pages mention.
- **Location signals:** An answer for a user in one city can differ from an answer for a user in another, even with the same words.
- **Model and product updates:** Engines ship changes often, and a new version can reshuffle which sources get cited.
- **Prompt wording:** “Best emergency locksmith in Charlotte” and “who can unlock my car fast in Charlotte” are close in meaning but can return different names.

RadarKit’s own terms say AI platforms vary by time and location, and no tool can promise a particular mention. That is exactly why you sample. You stop asking “were we mentioned?” and start asking “how often are we mentioned, and is that rate moving?”

## The Worked Example Used in This Guide

Every question below uses the same made-up business, so the numbers connect. Swap in your own details and the method holds for salons, roofers, physical therapy clinics, storage sites, law offices or any other local service.

| Setting | Value in this walkthrough |
| --- | --- |
| Business | A locksmith company with three locations (illustrative only) |
| Service | Emergency lockout help |
| Locations | Charlotte, Concord and Gastonia, North Carolina |
| Prompts | 4 per city, 12 in total |
| Engines | All six (four assistants plus the two Google AI features) |
| Date range | 28 days, with a matching 28-day window afterward for comparison |
| Goal | Learn how often the company is recommended for lockout help in each city, and what to fix first |

The business is fictional. The steps, formulas and RadarKit plan limits are real. Any sample answers in this guide are labeled as illustrations so you can see the format without mistaking them for measured data.

## 7 Questions About AI Visibility Sampling

Here is the full list before we go deep on each one:

1. What exactly are you sampling?
2. Which prompts belong in the sample?
3. Which engines and locations should each prompt run on?
4. How often should each prompt run?
5. How many answers do you need before you trust a number?
6. What should you record from every answer?
7. How do you compare results and choose the next action?

## Question 1: What Exactly Are You Sampling?

Most weak samples fail here. The team tracks “AI visibility” in general, mixes several services and cities into one score, and then cannot explain why the number moved.

Lock the scope before you write a single prompt. For the locksmith company, the scope reads like this: one business, one service (emergency lockout help), three named cities, six engines, 28 days. Anything outside that box goes into a separate sample.

### The check

Write a one-line scope statement and keep it at the top of your tracking sheet or project notes:

- Business name and the brand aliases AI might use
- The single service being measured
- The cities or region
- The engines
- The start date and end date of the window

### Why the date range matters

A defined window lets you compare like with like. If the first 28 days are your baseline, the next 28 days become the test period after any change you make. Without fixed windows, a busy week or a model update can look like progress or decline.

## Question 2: Which Prompts Belong in the Sample?

Your prompts are the questions you are sampling from, so they need to match how real customers ask. A set built only from branded searches will make visibility look far better than it is.

For local services I mix four prompt types:

- **Unbranded discovery:** “Who is the best emergency locksmith in Charlotte?”
- **Problem-first:** “Locked out of my house in Concord at night, who can help quickly?”
- **Comparison:** “Which locksmiths in Gastonia have the best reviews for lockouts?”
- **Branded check:** “Is [company name] a good locksmith in Charlotte?”

In the example, each city gets three unbranded or problem-first prompts and one branded prompt. That gives 12 prompts in total.

### Keep the set fixed

Once the window starts, do not swap prompts in and out. If you change the questions halfway, you cannot tell whether a rate moved because of your work or because the sample changed. Add new prompts at the start of the next window instead.

### Use paraphrases on purpose

Real people phrase the same need in many ways. Two or three wordings of the same intent give a fairer picture than one perfect keyword. Group those wordings under one topic so you can read them together.

## Question 3: Which Engines and Locations Should Each Prompt Run On?

Each engine builds answers differently. Perplexity leans hard on live sources, Google AI Overviews and AI Mode sit inside Google results, and ChatGPT can answer with or without browsing. A business can do well on one and barely appear on another.

### Engines

If budget allows, run all six. Customers do not all use the same assistant, and a blended score hides big gaps. Always keep a per-engine view next to the total.

### Locations

Location affects local answers. There are two ways to handle it:

- **Name the city in the prompt.** “Emergency locksmith in Gastonia” tells the engine where you mean, even when the request comes from elsewhere in the country.
- **Ask from inside the city.** Some tools can send the prompt from an IP located in a chosen city, which is closer to what a resident sees.

RadarKit sends prompts through residential IPs. Its Lite, Growth and Pro tiers reach 50+ countries, and targeting by state or city is reserved for Enterprise. So for the three locksmith cities on a self-serve tier, I would pick a United States location and put Charlotte, Concord or Gastonia right in the question.

### Logged-out, consistent conditions

Personal history can change answers. A clean, consistent setup for every run keeps the sample comparable. Doing this by hand in your own browser is hard, which is one reason teams use a tracker for AI visibility sampling.

## Question 4: How Often Should Each Prompt Run?

Frequency decides how much of the natural variation you capture. Ten runs fired in the same minute mostly measure randomness in wording. Ten runs spread over ten days capture changes in sources, updates and day-to-day shifts.

### The answer-volume formula

Use one simple formula to plan any AI visibility sampling project:

**Answers collected = number of prompts x number of engines x scheduled runs**

For the locksmith company on a daily schedule over 28 days:

- 12 prompts x 6 engines x 28 runs = 2,016 answers in the window
- Per city: 4 prompts x 6 engines x 28 runs = 672 answers
- Per city on one engine: 4 prompts x 28 runs = 112 answers

Slow the refresh to once every 72 hours and those 28 days hold only nine or ten passes for each prompt. The total drops to somewhere between 650 and 720 answers, which is about 220 to 240 for each city.

### Spread runs across the window

Daily runs are the easiest way to spread a sample evenly. If your plan refreshes less often, extend the window rather than squeezing more into a short burst. A 72-hour cadence over six weeks can give a steadier read than a daily cadence over one week.

### When to add a fresh run

A manual “run now” check is useful right after you publish a page, update a listing, or notice a model update. Treat that run as a spot check, not as a replacement for the scheduled sample.

![RadarKit plan table showing prompt frequency and responses analyzed for AI visibility sampling](https://jayyseo.com/wp-content/uploads/2026/11/ai-visibility-sampling-radarkit-prompt-frequency.jpg)

## Question 5: How Many Answers Do You Need Before You Trust a Number?

Most guides on this topic spend the bulk of their time here, and for good reason. Small samples swing wildly. A rate built on ten answers can be off by a lot.

### A quick margin-of-error guide

When the metric is a simple named or not named, statisticians use a worst-case 95% margin: take 0.25, divide by n (your answer count), find the square root, and multiply by 1.96. The results look like this:

| Answers in the sample (n) | Worst-case margin, plus or minus |
| --- | --- |
| 10 | about 31 points |
| 25 | about 20 points |
| 50 | about 14 points |
| 100 | about 10 points |
| 200 | about 7 points |
| 400 | about 5 points |

So a rate of 40% from 10 answers could really sit anywhere from about 9% to 71%. The same 40% from 400 answers is pinned to roughly 35% to 45%. That gap is the whole case for planned AI visibility sampling.

### More prompts usually beat more repeats

Repeat runs of one prompt tend to look alike, so they carry less new information than the same number of answers spread across different prompts. If you want a reliable overall rate, add more prompts first. Add more repeats when you need to report on a single prompt that keeps flipping.

### Reading the locksmith example

With 112 answers per city on one engine, a city-level rate on that engine has a margin of about 9 to 10 points. The full 672 answers per city across all six engines tightens the blended city rate to around 4 points, as long as you remember the engines behave differently. That is enough to spot a big gap between Charlotte and Gastonia. It is not enough to celebrate a 3-point rise.

## Question 6: What Should You Record From Every Answer?

A sample is only as useful as the fields you capture. “Mentioned or not” is the starting point, but the next action usually comes from the details around it.

### The fields I log

- **Mentioned:** yes or no for your business
- **Position:** where you appeared in the list, if listed
- **Other businesses named:** every competitor in the answer, with rating and review count if shown
- **Sources:** pages the answer cited with a link, and pages it read without linking
- **Searches run:** the background web queries the engine made before answering
- **Sentiment:** how the answer described you, if it did
- **Run details:** engine, location, date and time

### A sample log (illustrative values only)

The rows below are made up to show the format. They are not measured results.

| Run | Prompt | Engine | Named? | Position | Businesses named | Sources cited |
| --- | --- | --- | --- | --- | --- | --- |
| Day 1 | Emergency locksmith in Charlotte | Perplexity | Yes | 3 | Competitor A, Competitor B, you | Yelp listing, local news roundup |
| Day 1 | Locked out in Concord tonight | ChatGPT | No | n/a | Competitor C, Competitor D | Google Business Profiles |
| Day 2 | Emergency locksmith in Charlotte | Perplexity | Yes | 2 | You, Competitor A | Yelp listing, your service page |
| Day 2 | Best locksmith reviews in Gastonia | Gemini | No | n/a | Competitor E, Competitor F | Directory list page |

### Why appearance rate beats rank

Rank inside an AI list jumps around between runs. The share of answers that include you holds much steadier and compares cleanly across windows. Keep average position in the report, but lead with appearance rate.

![RadarKit records AI answers from real chat interfaces across locations for AI visibility sampling](https://jayyseo.com/wp-content/uploads/2026/11/ai-visibility-sampling-radarkit-real-chat-locations.jpg)

## Question 7: How Do You Compare Results and Choose the Next Action?

Now the sample turns into a decision. Compare in a fixed order so you do not chase noise.

### Step 1: Compare against the margin

Is the change between two windows bigger than the margin for that sample size? If Charlotte went from 38% to 41% on 112 answers, that is inside the noise. If Gastonia went from 12% to 30% on 672 answers, that is a real shift.

### Step 2: Split by engine and city

Look for one weak cell rather than one weak total. A business can be strong on Perplexity in Charlotte and nearly absent on Gemini in Gastonia. The fix for each is different.

### Step 3: Read the sources behind the winners

Open the cited pages for the answers where a competitor won. Directory listings, local roundups, review sites and service pages all point to different work.

### Step 4: Match the evidence to an action

| What the sample shows | Likely next action |
| --- | --- |
| Named on branded prompts, rarely on unbranded ones | Earn mentions on the third-party lists and roundups AI cites |
| Competitors win with directory or listing sources | Update and complete those listings and your Google Business Profile |
| Background searches use wording your site lacks | Add a clear page section that answers that exact question |
| You appear, but low in the list | Strengthen reviews, service details and proof on cited pages |
| Weak on one engine only | Study that engine’s cited sources and fill the gap there |

### Step 5: Re-sample with the same setup

Make one change, then run the next 28-day window with the same prompts, engines and locations. If the rate moves beyond the margin, you have evidence the change worked.

In the locksmith walkthrough, a plausible outcome looks like this: branded prompts go well, unbranded prompts lag in the two smaller cities, and directory listings sit behind the bulk of rival wins. That points to cleaning up directories and listings in Concord and Gastonia first, followed by a second window to check the effect.

## Common Mistakes in AI Visibility Sampling

- **Reporting single runs:** A screenshot of one answer is a story, not a measurement.
- **Changing prompts mid-window:** It breaks the comparison.
- **Blending engines:** One total score hides the engine that needs work.
- **Short bursts:** Many runs in one hour measure wording randomness, not real change.
- **Chasing rank:** Position swings more than appearance rate does.
- **No fixed date range:** Without a start and end, trends cannot be compared fairly.

## RadarKit for AI Visibility Sampling

[RadarKit](https://radarkit.ai/) tracks how AI assistants talk about brands and local businesses. Instead of calling model APIs, it types each prompt into the consumer apps people actually use: ChatGPT, Perplexity, Gemini and Copilot, along with the AI Overviews and AI Mode results inside Google. Every reply is stored. For AI visibility sampling, that takes care of the schedule, the consistency and the record keeping, which are the hardest parts to manage by hand.

![RadarKit homepage showing Visibility Score and Brand Reputation for AI visibility sampling](https://jayyseo.com/wp-content/uploads/2026/11/ai-visibility-sampling-radarkit-homepage.jpg)

### How RadarKit Maps to the Seven Questions

A project holds the scope from question 1, and topics hold the prompt groups from question 2. You pick engines for question 3. Question 4 is covered by the plan’s refresh cycle, which is once every 72 hours on Lite and once a day on Growth or Pro. For question 5, the Visibility Score reports what share of prompt and engine runs mentioned the brand, set against the period before. Local Radar gives the appearance rate for local answers, such as the 3 of 9 answers shown in the feature demo, with each rival business listed. Citation Data and Query Fanout supply the sources and searches that question 6 asks you to log.

### RadarKit Key Features

- **Consumer-app tracking:** Each prompt is asked inside the real ChatGPT, Perplexity, Gemini and Copilot apps, plus Google’s AI Overviews and AI Mode, using residential IPs, with 50+ countries to choose from.
- **Prompts and topics:** Group the questions customers ask under themes. They rerun automatically, and a run-now button gives an instant reading whenever you need one.
- **Visibility Score:** A percentage of prompt runs, across models, in which your brand came up, with the change from the last period.
- **Brand Reputation:** A separate score for the way AI engines describe your brand, followed over weeks and months.
- **Average Position:** The rank you usually hold when you are listed, compared with the other domains in the same answers.
- **Sentiment:** Tags each mention as positive, mixed or negative for every model and in total, and pulls out phrases that keep repeating.
- **Local Radar:** For local prompts, it shows if you were recommended, your appearance rate and rank, and each competing business with its category, stars and number of reviews. Listing platform filters (Yelp, Google Business Profile and others) show which prompts every business wins. Targeting by state or city is an Enterprise feature.
- **Shopping Radar:** Captures AI shopping carousels: the product, the store, the price, the rating and the slot, along with which retailers come out on top.
- **Ads Radar:** Records the sponsored spots around AI replies: the advertiser, the headline, the copy, the landing URL and the creative, plus how frequently each ad turns up.
- **Citation Data:** Splits sources into cited and consulted, flags which ones link back to you, and shows the page formats engines like to cite.
- **Page Radar:** Keep watch over a single URL, a guest post for example, and learn which engines reference it and for which questions.
- **Query Fanout:** Reveals the web searches a model performed before it replied, and how often each model searches in the first place.
- **Competitors and Share of Voice:** Lists rival brands and domains from the same answers, sorted by share of voice, with each rival’s usual placement beside it.
- **Create Content:** Give the Content Agent a prompt. Before it drafts anything in your voice, it reviews the sources AI cites, the top Google results, the fanout searches and any People Also Ask questions. Formats on offer: blog posts, list-style posts, head-to-head comparisons, alternatives roundups, first-person reviews and FAQ pages.
- **Optimize Content:** Send an older post in to be rebuilt around the entities and fanout terms engines already reward, polish it, and push the finished version to WordPress.
- **AI Traffic Analytics:** Connect GA4 in read-only mode to see sessions referred by AI assistants right beside your visibility trend.
- **Plan Agent:** Hooks into Search Console on any tier, reads it next to your citation findings, and writes a gap list that the other agents then work through.
- **Outreach Agent:** Tracks down websites that cite your rivals, writes outreach in your brand voice, sends it through Gmail once you sign off, and handles follow-ups.
- **Reddit Agent:** Shows the Reddit threads that AI answers lean on in your category, and where your brand is still absent from them.
- **Chat Agent:** Lets you question a project in everyday language, with replies drawn from the live data.
- **API and MCP access:** Each plan includes REST API keys and an MCP server, so Claude, ChatGPT, Codex or Cursor can read your data directly.
- **Exports, branded PDFs and Client Portal:** Download answers, sources, rival lists, rankings, fanout and radar results as files. Lite includes 2 branded PDF reports per month, Growth 10 and Pro 20, and clients can follow along in a view-only portal or by shared link.
- **Full history:** No plan trims old data, so this period can always be lined up against the last one.
- **Projects and workspaces:** The project allowance climbs from 2 on Lite to 5 on Growth and 10 on Pro. Enterprise is unlimited.
- **Project settings and seats:** Keep your brand kit, the alternate spellings of your name and your rivals’ names, your time zone and user roles in one place. Seat count: 2 extra on Growth, 5 extra on Pro, unlimited with SSO on Enterprise.
- **Free GEO tools:** No-signup freebies include a validator and a generator for LLMs.txt files, plus a Chrome extension for AI SEO work.

### RadarKit Pricing

![RadarKit pricing plans compared for AI visibility sampling](https://jayyseo.com/wp-content/uploads/2026/11/ai-visibility-sampling-radarkit-plans.jpg)

Here is what the [RadarKit pricing page](https://radarkit.ai/pricing) shows. All four plans include the same six engines.

| Plan | Price | Projects | Prompts | Refresh | Responses analyzed |
| --- | --- | --- | --- | --- | --- |
| Lite | $29 per month | 2 | 15 | Every 72 hours | 900 per month |
| Growth | $79 per month | 5 | 50 | Daily | 9,000 per month |
| Pro | $139 per month | 10 | 100 | Daily | 18,000 per month |
| Enterprise | Custom, billed annually | Unlimited | Custom | Daily or weekly | Custom |

A few extra details matter for sampling. Lite, Growth and Pro cover 50+ locations, while Enterprise covers 70+ and adds tracking at the state and city level. Growth comes with 2 extra seats and Pro with 5, plus priority support. Enterprise adds SSO, unlimited seats, custom contract terms and a dedicated manager.

You can try any self-serve plan free for 7 days without paying upfront. After the trial converts, RadarKit offers a full refund if you get in touch within 48 hours of being charged.

In the locksmith case, the 12 prompts fit on Lite. The catch is volume: a 72-hour cycle across 28 days yields roughly nine or ten passes per prompt. Growth runs every prompt every day and leaves space for extra paraphrases, which narrows the margins from question 5.

### Best For

- Service businesses with one or many locations that need appearance rates they can defend
- Agencies running AI visibility sampling for a roster of clients
- Lean teams that want tracking, source evidence and follow-up work in a single dashboard

## Final Thoughts

AI visibility sampling turns a moving target into a number you can trust. Set the scope, fix the prompt set, choose engines and locations, spread the runs, size the sample, record the details, and compare against the margin before you act.

The locksmith company is invented, yet the steps carry over to any local service. Run one clean window, make one change based on the evidence, and run the next window the same way. For more on this, see how to [build a baseline for local AI visibility](https://jayyseo.com/how-to-set-a-local-ai-visibility-baseline-5-steps/), how to [track local AI visibility changes](https://jayyseo.com/how-to-track-local-ai-visibility-changes-5-step/), and how to [compare AI visibility across cities](https://jayyseo.com/how-to-compare-ai-visibility-across-cities/). If you are weighing tools, my guide to [checking an AI visibility tool’s data quality](https://jayyseo.com/check-ai-visibility-tool-data-quality/) and the breakdown of [branded vs unbranded local prompts](https://jayyseo.com/branded-vs-unbranded-local-prompts-3-differences/) are good next reads.

## FAQs

### What is AI visibility sampling?

It is a planned way to measure how often AI assistants recommend your business. You run a fixed set of prompts on chosen engines and locations several times over a set date range, then report the share of answers that named you.

### How many times should I run each prompt?

Spread the runs across the window instead of firing them together. Daily runs over four weeks give about 28 per prompt per engine, which is a solid base. If you need a tighter overall number, add more prompts before adding more repeats.

### Why does the same prompt give different AI answers?

Models choose words with some randomness, many engines search the live web, and location and product updates change what they find. That is why a rate from many answers is more useful than any single answer.

### Can RadarKit handle AI visibility sampling for several locations?

Yes. Each location or brand can have its own project, and Lite allows 2, Growth 5 and Pro 10. Write the city name into every prompt. The self-serve tiers ask via residential IPs covering over 50 countries, while state or city targeting needs an Enterprise contract.

### How often does RadarKit refresh prompts?

Tracked prompts rerun every 72 hours on Lite and every day on Growth and Pro. Enterprise contracts can choose daily or weekly. After a change on your site, you can also trigger one prompt instantly for a spot check.

---

Source: https://jayyseo.com/ai-visibility-sampling-7-questions/
