How to defend an AI analysis in front of a client
Six ways an AI analysis goes wrong, six checks that catch them before the call, and one prompt that gets you all six with every answer.
If you use AI to go faster, you inherit the job of proving it's right
If you're not using AI to accelerate your quest for insights and measure the performance of the efforts you're making to grow sales, lift retention and move your KPIs, you're probably behind. And you're probably spending more time wrangling data than you should.
But it's a catch 22, because when you use AI to accelerate your analysis you're handed a new problem. You have to audit it, and you have to be able to show that the analysis is accurate.
Execution is everything, but insight is what separates you
Execution is everything. But what separates you is insight, true, non-intuitive, non-obvious insight, that is defendable and trusted. You can have all the execution in the world, and if it doesn't yield non-obvious insights, you're one of the pack and you're not going to break out.
The super strategists are the ones who get a lot more quality work done with AI, and it's still their work. If you're at an agency, this is why they pay you. Yes, you might be babysitting an AI agent to do some of your work. But your boss or your client does not have the capacity or the expertise to babysit an AI agent the way you can. And babysitting an AI agent is important work in itself: directing it, teaching it, managing it, and executing on what it gives you.
Checking your work was always the job, long before AI
I worked in investment banking. I was pulling insights for the CEO at Roblox, analyzing user data to find product metrics, and building our search algorithms. I have lived and breathed in spreadsheets. I have designed personalized recommendation algorithms, and systems that sync profile properties to Klaviyo so everybody gets a tailored recommendation.
Manual or AI, you have to check your work. If you're not checking your work, the mistakes surface eventually, and that was true long before any of this.
The further you are from how the data was pulled, the harder it is to defend
In some cases you went further than you realized, or further than you could have gone yourself. The agent surfaced data you weren't even considering. And rather than exporting the data yourself, the agent was given it through an MCP server or straight out of the warehouse.
The more removed you are from the work that's getting done, the harder that work is to check and to trust. When a client challenges it as inaccurate, rightly or wrongly, it's much harder to defend, because you may not know exactly where the data came from.
A reasonable-looking filter broke four of six numbers in one report
A retention strategist was trying to explain a decline in her client's Klaviyo revenue in the context of the broader business, across other channels like paid social. She had an AI agent plugged in to a warehouse, and the agent found Shopify data with traffic broken out. It did the analysis, wrote the queries, pulled the data. She pulled the insights together and presented them, and the client said: this data is wrong, this data is wrong, this data is wrong.
The analysis was flawed, but the part he pushed back on was the part that was right.
Four of the six headline numbers in that report were wrong, and one filter did all of it. Shopify started sorting website visits into real people and bots on 7 October 2025. Before that date there was no such label, so every old visit is stamped human by default, not because anyone checked but because the label did not exist yet. The analysis filtered to human visits only and then compared this September against last September, which means it compared last year with bots included against this year with bots stripped out. That is not the same measurement twice.
And bots are not spread evenly. They pile up in email. Of that brand's Klaviyo-tagged visits in September, 62% were bots, against under 2% of its Meta traffic. So the filter gutted the email numbers and barely touched Meta.
Which is why the one number he challenged, Meta sessions up 58%, is the one that held up. Correcting the filter moved it to 59.9%, and his own reasoning pointed the other way too. Flat spend with falling CPMs buys more impressions, and more impressions at the same click rate means more clicks and more sessions. Cheaper CPMs are a reason to expect more traffic, not less.
And none of it mattered, because nothing in the report let her check any of it in the room. The agent was working without context on that data, and without the skill that trains it to help a strategist audit and defend the numbers.
Even when the work is correct, a net new insight is one the stakeholder has never seen before. Their default position is skepticism, and it should be. Before you change how a business runs and make decisions off the back of some data, you'd better be confident that data is correct.
So you're going to get challenged. Anything useful you put in front of a client, they will challenge.
So the question is what every retention strategist should do to be ready to defend the analyses their agents produce.
Part one: where an AI analysis goes wrong
Your data comes from one of four places, each with a different audit problem
1. A CSV you exported. You go into Klaviyo reports and pull flow performance or campaign performance. You go into marketing analytics and pull a saved export. In Shopify it might be a ShopifyQL pull or a product performance report. The advantage is that your client can run the same export in their own account, so it's obvious how to recreate the source data.
2. A platform's MCP server. An official one like Klaviyo's or Shopify's. These are typically free and they get you aggregated data straight from the source platform, fast. The downside for a retention strategist is that they don't combine. When the agent retrieves that information, it has to write code to join the pieces together, which puts more of the processing burden on the agent, gives you more to audit, and leaves more room for blunders.
3. A data warehouse. Warehouse data offers some of the most powerful insights, because a warehouse that's set up correctly can do deep cross platform analysis, and deeper analysis within a single source.
4. A browser screenshot. Not the most common, but increasingly common. More agents are being designed to use browsers well. An agent can open the browser, take a screenshot, save it, and read it through a vision tool.
The agent turns your data into new information three ways, and they are not equally safe
Once you hand that data to an agent, it starts doing all sorts of wizardry, and that's where the problems start. It can turn your data into new information in three ways, and they are not equally safe.
It can do LLM math. This is the most dangerous, and the least reliable. Ask a model what 1,025 times 943 is. If it answers without saying "let me write some code or use a tool that multiplies these together," it's at high risk of being wrong, because these models are designed to predict the next token you want to see, not to do complicated mathematical operations.
Most agents are trained to know this about themselves and will avoid it by default. But every so often one will tell you your flow performance fell from one number to another and it has swapped the two. It doesn't happen often, and when it does it's the truest of the true black box, because there's no trace of how it got that number.
It can use a tool. These range from purpose built tools to bespoke ones. Flaunt's system has a very simple tool that's just called math, because that part can't be left to chance.
It can write code. This is what the most advanced agents do now, because code is flexible and handles the easy operations and the hard ones equally well. The agent takes your source data and mixes, matches and combines it in a language like Python, doing the job that used to belong to a data science notebook or a very complicated spreadsheet.
And this is where it gets hairier. If you'd built that spreadsheet yourself, you'd know every operation in it, because you built the pivot tables. Here, the only way to audit it is to read the code, and now you're in unfamiliar territory. The agent can do more than you could have done in a spreadsheet, and it does it faster. But you can't check it unless you read Python, and in a normal chat interface it can be very difficult even to find the code it ran.
Six ways it goes wrong, and the last one has nothing to do with the data
1. It misunderstands your request. Then the whole thing is off the rails. You end up with results built on an entirely different data set than the one you had in mind, described in language reasonable enough that you present it anyway.
2. It understands you, but it makes an assumption. This is the most likely one. Sometimes the agent knows it is making an assumption, and sometimes it has no idea. Either way it will not catch that assumption unless you ask it to check its own work. It's analyzing the right data set, but it has filtered something out.
That's the traffic example from earlier. The agent made the mistake a human without context would make: it excluded bot sessions to get human sessions only. Shopify didn't start labeling sessions as bot or human until October 2025, so excluding bots lowers this year's number and leaves last year's alone.
What makes that one hard to catch is that it doesn't move everything by the same amount. Bots pile up in email and barely show up in paid social, so the same filter can destroy one channel's numbers and leave another channel's intact. A single wrong assumption doesn't produce a single wrong number. It produces a report where some numbers are correct and some are wrong, and nothing on the page tells you which ones.
The way to prevent this in the first place is to give your agent a lot of context on the data you work with. Scrape the documentation. Invest in teaching it everything you know. Take the best practices from everywhere else, and study how Shopify, Klaviyo, Skio and Recharge actually present their underlying data to users in their own dashboards.
3. It has a bug. These agents write code to do the work you'd do in a spreadsheet, and there might be a small bug it didn't catch.
4. It blunders in the last mile. The data was right. The code was sound. The processing was correct. The context was accurate. And then, delivering the answer, it switched two numbers, so flow revenue that went from X to Y gets reported as Y to X.
5. The warehouse is out of sync. A warehouse gives you more power and deeper insight, and it also sits a layer away from the source of truth. There's a processing layer between the platform and the warehouse, and every so often it drifts. Usually when that happens it's obvious.
6. The analysis is right and you can't defend it. This one has nothing to do with the data. The analysis holds up, the client challenges it, and you don't know exactly what you're presenting well enough to say so. As soon as it's challenged, the easiest thing to say is "I used AI for this, let me double check it," which hands the doubt straight back to the client.
Part two: what to do before the call
You have an analysis that looks great. Maybe you've done everything right, and you're using tools that make sure the data comes with strong context, so the agent knows the gotchas. Here's how I get from "the agent did something good and it looks right" to "this is an insight I can defend."
Your tool should be doing most of this for you already
Doing this with less human effort is becoming a required feature of any AI data analysis tool. Showing how you got there should be part of delivering the answer, not a separate step. The agent should check its own work and correct itself. It should give you traceability, so you can hover over any number and see its source.
In Claude, some of those capabilities may not be there natively. Even in Claude, an expert data platform's MCP should come with skills and context that make every piece of work follow a protocol, which raises the odds of an accurate analysis significantly. And every analysis should come with its methodology attached.
But let's say you're not using those tools.
1. Tie a few numbers back to the source of truth, because that is what your client will do
This is the number one thing you can do.
You don't need to check every number, especially if you ask the agent to catch its own slips:
Prompt. Double check whether you misgenerated any numbers.
The agent is usually very good at catching its own slips. What it can't do for you is decide what counts as truth. So wherever the information came from, know what the source of truth is. For Klaviyo, that's a number in the campaigns dashboard or the business review dashboard. For Shopify, it's a report like product performance, the overview dashboard, or a cohort analysis. Depending on your relationship with the client, you may not always have access to those directly.
Work with the agent to pick a few high level numbers that should tie out. If the analysis says certain products generated a certain amount of revenue in a window, open the product performance report in Shopify, pick two products, and check the numbers are in the ballpark. If you're working purely in Klaviyo data and it reports new customer revenue, that figure isn't easy to verify inside Klaviyo, but it should be close to what you'd see in Shopify.
This is exactly what your client is going to do. In that example, it is exactly what the client did. He opened a dashboard the strategist did not have access to and checked the numbers himself.
2. Ask it to attack its own methodology, and it usually finds something
Prompt. Audit your methodology and try to find some flaws.
Give it an adversarial disposition. When you ask it to find flaws, it usually finds some, even when the analysis is broadly right.
3. Make it explain the method simply, so you can repeat it back to a client
Prompt. Explain the methodology you used for this analysis like I'm five.
That prompt keeps the explanation out of hypertechnical language, so you can repeat the method back to a client.
4. Get the steps to rebuild it yourself, in a tool you actually speak
Prompt. Give me the steps to recreate this myself, from the source data through to the Excel analysis.
If you're in a tool like Claude, you can go further and have it rebuild the whole thing somewhere you're fluent:
Prompt. Recreate this with the same methodology in a spreadsheet, with raw data tabs for the flows export and the campaigns export, and Excel formulas doing the analysis.
Speaking Python might be challenging for you, but speaking in Excel formulas is probably easier. And the raw export matters on its own, because a lot of clients, especially DTC operators, run their business in spreadsheets. A raw export is a good way to share the underlying data, and it helps a stakeholder build trust in the analysis.
Even if you never build the spreadsheet, working out how you'd recreate it builds your own trust in a language you understand.
5. Check how it grouped things and what it weeded out
Investment bankers run this same kind of check before sending a complicated financial model to a company they are about to acquire.
Once you know what information was used and where it came from, look at the key processing steps. In a product level analysis, how did it group? In a campaign analysis, how did it group the campaigns? If the agent counted every campaign with the word promo in its name as a promo, that grouping might not hold up. Understand how it grouped things, what it weeded out, and what it combined.
6. Go fishing for outliers, because it will not control for them on its own
Every good insight yields more questions. People immediately ask why, and then whether it's a flaw in the analysis. They challenge the methodology. Did you account for free products? Did you exclude wholesale customers?
So look for anything that could be producing this result without it being a real effect. Are there wholesale orders at $3,000 an order in a business with a $60 average order value, throwing everything off? Are you including a channel like TikTok Shop that behaves nothing like the main Shopify site?
Then take it back to the agent and go one level deeper:
Prompt. Are there any outliers in this analysis that could be driving the result, that I should control for?
It's very good at finding anomalies when you point it at them. It's not good at knowing to control for them on its own. The more advanced your tool, and the more context you've built up on this client, the better those odds get.
This is only more work if you run it as a second pass
Running all six checks after the analysis is finished does add work.
So don't run it as a second pass. Every one of those six prompts is something you can ask for once, up front, and then get with every answer from then on. Paste the prompt below into your agent, and leave it there.
Prompt. Read this guide first: https://www.flaunt.ai/blog/how-to-defend-an-ai-analysis-in-front-of-a-client
From here on, an analysis isn't finished when you have the answer. It's finished when I can defend the answer in front of a client. So before you run anything, tell me what you think I asked for. Then with every analysis, give me:
- The tables and the date ranges you used, and how you filtered the dates.
- Every filter you applied, and especially anything you excluded. Where you excluded rows, show me the number both with and without the exclusion.
- Anything that changed in the source data partway through the period I am comparing, including any field that only started being populated partway through.
- The assumptions you made that I did not give you.
- A second check of the headline numbers, calculated a different way, and whether the two agree.
- The outliers you found, and what you did about them.
- The method in plain language, as if I have to repeat it to someone who does not work in data.
- How I would get to this number myself in the source platform, without you.
- Which of your numbers you are least confident in, and why.
Do not do arithmetic in your head, write code for it. Before you give me the final summary, check every number in it against the query output. Then give me the three questions a skeptical client is most likely to ask about this, and your answer to each.
The takeaways
It's going to get challenged, and that is the opportunity
If it's a useful business insight, your stakeholder may reject it at first, because it challenges an assumption they hold about their own business. That isn't something to be afraid of. It's the opportunity to prove a deeper level of value, the kind that makes you genuinely hard to replace. Being challenged also raises the stakes.
Show up with the sources, the checks and the method, and you're not scrambling
Come in with the sources of truth, the high level verifications you've already run, and your methodology.
Then when they push back, and they will, you're not scrambling. You already know where the number came from and you've already checked it, and so the client will trust you as a source of truth for non-obvious insights that can move their business forward.
And if the stakeholder says the insight is powerful enough that they want to run it themselves, you should be able to explain how they'd do it without AI at all.
Answer the first order questions before they're asked, and the meeting moves to the so what
Show up and say: here's the insight, here's where the data came from, here's how I checked it's accurate, here are the outliers I considered and controlled for. Given the speed of AI, you can go further and answer the first round of follow-ups before the meeting. The first order questions are usually about outliers. The second order ones get closer to the so what.
Do that, and your client considers you irreplaceable. They keep you longer than they otherwise would, because they know you're looking out for their business and helping them find the truth. And truth is the most important thing to growing a business.
You can also use a tool like Flaunt that aggregates all of that context for you, so your agents work from that context by default.
So deliver breakout results. Show up to your meetings with insights, non-obvious insights you can stand behind. Deliver outlier results.
Do these things and you'll be irreplaceable.
Connor Kelley helps agencies like New Standard Co, Homestead Studio and Power Digital build custom AI software. He is the CEO of Flaunt, the AI Data Analyst for Retention Strategists that answers any question 1000x faster with verifiable accuracy and makes recommendations to drive LTV.
