AI Pilot Project Scope: How Agencies Avoid the Data-Quality Trap

AI pilot project scope is becoming one of the most important commercial documents an agency can create. As more ecommerce brands buy AI assistants for support, sales, and product discovery, the apparent simplicity of “connect the catalog and answer customer questions” often collides with a far messier reality: incomplete, contradictory, and ownerless business data.

A recent post in r/SaaS captures the problem neatly. A small services business sold an ecommerce AI-assistant pilot, only to discover that the client’s website and spreadsheets conflicted, product IDs were not reliably unique, descriptions were missing, and key customer-facing attributes were scattered across systems. The agency spent nearly a month cleaning, indexing, and constraining the knowledge base—then faced refund pressure after the client found issues during a short set of live test conversations. (reddit.com)

The lesson is not that clients are unreasonable or that AI pilots are doomed. It is that an AI pilot is a joint delivery process with dependencies, not a switch that an agency flips. If the scope describes only the intended outcome while ignoring the condition of the source data, the acceptance criteria, and the client’s responsibilities, both sides can be technically right and commercially disappointed.

Why AI pilots fail before the model is even involved

Most client conversations start with the visible layer: the chat interface, the assistant’s tone, product recommendations, and anticipated ticket deflection. That is understandable. Those are the parts customers see.

But the operational layer determines whether that visible experience can be trusted. An assistant can only retrieve, reason over, or cite the information made available to it. It cannot reliably resolve a conflict between a product page that says one thing and a spreadsheet that says another unless someone defines which source wins. It cannot supply a warranty term, sizing dimension, compatibility rule, or delivery exception that the business has never recorded.

The r/SaaS account is a classic example of a hidden implementation risk becoming unpaid delivery work. The initial brief sounded bounded: connect product information, then answer a defined set of questions. The actual work expanded into catalog remediation, information architecture, knowledge-base design, guardrails, and testing. (reddit.com)

That distinction matters because these are different services with different economics:

• Assistant configuration turns reliable, structured information into a usable customer experience.
• Data remediation identifies, reconciles, enriches, and governs unreliable information.
• Workflow design determines what the assistant can do, what tools it can access, and when it must hand a conversation to a person.
• Evaluation measures whether the deployed system meets the agreed quality and safety bar.
• Ongoing operations maintains information freshness, reviews failures, and updates policies as the business changes.

A client may perceive all five as “the chatbot.” An agency that prices them as one undifferentiated setup fee is taking on a major margin and expectation risk.

The hidden dependency: a source of truth

The phrase “source of truth” can sound like project-management jargon. In a customer-facing AI system, it is a practical safety control.

A source of truth is the explicitly designated system, file, or owner whose version of a fact governs when competing information exists. For an ecommerce merchant, there may be separate authorities for product specifications, inventory status, price, promotions, shipping estimates, returns, and policy language. The important point is not that every company must consolidate everything into one database. It is that each answerable fact needs a documented owner and precedence rule.

Why websites and spreadsheets are not automatically knowledge bases

A website is often optimized for merchandising, not accuracy at the field level. Product copy may be written for search or brand tone. Collection pages may summarize or simplify details. Old variants can persist in search indexes. Specifications may sit in PDFs, supplier sheets, a product information management system, or a staff member’s inbox.

Spreadsheets introduce a different set of problems. They may contain useful operational fields but lack validation rules, stable IDs, controlled vocabularies, timestamps, or a clear owner. When the same product has two names, duplicate identifiers, and different values in several tabs, indexing the sheet does not solve the underlying ambiguity. It simply makes ambiguity available to the model.

A retrieval-augmented generation system, or RAG system, can reduce unsupported answers by grounding responses in supplied information. However, retrieval does not magically establish truth. If the retrieved documents conflict or omit the exact attribute a shopper needs, a model may produce an uncertain answer, decline to answer, or—without sufficient guardrails—make an unsupported inference.

This is why “we have a website and a CSV” should never be interpreted as “our data is ready for AI.” It should mean “we have candidate inputs for a data-readiness review.”

The minimum source-of-truth matrix

Before a build starts, the agency and client should jointly complete a simple matrix. It does not need enterprise governance software. A spreadsheet, shared database, or project document is sufficient if it is maintained.

The crucial field is not merely the link to the source. It is the owner. A file cannot accept responsibility for clarifying a contradiction. A named person can.

A better way to define AI pilot project scope

A useful AI pilot project scope does not promise that the assistant will “work.” That phrase is too subjective. Instead, it specifies the operating domain, the inputs, the intended behaviors, the boundaries, and the evidence required to decide whether the pilot succeeded.

The goal is not to make the contract hostile or overly legalistic. It is to turn assumptions into shared decisions before the agency has invested weeks of specialized labor.

The five layers every scope should separate

A strong pilot statement of work separates at least five layers.

1. Business objective

   • Example: reduce repetitive pre-purchase product questions for one category of products.
   • Not: automate customer support generally.

2. Knowledge scope

   • Identify the catalog, policy documents, FAQ content, and integrations that are included.
   • State which sources are authoritative and which are reference-only.

3. Conversation scope

   • Define supported intents, such as product dimensions, basic care instructions, compatibility, delivery-policy questions, and store-policy questions.
   • Explicitly list unsupported intents, including refunds, order changes, medical or legal guidance, bespoke product recommendations, or account-specific requests if the required tools are not included.

4. Delivery scope

   • Detail data-audit hours or record limits, configuration work, evaluation rounds, launch support, and reporting.
   • Separate implementation of new integrations, image understanding, or agentic workflows from the base pilot unless they are deliberately included.

5. Acceptance scope

   • Set test cases, quality thresholds, escalation rules, review responsibilities, and the decision process at the end of the pilot.

When these layers are collapsed into one vague promise, every discovered limitation feels like a failure. When they are separated, the team can correctly classify a problem: model behavior, retrieval quality, missing information, unsupported intent, integration defect, or a new request.

A pilot is an experiment with production safeguards

“Pilot” should not mean “unfinished production system sold at a discount.” It should mean a time-bounded validation of a specific hypothesis under defined conditions.

For example, a pilot hypothesis might be: Given a verified catalog of 500 SKUs and approved return-policy content, the assistant can correctly resolve or safely escalate at least 85% of the 75 agreed pre-purchase test scenarios for Category A.

That is measurable. It allows a client to assess value and gives the delivery partner a fair standard. It also prevents an eight-conversation test from becoming the sole basis for judging a month of discovery, remediation, configuration, and evaluation work.

NIST’s Generative AI Profile frames AI risk management as something organizations should incorporate into the design, development, use, and evaluation of AI systems. For small agencies, that does not require a giant compliance program. It supports the practical discipline of mapping risks, measuring system behavior, and managing the gaps before expanding deployment. (nist.gov)

Sell the data audit before the AI build—or cap it inside the pilot

The biggest commercial question raised by the original post is whether a data audit should be sold separately or included in the pilot. There is no universal answer, but there is a clear principle: unbounded data remediation should never be implied inside a fixed-price AI pilot.

The right packaging depends on what the agency can observe before signature and how much uncertainty the client is willing to absorb.

Option 1: a paid discovery and data-readiness phase

This is usually the cleanest option when the data estate is unknown, fragmented, or likely to be poor.

A standalone discovery engagement can include:

• Inventory of candidate sources and their owners.
• Sample-based checks for completeness, duplication, stale content, and field consistency.
• Definition of canonical IDs and required attributes.
• Identification of unsupported customer questions due to missing data.
• A source-of-truth matrix and remediation backlog.
• A technical recommendation for retrieval, live data access, image handling, and handoff.
• A fixed estimate for the implementation pilot that follows.

The advantage is commercial clarity. The client buys a useful deliverable even if the AI build never proceeds. The agency avoids estimating implementation from a sales call and two sample files.

The potential downside is sales friction. Some buyers want a single number and are reluctant to fund “discovery.” That objection is real, but it can be handled by framing discovery as risk retirement. A business would not expect an agency to quote a database migration, ERP integration, or custom analytics implementation without inspecting the data. An AI assistant that makes customer-facing claims deserves the same seriousness.

Option 2: a capped audit embedded in the pilot

A capped audit is appropriate when the sales cycle demands a fast start, the dataset is modest, or the agency already has evidence that the inputs are reasonably mature.

The scope should state a hard cap in measurable terms, such as:

• Up to 10 hours of source normalization.
• Up to 500 product records assessed.
• Up to three supplied data sources.
• One consolidated remediation report.
• No manual creation of new product attributes without written change approval.

Once the cap is reached, the agreement should offer choices: client remediates the identified issues, purchases an additional data-work block, narrows the pilot to clean categories, or pauses the project. The point is not to nickel-and-dime a client. It is to ensure that a major data-quality problem is treated as a real project decision rather than quietly absorbed by the agency.

A hybrid approach is often best

For many small agencies, the practical answer is a low-cost readiness sprint followed by a production-minded pilot. The readiness sprint can be credited toward the pilot if the client proceeds, which reduces friction without making the agency’s diagnosis free.

A simple structure could be:

1. Readiness assessment: fixed fee, one to two weeks, sample-based audit and implementation plan.
2. Pilot build: fixed price based on the verified scope, with a stated data-cleanup allowance.
3. Expansion or launch: separate phase after pilot acceptance, covering scale, integrations, monitoring, and operational ownership.

This structure also gives the client an off-ramp. If the audit reveals that their product content needs a major overhaul, they can solve that business problem first instead of paying for an assistant that will inevitably be constrained by weak inputs.

Separate defects from missing client data

One of the most useful clauses in an AI statement of work is a plain-language taxonomy of issue types. Without it, every unsatisfactory interaction becomes “the AI is wrong,” even when the source material was missing, internally inconsistent, or outside scope.

That does not mean an agency should hide behind the taxonomy. The agency remains responsible for designing appropriate fallback behavior. If information is absent, the safe experience is usually not a confident guess. It is a short explanation, a clarifying question, or a handoff.

A practical issue taxonomy

Use categories that a nontechnical stakeholder can understand:

• Implementation defect: the system did not behave according to the agreed specification despite the needed authoritative data being available.
• Knowledge defect: the approved source contains incorrect, contradictory, incomplete, or stale information.
• Coverage gap: the user asked for information or an action that is not in the agreed pilot scope.
• Experience improvement: the answer is factually acceptable but could be shorter, clearer, better formatted, or more on-brand.
• Enhancement request: the client wants a new capability, integration, channel, language, product category, or workflow not included in the signed scope.

The original r/SaaS post described several different categories at once: overly long answers, photo questions that needed another setup pass, questions that could not be answered because the source did not contain the answer, and scenarios requiring clearer escalation to humans. (reddit.com)

Those should not all be treated identically. Excessive answer length may be a configuration or experience issue. Weak photo question handling could be a multimodal capability and data-design issue. An unanswerable question may be a knowledge gap. Missing human handoff rules are a workflow-design issue. Each deserves remediation, but each has a different owner, cost, and timeline.

Use written test cases, not vibes, to decide whether the pilot worked

A pilot cannot be fairly evaluated by asking a stakeholder to try a few questions and report whether it “feels ready.” Real chats are valuable, but they are noisy. They may overrepresent unusual questions, use ambiguous language, or reveal new use cases that were not part of the original brief.

That is why a test suite should be agreed before launch. Anthropic describes evaluations as tasks that measure specific capabilities or behaviors; in a customer-support context, an eval suite can cover categories such as refunds, cancellations, and escalations. (anthropic.com)

What a good ecommerce AI test set includes

Build a set of 50 to 150 cases for a narrow pilot, depending on the catalog and intended complexity. Each case should include the customer prompt, known context, expected outcome, and pass criteria.

Include a balanced mix of:

• Direct factual queries: “What are the dimensions of Model X?”
• Ambiguous queries: “Will this fit my setup?”
• Contradictory-source checks: questions about facts known to be inconsistent before cleanup.
• Negative cases: questions for which the assistant must not invent an answer.
• Policy questions: shipping, return, warranty, and eligibility queries.
• Escalation cases: account-specific issues, complaints, safety-sensitive queries, and cases needing human judgment.
• Tone and brevity checks: whether the answer is clear enough to act on without becoming an essay.
• Adversarial or edge cases: misspellings, incomplete SKU names, multiple products in one message, and requests that conflict with policy.

For each case, define the expected response type rather than forcing a single exact wording. For example: “Correctly identify that compatibility cannot be verified from approved data; ask for device model or offer handoff.” This is more durable than matching one sentence word for word.

Measure more than answer accuracy

Customer-facing assistants should be measured on several dimensions:

A strong acceptance criterion combines these rather than relying on a single percentage. A system that answers 90% of questions but confidently gives wrong warranty guidance is not ready. Conversely, a system that safely escalates a small share of complex questions may be delivering real value even if it is not fully autonomous.

OpenAI’s developer materials similarly emphasize evaluating RAG systems rather than assuming that retrieval makes them reliable. Its examples cover measuring the quality of a retrieval-backed application and assessing how image understanding changes customer-service workflows that include photos. (developers.openai.com)

Human handoff is a feature, not an embarrassing fallback

The original project found that some scenarios needed clearer human handoff. That should not be treated as proof that the assistant failed. In a well-designed support workflow, handoff is part of the product.

The mistake is waiting until live users expose edge cases before defining what handoff means. Does the assistant merely say “contact support”? Does it collect the necessary details first? Does it create a ticket, route to a queue, send a transcript, or alert a team member? What happens outside business hours? These are product and operations decisions, not last-minute prompt edits.

Define escalation before the first live chat

The scope should include a handoff policy that names trigger categories and the intended behavior. Common triggers include:

• The required information is unavailable or contradictory in approved sources.
• The user wants an order-specific action, refund, cancellation, replacement, or account change.
• The question relates to safety, legal commitments, regulated advice, or a high-value commercial exception.
• The assistant has low confidence after a defined number of clarification attempts.
• The user explicitly asks for a person or expresses dissatisfaction.
• An image cannot be reliably connected to a product or condition assessment.

The assistant should then do something useful: state the limitation in simple language, collect relevant order or product details where permitted, preserve the conversation context, and route it correctly.

For agencies, this reframes a sales conversation. You are not selling “zero human involvement.” You are selling a controlled system that handles routine questions efficiently and recognizes when a person is better positioned to act. That is a more credible promise, and it protects both brand trust and customer experience.

Why photo-based questions deserve their own scope line

Photo questions are easy to underestimate. A shopper may upload an image and ask, “Is this the part I need?” “Which color is this?” “Will this damage wash out?” or “Does this look like the same model?” These prompts can require image understanding, product matching, access to image assets, variant-level metadata, and sometimes human judgment.

The Reddit account specifically notes that photo-based questions needed an additional setup pass. (reddit.com) That is unsurprising: image inputs are not simply another text field. The image may be low resolution, poorly lit, incomplete, or depict a product from an older generation. A catalog may have lifestyle photography without the metadata required to map the image to an exact SKU.

A pilot should state which image tasks are supported. For instance, the assistant might be allowed to explain information visible in a product image supplied by the company, but not authenticate a customer-owned item, determine damage, or guarantee compatibility from a photo alone.

If image-based matching is a core commercial requirement, treat it as a specific workstream. It needs dedicated test cases, confidence thresholds, human-review logic, and data checks for image-to-product mappings. Do not hide it inside a generic “multimodal assistant” bullet.

Feedback rounds need ownership, timing, and limits

Client collaboration is not a burden that makes an AI service useless. It is how a pilot becomes aligned with the business’s actual policies, terminology, exceptions, and customer expectations.

However, collaboration must be organized. Random feedback from several stakeholders in chat threads, emailed screenshots, and ad hoc calls can create endless rework. One person requests shorter responses, another asks for more explanation, and a third introduces a new product category without realizing it changes the knowledge scope.

The consolidated-feedback model

Assign one named client-side feedback owner. That person can collect input internally, prioritize it, resolve conflicting stakeholder opinions, and submit consolidated feedback by a stated deadline.

Then set a fixed revision model, such as:

1. Agency delivers the test environment and evaluation report.
2. Client has five business days to test the agreed cases and add a limited number of representative real-world examples.
3. Client submits one consolidated issue log, using the defect taxonomy.
4. Agency completes correction round one.
5. Client re-tests only corrected scenarios plus agreed regression tests.
6. A second, final correction round addresses confirmed in-scope defects.
7. New features, new data sources, or major changes become a change request or phase-two item.

This protects the client as much as the agency. It creates an orderly path to resolution instead of leaving stakeholders uncertain whether their feedback was received, prioritized, or quietly deferred.

The commercial model: charge for uncertainty, not just output

The uncomfortable truth in AI services is that the client often sees a conversational interface while the agency absorbs the uncertainty beneath it. That uncertainty includes data quality, system access, stakeholder availability, business-rule ambiguity, implementation complexity, and evolving expectations.

A sustainable agency model makes uncertainty visible and priced. It does not punish the client for having imperfect operations. It gives them a clear choice about how to solve the problem.

Contract language to adapt

This is not legal advice, and agencies should have counsel adapt terms for their jurisdiction and service model. But operationally, language like the following can prevent misunderstandings:

The pilot is a limited validation engagement, not a production-wide launch. The assistant’s performance depends on the completeness, accuracy, accessibility, and timeliness of Client-Provided Materials.

Agency will perform up to [X] hours or [Y] records of data assessment and normalization. Material remediation, creation of missing customer-facing facts, reconciliation of conflicting records, and new data integrations are outside the included allowance unless approved in writing.

The assistant will answer only within the agreed intent and knowledge scope. Where approved sources do not support a reliable answer, the assistant may ask a clarifying question, state that the information is unavailable, or route the conversation for human support.

Acceptance will be assessed against the agreed test cases and severity criteria. Requests beyond the agreed knowledge sources, channels, features, or workflows are enhancements and will be separately estimated.

This language does not eliminate disputes. It does change the conversation from “why doesn’t it do everything?” to “which specific capability, source, or acceptance criterion should we change?”

What agencies should do differently at the sales stage

The best time to prevent a pilot dispute is before the proposal is sent. Sales teams and founders should resist the temptation to make an AI system sound effortless in order to win a deal.

Instead, qualify for readiness early. Ask pointed questions that reveal whether the client is buying an AI implementation, a data-cleanup engagement, or both.

Ten discovery questions worth asking

1. Which system is authoritative for each customer-facing fact the assistant will answer?
2. Who can resolve contradictions between the website, catalog, policies, and operational systems?
3. How are product IDs, variants, bundles, and discontinued items represented?
4. Which three questions do customers ask most often, and what is the approved answer source for each?
5. What information is missing today that support staff know from experience but cannot point to in a system?
6. Which requests must always reach a person?
7. Are stock, price, order, and shipping answers expected to be real-time? If so, which APIs or platform permissions are available?
8. Are customer-uploaded images in scope? What business decision should the assistant make from them?
9. Who is the single feedback owner, and how quickly can they validate business facts?
10. What would count as a successful pilot after 30 days: lower response time, safer coverage, fewer routine tickets, higher conversion, better support availability, or something else?

The answers help agencies avoid a familiar trap: designing an impressive demo around ideal data, then discovering that the commercial product must operate under much less ideal conditions.

The broader lesson: AI readiness is operational readiness

The original r/SaaS post is valuable precisely because it does not portray either party as entirely wrong. The client bought an outcome and reasonably expected a useful service. The agency performed real work and reasonably believed that a brief test period did not erase the value of data cleanup, system setup, and safer knowledge-base design. (reddit.com)

The gap was not simply technical. It was an expectation gap created by an incomplete definition of “pilot.”

AI exposes weaknesses that businesses have often tolerated for years: duplicate product records, undocumented exceptions, stale policy pages, scattered ownership, and support knowledge that lives only in people’s heads. Those weaknesses are not caused by the model. But customer-facing AI makes them harder to ignore because the assistant has to make a visible decision in real time.

That can be uncomfortable, but it is also valuable. A readiness audit can reveal where the customer experience was already inconsistent before any AI system was introduced. Agencies that position this work as a strategic deliverable—not invisible setup—can create better systems and healthier client relationships.

A practical AI pilot project scope checklist

Before accepting payment or beginning a build, confirm that the following are written down:

• The narrow business objective and user group.
• Included customer intents and explicitly excluded intents.
• Every authoritative data source and the owner for each source.
• Data-readiness assumptions, audit limits, and remediation pricing.
• Required integrations, access dependencies, and client deadlines.
• Rules for conflicts, missing information, uncertain answers, and unsupported questions.
• Human-handoff triggers, routing method, and responsibility after handoff.
• Supported channels, languages, image inputs, and product categories.
• An agreed test suite with pass criteria and severity definitions.
• A named client feedback owner and a consolidated review process.
• The number of included correction rounds and their deadlines.
• A written distinction between defects, knowledge gaps, and new requests.
• Pilot success metrics and the decision options at the end: launch, expand, remediate, pause, or stop.

The agency in the Reddit discussion arrived at many of these improvements after a difficult project: a written source of truth, visible and capped data cleanup, test cases, a named feedback owner, consolidated correction rounds, and defined human handoff. (reddit.com) Those changes are not bureaucratic overhead. They are the operating system for delivering AI responsibly at a profit.

FAQ

What is an AI pilot project scope?

An AI pilot project scope is a written definition of what an AI system will do, which data and workflows it may use, what it must not do, how it will be tested, and what each party must provide. It should distinguish a limited validation engagement from a full production launch.

Should an agency charge separately for AI data cleanup?

Usually, yes—especially when data quality is unknown or inconsistent. Agencies can sell a standalone data-readiness assessment or include a strictly capped audit allowance in the pilot. The key is that extensive remediation is visible, measurable, and separately approved rather than assumed to be free setup work.

How many revision rounds should an AI pilot include?

There is no universal number, but one or two consolidated correction rounds are common and manageable. The agreement should specify who submits feedback, when it is due, how issues are categorized, and when a request becomes a separately priced enhancement.

What should happen when an AI assistant does not know the answer?

It should not guess. Depending on the agreed workflow, it can ask a clarifying question, explain that approved information does not support a reliable answer, or hand the conversation to a human. Safe abstention and escalation are valuable product behaviors, not evidence that the system is broken.

Are customer photo questions included in a standard ecommerce AI pilot?

They should not be assumed to be. Photo-based questions may require image understanding, product-image mapping, special data preparation, confidence thresholds, and human review. If image tasks matter to the client, define them as a separate, testable scope item.