Usability Testing for SaaS: What Five Users Really Reveal

Usability testing for SaaS is often treated as a luxury for teams with research budgets. A recent Reddit experiment makes the stronger case: five brief phone recordings may not tell you whether people will pay, but they can reveal why interested visitors never reach the moment where payment becomes possible.

The experiment, posted anonymously in r/SaaS, involved five U.S.-based iPhone owners who were paid $10 to visit an unfamiliar AI job-search product, speak their thoughts aloud, and answer three basic questions: what the product was, who it was for, and whether they would take the next step. Four said they would use it. None said they would pay. Only two could clearly explain what the product actually did. (reddit.com)

That result is more useful than the usual binary verdict of “good idea” or “bad idea.” It separates three problems founders frequently collapse into one: desirability (do people want the outcome?), comprehension (do they understand the offer?), and conversion readiness (can they smoothly act on that interest?). The product may have had some demand. Its landing page still failed to turn that demand into a clear, credible path to action.

The Reddit test was small, but the signal was real

The original poster did not run a statistically representative study. Participants were recruited through Craigslist, received a small incentive, and saw a single site for just three to six minutes. That makes the sample far too limited to estimate market size, conversion rate, or willingness to pay.

But that is not what a qualitative test is for.

In usability research, a small round of task-based observation is designed to find recurring interface and comprehension problems, then feed an iteration cycle. Nielsen Norman Group continues to recommend small qualitative studies—often around five participants—for uncovering major usability issues, while emphasizing that larger samples are required when teams need statistically reliable metrics or comparisons. (nngroup.com)

That distinction matters because founders routinely misuse both directions of evidence:

• They treat five positive interviews as proof that a product will sell.
• Or they dismiss five sessions entirely because the sample is too small to be “scientific.”

Both reactions miss the point. If two independent people, on different phones, say that content is overlapping and controls appear broken, that is not a market-sizing finding. It is a high-priority defect report. The right response is not to debate sample size; it is to reproduce the issue, fix it, and retest.

What the five sessions actually uncovered

The Reddit post identified three practical failures. Each could damage conversion independently. Together, they explain why a visitor could want an AI product yet leave without understanding it, trusting it, or reaching pricing.

1. Information density created disorientation

Two participants described the page as visually overwhelming and said they felt lost. Importantly, they still expressed interest in the product’s broad promise. That means “too much text” should not automatically be read as a preference for minimalist copy.

It may instead signal that the page has failed at information hierarchy. Visitors do not know what to read first, what claim matters most, or what action they are expected to take. On a landing page, dense copy becomes especially harmful when the visitor is arriving with a specific problem and limited patience.

For an AI job-search product, the first screen should make the chain of value unmistakable:

1. Who it serves: job seekers not getting interviews or callbacks.
2. What it does: for example, analyzes application materials, tailors resumes, or identifies gaps.
3. What changes: stronger applications, faster iteration, or more targeted outreach.
4. What to do next: try it with a resume, view an example, or start free.

If users need to scroll, infer, and translate several abstract claims before they can explain the product, the page is doing too much work in the wrong order.

2. A mobile layout bug impersonated a positioning problem

Two participants reportedly saw elements collide or get cut off on their phones. That is a crucial finding because poor responsive behavior can masquerade as vague messaging. A page with layered text, clipped buttons, and broken spacing feels “busy” even if its copy is perfectly reasonable.

Responsive design is not simply shrinking a desktop page onto a smaller viewport. Google’s web.dev guidance describes responsive design as adapting layout to both user needs and device capabilities; its interface-pattern guidance specifically warns against treating a small-screen experience as merely a reduced desktop layout. (web.dev)

The practical implication is severe: founders can spend days rewriting a headline when the immediate problem is a CSS rule, a fixed-width component, a translated element, a stacking context, or an untested browser-specific rendering edge case.

Before changing positioning, inspect the experience on real devices. Test at least a small matrix of current iPhones and Android devices, plus a throttled network connection. Simulators are useful, but web.dev notes that browser developer tools can emulate device modes and network conditions; they should complement, not replace, testing on real hardware. (web.dev)

3. Slow interaction looked like a broken product

One participant repeatedly clicked the pricing link and spent much of the session thinking the site was down. The page may eventually have responded, but the user’s interpretation was still rational: a tap that produces no visible acknowledgement invites another tap, then confusion, then abandonment.

This is a reminder that perceived performance is a conversion issue, not only an engineering metric. A delayed route transition, a client-side hydration bottleneck, an overloaded analytics script, or a slow third-party pricing widget can all make a functioning page feel unavailable.

Google’s Core Web Vitals framework includes Interaction to Next Paint (INP), a responsiveness metric focused on the delay between a user interaction and the next visual update. While a good INP score does not guarantee a good conversion flow, it captures an important part of the experience this participant described: the page should visibly respond when someone tries to act. (web.dev)

Why analytics could not explain the failure

The most compelling observation in the Reddit post was that conventional analytics could show an exit but not the reason behind it. A bounce, short session, or abandoned funnel does not inherently distinguish among these possibilities:

• The visitor did not want the product.
• The visitor wanted it but could not understand it.
• The visitor understood it but could not find the right CTA.
• The visitor tried to continue but encountered a broken mobile layout.
• The visitor tapped a link and assumed the site had failed.
• The visitor reached pricing and decided the value was not worth the cost.

Those behaviors can all produce nearly identical top-line metrics.

Google Analytics 4 is event-based: it can record page views, clicks, purchases, crashes, and custom interactions, and event parameters can add useful context about which button or product a user touched. (support.google.com) That is valuable instrumentation, but an event log records that a click happened; it does not reveal the visitor’s mental model, hesitation, or interpretation of a delayed response.

The answer is not to abandon analytics. It is to combine three forms of evidence:

A healthy research loop moves between them. Analytics identifies the suspicious step. Recordings or monitoring reveal the mechanical behavior. A short usability test explains the human interpretation. Then an experiment measures whether the fix improved the outcome.

The most important distinction: “I would use it” is not “I would pay”

The post’s numbers—four out of five willing to use the product, zero out of five willing to pay—should not lead to a simplistic conclusion that the product has no business model. But they should stop a founder from treating stated interest as revenue validation.

Usage intent is cheap. Payment requires a different stack of beliefs:

• The problem is painful enough to deserve a budget.
• Existing tools or manual work are inadequate.
• The product’s outcome is credible.
• The buyer trusts the company with relevant data.
• The pricing model feels proportionate to the value.
• The visitor knows what happens after they sign up.

An AI job-search tool also faces a particularly difficult willingness-to-pay environment. Job seekers may be under financial stress, may see free alternatives everywhere, and may be unsure whether an AI-generated resume or application will materially change their results. Even a great product must make the benefit concrete without making claims it cannot substantiate.

The community reaction to the post raised a valid concern: a $10 incentive can make participants more agreeable, while saying “yes, I would use this” costs them nothing. One commenter contrasted interview feedback with an actual behavior signal: a free trial without a credit card that reveals whether someone returns after a few days. That critique is not a reason to discard the recordings. It is a reason to assign the recordings the right job.

Use qualitative sessions to diagnose friction. Use behavior to validate demand and pricing.

A better SaaS validation ladder

Founders need different tests for different risks. Trying to answer every question with one five-person study creates bad decisions.

Stage 1: Test comprehension

Show a cold visitor the page for 10 to 20 seconds, then ask them to explain the product in their own words. Do not ask, “Do you understand?” That question tends to produce polite agreement.

Instead ask:

• “What do you think this company helps people do?”
• “Who is this made for?”
• “What result would you expect after using it?”
• “What would you click next, and why?”

If their answer differs from your intended positioning, fix the page before buying more traffic.

Stage 2: Test task completion

Give users a realistic scenario rather than asking them to browse freely. For example: “You have sent 30 applications, are receiving few responses, and want to improve your application materials before applying to another role.”

Then observe whether they can find pricing, identify the free plan’s limits, start a trial, and understand the first onboarding step. Usability testing is most informative when participants attempt realistic tasks while the researcher observes behavior and listens to their commentary. (nngroup.com)

Stage 3: Test activation behavior

Once the page and workflow are understandable, measure what people do after signup. Define activation around an action that plausibly leads to retained value—not just account creation.

For an AI career product, activation might mean:

• Uploading or pasting a resume.
• Completing the first job-targeting workflow.
• Generating and saving a tailored application asset.
• Returning to use the tool for a second job listing.

A no-card trial can be useful here because it reduces payment friction while preserving the signal of effort. Someone who comes back and completes meaningful work has given stronger evidence than someone who says they like the concept.

Stage 4: Test willingness to pay with a real trade-off

Do not ask, “Would you pay $19 per month?” Ask users to choose. Put a meaningful paywall or upgrade moment after they have received enough value to understand the product, then measure upgrade behavior.

You can test packaging before changing everything at once: monthly versus annual framing, credit-based usage versus subscription, trial length, feature gating, and whether a free tier encourages genuine adoption or simply absorbs demand that would otherwise convert.

A pricing test should still be ethical and clear. Do not invent scarcity, hide renewal terms, or use ambiguous credits to manufacture upgrades. The goal is to learn whether value and price meet—not to trap a confused customer.

How to run a useful $50 usability test

The Reddit post is a good model for scrappy testing, with several upgrades that make the findings more actionable. You do not need a formal lab, but you do need a repeatable protocol.

Recruit for the decision, not just device ownership

The original test screened only for U.S. location and iPhone ownership. That is enough to catch mobile breakage, but weak for validating a job-search product’s message.

For a B2C job-search tool, recruit people who have applied for roles in the past 60 to 90 days. For B2B SaaS, recruit users who influence the workflow, not merely people who resemble the audience demographically. A recruiter, hiring manager, or career coach may be a valuable secondary participant, but should not replace the job seeker if the job seeker is the buyer.

Segment sessions by user type if you serve materially different audiences. An entry-level candidate, an executive candidate, and a career coach may each interpret “AI application help” differently.

Use neutral instructions

Avoid leading questions such as “Can you find the pricing?” or “Do you like this AI assistant?” Start with a scenario, ask the participant to think aloud, and intervene as little as possible.

A lightweight script can be:

1. “Please open this page and narrate what you notice.”
2. “Imagine you are trying to solve this problem today. What would you do?”
3. “What do you think this product does?”
4. “Who do you think it is designed for?”
5. “What would make you take the next step—or stop?”
6. “If you saw pricing, what would you expect to get for that amount?”

Do not explain the product during the task. Every clarification you provide removes evidence about what the page failed to communicate.

Capture the right artifacts

Ask for screen recording and voice, then write a short observation sheet for every participant. Record exact timestamps for confusion, backtracking, repeat taps, dead links, unrecognized terminology, and unexpected interpretations.

Tag each observation under one of four categories:

• Comprehension: they cannot explain the offer or intended outcome.
• Navigation: they cannot locate a feature, CTA, price, or next step.
• Technical performance: the interface delays, overlaps, errors, or appears unresponsive.
• Value and pricing: they understand the offer but reject the trade-off.

This prevents a technical issue from being mislabeled as a marketing issue—and prevents a pricing objection from being “fixed” with prettier buttons.

Fix the homepage in the correct order

When a test exposes several problems, teams often start with the most intellectually interesting one: rewriting messaging. That may be premature.

The correct sequence is usually:

1. Eliminate reproducible breakage. Fix clipped layouts, overlapping text, broken controls, route failures, and severe response delays first.
2. Make the primary promise legible. Use a direct headline, an outcome-oriented subhead, and one primary CTA.
3. Show the mechanism. Explain how the AI product gets from input to outcome with a short visual, sample output, or three-step flow.
4. Establish credibility. Add honest evidence: examples, constraints, privacy handling, customer outcomes, or transparent product limitations.
5. Clarify the commercial offer. State free-plan limits, paid-plan benefits, pricing cadence, and what the visitor receives after taking action.
6. Retest the changed page. Do not assume the fix worked because it feels clearer to the team that built it.

This sequence matters because a great headline cannot compensate for a button that appears untappable. Nor can fast loading compensate for a visitor who cannot tell whether the product helps them write a resume, find jobs, prepare for interviews, or automate applications.

Mobile QA should become a conversion discipline

The mobile overlap report is a useful warning for every founder building with modern component libraries, AI-assisted code, visual builders, and rapid deployment workflows. Fast shipping can make it easy to create a desktop-perfect page with fragile breakpoint behavior.

Responsive failures often emerge from patterns that seem harmless in isolation:

• Absolute-positioned decorative elements that cover copy at narrow widths.
• Fixed-height hero sections that cannot accommodate text wrapping.
• Long words, badges, or pricing labels that overflow flex containers.
• Animated sections that shift after fonts or images load.
• Modals, cookie banners, or chat widgets blocking primary CTAs.
• JavaScript-driven navigation that has no immediate pressed, loading, or pending state.

Build a pre-launch mobile checklist around actual conversion tasks, not visual inspection alone. Can someone read the headline at 320 to 390 pixels wide? Can they tap every primary CTA with one thumb? Does the page preserve its hierarchy when browser text size increases? Does the pricing route respond quickly enough to communicate that the tap registered? Is there a visible loading state when it cannot?

For layout movement, browser tooling can help identify layout shifts; web.dev notes that DevTools-based tools draw on the Layout Instability API for debugging unexpected shifts in Chromium browsers. (web.dev) But field observation remains essential because users react to the total experience, not to a single diagnostic score.

Turn qualitative findings into better instrumentation

The Reddit author was right that standard analytics cannot hear a frustrated user say, “I think this is broken.” Still, qualitative research should improve the analytics setup afterward.

Once a session reveals likely failure modes, add instrumentation that lets you monitor their scale:

• Record pricing_cta_click with CTA location and viewport category.
• Record pricing_page_view and compare it against CTA clicks.
• Record client-side route errors and failed API requests.
• Track repeated clicks on the same CTA within a short interval as a possible unresponsive-interaction signal.
• Measure time from CTA interaction to destination render.
• Segment landing-page behavior by device class, browser, screen width, and connection quality where privacy and your analytics policies permit.
• Track trial start, first-value event, return visit, upgrade prompt view, and purchase separately.

Google Analytics supports custom events and parameters for granular context, but every event should map to a decision. (support.google.com) If your team cannot explain what action a metric would trigger, the event is probably dashboard decoration.

For example, a large gap between pricing CTA clicks and pricing-page views on mobile should trigger a technical investigation. A healthy pricing-page view rate followed by low trial starts points more toward messaging, offer clarity, or trust. Strong trial starts but weak activation suggests the landing page may be promising an outcome that onboarding does not deliver quickly enough.

Critical users are often your most valuable users

One of the post’s best insights was counterintuitive: the most critical reviewers were also among the people most interested in the product. That tracks with how motivated users behave. They keep trying because they want the outcome, and their frustration becomes visible precisely because the product has earned some attention.

Founders should therefore avoid categorizing feedback as either “positive” or “negative.” Better labels are:

• Is the observation specific?
• Did it occur during a meaningful task?
• Can it be reproduced?
• Does it block comprehension, trust, activation, or payment?
• Does it appear across more than one participant or data source?

A user saying “this sucks” is hard to act on. A user who taps a pricing link four times, waits, then says they think the site is down has handed you a prioritized engineering and conversion task.

The same principle applies to prospects who ask difficult sales questions. Objections about data privacy, workflow fit, implementation time, or pricing are often signs of real consideration. Silence is not automatically a better signal.

What founders should do this week

If your SaaS landing page receives traffic but produces unclear conversion results, run a focused round of usability testing for SaaS before rewriting your whole strategy.

Start with five sessions, but treat them as an early diagnostic pass, not a referendum on your business. Recruit people close to your actual audience, give each a realistic scenario, watch them on the devices that matter, and ask them to narrate without coaching.

Then make one prioritized change at a time. Repair obvious mobile and performance defects. Simplify the first-screen message. Clarify what the free experience includes and what paid access unlocks. Instrument the new flow. Finally, measure real behavior: activation, return use, and payment at a genuine value threshold.

The lesson from the Reddit experiment is not that $50 of research can replace product strategy. It is that a small amount of direct observation can stop you from solving the wrong problem. When interested people leave, do not immediately conclude that demand is weak. First find out whether they were unconvinced—or simply unable to proceed.

FAQ

Is five users enough for usability testing for SaaS?

Five users can be enough for an early qualitative round intended to uncover major, recurring usability issues. It is not enough to estimate conversion rates, market demand, or statistically compare designs; those questions require larger samples and quantitative methods. (nngroup.com)

Can paid test participants provide reliable feedback?

Yes, if you use them to observe concrete behavior rather than to validate compliments or stated purchase intent. Compensation can encourage participation and completion, but it does not make “I would pay” a reliable proxy for an actual purchase decision.

What should I ask in a SaaS think-aloud test?

Ask participants to describe what they think the product does, who it is for, what they would do next, and what made them hesitate. Give a realistic task and avoid explaining the product or asking leading questions.

Why do analytics and usability tests disagree?

They usually answer different questions. Analytics captures events and outcomes, while usability testing reveals the interpretation behind those behaviors—such as whether a user abandoned because of low interest, confusing copy, a hidden CTA, or a perceived technical failure.

Should I fix messaging or mobile bugs first?

Fix reproducible mobile breakage and responsiveness issues first. A cluttered or unresponsive interface can create the appearance of weak positioning, making copy changes difficult to evaluate fairly.