A/B Testing in Mobile Apps: How I Experiment With App Store Pages, Paywalls, Onboarding, and Pricing

I’ve come to think of A/B testing as one of the most useful tools in mobile app development.

Not because it gives me a magical answer to every product question, but because it forces me to replace opinions with evidence.

When I’m deciding whether a paywall should appear earlier, whether an annual plan should be emphasized, whether onboarding should be shorter, or whether a different App Store screenshot communicates the product better, I don’t want the loudest opinion in the room to win.

I want to test it.

In my mobile apps, I typically experiment with App Store custom product pages, paywalls, onboarding flows, pricing, offers, feature exposure, copy, and user journeys.

This article explains how I approach A/B testing, starting from the fundamentals and moving into the tools that make experimentation much easier today, including RevenueCat, Adapty, and Superwall.

What is A/B testing?

At its simplest, A/B testing means showing different versions of an experience to different groups of users and measuring which version performs better.

Imagine I have a subscription paywall.

The current version looks like this:

• Headline: “Unlock Premium”
• Monthly plan: €9.99
• Annual plan: €59.99
• Free trial: 7 days

I create a second version:

• Headline: “Get the Most Out of the App”
• Monthly plan: €9.99
• Annual plan: €49.99
• Free trial: 3 days

Instead of choosing one based on intuition, I randomly assign users to the two versions.

Some users see A, the control.

Others see B, the treatment.

Then I measure what happens.

                    Users
                      |
              +-------+-------+
              |               |
              v               v
         Variant A       Variant B
         Control         Treatment
              |               |
              v               v
          Conversion       Conversion
              |               |
              +-------+-------+
                      |
                      v
                 Comparison

The important part is that I’m not simply asking:

“Which version converts better?”

I’m asking:

“Did changing this specific part of the experience cause a meaningful difference in user behavior?”

That distinction becomes extremely important as an app grows.

Why A/B testing matters so much in mobile apps

Mobile apps have a particularly interesting funnel because there are many places where users can drop off.

For a subscription app, the journey might look something like:

Ad / Search / Social
        ↓
App Store Product Page
        ↓
Download
        ↓
First Launch
        ↓
Onboarding
        ↓
First Value
        ↓
Paywall
        ↓
Offer Selection
        ↓
Purchase
        ↓
Activation
        ↓
Retention

There isn’t one conversion rate.

There are many.

A change at the beginning of the funnel can affect everything downstream.

For example, I might increase App Store conversion by creating a more aggressive product page.

That sounds great.

But if those new users don’t understand the product and churn immediately, I haven’t necessarily created a better business.

That’s why I try to think about experimentation as funnel optimization, rather than simply conversion-rate optimization.

What can I A/B test in a mobile app?

Almost anything that influences user behavior can potentially become an experiment.

In practice, I usually divide experiments into a few categories.

1. App Store product pages

The App Store is the first conversion point for many apps.

I can experiment with:

• App icons
• Screenshots
• Screenshot ordering
• App previews
• Promotional messaging
• Feature positioning
• Localization
• Audience-specific messaging

Apple provides two particularly useful concepts here: Product Page Optimization and Custom Product Pages.

Product Page Optimization lets me test different versions of the default App Store product page. Apple currently allows up to three treatments against the original page, and the treatments can include different icons, screenshots, and app previews.

Custom Product Pages are slightly different.

Instead of trying to find one universal page that works for everyone, I can create pages for specific audiences or campaigns.

For example:

Default Product Page
        |
        +--- Fitness audience
        |
        +--- Meditation audience
        |
        +--- Productivity audience
        |
        +--- Students

Apple currently allows up to 70 custom product pages per app. Each can have different screenshots, previews, promotional text, and keywords, and each has its own URL.

I find this particularly useful when the same product has several different value propositions.

A fitness app might be both a:

• calorie tracker
• workout planner
• AI coach
• weight-loss companion

Trying to communicate all of that in the first three screenshots can create a very confusing product page.

A custom product page allows me to say:

“This page is specifically for people looking for an AI workout coach.”

That’s much more focused.

And Apple provides analytics for these pages, including product page views, downloads, conversion rate, and downstream sales or subscription metrics.

2. Onboarding

Onboarding is another area where I frequently experiment.

A common mistake is assuming that onboarding should explain everything.

I don’t necessarily agree.

The goal of onboarding isn’t to teach the entire product.

The goal is to get the user to the first meaningful value as quickly as possible.

For example, I might test:

Version A

Welcome
↓
Choose your goal
↓
Choose your experience level
↓
Choose your interests
↓
Enable notifications
↓
Create account
↓
Paywall

against:

Version B

Welcome
↓
Choose your goal
↓
Create account
↓
Paywall

The second version might look almost too simple.

But that’s exactly why I want data.

Perhaps users don’t need five configuration screens before seeing the product.

Perhaps personalization is valuable.

Perhaps it’s actually hurting activation.

A/B testing helps me discover which assumption is closer to reality.

Tools such as Adapty explicitly support A/B testing for onboarding flows in addition to paywalls and pricing.

3. Paywalls

Paywalls are probably the most obvious place for subscription apps to experiment.

And there are many variables to test.

I can test:

• Headline
• Subtitle
• CTA
• Layout
• Product ordering
• Monthly vs. annual emphasis
• Trial length
• Trial vs. no trial
• Pricing
• Discount messaging
• Social proof
• Feature lists
• Images
• Videos
• Testimonials
• Guarantee messaging
• Number of products
• Placement of the paywall

For example, I might test:

A: Annual → Monthly
B: Monthly → Annual

Or:

A: "Start Free Trial"
B: "Continue"

Or:

A: 7-day free trial
B: 3-day free trial

Or even:

A: Paywall immediately after onboarding
B: Paywall after the user completes their first action

The last one is particularly interesting because I’m no longer testing just the paywall.

I’m testing when the user sees the paywall.

That distinction matters.

The metric I optimize is not always conversion

This is probably one of the most important lessons I’ve learned from experimentation.

A/B testing becomes dangerous when I optimize the wrong metric.

Suppose Paywall B produces:

Paywall A
Conversion: 5%

Paywall B
Conversion: 7%

At first glance, B is obviously the winner.

But imagine the revenue looks like this:

Paywall A
Conversion: 5%
Average revenue per purchaser: €40

Paywall B
Conversion: 7%
Average revenue per purchaser: €20

Now the decision isn’t so obvious.

And it becomes even more complicated when I consider:

• Refunds
• Trial-to-paid conversion
• Subscription renewal
• Churn
• Lifetime value
• Retention
• Revenue per install

A paywall can produce more purchases but less long-term revenue.

That’s why I prefer thinking in terms of the business metric behind the experiment, not just the easiest metric to measure.

For subscription apps, I might look at:

• Purchase conversion
• Trial start rate
• Trial-to-paid conversion
• Revenue per install
• Revenue per paying user
• Renewal rate
• Churn
• Subscriber retention
• LTV

RevenueCat’s experimentation system is designed around this broader subscription lifecycle. Its experiments compare Offerings and analyze the resulting subscription lifecycle rather than stopping at the initial conversion.

The basic anatomy of a good A/B test

Before creating an experiment, I try to answer five questions.

1. What am I changing?

I want the variable to be clear.

For example:

Change annual-plan positioning on the paywall.

Not:

Make the paywall better.

The second statement isn’t really a hypothesis.

It’s a wish.

2. Why do I think it will work?

This is the hypothesis.

For example:

I believe emphasizing the annual plan will increase subscription revenue because users will perceive the annual option as better value.

Now I have something I can actually test.

3. What is my control?

The control is normally the current experience.

Control
Existing paywall

The treatment is the proposed change.

Treatment
New paywall with annual plan emphasized

Without a good control, it’s difficult to understand what the treatment actually changed.

4. What metric determines success?

Before launching the experiment, I should already know what success means.

For example:

Primary metric: revenue per install.

Secondary metrics:

• Paywall conversion
• Trial starts
• Trial-to-paid conversion
• Refund rate

This prevents me from looking at ten different metrics after the experiment and choosing whichever one happens to look good.

5. What would make me ship the change?

I like defining the decision before seeing the result.

For example:

If Variant B produces a statistically reliable improvement in revenue per install without materially increasing churn, I’ll ship it.

That sounds simple, but it prevents a lot of post-hoc rationalization.

Statistical significance isn’t the whole story

One of the easiest mistakes to make with A/B testing is declaring a winner too early.

Imagine I launch an experiment and after one day I see:

A: 4.1%
B: 5.2%

It is tempting to say:

B wins!

But I don’t know that yet.

The sample might simply be too small.

Random variation can create large differences early in an experiment.

I therefore want to distinguish between:

Observed difference

and

Evidence that the difference is real.

Those aren’t the same thing.

Apple’s own Product Page Optimization reporting includes estimated conversion rate, conversion lift, and confidence, and uses statistical analysis to determine whether treatments are performing better or worse than the baseline. Apple currently describes 90% confidence as the threshold at which a treatment may be labeled “Performing Better” or “Performing Worse.”

I don’t treat “90%” as a universal magic number for every experiment, though.

The right statistical approach depends on the experiment, sample size, metric, and decision I’m making.

The broader lesson is simpler:

Don’t stop an experiment just because one variant is temporarily ahead.

Sample size matters

A/B testing with 100 users is fundamentally different from A/B testing with 1 million users.

If I have:

A: 5 conversions / 100 users
B: 7 conversions / 100 users

I shouldn’t confidently conclude that B is better.

The difference might disappear with more users.

This is one reason I prefer estimating the required sample size before launching an important experiment.

I need to understand:

• Current conversion rate
• Minimum improvement worth detecting
• Expected traffic
• Number of variants
• Desired statistical confidence
• Expected experiment duration

There is also an important trade-off here.

If I create ten variants instead of two, I spread traffic across more groups.

Apple explicitly notes that adding more treatments can make product-page tests take longer to reach a conclusive result.

So I usually prefer:

Fewer, stronger hypotheses

over:

Many tiny variations

One variable at a time?

This advice is useful, but I don’t treat it as an absolute rule.

If I test:

A:
Blue CTA
Annual first
"Start Free Trial"

B:
Green CTA
Monthly first
"Get Premium"

and B wins, I don’t know which change caused the improvement.

Was it:

• the color?
• the product order?
• the copy?
• the combination?

That’s why, when I’m trying to learn something specific, I prefer to isolate the variable.

For example:

A:
Annual first

B:
Monthly first

Everything else stays the same.

Now the experiment answers a much more useful question.

There are situations where testing complete experiences is perfectly reasonable. But I need to understand that I’m testing the combined experience, not a single isolated variable.

A/B testing is really about hypotheses

This is how I think about experimentation:

Observation
    ↓
Problem
    ↓
Hypothesis
    ↓
Experiment
    ↓
Data
    ↓
Decision
    ↓
New learning
    ↓
Next hypothesis

For example:

Observation: Users abandon onboarding before reaching the paywall.

↓

Hypothesis: Onboarding is too long.

↓

Experiment: Reduce onboarding from six screens to three.

↓

Measurement: Activation rate + subscription revenue per install.

↓

Result: Activation improves but revenue decreases.

↓

Learning: The removed screens might have been important for purchase intent.

↓

Next hypothesis: Keep the personalization screens but remove the account creation step.

That’s much more powerful than simply asking:

“What should we change?”

RevenueCat, Adapty, and Superwall

Historically, running these experiments often meant building a lot of infrastructure ourselves.

I needed some combination of:

• Remote configuration
• User bucketing
• Variant assignment
• Experiment persistence
• Analytics events
• Paywall configuration
• Subscription management
• Revenue attribution
• Experiment dashboards

That’s a lot of engineering work.

Today, tools such as RevenueCat, Adapty, and Superwall remove a significant portion of that infrastructure.

They aren’t identical products, but they solve overlapping problems around subscription monetization and experimentation.

RevenueCat

RevenueCat Experiments documentation

I think of RevenueCat primarily as a subscription infrastructure layer that has expanded into experimentation and paywall tooling.

RevenueCat Experiments let me create experiments around Offerings and test things such as:

• Pricing
• Trial length
• Trial availability
• Subscription duration
• Product combinations
• Product ordering
• Paywall design
• Paywall metadata

RevenueCat currently supports experiments with 2–4 variants and can use Placements to control which Offering is shown at different paywall locations.

One thing I like about this model is that the experiment is connected to the actual subscription infrastructure.

Instead of thinking only about:

“Did the user click Subscribe?”

I can think about the resulting subscription lifecycle.

RevenueCat also has remotely configurable Paywalls, meaning the paywall itself can be changed without shipping a new app update.

That creates an important development loop:

Build SDK integration once
        ↓
Configure remotely
        ↓
Create experiment
        ↓
Collect results
        ↓
Iterate

rather than:

Change UI
   ↓
Code
   ↓
Build
   ↓
QA
   ↓
App Store review
   ↓
Release
   ↓
Wait for adoption
   ↓
Measure

For teams running frequent experiments, that difference is significant.

Adapty

Adapty A/B testing documentation

Adapty approaches experimentation with a strong focus on paywalls, flows, and onboarding.

Its current A/B testing functionality supports testing:

• Subscription prices
• Paywall design
• Paywall copy
• Trial periods
• Subscription durations
• Onboarding designs
• Flows

It also supports regular experiments for a single placement and cross-placement experiments for paywalls.

I find the concept of placements particularly useful.

Instead of having one generic:

showPaywall()

I can think in terms of:

onboarding_paywall
feature_paywall
settings_paywall
premium_content_paywall

Now the experiment can answer a more interesting question:

“Which experience works best at this specific point in the user’s journey?”

Adapty also supports audience-based targeting, allowing experiments and paywalls to be associated with particular user groups.

That becomes useful when a single global experiment isn’t enough.

For example:

New users
    → Experiment A

Returning users
    → Experiment B

Existing subscribers
    → No paywall experiment

Superwall

Superwall Paywall A/B Testing

Superwall is particularly focused on paywall experimentation and monetization.

Its model is built around campaigns, audiences, and remotely configured paywalls.

I can create multiple paywall variants and split traffic between them without requiring a new app release for each experiment. Superwall’s current documentation describes experiments as multiple paywalls assigned to an audience with configurable presentation percentages.

For example:

Audience: New users

Paywall A → 50%
Paywall B → 50%

or:

Paywall A → 60%
Paywall B → 30%
Paywall C → 10%

This is useful when I want to iterate quickly on monetization without turning every experiment into an engineering project.

Superwall also emphasizes audience targeting, offers, pricing, paywall placement, and holdout experiments as part of its experimentation workflow.

How I think about these tools

I wouldn’t choose a tool simply because it has an “A/B testing” checkbox.

The important question is:

How much of my experimentation infrastructure does this tool remove?

I care about things like:

Capability	Why I care
Remote configuration	Avoid shipping an app update for every experiment
Variant assignment	Reliably split users between treatments
Sticky assignment	Keep a user in the same variant
Audience targeting	Experiment with specific user segments
Paywall builder	Iterate without rebuilding UI
Subscription integration	Connect experiments to purchases
Revenue attribution	Measure business impact
Analytics	Understand why a variant won
Cross-platform support	Avoid duplicating experimentation infrastructure

RevenueCat, Adapty, and Superwall overlap in these areas, but their product philosophies and feature sets differ.

For me, the best choice depends on how my app’s monetization architecture is already built.

Sticky assignment is important

Imagine I assign a user to Variant B today.

Tomorrow they open the app again.

If I randomly assign them to Variant A, I have a problem.

The user saw:

Day 1 → Paywall B
Day 2 → Paywall A
Day 3 → Paywall B

Now the experience isn’t consistent.

I generally want a user to remain in the same experiment group for the duration of the experiment.

Conceptually:

User
  ↓
Experiment assignment
  ↓
Hash / persistent assignment
  ↓
Variant B
  ↓
Variant B
  ↓
Variant B

This is important for both user experience and measurement.

Superwall, for example, describes users as receiving a sticky assignment to a paywall variant for the life of the test.

Don’t forget the control group

Sometimes the most interesting experiment isn’t:

Paywall A vs Paywall B

but:

Paywall
   vs
No Paywall

This can answer a fundamentally different question:

Does this paywall actually create incremental value?

Imagine I have a premium feature.

I assume showing a paywall before the feature increases revenue.

But perhaps the paywall frustrates users and causes them to leave.

A holdout group can help answer:

Control
No new paywall

Treatment
New paywall

Then I can compare the downstream business impact.

Superwall explicitly supports holdout/value-of-paywall experimentation as one of its use cases.

Segmentation makes experiments much more interesting

An experiment that works for one group of users might fail for another.

For example:

              Variant A     Variant B

US users         5.2%         6.8%

German users     6.1%         5.9%

New users        7.0%         7.4%

Returning        3.2%         4.8%

Now I don’t simply have a winner.

I have a product insight.

Maybe Variant B is better for new users but worse for returning users.

This leads naturally to more sophisticated experimentation:

User
  ↓
Audience
  ↓
Experiment
  ↓
Variant

rather than:

Everyone
   ↓
Same experiment

However, I need to be careful here.

The more segments I analyze, the more likely I am to discover apparent differences just by chance.

So I treat segmentation as a source of hypotheses, not automatically as proof.

App Store testing and in-app testing are different

One distinction I find useful is separating acquisition experiments from product experiments.

App Store experiments answer questions like:

Which message gets more people to download?

In-app experiments answer questions like:

Which experience gets more users to subscribe and retain?

For example:

              Acquisition
                  ↓
        App Store experiment
                  ↓
              Download
                  ↓
               Product
                  ↓
        Onboarding experiment
                  ↓
              Activation
                  ↓
           Paywall experiment
                  ↓
             Subscription
                  ↓
          Retention experiment

Optimizing one layer without understanding the others can lead to misleading conclusions.

If my App Store page attracts people interested in “AI photo editing” but my onboarding immediately pushes them toward a completely different use case, I have created a message mismatch.

The user expectation starts before they install the app.

The biggest mistake: testing random things

A/B testing can become addictive.

Once I have the infrastructure, I can test everything.

Button colors.

Icons.

Headlines.

Animations.

Prices.

Spacing.

CTA text.

And suddenly the team is running twenty experiments without knowing what they’re trying to learn.

I think that’s a mistake.

I prefer to maintain a hypothesis backlog.

For example:

H1: Shorter onboarding increases activation.

H2: Showing annual pricing first increases revenue per install.

H3: Showing the paywall after first value increases purchase conversion.

H4: Positioning the app as an AI coach improves App Store conversion.

H5: Removing the free trial increases revenue per paying user.

Then I prioritize the hypotheses by:

Expected impact
×
Confidence
×
Ease of implementation

This gives me a much better experimentation roadmap.

What I would test first in a new subscription app

If I were starting with a new subscription app, I wouldn’t immediately build a huge experimentation system.

I’d start with the highest-leverage parts of the funnel.

Experiment 1: App Store positioning

I’d test different value propositions through App Store assets.

For example:

A:
"Your Personal AI Assistant"

B:
"Get More Done With AI"

C:
"Automate Your Daily Work"

The goal would be to understand which problem users actually respond to.

Apple’s Product Page Optimization tooling can test alternate icons, screenshots, and previews against the original product page.

Experiment 2: Onboarding length

I’d test:

Long onboarding
vs.
Short onboarding

Then measure activation, not just onboarding completion.

Experiment 3: Paywall timing

I’d test:

Paywall immediately
vs.
Paywall after first value

This is often more meaningful than changing button colors.

Experiment 4: Pricing presentation

I’d test:

Monthly → Annual

against:

Annual → Monthly

without changing the actual prices initially.

This lets me learn about price framing before introducing a more complicated pricing experiment.

Experiment 5: Offer structure

Once I understand the baseline, I might test:

7-day trial
vs.
3-day trial
vs.
No trial

At this point, I would be much more careful about downstream metrics.

A trial can increase initial conversion while reducing the quality of subscribers.

I try to avoid changing everything at once

One of the most frustrating experiment results is:

Variant B won, but we don’t know why.

That’s often self-inflicted.

If I change:

• headline
• price
• layout
• CTA
• product order
• trial length

all at the same time, I might get a big improvement.

Great.

But I’ve learned very little about which levers matter.

Sometimes that’s acceptable.

If my goal is simply to find a substantially better overall experience, a full redesign experiment can make sense.

But if my goal is learning, I prefer more controlled experiments.

There are really two different objectives:

Optimization

Find a better experience.

Learning

Understand which variable changes user behavior.

The best experiment design depends on which objective I have.

Experimentation should become a feedback loop

The real value of A/B testing isn’t one winning variant.

It’s the accumulation of knowledge.

Suppose I run these experiments:

Test 1
Annual-first paywall
→ +8% revenue

Test 2
Shorter onboarding
→ +12% activation

Test 3
Paywall after first value
→ +6% subscription conversion

Test 4
More aggressive discount
→ +10% conversion
→ -15% revenue per subscriber

After a while, I start building an internal model of my users.

I learn that:

• users need to experience value before purchasing
• annual plans perform well
• aggressive discounts attract lower-value subscribers
• shorter onboarding improves activation
• certain value propositions resonate with specific audiences

That’s much more valuable than simply knowing:

“Experiment #17 won.”

A/B testing doesn’t replace product thinking

This is probably my biggest takeaway.

A/B testing is not a substitute for product judgment.

It is a tool for improving product judgment.

If I test 50 random ideas, I don’t necessarily end up with a great product.

I can optimize the wrong thing.

I can optimize short-term revenue while damaging retention.

I can optimize conversion while attracting the wrong users.

I can optimize onboarding completion while reducing activation.

Data tells me what happened.

It doesn’t automatically tell me what I should care about.

That’s still a product decision.

My practical A/B testing checklist

Before I launch an experiment, I try to make sure I can answer these questions:

• What user problem am I trying to solve?
• What is my hypothesis?
• What exactly is changing?
• What is the control?
• Who is included in the experiment?
• How are users assigned to variants?
• Will assignments remain stable?
• What is my primary metric?
• What are my secondary metrics?
• How much traffic do I need?
• How long should the experiment run?
• What would make me declare a winner?
• What would make me conclude that the experiment is inconclusive?
• Could seasonality or another product change affect the result?
• What happens after the experiment ends?

If I can’t answer these questions, I probably shouldn’t launch the experiment yet.

The most important mindset shift

For me, A/B testing became much more useful when I stopped thinking about it as:

“Let’s test two designs.”

and started thinking about it as:

“Let’s learn something about our users.”

That small change completely changes how I approach experimentation.

A good experiment isn’t necessarily one that produces a winner.

Sometimes the best result is:

No meaningful difference.

That tells me I can stop spending engineering time on that particular idea.

Sometimes the result is:

Variant B converts better but produces less revenue.

Now I’ve learned something about the relationship between conversion and monetization.

Sometimes:

The treatment works for new users but not returning users.

Now I have a segmentation hypothesis.

And sometimes:

Our hypothesis was simply wrong.

That’s not failure.

That’s exactly what experimentation is supposed to uncover.

Final thoughts

Mobile apps give us an enormous number of opportunities to experiment.

I can test the first impression in the App Store, the onboarding experience, the moment I ask for money, the pricing structure, the paywall design, the offer, and eventually the entire monetization funnel.

Apple gives us native experimentation capabilities for App Store product pages, including Product Page Optimization and Custom Product Pages.

Inside the app, platforms such as RevenueCat, Adapty, and Superwall make it much easier to remotely configure paywalls, split users into variants, target audiences, and connect experiments to subscription outcomes.

But the tooling isn’t the most important part.

The most important part is having good questions.

I don’t want to run more A/B tests just for the sake of running more A/B tests.

I want every experiment to teach me something.

Because over time, that’s what turns A/B testing from a growth tactic into a product development system:

Hypothesis → Experiment → Evidence → Learning → Better Product → New Hypothesis.

And for me, that’s the real value of experimentation.