%/email-for-growth-teams providers ↗
guide

A/B Testing Email

Email A/B testing is only useful when one variable changes, the audience split is valid, and the winning rule is chosen before the send. This guide keeps tests small enough to run and strict enough to trust.

last updated 2026-05-07 4 sections
section 01

What to test first

Start with variables that map to the metric being measured. Subject lines affect opens. CTA copy and offer framing affect clicks. Journey timing and audience rules affect conversion.

variableprimary metrickeep constant
Subject lineOpen rate where opens are still useful.Audience, send time, body, CTA.
CTA copyClick rate.Subject, audience, send time, layout.
Send timeClick or conversion by window.Subject, body, audience.
Journey branchConversion or retention.Entry criteria and attribution window.
section 02

Sample size limits

Small lists produce noisy tests. A test can still be useful, but it should be treated as directional unless the sample is large enough and the effect is clear.

  • ok Pick the success metric before sending.
  • ok Avoid testing more than one variable in the same split.
  • ok Do not stop a test early because one variant leads after a few clicks.
  • ok Use holdouts for journey-level impact, not only copy changes.
section 03

Holdouts vs split tests

A split test compares two versions of an email. A holdout compares sending the journey against not sending it, or sending a lighter version. Holdouts answer whether the email should exist.

test typequestion answeredgood fit
Subject splitWhich wrapper earns more attention?Broadcast and launch emails.
Content splitWhich message drives more action?Activation and upgrade prompts.
Journey holdoutDoes this automation change behavior?Lifecycle programs and dunning.
Frequency holdoutCan fewer sends achieve the same result?Retention and reactivation.
section 04

Promotion rules

A winner should become the default only when the result matches the goal and does not create a secondary problem. More opens with fewer conversions is not a win for most lifecycle email.

  • ok Check the primary metric first.
  • ok Review complaint, unsubscribe, and reply quality before promoting.
  • ok Keep a record of variant, audience, date, and decision.
  • ok Retest when the audience or offer changes materially.

related startup email pages

pillar

Email marketing for startups

Strategy from waitlist to activation.
examples

Startup email examples

Concrete examples and teardown notes.
templates

Startup email templates

Copyable packs by lifecycle moment.