Why Subject Line Klaviyo A/B Testing Is a Waste of Time
Most eCommerce marketers spend hours split testing subject lines. They tweak emojis, change capitalisation, and rewrite hooks, hoping a two percent bump in open rate will translate to more sales. It rarely does.
Apple killed the open rate metric in 2021. When they released iOS 15, they introduced Mail Privacy Protection. This update fundamentally changed how email pixels fire. If you still optimise your campaigns based on open rates, you are optimising for bots.
I see this pattern across dozens of ad accounts and Klaviyo dashboards every month. Brands run standard subject line tests, declare a winner based on a 65 percent open rate, and wonder why their revenue stays flat. The data they rely on is completely fabricated by background machine processes.
To grow your email revenue, you must stop looking at top-of-funnel vanity metrics. You need to shift your focus to bottom-line metrics that reflect actual human behaviour.
Apple Mail Privacy Protection and Klaviyo A/B testing flaws
Apple’s Mail Privacy Protection automatically pre-loads email content in the background. When an email hits an Apple Mail inbox, the Apple proxy servers download all the images. This includes the tiny, invisible tracking pixel Klaviyo uses to record an open.
Because the server downloads the pixel, Klaviyo registers the email as opened. The customer might never actually look at the email. They might delete it without reading it. But your dashboard will still show a successful open. This artificially inflates open rates to near 100 percent for any subscriber using an Apple device.
This is a massive problem for the Australian market. Apple devices dominate mobile usage in Australia. When we look at the data across our client accounts, over half of most subscriber lists use iOS devices. This makes your aggregate open rate data completely unreliable.
If you run standard subject line klaviyo experiments, the platform uses open rates to determine the winner. You might test a short subject line against a long one. The test runs for four hours. Klaviyo looks at the open rates, crowns a winner, and sends that version to the rest of your list.
But the winner was not chosen by your customers. The winner was chosen by which segment had more Apple Mail users checking their phones during that four-hour window. You are feeding the bots.
We audited a seven-figure apparel brand last month. They spent weeks testing different emojis in their subject lines. Their average open rate sat at 62 percent, but their click rate hovered around 0.8 percent. They thought their emails were performing well. In reality, their Klaviyo deliverability was suffering because internet service providers saw massive open rates with zero engagement.
Key revenue metrics for effective email A/B testing in Klaviyo
You must pivot your attention away from opens and toward metrics that require actual human intent. A bot can open an email. A bot will not click a link, browse a product page, and complete a checkout.
Click-through rate (CTR) and placed order rate (POR) reflect real customer engagement. CTR tells you if the content inside your email was compelling enough to drive traffic to your store. POR tells you if that traffic actually converted into sales.
But the ultimate North Star metric for email growth is Revenue Per Recipient (RPC).
RPC calculates the total revenue generated by an email divided by the number of people who received it. This metric is crucial because it accounts for both conversion rate and average order value. If you want to generate more revenue from your list, every test you run should aim to increase your RPC.
Focusing on bottom-of-funnel conversion metrics completely changes your testing hypothesis.
Consider a standard discount test. You send Version A offering 20 percent off. You send Version B offering 10 percent off. Version A will almost certainly get a higher click-through rate and a higher placed order rate. Customers love big discounts.
But Version A destroys your profit margin. When you measure the test by RPC, you might find that Version B actually generated a higher net profit per subscriber. The lower discount resulted in fewer total sales, but the retained margin made the overall campaign more profitable.
According to Klaviyo’s benchmark data, top-performing brands consistently optimise for conversion over superficial engagement. When I was running my own stores, shifting our focus from CTR to RPC changed how we structured every campaign. We stopped trying to get cheap clicks and started focusing on high-intent buyers. If you are looking to align your testing strategy with true revenue metrics, our free Klaviyo audit covers the exact performance framework we use across our client accounts.
Email content elements to test beyond subject lines
Once you accept that subject line testing is flawed, you can focus on the variables that actually move the needle. You should test the internal body content of your emails. This is where you convince the customer to buy.
Testing the body of your email directly impacts your click-through rate and your placed order rate.
Offer incentives and structural hooks
The most impactful test you can run is your offer structure. Do not guess what your customers want. Test it.
You can test a percentage discount against a dollar-off amount. A 15 percent discount on a $100 product saves the customer $15. Offering a flat $15 off often converts higher, even though the monetary value is identical. The human brain processes flat dollar amounts faster than percentages.
You should also test physical offers like a free gift with purchase. We ran this exact test for a homewares brand last quarter. We split their welcome flow. Half the traffic received a 10 percent discount code. The other half received a free set of coasters with any purchase over $80.
The coasters cost the brand $4 landed. The 10 percent discount cost them an average of $12 in lost margin per order. The free gift offer increased their average order value by $22 and lifted their placed order rate by 1.4 points. Evaluating the margin impact of different discount types is how you scale profitably.
Design and CTA placement
Most eCommerce emails look identical. They feature a massive lifestyle image at the top, followed by a headline, a block of text, and a button buried at the bottom.
Test your call-to-action (CTA) positioning. People skim emails on their phones. If your CTA sits below the fold, a huge portion of your audience will never see it. Move your primary button above the fold. Make the button colour contrast sharply with the background. If your brand colours are black and white, use a bright blue or green button.
You should also test plain-text formats against visual, image-heavy designs.
Founders often assume customers want beautiful, highly designed emails. I have found the exact opposite to be true in many accounts. Plain-text emails look like they came from a real person. They bypass the mental filter customers use to ignore marketing material. We tested a plain-text letter from the founder against a polished HTML design for a skincare brand. The plain-text version generated an RPC of $0.18, while the heavily designed version only managed $0.11.
Klaviyo A/B testing setup focused on Revenue Per Recipient
Setting up these tests requires a different approach inside Klaviyo. You cannot use the standard campaign A/B testing tool if you want to optimise for revenue over time. You need to build split branches inside your automated flows.
Automated flows generate the highest return on investment for any eCommerce brand. Your Klaviyo welcome flow and abandoned cart flow run 24 hours a day. Optimising these sequences compounds your revenue over months and years.
To test content properly, open your flow builder. Drag a “Conditional Split” block into the sequence. Set the condition to “Random Sample” and choose 50 percent. This splits your incoming traffic evenly down two separate paths.
Place your control email down Path A. Place your variant email down Path B.
You must configure your sample sizes and test duration for statistical significance. A test run on 200 people over a weekend tells you nothing. You need volume. For most brands doing under $5 million a year in revenue, a flow klaviyo a/b testing branch needs to run for at least 30 days to gather enough data.
Do not touch the test while it runs. Let the data accumulate.
When you are ready to review the results, click on the email blocks. Change the reporting metric in the Klaviyo sidebar from “Open Rate” to “Placed Order Rate” or “Revenue”. Look at the total revenue generated by Path A divided by the number of deliveries. Compare that to Path B. The path with the higher RPC is your winner. Delete the losing branch, set the winner as your new control, and build a new test.
Next steps for scaling your eCommerce Klaviyo strategy
The days of hacking subject lines for cheap open rates are over. Apple forced the industry to mature, and your email strategy must mature with it.
Your first step is to audit your existing email campaigns. Go into your Klaviyo account and turn off any active subject line split tests that rely on open rates. They are feeding you bad data and wasting your traffic.
Shift your focus to high-use flow optimisation. Build out random sample splits in your welcome series and your abandoned checkout flows. Test your offer structures, evaluate your margin impact, and experiment with plain-text layouts. Base every decision on Revenue Per Recipient.
Building a systematic, revenue-driven testing calendar takes time. It requires discipline to ignore vanity metrics and wait for statistical significance. But this is exactly how the top-performing brands scale their email revenue month over month.
Your Klaviyo account is probably costing you more than you think
Most Shopify stores we audit have at least 5 of the same 24 revenue-killing issues in their Klaviyo setup. The free Klaviyo Audit catches them in 48 hours.
If you want an expert set of eyes on your current setup, we can review your account. You can request a free Klaviyo audit from our team to see exactly where your flows are leaking revenue.
