A/B Testing

What is A/B Testing and How Does it Help?

A/B Testing lets you test up to five different email variants (A, B, C, D, E) with different subject lines or email bodies. This helps you identify which version performs better for opens and replies in your cold email campaigns.While composing your email step, you can add multiple email variants (up to 5 for each step: A, B, C, D, E variants). Each variant can have a different subject and email body. After starting your campaign, you will get detailed analytics via the Stats tab and will be able to see which of the test variants works better: more email opens stand for a better subject and more replies for a better email body.

When you add a new email variant, SmartReach will split the prospects in your campaign as evenly as possible, with each variant going out randomly to approximately 50% of the prospects. For example, if you have a list of 200, the system will automatically send variant A to 100 and variant B to the other 100.

Email steps are split up randomly according to the number of variants on the step. Each step is randomized independently from each other, so if [Day 1: Opening] Step variant was A, it doesn't mean this person gets A variant in the following steps as well.

Setting up A/B testing

Step 1: Add an Email Step: During campaign creation, add an email step in the campaign setup process.

Step 2. Check the results after launching your campaign: Launch the campaign (e.g., using Campaign Soft-Start). Smartreach.io evenly and randomly splits prospects across active variants. For example, with 200 prospects and two variants, each variant is sent to approximately 100 prospects.

You can see the detailed results for each step and its variants by selecting your campaign → Reports →Scroll down at the bottom to see the results of the report

You can also check it in the template section of the reports

Delete an Underperforming Variant: In the email step editor, Click on the under performing variant and click on the delete icon on top to remove the step that does not perform well. You can also choose to turn it off for the time being by pausing it with the toggle switch next to the variant

What happens when I delete an email variant ?

i) The variant and its reporting stats will be deleted.

ii) The existing variants will be renamed. Let's say you had A, B and C variants, and you delete the B variant. Then the C variant will be renamed to B, so now you will have two variants: A and B (which was earlier C).

iii) On deleting a variant, prospects will continue receiving emails using the rest of the variants for the step.

For example: Let's say you have a campaign which has a Step-1 with two variants A and B. You have added 100 prospects to the campaign and started the campaign with the Campaign Soft-Start. Let's say that on the first day, 20 prospects each have received variants A and B (so 40 prospects have received the Step 1 till now). Now, you delete variant B from the Step-1. Then the other 60 prospects in the campaign will receive variant A of Step-1.

Best Practices for A/B Testing

Test One Element at a Time: Focus on testing either the subject line (for open rates) or email body elements (e.g., headings, body text, call-to-action, images, layout) for reply rates to clearly identify what impacts results.

Random Distribution: Each step’s variants are randomized independently, so a prospect receiving variant A in one step may receive variant B in another thus you should be able to track performance of each variant with precision.