Improve the Effectiveness of your Flows with A/B Testing

A vital part of building something is testing it out: build, measure, learn and adjust. At TextIt, we believe there’s no such thing as too much testing. To that end, we provide multiple tools to enable you to test your flows both inside TextIt and out. 

Within TextIt, you can use the 'Split randomly' RuleSet to create equally distributed pathways through which your contacts will randomly pass, enabling you to test the efficacy of your flows. We also provide a handy simulator on the same page you build your flows to allow you to test them as you build. 

Outside the platform, we encourage you to use your 1,000 complimentary credits to measure flow performance and conduct A/B tests pilot users.

A/B testing (also called split or multivariate testing) is a method you can employ to test the effectiveness of two or more versions of a single message, flow or campaign with respect to a desired outcome. It's a type of controlled experiment conducted on similar audiences, the initial version being the control and any additional versions being treatments.

Getting Started

  • Define objectives: What are you trying to achieve with your flow, distribute information, collect data or something else?

  • State your hypotheses: Which version of your flow do you think will perform best? Why?

  • Create a list of the things want to test: For example, flow length, message length, vocabulary and method of engagement (will the contact prompt the flow through a keyword or phone call, or will the flow start on a schedule?).

How to conduct an A/B test

Test one item at a time. 

Let’s say version A addresses your contacts by name while version B doesn’t. When testing two versions of a flow, make sure you’re only measuring the impact of a few changes at a time. If version A is too dissimilar to version B, you’ll have trouble isolating the variable responsible for the result.

Send the flow to similar representative samples. 

Show version A and version B to similar groups and ensure each is representative of the entire target population. If you show version A to only experienced contacts and version B to only new contacts, your results may be skewed. Ideally, you want to split your target population evenly.

Be patient. 

One of the best things about TextIt is time to deployment. You can sign-up, connect a channel and send messages in as little as 5 minutes if you want to, but this doesn’t mean everyone should take that approach. Being that SMS is just as much an extension of an organization's brand as email, larger deployments need to conduct tests to make sure flows are well-received and driving results.

Creating flow versions

Creating a new version of a flow is simple: click the gear icon menu in the flow editor and select the “copy” option. This will produce an exact copy of the flow for you to edit and rename. 

You can link your versions using a 'Split randomly' RuleSet.

Evaluating the results of an A/B test

Completion Rate

In addition to the primary marker of success you select when you identify your objectives, we provide a completion rate for each flow represented by a percent value. The completion rate allows you to see how many of the contacts that have entered the flow have finished it. Comparing the completion rates of two versions of a flow is easiest way to measure effectiveness.

Opt-Outs

Another way to evaluate the effectiveness of a flow is to track opt-outs. Learn more about tracking and managing opt-outs here.

Questions? Comments? Let us know! We love hearing from you.