Lessons on SMS Survey Design: A Factorial Experiment on SMS Survey Participation using TextIt

by Kellan Alexander

Laterite, a Rwanda and Ethiopia-based research and advisory firm, recently used TextIt to implement an SMS survey with agricultural cooperatives in Rwanda. This survey was unique in that it was also used to measure the impact of survey delivery on participation rates. We spoke with Josep Casas and Mallory Baxter of Laterite's research team to learn more about their findings.

Laterite

Laterite provides data collection and econometric analysis services to government agencies and ministries, international NGOs, think thanks, academic institutions, and development organizations. They conduct the vast majority of their work in countries in which they have an established presence in order to build a strong understanding of the local context.

conTEXT

Using TextIt, Laterite has developed an SMS-surveying solution called conTEXT that is operational on all of Rwanda’s telco networks. It allows the company to send reverse-billed SMS surveys (i.e. at no cost to respondents) as well as airtime incentives on completion of those surveys.

The Experiment

Using conTEXT, Laterite recently conducted an SMS-based survey with members of agricultural cooperatives in Rwanda. They were hired by SPARK International to assess its Cooperative Support Programme (CSP), which targets agricultural cooperatives and SMEs operating in bean, Irish potato, maize and horticulture value chains with an aim to increase food security and stability by accelerating agri-business development, promoting rural economic growth and generating jobs in Rwanda's agricultural sector. The project's budget for interviews was limited, so Laterite and Spark decided to use conTEXT to collect complementary data from cooperative members.

This was the first SMS survey Laterite had carried out in Rwanda, so they wanted to use it to determine the best way to deliver future surveys. The key question was: can small tweaks to SMS survey delivery increase participation, and if so, which? Laterite's research team designed a factorial experiment to explore this question. They tested five factors they thought might impact cooperative members' willingness to participate:

  • Introduction Type: The majority of cooperative members (85%) were introduced to the survey via a phone call while others were introduced to the survey directly via SMS (15%). The call was scripted and focused on potential questions participants might have regarding the content and purpose of the survey.
  • Time of Day: Half of the cooperative members received their surveys in the morning (10 AM) while others received them in the afternoon (4 PM).
  • Personalization: Half of the cooperative members received messages that were personalized (addressed them by name in the introduction using TextIt’s substitution variables) while others received generic messages.

  • Financial Incentive: Half of the cooperative members were informed before the survey that they would receive an incentive of RWF 100 in airtime for completing the survey and received this incentive on survey completion. The other group did not receive any reward.
  • Survey Expiration: Half of the cooperative members were made aware of the fact that the survey would expire after 1 week (as flows can be configured to expire after a period of inactivity) while others were not.

Laterite tested the impact of the phone introduction by randomly allocating 15% of the sample to the SMS Intro group and the rest to the Phone Intro group. Individuals in the phone intro group were called by the enumerator team before the TextIt flow was initiated. Once consent had been obtained following the initial introduction, Laterite implemented a factorial experiment with the remaining factors.

In a factorial experiment, an intervention is broken down into factors and levels, e.g. the factor representing time of day has two levels: morning and afternoon.  In this experiment, Laterite was testing 4 strategies–each with 2 levels–resulting in 16 (2^4) possible experimental conditions. This means that they had to deliver the survey in 16 unique ways. Using a series of “Split by Contact Field” RuleSets, Laterite could create different routes inside the TextIt flow so that each candidate received the messages corresponding with their own combination of factors.

While analysis is on-going, preliminary results suggest that pre-survey phone calls and message personalization had the largest impact on increasing participation rates. Interestingly, financial incentives appear to have had no effect.

Preliminary Results

Introducing the survey via phone call instead of SMS increased the participation rate by 18% while message personalization led to an increase of 8%. Other factors weren't statistically significant. 

Figure 1. The effect of each of the factor on participation rates.

DesktopTextIt20BlogPicture2png

*Green factors were statistically significant while red factors were not

Laterite also tracked the number of errors that occurred for each type of question, allowing them to identify which questions work best. In this case, 'error' is defined as an invalid response as determined by the response rules present in each RuleSet. For example, Laterite included a question that asked participants to reply with their gender and age separated by a plus sign delimiter (e.g. "female+30"). They found that 37% of respondents could not answer the question correctly on the first try while 10% raised an error at least 3 times. Message form questions–as these are called–are unique in that they enable you to collect multiple values in a single message, reducing the number of questions required. To ensure message form error rates are low, you'll want solicit feedback from your contacts and test other value combinations and delimiters (spaces and periods) to determine what works best in your project's context. 

Moving Forward

Laterite’s research team is currently working to understand the links between these factors, survey completion, and attrition between survey rounds. Among their questions is whether a single phone introduction has a lasting effect on participation rates. Laterite is also looking to replicate this experimental design in other contexts to build a body of evidence around the optimal delivery of SMS surveys. 

Questions? Comments? Let us know! Questions for Laterite can be directed to Josep or Mallory, linked above, or its website.