Click here to access Github repo containing relevant code or continue reading below on for non-technical description of the project.
Technologies/Skills Used
- A/B testing
- R
data.table,sandwich,tidyverse,lmtest, andstargazer
Problem Description
570 million people in Sub-Saharan Africa lack a connection to an national electric grid. Gasoline powered generators and kerosene laterns rely on expensive fuel while emitting tons of dioxide and other air pollutants. Off-grid solar home systems, like the d.light device shown below, offer a zero carbon power solution that is more cost effective in the longer term than conventional energy technologies.
However, the upfront costs of solar home systems often exceed families’ ability to pay out of pocket so many customers pay for these systems on a pay-as-you-go payment plan. Solar retailers with customers buy products on credit send text message reminders in an attempt to improve customer repayment. In conjunction with a solar retailer in Sub-Saharan Africa, Catalyst Off-Grid Advisors, and two other fellow graduate students at the UC Berkeley School of Information, I ran an A/B test to answer the following question: “How do text message reminders that implement minority norm framing rather than the previously utilized loss framing affect the timing of repayments for pay-as-you-go solar customers who buy at-home solar products on credit?”
The Experiment
To measure the effect of text messages, we tracked customers days past due at time of repayment, which is a measure of the amount of time elapsed between the expected payment date and actual payment date. We also recorded collection rate, a ratio comparing the amount expected to the amount actually paid. We ran a 5 week field experiment using blocked random assignment to assign treatment or control on a sample of 760 customers in June and July of 2022. As customers approached a payment due date, they were sent upcoming reminder messages with the control or treatment wording. In the case that customers did not pay by the expected payment date, they were sent late payment reminders 1, 7, 14, 21, and 28 days after the scheduled payment date. Sample upcoming reminders in English (actual messages were sent in English and another language depending on customers preferred language) are shown below:
Prior to the experiment, we also completed a systems test, satisfactory power analysis, and covariate balance check.
Results
We failed to detect a statistically significant effect of the treatment on the average days past due between treatment and control groups, even when controling for a variety of customer attributes and including fixed effects of day of the week. All analysis was conducted in R using various statistical libraries such as data.table, sandwich, tidyverse, lmtest, and stargazer.
These findings held true even when switching the outcome variable from average days past due to average collection rate. Despite the inability to detect a treatment effect, one positive outcome of the experiment was a new way to segment customers. We are also currently exploring different phenomenon that we wish to test with a field experiment with the goal of making an impact for solar retails fighting both climate change and the global energy deficit.