PPF Blog

AskThem Experiment 8: split-test of email subject headings

This is the writeup of Experiment Eight of our open-analytics experiments on AskThem, conducted with support from Google Civic InnovationBackground on this project.

Previous tests have been run on Optimizely and Google Analytics – this focuses on Action Network, an organizing platform that we primarily use as our email mailing list solution. It’s a terrific, affordable, user-friendly service and I’ve been excited to evangelize it – shouts Jason Rosenbaum, j_ro, their stellar & responsive & bright CTO.

In announcing AskThem to our email list, we were primarily communicating with users of our previous non-profit flagship project, OpenCongress.org. So in introducing the site to them, we tested three variants of subject headings, with three twists on mentioning the OC brand while describing the value proposition of AskThem. Before going to our entire list, following best-practice, I tested five variants with 2500 recipients apiece ::

Mailings 12 & 13 ::

AskThem_Experiment_8_mailings_12-13

 

 

 

 

 

 

 

Mailings 11 & 10 ::

ASkThem_experiment_8_mailings_11-10

 

 

 

 

 

 

 

Mailings 8 & 7 ::

ASkThem_experiment_8_mailings_8-7

 

 

 

 

 

 

 

Mailings 6 & 5:

ASkThem_experiment_8_mailings_8-7

 

 

 

 

 

 

 

… the goal is to track both opens & clicks, and both matter in different weights at different times, but we’ll prioritize clicks, to get eyeballs onto our new site –  and with that, the “AskThem.io launches today” message won with 50 clicks, while the “We the People” message came in a close second with 48.

From this, it’s clear that the timeliness and novelty of a new site prompted clicks to see the new pages – but that’s not a reliable pitch, necessarily, once a platform & brand has launched. The product comparison to “We the People” proved pretty popular as well, but that might fade too. Currently we’re experimenting with regionally-specific updates of good questions from other people in your area, and we’ll continue to promote questions that receive an official response, as that’s our site’s core function. As we gather more users and generate more questions, we hope to have more funny & entertaining questions to report. We’re also working to tie-in questions to events, such as city council meetings or committee hearings or campaign debates, and to work with media partners to enable their site readers to ask questions of public figures in the news. AskThem already integrates city council agendas in a handful of cities, such as Philadelphia; what issues are you most interested in asking about, at different levels of government? From here, future email subject heading tests can try more fanciful and timely wordings, and to our full mailing list.

From these revealing & crucial analytics experiments, my open-source development team and I now have empirical evidence on how we can enhance our non-profit platform. From here, I have other good ideas to connect visitors to their local elected officials – but we need charitable funding support to continue operating and move towards sustainability. Please support our non-profit work. Questions, comments: david at ppolitics.org.

This entry was posted in Uncategorized. Bookmark the permalink.

Take Part

Your email address will not be published. Required fields are marked *