This is the writeup of Experiment Four of our open-analytics experiments on AskThem, conducted with support from Google Civic Innovation. Background on this project.
Like experiment three, this test was on our radar screen, but was boosted in priority from a video Skype conversation with our non-profit advisor Tom Steinberg of mySociety (UK). Many thanks to Tom for his experience & insights, as before.
Experiment Four centers around another homepage above-the-fold copy value proposition, and how relatable to make it. Tom suggested a copy tweak – see background in previous exp. 3 – currently first variant of single-button simplification copy is “ask questions and get answers / from every elected official nationwide and any verified Twitter account”, with the single-button up top:
… which can be contrasted with, “Where people like you /Â Ask questions & get answers /Â From politicians, celebrities, and any verified Twitter account.” Also single-button up top. Note the terms “politicians, celebrities” – whereas typically our AskThem marketing descriptions use “elected officials” (more technically accurate) and “public figures” (as opposed to potentially-frivolous connotations of “celebrities”), to emphasize our civic mission.
… running in parallel with experiment three, this test number four is ongoing – and tests the same two goals of  a) overall engagement (any click on the page) and b) specifically clicks on the orange “ask-your-own-question” button. Results so far on engagement ::
… shows the “people like you” copy with -6.6% effect on overall engagement vs. the baseline (see 3.a) compared to a -2.1% effect for the “Ask questions and get answers” copy. There’s early evidence it’s less effective than the simpler copy and the original, more-full page design, which showcases more sample question content.
But on focusing clicks on the button, “people like you” is almost exactly as good as the simpler copy so far:
… each over a 9.5% improvement, so more data is needed – and if the single-button simplification is implemented fully in production code, as seems likely, then a straightforward A/B test of the “people like you” copy could work. User survey could also elcuidate the question of if visitors want to ask questions to politicians (vs. elected officials), if they understand the difference (the former could include candidates, appointees and other public figures), and if one is more appealing. Same for questions to celebrities, which AskThem wasn’t designed to foreground, and which hasn’t been as heavy a use case as our questions to city council members.
When in doubt, though, follow the advice of Tom Steinberg, is my mantra. So we’ll get more results when we’re next able to streamline the homepage offerings. From these revealing & crucial analytics experiments, my open-source development team and I now have empirical evidence on how we can enhance our non-profit platform. From here, I have other good homepage revisions to connect visitors to their local elected officials – but we need charitable funding support to continue operating and move towards sustainability. Please support our non-profit work. Questions, comments: david at ppolitics.org.