Experiments for the risk-averse

By Posted in - Experiment on August 6th, 2015 0 Comments

You’ve decided you want to do something a little bit out of the ordinary for your next marketing push. But you’re also a little bit concerned about getting it right…

Tracy has already written about the risk-averse nature of education marketing, and how our own fears can sometimes hold us back. With that in mind, I’ve got a few ideas on how you can either test or refine your risky experiment.

Step 1: Do some pre-experiment experiments

If you’re doing something you’ve never tried before, you might want to attempt to gauge some reactions beforehand. While you won’t be able to replicate the large-scale effects of going live with a new idea, you’ll at least be able to get a feel for how your experiment might play out. There are a couple of ways you can do this…

Focus groups

Sitting around a table with some of your target audience is a great way to explore what parts of your experiment might work (and what really won’t). As well as any discussions you want to have, you can also run small experiments with your groups. Depending on what you want to test, you can try various methods.

If you wanted to see how your audience would respond to a certain type of visual content, you could set up sentiment reaction dials for them to use while viewing your content. Alternatively, you could show them each piece of content and talk through their responses as a group – it all depends on what kind of information you want to have to be able to produce your content effectively.

Remember that there’s always a risk with focus groups that people will just say what they think you want to hear. Try to make sure you tease out their true feelings, even if that means hearing that they don’t like your ideas


If you really want to dig into your audience’s thoughts on your experimental idea, a series of interviews is a good option. One-to-one conversations can lead to more revealing insights than a group situation, and you can pursue lines of questioning with an individual that might waste other people’s time in a focus group.

This technique is very useful for getting qualitative results on potential options for a new approach. If you have a a new campaign idea to try out, for example, you could carry out a series of interviews in which you ask your participants to choose between different options on screen, and get them to explain the reasoning behind their choices.

By conducting a series of interviews with different members of your target audience, you’ll be better able to predict how an experiment might work when you run it for real.

Remember that with focus groups and interviews the information you’ll get back is subjective – but that doesn’t make it invalid. We’ve all seen episodes of The Apprentice when teams didn’t listen to feedback from their potential customers. It never ends well.



Step 2: Don’t be afraid to go live

Sometime you can only really test your ideas in a live environment. But that’s OK! There are ways to track the effects of your experiments so that you can gradually improve them as you go along.

A/B testing

I’m a big fan of A/B testing (and its more detailed cousin multivariate testing). In case you don’t know what it is, this is a method of experimentation that allows you to test different approaches with different sections of your audience, usually in a live environment.

As an example, you could send two different versions of a marketing email to small sections of your database to test which subject line is more likely to encourage recipients to open the email. Or you could show different versions of a mailing list signup page to new website visitors to see which layout brings in more contact details.

What’s great about A/B testing is that it doesn’t have to affect your entire audience. If you’re worried about the negative effects of trying a new approach, you can limit your test to a small portion of your audience. You’ll still get some very valuable data, but you won’t suffer terribly if your experiment is a disaster…

If you want a few examples of A/B tests and the difference they can make, check out this Hubspot blog post.

As with any of the other techniques I’ve mentioned in this post, it’s important to make sure you track the effects of your experiments. AB testing can give you great data to make decisions with in the future, but it’s no use if you just ignore it and plough on with your chosen route.


Running an experiment using email, social media or your website means that you’ll typically have some kind of tracking system built in. Once your experiment is over you can look at the data you’ve gathered and make a decision on whether or not it’s been successful.

Obviously it’s up to you what constitutes “success”. You might want to see a 20% jump in email sign-ups, or 1000 new Facebook likes, or an increase in the amount of time visitors spend on your website. If “success” is a quantifiable measure, then analytics tools can help you out.

Where analytics really come in handy is helping to convince those people who like to see data-driven support for a new approach. By showing evidence that your experiment has had clear effects, you can pave the way for more bold moves in the future.

Gather that evidence

As I’ve mentioned a couple of times in this blog post, the key thing with experimentation is to record your results. You’ll feel more confident about your new approach if you have evidence to back it up, and you’ll hopefully be able to take a few steps towards becoming a little more experimental.

If you’ve got a scary idea you want to run by us, we can probably help you out. Just let us know and we’ll see if there’s anything we can do to calm your fears.

Please leave a Comment