Tackling the tricky question of attributing impact

A question we often get asked is ‘how do I know that the difference I see in young people’s well-being is due to my programme, or due to other causes? This question is not new for researchers and one that you will find comes up in any survey or research that you do.

The simple answer is that you can never be 100% certain. In their laboratories, scientists put enormous effort (and money) into isolating different factors or conditions in their experiments. This allows them to focus their experiment on what they want to test.

Unfortunately, social scientists don’t have this luxury. Because they work with people, it is impossible to recreate laboratory conditions.

Instead, the nearest thing to recreating lab conditions is to use a ‘control group’. This is a comparison group, drawn from a population with similar characteristics and selected at random, which allows you to say ‘what would have happened’. By comparing the results of your experiment with what happens to the control group over the same period of time you can isolate the impact of what you are testing.

It is always worth thinking about whether you could run a control group within your survey. However, it may not be possible or desirable. For one, it is expensive and it is not worth doing unless you do it properly. And second, if you work with a particularly vulnerable group it may not be something you want to do as it might mean not providing a service to a comparison group that you know really needs it.

Without a control group your results will still give you a good sense of the difference you make to young people’s lives. When presenting your results it can be useful to talk about ‘contribution’ rather than attribution – recognising that there are many influences on young people’s lives and your work is one of them.

If you would like to use a control group and need help, you can contact us about our consulting services by emailing wellbeing@philanthropycapital.org.


About John Copps

John is part of NPC's research and consulting team and is the founder of NPC's Well-being Measure, a social business that provides an online tool to measure young people’s well-being. He has eight years experience of research and consulting, and is passionate about how data can be used to improve the performance of organisations. John is a regular contributor to NPC's blog and has also contributed to pieces for BBC Radio, the Guardian, and the Financial Times. John is a governor of a secondary school.
This entry was posted in General Well-being. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s