Debunking Research Impact: Theories of Change

Most research institutions are asked to do it, but for many it is an undertaking that is approached with a certain trepidation and even reluctance. To what am I referring? Demonstrating the non-academic impact of research.

What’s all the fuss about?

Researchers and their institutions are increasingly being asked to demonstrate research uptake and non-academic impact, be it for the Research Excellence Framework or to meet funder requirements. Although an abundance of options exist, because this is a relatively new ask, academics and research uptake/communications specialists are grappling with how best to approach demonstrating impact/uptake in a systematic and rigorous fashion.

Fear not!

Recognising this challenge, as part of its Annual Symposium, on 23rd September LSHTM held a ‘Demonstrating Impact Workshop: Toolbox and Genius Bar’. The workshop brought together presenters from a range of institutions including LSHTM, the Overseas Development Institute (ODI) and Altmetric who introduced a range of practical tools that could be used to plan, track and describe the non-academic impact of research.

As a core facet of SHARE’s work is ensuring research uptake – that our research reaches the right people at the right time in the right format so that it can help improve sanitation and hygiene policy and practice globally and nationally – I was delighted to be invited to present at this workshop and participate in the Genius Bar.

Theories of Change

Having heard from the organisers – Agnes Becker of the IDEAS project and Annie Holms of the Strive Consortium – I kicked off the workshop with a presentation on SHARE’s approach to planning for impact: our Theory of Change (ToC).

Essentially, a ToC is a participatory planning process used to identify the change you want to see, how it will be reached, and what will be used to measure progress along the way. ToCs are formulated through a backwards mapping process, requiring those involved to think first about the long-term goal, then the intermediate outcomes, outputs and activities required to cause the desired change.

Obviously this kind of top-level conceptual framework is useful for thinking broadly about an organisation’s goals, but how can it be used to demonstrate impact?

In practice this will be very specific to each organisation, but essentially ToCs can help you to demonstrate outcomes because they guide you to the evidence you need to collect to assess whether a project has made a difference.

For example, the ‘outcomes’ stage of the ToC encourages you to ask yourself “in what ways can we show that users have benefitted from our work/research” and thus throws up a plethora of qualitative and quantitative methods you could use to answer this question:

• Pre and post assessments of users e.g. workshop feedback forms, case studies and qualitative research
• Direct observation of behaviour i.e. is our work cited in global non-academic meetings/presentations/publications?

By collating evidence for each stage of the change process, you are able to show whether these results were consistent with the expected theory and, if there is positive evidence against each element of your ToC, then you have the basis for a narrative that the research has impacted.

Tools abound

ToCs are just one of many tools that can be used to plan, track and describe the non-academic impact of research. In fact, at SHARE we use our ToC to guide the formulation of ‘outcome mapping’ (OM), the topic of the workshop’s second presentation delivered by Jessica MacKenzie of ODI. At ODI they use a Rapid Outcome Mapping Approach (ROMA) to track impact. It captures:

• The key stakeholders one wishes one’s research to influence
• The things one wishes these stakeholders will do differently based on that research
• Progress makers: simple indicators that help you measure the achievement of these aims
• The activities one will undertake to achieve these

Next we heard from Natalia Madjarevic of Altmetric, a London-based start-up, about a new tool they’ve developed to track and analyse the online activity around scholarly literature. Intended to be complementary to traditional citation-based analysis of research impact, it looks at alternative metrics (hence the name Altmetric) such as news stories, social media, blogs, post-publication peer review (e.g. Publons, PubPeer), online media (e.g. Wikipedia, Youtube, Reddit) and policy documents to ascertain the real-time, attention/societal impact of a research output.

Finally we heard from two LSHTM staff members about their efforts to describe the impact of research. First up was Shirine Voller of IDEAS who showcased the Research Impact Framework IDEAS uses to describe the impact of maternal and newborn health grants funded by the Bill & Melinda Gates Foundation. This interview-driven framework succinctly describes the research-related, policy, service (health/intersectoral) and societal impacts of those grants and the mechanisms through which these impacts were achieved.

Katie Steels of the LSHTM Press Office closed the workshop by summarising the how the media, and the LSHTM Press Office in particular, can be used to maximise the impact of research. She gave some excellent examples of media coverage garnered thanks to the Press Office’s proactive and reactive efforts on research ranging from the genetics of mosquito bites to the discovery of Ebola.

What could we do better?

Having heard about just a few of the tools available for demonstrating impact, the floor was opened for an interactive brainstorming session on how LSHTM could help support efforts to secure and demonstrate research impact. We will shortly be submitting a list of recommendations to the School and, in the meanwhile, I look forward to trying out some of these exciting news tools! 

Videos and presentations of these tools will be available soon, so watch this space!