Experiments and trials
An experiment is a way of trying something new while putting in place the necessary structures to find out if it works
We often don’t know how innovations will work out so an experiment allows us to test new ideas and, based on what we learn, develop them further. Nesta works with governments around the world to help them to experiment more effectively.
Experiments and trials as an innovation method
An experiment is a way of trying something new while putting in place the necessary structures to find out if it works.
We often don’t know how innovations will work out so an experiment allows us to test new ideas and, based on what we learn, develop them further. Experimenting also means that any solution can be treated as a work in progress, which tweaking and tinkering can improve.
There are several ways to experiment, depending on the context. Sometimes it can involve working closely with the people who will use the end solution to see how an innovation works in real-life. Other times, it can involve using more robust evaluation methods - such as Randomised Controlled Trials (RCTs) - to test an idea and create evidence to support it.
As an innovation method, experimenting on a small scale and treating it as a repetitive process allows people and organisations to explore new solutions without wasting time and resources on initiatives that do not work.
Our work on experiments and trials
Nesta was an early promoter of experiments as an innovation method.
In 2011, we published ‘State of Uncertainty’, a report calling for a more experimental approach to be taken in innovation policymaking. The idea was that innovation policy would work better if it were modelled on experimental science and focused on minimising the uncertainty entrepreneurs face.
Then, in 2013, Nesta led on one of the first Randomised Controlled Trials (RCTs) to assess whether a popular business support scheme - Creative Credits - worked effectively. The resulting report revealed important insights about the policy that normal evaluation methods used by government would not typically show.
In 2014, a new joint venture was created through a partnership between the Cabinet Office, Nesta and employees of the Behavioural Insight Team, also known as the government’s nudge unit. The Behavioural Insights Team focuses on carrying out experiments to better inform public policy.
In the same year, Nesta launched the Innovation Growth Lab (IGL). This global collaboration of governments, foundations and researchers develops and tests different approaches to support innovation, entrepreneurship and growth. It aims to make policy in this area more experimental and evidence-based, while also building the capacity of people and institutions to do their own RCTs.
Since it was founded, IGL has funded over 30 RCTs in innovation, entrepreneurship and business growth, and partnered with more than a dozen organisations to promote the use of RCTs in this field.
IGL also hosts a global conference each year, bringing together senior policymakers, practitioners and researchers from across the globe to explore future innovation and entrepreneurship policies.
In parallel with this work, Nesta’s Innovation Skills team continues to do a lot of work to promote an experimental culture at government level. Most recently, it initiated ‘States of Change’ - a programme working with progressive governments and public innovation practitioners around the world to strengthen their capacity to experiment.
In 2017, the IGL Experimentation Toolkit was created. This toolkit is designed to build an understanding of how adopting an experimental approach can be used to make policies more effective by successfully designing and delivering trials. As part of its ongoing work, Nesta continues to update and customise the toolkit.
Creative Credits: A randomized controlled industrial policy experiment is a Nesta study that used an RCT to see if a popular business support scheme called ‘Creative Credits’ worked effectively.
The pilot study, which began in Manchester in 2009, was structured so that vouchers, or 'Creative Credits', would be randomly allocated to small and medium-sized businesses applying to invest in creative projects such as developing websites, video production and creative marketing campaigns, to see if they had a real effect on innovation.
The research found that the firms who were awarded Creative Credits enjoyed a short-term boost in their innovation and sales growth in the six months following completion of their creative projects. However, the positive effects were not sustained, and after 12 months there was no longer a statistically significant difference between the groups that received the credits and those that did not.
Nesta published a report on the study, which argued that these results would have remained hidden using the normal evaluation methods used by government, and called for RCTs to be used more widely when evaluating policies to support business growth.
Nesta tools and resources
Other useful reading material
- State of Uncertainty: Innovation policy through experimentation (2011)
- Creative Credits report (2013)
- Better Public Services Through Experimental Government (2015)
- States of Change