Aims and objectives
Introduction
How do we know that the Open Government Pioneers UK initiative is making a difference?
- The evaluation of the core activities funded by the Big Lottery Fund UK was delivered in partnership with an external consultant with expertise in this area selected by the Project board.
- The evaluation of the equality activities involved an equality impact assessment and further consultancy budget for specific equality expertise was allocated for this.
We want the learn from the mistakes we make and build on the successes we create. Is there evidence that a diverse range of citizens and civil society are any closer to engaging the decisions that affect them as a result of the Open Government Pioneers Project UK?
Objective
Track progress towards the following four big outcomes:
- Empowered engaged citizens, leveraging their knowledge of SDGs.
- Political representatives have greater awareness of SDGs and types of actions required.
- Government officials have a better plan for involving citizens.
- Open government seen as instrumental, and a common good towards progressing SDGs.
Overall approach
- To use an open and agile approach to monitoring, learning and evaluation
- To assess how successful the project is at opening up government progress against the Sustainable Development Goals to citizen participation
- To inform future activities that support Open Government approaches to delivering the Sustainable Development Goals
Beneficiary involvement
Beneficiary | Involvement | Sharing learning |
Influencers - politicians, government officials, institutional partners | As critical friends, advisors on positioning of the initiative and its outputs | Learning shared via briefings and presentations |
Participants - civil society activists, engaged citizens | Directly involved in shaping the project's design, analysis and presentation from the start (using the Wiki and other digital platforms) | Learning shared via collaborative Wiki and through Forums, including Forum polling |
General Public | e-petitions, social media, comments on new sites | Learning shared via mainstream media reflective articles |
How the project has been evaluated
STREAM | INDICATORS | EVIDENCE | COLLECTION METHOD | RESPONSIBILITY | WHEN REPORTED |
Awareness (Capacity) | Articulation of issues | Events, blogs, forum posts, wiki posts, website updates | Analytics & narrative report | All | Bi-annually |
Publicity & Promotion | Tweets, articles, media coverage | Analytics & narrative report | All | Bi-annually | |
Sharing (Learning) | New approaches & tools used | Itemisation of new tools & methods used | Narrative report | All | Bi-annually |
Methods & approaches shared with project partners and other stakeholders | Details of new tools & methods shared | Narrative report | All | Bi-annually | |
Participation (Engagement) | New participants/members | Forum registrations, twitter followers | Analytics & narrative report | All | Bi-annually |
New collaborations | Number of new partners & collaborators | Analytics & narrative report | All | Bi-annually | |
Digital platform interaction | Number of website/platform visits & interactions | Analytics | All | Bi-annually | |
Event & webinar attendees | Number & demographic | Analytics | All | Bi-annually | |
Influence (Outcomes/Impact) | Progress with SDGs | Implementation plan | Narrative report | All | Bi-annually |
Citizen involvement | Participation opportunities created | Narrative report | All | Bi-annually | |
Policy influence | Influence on government policy, for example, the PfG, SDG implementation plan | Analytics & narrative report | All | Bi-annually | |
Transparency & accountability of process | Government utilisation of OG methods in the planning & implementation process | Narrative report | All | Bi-annually |
Agile learning approach
- Initial data collected in the first quarter of the project informed a baseline (against the Data subjects in the table below)
- An assessment was made of any changes when reported (quarterly, monthly or annually as appropriate). Advice from Reference Group partners was sought as appropriate.
- Recommendations were made to the Project Board by the Delivery Group to change project elements based on this assessment.
Data underpinning evaluation
Stream | Data subject | Data | Relevant targets | Collection method | Responsibility | When reported |
Capacity | Opengov forum | Registrations
Posts and replies |
Increase forum registrations from 1000 to 5000 and aim for fivefold increase in each nation | Website usage stats | Each nation lead partner | Quarterly |
Capacity | Opengov pioneers Wiki | Visits
Active contributions Citations |
Tbc after setting baseline | Website analytics
Google analytics |
SCVO | Monthly |
Capacity | Opengov collaboration | Contributions to crowdsourcing platforms
New initiatives between citizens sparked as a result of project activities |
Submissions from 20% of members during lifecycle of project | Crowdsourcing platform analytics
Forum polling |
Involve UK | Quarterly |
Engagement | Opengov website | Blog reads
Google ranking |
tbc | Website analytics
Google analytics |
Involve UK | Quarterly |
Engagement | Positioning of Open Government agenda in delivery of SDGs | Stakeholder analysis | Qualitative | Influencer interviews / meetings | Each nation lead partner | Quarterly |
Engagement | Impact on SDG progress | Policy documents (government) | Qualitative & # of citations | Rapid review and appraisal | Consultant | Annually |
Learning | Knowledge of what works in using Open Government to progress SDGs | Blogs and articles from members | Submissions from 20% of members during lifecycle of project | Media analysis | Each nation lead partner | Annually |
Learning | Engagement between Pioneer civil society on what works | Pioneer forum engagement
Posts and replies Forum polls |
Tbc after setting baseline | Website usage stats | SCVO & Involve UK | Quarterly |
Learning | Engagement between UK nations on what works | Stakeholder analysis | Qualitative | Influencer interviews / meetings | NIEL | Annually |
Independent evaluator
We needed to ensure that the project success was properly and independently assessed by an independent evaluator. This was for the following reasons:
- The use of an independent evaluation consultant was agreed by project core partners and specified in the bid to the Big Lottery Fund UK. There would be an expectation of this by all partners who have reviewed the application and the plans which was published on the project Wiki at that time
- An independent evaluation would allow a more objective evaluation of the success of the project than project core partners interviewing or getting feedback themselves from project beneficiaries
- A single independent evaluation rather than separate evaluations in each UK home nation would allow us to stay within budget and capture differences and common experiences across the nations.
- We used the process of recruiting the independent evaluator to generate a conversation between existing and prospective delivery partners on our evaluation approach. In particular, we sought advice on how to get a clearer line of sight between the project output delivery and progress towards outcomes
We openly advertised and recruited an evaluation partner (consultant) to deliver the independent evaluation and the equalities evaluation. The project's Delivery Group's recommendation to the project board was to use a single evaluator for both the main and equalities evaluation of the project. This recognised the importance of embedding equalities in the overall evaluation approach. It also fit into the outcome Empowered engaged citizens, leveraging their knowledge of SDGs, emphasising the focus on engaging people whose voices are least heard.
In order to inform the process used to recruit the evaluation consultant, we drew on the pilot approach we used to recruit the equalities trainer for the project team. This involved an open approach, where potential delivery partners self-identified their interest on this Wiki on the Partners page. They were subsequently added to the Opengovernment Pioneers Project Partners forum and were encouraged to comment on the overall process and brief openly to other partners before making their pitch.
Access the evaluator tender brief here.
The evaluation will involved interviews with project partners, project participants and government, and was supported by the data collected as part of the project's monitoring activities and a discussion with the project board and input from project's forums around evaluation priorities. The evaluator was expected to make recommendations about how to continue and develop the engagement of citizens and civil society in using open government to secure progress towards SDGs.
Mutualgain was appointed as the independent evaluator. See Partners page for more about them.
Evaluation Outputs
You can access all the outputs from the evaluation activities at this webpage - www.mutualgain.org/ogpeval
You can access the full draft of Mutualgain's independent evaluation report (Nov 2018) here. It is in PDF format.
Self-evaluation
It is important that this project does not rely completely on the independent evaluation to understand the difference it is making.
Therefore:
- The UK Conference included a workshop component to facilitate stakeholder analysis, where project participants collectively deliberated around what works in delivering open government approaches to SDGs.
- The Activity sprint on international learning in Summer 2018 also provided international level learning about what works and external perspectives on how the project is making a difference based on International webinars
- Crucially, the independent evaluation was accompanied by six monthly self-evaluations by each core partner for each home nation which was captured and reported through this Wiki.
Timescales for evaluation
Milestone | Timescale | Status |
---|---|---|
Evaluation plan on Wiki
Delivery Group Self-evaluation |
May 17 | Completed |
Meeting with Project Board
Paper outlining equalities impact proofing project activities |
Jun 17 | Completed |
Recruitment process agreed | Aug 17 | Completed |
Recruitment of evaluators
Delivery Group Self-evaluation |
Dec / Jan 18 | Completed |
Evaluator session with Project Board | Jan 18 | Completed |
Evaluation delivery
Delivery Group Self-evaluation |
May 18 | Completed |
Evaluation report and recommendations | Jul 18 | Completed - see here |
UK wide sprint to review project and legacy | Sep 18 | Completed - see here |
How did we ensure a strong equalities focus?
This Open Government Pioneers Project was primarily about raising awareness and supporting the implementation of the SDGs through Open Government principles and methods, with an emphasis on reaching groups who have difficulty in having their voices heard.
We used the following approaches to ensure an equalities focus is embedded at all stage and all activities of the project:
- Prioritising budget and engagement with partners and groups supporting those without a strong voice in opengovernment and SDG discussions
- Securing strong buy-in to this project from a variety of equalities-focused civil society organisations
- Ensuring all project activities seek wider engagement than the 'usual suspects' - e.g. using livestreaming, non-jargon language, accessibility checks for digital platforms.
- Ensuring all project activities are as open and accessible as possible at the design stage to people who face additional barriers to engaging the decisions that affect them.
- We will learn from the existing evidence base available to us, starting with the research and reports referenced at Analysis and theory of change#Who participates.3F
Project Impact
Updated March 2019
Overall impact assessment
- The project has facilitated a significant renewal of interest in the Sustainable Development Goals which had been flagging since the launch of the SDGs in late 2015, and has been integral to other SDG initiatives such as the UKSSD report.
- The project has substantially increased interest and visibility in the Open Government Partnership and action plans amongst the policy communities in Scotland and Wales, and has begun to provide visibility to wider audiences, including as a global exemplar on the international opengoverment and SDG stage.
- The project has secured a greater appreciation by government and civil society participants across the UK and internationally and has engaged a broader constituency of interests in the open government partnership of the link between OGP and SDGs in Northern Ireland and England.
- The project is beginning to broaden awareness and interest in OGP and SDGs in the broader public and media. There has been a very positive Open Government Partnership international evaluator's report on the project's structures and engagements, and recommendations around governance which have now been secured for the future.
- The project has brought both engagement and leadership around open government and SDGs to young people, and has pioneered engagement with people from minority and disability groups.
Impact against outcomes
NB. Text in italics is due for an update.
Outcomes | Beneficiary | Impact assessment | Areas for improvement |
Political representatives have greater awareness of SDGs and types of actions required. | Influencers - politicians, government officials, institutional partners | UK level:
Scotland:
Northern Ireland:
Wales:
|
UK: Continued need to push the SDG up on government priorities (although some success as they are now working on National Voluntary Review to the UN). We also need to ensure the proposals for reform are incoporated in to long-term government action.
NI: political literacy around the SDGs is still low. Continue to raise understanding through the Pioneer conference and include OG Implementation Group in deliberative citizen engagement training. |
Government officials have a better plan for involving citizens. | Participants - civil society activists, engaged citizens | UK Opengov networks:
Scotland:
Northern Ireland:
Wales:
England:
|
In Scotland, we need to explore how to improve enthusiasm in engaging government in partnership on opening up government.
NI: Continue to push for the OG commitments to be progressed by the NICS Board. This will be difficult. Wales: It will be important to engage civil society from the outset of developing themes and commitments rather than just consulting them. There is a risk of this at the moment, and our project will be key to making sure this happens. In UK/England, we need to ensure that the projects that form part of the Innovations in Democracy Fund are successful and impactful. |
Empowered engaged citizens, leveraging their knowledge of SDGs. | General Public | Scotland:
Northern Ireland:
Wales:
England/UK:
|
Scotland: We will need a better plan in place to support people who are interested in hosting meetups, as simply providing more staff time will exhaust team's resources.
NI: Encouraging teachers to adopt our framework/methodology to run their own democracy games sessions in schools, so the approach can be sustained after the lifetime of the project. Adjust the methodology for 16-24 year olds. Wales: There is a need to share the learning from these projects with Welsh Government and other relevant public bodies. |
Open government seen as instrumental towards progressing SDGs. | Media, academia, funding bodies, policy professionals, charities, private sector, international networks | UK wide
Scotland
Northern Ireland:
Wales:
England:
|
A new plan may be needed to bring back the numbers of people engaging with the network through the forums. While the project has had success in building more interest in open government approaches to SDGs, it is clear that breaking into the mainstream media remains a challenge. |
Engagement statistics
Stats are cumulative totals, unless otherwise specified
Indicator | Commentary (updated Jan 2018) | Nov 2016 | Mar
2017 |
May
2017 |
Dec 2017 | July 2018 | End Feb 2019 |
---|---|---|---|---|---|---|---|
Pioneers project partners (organisations registered on forum as Pioneers Project partners) | Most organisations have one individual registered. Delivery partner organisations have 2-3 people registered. See https://forum.opengovernment.org.uk/groups/opengovpioneers | 0 | 55 | 57 | 87 | 103 | |
Open Government Forum registrations | We have seen a doubling in forum memberships for Scotland and Wales, and a notable increase for both UK and NI forums. | All: 1149
Scot: 121 Wales: 29 NI: 299 |
All: 1310
UK: 402 Scot: 160 Wales: 34 NI: 326 |
Wales: 36 |
All: 1595
UK: 428 Scot: 260 Wales: 63 NI: 378 |
OLD FORUM:
All: 1749 UK: 453 Scot: 316 Wales: 67 NI: 396 NEW FORUM: All: 191 UK: 63 Scot: 45 Wales: 20 NI: 17 |
NEW FORUM:
All: 378 UK: 119 Scot: 97 Wales:39 NI: 48 |
Open Government Forum conversations | Activity around posting and replies are in line with numbers of registrations. | Wales: 12 | UK: 411
Scot: 164 Wales: 28 NI: 185 |
OLD FORUM:
UK: 455 Scot: 229 Wales: 44 NI: 193 NEW FORUM: UK: 33 Scot: 15 Wales: 4 NI: 3 |
NEW FORUM:
UK: 128 Scot: 59 Wales: 11 NI: 7 | ||
Wiki visits | Unable to track visits due to Miraheze platform restrictions - unlikely this will change soon. | ||||||
Wiki contributions | Registration not required to contribute anonymously. Therefore we anticipate those that choose to register plan to contribute regularly or on multiple occasions. | Wales: 1 | 32 contributors signed up
888 page edits |
50 contributors signed up
1372 page edits |
75 contributors signed up
1,849 page edits |
||
Wiki citations (currently active) | Difficult to track this properly due to limitations of platform. May need to review this indicator. Currently using Google links | 150 | 247 | 230 | |||
Digital platform unique users (globalgoals.scot) | N/A | N/A | 167 | 1296 | |||
Digital platform page views (globalgoals.scot) | Platform has been soft launched and made available to those taking an interest in SDGs. Platform has not been launched yet formally through mainstream media. | N/A | 147 users, 741 including returning | 761 users, 1663 including returning | 4766 users, 12,618 including returning | 7047 users
32004 visits |
|
Digital platform contributions | *Contributions largely posted by SCVO for now | 120* | 115 published | ||||
Open Government Network blog reads | analytics> behaviour>content drilldown> /2016/12/ + /2017/ + /2018/ + /2019/ (unique page views) stats are cumulative from end Nov 16 | 648 | 1002 | 2775 | 5854 | 8202 | |
Open Government Network blog/article contributions | statistics are cumulative starting end of Nov 16 (latest stats are until 03/08/2018) | Wales: 1
NI:1 Scot: 2 UK: 5 Mixed: 4 |
Wales: 1
NI: 4 Scot: 3 UK: 9 Mixed: 4 |
Wales: 1
NI: 12 Scot:22 UK: 14 Mixed: 6 |
Wales: 5
NI: 24 Scot: 48 UK: 32 Mixed: 10 |
Wales: 8
NI: 28 Scot: 50 UK: 59 Mixed:10 | |
Meetup.com registrations | Trebling of meetup.com registrations which comprises mainly people not already members of the opengov forums. i.e. engaged citizens. | UK (Scot and Wales): 163 | UK (Scot and Wales): 200 | UK (Scot and Wales): 331 | UK (Scot and Wales): 397 | ||
Twitter followers | Substantial increases for Wales and Scotland as these are the new networks. | Scot: 175
Wales: 104 |
Wales: 143
Scot: 231 |
UK: 1997
Wales: 346 Scot: 538 NI: 1320 |
UK: 2246
Wales: 469 Scot: 816 NI: 1470 |
UK: 2419
Wales:531 Scot: 978 NI: 1627 |
Project Legacy
This section summarises and assess legacy options to build on the project outputs.
Legacy planning suggestions
If there is a more detailed concept note available, then the Proposals title will be hyperlinked to it.
Proposals | What? | Why? | How? | When? |
---|---|---|---|---|
Equalities co-production initiative | Supporting a partnership of Equalities organisations across the UK to host an open conversation around engaging people furthest from decision making into plans around SDGs and Open Government. | The Opengovpioneers project opened up engagement with equalities groups, but found a much more intensive capacity building exercise is needed to engage those furthest from the decisions that effect them in using opengov as a way to progress SDGs that affect them. | A consortium of equalities orgs led by a key equality network in each UK home nation, back by an online wiki to document and plan progress.
There is an opportunity for the opengovpioneers project to host an initial engagement event, that can be used to develop a proposal to BIG UK. |
Early 2019 |
UNHS Advocate Programme | A platform for collaboration to contribute towards the SDGs. | To give businesses, social enterprise, individuals, and non-government organisations the opportunity to work together to contribute to the SDGs. | By providing an online platform that provides opportunities for each user to contribute to the SDGs.
Upon completion of specific milestones, users receive recognition. |
Currently in development.
All 4 user groups (businesses, social enterprise, individuals, and non-government organisations) can sign up now to be notified of its launch date. https://www.unhscotland.org.uk/advocate-program |
Focus on Equalities - Food Poverty | Food poverty is a growing issue leading to (and resulting from) a range of inequalities. Aim is to support partnership of groups which will progress activity around this and related SDGs (particularly 1,2,3). Develop partnership approach with other cities to progress. | Pioneer project has been good at reaching equalities groups, but there is an argument for taking a more focused approach, and targeting specific issues or SDGs where we feel we can have impact. One such area is food poverty. | Linked to proposal around equalities co-production (above). There is a local food poverty equalities event planned for Autumn, which would bring a range of groups together. Belfast Food Network is the key co-ordinating body - opportunity to link with Nourish Scotland. | Early 2019 |
Open Glasgow initiative | Supporting Glasgow City Council and Glasgow civil society to consider whether to apply for the OGP local programme next year. | Glasgow is in a good place politically, technologically and socially to consider an Open Government Glasgow action plan. | Initial conversations led by GCC and GCVS with SCVO support, documented openly at this wiki. | Late 2019 |
Open Belfast Pioneer City | Belfast to be a Pioneer City championing open government at local level in NI | Belfast has been transformed over the past 2 decades, and the City Council has just developed a new Community Plan - the Belfast Agenda. | Potential for civil society to support BCC in development of OGP Action Plan; progress range of civic engagement methods to strengthen new plan; opportunities in connecting cities such as Glasgow/Belfast. | 2019 |
Securing the devolved civil society opengov networks | Independent civil society-led networks in Scotland, Northern Ireland and Wales, building on the infrastructure set up by the opengovpioneers project, to maintain the network, forums and engagement with Government. | The BIG lottery funded Opengovpioneers project was designed to build the infrastructure, visibility and engagement with Open Government and using this to advance the SDGs. However, the BIG lottery was an accelerator project, and not designed for maintenance of the network. Funding ends 2018, therefore a new plan and delivery model and funding partner(s) is required to secure a sustainable network. | As part of the legacy work, the Opengovpioneers project team will host a Mini-Summit with project partners on opportunities and interest in leading the next phase of the opengov movement. | Late 2018 |
Gendered approach to securing the SDGs across the UK | Partnership of Open Government pioneer civil society organisations across the UK Home nations using a gendered approach to open government as a means to securing SDGs | We suspect that Gender equality has become one of the single most important factors holding back progress on the SDGs, but Opengovpioneers ddn't take a gendered lens to engagement in the project. Addressing this should help speed progress to achieving the SDGs. | A project led by existing core partners of the opengovpioneers project to advance a gender equalities based approach to the SDGs using open government approaches. Linking into existing civil society networks and groups that focus on gender. | Late 2019 |
Open Government 'beyond the usual suspects' | Partnership of Open Government civil society organisations across the UK using new methods to engage a wider group of people on Open Government and the next Open Government Action Plans for each of the nations. | The key principles of Open Government of transparency, accountability and participation could be game changing for a much wider representation of the public, and yet to date we have had limited engagement with 'harder to reach' groups. This project would have a particular emphasis on engaging more diverse groups of people in the project and development of the next plans, and would follow on from the issues identified in the Welsh network's equality projects. | A project led by existing partners, but working directly with groups that specifically represent groups of people that haven't been engaged to date. | 2019
Phase one report now available |
Integrate Open Government into Welsh Government’s Third Sector Scheme | Aim to significantly increase third sector involvement in Welsh Government policy and practice, particularly amongst equality groups, by introducing open government approaches to existing mechanisms for engagement between Welsh Government and the third sector. | Devolution introduced a statutory commitment for Welsh Government to work with the third sector, ground-breaking at the time. Through this, Welsh Government has set out ambitions to increase engagement. This includes regular dialogue between government and 25 pan-Wales third sector networks, that include equality groups. However, the approaches have become outdated. They don’t reflect new commitments for involvement in the SDGs or Wellbeing of Future Generations Act. It’s failed to use digital effectively. Using open government approaches could significantly increase meaningful participation and engagement. | The third sector networks are already demanding change and are clear they want to open up the process. The Wellbeing of Future Generations Act is a strong incentive for government. The project would draw on the work done to date, the existing mechanisms and networks and energy for change inside and outside government. The Open Government approaches that we’ve learnt to date, continued engagement with the Open Government Partnership, partners across the UK and the Open Government Network in Wales would support us to find practical solutions to achieving this. | 2019 (begin early 2019) |
Enlightenment 2.0 | Scottish initiative to rethink progress - what it's all about and how to make it happen, inclusively | The original Scottish enlightenment was recognised as a paradigm shift in the way people thought about progress. We hope to do the same again given the rise of populism and global regression. Enlightenment 1.0 was focused on educated elites. Enlightenment 2.0 should be based on human rights, responsibility to the planet and inclusion. | Network, wiki page, strong cross-sector collaboration focused on individuals rather than organisations | From Dec 2018 |