Getting the Most Out of Big Data for National SDGs Monitoring

Getting the Most Out of Big Data for National SDGs Monitoring

By Cameron Allen

Photo by Jack Moreh on Shockvault

The SDGs present an unprecedented monitoring challenge, and the potential for ‘big data’ to support these efforts has incited considerable enthusiasm. 

‘Big data’ describes large volumes of high velocity, complex, and variable data. This covers commercial, sensor, mobile, and online data from searches and social media. For national statistical systems, big data sources have a strong value proposition, including increased coverage, improved timeliness, and granularity of their statistical products as well as lower production costs and reduced respondent burden. 

There have been major developments over the past decade in how countries use big data for statistical production and decision-making.  Mature applications include web scraping or scanner data for price statistics, social media data for consumer confidence indexes, mobile data for mobility and tourism statistics, and satellite data for agricultural and environmental statistics.

Countries are also experimenting with big data sources to support national SDGs monitoring, including in the Philippines, Colombia, and Ghana, among others.  Many of these initiatives are the result of collaborations between National Statistical Offices (NSOs) and a range of partner organizations, including technical experts, knowledge brokers, private sector organizations, international organizations, and donors.  These initiatives also draw upon innovative methods for harnessing big data for SDGs monitoring that have been pioneered by the global research community.

However, there is a lack of guidance for countries seeking to get the most out of ‘big data partnerships’ that can support their national SDGs monitoring, and limited information sharing between practitioners and experts at the forefront of new innovations.  Our new research led by SDSN TReNDS and funded by Partners for Review aims to address these gaps.  

We start with a comprehensive review of the scientific literature on the latest innovations in harnessing big data for SDGs monitoring.  We review 100 recently published datasets which could support national monitoring of 15 goals, 51 targets and 69 indicators. We find that satellite or Earth Observation data is by far the dominant source of big data being used, and that derived datasets can serve a range of monitoring objectives.  With many new datasets emerging from leading experts, there is a growing need to ensure that they can be integrated with national reporting obligations. Drawing on recent global platforms and scalable applications, we offer some solutions to help bridge this divide.

We also interviewed a range of experts and statisticians to glean their experience in leveraging partnerships to harness big data for SDGs monitoring.  We find that the partnership landscape is dynamic and diverse, with a broad range of partners and working modalities. This goes beyond traditional partners to include local champions, convenors, brokers, regulators, and privacy partners. The maturity of NSOs - in terms of their organizational awareness and literacy, technical and human capabilities, governance, and legal aspects – determines which mix of partner capabilities will be most effective in harnessing big data.

While some NSOs have yet to commence their big data journey, others are quite advanced. We propose a maturity model to explore how big data maturity influences successful partnerships, ranging from nascent through pre-adoption to early implementation, proficiency and maturity.  As NSOs move through these five stages, they require different partners and partnership models to gain greater value from their investments. They also face a ‘chasm’ – requiring concerted commitment and investment to move beyond pilot projects so that big data sources are embedded within their statistical production pipelines. 

The maturity model proposed in our report highlights areas where NSOs and their partners can target joint efforts and provides a set of practical guidance for NSOs seeking to get the most out of successful partnerships. This includes getting big data partnerships off the ground, building a business case and aligning incentives, navigating the governance and regulatory landscape, building technical and human capabilities, and sustaining partnerships.