Household surveys shape policy investment
A case study of the Living Standards Measurement Survey
From TReNDS’ and GPSDD’s series on the Value of Data
Overview
Household surveys are a powerful analytical tool that can shed light on how households interact with services and how interventions affect their wellbeing. Household surveys are therefore particularly relevant for policy analysis and guiding government investments. The Living Standards Measurement Survey (LSMS), housed within the Development Data Group of the World Bank, is one of the largest and longest-running household survey programs. To date, it has supported more than 100 LSMS-based national surveys. It has had a profound impact on many countries’ policies and investments. In Nicaragua, for example, national-level LSMS data were used to improve the targeting of social programs associated with the Emergency Social Investment Fund. This resulted in the government diverting funds away from regressive sanitation projects and towards progressive education programs in extremely poor communities. While household survey programs have been critiqued for diverting limited national resources away from administrative data systems, they provide an important complement to these and other data sources by shedding light on the behavior and well-being of households to better target resources and services, particularly in environments where other data sources and tools may be weak or under-resourced.
Key insights
The LSMS has supported more than 100 national, LSMS-based surveys.
It costs approximately US$1.7 million per survey per country.
In Tanzania, LSMS-supported household survey data showed that about 80 percent of farmers do not use current extension services, resulting in the government adopting a new approach to local livestock support services.
Written by Jessica Espey (SDSN TReNDS Director) with considerable input from and thanks to Dr. Gero Carletto (Lead Economist and Manager, LSMS Program, World Bank) and Gbemisola Oseni (Senior Economist, LSMS Program, World Bank), to Shaida Badiee and Eric Swanson (SDSN TReNDS members and Directors of Open Data Watch), and Jay Neuner (SDSN Communications Manager).
Table of contents
Context | Description of the Data Solution | Implementation | Costs and Funding | Return on Investment | Ongoing Challenges | Conclusion | References
Context
Government expenditures on social services like health and education, as well as public service infrastructure, represent a huge amount of annual public expenditure. Last available global estimates suggest governments spend as much as 14 percent of total expenditures on education (as of 2014) and 15 percent on healthcare (as of 2011) (World Bank, n.d.). It is not unusual for this to account for 2 to 3 percent of GDP, which can range up to hundreds of millions of dollars. Decisions on how best to allocate these resources are, or should be, based on national data. Administrative data is one such source that helps us understand the quality and governance of public services, as well as users’ interaction with the services. However, this data cannot explain household behavior and how social policies and programs might affect the wellbeing of individuals and households.
Household surveys aim to provide reliable data on demographic and socioeconomic characteristics of a population. Typically, the survey collects data from a national sample of households, randomly selected from a list of households that is derived from the census. These surveys complement information gathered by the census and administrative data by getting a better picture of households’ daily lives and challenges, e.g., understanding income dynamics, dependencies, access to services, and access to food. They also have the advantage of being able to gather information on people outside of formal systems or who are not accessing government services – for example, children out of school.
In countries where administrative data systems are weak, household surveys are particularly important. Not only can they provide detailed information on household wellbeing and behaviors, but they can also help fill gaps in civil registration systems, giving governments a more holistic understanding of their population.
The importance of household surveys for national and international data collection is highlighted by the indicators identified to monitor the global Sustainable Development Goals (SDGs). According to one estimate, surveys – including household, consumption, agricultural, and labor force surveys – will be the key source of information for producing more than 26 percent of SDG indicators (SDSN 2015). Increasing their quality, frequency, and coverage is therefore crucial for tracking progress on the SDG agenda over time. Household surveys can also support this agenda’s objective to “leave no one behind” by providing a better picture of the situation of the poorest and most vulnerable. Although household survey sample sizes are usually limited (United Nations Department of Economic and Social Affairs 2005), the surveys can often be designed to include more consistent stratification variables or to ask specific detailed questions pertinent to our understanding of exclusion. Examples include time-use and disability, both of which are often excluded from national censuses.
Description of the Data Solution
The Living Standards Measurement Study (LSMS) is a global household survey program housed within the Survey Unit of the World Bank’s Development Data Group (World Bank, n.d.). It supports the production of multi-topic household surveys by providing technical assistance to national statistical offices (NSOs). Since its inception, the focus of the LSMS has been “understanding, measuring and monitoring living conditions, the interaction of government spending and programs with household behavior, ex ante and ex post assessments of policies, and the causes of observed social sector outcomes” (Scott, Steele, and Temesgen 2005). LSMS surveys therefore rely on multiple instruments and innovative technologies to obtain data needed for these purposes (World Bank, n.d.).
The LSMS technical team, based in Washington, D.C., collaborates with NSOs worldwide on methodological design of national surveys, survey topics, levels (e.g., household or individual), georeferencing, and use of new technologies such as computer-assisted platforms and sensors for direct measurement. Specific instruments used to support country-level LSMS implementation include:
models of household questionnaires for collecting information at the household and individual levels, as well as at the level of household economic activities (such as agriculture and home businesses);
community questionnaires for collecting data on the environments in which households function, with a focus on the available services, economic activities, access to markets, and social capital;
price questionnaires administered in each area where households are located, to support cost-of-living adjustments; and
facility questionnaires, administered to local service providers to obtain information on the types and quality of services available to households (Scott, Steele, and Temesgen 2005).
The team also conducts methodological tests to ensure the quality and robustness of different countries’ household survey methods. For example, they have completed a range of consumption, income, and labor experiments to verify LSMS results (World Bank, n.d.). In 2018, the U.K. Department for International Development provided the LSMS team with a grant to conduct methodological validation work in the area of agricultural productivity (World Bank, n.d.).
Implementation
The first LSMS surveys were piloted in Côte d’Ivoire and Peru in 1985. These two surveys were pilots testing the methodology to determine the usefulness and quality of the data that could be obtained (Scott, Steele, and Temesgen 2005). They proved so successful that LSMS has since gone on to support more than 100 LSMS surveys in more than 100 countries.
LSMS surveys are typically a collaborative effort among the World Bank LSMS technical unit, the host government (usually led by the NSO), the principle users of the data from across government, and bilateral and multilateral donor organizations. Each effort starts from the same core concepts and technical advisory documents provided by the World Bank team and is thereafter highly customized to suit the needs of the country in question. The NSO–working with the LSMS technical team–takes the lead in designing the questionnaire, the sample, and the fieldwork methodology “using the techniques found by the LSMS to be most effective” (World Bank, n.d.).
As a result of this customization, LSMS surveys are highly varied; for example, Scott et al., highlight the case of Bosnia and Herzegovina where, in 2001, the health module was expanded to incorporate questions on depression in an attempt to identify links between mental health and other aspects of welfare and labor market participation (Scott, Steele, and Temesgen 2005). Another example is Guatemala, where a module on social capital was added to its 2000 survey to collect information on the social dimensions of poverty, “such as participation in community/government programmes and collective actions, causes of exclusion in the society, perceptions of welfare, and perceptions of, and access to, justice” (World Bank, n.d.).
Costs and Funding
In 2015, the Sustainable Development Solutions Network (SDSN), PARIS21, Open Data Watch, and a consortium of development data specialists produced an estimate of the resources required to produce data and statistics to monitor the SDGs. As part of this, they consulted with LSMS and other household survey groups to establish the average costs of conducting household surveys in each country. An estimate provided by the LSMS team at the time of publication indicated that a standard LSMS survey costs approximately US$1.73 million per country, with operations accounting for approximately US$1.2 million and field support accounting for US$500,000. The costs were broadly comparable to ICF International’s Demographic and Health Surveys, which gather health and population data from developing countries; the total average cost for these surveys is US$1.6 million (ICF International, n.d.; SDSN 2015). Another estimate of the costs of LSMS surveys was produced in 2005 by Scott et al.
They concluded that “LSMS survey costs range from US$400,000 to US$1.5 million, depending on the country and the year. On a per- household basis, this is commensurate with other complex surveys such as Income and Expenditure Surveys and Demographic and Health Surveys” (Scott, Steele, and Temesgen 2005). Considering the 10-year time difference between estimates, the findings are broadly comparable.
In both studies, “operations” or “salaries” are found to be the largest outlay, including the salaries of interviewers, supervisors, data entry operators, anthropometrists, and drivers. According to LSMS staff, the field staff teams are large relative to the sample size, “owing to the high supervisor-to-interviewer ratios (typically 1 to 3), the size of the questionnaire and the use of direct informants which limits the number of households that can be visited per day, the inclusion of data entry in the field teams, and the provision of transport to each team member to ensure the mobility and integrity of the team by providing each with transport” (Scott, Steele, and Temesgen 2005). The majority of LSMS surveys are funded by a variety of sources, including government budgets (from the NSO or from other ministries), bilateral donations, and multilateral donations and credits.
The contributions of each party vary substantially by country, given the heterogeneity of each survey with some surveys covering 2,000 households, others covering 4,000; some countries only pursuing one survey, while others undertake panel sets; and each with specific modules tailored to suit individual country needs (Scott, Steele, and Temesgen 2005; Grosh and Glewwe 1998; World Bank, n.d.).
The value that governments ascribe to LSMS-based surveys is highlighted by the number of countries that have conducted two or more of such surveys: Of the 32 countries that have made their LSMS- based survey microdata available via the World Bank, 59 percent have conducted two or more surveys (including Indonesia, which has conducted and shared microdata for 19 surveys), showing the value that governments place upon these surveys for policymaking (World Bank, n.d.).
Return on Investment
Since the early 1990s, there have been cost-benefit analyses of LSMS surveys. For example, Grosh (1991) highlighted the impact that the LSMS survey in Jamaica had upon nutrition and food policy (Grosh 1991). Through the LSMS survey results, the government was able to quantify the benefits poor households received from major nutrition programs and showed that the food subsidy, in particular, was highly regressive. As a result of this analysis, the government decided to eliminate food subsides. Scott et al. (2005) cite another example from Nicaragua, where national-level LSMS survey data was used to improve the targeting of social programs associated with the Emergency Social Investment Fund (Scott, Steele, and Temesgen 2005; World Bank 2000). The World Bank-sponsored study showed that sewerage projects were highly regressive, while latrines and primary education projects were progressive, reaching the 17 percent of the population classified as extremely poor.
Other studies (not specific to the LSMS program) have shown the immense cost-benefit of household surveys for the effective design of policies and interventions. For example, Shortall (2009) showed how an inexpensive household survey (of US$11,000, or US$7 per household reached) fundamentally shaped a large-scale financial services program run by ShoreBank International, the United States Agency for International Development (USAID), and in-country partners in the disaster-affected area of Pakistan’s Azad Jammu and Kashmir region following a 2005 earthquake (USAID and ShoreBank 2008).
In an attempt to assess and quantify their impact, the LSMS team itself has conducted evaluations of their policy relevance, and cost effectiveness in a number of countries, including Uganda, Tanzania, Malawi, Ethiopia, and Nigeria. The examples cited below were provided by the 2018 LSMS team (Carletto and Oseni 2018).
Improving Access to Livestock Services
In both Uganda and Tanzania, LSMS-supported national surveys have been crucial in showing governments that limited access to livestock services is one of the binding constraints that prevent farmers from making productive use of their animals.
In Uganda, the Uganda National Panel Survey (UNPS) was the only statistically accurate source of information that could systematically identify the challenges faced by livestock farmers. As a consequence, the Agricultural Sector Strategic Plan 2015/16-2019/20 proposed to establish a “single spine agricultural extension system” to “increase farmer access to relevant information, knowledge and technology through effective, efficient, sustainable and decentralized extension services” (FAO 2015). The Ministry of Agriculture, Animal Industry, and Fisheries is collaborating with the Makerere University Business School to identify potential public-private partnerships to operationalize the single spine agricultural extension system.
In Tanzania, the National Panel Survey (TZNPS) has been used by the relevant ministries in collaboration with the NSO to show how livestock services affect farmers’ productivity. In particular, the TZNPS analysis showed that about 80 percent of farmers do not use extension services (FAO 2015). The TZNPS data were the only statistically accurate source of information that provided this evidence in the country. As a result of this finding, the Ministry of Livestock and Fisheries Development has invested resources in identifying effective options to improve the system of livestock extension (United Republic of Tanzania Ministry of Livestock and Fisheries Development 2015). They conducted, with support from the Food and Agriculture Organization of the United Nations (FAO), a randomized control trial to look at the impact of cost recovery mechanisms for improving farmer access to livestock extension services. The objective of this work has been to identify budget-neutral policy reforms that improve the “on-the-job” performance and efficiency of extension officers (United Republic of Tanzania Ministry of Livestock and Fisheries Development 2015).
Understanding Extreme Weather Events
The infrastructure created by the LSMS-Integrated Surveys on Agriculture program (LSMS-ISA) is being used to study the impact of extreme weather events in Malawi, including floods in 2014 and 2015 and drought in 2015 and 2016 (McCarthy et al. 2018). The presence of a successful household survey system that could be used in a timely manner has allowed for the study of the actual events, as opposed to simulations. From October 2015 to December 2015, the Malawi NSO– with technical assistance from the LSMS team–conducted the Malawi Flood Impact Assessment. The assessment looked at household coping strategies during the crisis and household risk management methods, providing insight on how best to support households coping with isolated and covariant shocks (McCarthy et al. 2018).
Following these historical floods, large areas of Malawi were then subject to drought conditions in the following 2015-2016 season. The Drought Impact Assessment was developed using available data from the 2010, 2013, and 2016 waves of Malawi’s Integrated Household Panel Survey (Carletto and Oseni 2018). Similar to the flood impact assessment, the analysis was used to inform the government’s response, including the development of the Malawi Drought Post-Disaster Needs Assessment and a recovery plan (World Bank, United Nations, and European Union 2016). The recovery plan aims to increase the productivity and sustainability of agriculture by promoting diversified and climate-smart agriculture and improving irrigation infrastructure (World Bank, United Nations, and European Union 2016).
Access to Safe Water
In 2016, a water quality survey was conducted by Ethiopia’s Ministry of Water, Irrigation and Electricity in collaboration with the LSMS- ISA project and with support from the World Bank, UNICEF, WHO, and WHO/UNICEF’s Joint Monitoring Programme for Water Supply, Sanitation, and Hygiene. The survey, “the first of its kind in the country, was part of the [Ethiopia Socioeconomic Survey or] ESS, which is itself a collaborative project of the [Central Statistics Agency of Ethiopia or] CSA and the World Bank team for the Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMS- ISA). ESS is a nationally representative panel survey that began in 2011. In 2016, during the third wave of the panel, the water quality test module was added” (Central Statistical Agency of Ethiopia 2017).
The results showed that only 13 percent of households have access to safely managed sources of drinking water (Central Statistical Agency of Ethiopia 2017). According to the LSMS team, the findings were highlighted by the media, thereby encouraging the government to act. As a result, the government plans to improve water quality though three major initiatives: (i) implementation of water safety plans in 36 water schemes in four regions with support from WHO, (ii) implementation of water safety plans in all UNICEF-supported water supply projects, and (iii) installation of on-site chlorine generation equipment.
Ongoing Challenges
There is extensive literature on the methodological challenges and limitations to using household and expenditure surveys–particularly to measure poverty, which is outside the scope of this paper. Frequently debated issues include whether to measure income or expenditure (Meyer and Sullivan 2003; Grosh and Glewwe 2000); the use of diaries or recall methods (Wutich 2008); or the determination of poverty lines (Vecchi 2008). There is also a question around the international comparability of Household Income and Expenditure Surveys, as they are designed to serve national decision-making and therefore ignore the problem of converting income and expenditure levels in one currency to another currency for the purpose of international comparison. The following section considers three major challenges affecting the ongoing use and expansion of household survey programs such as LSMS.
Disaggregation
An ongoing burden placed upon household surveys is the need for disaggregated data. Although they are intentionally small and intended for analytical purposes (not detailed demographic monitoring, as per a census), NSOs and survey teams are often asked for detailed information on the demographics of surveyed recipients (Chuwa 2016). One major reason for limiting the sample size of household surveys to between 2,000 and 5,000 households is to maintain quality and balance sampling errors. However, as highlighted by Penneck (2007), these small sample sizes can be a problem if they are being used in place of a more expensive large-scale survey but then prove to have limited use for the government (Penneck 2007). This is a particular concern if investment in household surveys is diverting funds away from bolstering administrative data systems that have the potential to be more comprehensive and to gather recurrent data.
Intrahousehold Dynamics, Including Gender and Age
A second challenge to the utility of household surveys is intrahousehold dynamics. Household surveys usually rely on one or two informants per household who report on the status of the other household members. They do not always capture the experiences of other household residents in any detail, particularly women and children, creating inherent gender and age biases. (For more detailed discussion of this profound challenge, see the International Household Survey Network’s (IHSN) July 2015 report “How well are gender issues covered in household surveys and censuses?”). They are also based on the assumption that the household “encompasses a notion of a bounded, largely impermeable, unit,” which anthropology has long critiqued (Randall and Coast 2015).
Access and Use
A fundamental challenge to household survey programs, relates to the accessibility of the underlying data. In recent years there have been considerable improvements in archiving, documenting, and making LSMS surveys accessible thanks to the efforts of the IHSN and its National Data Archive tool, which the network offered to countries to use for documenting and archiving microdata (Badiee 2018). But many data sets are still not open and accessible, making it difficult for researchers and users (including various government policymakers) to access them and tease out policy insights (Badiee 2018). Part of the reason for this inaccessibility is the necessity to safeguard the respondents’ confidentiality, but recent advancements in anonymization techniques could go a long way to help improve access to microdata.
Conclusion
Household surveys are a powerful analytical tool that can help governments to better understand the social and economic conditions experienced by their residents, as well as how they interact with and use services. If carefully designed, they can show the benefits or limitations of one intervention over another, and can therefore help governments to prioritize limited resources. For the duration of the SDG agenda, they will play a vital role in helping to guide policy decisions and investments, particularly for low-capacity countries. However, household surveys cannot replace effective widespread administrative systems that can provide much larger recurrent data samples, as opposed to just showing a representative sample at one static moment in time. Governments and donors deliberating the relative value of different data systems should consider household surveys an essential short-term input. This input needs to be coupled with investments in long-term system-building to ensure all countries have comprehensive data systems that can effectively guide policy and decision-making to achieve sustainable development.
References
Badiee, Shaida. 2018. Interview by Jessica Espey.
Carletto, Gero, and Gbemisola Oseni. 2018. Email Interview by Jessica Espey.
Central Statistical Agency of Ethiopia. 2017. “Drinking Water Quality in Ethiopia: Results from the 2016 Ethiopia Socioeconomic Survey.” https://washdata.org/sites/default/files/ documents/reports/2018-07/Drinking-water-quality-ethiopia-ESS-2016.pdf.
Chuwa, Dr. Albina. 2016. “Household Surveys in the 21st Century: Challenges and Opportunities in an Evolving Environment.” In Global Preparatory Seminar for the United Nations World Data Forum. Guillin, China. https://unstats.un.org/sdgs/files/meetings/ wdf-meeting-01/Panel 4 - 3-Tanzania.pdf.
FAO. 2015. “Livestock Data and Statistics for Evidence-Based Policies: Lessons from Country Experiences.” In African Commission on Agricultural Statistics Twenty-Fourth Session, 1–10. Kigali, Rwanda.
Grosh, Margaret E. 1991. “The Household Survey as a Tool for Policy Change: Lessons from the Jamaican Survey of Living Conditions.” 80. http://documents.worldbank.org/ curated/en/877191468774897996/The-household-survey-as-a-tool-for-policy- change-lessons-from-the-Jamaican-survey-of-living-conditions.
Grosh, Margaret E., and Paul Glewwe. 1998. “Data Watch: The World Bank’s Living Standards Measurement Study Household Surveys.” Journal of Economic Perspectives 12 (1): 187– 96.
Grosh, Margaret E., and Paul Glewwe. 2000. “Designing Household Survey Questionnaires for Developing Countries.” https://ideas.repec.org/b/wbk/wbpubs/25338.html.
ICF International. n.d. “The Demographic and Health Surveys (DHS) Program.” https:// www.icf.com/resources/projects/research-and-evaluation/demographic-and-health- surveys.
McCarthy, Nancy, Talip Kilic, Alejandro de la Fuente, and Joshua M. Brubaker. 2018. “Shelter from the Storm? Household-Level Impacts of, and Responses to, the 2015 Floods in Malawi.” Economics of Disasters and Climate Change, 1–22. https://doi.org/10.1007/ s41885-018-0030-9.
Meyer, Bruce D., and James X. Sullivan. 2003. “Measuring the Well-Being of the Poor Using Income and Consumption.” 9760. https://www.nber.org/papers/w9760.
Penneck, Stephen. 2007. “Using Administrative Data for Statistical Purposes.” In Third International Conference on Establishment Surveys (ICES-III). Montreal, Quebec, Canada. https://ww2.amstat.org/meetings/ices/2007/proceedings/ICES2007-000077.PDF.
Randall, Sara, and Ernestina Coast. 2015. “Poverty in African Households: The Limits of Survey and Census Representations.” The Journal of Development Studies 51 (2): 162– 77. https://www.tandfonline.com/doi/pdf/10.1080/00220388.2014.968135.
Scott, Kinnon, Diane Steele, and Tilahun Temesgen. 2005. “Chapter XXIII: Living Standards Measurement Study Surveys.” In Household Sample Surveys in Developing and Transition Countries, 523–49. UN Statistics Division. https://unstats.un.org/unsd/
hhsurveys/pdf/household_surveys.pdf.
SDSN. 2015. “Data for Development: A Needs Assessment for SDG Monitoring and Statistical Capacity Development.” https://www.sdsntrends.org/research/2015/7/15/data-for- development-a-needs-assessment-for-sdg-monitoring-and-statistical-capacity- development.
United Nations Department of Economic and Social Affairs. 2005. “Designing Household Survey Samples: Practical Guidelines.” https://unstats.un.org/unsd/demographic/ sources/surveys/Handbook23June05.pdf.
United Republic of Tanzania Ministry of Livestock and Fisheries Development. 2015. “Tanzania Livestock Modernization Initiative.” Vol. 33. https://livestocklivelihoodsandhealth. org/wp-content/uploads/2015/07/Tanzania_Livestock_Modernization_Initiative_
July_2015.pdf.
USAID, and ShoreBank. 2008. “Advancing Microfinance for Post-Disaster Economic Reconstruction (AMPER) Project in Azad Jammu and Kashmir: Baseline Household Survey.” https://www.marketlinks.org/library/amper-project-azad-jammu-and- kashmir-baseline-household-survey-2007.
Vecchi, Giovanni. 2008. “Poverty Lines: Principles and Common Practices.” In Serbia National Poverty Analysis Workshop. http://siteresources.worldbank.org/PGLP/Resources/ poverty_lines.pdf.
World Bank. n.d. “Data Catalog.” https://datacatalog.worldbank.org/search/datasets?search_ api_views_fulltext_op=AND&query=LSMS&nid=&sort_by=field_wbddh_modified_ date&f%5B0%5D=type%3Adataset.
World Bank. n.d. “Domestic General Government Health Expenditure (% of General Government Expenditure).” https://data.worldbank.org/indicator/SH.XPD.GHED.GD.ZS
World Bank. n.d. “LSMS.” http://surveys.worldbank.org/lsms.
World Bank. n.d. “LSMS Dataset Finder.” http://iresearch.worldbank.org/lsms/ lsmssurveyFinder.htm.
World Bank. n.d. “Survey Methods.” http://surveys.worldbank.org/lsms/our-work/survey- methods.
World Bank. n.d. “World Bank Open Data.” https://data.worldbank.org/.
World Bank. 2000. “Nicaragua: Ex-Post Impact Evaluation of the Emergency Social Investment Fund (FISE).” http://documents.worldbank.org/curated/en/926901468775574549/ Nicaragua-Ex-post-impact-evaluation-of-the-emergency-social-investment-fund- FISE.
World Bank, United Nations, and European Union. 2016. “Malawi Drought 2015- 2016: Post-Disaster Needs Assessment.” https://openknowledge.worldbank.org/ handle/10986/25781.
Wutich, Amber. 2008. “Estimating Household Water Use: A Comparison of Diary, Prompted Recall, and Free Recall Methods.” Field Methods 21 (1): 49–68.
A joint project of