By Barbara Adams and Karen Judd
Download this briefing (pdf version).
A common theme that ran through the 50th Session of the UN Statistical Commission, March 2019, was the often tense interface between data and policy-making and the asymmetrical power dynamics that shape it. This was evident in the several reports submitted for consideration by the Commission. One from the UN Statistics Division (UNSD) reported on the federated system of data hubs, designed to integrate new data sources into a platform which is accessible to National Statistics Offices (NSOs) and creates comparable data among users. Another was a proposal by the High-Level Group for Partnership, Cooperation, and Capacity Building for the 2030 Agenda for Sustainable Development for a UN Chief Statistician to enhance the voice of statistics in UN policy processes.
The Commission also took stock of the state of indicators to measure progress on the Sustainable Development Goals (SDGs), aware that 2020, one year away, will be the opportunity to reopen the Global Indicator Framework. The indicator framework was accepted on a provisional basis by the Commission. While refinements would continue to be made as the technical work progressed, more serious revisions such as the addition or subtraction of an indicator or indicators will be taken up by the Commission in 2020.
Professor Joseph Stiglitz co-author, with Martine Durand, of a new report called “Going Beyond GDP: Measuring what Counts for Economic and Social Performance”, kicked off the sessions by saying that what is important in assessing progress is to focus first on the goal; before measuring progress on some predetermined targets and indicators, policy-makers at all levels need to ask what kind of society they hope to see. His presentation on economic statistics focused on ways to increase links between economic activity and the outcomes for people and society.
From data producers to data stewards
Propelled by national demand for greater access to and control over Big Data, panelists at the seminar on financing data and statistics side event were asked how National Statistical Offices (NSOs) could be better supported in “their institutional transformation from producers of statistics to data stewards”.
“In an apparent paradigm shift, there is now an overriding consensus that both developed and developing statistical systems should actively pursue and accelerate the integration of new data sources—public and private—in their statistical products and services…It is evident that new sources of data can be harnessed for statistical purposes and to better meet policy makers needs.” [UNSD Summary report]
This raised the need to address the relevance of NSOs, requiring integrity, professionalism, confidentiality and data consistency.
Austria’s director general of statistics Konrad Pesendorfer asked how active a role they should play in the data and policy-making context, noting that some feel that NSOs already drive policy too much.
One challenge for national accounts is how to capture relevant bits of the global value chain. Today’s labour markets also pose a measurement challenge, as there are no longer mostly 9-5 jobs with social security. NSOs need to think regionally, across borders, with understanding of the context as well as statistics and measures. New trends may not be comparable across countries.
Stiglitz, speaking on the policy-evidence interface, was careful to distinguish between policymakers and politics, especially regarding pressure from the latter. He said with regard to chain weighted indices of GDP an independent NSO is essential, emphasizing the need to be immune to political pressure – “data is not information; it needs to be organized” accounting systems do this, so accounting frameworks need to be updated constantly.
“NSOs must report to society as well as governments”, he added. They create a public good which civil society and governments can use for different ends, but they need to resist political pressure. He reminded statisticians that they are not just reporting to government but reporting to citizens. “You do want to influence policy makers but society has created you to serve society. We’ve created a public good so that citizens can agitate and ask what is going on, and governments can respond.”
South Africa’s chief statistician Risenga Maluleke agreed, saying NSOs must not see themselves only as employees but as public servants, placing the user at the center. “The numbers we produce have to have value for human life.” So too did Joseph Zveglich, deputy chief economist of the Asian Development Bank: “We must look at statistics as a public good, available to anyone, at least on a partial basis.”
Maluleke stressed the need to look at user demand for data and statistics not just the supply, particularly the demand for big data when already the private sector is stepping in. He added that they are trying to do this in South Africa by partnering with the private sector but “such partnerships are not strong”.
Pesendorfer said that if we are talking about policy transformation, what we measure determines what is important. But while we need good measurement – measurement is “a necessary but not sufficient condition of good policy”. Who actually decides what to measure? “In Austria it is totally policy driven – either by politicians or by the EU.”
UNSD head Stefan Schweinfest said that as efforts increased to leave no one behind, NSOs were now shifting their focus to include new sources of data, including individual perceptions, as how realities are perceived influences economic and political decisions.
From data stewards to data advocates?
The statistics-policy nexus was highlighted by the proposal to create a Chief Statistician in order to enhance the voice of statistics in UN policy processes, including UN reform, first proposed in 2016. The report of the High-level Group for Partnership, Coordination and Capacity-Building for statistics for the 2030 Agenda for Sustainable Development (HLG-PCCB), which submitted this recommendation, stated that such a position “can help shape the interaction between the technical and political aspects of the work on indicators” and that it will define mechanisms to make recommendations to the Inter-agency and Expert Group on the Sustainable Development Goal Indicators (IAEG-SDGs) “on strategic issues at the country level, including the use and interpretation of indicators and means of implementation”.
The proposal was received by markedly conflicting views, with 18 countries supporting, and 14 countries opposing. Donor countries were split, with the USA, Sweden, Switzerland, Argentina, Italy, South Africa, and China supporting and Nordic countries (Norway, Denmark, Finland) and EU countries (Germany, Ireland, Poland, Slovenia, Croatia, Portugal) plus Japan and New Zealand opposing. Norway stressed need to see how this would play out in the ongoing UN reform process before deciding, and instead focus on strengthening existing mechanisms including the mandate of the Committee of Chief Statisticians which already exists. Francesca Perucci, on behalf of the Statistics Division reiterated the urgent need for better coordination and voice and support, as recommended by the High-Level Group, but noted that systematic consultations are necessary before taking the proposal to the Economic and Social Council (ECOSOC).
Evidence-based policy in the work of ECOSOC
ECOSOC Vice President Mona Juul, speaking on behalf of ECOSOC President Inga Rhonda King, introduced the report of the IAEG-SDGs in measuring progress towards Sustainable Development. She stated that ECOSOC serves as the principal organ to coordinate the economic, social and related work of the United Nations, and that functional commissions, such as the Statistical Commission, drive its work: “The Statistical Commission, serves as the hub of the global statistical system.” She explained:
“By consolidating the links across ECOSOC bodies and forums, data and statistics can better inform the policy discussions, leading to more data and statistics used in evidence-based policy making. As countries prepare Voluntary National Reviews, strong, sustainable statistical systems provide the foundation for the collection, production and dissemination of this evidence.”
Under Secretary-General Liu Zhenmin, head of the UN Department of Economic and Social Affairs (DESA) echoed this, saying the Statistical Commission brings the voice of statistics into the policy process, enabling countries to confront some major challenges, including use of nonofficial data and its integration with official data so as to inform decision-making.
Streamlining the Global Indicator Framework and reducing UN agency reporting demands
Member States welcomed the latest IAEG-SDGs report on the global indicator framework and the advance of the federated system of data hubs. They specifically focused on the need to eliminate overlap and reduce the burden on NSOs in reporting to UN agencies.
Germany, Netherlands, Austria and Croatia all expressed this concern. Germany stated that 2020 provides an opportunity to improve the global indicator framework and avoid overlap – “so we can get a lean and consistent framework that will not impose reporting burden on NSOs”.
Croatia also cited the need to eliminate burdensome requests by UN agencies as well as to strengthen national ownership, not only on the statistical side but also on the analytical side of reporting. So too Austria, cited the need to limit burdensome data verification requests by UN agencies, especially those not appropriate at the national level, and stressed that the source of UN agency data for which verification was being sought must be specified. He strongly urged the IAEG-SDGs to limit the number of indicators in the framework and encouraged more coordination and use of overarching indicators and those really needed.
Belarus wished to see guidelines on how the work of UN agencies and NSOs should be ordered when establishing national level indicators and “find ways to compromise”. South Korea expressed concern about multi-custodian agencies for many of the goals, all using different methods of calculation. An example cited is indicator 16.1.1 on victims of homicide, for which the data is produced by the host ministry but also by Organisation for Economic Co-operation and Development (OECD) and the World Health Organization (WHO). Suriname added with regard to data and reporting flows that the guidelines on reporting to international and supranational agencies must be reformulated and reiterated that a country cannot report if there is not a national report.
Ghana expressed concern regarding indicator 16.7.1 on institutional representation by sex, age, disabilities and population groups, and welcomed greater attention to national level representation but noted that it should not include private sector institutions. Ghana requested that the word ‘public’ be inserted before the word ‘institutions’. Suriname also stated that this indicator needed greater clarification.
Several countries (Oman, Ghana, Philippines, India, Egypt) welcomed the use of proxies identified in the IAEG-SDGs report (see GPW UN Monitor #2) but Oman also thought more improvements could be made to indicators for SDG 6 on access to affordable energy.
Moving from methodology to measurement
As many Tier III indicators had now moved up to Tier II, several Member States expressed concern with Tier II indicators, meaning those with an agreed methodology but an insufficient number of countries compiling and analysing the data. Colombia welcomed the IAEG-SDGs’ work on Tier III indicators but stressed the need to keep working with custodian agencies to move from methodology to measurement. Brazil also welcomed the movement of indicators up from Tier III, but felt that some of them would benefit from wider consultations; for example, indicator 2.4.1 on the proportion of agricultural area under productive and sustainable agriculture requires a debate on the appropriate methodology. And South Africa, speaking on behalf of the South-South network singled out the lack of depth in poverty measurement, particularly Indicator 1.2.2, on the proportion of people of all ages living in poverty in all its dimensions according to national definitions. He stated: “Many countries have multidimensional poverty statistics but there is no official channel through which to report on progress”.
Streamlining the Global Indicator Framework in 2020: how and for and by whom?
The status of the global indicator framework will under-go a comprehensive review in 2020. The IAEG-SDGs has proposed to the Commission the removal of Tier III indicators for which there was little or no progress on an agreed methodology. This would allow the addition of a select number of indicators, from a list of 37 included in the IAEG-SDGs’ report to the Commission in March 2018. This opportunity was specifically supported by the Pacific Community, representing 22 countries, which stressed that SDGs 13 and 14 need a lot more work, as many of the indicators are stuck at Tier III.
Removing Tier III indicators could also allow the addition of indicators long sought by civil society – including a measure of inequality under SDG 10 on inequalities within and between countries, an indicator for civil justice under SDG 16 on access to justice, and additional measures of progress to complement GDP under SDG 17.
Germany addressed the need to avoid overlap, noting that 2020 would be the opportunity to revise the framework, but without adding additional indicators.
The UK and Sweden indicated another option, using 2020 as an opportunity to add indicators that advance some positions developed at the OECD. The UK noted that while the Addis Ababa Action Agenda agreement focused on mobilizing all types of resources, there is unfinished work in measurement, specifically regarding the share of resources, including from the private sector, for countries on the one hand to reach the goals and the share that goes to global public goods: “Thus 2020 is an opportunity to improve the global indicator framework by adding the Total Official Support for Sustainable Development – TOSSD framework.” Sweden, supported by the Philippines, stated that the TOSSD methodology should be considered as a way to bridge the gap between the indicator framework and financing. Many NGOs have criticized this “total” figure, as it adds state subsidies and private investment independent of its contribution to the SDGs but does not subtract the transfer of profits and royalties or divestment.
Measuring the impact of the private sector more broadly
Regarding SDG 12 on sustainable consumption and production, for target 12.2 on sustainable management and efficient use of natural resources, the key indicator 12.2.1 on “material footprint” per capita and per GDP is still in Tier III, while indicator 12.2.2 on material consumption is in Tier I. The same (former) indicator is used for target 8.4 on decoupling economic growth from environmental degradation. The accounting used to measure material consumption transfers environmental damage caused by the products consumed out of the country they were consumed in. (See GPW blog: “Decoupling is the glue that holds “sustainable” and “development” together, material footprint is its indicator”).
Target 12.6 encourages companies to adopt sustainable practices and integrate sustainability information into their reporting cycle. To date the indicator, 12.6.1 on number of companies publishing sustainability reports, is still at Tier III. UNCTAD, as part of the intergovernmental group on international standards, accountability and reporting, is bringing the private sector together with governments and NGOs to identify core indicators for measuring private sector contributions within the sustainable development framework. UNCTAD cites progress with the development of standards for transnational companies, but the need for a more flexible version for small and medium enterprises.
Jørgen Elmeskov, Statistics Denmark reported that they have set up a partnership, with MPs, NGOs, politicians all interested in monitoring the SDGs. He acknowledged that it is all about promotion for the companies, and the need to know an investment is sound for investors. He stated: “If we do not impose pretty strict rules we will not get far”, and “need a comparable, comprehensive view of what enterprises are doing.” This means looking at “products, processes and motives” at the appropriate level.
Juan Daniel Oviedo, Statistics Colombia, reported they are engaging in partnerships but are a long way from having good reports from the private sector. The main problem is the lack of incentive. He stressed the need to strengthen the measurement and systemization of the information on sustainability, getting information on the intensity of resource use and environmental impacts.
Bernard Frey of the UN Global Compact, speaking of trends in corporate reporting, said 72% of corporations mention SDGs in their annual reports, and 54% include them in business strategies. While most companies are not really clear as to why they should mention the SDGs, for those who do, it is mostly to attract investment, or to explain to shareholders how business strategy contributes to sustainable development.
He acknowledged the risk of SDG washing and cherry picking; corporations may have other issues, affecting labor and the environment, calling this an outstanding challenge. To which Elmeskov commented: “We need to distinguish between incentives to report and incentives to behave. ”
Frey pointed out that companies tend to choose to report on SDGs that are easier for them, such as a pharmaceutical company reporting on SDG 3 (health). What is measured are their activities or output but not actual impact. He used the example of a solar panel manufacturer, reporting that they are contributing to emissions reduction (13.2), but not on the high number of work related injuries (8.8) or a large environmental footprint (8.4).
While UNCTAD is charged with coming up with an agreed methodology by the end of 2019, it is clear that they have a long way to go. Is it possible that this might become one of the Tier III indicators to be eliminated?
From policy making to politics in the interpretation of the 2030 Agenda and the targets and indicators by which to measure progress
A far more critical view of the role of politics, rather than policy making per se, on the selection and measurement of SDG indicators was expressed by panelists in a side event with authors of a special issue of the Global Policy Journal, “Knowledge and Politics in Setting and Measuring the SDGs”.
In-depth case studies and analyses of the governance landscape of the SDGs, plus a number of commentaries from stakeholders, detail the ‘slippage in ambition’ of the scope and intent of the SDGs, as different priorities of developed and developing countries and of the UN agencies operate in the selection of targets and indicators, particularly with regard to human rights.
Pali Lehohla, former head of Statistics South Africa commented in the Journal: “The facade of measurement innocence fails to answer the vexed question of why the United Nations Statistics Commission and the Human Rights Commission never ever sat side by side despite being Siamese twins.”
Sakiko Fukuda-Parr, co-editor of the special journal edition and author of an article on inequalities, observed with others that “measurement is never an innocent matter where facts speak for themselves”. Rather, as civil society representatives have stated often, what is measured, who does the measuring, and who finances it are all subject to the political priorities of different players.
Manjari Mahajan, author of an article on the shifting landscape of health metrics, spoke about the influence of a single donor in not only determining what gets measured but in sidelining NSOs and the WHO, which are charged with compiling and analysing data and statistics for SDG 3 on health.
Her article focuses on the Institute of Health Metrics and Evaluation (IHME), primarily funded by the Bill and Melinda Gates Foundation. Resources of over US$ 1 billion over ten years has gone to measurement, primarily to a measure called the Global Burden of Disease (GDB). This measure provides estimates on the magnitude of health loss from disease, injuries and risk, by sex, age and other population groups. It is distinct from the approach of the WHO, in that it does not rely on NSOs exclusively for data but takes data from insurance companies, hospitals, private servers, and so on, in “stark contrast with reliance on traditional sources of statistics”. Indeed, “reflecting the underlying assumption that more data is better, super computers and modeling capacity have far outstripped the capacity of NSOs.”
IHME has some 3000+ partnerships, including universities and the private sector, with well-funded PhD programmes creating “a new generation trained in how to think about numbers”.
With regard to the SDGs, a MOU signed between IHME and the WHO in 2018 has resulted in “erasing a competitive landscape,” Mahajan said. Asking: “What are the implications of this hegemonic influence?” She answered: “It has sidelined and coopted WHO from within.” It has also sidelined NSOs, she added, as none of the IHME funds are used to build their capacity, and by drawing on other sources of data further undermines NSOs.
Furthermore, a lot of the data is protected by proprietary agreements, raising question of transparency and accountability. “IHME produces a particular kind of knowledge which is presented as global knowledge,” she said.
Reflecting the thoughts of many of the contributors she commented: “Statistical methods are meant to hold governments accountable but need a discussion of how to hold methods themselves accountable.”
A related challenge to partnering with philanthropy or the corporate sector has to do with privately held data sources, referred to loosely as ‘Big Data’.
Steve MacFeely, of the Centre for Policy Studies, University College Cork, acknowledged that in offering an opportunity to link data from different sources and different goals, Big Data responds to one of the objectives of the SDGs, that is, moving from reporting in silos on different goals to seeking an integrated assessment of progress. “It offers a strategic opportunity for developing countries to jump ahead”, he said, “but there are tricky issues of sovereignty, and much of Big Data is proprietary”.
Does it also offer opportunities for UN agencies to become data brokers? “These questions raise issues of ownership, quality, access and privacy”, he stated, “while measuring digital activity creates its own challenges – legal, IT, governance and ethical, among others”.
“Why is Open Data only with regard to public sector?” he asked. “None of the private sector data is open. I am concerned that data will be at the heart of human rights abuse – and it is important that the UN not be part of this.”
Addressing many of the criticisms, Francesca Perucci of the Statistics Division said that the statistics community inherited the tensions from the SDG policy discussions, which they were not ready for, so that the political and technical aspects got compressed within a very short time frame. She added that UN agencies also played a role, with their own marching orders, governing bodies and agendas. With regard to capacity building, a mandate of the Division, she stated that the current way of financing means one single donor can decide the policy agenda.
Speaking specifically of SDG 10, Fukuda-Parr voiced the still unaddressed criticism that there is no target or indicator to measure the distribution of wealth, despite the fact that two frequently used indicators already exist: the Gini coefficient, which is sensitive to shifts in the middle income sector, and the Palma ratio, which measures the distance between the top and the bottom of the national income scale. This raises the question: What is the policy goal? “The choice of an indicator that measures the income of the bottom 40 percent of the population ignores the role of the top 10 percent in capturing policy.”
Shahra Razavi, head of the data and information section at UN Women observed in another commentary that ‘the slippage in ambition’ between the 2030 Agenda and the targets and indicators certainly applies in the case of women’s rights, “with the ambition to have gender equality as a crosscutting issue sometimes evaporating as targets were set and the indicators selected”.
She noted that gender specific indicators under Goal 16 on access to justice suffer from the same limitations identified by the authors of the article on SDG 16, namely the privileging of crime victims and criminal justice, rather than the gender-specific consequences of lack of access to civil justice. “The gender-specific indicators put the spotlight on women as victims of intentional homicide, conflict-related deaths, human trafficking and sexual violence, with close to no attention to the more complex and process-oriented institutional dimensions of women’s everyday legal problems….”
The year 2020 might provide the opportunity to address these limitations, as the IAEG-SDGs has said it will eliminate “orphan indicators” for which there is no methodology, and, in a few exceptional circumstances, add new indicators. Will this be the opportunity to finally get an indicator on inequality (SDG 10) as well as an indicator on access to civil justice (SDG 16)? In response to the call from Joseph Stiglitz in presenting his report, will the IAEG-SDGs agree on indicators of progress to complement GDP? All three of these indicators were among the list of 37 reportedly under review (see GPW briefing #22: “The Ups and Downs of Tiers: Measuring SDG Progress”).
Also, since “decoupling” GDP from material consumption seems to be key to holding together “development” and “sustainable”, will the IAEG-SDGs finally adopt a methodologically agreed material footprint indicator (SDG 12)?
What will make the final cut?