What primary questions could the Global Data Barometer include to address data values?

This is the final post in a series produced as part of the analysis for the Measuring Data Values Around the World project.

The next edition of the Global Data Barometer is likely to be developed over Autumn 2023. In this post we consider possible additional questions that might be included as part of the Barometer’s expert survey to address key concepts from the Data Values framework.

In doing this, there is a delicate balance to strike between identifying questions that speak directly to elements of the Data Values framework, with designing questions that are reasonable for generalist researchers to complete primarily through desk research as part of a larger survey, and that fit within the overall framework of the Barometer.

Furthermore, the first edition of the Barometer made a design choice to focus on including questions that measured country performance against global norms, rather than against aspirations. That is to say, inclusion of questions in the first edition of the Barometer depended on being able to ground both the overall question, and the individual elements within it, in one or more principles or statements that could be shown to have broad international agreement. Although broadly consulted on, the Data Values Manifesto and white paper do not currently have this status, and so, if the principles from the first edition of the Barometer were followed then either (a) questions for inclusion would need to identify _other _normative foundations; or (b) questions on Data Values could be included as ‘informative’ rather than ‘normative’ components of the Barometer (i.e. they might be reported on, but not included within the calculation of pillar and overall Barometer scores). Alternatively, the next edition of the Global Data Barometer may relax this condition, and take a more forward-looking and opinionated approach overall to setting the agenda and norms on data for the public good.

Suggested questions

Based on the considerations, longlist, and early question development described below, we have identified four priority questions that could be put forward for consideration in the next edition of the GDB. In making these initial suggestions we recognise that the GDB is developed through a consultative process, with the final survey decided in consultation with thematic partners.

Our suggested questions are:

• To what extent are there established mechanisms to enable broad and inclusive public input into the design of key public datasets?
• Is there evidence that changes have been made to the design (or use) of key public datasets as a result of public engagement?
• To what extent is there broadly accessible critical data literacy education in the country?
• To what extent are civil society groups equipped to engage with governments on issues of data governance?

In addition, we suggest that the data protection indicator should be updated to assess whether frameworks provide for collective rights of action, and specific rights in relation to automated decision making, and opportunities should be sought to incorporate sub-questions on public engagement frameworks and practice within other draft indicators.

Initial considerations

Kinds of questions: evidence or extent

There are two kinds of questions we may want to ask: exploratory and evaluative.

Exploratory questions: Is there evidence that/of…

These questions would ask researchers to locate one or more examples of a particular framework/intervention/outcome, without necessarily assessing how far these examples demonstrate that the concept measured is ‘mainstream’ or ‘embedded’ within the country.

This kind of question is useful when we might expect there to be countries with no examples at all of a given practice, and where a small number of examples may be a positive sign of engagement with the concept.

For example, in early editions of the Open Data Barometer (the Global Data Barometer’s predecessor), it was useful to ask about the presence of events like hackathons to support engagement with open data: as there were many countries where they were yet to take place. Within a few years, many countries had _at least one _open data hackathon that had taken place (sometimes donor supported) which made this question less able to discriminate between countries.

Evaluative questions: To what extent…

These questions seek to provide a comparative measure between countries based on adoption or implementation of some concept, with the answer provided through a series of sub-questions that look at the existence of some measure, the elements of that measure, and the extent to which that measure is in place (i.e. only visible in a limited number of settings, or evident throughout the jurisdiction).

To be researchable, these questions can’t ask researchers to check every _possible place where a concept should be enacted (e.g. checking that _every government agency has consultation mechanisms for the design of datasets), but instead need to suggest heuristics for assessing how widespread a framework, capability, or practice is.

It is worth noting that the earlier Open Data Barometer based ‘To what extent…’ questions on a different method, inviting researchers to provide a 0 – 10 score based on their qualitative judgement and following a rubric. This model is more flexible, but also is vulnerable to differences in subjective judgement. It may, however, be possible to pair country researcher evidence collection, with subjective rating of evidence by a smaller number of researchers, with assessments checked for intercoder reliability.

From exploratory to evaluative: measuring sustained practice

There are some cases in which an exploratory question might be able to provide an evaluative metric over time. While, for example, activities like open data hackathons were ultimately short-lived models of engagement, if we were to ask whether, in the last two years covered by the study, there was any evidence of public engagement around a key public dataset, we might expect to find at least one new example every two years. Looked at across multiple editions of the Barometer, an exploratory/observational question like this might help draw distinctions between countries that having very limited or ad-hoc efforts at engagement (perhaps donor supported), and those that have integrated and embedded new forms of practice.

Question long-list

The Data Values framework addresses issues of inclusion and participation across three distinct areas of data governance:

• Design: Dataset design and collection
• Use: Data sharing and use
• Accountability: Accountability and redress

Recommendations under the framework and manifesto cover issues of investment, capability, process and outcomes. Barometer questions can be designed to emphasise different aspects of these recommendations, such as the presence of frameworks and formal mechanisms or evidence of supporting interventions, capabilities, practices, and outcomes.

In the table below a set of potential questions have been outlined from an initial brainstorm. While only some of these have been developed further in the following sections, the full list is presented to support discussion of alternative options.

N	Question	Assessment	Select?
	Understanding support for public participation to shape data (Manifesto points 1 & 2; Barometer Governance and Capability pillars)
D1	To what extent do relevant laws, regulations, policies, and guidance provide a framework for inclusive and participatory governance of key public datasets?	Ruled out because we currently lack clear evidence on the kinds of laws, regulations, policies and guidance that a country may be reasonably expected to have in place to provide a framework for inclusive and participatory governance of public datasets. As references to participatory engagement are likely to be spread across many laws or policy areas, requiring researchers to check all the potential areas of relevance would be burdensome.
D2	To what extent are there established mechanisms to enable broad and inclusive public input into the design of key public datasets?	This option offers a route to a question that is mid-way between a Barometer ‘Governance’ and ‘Capability’ question, by asking about mechanisms, and allowing us to identify cases where these are as a result of comprehensive government action vs. ad-hoc agency decisions. This should generate evidence which might be used to develop D1 in future.	Yes
D3	Is there evidence that public agencies have made credible commitments to involve affected communities in decisions about the design and use of key public datasets?	It may be possible to find secondary data proxies for D3 in the form of commitments to the Fundamental Principles on Official Statistics, or future country signature to documents like the Data Values Manifesto, or, if it includes appropriate terms, the forthcoming UN Digital Compact. This makes this of lower value as a primary question.
	Understanding capability and data skills(Manifesto point 3; Barometer capability pillar)
C1	To what extent are civil society groups equipped to engage with governments on issues of data governance?	While it may be complex to judge how well equipped civil society are, a civil society able to engage on data issues appears to be a vital component of delivering data values, and no other global measures provide insight on this topic.	Yes
C2	To what extent is government providing support for affected communities to engage around issues of data governance?	For many countries, support for affected communities to engage around issues of data governance may come from sources other than government, such as international donors. This makes C2 tricky to treat as a normative measure (support from other sources may be as desirable as having government support.
C3	To what extent is there broadly accessible critical data literacy education in the country?	There are significant global primary data gaps about data literacy in general. However, other measures proposed, or that might become available as secondary data, focus primarily on technical skills. This question could add to understanding of how far critical skills are available.	Yes
	Understanding practices and outcomes (Manifesto points 1 & 2; Barometer Availability & Use and impact pillars)
I1	To what extent is there evidence that affected communities are actively involved in shaping the design and collection of key public datasets?	This focuses on process as opposed to outcomes or impacts, and risks some duplication with Dx questions about mechanisms.
I2	Is there evidence that changes have been made to the design (or use) of key public datasets as a result of public engagement?	It would theoretically be possible for a change to be made which is regressive (e.g. initiating a data use, or stopping data collection in ways that make minority groups more vulnerable). A question might need to include stronger normative content to make sure evidence can be counted as a positive evaluation of the country
I3	Is there evidence that data collection (and use) practices have been altered to take account of the needs of commonly marginalised communities?	As a ‘evidence’ question this does not try to provide full evaluation, but assumes that the presence (or absence) of evidence in response to this question (particularly when limited to changes during the two-year study period) will be able to offer some discrimination between countries. We assume that all countries have changes they need to make to improve data collection, and so the absence of change in any two year study period can be reasonably judged as negative.	Yes
I4	Is there evidence that public authorities have engaged with historically marginalised groups in revisiting and revising statistical indicators	This has a similar focus to I3, but places emphasis on evidence of engagement, as opposed to evidence of change. It may be easier to research in some cases, but is less likely to generate ‘bright spots’ examples to build on.
	Understanding accountability and redress
A1	To what extent are there mechanisms to allow communities impacted by data collection or use to raise concerns about potential harm?	This could be particularly addressed by adding redress questions into existing data protection indicators.

Developed questions.

This section offers an initial outline of selected questions that might be included in a future Barometer expert survey, including draft survey guidance.

Common definitions

Questions make use of the following common terms:

Data governance refers to decisions made about the data to collect (including the ways data will be disaggregated, and the way variables are described), the way it is collected, how data is secured and/or shared, how data is used, and how concerns about data are addressed.

Key public datasets refers to any significant dataset used to support public sector decision making or the delivery of public services. For example, the Census, health service records, household or farm surveys, budget and spending data, or operational data from the delivery of frontline services.

Affected communities refers to any group (constituted or not) that feel effects of the collection or use of a particular public dataset. Particular attention should be paid to marginalised groups who may have additional needs, or whose needs are often not taken into account adequately in public policy.

To what extent are there established mechanisms to enable broad and inclusive public input into the design of key public datasets?

Definitions and identification

Mechanisms for public input into data governance may take many forms. This question is focussed on public input that addresses questions around:

• The data that should be collected and how;
• The categories that should be used in data collection and publication;
• How data should be governed, including arrangements for sourcing data from, or sharing data with, the private sector.

Mechanisms might include:

• Parliamentary processes in which elected representatives explicitly debate data collection measures, and seek input through committee hearings, written submissions or other processes;
• Public consultations prior to the introduction of a new data collection process, or to review and update an existing process;
• Advisory groups that provide a channel for regular public feedback on data collection plans and processes;
• Public dialogues that foster in-depth discussion of particular data gaps, challenges, design or data sharing choices.

Look for evidence that a mechanism is inviting public input (either directly, or through civil society intermediaries), as opposed to only inviting input from official stakeholders or expert groups

[ Examples of mechanisms that might be relevant to this question should be included in the final guidance. ]

In the first instance, look for mechanisms that are initiated or managed by the government. If you cannot find any government-led mechanisms, you may answer with respect to non-governmental (outsider) mechanisms.

Research guidance

First identify any significant decisions about the design of key public datasets that may have taken place during the study period (for example, decisions about an updated census, creation of a new national register, or digitisation of a previously paper-based administrative process).

Then look for any evidence that public input was sought, or given, to shape decisions.

Assess the mechanism by which public input was sought. Was this a one-off activity? Or did this make use of an established wider mechanism? Check if use of this mechanism is mandatory or optional when data design decisions are made.

Repeat this process for multiple decisions, aiming to look at the practices of a number of government bodies.

A complementary research strategy is to look for information about public participation mechanisms in place in the country (such as policies on public consultation, or information on the national government website about advisory bodies). Look for instances of these mechanisms being applied to data.

Sub-questions

Note: these are early drafts and need further refinement.

• Existence
  • Do mechanisms exist to enable public input into the design of key public datasets?
    • No mechanisms were identified. Any public input identified was ad-hoc or one-off.
    • Mechanisms exist, but are not led by the government.
    • The government has and makes use of one or more established mechanisms to seek public input into the design of key public datasets.
• Elements
  • Mechanisms for public input into design of key public datasets include:
    • parliamentary oversight
    • formal consultations
    • advisory groups
    • public dialogue
  • At least one of these mechanisms
    • Explicitly seeks to involve affected communities
    • Explicitly seeks to gather representative views from across the population
    • Supports deliberation between different stakeholder groups
    • Provides support for marginalised populations to engage
    • Has had a demonstrable impact in making datasets more inclusive (e.g. adding new disaggregation, or addressing problems of missing data)
• Extent
  • How widely used are these mechanisms?
    • They are used by a small number of government bodies, or with respect to a small proportion of key public datasets;
    • They are used by a large number of government bodies, or with respect to many different key public datasets;
    • They are used whenever there are significant decisions to be made about the design of key public datasets.
• Examples
  • Please list examples and tag them with the elements they relate to
  • Include a flag for ‘best practice’ examples to be surfaced.

To what extent are civil society groups equipped to engage with governments on issues of data governance?

Definitions and identification

Civil society groups may be formal associations, or informal groups who have organised in order to represent the interests of a particular community.

Engaging with government on issues of data governance might include (but is not limited to):

• Responding to government consultations or participating in advisory groups around data collection and use;
• Campaigning on issues of data collection and use, such as calling for better gender disaggregation or data, addressing the invisibility of certain populations in collected data, or calling for indigenous data sovereignty;
• Taking legal action to protect data rights (e.g. opposing transfers of public data to the private sector without adequate transparency and debate);
• Supporting individuals to seek accountability and redress over misuses of data.

Being equipped to take on these kinds of activities involves civil society groups having:

• Relevant awareness and data literacy
• Resources and capacity
• Opportunities to engage

Research guidance

This question invites your subjective assessment of civil society capability with respect to engaging government on data governance topics.

To support your assessment you will need to gather evidence by:

• Looking at a range of civil society organisations active in different sectors (e.g. health, education, women’s rights, racial justice);
• Finding examples of how organisations have engaged with government;
• (Optionally) talking to leading civil society organisations about how confident they feel in addressing data governance issues.

For this question you may judge that organisations are equipped to engage even if there is not evidence that they have actually engaged. For example, if there are generally high levels of data literacy in the population, strong civil society freedom, a well resourced civil society, and programmes offering support for civil society to understand and engage with data issues.

Sub-questions

• Existence
  • Are there civil society groups equipped to engage with the government on data governance issues?
    • No. There are few, if any, civil society groups equipped for this.
    • Limited cases. A small number of civil society groups are equipped to engage with the government on data governance issues. These may be specialist organisations whose main focus is on data.
    • Yes. There are many civil society groups equipped to engage with the government on data governance issues, representing a broad range of sectors and communities. Leading civil society organisations are amongst those equipped to engage.
• Elements
  • There are groups equipped to engage on:
    • Data disaggregation – including gender disaggregation
    • Privacy and security
    • Data sharing with the private sector
    • Governance of automated decision making
  • Inclusion
    • Civil society groups representing minority communities are equipped to engage with government on data governance issues
    • There are capacity building programmes or specific funding opportunities to help civil society engage on data governance issues
• Extent
  • (To be refined: overlaps with Existence at present)

Alternative structure

This question could alternatively be structured using a rubric based expert score as follows:

To what extent are civil society groups equipped to engage with governments on issues of data governance?

0	3	5	7	10
There is no evidence of civil society being able to engage with data governance issues at all.	There are isolated cases where civil society groups are equipped to engage with government on data governance issues.	There are a number of cases of civil society groups equipped to engage with government on data governance issues, but most groups lack relevant data literacy and awareness to identify or act on data governance issues that affect their communities.	Many civil society groups can identify data governance issues that affect their communities, but only a small number are equipped to take action and engage with government to secure changes as a result.	Many civil society groups can identify data governance issues that affect their communities, and the majority of these have, or can access, the resources, skills and experience they need to engage effectively with government.

Is there evidence that data collection and use practices have been altered to take account of the needs of commonly marginalised communities?

Definition and identification

A change in a data collection or use practice might involve:

• Changing the categories used to collect data or introducing new data collection processes
• Introducing or stopping the use of an automated decision-making process
• Updating guidance on how data should be used to address issues of bias

Research guidance

Identify significant decisions made with respect to key public datasets over the study period through reviewing the literature, examining government websites, and talking to key informants.

Identify if any of these involve changes to address the needs of communities that are commonly marginalised, excluded or subject to discrimination in this country.

Sub-questions

For each example identified, please provide:

• A brief description of the changes made to data collection or use practices.
• Whether or not this was related to the operation of a public engagement mechanism (see tags from questions above)
• Which minority communities were affected

To what extent is there broadly accessible critical data literacy education in the country?

Definitions and identification

Critical data literacy refers to the ability to critically analyze, interpret, and make informed judgments about data. It involves a combination of data literacy skills, critical thinking and contextual awareness and emphasizes the importance of questioning the sources, biases, and limitations of data, as well as considering the ethical, social, and political implications associated with its collection, analysis, and use.

Research guidance

Look for evidence of data literacy related courses and curricula and examine how far these address critical data literacy skills.

Sub-questions

• Existence
  • Where is there evidence of critical data literacy education in the country?
    • There is evidence in primary or secondary schooling
    • There is evidence in University education
    • There is evidence in public administration training programmes
    • There is evidence in training available to civil society
• Elements
  • Accessibility
    • [ This question needs further development ]
  • Embedding
    • Competency frameworks or assessments for officials making decisions around data require critical data literacy
• Extent
  • [ For further development ]