Disaggregated data is important for two principal reasons:
1) To make information "actionable" and
2) to identify vulnerable and marginalized groups.
Actionability implies greater clarity on the steps that governments can take to improve their scores on an indicator. The trend toward more actionable indicators can be seen both at global and national levels, which emphasizes the demand for better national-level data also from global indicator producers. An actionable indicator is one in which data allow disaggregation to pinpoint bottlenecks and inefficiencies within the public administration and wider systems of accountability. In addition, data should be able to correctly attribute change to policy initiatives and tell to what extent observed changes are the results of government actions or caused by external factors.
Identification of vulnerable groups requires data that can be disaggregated by income, gender and other markers such as region, age, ethnicity, religion, sexual orientation, disability and so on, depending on the scope, country context and objective of the assessment. This requires household surveys with sample sizes large enough to allow extensive disaggregation of the results. It also may require individualized questionnaires targeted toward vulnerable groups, such as a women's questionnaire. Qualitative research techniques such as focus groups, consultations and interviews may offer insights for formulating questionnaires and interpreting results, as well as more in-depth information about social groups.
Both of these two entry points emphasize the need for country contextualization of indicators, and the need to secure primary, bottom-up and robust data produced within countries.
The following information outlines the main kinds of data and their various advantages and disadvantages. Types of data include:
Qualitative versus quantitative data
Subjective versus objective data
Household survey or census data
Expert or focus group interview data
Qualtitative versus quanitative data
Governance data can be quantitative or qualitative in nature. In broad terms, qualitative data are comprised of words, whilst quantitative data are comprised of numbers and statistics. Nevertheless, it is possible to modify and transform qualitative data into quantitative data. Rather than regard these methods as contradictory, it is constructive to see them as complementary. Gathering both quantitative and qualitative information helps refine methodologies and enriches analyses. The design and use of quantitative measurement methods and tools must be informed by accurate qualitative information, including in-depth documentation of perceptions and experiences of target populations and of assumptions and expectations of different stakeholders. Qualitative information is essential not only to ensure proper design of survey questionnaires, but also to focus statistical analysis on relevant issues and to provide appropriate contextual frameworks for an effective policy-oriented interpretation of quantitative data. Quantitative and qualitative data can and should interrelate in order to properly and comprehensively inform governance monitoring exercises. Broadly speaking, qualitative data, such as data gathered from expert interviews and focus groups, are more flexible, allowing in-depth exploration of the meaning behind concepts and events, enabling a clear understanding of the situation, and exposing the motivations and patterns of association between factors. One main disadvantage, however, is that sample sizes are often small and do not allow for representative data to be collected. The generalization of findings can thus be an issue. In addition, expert interview methods rely heavily on respondents being reasonably articulate; and processes behind the analysis of the data are not always transparent or replicable.
Quantitative data, usually based on surveys, are extremely useful in producing statistics. Moreover, if random samples are used, estimates are precise and inferences can be made about the target population. Other advantages of quantitative data include their capacity to measure the extent, size and strength of observed phenomena; their usefulness in determining the importance of given factors in influencing the outcomes; and their use of standardized procedures, which allow for replication of results. However, quantitative data collection methods can be costly, especially if the target population is hard to reach. Sampling frames are not always available, the use of structured interviews can sometimes hinder a detailed exploration of reasons behind specific actions or decisions, the use of standardized questionnaires means little flexibility, and key concepts must be clearly defined and translated into meaningful survey questions.
Subjective versus objective data
Just in the same way as quantitative and qualitative data can be seen as complementary tools, so can objective and subjective data. Objective data deal with observed/experienced facts or situations (such as levels of income and consumption, housing conditions, level of education, cases of corruption), whilst subjective data are linked to perceptions and assessments of the people being surveyed (such as degrees of satisfaction with living conditions, and opinions on how institutions operate and on the policies they implement). Subjective data is often considered “second best,” and are criticised as being less reliable than objective data. However subjective data is very important in governance measurement and assessment exercises. The way in which a population perceives a given issue is extremely important in understanding local concerns, even if there is little “objective” justification for the perception. Perceptions can be early signs of significant events to come, such as conflict, condemnation, or the overthrow of a regime.
Household survey or census data
Household survey data are almost always based on a random sample of a population or sub-population; in contrast, a census attempts to collect data from an entire population. The questionnaires are designed to capture the desired information. Both surveys and censuses can be focused on capturing factual data (e.g., household income), or perceptions/opinions (e.g., what the respondent believes is a reasonable wage in his/her community), or both. There are many different ways of carrying out a survey: face-to-face, on the telephone, by mail, email or the Internet, etc. Survey and census data are almost always quantitative, although a survey may contain a qualitative component. Censuses allow for socioeconomic data to be gathered, which can be extremely useful in providing information on the experiences of governance across different segments of the population. Censuses also can show the percentages of individuals entitled to vote, voter participation by age and gender, and so forth. Survey data are particularly important for monitoring the equality and effectiveness of governance reforms. In contrast to administrative data, survey data can go beyond the reporting of events and conditions to capture the experience, perceptions and attitudes of individuals providing or receiving public services. Surveys should therefore be used to supplement data available from administrative sources.
Household surveys and censuses can be useful as a platform on which governance and human rights measurement exercises can be developed. For example, governance- and democracy-related questions can be grafted on to regularly implemented household survey instruments.
A survey, in the statistical sense, is a data collection effort that focuses on facts or opinions related to human populations. The word "survey" can refer specifically to the survey instrument or to the entire process by which the data are collected. Surveys of human populations and institutions are common in political polling and government, health, social science and marketing research. Surveys are very useful in producing data and statistics on the extent, size and characteristics of observed phenomena; in determining the importance of given factors in influencing outcomes; and, since they use standardized procedures, in allowing the reproduction of results. However, the effectiveness of surveys in measuring and monitoring democratic governance and human rights issues is limited. Most of these limits affect the accuracy of the information reported; for example, translation or interpretation issues may arise, especially in the case of surveys with international or multinational coverage. Indeed, surveys are prone to cultural biases -- and when applied in more than one country, or where there are different languages used within a country, it is crucial to make sure that the concepts being measured, and the wording of questions being asked, are accurately translated into the local language. Other features include:
Surveys, especially face-to-face surveys, can be extremely expensive.
Survey data are relatively easy to disaggregate by including appropriate questions from the start. However, survey data are subject to sampling error as well as to multiple sources of non-sampling error, which means that in order to obtain reliable estimates (meaning estimates with a small confidence interval) at a disaggregated level, the sample size needs to be very large. Drawing large samples is expensive, and unfortunately, non-sampling error tends to increase as sampling error decreases.
Surveys that involve writing or are conducted via telephone assume that the target population can read and write and/or that they have access to a phone. In many parts of the world, this is simply not true.
In the case of surveys about democracy and human rights, a challenge exists in conveying, asking about and measuring concepts that might be new to the population.
Like expert interviews and other data collection methods, such as individual human rights violation reports collected by human rights commissions and civil society organizations, survey samples must be designed to be representative of the population or the data produced will not be representative and will lead to over- or under-reporting of events and violations. Issues of selection bias are thus extremely important.
In general, a tradeoff exists between smaller, less expensive samples and larger, more representative ones.
In many countries, high-quality data already have been collected by the government in the pursuit of its administrative duties. For example, data are collected from citizens when they pay taxes, or when they register for social security or medical benefits. Though such data are not primarily collected to inform policies, they may be used as such – for example, medical data that include date of birth might be used to inform decisions regarding university scholarships designed to encourage geriatric specialists for an aging population. It is possible, by combining different administrative data (health, schools, police) to build an overall picture of the degree to which governments do or do not respect basic democratic governance and human rights principles, including principles of equality of access and enjoyment of public goods and services. However, administrative data provide limited scope for in-depth analysis. Furthermore, disaggregating administrative data is much more difficult than with survey data because the forms used to collect the data typically use categories that are too broad for useful analysis. Proxies may be necessary to disaggregate data: for example, neighbourhood may be a good proxy for income.
Expert or focus group interview data
Expert or focus group interview data can consist of either the transcript or summary of an interview. Usually, interviews are aimed at reaching a specific section of the population and can be individual or organized in the form of focus groups. In addition, interviews can be more or less structured, depending on their specific objective and on the amount of information already in the hands of those in charge of data collection. Such data are collected via very small population samples and are almost always qualitative.
Expert or focus group interview data are an extremely useful source of information for democratic governance and human rights measurement exercises. Whether they are individual or group interviews, data collected via this method allow for in-depth analysis of a large variety of democratic governance- and human rights-related topics, and for understanding of perceptions, needs and concerns of the population(s) under study.
Events-based data consist in the recording of events. Such data include newspaper articles and other news sources; individual records collected by NGOs (testimonies, etc.) or specialized private or public bodies; and information collected by independent researchers. Such data can be either quantitative or qualitative. In terms of governance, democracy and human rights indicators, events-based data are essential, because they provide data concerning the incidence of a series of critical elements which, in turn, reveal the democratic governance and human rights situation in particular areas. However, one of the main limits associated with this type of data is that findings are usually based on individual records, such as a report of a human rights violation before a Human Rights Commission; since these are not based on representative samples, they cannot be generalized to the entire population.
Because construction of this data depends on the filing of complaints/reports, inherent limitations exist: On human rights issues, for example, the data could be affected by selection bias and will not correctly capture the full extent and magnitude of human rights violations or the characteristic features of victims. In sum, events-based data can suffer from problems of under- or over-reporting of events.
These data tend to be presented in the form of qualitative statements authored by the expert and can be based on the legal situation of a country (e.g., data relating to adoption of specific legislation and treaties, often used to measure statements of intent) or on reports from civil society, the media or official sources. Numerous governance indicators available in international databases are constructed from "expert" assessments by consultants, researchers, development workers, decision makers, senior civil servants, politicians and others. Even though the number of surveys directly documenting the views and experiences of citizens is increasing rapidly, indicators based on that approach remain relatively scarce. The relevance of indicators based on the experts' opinions has to be assessed in relation to those based on surveys amongst individuals and households.
In the absence of household survey data, or where such data are fragmentary, assessments by experts usefully complement existing information. Because this source of data is essentially qualitative, it is potentially very useful for measurement exercises in preliminary and exploratory phases. Expert interviews are relatively inexpensive, for both obtaining information and allowing for in-depth study and analysis of a given situation. A major limitation, however, is that results, while interesting, cannot be generalized, since they are neither representative nor comparable. In addition, good expert interviews rely heavily on the extensive and high-quality training of interviewers.
As is the case with most data collection methodologies, expert judgments should be complemented and combined with other kinds of data.
This type of study and data collection method relies on existing qualitative data and usually aims to find out what the general (i.e., legal, economic, political and social) situation of a country is like. This may be based on material such as domestic legislation and reports published by the media, government, civil society and/or academia. By providing important background information, desk studies can inform the preliminary and exploratory phases of governance assessments.
Desk studies are extremely useful for collecting hard facts and existing data on the situation in a country. Even so, the main limitation of this data is that it relies completely on the quality of the information already published, does not necessarily supply representative data, and might exclude information on very recent developments.