How to measure disability: survey usability versus usefulness

The concept of disability is complex and its definition often changes depending on who you ask. A comprehensive view of disability should include functions and structure, limitations in activities or restrictions in participation as well as characteristics of both the person and their environment. These factors already make it tricky to measure disability itself, and this task becomes even more problematic in countries where disability is stigmatised, and people with disabilities are excluded from the scope of conventional official surveys. If every country adopts its own unique approach to defining and measuring disability, and understands disability differently or adopts a broader approach to a subject – at an international scale, you are left with a bunch of numbers that do not compare.

Source: Table taken from Jennifer H. Madans’s presentation ‘Overview of Disability Measurement and the Washington Group Short Set’ (available online here). Reproduced by kind permission. Data from https://unstats.un.org/unsd/demographic/sconcerns/disability/disab2.asp Note: Sources and methodologies are country specific; L-M = lower-middle

A year after the adoption of the Millennium Development Goals, the Seminar on Measurement of Disability identified severe gaps in measurement and comparison of data on disability cross-nationally. To address these, in July 2001 in New York, the UN Statistical Commission authorised the formation of a Washington Group (WG) on Disability Statistics. This group, which now includes members from 135 countries, several international organisations and disabled people organisations, has spent the last 16 years developing and testing simple and effective means of measuring and obtaining comparative data on disability across countries.

Their solutions are simple and nationally attainable – the Short Set of six Disability Questions to be incorporated into censuses and surveys which primarily collect data on other topics, and the Extended Question Set on Functioning that comprises 37 questions for a more comprehensive assessment of disability in a given population. These questions cover main functional domains such as vision, hearing, mobility, cognition, affect (anxiety and depression), pain, fatigue, communication and upper body functioning – domains which are driven by the World Health Organization International Classification of Functioning, Disability and Health classification. Twenty-two countries have to date incorporated WG questions in their surveys or censuses – in some cases creating a baseline, in others (such as Uganda), starting to monitor the same disability indicators over time.

It is clear that the WG is making significant progress in ensuring internationally comparable basic data on disability is reported and simple, short and easy questions are in place to be readily incorporated into national systems. It came as a surprise to many that a second survey was launched earlier this year, also focused on disability data.

“The motivation for the Model Disability Survey (MDS) project of the World Health Organization and the World Bank arose from the recognition, clearly enunciated in the 2012 joint World report on disability, that there is a worldwide need to collect comprehensive, comparable and relevant disability information that countries can use with confidence”. The MDS provides both a comprehensive (over 320 questions) and a brief survey (15 questions). The intention is to use the former as a general population survey with modules that aim to provide detailed and nuanced information on people with and without disabilities; the latter is a set of supplementary questions that can be added to any survey on any other topic.

I was curious to see how the two compared to each other. I found that:

• The short WG short set of disability questions is indeed present verbatim in the complete MDS.

• However, the brief MDS has (out of six WG questions) three questions that are similar but formulated differently, and three that are related to the WG questions but differ in scope. This will pose a problem when comparing disability data reported by the WG questions and the brief MDS.

•   Meanwhile most of the WG questions from the extended set (except for three) find their equivalents in the long MDS questions.

• Surprisingly, only 19 out of 40 of the brief MDS questions can be mapped back to the WG questions, and in a varying scope.

• Finally, the unique MDS questions are divided into sections on environmental factors, functioning, capacity and health conditions, and personal assistance and assistive products.

As a researcher with a doctorate in biological sciences, a disabled person, and a humanitarian, I applaud the detail and the depth of the recently introduced comprehensive MDS survey, and see the wealth of previously neglected data that could potentially be gathered on a particularly vulnerable group of people around the world. As a pragmatist, however, I doubt that the comprehensive MDS will be used as widely as existing household surveys, such as DHS, MICS or LSMS. Like these existing and well-established surveys, it is lengthy (120–150 minutes) and will take time – and a commitment to sustained funding over time – to plan, survey, recode and standardise. I fear that without committed and long-term resources, countries will carry out the brief MDS survey of 40 questions.

If that is the case, with my Joined-up Data Standards hat on, I cannot help but notice that WHO’s brief MDS stands in direct competition with the WG short set of questions.

Why?

The Washington Group has already made international reporting on disability a reality, with 22 countries to date incorporating and reporting on these questions in their censuses and surveys. Introducing the brief MDS as a direct and competing alternative negatively impacts the value of this achievement – the time and effort put into it over more than a decade, the extensive funding that supported the Group’s work, and investment in surveys with questions that are no longer useful.

So rather than competing and thereby compounding the issues around interoperable and comparable data, an alternative approach could be to continue using the WG set of questions for brief international reporting, and the complete MDS survey where countries can afford to conduct a full-scale disability survey within their borders with regular frequency. The bottom line is that surveys are costly. Getting them right, and getting usable data, can take years. To me, it’s essential that data collection and use is simplified and supported – and the processes around that should enable and encourage take-up. Introducing two different systems of collecting the same information stands against that and international organisations should make the data world easier instead of complicating it further.

This blog was written as a part of the Joined-up Data Standards project, a joint initiative between Development Initiatives and Publish What You Fund.