Donor-driven information on results doesn’t work…
This is because there is no agreed common approach to results reporting.
- With few exceptions (such as the WHO’s Global Health Observatory) there are no international standards governing machine-readable indicators. Most providers of development cooperation make up their own indicators. Those using common indicators often rely on free-text descriptions, which are then difficult to compare.
- Most published data on results are distributed in documents, making it very difficult to compare across organisations and countries. The International Aid Transparency Initiative (IATI) has a perfectly adequate framework for structured reporting of indicators, but few publishers use it.
- Most providers will openly admit that it is not in their interest to admit failure.
- Most providers would hopefully also agree that they themselves are not the best judges of their own activities. Nor are expensive M&E consultants being paid from the same pot of money.
But, most importantly, while the reporting of activity-level outputs is a critical and necessary piece of information in assessing development effectiveness, relating outcomes and impacts to particular activities (let alone particular agencies) is a highly dubious venture. Most development activities take place in an environment where multiple interventions from multiple agencies are interacting with each other. If, for instance, I build a school, it would be presumptuous of me to attempt to measure its effect on the local economy without placing my efforts in a pot with all other initiatives and contextualising it with locally collected data covering a range of often complexly interrelated indicators.
Sub-national socio-economic indicators are a goldmine…
In most countries the national statistical system produces loads of data. In addition to budgets, censuses and household surveys, clinics and schools submit data to health and education management information systems. Agricultural extension officers do likewise. All the data that is required to assess the impact of aid is available in the neighbourhood of the intervention.
If top-down data on inputs and outputs from development activities could be joined up with bottom-up impact data collected through the national statistical system, development effectiveness would take a great leap forward.
That is, at least, the theory…
There is, however, a problem…
On the one hand most donors are not yet geocoding their data, nor are they releasing structured data on outputs. The IATI channel for this data is tried and tested, but uptake has been slow.
On the other hand, most local socio-economic data is currently not publicly available. This is because the culture of open data has not reached many governments, and development data is often viewed as a political hot potato.
Furthermore, much of the data (whether public or not) is of variable quality. Many countries do not have the resources and sufficient skilled personnel to insure routine and accurate collection of all required data. In a number of countries donor-funding of national statistics offices has been accompanied by conditions that have diverted data collection to meet specific donor requirements.
Towards a solution…
There is no quick fix to this problem, but a roadmap towards a solution is opening up. The Secretary General of the United Nations is spearheading a Data Revolution for Sustainable Development that seeks a step change in information available to decision makers, and their monitors, both nationally and sub-nationally.
The Data Revolution: Keeping the basics simple
What should happen next?
Donors can play an important role in this process.
- Firstly, by recognising that they cannot evaluate their own activities without data gathered in the environs of the point of delivery by the national statistical system, they will help drive demand for better collection and release of sub-national data in all recipient countries. This demand will lead to an awareness of the importance of funding initiatives aimed at improving the capacity of the national statistical system.
- Secondly, by recognising that the impact data that they require can only be credible if it is collected independently of their own activities, they will in turn realise the need to respect and support an independent and sustainable national statistical system.
- Thirdly, by publishing geocoded data on their activities to the IATI standard it will be possible to compare them not only with other aid programmes but also with domestic initiatives. This means a more rounded assessment of development can be conducted.