Some of the standard questions that come up in the field of development economics, which is concerned with determinants of poverty and policies to alleviate it, are as follows: Does microcredit alleviate poverty? Are policies of financial inclusion effective in helping the poor who are self-employed to save, invest, and raise their incomes? Did the Mahatma Gandhi National Rural Employment Guarantee Act (MNREGA)1 raise wages by providing an alternative source of employment for rural labourers in India? Is it availability of textbooks or mid-day meals or better health and sanitation that can improve the educational attainment of children from poor families in rural areas?
The main challenge in answering these questions is establishing a connection between cause and effect. If the questions sound straightforward and the approach sounds simple, then you have not encountered the word that is to academic seminars what spells are to Harry Potter and his friends: ‘identification’. The moment this comes up, a hushed silence descends on the room and the speaker launches an intense defence of how their analysis establishes a robust path from cause to effect. The problem is that in the real world everything changes at the same time and so it is hard to identify what is a cause and what is an effect. For example, if the poor save less, is it because low incomes cause low savings or is it the case that low savings cause low incomes? Similarly, it is hard to figure out the effect of one cause from that of another: maybe expansion of bank branches does facilitate saving but simply establishing a correlation between the two is not sufficient as some third factor (such as rising wages in the region) could be driving both, creating a spurious correlation. Theory can justify all these lines of arguments but cannot tell whether we should focus on policies that will raise income and therefore boost savings, or whether we should prioritise policies that will enhance savings opportunities for the poor and thereby eventually raise income.
The strength of randomised controlled trials (RCTs)
Standard empirical methods try to find some externally driven change in the environment that changes one factor, and then follows the line of causality. To continue with the example of savings, the Government of India has enthusiastically pushed the Jan Dhan Yojana2 in the last few years and one could try to see if those who were brought under this scheme were able to save more than those who were not. The trouble is that the government may have chosen to prioritise some areas over others for a reason (for example, they were poorer) and so this is not a clean comparison. Similarly, those who chose to have an account may be thriftier and so we cannot use their behaviour to judge how the average person would react to such a scheme. Finally, even if we see a positive effect on the savings and incomes of those who signed up for these accounts, it could well be that something else was going on that drove both trends – maybe wages were rising due to a rise in export demand, or due to a government programme of road construction.
This is where the strength of randomised controlled trials (RCTs) lies. RCTs is a technical phrase only heard within the confines of academic and policy worlds until 14 October 2019, when the Nobel Prize in economics was awarded to Abhijit Banerjee, Esther Duflo, and Michael Kremer for pioneering the use of RCTs in development economics. Following experimental trials in medicine, RCTs use a key insight that can be traced back to The Design of Experiments (1935) by Ronald Fisher, an eminent British statistician and geneticist: you select two groups that are similar and then randomly select one to receive the treatment (a drug, or a policy) being tested and then compare the outcome of this group (called the ‘treatment’ group) with that of the other group (called the ‘control’ group). If the difference is statistically significant, that is attributed to the treatment. The very design of the study eliminates the standard problems mentioned above.
The key innovation here is not coming up with the idea of randomisation – but applying it in real life with programmes and interventions that directly affect the lives of the poor. From testing drugs to placing government programmes as well as those carried out by NGOs (non-governmental organisations) on a randomised basis across villages, households, and organisations, takes quite a leap of imagination.
Using this method in economics has altered our views about what policies work and what do not. Take the example of microfinance, which serves more than 100 million people, mostly women, belonging to the poorer sections of society worldwide. Muhammad Yunus of Bangladesh is viewed as the leader of the microfinance movement for singlehandedly creating the most famous and successful microfinance institution (MFI) of the modern era, the Grameen Bank of Bangladesh. In 2006, Yunus and the Grameen Bank jointly won the Nobel Peace Prize for their contribution to reducing world poverty.
But, is microfinance effective in reducing poverty? If we merely compared those that have access to microfinance and those that do not, we would not get a satisfactory answer for the reasons mentioned above. Banerjee and Duflo, together with their colleagues, studied the impact of access to microfinance on the creation and profitability of small business as well as various measures of standard of living by working with Spandana, an MFI. They randomly selected half of around 100 slums of Hyderabad where a new branch was opened (the ‘treatment’ group), while in the remaining half of the slums no branch was opened (the ‘control’ group).
Before the programme was carried out, the control and treatment slums looked very similar in terms of population, average debt outstanding, businesses per capita, per capita expenditure, and literacy. What about the effect of the programme on the treatment slums? Small business investment and profits of pre-existing businesses increased, but consumption did not significantly rise. Durable goods expenditure increased, which suggests that loans were mostly used to purchase these. The study found no significant changes in health, education, or women’s empowerment. This research and a set of other studies in different countries have changed our views about the role of microfinance in alleviating poverty. While access to small loans is undoubtedly useful for expanding existing businesses and funding consumer durable goods, and may also help recipients to tide over temporary gaps between income flows and consumption needs, it is no longer seen as a magic bullet for solving the problem of poverty.
Where do RCTs fit into the broad scope of the field of development economics?
Development economics is concerned with a much broader set of issues than evaluating specific programmes relating to health, education, or credit, where RCTs have been most frequently applied.
A central concern has been the process of structural transformation of an economy – how the population moves from agriculture to industry and services – and accordingly, how the sectoral composition of national income changes. This process involves not just a movement of resources (land, labour, and capital) but also a process of institutional change – from informal personalised transactions to more formal contractual arrangements and markets, and associated changes in social norms. These are the kind of issues that Simon Kuznets and Arthur Lewis, two earlier recipients of the Nobel Prize, dealt with.
RCTs, however, can mostly be applied to study problems at the micro-level where the implementation of an individual programme – whether it is by the government or a private organisation (like a MFI or an NGO) – can be done in a randomised way that allows for a statistically satisfactory evaluation of the programme’s impact, as outlined earlier. Clearly, as with any other tool of analysis, RCTs cannot be applied to every question of interest within the field. And, as with any new method that attracts young researchers and research funding, there are grounds to worry that this will push out important research that uses other methods, including theory and empirical work that does not use RCTs. By their very nature, RCTs cannot be applied to broad macro-level issues or the more long-run aspects of development and institutional change.
However, one should note that a new generation of RCTs have come up that goes beyond evaluating programmes, and suggests that the frontier of their applicability can be pushed forward in creative ways. For example, a major focus of research in development economics has been to understand the contractual terms that prevail in land, labour, and credit markets in developing countries. A number of recent research papers have applied the tools of RCTs to vary terms of credit or tenancy, and have overcome some of the limitations of earlier work. Take the case of tenancy. My own work with Abhijit Banerjee and Paul Gertler showed how Operation Barga, a tenancy reform programme carried out in West Bengal in the late 1970s and early 1980s, changed tenancy arrangements and improved agricultural productivity. However, despite our best efforts, given the data we could not rule out the role of other policies that were carried out at the same time such as empowering the panchayats. In a recent RCT carried out in Uganda, the research team collaborated with the Bangladeshi NGO BRAC (Building Resources Across Communities) to induce randomised variation in real-life tenancy contracts. As part of their operations, BRAC leased plots of land to women from low socioeconomic levels who were interested in becoming farmers, effectively acting as the landlord. In the experiment, some tenants received a higher crop share (75%) and some a lower crop share (50%). The study, which was carried out by a group of researchers that included two of my former Ph.D. supervisers from the London School of Economics (LSE), Konrad Burchardi and Selim Gulesci, found that tenants with higher output shares used more inputs, cultivated riskier crops, and produced 60% more output relative to those in the control group. While these effects are reassuringly similar to those that we had found earlier, the nature of the new evidence ensures that the new study is not subject to the methodological limitations ours had to face.
Criticisms of RCTs from inside and outside the world of academic research
The main ‘inside’ criticisms of RCTs – from within the world of academic research (for example, by recent Nobel Laureate Angus Deaton) – are as follows.
First, while RCTs overcome some problems of evaluating individual programmes, the typically small sample size of these studies implies that the conclusions cannot be generalised to the whole population or extended to other environments. Moreover, there is the possibility that these studies may also be partly picking up the sheer effect of being observed by the researchers and the surveyors, which creates a bit of an artificial environment and therefore may give a biased picture of how the programme will work out when it is not being surveyed (the so-called ‘Hawthorne effect’).
Second, if some programme works well, we do not know if there is another programme that would have worked better.
Third, if a policy worked well, it is hard to infer the exact mechanism by which it worked – for example, does microfinance work by making credit more available or is it something that empowers women, or both?
There is some validity to each of these criticisms. However, every method has some limitations and to find a way forward one has to either come up with a better method or improve the existing method. Another promising direction is to harness the synergy of different methods – for example, it may be worth exploring how RCTs can be combined with other tools of economics, such as theory and simulation. Theory is good at coming up with alternative narratives that connect cause and effect, but it is not very good at determining what may be going on in a given environment. This is exactly as in medical science – theory gives us a first hunch as to what has happened while empirics are diagnostic tests which may confirm or disprove or modify the original hunch. A recent research trajectory that combines theoretical models with RCT evidence to carry out policy simulations that estimate the effect of hypothetical alternative policies tells us what else could work even better, as well as what the likely effect will be in a different environment.
Then there are ‘outside’ criticisms of RCTs.
Some wonder why academic economists should do policy evaluation. Should that not be left to policymakers? After all, as economists, we know the value of comparative advantage and specialisation. As much as science and engineering are different fields, should research not be separate from policy work, whether it is formulation of policy or its evaluation?
There is also some concern that, because RCTs require lots of funding, the missions of certain donor agencies and philanthropic organisations may distort the direction of research – as much as the profit motive of pharmaceutical companies can influence the agenda of medical research.
Then there are ethical considerations regarding experimenting on human subjects. These range from depriving those in the control group of a beneficial programme, to manipulating the behaviour of individuals in the treatment group, which raises questions of transparency and informed consent.
Another criticism is that, since policymaking happens in a political framework, to take a purely technocratic view about evidence-based policy and incremental improvements may be misguided at best, and at worst, the equivalent of putting band-aid on a serious injury.
Once again, there is some validity to each of these criticisms. But they provide a partial picture. Policymaking may be too important to be left to policymakers only. After all, we have seen too many instances of policy formulation that oversimplify problems and take a centralised one-size-fits-all approach. In the Indian context, some of the major policy shifts, such as demonetisation or goods and services tax (GST) implementation, or making the Aadhaar3 card mandatory, were done without any grounding in evidence or without first testing the waters. Yes, there are ethical considerations regarding the design of experiments, as well as the need for accountability regarding how well the research agenda fits the development priorities of a country. But that points to the need for developing a legal and ethical framework that governs research, not to abandon a particular method. It is also true that the kind of programmes that are studied offer incremental improvements but it is not the case that stopping doing these would unleash more major initiatives, whether on the part of the government or by other actors, including the people themselves.
To me one of the most significant legacies of the RCT research agenda is to put the importance of evidence at the centre of the table in the context of policy. Knowledge consists of knowing both what we know and what we don’t know. The demands of rigorous evidence make us acutely aware of the boundary between the two. Another welcome aspect of this research agenda is its emphasis on a bottom-up rather than top-down approach towards policymaking. The same policy may not work equally well everywhere or for everyone in the same place. Only evidence can help improve the effectiveness of policies by making them better suited to the specific needs of an area or a group of people. This can provide a much-needed corrective to the top-down, one-size-fits-all approach that, sadly, is a feature of centralised policymaking, whether in contemporary India or in the failed model of central planning.
Maitreesh Ghatak,
London School of Economics; IGC India
This article was originally published in Ideas For India
Stay updated with all the insights.
Navigate news, 1 email day.
Subscribe to Qrius
