feeds from sacw.net
print versiono o o
Economics’ Biggest Success Story Is a Cautionary Tale
Field experiments now dominate development economics—often at the expense of the world’s poor.
By Sanjay G. Reddy | October 22, 2019
When I was a graduate student of economics in the early to mid-1990s, a new idea was just starting to emerge in the field of global development: using randomized controlled trials (RCTs), of the sort that had long been common in medicine, to assess efforts to assist the poor. One of the very first of these studies tested the impact of eradicating parasitic worms on school attendance among children, with the researchers picking schools randomly to determine not only how children in these schools were affected but also neighboring ones. Researchers have since tested the impact of placing additional teachers in a classroom or monitoring teachers’ attendance with cameras; the effect of access to bank or microfinance loans; and even the effect of specific appeals made by candidates in an election campaign on voting behavior.
The growing interest in RCTs has culminated in the awarding of the Nobel Memorial Prize in Economic Sciences last week to several of its pioneers: Esther Duflo, Abhijit Banerjee, and Michael Kremer. The RCTs they have promoted were described by the Royal Swedish Academy of Sciences as having come to “entirely dominate development economics.†The prize committee suggested the rise to centrality of this previously marginal idea was evidence of scientific progress and of a breakthrough that much better enabled us to “improve the lives of the worst-off people around the world.†We should all be glad if it were so simple. The fact that RCTs now so thoroughly shape development economics may be less a success story than a cautionary tale.
The RCT trend has been fueled by two factors: one from within economics, the other from outside it. Within the discipline, RCTs promised to address a problem that had bedeviled economists’ efforts to empirically assess development programs—namely, that the people who fared better when there was a change in circumstances were often those who were also more motivated or better positioned in some way to take advantage of it. There was no sure way of telling apart which interventions seemed to work because of such factors from which ones worked, well, because they worked. A premise among those who used statistical methods to address this problem had been that experiments on people were not possible. The “randomistas†—as they later came to be called—cheekily turned this idea on its head, proposing precisely to try such experiments.
The factor from outside of the discipline fueling the rise of RCTs was a widespread collapse of faith in the ability of public policies to durably change economic fates (culminating in the worldwide wave of structural adjustment policies in the 1980s and 1990s, bringing about austerity and market-oriented policy reforms in almost a hundred countries) and doubt too about the efficacy of international aid in the face of ongoing economic stagnation in large parts of the world. In this context, there was a growing interest in distinguishing what worked in development from what didn’t, with the idea that successful small-scale interventions could be made bigger by receiving adequate support from nongovernmental organizations, aid agencies, private foundations, impact investment funds, and governments. There was too an inclination to elevate explanations of development failure and success centered on individuals making the best of their circumstances, perhaps with the help of specific interventions (hence, for instance, a growing fascination with microfinance).
Against this background, an RCT wave swept the world. There have been a thousand or more trials by now, and some of these have informed funding decisions and policies on a national or international scale. Indeed, some aid agencies and government bodies have strongly preferred funding interventions that are validated by an RCT. A movement of so-called ethical altruists also argues that it is only sensible to give money to interventions that have been found to be high impact according to an RCT and as a consequence focuses on a very narrow range of development initiatives, such as deworming and malaria treatment. The RCT movement has even started to touch rich countries, with RCTs being applied to determine the efficacy of different schooling methods in the United States and elsewhere. (In fact, RCTs had originated in an earlier era with experiments on the behavioral effects of income security and health insurance schemes in the United States in the 1960s and 1970s, but this had been largely forgotten.)
RCTs grew from strength to strength, celebrated by the media as a clever idea leading to a revolution in how poverty could be addressed; endorsed by politicians, who were often instrumental in enabling trials to be implemented; and receiving massive support from private and public funding agencies. RCTs received very little criticism from within the profession for perhaps a decade and a half. Then around 2010 the dam broke, as other economists, both those working on development and those interested in statistical methods, including some of great eminence within the mainstream of the discipline, began to point to weaknesses in the randomistas’ arguments.
This countercharge from within the discipline has had three critiques: insight, reliability, and adequacy.
The insight critique contested the proposition that RCTs had revealed significant new facts or provided new understanding of development processes that would not have been had otherwise. RCTs take a long time and cost a lot (running quite easily into hundreds of thousands of dollars each). But closer inspection reveals that they most often merely provide a validation of common sense. Whereas at times randomization seemed to reveal something surprising (for instance, claiming to have shown that microfinance was less effective than many had assumed), in other instances it simply told us what had been long expected (for instance, that providing treatment for communicable diseases benefits the community at large). One such finding—that providing preventative public health treatments at low or no cost, or better yet with incentives, leads to an increase in the number of people willing to accept them—is cited by the prize committee as having led to a change in the received wisdom in favor of user fees in primary health. This gets the history quite wrong, since such fees had long before that lost favor, due in part to activists, including prominent economists such as Jeffrey Sachs, who who had made it a prominent focus of his advocacy. I know because I was myself involved in this debate in the late 1990s, when the World Bank, World Health Organization, and other institutions were still promoting them.
RCTs cannot reveal very much about causal processes since at their core they are designed to determine whether something has an effect, not how. The randomistas have attempted to deal with this charge by designing studies to interpret whether variations in the treatment have different effects, but this requires a prior conception of what the causal mechanisms are. The lack of understanding of causation can limit the value of any insights derived from RCTs in understanding economic life or in designing further policies and interventions. Ultimately, the randomistas tested what they thought was worth testing, and this revealed their own preoccupations and suppositions, contrary to the notion that they spent countless hours listening to and in close contact with the poor. It is not surprising that economists doing RCTs have therefore been centrally concerned with the effects of incentives on individual behavior—for instance, examining the idea that contract teachers who fear losing their jobs will be more effective than those with a guarantee of employment.
But valuable innovations in everyday life, whether on the small or large scale, are likely to result from explorations of a more open-ended kind. This requires that people experiment with the institutions of which they are a part, which is not the same as conducting randomized experiments on other people. Policies (and reforms of policies) that go beyond one dimension are essential in a complex environment. For instance, better schools are likely to result both from measures dealing with teachers’ employment and ones dealing with curriculum, community participation, and funding arrangements. RCTs simply cannot advise us on how best to combine all of these, let alone on how to think creatively about them. Better schools may also result from changes that result from improvements in other domains beyond the individual school—for instance, safer neighborhoods, better drug policy, or lessened poverty. The actions needed to achieve better outcomes may sometimes only be possible to undertake at a level going much beyond the locality. A good example is provided by the iodization of salt, which has contributed not only to better health but may also have improved educational outcomes.
And so it should not be a surprise that RCTs played no role at all in some of the greatest development successes of the past (including the creation of a free and universal public education system and widespread public health measures in the 19th and 20th centuries in the United States and other countries). In medicine, it has long been recognized that interactions between drugs, and treatments generally, require that the results of individual RCTs be acted on with great care. In addition, improving health requires combining medical knowledge in complex ways from both societal and individual levels (for instance, public health measures such as closing sewers and individual measures such as eating nutritiously).
Many of the most important findings in development economics have come from broad comparisons between cases. For instance, the finding that some countries achieved high health and educational outcomes at low incomes, and that this in turn resulted in much lower population growth and other benefits, came from comparing their experiences, not from fine-grained statistical tests of household behavior. There is still considerable scope for applying comparative studies of this kind to gain important insights. Indeed, if one needs a sophisticated statistical method to identify an effect, then its relevance may be doubtful.
The reliability critique contests the idea that RCTs provided a sure means—indeed the gold standard—for inferring, although in narrow terms, what worked in development. Those who have made the reliability critique, including eminent, statistically minded economists (a few of whom are also Nobel Prize winners), have argued that RCTs suffer from two problems of reliability. The first, external validity, concerns whether the estimate of the effect of a treatment from the place that the RCT is administered, even if it is accurate there, can be transferred elsewhere, given differences in the behaviors of different populations, as well as in the prevailing environmental, institutional, and social circumstances. For instance, public health information may influence behavior more where the government is trusted than where it is not. It may even have the opposite effect from that intended if the government is held in great suspicion. The second concern, internal validity, is about whether the results from a given context are really meaningful and accurate even there. RCTs are designed to measure the average effect of a treatment in a population and cannot generally tell us how it affects different parts of that population (in an extreme case, which is encountered frequently in medical trials, it may harm some people even as it creates a benefit on average). The effect of an intervention may moreover change over time even in a single place due to learning and behavioral responses. For these and other reasons, it is necessary to take care in interpreting what RCTs have actually measured and in employing their lessons, even in the very same place that they have been implemented.
Since the ultimate justification for RCTs is to inform policies on a larger scale, these are very serious problems. If the results of RCTs cannot be generalized, or only partially, then the cost they involve becomes still harder to justify. Moreover, RCT proponents themselves admit their method can be applied only to interventions that affect individuals or local communities and which can then be scaled up—with all of the already noted difficulties that involves. Therefore, even on their own terms, RCTs can’t inform many, if not most, of the central questions in contemporary global development, especially those that require policies that go beyond widely replicating what works locally. For instance, although an RCT may help inform what causes a migrant to leave her home or what aids her integration elsewhere, it cannot tell us how to organize migration policy between countries. Indeed, although evidence is an important aid to policymaking, it is not sufficient. Designing appropriate policies and prioritizing among them requires reference to values to determine what is the appropriate trade-off between different goals and taking into account the political and social consequences of implementing them. For instance, in a post-conflict society, a policy intervention that risks being perceived as prioritizing the interests of one group, whatever an RCT may suggest would be its average impact on the persons benefitted, may be imprudent or even dangerous to implement.
To add to these concerns from within the field of development came another, largely from outside of it: an ethical worry concerning the acceptability of experimenting on people. RCTs involve treating people as means rather than ends (contrary to the spirit of the famous dictum of Immanuel Kant) with the idea that the knowledge gained will be of broader public benefit. As noted, it is not at all clear that such knowledge is gained, but even if so, the approach of RCTs is to experiment on people rather than to work with them. It is not a surprise that nearly every RCT involves treating poor people, usually also in poor countries, as their subjects (or is that objects?). It is often argued that the participants in trials receive benefits that they would not otherwise receive and therefore can have no complaint, but in an unjust world, the doling out of benefits on a random basis (when, for instance, it is known that some are poorer and more deserving or in more urgent need than others) can be hard to accept and may even lead to potential harms. For instance, a study on how economic incentives affect drinking by pedicab drivers in India found that they shifted their consumption from the daytime to the evening. One wonders how this may have affected intra-household violence or other outcomes not considered in the study.
Indeed, it is not clear that the randomistas have adopted the ethical protocols that have long been standard in medical research to ensure meaningful informed consent by individuals, prevent harms to subjects, and ensure that a trial is stopped when there is reason to believe either that it is causing such harms or that there are evident benefits that should be offered to all. These ethical concerns, and associated attitudes to the poor as suitable for experimentation, have been barely at all discussed by the randomistas (and do not figure in the discussion of their work by the prize committee). The administration of RCTs has suffered from more than a whiff of neocolonial attitudes. Arguably, all of the difficulties of RCTs stem from a single source: a failure to recognize the full personhood of those who are affected by interventions. Research and policy would be improved by having a less caricatured view of people, whether in deciding when it is alright to experiment on them, what motivates them and what mistakes they are likely to make, how it is determined “what works,†or in acting on these conclusions.
If RCTs now “entirely dominate†development economics, or worse, provide the basis for development policymaking, that is no cause for celebration. The roaring success of the randomistas tells us most of all about the historical moment in which they came to prominence: one in which defeatism or cynicism about public initiatives on a larger scale has been replaced by a focus on what works at the level of individuals and communities. But even there, what does work, really, remains an open question. The difficult question of how to fix broken institutions and help societies function better requires going beyond a biomedical metaphor of taking the right pill. Nobel or not, the debate must continue.
Sanjay G. Reddy is an associate professor of economics at the New School for Social Research in New York. Twitter: @sanjaygreddy
SEE ALSO:
Reddy, Sanjay. 2012. “Randomise This! On Poor Economics.†Review of Agrarian Studies 2 (2): 60–73
o o o
Development and Chnage, Volume50, Issue3, May 2019, Pages 735-762
All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development
by Florent Bédécarrats, Isabelle Guérin, François Roubaud
First published: 06 December 2017
https://doi.org/10.1111/dech.12378
ABSTRACT
Randomized control trials (RCTs) have a narrow scope, restricted to basic intervention schemes. Experimental designs also display specific biases and political uses when implemented in the real world. Despite these limitations, the method has been advertised as the gold standard to evaluate development policies. This article adopts a political economy approach to explore this paradox. It argues that the success of RCTs is driven mainly by a new scientific business model based on a mix of simplicity and mathematical rigour, media and donor appeal, and academic and financial returns. This in turn meets current interests and preferences in the academic world and the donor community.
https://onlinelibrary.wiley.com/doi/full/10.1111/dech.12378
o o o
https://africasacountry.com/2019/10/the-poverty-of-poor-economics
The poverty of poor economics
By Grieve Chelwa and Seán Muller
The winners of the Nobel Prize in Economics experiment on the poor, but their research doesn’t solve poverty.
This past Monday, the Swedish Academy of Sciences awarded the “Nobel Prize†in economics to Abhijit Banerjee, Esther Duflo and Michael Kremer for “their experimental approach to alleviating global poverty.†The prize in economics is not one of the original prizes endowed by Swedish armaments manufacturer Alfred Nobel. It was established in 1969 as a tag along and is endowed by the Bank of Sweden.
Banerjee and Duflo teach at MIT while Kremer is at Harvard. The trio have been at the forefront of pushing the use of randomized control trials (RCTs) in the sub-discipline of economics known as development economics. And partly as the result of their efforts, an ecosystem has developed in which the vampire squids with tentacles of influence across the globe are the “poverty action lab†JPAL, 3ie, and the World Bank’s development impact evaluation group (DIME). The main idea behind their work is that RCTs allow us to know what works and doesn’t work in development because of its “experimental†approach. RCTs are most well-known for their use in medicine and involve the random assignment of interventions into “treatment†and “control†groups. And just like in medicine, so the argument goes, RCTs allow us to know which development pill to swallow because of the rigor associated with the experimental approach. Banerjee and Duflo popularized their work in a 2011 book Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty.
Even though other Nobel prize awards often attract public controversy (peace and literature come to mind), the economics prize has largely flown under the radar with prize announcements often met with the same shrugging of the shoulders as, for example, the chemistry prize. This year has however been different (and so was the year that Milton Friedman, that high priest of neoliberalism, won).
A broad section of commentary, particularly from the Global South, has puzzled over the Committee’s decision to not only reward an approach that many consider as suffering from serious ethical and methodological problems, but also extol its virtues and supposed benefits for poor people.
Many of the trio’s RCTs have been performed on black and brown people in poor parts of the world. And here, serious ethical and moral questions have been raised particularly about the types of experiments that the randomistas, as they are colloquially known, have been allowed to perform. In one study in western Kenya, which is one-half of the epicenter of this kind of experimentation, randomistas deliberately gave some villages more money and others less money to check if villages receiving less would become envious of those receiving more. The study’s authors, without any sense of shame, titled their paper “Is Your Gain My Pain?†In another study in India, the other half of the epicenter, researchers installed intrusive cameras in class rooms to police teacher attendance (this study was actually favorably mentioned by the Swedish Academy). There are some superficial rationalizations for this sort of thing, but studies of this kind—and there are many—would never have seen the light of day had the experimental subjects been rich Westerners.
There are also concerns around the extractive nature of the RCT enterprise. To execute these interventions, randomistas rely on massive teams of local assistants (local academics, students, community workers, etcetera) who often make non-trivial contributions to the projects. Similarly, those to be studied (the poor villagers) lend their incalculable emotional labor to these projects (it is often unclear whether they have been adequately consulted or if the randomistas have simply struck deals with local officials). The villagers are the ones that have to deal with all the community-level disruptions that the randomistas introduce and then leave behind once they’ve gone back to their cushy lives in the US and Europe.
And while there is an increasing amount of posturing to compensate for this exploitation, with some researchers gushing about how they and their “native assistants†are bosom buddies, the payoffs of the projects (lucrative career advancement, fame, speaking gigs, etcetera) only ever accrue to the randomistas and randomistas alone. The extreme case is obviously this week’s award.
Beyond the ethics of the Nobel winners, their disciples, and the institutions they have created in their image are two serious methodological problems that fundamentally undermine their findings.
The first is that the vast majority of studies conducted using these methods (our rough guess is more than 90%) have no formal basis for generalization. In other words, there is no basis to believe that the findings of these studies can be applied beyond the narrow confines of the population on which the experiments are undertaken. This is simply fatal for policy purposes.
The prize giving committee addresses this only in passing by saying that “the laureates have also been at the forefront of research on the issue of [whether experimental results apply in other contexts].†This is misleading at best and false at worst. There are some advocates of randomised trials who have done important research on the problem, but the majority of key contributions are not by advocates of randomised trials and the three awardees have been marginal contributors. The more important point is simply that if the problem of whether experimental results are relevant outside the experiment has not been resolved, how can it be claimed that the trio’s work is “reducing world poverty?â€
The second contradiction is more widely understood: despite the gushing headlines in the Western press, there is simply no evidence that policy based on randomized trials is better than alternatives. Countries that are now developed did not need foreign researchers running experiments on local poor people to grow their economies. There is ample historical evidence that growth, development and dramatic reductions in poverty can be achieved without randomised trials. Randomistas claim that their methods are the holy grail of development yet they have not presented any serious arguments to show why theirs is the appropriate response. Instead, the case that such methods are crucial for policy is largely taken for granted by them because they think they are doing “science.†But while they are certainly imitating what researchers in various scientific disciplines do, the claim that the results are as reliable and useful for economic and social questions is unsupported. It is instead a matter of blind faith—as with the conviction many such individuals appear to have of a calling to save the poor, usually black and brown, masses of the world.
We do not have a view on whether these individuals ought to have been awarded the prize—prizes are usually somewhat dubious in their arbitrariness and historical contingency. But the claims made about the usefulness and credibility of the methods employed are concerning, both because they are unfounded and because they inform a missionary complex that we believe is more of a threat to the progress of developing countries than it is an aid.
o o o
Open Democracy, 18 October 2019
Impoverished economics? Unpacking the economics Nobel Prize
When the world is facing large systemic crises, why is the economics profession celebrating small technical fixes?
by Ingrid Harvold Kvangraven
This week it was announced that Abhijit Banerjee, Esther Duflo and Michael Kremer won the Economics Nobel Prize (or more accurately: the ‘Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel’). The trio of economists were awarded the prize for "their experimental approach to alleviating global poverty†.
On social media and in mainstream newspapers, there was an exceptional level of praise for the laureates, reflecting their existing rockstar status within development economics. The Financial Times even claimed that the Nobel “will help restore [the] profession’s relevance†. However, the widespread calls for celebration need to be considered with a cautionary counterweight.
The experimental approach to poverty alleviation relies on so-called Randomized Control Trials (RCTs). Inspired by studies in medicine, the approach targets specific interventions to a randomly selected group (schools, classes, mothers, etc), and then compares how specific outcomes change in the recipient group versus those who did not receive the treatment. As the groups are assumed to be otherwise similar, the difference in outcomes can be causally attributed to the intervention.
While the laureates were first pioneering this work in the 1990s in Kenyan schools, the approach is now widely considered the new “gold standard†in development economics, also sometimes simply called “New Economics†. The approach has become enormously influential among governments, international agencies and NGOs. The body of work pioneered by the laureates, or the randomistas as they are sometimes called, is meant to alleviate poverty through simple interventions such as combating teacher absenteeism, through cash transfers, and through stimulating positive thinking among the people living in poverty. Sound good so far?
While the laureates’ approach to poverty research and policy may seem harmless, if not laudable, there are many reasons for concern. Both heterodox and mainstream economists as well as other social scientists have long provided thorough critique of the turn towards RCTs in economics, on philosophical, epistemological, political and methodological grounds. The concerns with the approach can be roughly grouped into questions of focus, theory, and methodology.
Focus: tackling symptoms and thinking small
The approach that is being promoted is concerned with poverty, not development, and is thus a part of the larger trend in development economics that is moving away from development as structural transformation to development as poverty alleviation. This movement towards “thinking small†is a part of a broader trend, which has squeezed out questions related to global economic institutions, trade, agricultural, industrial and fiscal policy, and the role of political dynamics, in favor of the best ways to make smaller technical interventions.
The interventions considered by the Nobel laureates tend to be removed from analyses of power and wider social change. In fact, the Nobel committee specifically gave it to Banerjee, Duflo and Kremer for addressing “smaller, more manageable questions,†rather than big ideas. While such small interventions might generate positive results at the micro-level, they do little to challenge the systems that produce the problems.
For example, rather than challenging the cuts to the school systems that are forced by austerity, the focus of the randomistas directs our attention to absenteeism of teachers, the effects of school meals and the number of teachers in the classroom on learning. Meanwhile, their lack of challenge to the existing economic order is perhaps also precisely one of the secrets to media and donor appeal, and ultimately also their success.
The lack of engagement with the conditions that create poverty has led many critics to question to what extent RCTs will actually be able to significantly reduce global poverty. A further consequence of this impoverished economics is that it limits the types of questions we can ask, and it leads us “to imagine too few ways to change the world†.
Theory: methodological individualism lives on
In a 2017 speech, Duflo famously likened economists to plumbers. In her view the role of an economist is to solve real world problems in specific situations. This is a dangerous assertion, as it suggests that the “plumbing†the randomistas are doing is purely technical, and not guided by theory or values. However, the randomistas’ approach to economics is not objective, value-neutral, nor pragmatic, but rather, rooted in a particular theoretical framework and world view – neoclassical microeconomic theory and methodological individualism.
The experiments’ grounding has implications for how experiments are designed and the underlying assumptions about individual and collective behavior that are made. Perhaps the most obvious example of this is that the laureates often argue that specific aspects of poverty can be solved by correcting cognitive biases. Unsurprisingly, there is much overlap between the work of randomistas and the mainstream behavioral economists, including a focus on nudges that may facilitate better choices on the part of people living in poverty.
Another example is Duflo’s analysis of women empowerment. Naila Kabeer argues that it employs an understanding of human behavior “uncritically informed by neoclassical microeconomic theory.†Since all behavior can allegedly be explained as manifestations of individual maximizing behavior, alternative explanations are dispensed with. Because of this, Duflo fails to understand a series of other important factors related to women’s empowerment, such as the role of sustained struggle by women’s organizations for rights or the need to address unfair distribution of unpaid work that limits women’s ability to participate in the community.
Note that there is nothing embedded in RCTs that forces randomistas to assume individuals are rational optimizing agents. These assumptions come from the economics tradition. This is therefore not a critique of RCTs per se, but of the way RCTs are employed in the laureates’ work and in most of mainstream economics.
Method: If you didn’t randomize it, is it really knowledge?
While understanding causal processes is important in development economics, as in other social science disciplines, RCTs do so in a very limited way. The causal model underlying RCTs focuses on causal effects rather than causal mechanisms. Not only do RCTs not tell us exactly what mechanisms are involved when something works, they also do not tell us whether the policy in question can be reliably implemented elsewhere. In order to make such a judgement, a broader assessment of economic and social realities is unavoidable.
Assuming that interventions are valid across geographies and scale suggests that micro results are independent of their macroeconomic environment. However, while “effects†on individuals and households are not separate from the societies in which they exist, randomistas give little acknowledgement to other ways of knowing about the world that might help us better understand individual motivations and socio-economic situations. As it is difficult to achieve truly random sampling in human communities, it is perhaps not surprising that when RCTs are replicated, they may come to substantially different results than the original.
Not only do RCTs rarely have external validity, but the specific circumstances needed to understand the extent to which the experiments may have external validity are usually inadequately reported. This has led even critics within the mainstream to argue that there are misunderstandings about what RCTs are capable of accomplishing. A deeper epistemological critique involves the problematic underlying assumption that there is one specific true impact that can be uncovered through experiments.
Recent research has found that alternative attempts to assess the success of programs transferring assets to women in extreme poverty in West Bengal and Sindh have been far superior to RCTs, which provide very limited explanations for the patterns of outcomes observed. The research concludes that it is unlikely that RCTs will be able to acknowledge the central role of human agency in project success if they confine themselves to quantitative methods alone.
There are also serious ethical problems at stake. Among these are issues such as lying, instrumentalizing people, the role of consent, accountability, and foreign intervention, in addition to the choice of who gets treatment. While ethical concerns regarding potential harm to groups is discussed extensively in the medical literature, it receives less attention in economics, despite the many ethically dubious experimental studies (e.g. allowing bribes for people to get their drivers’ licences in India or incentivizing Hong Kong university students into participation in an antiauthoritarian protest). Finally, the colonial dimensions of US-based researchers intervening to estimate what is best for people in the Global South cannot be ignored.
Why it matters: limits to knowledge and policy-making
There will always be research that is more or less relevant for development, so why does it matter what the randomistas do? Well, as the Nobel Committee stated, their “experimental research methods now entirely dominate development economics†. A serious epistemological problem arises when the definition of what rigour and evidence means gets narrowed down to one single approach that has so many limitations. This shift has taken place over the past couple of decades in development economics, and is now strengthened by the 2019 Nobel Prize. As both Banerjee and Duflo acknowledged in interviews after the prize was announced, this is not just a prize for them, but a prize for the entire movement.
The discipline has not always been this way. The history of thought on development economics is rich with debates about how capital accumulation differs across space, the role of institutions in shaping behavior and economic development, the legacies of colonialism and imperialism, unequal exchange, the global governance of technology, the role of fiscal policy, and the relationship between agriculture and industry. The larger questions have since been pushed out of the discipline, in favor of debates about smaller interventions.
The rise of the randomistas also matters because the randomistas are committed to provoke results, not just provide an understanding of the situations in which people living in poverty find themselves. In fact, it is one of their stated goals to produce a “better integration between theory and empirical practice†. A key argument by the randomistas is that “all too often development policy is based on fads, and randomised evaluations could allow it to be based on evidence†.
However, the narrowness of the randomized trials is impractical for most forms of policies. While RCTs tend to test at most a couple of variations of a policy, in the real world of development, interventions are overlapping and synergistic. This reality recently led 15 leading economists to call to “evaluate whole public policies†rather than assess “short-term impacts of micro-projects,†given that what is needed is systems-level thinking to tackle the scale of overlapping crises. Furthermore, the value of experimentation in policy-making, rather than promoting pre-prescribed policies, should not be neglected.
The concept of “evidence-based policy†associated with the randomistas needs some unpacking. It is important to note that policies are informed by reflections on values and objectives, which economists are not necessarily well-suited to intervene in. Of course, evidence should be a part of a policy-making process, but the pursuit of ineffective policies is often driven by political priorities rather than lack of evidence.
While randomistas might respond to this by arguing that their trials are precisely meant to de-politicize public policy, this is not necessarily a desirable step. Policy decisions are political in nature, and shielding these value judgements from public scrutiny and debate does little to strengthen democratic decision making. Suggesting that policy-making can be depoliticized is dangerous and it belittles the agency and participation of people in policy-making. After all, why should a policy that has been proven effective through an RCT carry more weight than, for example, policies driven by people’s demands and political and social mobilisation?
While the Nobel Prize does leave those of us concerned with broader political economy challenges in the world anxious, not everything is doom and gloom. Firstly, the Nobel directs attention to the persistence of poverty in the world and the need to do something about it. What we as critical development economists now need to do is to challenge the fact that the Prize also legitimizes a prescriptive view of how to find solutions to global problems.
Secondly, the fact that a woman and a person of color were awarded a prize that is usually reserved for white men is a step forward for a more open and inclusive field. Duflo herself recognizes that the gender imbalance among Nobel Prize winners reflects a "structural" problem in the economics profession and that her profession lacks ethnic diversity.
However, it is obvious that to challenge racism, sexism and Eurocentrism in economics, it is not enough to simply be more inclusive of women and people of color that are firmly placed at the top of the narrow, Eurocentric mainstream. To truly achieve a more open and democratic science it is necessary to push for a field that is welcoming of a plurality of viewpoints, methodologies, theoretical frameworks, forms of knowledge, and perspectives.
This is a massive challenge, but the systemic, global crises we face require broad, interdisciplinary engagement in debates about possible solutions.
Thanks to Carolina Alves, Devika Dutt, Minna Lehtinen and Farwa Sial for helpful discussions on these issues and comments on a previous draft.
o o o
Macro dimensions of poverty brushed aside
by Pritam Singh (Visiting Fellow, Wolfson College, University of Oxford)
A serious methodological flaw of the RCT approach that has been pointed out by ethnographic studies of poverty is that its overemphasis on quantitative method has deprived it of the insights of qualitative methods. There is some recognition of the flaw by admitting that a mixed method approach can be useful, but the approach remains hostile to qualitative methods.
THE award of the 2019 Nobel Prize in Economics to Abhijit Banerjee and Esther Duflo of the Massachusetts Institute of Technology (MIT) and Michael Kremer of Harvard University deserves not only to be celebrated but also critically reflected upon.
Duflo is only the second woman economist to win this prize since it was instituted in 1969. The first was Elinor Ostrom for her analysis of economic governance, especially relating to ‘common’ ecological resources, thus raising the status of ecological economics. Duflo (46), is also the youngest-ever winner of this prize along with being the only woman among this year’s 16 Nobel Prize winners. Gender bias remains an old problem of this prize. The most celebrated woman economist of all times, Joan Robinson of Cambridge University, was never given this prize due to ideological reasons though her work on imperfect competition has been pioneering in breaking the assumptions of perfectly competitive markets.
It has been my experience on teaching development economics that whenever I taught the topic of gender bias in economic theory and policy, the women students have loved it and have felt inspired while most, not all, men students feel uncomfortable about being told that even the central concept of macro-economics i.e. gross domestic product (GDP) is gender-biased because it is based on measuring marketable goods and services, and consequently ignores women’s work in the household.
Abhijit Banerjee, Duflo’s husband and PhD supervisor, is a former student of Jawaharlal Nehru University. The faculty and students of this university have been under vicious attack for many years from the government of their own country. One hopes that this prize will make those who govern higher education in India feel proud of this university and value its academic resources — the faculty and the students.
The prize honours the field of development economics and draws attention to the global challenge of reducing poverty which continues to damage the lives of millions. Globally, more than 700 million people live in extreme poverty even if one accepts the questionable World Bank definition of poverty as living on less than $1.90 per day. Many millions live just marginally above this poverty line and are categorised as ‘vulnerable poor’. One in three children is malnourished, and most children in the developing world leave school without basic skills in reading, writing and mathematics.
Banerjee and Duflo have done work on India, and Kremer on Africa, especially Kenya. This prize will open more doors for their work, but it is also going to open to much sharper scrutiny than it has been subjected to so far. I read their book, Poor Economics: A radical rethinking of the way to fight global poverty, soon after it came out. I was impressed by their style but unconvinced by their argument.
The most serious criticism, from a methodological point of view, is that their approach individualises poverty and the route out of poverty. Since their randomised controlled trials (RCTs) method is borrowed from clinical medicine, one can take the example from that field to explain the problem with this method. A doctor can individualise an illness (say obesity), prescribe some medicine which can provide temporary relief to the patient, important in itself, but if that illness (obesity) is widespread, it will show a serious weakness of the medical profession if it were not to look into the general conditions of society that lead to obesity, namely, the pattern of food consumption, the nature of work and employment and the marketing of food products etc. As we cannot throw the entire blame for obesity on to an obese person, it is intellectually and morally flawed to blame a poor person for his/her poverty.
The RCT method believes that the problem a poor person faces involves small modifications (called ‘interventions’) in the existing methods of his/her dealing with the problem, and the question is merely of testing which small modifications will eventfully work. The method, therefore, celebrates ‘what works’ and gives the false impression of being very practical and not theoretical.
A still higher order of methodological problem emerges when a suggested intervention is ‘scaled up’ i.e. generalised without considering different contexts. The macro dimensions of poverty i.e. structure of property relations, the pattern of income distribution, the unequal bargaining powers between employers and employees, the nature of patriarchal relations, the hierarchy of caste relations and identity issues of majority-minority are just brushed aside as ‘confounding’ variables.
The other serious methodological flaw of the RCT approach that has been pointed out by ethnographic studies of poverty is that its overemphasis on quantitative method has deprived it of the insights of qualitative methods. There is some recognition of this flaw by admitting that mixed method approach can be useful, but the approach remains intrinsically hostile to qualitative methods. The econometricians have attacked the RCT approach from the opposite angle i.e. identification problem in selecting the samples for testing the effectiveness of chosen interventions. Moral philosophers have raised questions of ethics in choosing the poorest Third World people for experimentation.
The reason this approach is popular with politicians, aid agencies and global policy-makers is that they want to see quick results of their specific interventions rather than troubling themselves with the structural causes of poverty and the transformational changes required to deal with mass poverty.
o o o
Alleviating poverty with experimental research: The 2019 Nobel laureates
Oriana Bandiera 21 October 2019
The 2019 Nobel Prize in Economic Sciences has been jointly awarded to Abhijit Banerjee, Esther Duflo, and Michael Kremer “for their experimental approach to alleviating global poverty†. This column discusses the new laureates’ vision and their common interest in both understanding and addressing the persistence of poverty and the huge differences in living standards across countries.
https://voxeu.org/article/alleviating-poverty-experimental-research-2019-nobel-laureates
o o o
The limitations of randomised controlled trials
Angus Deaton, Nancy Cartwright 09 November 2016
In recent years, the use of randomised controlled trials has spread from labour market and welfare programme evaluation to other areas of economics, and to other social sciences, perhaps most prominently in development and health economics. This column argues that some of the popularity of such trials rests on misunderstandings about what they are capable of accomplishing, and cautions against simple extrapolations from trials to other contexts.
https://voxeu.org/article/limitations-randomised-controlled-trials
o o o
Randomized control trials may not suit India’s social schemes
by Indira Rajaraman (Livemint, 31 Oct 2019)
What works for a small-scale NGO-style intervention may not help the state’s implementation of it without elaborate checks
o o o
Journal of Contemporary African Studies, Volume 36, 2018 - Issue 4: Special Issue: African Perspectives on Corruption
When are experiments corrupt?
by Nimi Hoffmann
ABSTRACT
Between 2010 and 2011, the UK’s Department for International Development (DFID) funded a large-scale experiment on Kenyan schools. The policy experiment tested the effects of cutting teachers’ salaries and hiring them on short-term contracts, but the intervention failed after sustained opposition from teacher unions and parent associations. This article critically revisits the narrative of how this experiment was designed, implemented and interpreted, finding evidence that the experiment violated empirical logic. It examines whether the theory of neopatrimonialism can explain the ways in which vested interests may have undermined the empirical logic of the experiment. By doing so, the analysis tests the explanatory utility of neopatrimonialism, and casts light on the conditions under which policy experiments in Africa may be anti-empirical.
https://www.tandfonline.com/doi/full/10.1080/02589001.2019.1579893
