Literature tells us there are many dimensions of public policy success, and different actors in the policy process will likely focus on different dimensions. This paper asks how different actors in the policy process view policy success, and how much their views differ. It finds evidence that actors devising policy plans— designers—view success narrowly, as achieving near-term, programmed goals; whereas other actors involved in advocating for, authorizing, and implementing policies have a broader success perspective, paying more attention to non-program criteria like long-term impact, distributional and endurance success, and intertemporal gains that manifest in the way policies grow capability, political support, stakeholder satisfaction, and process legitimacy. Such finding raises a question about how policy objectives are determined when actors disagree, given that literature also tells us that policies are more likely to succeed when actors agree on what success is and how to achieve it.
Public policy work is hard, especially when one works in developing countries. It is even difficult to define what success looks like, and thus how to manage towards success. Literature helps manage such difficulty, providing studies that define the concept and show how it can be achieved. A core message from such is that success is multi-dimensional, and practitioner need to focus on multiple criteria when doing their policy work. But what dimensions and criteria matter? And do development practitioners really adopt this multi-dimensional view? Tackling such questions, the current paper reviews 45 applied studies from the public policy, project management and development evaluation literatures to see what they identify as key success criteria and if the practical studies (about development evaluation) are in sync with the more academic messages. Reading across all three literatures, I identify 30 potential success criteria in 6 categories or dimensions (program, impact and endurance, capability, political, stakeholder, and process). I find that the development evaluation literature focuses on a narrow set of 7 criteria, mostly in one dimension (program success) as compared to broader perspectives in the other literatures. This suggests that development practitioners have a narrow view on success, which is out of step with academic views on the topic. A conclusion proposes a broader approach for these practitioners.
Officials working on public policies must answer questions like ‘What does policy success mean?’ and ‘How should I pursue policy work in order to achieve success?’ These are difficult questions, but there are ways to respond. One way draws on what I call the program logic of policy success, which suggests that: (i) Success requires efficiently meeting goals that stakeholders view as relevant, (ii) by doing work focused on impacting high-level objectives through programs that deliver promised time-sensitive outputs and outcomes according to a clear, logical plan. I believe this logic dominates the global public policy community, as ‘the way’ officials and organizations should think about and do policy work. This paper tests such belief, showing that officials do think in this way and that this thinking is influenced by common budgeting and evaluation mechanisms. I conclude by asking if this way of thinking poses any concerns, especially if it biases policy organizations to produce some kinds of policy success and not others.
People management has an important role to play in improving public agency performance. This paper argues that a ‘Route Y’ managerial approach focused on supporting the empowered exercise of employee judgment will in many circumstances prove superior to conventional reform approaches steeped in ‘Route X’ monitoring and incentives. Returns to Route Y are greater when employees are or can become more “mission motivated” – that is, aligned with the goals of the agency in the absence of monitoring and extrinsic incentives. Returns to Route Y are also greater when monitoring is incomplete or otherwise likely to unproductively distort effort, thus lowering the returns to using performance-linked rewards and penalties. I argue that education systems are one (but far from the only) setting where Route Y is a lever worth focusing on in efforts to improve public performance in the developed and developing world alike.
As with all public policy work, education policies are demanding. Policy workers need to ‘know’ a lot—about the problems they are addressing, the people who need to be engaged, the promises they can make in response, the context they are working in, and the processes they will follow to implement. Most policy workers answer questions about such issues within the structures of plan and control processes used to devise budgets and projects. These structures limit their knowledge gathering, organization and sense-making activities to up-front planning activities, and even though sophisticated tools like Theories of Change suggest planners ‘know’ all that is needed for policy success, they often do not. Policies are often fraught with ‘unknowns’ that cannot be captured in passive planning processes and thus repeatedly undermine even the best laid plans. Through a novel strategy that asks how much one knows about the answers to 25 essential policy questions, and an application to recent education policy interventions in Mozambique, this paper shows that it is possible to get real about unknowns in policy work. Just recognizing these unknowns exist—and understanding why they do and what kind of challenge they pose to policy workers—can help promote a more modest and realistic approach to doing complex policy work.
In March 2021, the Confederation of African Football’s President, Patrice Motsepe, insisted that “An African team must win the World Cup in the near future.” This visionary statement is infused with hope—not just for an African World Cup victory but for a fuller future in which African men’s soccer competes with world soccer’s elite. This paper asks if there is any chance of this happening. It suggests a simple method to assess how a country competes as both a ‘participant’ and a ‘rival’ and uses this method to examine how Africa’s top countries compete in world soccer. This analysis points to a gap between such countries and the world’s best, which has grown in recent decades—even though some African countries do compete more over time. The paper concludes by suggesting that Africa’s hope of winning the World Cup is not impossible but demands more active work, focused particularly on ensuring top African countries compete with more high-quality competition more often. The conclusion also suggests that the research approach might be relevant beyond a study of African soccer. It could particularly help shed light on how well African countries compete (as participants and rivals) in the world economy.
It matters if public policies succeed in solving societal problems, but a dominant narrative holds that policies fail ‘often’. A large-sample study discussed in this paper suggests that this is not accurate, however. The most common policy result in this study is more ambiguous—what I call ‘successful failure’. Such result is achieved when a policy delivers enough low-level, short-term product to promise success, but ultimately (and repeatedly) fails to contribute to sustained high-level, long-term impact (addressing the problems citizens care about). Such ‘successful failure’ is endemic to public policy work, and a more pernicious result than outright failure: It allows policy design and implementation actors to associate with incomplete near-run success but insulate themselves from future failure (which they blame on factors and actors beyond their control) and simultaneously enjoy repeated demand for work (because problems are never really solved).
The enthusiasm for the potential of RCTs in development rests in part on the assumption that the use of the rigorous evidence that emerges from an RCT (or from a small set of studies identified as rigorous in a “systematic” review) leads to the adoption of more effective policies, programs or projects. However, the supposed benefits of using rigorous evidence for “evidence based” policy making depend critically on the extent to which there is external validity. If estimates of causal impact or treatment effects that have internal validity (are unbiased) in one context (where the relevant “context” could be country, region, implementing organization, complementary policies, initial conditions, etc.) cannot be applied to another context then applying evidence that is rigorous in one context may actually reduce predictive accuracy in other contexts relative to simple evidence from that context—even if that evidence is biased (Pritchett and Sandefur 2015). Using empirical estimates from a large number of developing countries of the difference in student learning in public and private schools (just as one potential policy application) I show that commonly made assumptions about external validity are, in the face of the actual observed heterogeneity across contexts, both logically incoherent and empirically unhelpful. Logically incoherent, in that it is impossible to reconcile general claims about external validity of rigorous estimates of causal impact and the heterogeneity of the raw facts about differentials. Empirically unhelpful in that using a single (or small set) of rigorous estimates to apply to all other actually leads to a larger root mean square error of prediction of the “true” causal impact across contexts than just using the estimates from non-experimental data from each country. In the data about private and public schools, under plausible assumptions, an exclusive reliance on the rigorous evidence has RMSE three times worse than using the biased OLS result from each context. In making policy decisions one needs to rely on an understanding of the relevant phenomena that encompasses all of the available evidence.
Core dual ideas of early development economics and practice were that (a) national development was a four-fold transformation of countries towards: (i) a more productive economy, (ii) a more responsive state, (iii) more capable administration, and (iv) a shared identity and equal treatment of citizens and (b) this four-fold transformation of national development would lead to higher levels of human wellbeing. The second idea is strikingly correct: development delivers. National development is empirically necessary for high wellbeing (no country with low levels of national development has high human wellbeing) and also empirically sufficient (no country with high national development has low levels of human wellbeing). Three measures of national development: productive economy, capable administration, and responsive state, explain (essentially) all of the observed variation in an omnibus indicator of wellbeing, the Social Progress Index, which is based on 58 distinct non-economic indicators. How national development delivers on wellbeing varies, in three ways. One, economic growth is much more important for achieving wellbeing at low versus high levels of income. Two, economic growth matters more for “basic needs” than for other dimensions of wellbeing (like social inclusiveness or environmental quality). Three, state capability matters more for wellbeing outcomes that depend on public production than on private goods (and for some wellbeing indicators, like physical safety, for which growth doesn’t matter at all). While these findings may seem too common sense to be worth a paper, national development--and particularly economic growth—is, strangely, under severe challenge as an important and legitimate objective of action within the development industry.
This paper re-examines why global collective action problems persist, and how to overcome them. Drawing on 140 interviews with campaigners, politicians, and businesses in 10 European countries, it suggests that many activists are stuck in a despondency trap. Never seeing radical reform, they lower their ambitions, and invest in more feasible but sub-optimal alternatives. This creates a negative feedback loop, in which the dearth of radical reform becomes self-fulfilling. But if reformists see advances at home and abroad, they may become more optimistic about collective mobilisation and break out of their despondency trap. This is shown by tracing the drivers of ground-breaking legislation. From 2018, large French firms must mitigate risks of environmental and human rights abuses in their global supply chains, or else be liable. This bill – the world’s first of its kind – was vociferously contested by businesses. But French campaigners and politicians persisted for four years, because they saw reasons for optimism. These include growing international support; public outcry; the French political culture (state intervention, and distrust of multinationals); together with a Centre-Left Government. Optimism galvanised relentless mobilisation. Legislative success in France then delivered a positive shock to activists across Europe, who were emboldened to launch similar campaigns and escape their despondency trap.
This is one of a series of working papers from “RISE"—the large-scale education systems research programme supported by the UK’s Department for International Development (DFID), Australia’s Department of Foreign Affairs and Trade (DFAT), and the Bill and Melinda Gates Foundation.
Support for gender equality has risen, globally. Analyses of this trend focus on individual and/or country-level characteristics. But this overlooks sub-national variation. Citydwellers are more likely to support gender equality in education, employment, leadership, and leisure. Why is this? This paper investigates the causes of rural-urban differences through comparative, qualitative research. It centres on Cambodia, where the growth of rural garment factories enables us to test theories that female employment fosters support for gender equality: potentially closing rural-urban differences; or whether other important aspects of city-living accelerate support for gender equality. Drawing on this rural and urban fieldwork, the paper suggests why social change is faster in Cambodian
cities. First, cities raise the opportunity costs of gender divisions of labour – given higher living costs and more economic opportunities for women. Second, cities increase exposure to alternatives. People living in more interconnected, heterogeneous, densely populated areas are more exposed to women demonstrating their equal competence in socially valued, masculine domains. Third, they have more avenues to collectively contest
established practices. Association and exposure reinforce growing flexibility in gender divisions of labour. By investigating the causes of subnational variation, this paper advances a new theory of growing support for gender equality.
Many development agencies and governments now seek to engage directly with local communities, whether as a means to the realization of more familiar goals (infrastructure, healthcare, education) or as an end in itself (promoting greater inclusion, participation, well-being). These same agencies and governments, however, are also under increasing pressure to formally demonstrate that their actions ‘work’ and achieve their goals within relatively short timeframes – expectations which are, for the most part, necessary and desirable. But adequately assessing ‘community-driven’ approaches to development requires the deployment of theory and methods that accommodate their distinctive characteristics: building bridges is a qualitatively different task to building the rule of law and empowering minorities. Moreover, the ‘lessons’ inferred from average treatment effects derived from even the most rigorous assessments of community-driven interventions are likely to translate poorly to different contexts and scales of operation. Some guidance for anticipating and managing these conundrums are provided.
This paper shows that countries may reduce labour repression if they perceive this as conducive to export growth. This paper traces what happened before, in the presence of, and then following the withdrawal of international economic incentives for pro-labour reforms in Vietnam and Bangladesh. The Government of Vietnam announced it would allow independent trade unions, in order to join the Trans-Pacific Partnership (TPP) and increase market access. Similarly, the Government of Bangladesh rescinded restrictions on unions, following global condemnation of Rana Plaza and fear of buyers leaving en masse. Both governments reduced labour repression to promote export growth. With high-level authorisation, Vietnamese and Bangladeshi activists and reformists became less fearful, and mobilised for substantive change. However, these economic incentives were short-lived: after Trump’s election, the USA withdrew from TPP; buyers continued to source from Bangladesh, and squeezed prices (without requiring labour reforms). Both governments then amped up labour repression - notwithstanding private regulation, economic upgrading, industry growth, and mass strikes.
Evaluations of development projects are conducted to assess their net effectiveness and, by extension, to guide decisions regarding the merits of scaling-up successful projects and/or replicating them elsewhere. The key characteristics of ‘complex’ interventions – numerous face- to-face interactions, high discretion, imposed obligations, pervasive unknowns – rarely fit neatly into standard evaluation protocols, requiring the deployment of a wider array of research methods, tools and theory. The careful use of such ‘mixed methods’ approaches is especially important for discerning the conditions under which ‘successful’ projects of all kinds might be expanded or adopted elsewhere. These claims, and the practical implications to which they give rise, draw on an array of recent evaluations in different sectors in development.
A defining task of development is enhancing a state’s capability for policy implementation. In most low- income countries, alas, such capabilities seem to be stagnant or declining, in no small part because dominant reform strategies are ill-suited to addressing complex non-technical aspects. This has been recognized for at least six decades – indeed, it was a centerpiece of Albert Hirschman’s understanding of the development process – yet this critique, and the significance of its implications, remain on the margins of scholarship and policy. Why? I consider three options, concluding that, paradoxically, followers of Hirschman’s approach inadequately appreciated that gaining more operational traction for their approach was itself a type of problem requiring their ideas to embark on ‘a long voyage of discovery’, a task best accomplished, in this instance, by building – and tapping into the distinctive insights of – a diverse community of development practitioners.
Globalization has fed significant economic gains across the world. The gains lead some policymakers in developing countries to believe in the potential of ‘catch up’—where they leverage the gains of an open world economy to foster rapid progress and compete with more developed nations. This belief is particularly evident in countries like Rwanda, where policymakers aspire to turn the country into ‘Africa’s Singapore’. This paper asks if such aspiration is realistic: Do developing countries really gain enough from globalization to catch up to more developed countries? The paper examines the world economy as a league in which countries compete for winnings (manifest in higher income and production). Wealthier countries are in the top tiers of this league and poorer countries are in the lower tiers. The paper asks if gains from the last generation of growth have been distributed in such a way to foster ‘catch up’ by lower tier countries, and if we see these countries ‘catching up’ by moving into higher tiers. This analysis of the world economy is compared with a study of English football, where over 90 clubs play in an multi-tier league system. Prominent examples of ‘catch up’ in this system include Leicester City’s rise from the third tier in 2008 to become first tier champion in 2015. The paper asks if such ‘catch up’ is common in English football, given the way winnings are distributed, and if ‘catch up’ is more common in this context than in the world economy more generally.
Observers claim that public policies fail ‘often’. This paper asks, ‘how often’? It is an important question, because public policies absorb resources to address major social issues. We should know if policies are proving bad social investments; routinely failing to solve focal problems at high costs. Unfortunately, it is not easy to assess this. Many public policy organizations—governments in particular—do not provide accessible views onto overall success or failure. The World Bank does, however, provide such view—and it supports policy interventions one finds in governments across the world. The paper thus examines World Bank failure rates. It finds that there are different answers to the ‘how often’ question, depending on responses to a second question, ‘what is failure anyway?’ In studying both questions, the paper identifies a bias in the World Bank—and probably all organizations adopting rational ‘plan and control’ policy processes—to measuring ‘project and product success’ rather than a broader view of success as ‘problems are solved with development impact’. This means that policy organizations like the Bank judge success based on whether planned products are delivered through an efficient process; not whether policies solve the problems that warranted intervention in the first place, or whether the policies promoted development outcomes. Is this how citizens would want their public policy organizations to conceptualize success?
Governments across the world regularly pursue reforms that achieve less than was originally expected or is needed to make the state function better. The limits to reform success are often obvious in even the early days of reform, where gaps and weaknesses manifest. Many governments have no mechanisms built into their reform processes to see these gaps and weaknesses, however, and persist with predefined reform plans instead of adapting designs to close the gaps and address weaknesses. One antidote to this challenge is to create reflection points where reformers scrutinize their progress to identify weaknesses, reflect on these weaknesses, and adapt their next steps to address the weaknesses. In the spirit of John Kingdon’s work on ‘policy windows’, we call these reflection points ‘adaptation windows’—moments where reformers acknowledge problems in their reforms, adapt reforms to address such, and mobilize support for this adaptation. This paper discusses an effort to open an adaptation window for reformers to ‘see’ and then respond to public financial management (PFM) reform gaps and weaknesses in Mozambique. The paper details why and how this work was pursued, and also reflects on results of the government’s reflection at the adaptation window.
In rich and poor countries alike, a core challenge is building the state’s capability for policy implementation. Delivering high-quality public health and health care – affordably, reliably, at scale, for all – exemplifies this challenge, since doing so requires deftly integrating refined technical skills (surgery), broad logistics management (supply chains, facilities maintenance), adaptive problem solving (curative care) and resolving ideological differences (who pays? who provides?), even as the prevailing health problems themselves only become more diverse, complex and expensive as countries become more prosperous.
The current state of state capability in developing countries, however, is demonstrably alarming, with the strains and demands only likely to intensify in the coming decades. Prevailing ‘best practice’ strategies for building implementation capability – copying and scaling putative successes from abroad – are too often part of the problem, while individual training (‘capacity building’) and technological upgrades (e.g., new management information systems) remain necessary but deeply insufficient. An alternative approach is outlined, one centered on building implementation capability by working iteratively to solve problems nominated and prioritized by local actors.