Evaluation Framework 2024: How We Work

 

Purpose

The Life You Can Save envisions a world where there is no global poverty. Our mission is to improve the lives of people living in poverty by changing the way people think about and donate to charity. As such, we seek to understand the complex nature of global poverty and use robust evidence to recommend highly impactful nonprofits and engage with donors on best practices.

This evaluation framework provides a summary of how we explore the process of creating high impact giving recommendations.

1. How we define poverty

Leading experts at the Oxford Poverty and Human Development Initiative and United Nations Development Programme have collaborated with stakeholders to develop a comprehensive understanding and measurement of the multidimensional and interlinked nature of poverty. The development of a multidimensional poverty index (MPI) provides a complex view and indicators of poverty and wellbeing, and moves beyond an income threshold to define poverty. Understanding the experience of poverty can be challenging for individuals in high-income countries. Many of us find it difficult to truly understand what it means to live on less than $2.15 a day.  Poverty encompasses various dimensions that profoundly impact individuals’ freedom to lead the lives they aspire to. At The Life You Can Save, we use the Global Multidimensional Poverty Index (MPI) to guide our understanding of poverty and exploration of causes, interventions and nonprofit recommendations. The MPI was developed by Sabina Alkire and Maria Emma Santos, scholars from Oxford University, and has been included in the UNDP’s Human Development Report since 2010.

The MPI defines three fundamental dimensions of poverty: health, education, and standard of living. These dimensions were selected through a comprehensive process that involved participatory surveys among individuals experiencing poverty, expert consultations, alignment with public consensus (including the Universal Declaration of Human Rights and Sustainable Development Goals), and philosophical examination. Our use of the MPI is based on its robustness, broad consensus within expert communities, and availability of reliable data.

2. How we measure poverty

The MPI breaks down the three dimensions of poverty into ten indicators. A person is identified as being multidimensionally poor if they are deprived in more than a third of the (weighted) indicators. The intensity of poverty is then measured by the percentage of deprivations they are simultaneously experiencing. The figure below shows the indicators and their weights.

Source: Alkire, S., Kanagaratnam, U. and Suppa, N. (2020). ‘The global Multidimensional Poverty Index (MPI): 2020 revision’, OPHI MPI Methodological Note 49, Oxford Poverty and Human Development Initiative, University of Oxford.

3. How we identify those most in need (Filter 1)

We focus on countries and regions characterized by the highest levels of multidimensional poverty, as well as higher numbers of people experiencing it. In these areas, a significant portion of the population grapples with deprivation across multiple dimensions, making them particularly vulnerable and underserved. We believe that focusing our efforts here is essential, as it indicates a concentration of structural barriers hindering human development. These obstacles span a wide spectrum, including issues such as limited access to healthcare, educational opportunities, and economic prospects, as well as deprivations stemming from entrenched social norms, conflicts, natural disasters, and the impacts of climate change.

4. What drives deprivation across these indicators?

Within the context of multidimensional poverty, various underlying factors contribute to deprivation across its three dimensions. Our team is committed to conducting comprehensive reviews of the scientific literature and data, identifying the primary drivers of poverty within the most deprived regions. This effort enables us to gain a nuanced understanding of the critical challenges faced by vulnerable populations, whether they reside in specific countries or regions, belong to particular ethnic groups, or fall within specific age groups. Moreover, we explore how poverty intersects with various household and community characteristics, ensuring a holistic analysis of the multifaceted nature of deprivation. Examples of key drivers are malaria and diarrheal diseases for child mortality, school costs and gender norms for education, and lack of access to markets or lack of capital for standard of living.

4.1. How we select drivers to focus on (Filter 2)

To identify drivers to focus on, we rely on three main criteria: 

1. Scale. Does the driver affect a vast majority of people in a given context? This ensures that we consider the most important problems, including those that affect a vast number of people as well as those that cause extreme suffering.
2. Solvability. Would additional resources go a long way towards solving the driver? This keeps our focus on problems that can in principle be solved using existing interventions that are backed by rigorous evidence.
   
3. Neglectedness or additionality. Will donor contributions add substantially to the current resources available to address the driver (i.e., by providing a given number of dollars or person-hours). This keeps our focus on drivers that get insufficient attention, so that our contribution has a high impact and doesn’t crowd out other efforts.

5. How interventions are selected (Filter 3)

Several interventions are available to tackle key drivers of poverty, and their design can vary according to contextual factors, the specific needs of beneficiaries, and the philosophical principles of implementing organizations. For example, some interventions involve active participation of beneficiaries in the design phase, tailoring solutions to their unique circumstances, while others may originate in different settings and be applied across various geographical areas. It’s important to recognize that there isn’t a one-size-fits-all approach to driving change and creating impact. However, extensive research has indicated that certain interventions can generate higher levels of impact than others.

At The Life You Can Save, we identify the most impactful solutions to key drivers of poverty. To do so, we will assess interventions across the following criteria:

• Evidence: Is the intervention backed by rigorous evidence? 
• Scale: Does the intervention have the potential to reach a high number of beneficiaries? This will depend both on its complexity and the cost per beneficiary. 
• Depth: How many MPI indicators does the intervention address?
• Durability: Is the change achieved by the intervention sustained over time?

How we use rigorous evidence to find the most impactful solutions:

When looking for the best solutions, we look for evidence on the impact of different interventions. One of the keys to understanding the impact of an intervention is to understand the evidence behind it. Our team goes through impact evaluation studies of different interventions, looking for places where evidence gathered through different methodologies converge. We do this to mitigate the bias and limitations inherent to the research methodologies such as publication bias, limited number of replication studies, and statistical error. While RCTs are often considered the gold standard in evidence-based programming, we believe that RCT results should be considered alongside studies using different methodologies. In an RCT, a representative sample of potential recipients, such as individuals or communities eligible for the treatment, is randomly divided into ‘treatment’ and ‘control’ groups. The intervention is then administered exclusively to the treatment group, allowing for a rigorous comparison of outcomes between the two groups. When there is a substantial number of recipients in each group, this approach yields a highly precise and attributable measure of the intervention’s effects. In cases where conducting an RCT is not feasible, alternative quasi-experimental methods can be employed to approximate this level of measurement. In addition to experimental and quasi-experimental evidence, we will look for other types of impact evaluation, such as qualitative and longitudinal work, as well as specific case study research. 

The quality and type of evidence available varies substantially depending on the dimension of poverty, the outcome that needs to change, and the nature of the intervention itself. For example, many of our recommended nonprofits work closely with the government to scale promising interventions in the countries where they operate, which is much harder to track than simple service delivery. Some methodologies are more suited to particular dimensions and strategies, some outcomes are harder to measure than others, and some interventions are much more studied. This leads to a few issues:

1. While in some cases these interventions will have been created and tested by an organization we’re considering (e.g. Evidence Action’s safe water dispensers), in most cases we will have to rely on a body of research that may or may not involve the nonprofits we consider. We then take into account:

• • External validity: can the results of findings from one context (geography, time period, implementer) be applied to another? For example, the impact of cash transfers in Kenya in 2015 will differ substantially from their impact in Liberia in 2022. Such external validity is highest if the nonprofit under review has implemented the intervention itself, if the intervention was implemented in a similar context, and if there are a high number of studies of the intervention, as summarized in systematic reviews and meta-analyses.
  • Internal validity: do the findings reviewed accurately measure the intended research question. Can the observed effects be confidently attributed to the intervention rather than other factors? Internal validity can be improved by high-quality evaluations that include clear metrics of implementation and impact, a rigorous attempt at identifying causality, and assessment of the internal and external validity of the findings. 
  • Attribution: can the impact estimates be causally attributed to the organization? For example, it is easy to measure the impact of fistula surgery in a population where no one has access to fistula surgery—effectively, the entire population has a baseline of zero surgeries and the intervention’s impact is equivalent to the number of surgeries provided. However, if there are multiple providers, this can be harder to assess. 

2. This kind of evaluation research excels at precise measurement of the causal impact of clearly defined interventions, but is ill-suited to large and complex interventions where treatment and control groups of sufficient size are hard to define. It is also ill-suited to measuring long-term impacts, or teasing apart the mechanisms through which impact does or does not occur. Similar to earlier, following only this type of evidence can lead us to leave out highly impactful and cost-effective, but harder-to-measure interventions—for example, policy advocacy for change at the level of the country or the world. In the latter instances, for example changes in national nutrition and climate policies, more appropriate methods are case studies and legislative reviews. These qualitative approaches though are less able to tell us whether the impact was caused by a particular charity’s intervention (for example, would the government have changed its policy anyway? Would other organizations have stepped in?), i.e. whether the findings have internal validity.
   
   Rather than only considering one kind of evidence, we seek the best quality evidence possible given the cause area and strategy—e.g. RCTs for direct delivery of health services, carbon market analysis and policy review for climate change advocacy interventions, etc.This has several consequences:

• By considering different types of evidence based in part on what solutions are most impactful in a given dimension (e.g. poverty graduation models to improve standard of living), and in part on the strategy and vision of each nonprofit, we are able to welcome diversity in mechanisms to drive change. Among the common mechanisms we consider are: (i) providing essential information, goods and/or services directly to people, (ii) helping the government create and/or scale effective interventions, (iii) helping the private sector create and/or bring effective interventions to market, and (iv) pushing for large-scale policy or systems change through advocacy. 

6. Identifying nonprofits implementing high impact interventions

Our goal is to identify outstanding organizations focused on high impact solutions. To streamline our selection process, we begin by compiling a longlist of organizations that operate in the areas with the highest need. We build a pool of nonprofits using different sources, and then vet them in-house before considering whether or not to recommend them. Our data come from the organizations’ program documents, review of studies by independent evaluators, conversations with sector experts and philanthropists, and field visits by us or our partners. It’s important to note that our team encounters several constraints that serve as an initial filter, determining which organizations we can effectively evaluate.

6.1. Our constraints (Filter 4)

We face three main constraints when searching for highly impactful organizations:

• Network: We often rely on a network of experts, fellow evaluators, and funders to identify organizations to support. This facilitates our own work as well as reducing our burden on nonprofits. It allows us to rely on our partners to gather information and due diligence on organizations, without having to ask them to produce extensive documentation or reports.  Although we strive to expand the network of partners we work with, we know it will always have limits and we will lead us towards organizations that are well-known and supported by others.
• Language: Closely related to network constraints, our team’s language limitations will prevent us from finding and evaluating organizations unknown to our predominantly Anglophone network, or whose websites are not in languages known to us.
• Bandwidth: While we can overcome the first two constraints by seeking out organizations beyond our networks or hiring translators and local experts to find organizations working in languages beyond ours, we face bandwidth and budget constraints. 

6.2. Due diligence and organizational impact criteria (Filter 5)

Organizational impact

Identifying impactful interventions through evidence is not sufficient to achieve impact, as this lacks two key steps:local adaptation and implementation effectiveness. Any successful intervention needs to be adapted, developed and implemented by effective organizations, ones that can reduce uncertainty across the various levels of their work. The first step we take to evaluate the work of nonprofit organizations is to combine the insight gathered through our review of evidence with Realist evaluation methodologies (Pawson and Tilley, 1997). Secondly, we rely on the  work of organizational scholars Christian Seelos and Johanna Mair (2017) to gather data from nonprofits that we can use to test our hypotheses about an organization’s impact. 

Realist evaluation

The realist evaluation methodologies were first developed by Pawson and Tilley (1997) with the purpose of making the results of impact evaluation and scientific research more useful to decision makers. The approach goes beyond answering the question of “what works?”, focusing on “what works in which circumstances and for whom?”. We rely on this work to overcome the limitations in replicability of RCT results and other kinds of evidence. Realist evaluation is a methodology that looks for the mechanisms enabling particular outcomes, also known as generative mechanisms. In the context of human development and social change, these can range from socio-cultural factors, environmental characteristics, and economic structure of the context under investigation. The beneficiaries of a particular programme will respond to that programme in ways that are tied to the context they are in, and interventions should be evaluated against that backdrop. This evaluation approach is based on a programme theory (Pawson and Tilley, 1997): a hypothesis of why an intervention might work in a given context. In our evaluation process, we will come to a new organization with a particular programme theory, based on our preliminary review of the impact evidence across different contexts. 

Knowing “which” interventions work in some context is not sufficient for decision making: evaluators need to understand which intervention works for certain beneficiaries in a certain context. Crucially, published research can answer the former question, but rarely the latter: RCT results and other kinds of evidence are not always replicable (Vivalt, 2020; Deaton and Cartwright, 2017). To overcome this difficulty in applying evidence-based interventions in real-world scenarios, the realist-evaluation methodology looks for the generative mechanisms of particular outcomes by asking a preliminary question: “why” does an intervention work? The answer to this question is formulated as a programme theory that specifies the generative mechanism of particular outcomes as a function of a variety of factors. In the context of human development and social change, these range from socio-cultural factors, to the environmental characteristics and economic structure of the context under investigation. Once we have this understanding, we can finally assess the effectiveness of interventions for specific subjects with features X in a specific context/environment with features Y. In our evaluation process, we will come to a new organization with a recommendation based on a particular programme theory, based on our preliminary review of impact evidence and iterative optimization based on further data collection. The process will follow these steps:

1. Assessing an organization’s intervention against the external evidence we reviewed. As mentioned in previous sections, we will review evidence on different interventions in order to identify those that appear to have the highest impact. We will then compare the interventions in the literature with the programmes implemented by the organization we are evaluating. This step will be the basis of our programme theory for the organization we are evaluating.

2. Programme Theory. Having reviewed organizational documents and information about the context in which they operate, we will develop a Programme Theory based on step 1, i.e.  which mechanisms may be generating positive impact in the context where the nonprofit organization is operating

3. Data collection. We will have meetings with the nonprofit organizations to collect the data we need to test our programme theories. Data will include experimental data gathered by the nonprofit, monitoring and evaluation data, as well as conversations and interviews with programme staff and key stakeholders. This process will allow us to have a deeper understanding of what mechanisms are at work in their programmes and the impact they may have.

Data Collection

To collect data on the context where organizations work and the mechanisms driving change, we draw on the work of organizational scholars Seelos and Mair (2017). Seelos and Mair focus on an organization’s  impact creation logic, i.e. the knowledge, processes and systems that drive high-impact organizations. An organization’s impact creation logic is defined around the knowledge it possesses to create impact and how it learns from past programming and failures to reduce key uncertainties in the implementation and the outcomes of interventions. Seelos and Mair (ibid.) define the impact creation logic as being made of three dimensions and related uncertainties. 

• Problem Space: what is the problem that the organization is trying to address? What are its underlying causes? An effective organization would demonstrate a full understanding of the issues they are trying to solve. This understanding comprises economic, political and normative causes of certain deprivations  (lack of access to schooling, low-income, malnutrition, etc.), as well as the consequences of the deprivations (vulnerability, poor wellbeing, mental health, etc.). A higher level of knowledge in this domain increases the changes of an organization being able to get to the root causes of a problem. Problems are not only technical, involving economic or information/knowledge barriers, but also relational (Seelos, 2014). The latter category include normative challenges such as beliefs, traditions and cultural norms, as well as power and politics dynamics such us ethnic tensions, socio-cultural barriers and political or economic interests.

• Solution Space: how is the organization addressing this problem? Is the solution effective and appropriate to address the problem? An effective solution would take into account the contextual barriers to implementation, deal with adoption uncertainty and the potential unintended consequences it may cause on the ground. The way organizations can address uncertainty in this domain is by iterating different solutions and adapting to feedback from their beneficiaries. Organizations that reach higher levels of knowledge about their solutions in context are more likely to have higher impact

• Identity Space: why is the organization focusing on this problem and solutions? When an organization’s sense of purpose is aligned with the target problem and its solutions, then it is more likely to overcome setbacks and managerial challenges in delivering positive impact. A strong understanding of why an organization works in a certain way rather than another helps strengthen organizational cohesion and managerial effectiveness. 

In our evaluation process, we will draw on this body of work and concepts to assess the levels of knowledge that an organization possesses across domains, how this knowledge is acquired (e.g. through iterating solutions, interacting with beneficiaries, adapting to new contexts) and how diffused it is across the organization (e.g. management, implementation teams, etc.). 

A commitment to learning and improvement

In addition to evaluating organization’s knowledge across these dimensions, we want to identify those who are constantly looking to increase their impact by identifying blind spots, partial successes and areas for improvement. Once again, we draw on the work of Seelos and Mair (2017) who describe key attitudes common to high-impact organizations, i.e. constructive dissatisfaction and constructive suspicion.

• Constructive dissatisfaction is the attitude of those organizations that work to constantly raise their ambitions and use their impact data not only to report to donors but also to improve current performance.

• Constructive suspicion is the attitude of those organizations who are constantly critical of their work and approaches, trying to seek out areas for improvement, dissatisfaction among beneficiaries or unoptimized processes.  

A way to better understand how organizations implement these two approaches can be by exploring how organizations learn about the contexts they are working in and the solutions they seek to implement. These include systems for feedback, data collection and monitoring, and rapid and collaborative organizational learning processes.

High impact 

We define high-impact organizations as those that consistently achieve measurable, evidence-supported, and sustainable improvements in the lives of individuals, which would not have happened without their contribution. We support these organizations to ensure that every dollar or contribution invested in them can go the furthest in generating positive outcomes and reduce suffering. In this regard, we will continue monitoring the three criteria that are key to an organization’s impact: scale, depth and durability.

In our evaluation and monitoring of nonprofit organizations, we will take into account these three dimensions of impact and consider how organizations can achieve similar levels of impact through different strategies. 

Examples:

1. Nonprofit A provides a narrow intervention that it distributes to millions of people. The intervention prevents people from contracting disease X but it cannot help beneficiaries in other ways. The intervention must be repeated every year. Their impact can be visualized as follows:

2. Nonprofit B provides an educational programme that supports beneficiaries in various aspects of their life. The intervention has proven to increase wellbeing, income and health of beneficiaries. The cost of the intervention is high and the organization cannot directly reach vast amounts of people directly. A longitudinal study showed intergenerational effects of the intervention. Their impact can be visualized as follows:

Track record

We want to recommend organizations that can deliver impact. Therefore, we look at a nonprofit’s track record. The key elements we consider are whether the organization has achieved its yearly targets, whether their impact is accurately measured, and whether they make a strong enough case for attribution (change would not have happened without them). Additionally, we will consider the ways in which the organization responds to feedback from their partner communitiesand how they innovate and produce new evidence as a result.

Cost

We analyse cost per beneficiary to ensure that the nonprofit’s intervention is cost-effective within a reasonable range—for example, compared to other organizations seeking to change the same outcome, and do not try to claim that an intervention or the organization that implements it is the most cost-effective. We consider that cost-effectiveness calculations are built on subjective valuations such as the value of life at different ages, the extent of suffering alleviated by a given intervention, and our best guesses about recipient utility and preferences. Small changes in many of these underlying assumptions can lead to widely differing answers. Our goal is to provide donors assurance that giving money to an organization is a great bet, highly likely to do good, and cheap enough to be scalable, durable, and/or have a deep impact on people’s lives. 

Transparency 

The organization should have robust monitoring and evaluation systems and learn from it. A transparent organization will be proactive about publicly sharing data and internal research, as well as publicly admitting mistakes and showing that it is learning from them.  Data for these metrics will come from our interactions with each nonprofit (requests for documentation and data, etc.), document review, and consultations with experts, donors, and other implementers. 

Proximate leadership

Leaders who intimately understand the communities they serve are best positioned to identify and implement impactful solutions. Despite this, the current funding landscape does not fully reflect this principle, with only 1.2% of humanitarian funding directed to local organizations in 2022. We assert that for social change to be truly sustainable, there must be a paradigm shift towards supporting organizations led by individuals deeply connected to partner communities, while prioritizing measurable impact. Critics of foreign-driven and top-down intervention such as the economists William Easterly (2006, 2007) and Owen Barder (2014) have long advocated for a shift towards bottom-up solutions. Easterly’s critique spurs from the tendency of Western-led organizations to overlook the complexity and diversity of local conditions, leading to a lack of sustainable impact, lack of agency among local communities and ignoring local knowledge and grassroots innovation (ibid.). Similarly, postcolonial thinkers have long advocated for local participation and ownership of the development process (e.g. Rodney, 1972, Escobar, 1995) while economists  like Dani Rodrik and Ha-Joon Chang stress the importance of context-specific approaches to development, avoiding interventions and policy prescriptions that do not consider the unique economic, social, and political contexts of individual countries and regions.

Our team actively seeks out organizations where decision-making is intertwined with community buy-in, local staff actively contribute to program planning and implementation, and community members assume key roles. We do this because we believe the sector needs stronger feedback loops between decision makers and the intended beneficiaries. This approach is grounded in the belief that upholding the dignity of those receiving assistance, ensuring the long-term viability of our programs, and contributing to a fairer and more equitable development sector necessitate empowering local leaders and fostering grassroots solutions.

Respect for dignity

Research shows that international development beneficiaries deeply value respect for dignity and that, unfortunately, they are also frequently denied. Even when people get the material aid they need, they frequently come away from interactions with international development service providers feeling bruised, mistreated and ignored. In order to build a more equitable and just social sector, it is essential that any charity considered effective is also respectful of its beneficiaries’ dignity, their priorities, and needs. 

The Life You Can Save partners with IDInsight’s Dignity Initiative to integrate dignity in our evaluation framework. IDinsight describes three main pathways to consider when reflecting and operationalising the concept of dignity. These are:

• Equity: Treat beneficiaries as if they were equals and reduce power asymmetries.
• Agency: Give beneficiaries choices and meaningful chances to consent.
• Representation: Offer beneficiaries the chance to be and feel seen by the institution delivering aid.

We assess nonprofits on how well they respect dignity alongside other impact factors in deciding whether to recommend them to donors. In our evaluations, we will consider the following criteria:

1. Priority: When looking at a nonprofit’s website, reports or listening to a presentation, evaluate whether the service they are providing is something people in the area value and are calling for. Most importantly, how does the charity know this is something people need and want? A respectful nonprofit will make reference to participatory processes of preference elicitation, willingness to pay and will transparently discuss trade offs with other priorities. 
2. Interactions with beneficiaries: A respectful nonprofit will reference concrete steps taken to ensure that beneficiaries are treated fairly and in the way they wish to be treated. Donors should look out for references to the three pathways in nonprofit’s websites, social media, presentations and reports. 
3. Culture and People: Internal culture is key to developing respectful interactions both within and outside the organization. Donors should look for examples of concrete steps organizations take to ensure staff are treated with respect.
4. Listening and Learning: A highly impactful and respectful organization needs tools and mechanisms to know that they are treating people with respect. A nonprofit that respects people’s dignity will mention ongoing and proactive measurement of beneficiaries’ experiences, feedback mechanisms, and will show a track record of adapting their programs based on these mechanisms. 

Systems strengthening

One of the primary criticisms of philanthropy revolves around its potential to foster dependency on foreign aid and donations (Easterly, 2006; Moyo, 2009), induce complacency and erode the accountability of  local governments who are not committed to development (Dearcon, 2022), and stifle innovation in addressing community challenges (Clayton et al., 2018). Recognizing the validity of these concerns, we are committed to enhancing our philanthropic efforts by incorporating these critiques into our evaluation and monitoring processes for nonprofits. However, we also acknowledge that the possible unintended consequences of foreign aid need to be set against their proven benefits (Barder, 2014) in terms of improvements in health, education, livelihoods and beyond. Recognising the complexity of calculating the tradeoff between the benefits and potential harm caused by foreign aid, our team works to identify organizations and interventions that can lead to sustainable systems strengthening.  

Our approach emphasizes assessing an organization’s capacity to strengthen local systems and drive sustainable change. Ideally, nonprofits could stop relying on philanthropic funding within a few years, either because the identified problem has been resolved or because other entities, such as governments or markets, have assumed responsibility for their initiatives.

While acknowledging the risks associated with dependency, we recognize that philanthropic investments play a crucial role in fostering innovation. Nonprofits can often introduce unconventional approaches beyond what is typically feasible for government entities, bound by accountability to elected officials and taxpayers. Additionally, philanthropy can act as a catalyst for innovative community action and for creating new markets, driving social change, and addressing issues that may remain unattended due to market failures and contextual factors.

Impactful philanthropy can help organizations on the ground in building sustainable impact, supporting collaboration with governments and key community stakeholders. Successful nonprofits aim to implement cost-effective solutions at scale, working through formal institutions to reduce the long-term dependence on external aid. It is crucial for these organizations to cultivate partnerships that facilitate a gradual decrease in direct involvement in target communities, paving the way for governments or markets to assume responsibility.

While recognizing the importance of long-term cultural change interventions, we understand that sustained philanthropic investments may be necessary for success across diverse geographies. In our evaluation process, we prioritize and support nonprofits that strategically plan to transition away from direct community involvement, creating effective off-ramps for governments, communities, or markets to assume a more prominent role.

7. Recommending high impact nonprofits

After following steps 1 to 6, we will develop a list of nonprofits that we can confidently recommend as high-impact funding opportunities. These will cover all three dimensions of poverty, a wide range of theories of change, and interventions. The list will be dynamic and subject to new evidence, contextual changes within the countries of operation, and organizational changes. 

7.1. Uncertainty and confidence levels

In the process of evaluating nonprofits, it’s important to acknowledge that all assessments inherently involve a degree of uncertainty and necessitate professional judgment from our research and evaluation team. The evidence and data available for different criteria may vary, leading to situations where we must rely on our expertise and judgment to determine whether an intervention aligns with our criteria or not. In our case, it informs how we assess nonprofits for our confidence in their alignment with evaluation criteria. We categorize recommendations by high and medium confidence levels, indicating where we have stronger or weaker certainty.

We believe that it would not serve the best interests of our donors to exclude interventions or organizations solely due to a subjective threshold of evidence supporting their impact. We maintain that the level of risk a donor is comfortable with regarding their donation should be a decision made by donors themselves, not imposed by our team. Our commitment is to provide transparent and informative evaluations, empowering donors to make informed choices aligned with their preferences and values while providing guidance, resources and explanations.

8. Our impact

Our impact can be conceptualized as the additional impact our donors have based on our work. Additionally, we want to use funds to strengthen the ecosystem of high-impact nonprofits in countries with high incidence of multidimensional poverty. Relying on a strong evaluation framework allows us to confidently advise donors and increase their impact across these spheres. 

To put this graphically, a donor wanting to achieve more impact has two options:

Source: Effective Philanthropy

Therefore, we rely on a number of key metrics to track our impact:

For our marketing and fundraising efforts:

Our key metric here is Money Moved, i.e. the funds that we moved to high-impact nonprofits. This gives an indication of how successful we are at our mission of improving the lives of people living in poverty by changing the way people think about giving and how they donate.

For our research and evaluation efforts:

We will consider how the funds we raise relate to impact, to assess how we can improve our recommendation list year after year. Our research and evaluation team’s success can be tracked through:

• Impact metrics across our portfolio. These include but are not limited to reductions in child mortality, learning outcomes, surgeries performed, income increases. These metrics should be an indication of the quality of our research and evaluation work as well as the support we provide to our partner nonprofits. We know that the most impactful interventions need not be the most costly. As such, the correlation between money moved and outcomes achieved might not be direct; smaller donations to highly impactful organizations can wield a far more substantial influence than larger contributions to less impactful ones, making our team’s role crucial to donors.
  
• RNP success: This metric helps us assess our ability to support and strengthen the ecosystem of high-impact organisations. Key questions we seek to answer are:
  Do our grants achieve what they set out to achieve, i.e. do our recommended nonprofits achieve their goals year by year? Do nonprofits access more funding and scale their programmes? If the organizations we support manage to scale successfully and attract considerable amounts of funding, we can consider this a success for our team. 

We will track and  improve our work based on the metrics above, by monitoring new research and evidence, connecting with other high-impact donors to catalyze funds, and influence more organizations to donate to our recommended list.

Limitations

Risks of overemphasizing data and metrics

The framework heavily relies on data and metrics to evaluate nonprofits. While it’s essential to assess impact, international development is often complex and context-specific. An overemphasis on quantitative data might overlook qualitative aspects, cultural nuances, and long-term, indirect impacts that are challenging to measure, including policy and systems change. 

Lack of a risk assessment methodology

Assessing risks related to political instability, changing regulations, environmental factors, and local partner relationships is important in international development.

Difficulties in accounting for contextual factors

The framework does not adequately account for the importance of local context and cultural nuances. Successful interventions in one region may not be directly transferable to another, and the framework’s focus on global indicators may not address these disparities.