Table 2 Description of ALNAP-DAC evaluation criteria used to design the study’s analytical framework (adapted from ALNAP guide 2006)

Relevance

It evaluates the synergies between the intervention, national priorities and local needs. It can be used in all evaluation types

Effectiveness

It indicates to what extent an intervention was successful in bringing about the intended results. It also implies timeliness and can be used in sectoral or organisational evaluations

Efficiency

It measures all resources (monetary and nonmonetary) required to achieve the desired output and compare alternatives to achieve the same output. Thus, it can be used in all evaluation types with adequate financial data

Effects

“Effects” is not a standard criterion, but it was intended to replace “impact”. “Impact” needs a comprehensive assessment that involves the affected population, which cannot be covered in this study. Alternatively, the study looked at various “effects” on the national health system, humanitarian response coordination, Cluster partners and a vulnerable population

Connectedness

It ensures that short-term interventions are linked with long-term plans and can be applied in evaluations that focus on coordination between organisations and stakeholders. It is adapted from the concept of “sustainability”

Cross-cutting theme

 Participation

Participation means that primary stakeholders (Cluster partners) must be fully engaged in all phases of developing and implementing health interventions

Among the eight cross-cutting issues, participation was relevant to the case of Yemen as it reflects how active was the Cluster in engaging partners. In addition, it provides a glimpse of the coordination and synergies between the Health Cluster and its members and highlights the power dynamics within the Cluster

Excluded ALNAP-DAC criteria

 Coverage

Coverage was not selected because it needs identification of the people who benefited from humanitarian action and why they have benefited or excluded from that action (ODI 2006), which is beyond the scope of this study. However, as coverage has an indirect link with “effectiveness”, some aspects of the coverage, i.e. identifying vulnerable groups within the Health Cluster partners’ response, has been included in the study

 Coherence

Coherence focuses on incorporating a humanitarian dimension into global policies. Measuring the coherence of the Health Cluster as a coordination mechanism against international policies is too broad and does not fit the purpose of this study

 Impact

“Impact” was replaced by “effects”. Assessing impact needs a specific impact evaluation approach (Steets et al. 2009, 2010). It also requires collecting feedback from the affected population (Hallam 1998), which is beyond the researcher’s capacity. Previous evaluation reports of the Cluster Approach employed “effects” as a criterion instead of “impact” for comparable reasons (Steets et al. 2009)