Challenges of monitoring and evaluation of partnerships in R&I

Monitoring and evaluation of R&I partnerships can shed light in many aspects. Yet, they are bounded by certain challenges. Expectations must be managed in this regard.

The challenges of monitoring and evaluation of partnerships in R&I include:

• Defining monitoring and evaluation to sufficiently capture or satisfy different partners' or stakeholder groups' concerns.
• The timeline for evaluation and communicating the results, because certain outcomes and impacts from R&I partnerships or the funded projects are unlikely to manifest until after the lifetime of the project itself or after a longer period of the lifetime of the partnership.
• If a relatively long period has been left in order to allow long-term impacts to develop, there is a risk that participants may be difficult to track down and locate or may not fully recall the details of a project to be able to answer detailed questions. There is a trade-off to be decided here.
• While some indicators can be relatively easy to monitor and measure (such as joint publications, co-patenting, licensing, etc.), others - particularly those relating to behavioural changes and longer term societal impacts - are much harder to quantify. However, they can be documented through qualitative means and impact narratives.
• Although certain indicators may capture the overall outcomes of a programme, the outcomes and experiences of individual projects and relationships may be difficult to discern. The true benefits of collaboration are more complex and involve longer term relationships, or behavioural changes amongst the partners engaged. Hence, quantitative data must often be reinforced with more detailed qualitative interview or case study approaches.
• The issue of added value: a truly comparable sample of non-participants is difficult to create to conduct counter-factual analysis. The existence of a detailed ex-ante assessment of the conditions and relationships prevailing before the initiation of the partnership/programme that could serve to benchmark the changes caused by the launch of the partnership/programme would help fill this gap.
• Closely linked to the above issues of counter-factuality and benchmarking is the issue of attribution: whether the observed performance can actually be attributed to the specific activities/programme, or whether it has occurred due to other, external, factors. This problem intensifies in the case of long-term impacts that need more time to manifest. Thus, as we refer to longer-time scales we can only talk about contribution and not attribution of an intervention to long-term outcomes and impacts.
• Unintended effects are also hard to identify and subsequently measure, although they are not to be neglected. A rather open and exploratory approach is needed in evaluations to enable this.
• Lack of appropriate indicators: this is a frequently encountered problem in evaluation and one consequence is the development of, sometimes, inappropriate 'proxy' indicators. Special attention is needed not to fall into this trap.