Summary report of the Horizontal Evaluation of the Genomics R&D Initiative

Summary report of the Horizontal Evaluation of the Genomics R&D Initiative - Final Report (PDF, 687 KB)

Prepared for:
National Research Council

Prepared by:

Goss Gilroy Inc.
Management Consultants
Suite 900, 150 Metcalfe Street
Ottawa, ON K2P 1P1
Tel: (613) 230-5577
Fax: (613) 235-9592
E-mail: ggi@ggi.ca

Date:
November 15, 2016

Program description

The Genomics Research and Development Initiative (GRDI) was created in 1999 and is currently implementing its Phase VI round of funding. The program's objective is to establish and maintain core genomics R&D capacity in the following federal departments and agencies (DAs):

  • Agriculture and Agri-food Canada (AAFC);
  • Environment and Climate Change Canada (ECCC);
  • Fisheries and Oceans Canada (DFO);
  • Health Canada (HC);
  • Public Health Agency of Canada (PHAC);
  • National Research Council (NRC);
  • Natural Resources Canada (NRCan); and,
  • Canadian Food Inspection Agency (CFIA) – since Phase VI.

A total of $59.7 million was invested in support of genomic R&D as part of Phase V. Research supported by GRDI seeks to uphold regulatory, public policy, and operational mandates in important areas such as health, food safety, sound management of natural resources, a sustainable and competitive agriculture sector, and environmental protection, with strong collaborations with university and private sectors. In Phase V (2011-14), GRDI supported 73 of these individual DA-led mandated research projects. As well, the program funded two shared priority projects (SPPs) between January 2012 and March 2016: Food and water safety in Canada through an integrated federal genomics initiative (also referred to as the Food and Water Safety or FWS project); and Protection of Canadian biodiversity and trade from the impacts of global change through improved ability to monitor invasive alien and quarantine species (also referred to as the Quarantine and Invasive Species or QIS project).

There are two types of end-users for GRDI-funded projects: internal and external end-users. Internal end-users are the most common type for GRDI-funded projects and are those inside federal government (e.g., scientists in the labs, inspectors on the ground, border agents, trade negotiators). External end-users are those outside federal government (e.g., industry using a patented technology, industry changing their processes due to policy change, international regulatory agencies using/adopting the technology).

Scope and methodology

GRDI foundational documents for Phase VI funding require a horizontal evaluation of the program in 2015-16. The evaluation focused on Phase V of GRDI, which spanned fiscal years 2011-12 to 2013-14. However, fiscal years 2014-15 and 2015-16 were included insofar as work associated with SPPs continued to be conducted during these years. The evaluation addressed nine questions in three issue areas: relevance; performance- effectiveness; and performance – efficiency and economy. The methodology employed multiple lines of evidence, including both qualitative and quantitative and relying on both primary and secondary sources of data. The main limitations of the methodology included: validity and reliability of administrative data; and a likely underestimation of the extent of collaborations in SPPs assessed through the bibliometric study.

Overall evaluation findings

The evaluation found that, while the needs of DAs for genomics R&D funding have evolved, there is a continued need for GRDI. For many DAs, the need continues to relate to capacity building (in both basic and advanced techniques). Several DAs have also seen the need move towards the application of genomics-based technologies and approaches. In addition, there is clear alignment of GRDI with the mandates and priorities of participating DAs and the federal government more generally. The evaluation confirmed that the program is consistent with, and contributes to, participating DAs' legislated mandates related to the health and safety of Canadians as well as the sustainability of Canada's natural resources (via regulatory activities) and support for industry (via economic development and regulatory activities). The evaluation further found that there is little duplication with other similar organizations or federal programs.

In spite of large amounts of funds being leveraged and some DAs making investments in genomics R&D, it is clear that the need for continued support for these types of projects does not show signs of abating. In the absence of GRDI, the evaluation found that DAs' abilities to deliver on their mandates in the future might be negatively impacted as it is unclear what amount of DA funding for genomics R&D would be available given other DA priorities.

The continued need for GRDI is taking place within a context of a decreasing real value of the funding (due to inflation since inception in 1999 without commensurate increases in funding), funding being shared among more participating DAs, SPPs vying for part of the GRDI funding envelope and increasing costs associated with the research. Research costs are increasing because more effort is now being focused on applications and transferring knowledge to end-users, which are more resource intensive phases of R&D.

In terms of effectiveness, the evaluation found that Phase V projects have been successful in the development of innovative knowledge and technologies and influencing evidence-based public policy. The program has produced expected outputs and exceeded targets related to these. Moreover, the evaluation confirmed that many GRDI projects have successfully transferred knowledge and technologies to end-users, both internal and external to the federal government. Many more GRDI projects have the potential to similarly impact end-users. Nevertheless, there continue to be opportunities, especially for the SPPs, to build on the lessons and best practices from Phase V in subsequent phases of the program. Some of these have already been addressed. In particular, content outlined in the Innovation Management Strategy (IMS), though intended for SPPs, offers examples and tools to ensure end-user engagement and take-up of GRDI-funded knowledge and technologies that could be considered beyond the SPPs.

In addition to the transfer of knowledge and technologies, the evaluation found evidence that GRDI-funded projects are likely to have real and lasting longer-term impacts, most of which remain anticipated or potential impacts at this time. These impacts are significant and are expected to lead to billions of dollars in benefits for end-users (through avoided costs, increased trade, etc.). More specifically, the evaluation found that GRDI-funded projects are likely to result in: improved public health and wellness; avoided health system costs; efficiencies and avoided losses for the public and private sectors; environmental sustainability; improved detection of invasive species; improved strains/traits of plants, trees and animals of commercial value; and improved fisheries tracking and management.

The introduction of interdepartmental SPPs in Phase V has proven to be a strong feature of the program and the evaluation found that there is a continued need for this type of interdepartmental collaboration. The two SPPs funded in Phase V were found to be well-managed and have achieved significant results, which likely would not have occurred in the absence of GRDI funding for the interdepartmental nature of the projects. As well, the degree of interdepartmental collaboration stemming from the SPPs has been significant. However, there is no consistent or planned approach related to what will happen to ongoing SPPs when the GRDI funding ends. Consequently, there is a risk that the results achieved to date and the potential for further impacts might be slowed or otherwise diminished. Some concerns were also raised about the selection of SPPs (including the role of scientists), and a review of related documents confirmed some lack of transparency in terms of how sub-projects are selected. As well, since sub-projects are ultimately directed by the Scientific Coordinator from one participating DA (with support from Sub-Project Leads), there is the potential for the perception of conflict of interest. Finally, the QIS case study suggests that mid-year reporting may be overly burdensome (on top of annual reporting). Both case studies brought the suggestion that there should be more interaction between the various scientists contributing to the SPP goals.

Other challenges associated with the SPPs included those pertaining to: end-user engagement; sharing of data/materials and building bioinformatics capacity; management of IP; a short timeframe for projects of this magnitude; interdepartmental financial and administrative issues; and delays associated with the hiring of highly qualified personnel. The IMS was developed with a focus on the first three challenges in particular (among other matters), and is being applied during Phase VI SPPs.

In terms of efficiency and effectiveness, the evaluation found that the program is both efficient (in terms of its horizontal governance) and effective (in terms of its use of collaborations, leveraging of funds and delivery approach). The evaluation did not reveal any significant opportunities for improvement in program governance or delivery. As well, the evaluation found that the program has made improvements in terms of its performance measurement approaches and systems. However, a few challenges remain, including reporting inconsistencies and the lack of mechanisms to compile and/or track supporting evidence by project.

Recommendations and Management Response

Recommendations Management response
1. The ADM CC should consider exploring and formally defining how SPP sub-projects are selected (including how and when the input of scientists is considered). Part of this selection process should consider and evaluate the potential for conflict of interest during the approval process as well as the input of scientists.

Accepted.

The GRDI will update its Governance Framework and articulate the SPP planning process more clearly, particularly vis-à-vis sub-project development and selection.

2. The ADM CC should consider requiring SPPs funded in Phase VI and beyond to develop a plan exploring how the technology/knowledge developed during the project will be transferred to, and used by, end-users outlining follow-on work that should be conducted by the partners to ensure such transfer and identifying potential funding sources to maximize uptake.

Accepted.

SPPs will include a technology/knowledge transfer plan that identifies clients and end users who will be engaged throughout the project.

SPP teams will periodically review and update the initial technology/knowledge plan in order to maximize uptake. At the end of the project funding period, GRDI will identify follow-on work required to fully implement the plan, identifying potential resources and fostering external linkages as appropriate

3. The ADM CC should consider and implement additional opportunities to increase communication between DAs participating in SPPs to improve collaboration and joint problem solving, for example by supporting interactive mid-year sessions.

Accepted.

Internal communications plans to facilitate collaboration and joint problem solving will be required in SPP Management Plans. GRDI will support additional interactive team sessions

4. The ADM CC should explore additional opportunities to improve performance measurement and reporting, including: Accepted.
  • Ongoing monitoring on Phase V SPPs (and the next evaluation should also explore what has happened with these projects). SPP leads should be consulted regarding what would be the most meaningful set of indicators to track that are manageable in terms of reporting burden
In consultation with Phase V and Phase VI SPP leads, GRDI will develop meaningful methodologies for examining the ongoing legacy and impact of SPPs. SPPs will be required to identify end-users and stakeholders that have benefited, or could benefit, from project outputs for follow-up assessments. An adoption and impact study of Phase V SPPs will be undertaken as part of the next GRDI evaluation.
  • Implementing a database to capture information by project, which would allow for searching and analysis by key project characteristics (such as project type, impact area, DA, GRDI funding phase, GRDI funding, DA-leveraged funding, HQP, etc.) and would encourage projects with similar objectives to communicate with one another. This database should be accompanied by a detailed definition of variables and indicators to be reported.
GRDI will define key project characteristics to be reported, and develop a database which can be used to foster interdepartmental communication and support reporting activities.
  • Streamlining SPP reporting, for example by revisiting the number of indicators to be reported in SPP mid-year reports. Interactive mid-year sessions rather than detailed mid-year reports, and revisiting the number of indicators, are examples of opportunities for streamlining and improvements in communications.
GRDI will streamline SPP mid-year reporting to the ADM CC and will revisit the variables and indicators reported annually for mandated and SPPs.
Date modified: