ARCHIVED - Evaluation of the Genomics R&D Initiative (GRDI)

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Purpose

This document presents the key findings of the horizontal evaluation of the Genomics Research and Development Initiative (GRDI). Conclusions and recommendations stem from these key findings, which are based on the integrated analysis of multiple lines of evidence. In 2010, the GRDI Assistant Deputy Minister (ADM) Coordinating Committee mandated the Planning and Performance Management Directorate of the National Research Council Canada (NRC)’s Strategy and Development Branch to lead an evaluation of the GRDI. An Interdepartmental Evaluation Working Group (IEWG) was established, to support the evaluation process. An independent firm, Science-Metrix, was contracted to design and implement this evaluation and a mixed-team approach was used to conduct the fieldwork (evaluators from both Science-Metrix and NRC-SDB-PPM).

Evaluation Objective and Approach

The main objective of the present evaluation is to assess the performance and relevance of the GRDI with regard to its targeted outcomes. Five core issues related to performance and relevance are assessed in accordance with the Treasury Board Policy on Evaluation. A horizontal evaluation was performed in 2006 and addressed program design and delivery issues by examining the first two phases of the Initiative (1999-2005). The present evaluation focuses on the achievement of outcomes (impacts) of research and development (R&D) supported during Phase III of GRDI (2005-2008).

Although it draws on data collected from individual departments/agencies, the evaluation examines the Initiative as a whole, rather than the performance of each individually. The evaluation approach was comprised of the following three methods:

  1. A management, delivery and output review, consisting of a review of program documents and external literature, and interviews with stakeholders, both internal and external to participating departments/agencies.
  2. An impact assessment through 15 project reviews, including review of project documentation and interviews with federal project leaders, collaborators and users of R&D results.
  3. An impact web survey of project collaborators and users of R&D results.

In total, 158 distinct individuals were consulted through this evaluation (44% were external to participating departments/agencies).


Profile of the GRDI

  • Objective: The GRDI objectives focus on building and maintaining genomics human resource and infrastructure R&D capacity in Canadian federal science-based departments/agencies and developing comprehensive networks of research collaboration in the field.
  • Multidepartmental profile: The GRDI is a multidepartmental funding initiative. Currently, the following seven departments and agencies participate: Agriculture and Agri-food Canada; Environment Canada; Fisheries and Oceans Canada; Health Canada; Public Health Agency of Canada; NRC; and Natural Resources Canada.
  • Funding phases: Launched in 1999, the GRDI has undergone four phases of funding and activities, each of which has been allocated in three-year blocks. While Phase I (1999–2002) focused on capacity building and Phases II and III (2002–2008) on the development and application of genomics research testing procedures and tools, the current Phase IV (2008-2011) aims to sustain and expand the genomics activities undertaken in the first three phases.
  • Funding profile: Since its inception, the Initiative has been supported with approximately $19.9 million in annual federal funding, for an average of $60 million per 3-year phase and a total of $234 million over the 12-year period. Funding allocations vary between the seven participating departments and agencies and have remained unchanged since the inception of the Initiative in 1999. The GRDI funding allocations are expected to be supplemented with resources from both internal and external sources.
  • Management and governance: Each department and agency is also responsible for identifying targeted strategic R&D areas and establishing a competitive peer-review process for project selection and funding, as well as managing and reporting on progress and performance. Initiative-wide governance is provided by the ADM Coordinating Committee, which oversees collective management and the Interdepartmental Working Group (WG), which supports the work of the Committee. The GRDI does not have a formal secretariat; however, NRC facilitates the governance and management structure of the Initiative.

Key Findings

Performance: Achievement of Expected Outcome

  • The GRDI has enabled federal researchers to participate in and contribute to genomics research in a way that has kept pace with developments in the field, with the level of impact expected to increase in coming years. Investments in capacity building made during the first two phases have positioned researchers to now begin applying resources to the development and application of new or improved methods, products, processes or technologies in projects that are more translational and commercially oriented.
  • GRDI-funded researchers have made significant contributions to the development and advancement of fundamental genomics research.These advances have led to new and improved applications of this research towards more translational and commercially-oriented projects. Initial impacts are beginning to be seen on the regulatory and policy side, but remain fairly limited.
  • There have been a few instances of longer term impacts as a result of GRDI-supported research, although in many cases direct application of GRDI research results is limited by the lengthy timelines required for implementation and uptake. This includes developments in science-based regulations, policies and decision making, as well as impacts in the strategic outcome areas of improved health care, reduced environmental impacts and improved competitiveness of Canadian companies.
  • The GRDI has successfully addressed and satisfied stakeholder needs, despite the fact that most users of the R&D were not systematically identified or integrated into the research process. Where they occurred, close working relationships and effective communication between scientists, their departments, collaborators and users of R&D results were essential in designing mandate-driven, application-oriented research built on common objectives.
  • Collaborators and users of R&D results were satisfied with the manner and extent to which research results were transferred to interested parties. The most common means for transferring GRDI-related scientific knowledge and technologies were conventional research dissemination mechanisms. R&D results with potential for subsequent commercial applications were generally transferred through patent applications and material transfer agreements.
  • The Initiative has allowed participating federal departments/agencies and Canada as a whole to establish and consolidate their position as credible contributors to genomics research and applications at the national and international level. However, the profile and visibility of the Initiative itself and funded researchers could be further enhanced within Canada.

Performance: Demonstration of Efficiency and Economy

  • GRDI-funded departments and agencies have implemented specific processes aimed at maximizing efficiency, increasing transparency and reducing the redundancy of R&D project delivery and management processes. These processes have included priority setting, the selection of projects to align with these priorities and increased coordination activities between departments. Further opportunities for continued improvement in these areas are being considered and included in plans for future GRDI horizontal management activities.
  • Internal and external stakeholders ranked most management practices tied to the GRDI highly and believe that efforts to increase program efficiency have been successful. Some issues were raised at the departmental level, such as the inability to use GRDI funds for salary support, a lack of communication between GRDI management and departmental staff, and long-term planning challenges due to uncertainty surrounding project funding and program continuity.
  • According to Annual Performance Reports, GRDI investments have been complemented with additional resources from departments or other sources at a ratio of 1.5 times the GRDI investments. The financial review of Phase III projects revealed that a significant proportion of the leveraged funds came from the internal (A-base) funds of the respective departments/agencies and only a small proportion was leveraged from external sources. However, multiple lines of evidence cast doubts on the validity of data collected on leveraging and further efforts would be required to conclude on this issue.
  • The cost-effectiveness of R&D projects is perceived to be high by a large proportion of collaborators and users of R&D results involved in Phase III projects. However, cost data were not readily available across all departments for the evaluation; therefore, it was not possible to perform a cost-effectiveness analysis of GRDI funding overall.
  • The allocation of funding to support multiple departmental mandates constitutes the main added value of the horizontal nature of the Initiative. It enabled the federal government to establish genomics R&D capacity and to demonstrate its potential to address issues in several niche areas.
  • Preliminary evidence suggests that the fixed departmental funding distribution established at the inception of the Initiative impedes the overall cost-effectiveness of the Initiative. One of the most valuable features of the overall Initiative lies in the application of funding in support of multiple departmental mandates. However, these departmental funding allocations have remained unchanged since 1999, and preliminary evidence indicates that this fixed distribution may not represent the scientific advances in genomics and the current capacity of mandate-driven research within participating departments/agencies. Individually, Phase III projects are seen as being highly cost-effective and their ultimate merit and value in relation to the GRDI investment are seen to have been significant.
  • Given the fairly low level of multi- or inter-departmental collaboration in projects supported by the GRDI, perceptions of the cost-effectiveness and utility of the horizontal nature of the Initiative were mixed. Although some expressed concerns that this low level of collaboration may have diminished resource use efficiency and research returns, others believed that the interdepartmental delivery of projects is not necessarily appropriate for all types of research investigations and that moving towards a more horizontal delivery approach will further limit the GRDI resources available for some departments.
  • Evidence suggests that there are opportunities to further support specific inter-departmental genomics R&D projects in high profile priority areas where existing R&D capacity and progress of individual departments/agencies are complementary.Such integration would maximize both the efficiency and cost-effectiveness of targeted R&D projects by leveraging the existing research capacity, shared priorities and advancement of applications.
  • Overall, little duplication or overlap appears to exist between GRDI-funded research and other genomics research in Canada. Duplication of effort was largely avoided through the competitive processes used to select GRDI-funded R&D projects, which feature a combination of senior management decisions and peer-review of projects. Efforts are being made to increase collaboration between the GRDI and Genome Canada, but the current funding eligibility criteria preclude the formation of a more formal partnership.

Relevance: Alignment with Government Priorities

  • GRDI-supported R&D is directly aligned with government S&T priorities and the mandates and strategic objectives of the individual departments. A defining characteristic of the Initiative is its strong alignment between the objectives and results of GRDI research with government priorities and individual departmental mandates. GRDI-funded projects must demonstrate a clear correlation between project goals and these broader objectives in order to be recommended and approved, both by peers and senior management. The GRDI’s governing bodies work to ensure that this high degree of alignment is reached.

Relevance: Alignment with Federal Roles and Responsibilities

  • The role played by the federal government in this area is appropriate and necessary and the Initiative has supported mandate-driven genomics R&D that generated results that have not been achieved elsewhere. Research funded by the GRDI has answered to a specific need that is not being fulfilled by the other genomics R&D being conducted in Canada—it represents the single most important mechanism in Canada through which mandate-driven R&D results in genomics can be generated. The Initiative’s strategic alignment with federal government and departmental objectives and priorities is well suited to research that is exploratory, conducted in support of regulation or addresses specific issues of importance to Canada.

Relevance: Continued Need for Program

  • There is a continued need for an initiative that supports genomics R&D within select federal departments/agencies, as well as a need for the Initiative to be managed horizontally. Although the last decade has seen many significant changes to the context in which the GRDI operates, none have diminished the relevance of the Initiative or its value to Canadians. The existing horizontal structure is credited with facilitating interdepartmental collaborations and the integrated sharing of resources, which have helped to build a core capacity in many areas of applied genomics. However, more effective mechanisms for communication between GRDI senior management and program-level staff are needed to support interdepartmental coordination and program transparency.

Conclusion and Recommendations

The evaluation found that the GRDI as a whole is relevant and effective. To support ongoing program improvement, the evaluation process has identified design and delivery considerations to heighten the program’s likelihood of success.

Recommendations are presented below.

Recommendation 1:

Develop opportunities to support specifically interdepartmental genomics R&D projects with shared resources in high profile priority areas. Given the economic context and fixed resources available for research at this time, the support of such integrated projects should be small in scale in order to minimize the reduction of funds for ongoing departmental mandate-driven genomics R&D and to leverage the existing research programs and capacity and advancements of applications. In addition, the selection of high-profile areas of priority for Canada should build on existing complementary strengths, shared departmental priorities and strategic outcomes, and progress made by departments.

Response:
Accepted

Planned Action:
Phase V of the GRDI will include the launch of pilot interdepartmental projects with shared resources.

The GRDI will target about 20% of its funds to these pilot projects.

Approaches that encourage interdepartmental projects with shared resources will continue to be supported by the GRDI ADM Coordinating Committee.

The selection of high-profile areas to be supported by shared resources will use decision-making criteria related to strategic opportunities and federal science and policy priorities, the unique role and leadership of government scientists, departments capacity and strengths, benefits from an integrated federal genomics R&D approach, and strong potential for collaboration and leverage with Canadian and international research providers.


Recommendation 2:

Should the Initiative be renewed, a significant proportion of the funding to individual departments should continue in order to build on the research capacity and expertise generated in their respective niche areas. This continued support will allow the federal government to take full advantage of the demonstrated potential of genomics R&D in supporting departmental mandates and strategic objectives. Research funded through the GRDI is now well positioned to produce more operational impacts as it moves from the proof-of-principle stage into one more translationally oriented.

Response:
Accepted

Planned Action:
While Phase V of the GRDI is being realigned to support interdepartmental projects, 80% of the funds will support individual departmental/agency priorities and mandates through predetermined allocations.

In addition, some of the mandated research projects are performed in collaboration or partnership with other government departments/agencies, which aligns with recommendation 1.

Recommendation 3:

Should the Initiative be renewed, review the distribution of funding among participating departments/agencies. According to results-based management principles, allocations should consider the current and potential level of activity, capacity and performance of mandate-driven genomics R&D conducted in participating departments (including the scientific excellence, progress and potential impact within their respective areas). This will support strategic results-based management and accountability as well as transparency in the allocation of funds.

Response:
Accepted

Planned Action #1:
Within the limitations of using the existing GRDI funding level for Phase V of the GRDI, a shared priorities fund will be built from departments on a pro-rata basis of their allocations, meaning that departments receiving the highest share of the funds will also contribute the most. These funds will then be redistributed among participating departments to reflect their level of activity in shared priority projects. Recognizing the importance of their regulatory role, the CFIA will have the opportunity to participate in the shared priority projects.

Planned Action #2:
When seeking authority to implement the new GRDI framework for 2014 onwards, departments and agencies will be in a position to fully address this recommendation. Departmental business cases will form the key driver for determining the distribution of funding among participating departments to support core mandated research. The business cases already developed will be updated to reflect current priorities, level of activity, capacity and performance, and will be augmented to include consideration of support to genomics expenditures in each department.


Recommendation 4:

Develop and implement a communication strategy to increase the visibility and profile of the Initiative (including the profile of funded genomics R&D, departmental capacity, progress and performance reporting/evaluations), both within and outside of federal departments/agencies. This will increase awareness and facilitate opportunities for collaboration among stakeholders and other genomics initiatives at the federal, national and international levels. Importantly, this strategy should include specific means to increase levels of communication and exchange between GRDI stakeholders in participating departments (including current and potential GRDI-funded researchers, collaborators and users of R&D results).

Response:
Accepted

Planned Action:
Communications activities at the corporate GRDI level require dedicated resources. While some activities are currently undertaken (detailed integrated GRDI annual progress reports, the development of a web site for the GRDI), they are currently limited. Phase V of the GRDI proposes that NRC’s Coordination Office support the GRDI to manage interdepartmental projects and initiative-level activities, including the development and implementation of a communications strategy for the GRDI. The NRC Coordination Function will work with individual departments/ agencies to encourage them to promote GRDI when opportunities arise.

Recommendation 5:

Develop mechanisms that further integrate users of R&D results in all stages of genomics R&D projects’ life cycles in order to ensure proper alignment of scientific progress with targeted potential uses and expected impacts (as outlined in the Logic Model). Efforts should specifically be made to ensure effective interactions at the transfer and adoption phases in order to obtain feedback for continuous improvement and future development. The Initiative should consider the integration of dissemination and transfer plans in project proposals that identify the nature of user(s) involvement and expectations, as well as a knowledge transfer/translation strategy. This will allow the Initiative to increase its focus on the ultimate translation of R&D results.

Response:
Accepted

Planned Action:
Stakeholder engagement is already integrated in the project proposal templates and decision criteria of some GRDI departments. The GRDI Working Group will assess best practices and will develop guiding principles to support the integration of users of R&D results throughout the projects life cycles. This will include the development of dissemination and transfer plans, in the strategic and project planning activities of all participating departments.

The communication strategy developed to address Recommendation 4 will also raise users of R&D results awareness of GRDI. This increased awareness should provide for more effective interactions leading to knowledge transfers and translation of R&D results.

Recommendation 6:

Continue to improve the tracking and reporting of performance, specifically to ensure that reliable information on total departmental investments and expenditures related to GRDI is available and understood. This would include data for all types of contributions that complement GRDI funding, which are collected and made available for ongoing performance management, reporting and evaluation processes. Participating departments/agencies should put in place processes to collect detailed financial profiles of GRDI-supported project/activities, including expenditures. In addition, the GRDI WG should work with participating departments/agencies to conduct a scan of the funding landscape for overall departmental/agency genomics R&D activities in order to determine the materiality of the GRDI and the relative importance of genomics in departmental R&D activities. This will inform the review of the distribution of funding among participating departments/agencies (Recommendation 3) and could be done prior to each renewal of the Initiative.

Response:
Partially accepted

Planned Action #1:
A new Performance Measurement Strategy will be developed and implemented for the GRDI that will clearly identify accountabilities and reporting requirements. These requirements will capture leveraging information related to GRDI projects.

Planned Action #2:
To the extent possible, these requirements will also capture other genomics-related activities that are not supported by the GRDI, based on existing individual department systems. However, genomics activities are often treated as an embedded tool in many program areas and are thus not identified for tracking or reporting purposes. Detailed financial profiles of non-GRDI genomics activities are thus expected to be difficult to obtain. Moreover, this information falls outside the scope of GRDI management and performance measurement activities.

Date modified: