ARCHIVED - Evaluation of NRC's Genomics and Health Initiative (GHI)
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
The National Research Council (NRC) launched the Genomics and Health Initiative (GHI) in 1999 to exploit advances in genomics and health, building on NRC's expertise in its biotechnology research Institutes and its regional innovation networks across Canada. As NRC's first large-scale horizontal research program, GHI aims to encourage close collaboration between its research Institutes, and its partners in other government laboratories, the private sector and universities, both nationally and internationally. In addition to research collaborations, GHI focuses on transferring the knowledge developed at NRC in genomics and health to a variety of industry sectors.
The evaluation of NRC's GHI was carried out in 2005 by the Planning and Performance Management Directorate (PPM) of NRC's Strategy and Development Branch. This evaluation was carried out at the request of NRC's Senior Executive Committee (SEC), in accordance with NRC's evaluation cycle and Treasury Board of Canada Secretariat (TBS) policies. The primary reasons for conducting an evaluation of GHI at this time were to:
- Provide NRC Executives and Senior Management Footnote1 with information on GHI program progress, quality, and management.
- Collect information to support NRC strategic program planning and on-going management of the horizontal research program.
- Provide an analysis of the strengths and weaknesses of the horizontal research program delivery model as support to the NRC Renewal Project.
- Document the outputs and early impacts of GHI funding.
NRC Senior Executive Committee approved the Terms of Reference for this evaluation in March 2005.
The evaluation focused on Phase 2 of the Initiative (2002-2003 to 2004-2005) and the transition to Phase 3 Footnote2 (2005-2006 to 2007-2008). The evaluation issues examined came under the following broad headings:
- Cost-effectiveness / alternatives; and
- Program design and delivery.
Specifics on the evaluation questions can be found in Appendix A. In addition, the federal government's Expenditure Review Committee's questions were taken into account in the development of the evaluation issues.
The evaluation methodologies used included a document review, an analysis of administrative and performance data, a literature review, a technical peer review, case studies of funded research programs, key informant interviews, and a review of international programs. The project was guided by an Evaluation Advisory Committee (EAC) which had both internal and external representation. The EAC provided input into the evaluation methodologies and advice on the analyses and outputs of the evaluation.
Key evaluation conclusions and recommendations are summarized below.
Conclusions and Recommendations – Relevance
A fundamental issue addressed in evaluations is the ongoing relevance or rationale for the initiative. Issues of relevance focus on whether the initiative continues to respond to a public interest or need. GHI continues to serve the public interest and address federal government priorities. However, it is perceived by some that the link between GHI-2 and NRC priorities is not clear (i.e., the pillars of NRC Vision 2006).
Government support for genomics research across all federal departments, agencies and initiatives (e.g., NRC, Genome Canada, CIHR, etc.) has played a legitimate and necessary role in enabling Canada to develop capacity in the area of genomics research. This support continues to be valid given the characteristics of this research.
Recommendation 1: NRC should continue to fund the Genomics and Health Initiative and seek renewal of the Genomics R&D Initiative for a fourth phase.
Response: Discussions to address the Phase IV program and competition design will be initiated with the GHI Directors General Committee and in consultation with the GHI Program Coordination Committee. Recommendations from the GHI Evaluation and lessons learned from GHI-3 will be incorporated into the Phase IV program design.
Recommendation 2: NRC should ensure that, once strategic priorities are articulated through the Renewal Initiative, GHI's objectives clearly align with these.
Response: GHI's objectives will be reviewed after the NRC strategic priorities have been articulated through the Renewal Initiative. Revisions will be made to ensure that the objectives are clearly aligned with NRC strategic priorities. The VP (Life Sciences) will develop the revised objectives in consultation with the GHI Directors General Committee.
Conclusions and Recommendations – Success
This evaluation addressed the impact of the GHI-2 activities. It also examined the extent to which the impacts would have been achieved without GHI funding. The investment in genomics infrastructure and research in Phases 1 and 2 of GHI has led to beneficial contributions to the advancement of scientific and technical knowledge in a number of research areas.
In GHI-1, NRC was putting in place the building blocks for genomics research; during GHI-2, researchers put those genomics approaches to work in their research. While it is not realistic to expect market impacts in the short term from genomics research programs, GHI-2 did make progress towards creating new technologies for Canadian industry. Market relevance will be an important performance indicator in GHI-3.
GHI's goal of supporting and participating in the development of a "national innovation network" was poorly understood by Initiative participants. Networking within NRC was strengthened by GHI-2, but support and participation in the development of external networks on a formal basis varied across GHI-2 Programs.
GHI represents a first step towards a more integrated NRC. GHI was successful in fostering informal collaboration and greater integration across NRC Institutes in the area of genomics research. Some concern about the unevenness of collaboration across Institutes remains. There is confusion about the level of support a business development office can provide to researchers to help with entrepreneurialism.
The initial objectives of most of the GHI-2 Programs were very ambitious but progress has been made against them. The level of detail between the Program Charters and the Performance Reports was inconsistent making it difficult to compare actual outputs to intended outputs. GHI-2 has had a number of additional positive impacts, particularly in increasing the number of human resources trained in the field of genomics and health.
The evaluation identified a range of barriers/challenges. Some are within the realm of the Initiative to solve and others are outside GHI's sphere of control. Interviewees who identified administrative issues as challenges consider many of these to have been dealt with through the GHI-3 governance structure.
Expected results for GHI-2 were not clearly defined nor communicated (e.g., through a results chain or logic model).
Recommendation 3: A portfolio approach should be taken in future GHI phases, funding a balance of new basic research Programs and Programs that are more applied. For those Programs that propose "closer to market" applications, a market assessment study should be performed as part of the proposal process to examine the potential impacts of the work.
Response: In GHI Phase IV, a more formalized portfolio approach will be established as part of the program evaluation criteria for use by the GHI Expert Panel, and as a guide for NRC Senior Executives when making program funding decisions. The portfolio approach will be based on funding a balance of programs with shorter-term commercial potential as well as those with longer-term research objectives.
Lessons learned from the market analysis studies conducted by NRC institutes (e.g. BRI) and as part of the pilot study conducted by NRC Corporate Services, will be used to develop the specific requirements for market analysis studies that will be implemented into the GHI Phase IV evaluation criteria. The GHI Expert Panel will also be strengthened to include additional members with business and marketing expertise.
Recommendation 4: Efforts should continue to build upon the progress made in GHI-2 in integrating activities across NRC. In addition, the complementarity between GHI Programs and other genomics and health research across Canada should be strengthened through increased collaboration with organizations external to NRC.
Response: Integration and leverage were important elements in the GHI-3 proposal evaluation criteria. Proponents were encouraged to assemble integrated, multi-disciplinary research programs that involved more than one NRC institute, and to include research coordination and collaboration with other government departments and agencies, academia and/or industry. An inter-departmental Genomics R&D Coordinating Committee oversees the collective management and coordination of the federal Genomics R&D initiative, and ensures that collaborations between federal departments are pursued wherever relevant and possible. The requirement for inter-institute collaboration in GHI Phase IV research programs will be strongly encouraged and collaboration with organizations external to NRC will continue to be an important criterion in proposal evaluation.
Recommendation 5: A GHI specific Logic Model, which defines expected results in the near, medium and long term, should be established to facilitate more effective performance measurement. Objectives must be clearly stated and performance should be reported against the stated objectives at both the Initiative and individual Program level. Meaningful indicators that are linked to clear objectives or strategic plans need to be identified, agreed to (i.e., between management, researchers, VPs, etc), tracked and accurately reported upon. The need to track performance and the allocation of resources should be balanced against the associated administrative burden.
Response: NRC is currently leading an evaluation of the interdepartmental Genomics R&D Initiative. As part of this evaluation, a revised Results-based Management and Accountability Framework (RMAF) will be prepared for the Genomics R&D Initiative. The consultant used to revise the RMAF will be used to develop a GHI specific Logic Model, which will define expected results in the near, medium and long term.
Steps have already been taken in GHI-3 Program Charters to better define program objectives and key deliverables/milestones and research programs are required to submit quarterly reports that are focused on progress reporting against research objectives and milestones. In GHI Phase IV Program Charters there will be a focus on improving the definition of research objectives and how they link to the Initiative strategic plan, and on better articulating and linking key program milestones.
Recommendation 6: Efforts should be made to clarify the roles and responsibilities of the business development function and communicate it to researchers and business development officers so there is a common understanding of what activities are part of the function.
Response: NRC is launching a comprehensive review of NRC business activities to ensure we have the right business activities and the right support for these activities in the future. Specifically, the Terms of Reference for this Review call for examining activities with a view to revamping support to achieve the goals under our Renewal Plan, to capitalize on opportunities under Portfolio management; and to work better 'horizontally'. This Review will ensure that GHI issues and opportunities uncovered during the evaluation are addressed appropriately. This would include clarifying business development and marketing roles and communicating them across the Program and the Council.
Conclusions and Recommendations – Cost-Effectiveness / Alternatives
The evaluation examined cost-effectiveness in terms of whether the most appropriate and efficient means were used to achieve the Initiative's objectives, including areas such as alternative approaches and GHI's three-year funding cycle. There was general agreement that the competitive approach GHI is presently using is working well. It was agreed that this approach provides more accountability.
There are a number of sources of funding for biotechnology, genomics and health research within the federal government. External interviewees did not seem to have sufficient details on the various organizations and their roles to conclusively say whether GHI and other federal organizations supporting genomics research are complementary or overlapping.
Many of the GHI-2 Programs did not manage their research as a fixed three-year project, causing disruption when proposals for new phases were not funded. New programs tended to have a more difficult time than existing programs to get up and running. The three-year funding cycle, presently set by TBS, however, is generally supported by GHI-3 participants.
Objectives for the Initiative as a whole and for the individual GHI-2 Programs were not sufficiently clear or specific. In most cases targets were also not identified, making it difficult to determine if the allocated level of resources was appropriate. The GHI Coordination Office and Scientific Leaders have learned from the GHI-2 experience. The GHI-3 Program Charters list objectives that are based on the funding received and not the funding requested.
Recommendation 7: In the context of the upcoming Genomics R&D Initiative evaluation, an in-depth review of the science directions and research thrusts of departments involved in Genomics R&D as well as other federal organizations including CIHR, Genome Canada, the Canada Foundation for Innovation, and the Canadian Biotechnology Strategy should be undertaken to determine the extent to which the different programs are complementary or duplicative. In this regard, the status and/or results of the current review by the Minister of Industry of federal government's involvement and investments in genomics R&D need to be taken into consideration.
Response: The evaluation of the interdepartmental Genomics R&D Initiative will provide an in-depth review of the science directions and research thrusts of federal departments involved in Genomics R&D. A similar review (i.e. the Genomics Review) of other federal organizations including the Canadian Institutes of Health Research, Genome Canada, the Canada Foundation for Innovation, and the broader Canadian Biotechnology Strategy has been initiated and is being led by Industry Canada. These evaluations/reviews will provide an excellent examination of how these different programs could be more effective. The GHI Coordination Office, in consultation with the GHI Directors General Committee, will take steps to address any specific issues or concerns that are raised in the recommendations and associated management responses of these evaluations/reviews.
Conclusions and Recommendations – Program Design and Delivery
Issues related to the appropriateness and effectiveness of program delivery (e.g., organization, communication, resources, program performance monitoring/ management) were examined in the evaluation. Overall, the GHI model is supported internally as an effective way to foster collaboration at NRC. Improvements made to the model and implemented in GHI-3 are supported by initiative participants. Due to a lack of expertise in managing large, collaborative programs, the hiring of professional project managers for GHI-3 is supported by researchers.
The GHI-3 governance, accountabilities and program management framework is generally supported by GHI-3 participants as an improvement upon the governance of previous phases of the Initiative.
A balance between "bottom-up" and "top down" planning approaches is necessary to manage competing objectives of addressing social issues and fostering creativity in science.
The proposal process used in GHI-3, including the use of external peer reviews, is effective but there are issues around reviewer recruitment. For GHI-3, senior executives could have more clearly articulated and communicated the rationale for funding decisions and made the final Program selection process more transparent. The level of detail requested at the GHI-3 proposal stage for resource information was viewed as requiring a high level of effort.
The GHI-2 approach to performance reporting was not considered to be effective, and was not broadly supported by interviewees.
GHI has had a positive impact on NRC by moving the organization towards a culture that supports horizontal research programs built upon multi-disciplinary teams, cross-Institute collaborations and a project-based management approach.
Recommendation 8: The GHI Coordination Office should continue to help support the Scientific Leaders in the area of project management (e.g., training, reference materials, information sessions, workshops) with special attention given to those with less experience. The Coordination Office should help to facilitate the sharing of good management practices between experienced Program Drivers/Scientific Leaders and new ones.
Response: One of the key roles of the GHI Program Coordination Committee is to share best management practices amongst Scientific Leaders. This approach will be strengthened in GHI Phase IV by the introduction of a project management workshop to be held during the launch of GHI-4. The GHI Coordination Office, in consultation with participating institutes and Corporate Services, will develop the workshop. The workshop objective will be to provide guidance on GHI and institute program management requirements and performance management, and will include presentations on best practice from experienced GHI scientific leaders. Consideration will also be given to including presentations from professional project managers in the private sector. With respect to specific project management training for Scientific Leaders, the GHI Coordination Office can provide financial support and facilitate specific training to support the delivery of horizontal research programs. However, the training and development of Scientific Leader staff is an institute responsibility and any efforts in this area would require coordination and approval from institute management.
Recommendation 9: Implementation of the new governance and accountability structure put in place for GHI-3 should be monitored as to its effectiveness as Phase 3 progresses.
Response: The GHI Directors General Committee as well as the GHI Coordination Committee will monitor the effectiveness of the new GHI Governance and Accountability structure. This will be an annual agenda item for both committees, and recommendations made by the committees will be used to guide governance model revisions. The effectiveness of the structure and operation of the various committees will be a key issue for discussion. Any major changes to the governance and accountability framework would need to be agreed to by the GHI DGs Committee and submitted to SEC for formal approval.
Recommendation 10: Proposals for future phases should be streamlined, and should focus on the articulation of clear and realistic objectives and milestones. There should be more transparency in the final Program selection process, including better articulation and communication to the Scientific Leaders of the rationale used for final funding decisions. Consideration should be given to tracking the time taken to develop proposals for any future phases.
Response: Changes to streamline and focus proposals on the articulation of real objectives and milestones have been initiated in GHI-3 and this will be built upon in GHI Phase IV. Efforts to improve the transparency of the final program selection process were also introduced in GHI-3 and additional steps will be taken in GHI Phase IV. For example, a more formalized proposal evaluation system will be developed to provide specific feedback on each evaluation criterion. This information will then be used to create evaluation summary documents that will be communicated to each proposal proponent. The overall objective of this change will be to better articulate the rationale used in the decision making process.
Recommendation 11: The Programs Charters need to include specific plans on how the project will end in the event funding is discontinued after three years.
Response: Based on evidence presented in the evaluation, there is clearly a perception by some participants in GHI research programs that funding is likely to continue beyond the nominal three-years of program approval. In GHI-2 and GHI-3, competition guidelines indicated that programs were to be planned and funded for a limited duration (typically three years), and that associated research objectives and milestones were to be prepared on this basis. In GHI Phase IV, program duration and the process for funding renewal will be more explicitly presented in program documentation. Additionally, as part of the GHI Phase IV Program Charter development, a new requirement will be introduced that will require each program to prepare a closure strategy in the event that funding is discontinued. As part of the GHI-3 program closure strategy, proposals may be put forward to seek continuing funding for a short period to ensure the completion of critical work. Efforts will be undertaken for Phase IV funding decisions to be made six months in advance of GHI-3 completion so that adequate time is provided to implement closure strategy plans.
Recommendation 12: To make optimum use of external reviewers, an independent assessment of past performance by experts should be integrated into the Program selection process of any new GHI phases. Peers reviewers should be asked not only to review proposed work, but also provide an opinion on past performance at the same time. Specific questions relating to research completed in the previous phase (e.g., achievement of objectives, quality and relevance of the outputs/outcomes) should be included as part of the proposal review.
Response: GHI program proposals are required to include a Background Section that provides progress to date in areas directly related to the proposal, and a list of outputs (e.g. publications, patents, licensing agreements, etc.) related to the research. For existing GHI programs, peer reviewers and members of the GHI Expert Panel have used this section to assess past performance. In the GHI Phase IV Competition, this section of the proposal template will be strengthened and the requirement to explicitly report on progress towards research objectives and milestones in areas directly related to the proposal will be made a requirement. Additionally, the GHI-3 performance reports will be made available to the reviewers of GHI-4 proposals.
GHI-3 research programs are required to submit quarterly performance reports that are reviewed by program Steering Committees and by the GHI Directors General Committee. As well, it is planned for the GHI Expert Panel to conduct formal mid-term reviews of GHI research programs, with recommendations made to the Vice President (Life Sciences), who will determine if funding for a program will be continued, reduced or reallocated. These existing performance evaluation mechanisms are considered comprehensive, and the integration of additional, independent assessments of past performance as part of future GHI program selection processes are seen to be unnecessary.
Conclusions and Recommendations – Lessons Learned – Horizontal Research Management
One of the objectives of this evaluation was to provide an analysis of the strengths and weaknesses of the horizontal research program delivery model as support to the NRC Renewal Project. The general feeling internally is that people support the GHI horizontal model and believe it is working well. This is mainly, however, due to its uniqueness at NRC.
Recommendation 13: Before replicating the GHI model for other NRC horizontal initiatives, the following issues need to be taken into consideration:
- The effectiveness of a GHI-3 type governance framework;
- The matching funds approach and the affect it has on Institutes' ability to participate in multiple horizontal initiatives;
- The balancing of the competitive process and accountability requirements with the demands on NRC scientists to prepare proposals at the Institute and horizontal levels and the ability to find external experts to participate in reviews; and
- The establishment of a suitable funding cycle in which the desired impacts are achievable.
Response: NRC Senior Executive agree to take these issues into consideration before putting in place any future horizontal initiatives at NRC.
Report a problem or mistake on this page
- Date modified: