ARCHIVED - Horizontal Evaluation of the NRC Genomics R&D Initiative

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Executive Summary

Description of the Genomics R&D Initiative

In March 1998, the National Biotechnology Advisory Committee (NBAC) released a report recommending ways to position Canada as a leading global player in biotechnology by the year 2005. During the same period, the National Research Council and the Medical Research Council (now the Canadian Institutes of Health Research) held discussions with stakeholders as part of the Canadian Biotechnology consultations. Genome research was clearly identified as an important priority for Canadian biotechnology research and development (R&D).

The NBAC recommended that a top priority be placed on several actions, including political championship and increased funding to Canada's genome program. The February 1999 Budget provided $55 million in funding for genomics R&D in six departments and agencies under the Canadian Biotechnology Strategy (CBS).

The Genomics R&D Initiative was launched in 1999 and is currently in its third three-year phase:

  • Phase 1 – 1999-2000 to 2001-2002
  • Phase 2 – 2002-2003 to 2004-2005
  • Phase 3 – 2005-2006 to 2007-2008

Its stated objective is:

To build the capacity inside government laboratories to do... biotechnology research (related to genome sciences), which will strengthen the regulatory system and bring the benefits of revolutionary advances in research and technology to a variety of Canadian industry sectors and regions. The new technologies are expected to have a dramatic impact on industrial competitiveness and economic growth. They are also expected to bring significant social benefits, e.g. better therapeutics, cleaner environment and better management of natural resources.

Footnote1

Six departments currently receive funding under this Initiative: Agriculture and Agri-Food Canada (AAFC), Environment Canada (EC), Fisheries and Oceans Canada (DFO), Health Canada (HC), National Research Council Canada (NRC), and Natural Resources Canada (NRCan). It should be noted that throughout this report, that the Public Health Agency of Canada (PHAC), which was established as a separate agency in 2004, was considered to be part of HC. The administration of the Genomics R&D Initiative funds for both HC and PHAC was coordinated through HC's Departmental Biotechnology Office for Phases 1, 2 and 3.

Purpose of the Evaluation and its Intended Audience

It is the policy of the federal government of Canada that departments evaluate their key policies, programs, functions and initiatives strategically and cost-effectively and to use the findings in decision-making and reporting. The Phase III Program Framework (2005-2008) states that a targeted evaluation of the Initiative was to be conducted in 2005-2006. This interdepartmental, horizontal evaluation study focused on the Initiative's short-term outcomes given that it is too early to measure impacts and to address other longer term issues. The intended audience of this report includes:

  • Treasury Board;
  • the Interdepartmental Genomics ADM Coordinating Committee;
  • the Interdepartmental Working Group for the Genomics R&D Initiative;
  • program managers in the six funded departments; and,
  • the Canadian public.

Methodology

The evaluation addressed issues related to the relevance of the Initiative, its early success, its cost-effective or alternatives, and its design and delivery. The methodologies used for this evaluation included:

  • a document review;
  • 26 in-depth interviews with departmental managers;
  • 61 in-depth interviews with researchers;
  • 19 in-depth interviews with departmental stakeholders (including project partners, beneficiaries or others with an interest in the genomics R&D activities of specific departments); and,
  • 9 in-depth interviews with horizontal stakeholders (including representatives of Central Agencies, other biotechnology departments / programming, or others with an interest in genomics R&D).

While there are some imbedded strengths and weaknesses associated with each of these methodologies, overall, the approaches and sample sizes used for this evaluation resulted in a strong and reliable horizontal evaluation, which provided the evidence to conclude on all issues. Additionally, the overall evaluation methodology is strong because multiple lines of evidence were used for all issues.

For more details, please refer to Section 2.0 of the main evaluation report.

Main Evaluation Findings

The main evaluation findings are summarized according to the issue categories previously identified:

  • Relevance:

The evidence from documents and from management, researcher and stakeholder interviewees revealed that there is an ongoing and ever evolving need for an initiative that supports capacity building inside government laboratories to do genomics R&D. While specific departmental needs differ and while the Initiative has increased the genomics R&D capacity of the funded departments, there is an ongoing need to maintain and grow that capacity and therefore support for genomics R&D. Additionally, the evaluation revealed that there is a legitimate and necessary role for government in this area, particularly given the importance of genomics R&D in the context of the broader Canadian Biotechnology Strategy, and the need for credible research results to inform policy, regulation and other governmental decisions.

For more details, please refer to Section 3.0 of the main evaluation report.

  • Success:

The evidence from documents and management, researcher and stakeholder interviewee feedback indicates that the first phase of the Initiative was successful in building capacity inside government laboratories to carry out genomics research. It was uncovered that there was limited capacity in most of the six funded departments before the Initiative and that the labs now have the human resource capacity as well as the tools, equipment, infrastructure and networks required to undertake genomics R&D. This capacity has helped labs benefit through the ability to undertake other genomics R&D projects using the capacity built in earlier phases. Additionally, it has helped strengthen other areas of research in departments. There was evidence of use of the research results in other (non-genomics) applications. Additionally, the labs continue to benefit from this capacity through ongoing projects, use of previous results in other projects, and ongoing involvement of the scientists in projects. Additionally, through the projects, the departments have established formal and information collaborations with Canadian and international organizations (governmental organizations, universities, non-governmental organizations and private sector organizations).

The key facilitating factor, as identified by the wide range of interviewees (managers, researchers and stakeholders), is the additional focussed funding available to departments to do this type of research. This additional funding also facilitated the hiring and training of highly qualified personnel (HQP) and other technical staff.

However, there were also impediments to success. There was evidence that the money was insufficient to address the genomics research priorities in the departments, in particular in those departments where the funds are more limited. Additionally, the three-year funding cycles, caused a delay in the release of the funds in the first year of the first phase. This delay resulted in delays in the proposal approval process and in the release of funds. Since personnel needed to be hired for many of the Phase 1 projects, the delay in the release of funds was further impeded by the hiring process. This delay in the release of funds was therefore a major impediment as it did not allow enough time to complete the projects. Another impediment involved the uncertain nature of the funding (three-year funding cycles) which could lead to human resource challenges (i.e. attracting and retaining highly qualified personnel). Another major impediment was the result of a Treasury Board ruling (starting in April 2006) that, while federal labs may continue to participate in Genome Canada projects, they cannot receive Genome Canada funding except in special circumstances. This has a major negative impact on the types of projects and collaborations that became possible in Phase 3.

While it is fairly early to report longer-term impacts, there is early evidence that the Initiative is successful. Since the evidence gathered in interviews shows that the Initiative is incremental (genomics programming would not likely be in place in departments, many of the projects would not have been undertaken, others would have been delayed, of smaller scopes or otherwise negatively affected), the success to date can be directly attributed to the Initiative.

For more details, please refer to Section 4.0 of the main evaluation report.

  • Cost-Effectiveness/Alternatives

A review of other genomics R&D programming in Canada revealed that the Genomics R&D Initiative complements rather than overlaps or duplicates other federal or provincial initiatives related to genomics or biotechnology. Other organizations involved in genomics R&D either have broader mandates than just genomics R&D, target different groups, and / or cover a narrower field of genomics R&D (such as just human genomics). Managers, researchers and stakeholders confirmed that they were unaware of other programs of a truly comparable nature. However, interviewees noted that it was important for the researchers in the departments to seek opportunities to work in collaboration with these other programs or initiative and that there were many instances where such collaborations had taken place. However, during the third phase, a Treasury Board ruling that, according to government policy, government departments cannot receive funding directly from Genome Canada (except in special circumstances). This change greatly reduced the level of interaction and complementarity between the Genomics R&D Initiative and Genome Canada.

As a separate fund with specific allocations to each department, the funding structure was deemed appropriate. However, the amount allocated to some departments was noted to be inappropriate in the context of the needs and priorities of those departments. The cost-effectiveness of the Initiative was difficult to assess because most departments did not have specific information on the actual cost of the Initiative, especially during the first phase. Departments did not have systems set up to capture these costs. However, the costs associated with the interdepartmental nature of the Initiative were believed to be minimal (interdepartmental meetings, TB Submissions, horizontal planning and reporting requirements). Some noted the costs associated with program renewal every three years. Along the same vein, it was noted that the uncertainty about the longevity of the Initiative could affect the types of projects undertaken and therefore the possible effectiveness of the Initiative. While the three-year funding cycle for this Initiative was deemed appropriate at the project level, it was believed to have added burden and costs (preparations for next cycle, difficult for human resource management, writing proposals every three years, burden on external reviewers, etc.).

For more details, please refer to Section 5.0 of the main evaluation report.

  • Design and Delivery

Most managers, researchers and stakeholders interviewed noted that the position of the Initiative was appropriate within the larger government biotechnology strategy. Several noted that they did not believe that the Canadian Biotechnology Strategy could guide the Genomics R&D Initiative. The CBS is broader and was therefore not believed to be directly relevant. Therefore, interviewees strongly believed that a separate fund was needed.

The interdepartmental governance model for the Genomics R&D Initiative includes an Interdepartmental Genomics R&D ADM Coordinating Committee and a Genomics R&D Initiative Working Group. In addition, the Initiative is part of the overall Canadian Biotechnology Strategy governance structure. Departmental managers believed that the Genomics R&D Initiative governance structure was effective, particularly in light of the fact that it was of limited burden to them. The role of NRC as the lead was also viewed positively. It was, however, noted that this was not viewed as a truly horizontal initiative and that it was therefore not governed as one. Nonetheless, concerns were expressed with the limited involvement of the ADM Coordinating Committee and with the lack of formal terms of reference for the working group.

Departmental processes have reportedly changed since the initial phase of this Initiative. In particular, the project approval processes have been deemed to have been greatly improved with the advent of more rigorous peer reviews of the proposals.

As noted previously, most departmental systems were not set up to capture detailed information on the costs associated with this Initiative, especially during the first phase. Additionally, most departments were not set up to keep track of the funds levered internally or externally. Nevertheless, there is evidence of leveraging through A-base matching funds as well as through collaborations with other organizations on projects.

In addition to the limited information on costs and leveraging, there is also limited evidence of adequate systems to capture good and complete performance information. This is due in part to the evolution of the Initiative and thus to the types of projects (and therefore possibly changing performance information requirements in Phase 1 than in Phase 2 or 3). The lack of adequate systems to collect and capture performance information is also in part due to the uncertainty of the Initiative (i.e. it would not have been cost-effective to invest resources in a performance measurement system for an Initiative that was funded for three years). Nevertheless, the Initiative is now at a stage in its evolution where enough is known on the Initiative to develop and implement better performance measurement systems in those departments with limited information. As a new Results-based Management Accountability Framework (RMAF) is being developed for the Initiative, this concern should be addressed.

There were several improvements suggested to the Initiative. These are reflected in the conclusions and recommendations which follow.

For more details, please refer to Section 6.0 of the main evaluation report.

Conclusions and Recommendations

The findings outlined above (and in the main evaluation report) support the following conclusions and recommendations.

Conclusion 1

The Genomics R&D Initiative is relevant as a critical element of the broader Canadian Biotechnology Strategy and is complementary to other elements of this broader Strategy such as the Canadian Regulatory System for Biotechnology. Given that genomics is still a relatively new and emerging technology, there is an ongoing need for government involvement in this field. Additionally the research results are required to support departmental mandates, the development of new regulations as well as to help enforce existing ones. As such, there is a legitimate and necessary role for government in this area.

Recommendation 1

Federal support for the Genomic R&D Initiative as a separate initiative of the Canadian Biotechnology Strategy should continue.

Note: the rest of the recommendations in this report assume the continuation of the Genomics R&D Initiative.

  • Success

Conclusion 2

The primary objective of the Initiative was to build capacity in federal labs. There is extensive evidence that the Initiative has built capacity inside government labs to carry out genomics research. Phase 1 built basic capacity which continues to be strengthened. As such, while there has been much progress made in this regard, there continues to be a need to build and maintain capacity in federal labs.

Recommendation 2

Support for capacity building should continue as there is an ongoing and ever evolving need for building and maintaining capacity in genomics R&D. The Interdepartmental Working Group should develop a strategy which identifies the mechanisms needed to ensure that new capacity will continue to be supported and that the existing capacity is maintained.

Conclusion 3

The capacity that was developed in Phase 1 has been used in Phase 2. There is extensive evidence of ongoing or continued projects, use of the tools developed or research results, and ongoing involvement of the same scientists. As such, Phase 1 translated into benefits for Phase 2. The increased capacity has also helped strengthen the research carried out in other areas of the departments.

No specific recommendation is required.

Conclusion 4

While there is some evidence of interdepartmental collaboration, it is limited. For example, different departments were initially at different stages of genomics research. In other cases, there was little commonality in the issues being explored. As such, there was limited opportunity for collaboration. However, as the capacity of departments has evolved, there may be increased opportunities for interdepartmental collaboration in future phases.

There has, nonetheless, been extensive evidence of collaboration with other research entities. The research projects have involved collaborative efforts on a national and international level with universities, governmental organizations, non-governmental organizations as well as private sector organizations. As such, the Initiative has been successful in strengthening linkages with appropriate research institutions.

Some departments participated in Genome Canada Competition I and II projects. Effective April 2006, federal labs cannot receive Genome Canada funding except in special circumstances (as a result of a Treasury Board ruling). As a result, projects are negatively affected, not only in their scope, but in the ability of the government labs to continue working with established collaborators.

Therefore, while the Initiative has been successful in strengthening linkages with appropriate research institutions, its continued success in this regard has been hampered, particularly due to the impact of the TB ruling regarding Genome Canada.

Recommendation 3

The Interdepartmental Genomics R&D ADM Coordinating Committee should explore specific ways in which interdepartmental projects could be encouraged to address government-wide genomics R&D priorities. This could include a pool of money set aside for interdepartmental projects as well as other options. This Committee should also precisely articulate these priorities and revisit them as needs evolve.

Recommendation 4

The Interdepartmental Genomics R&D ADM Coordinating Committee should also work with Treasury Board to investigate opportunities for federal scientists to participate more significantly in Genome Canada projects.

Conclusion 5

The main facilitating factor of the Genomics R&D Initiative has been that it is a focused funding source.

However, there are other financial elements of the Initiative that have impeded its success. The total amount of money available has become an impediment not only because there have been no inflationary increases in funding, but also because there is a need to re-balance the funding envelope to ensure that all departments have sufficient funding to address strategic priorities.

The three-year funding cycle has resulted in uncertainty. This has affected the scope of some of the projects as well as the ability to attract and retain highly qualified personnel.

Finally, the timing of the funding (delays in year one of each phase) has led to delays in meeting project milestones and, for start-up projects, to delays in hiring the required people for the research teams.

Note: There are several conclusions which can be addressed through more overarching recommendations. These recommendations are presented at the end of this section.

One of these deals with financial elements of the Initiative. Recommendations linked to Conclusion 5 are therefore presented at the end of this section.

Conclusion 6

There are significant differences in the way in which departments are allocating resources for program management and other overheads. As such, this has resulted in significant differences in the proportion of the funds which are available for the projects in different departments.

Recommendation 5

The Interdepartmental Genomics R&D ADM Coordinating Committee should clarify the rules on how the funds are used with respect to program management and other overheads and ensure that those rules are enforced.

Conclusion 7

The Initiative is highly incremental. Specific genomics R&D departmental programming would not be in place in the absence of this Initiative. As such, the great majority of projects would not have taken place and / or would have been seriously negatively affected as a result of delays, changes in scope, less qualified teams or for some other reasons. Therefore, the impacts of the projects are highly attributable to the Initiative.

No specific recommendation is required.

Cost-Effectiveness/Alternatives

Conclusion 8

The Initiative complements other federal or provincial initiatives related to genomics or biotechnology without undue overlap or duplication. However complementarity with Genome Canada has been reduced in the last few years as a result of a recent Treasury Board ruling.

See recommendations 1 and 4

Note: Conclusion 5 is also directly relevant to the cost-effectiveness issue dealing with the funding structure, in brief:

The focused funding is a strength of this Initiative. Problems with the funding structure include the total amount of money available, its three-year funding cycle, and the timing of the funding.

Recommendations linked to Conclusion 5 are presented at the end of this section.

Conclusion 9

It is not possible to conclude on the Initiative's cost-effectiveness because there is insufficient information in most departments on the specific departmental and interdepartmental costs associated with this Initiative. This is no reflection on specific departmental performance as departments were not required to track costs (nor would it have been cost-effective for them to set up specific systems to do so for an initiative with three-year funding cycles).

Recommendation 6

The summative evaluation needs to address the issue of cost-effectiveness in a way to reliably conclude on the cost and effectiveness aspects of the Initiative.

The departments should therefore ensure that improved cost information is available. The specific cost-effectiveness evaluation requirements will be outlined in the revised RMAF for the Initiative. This should include methods for a more thorough cost-effectiveness analysis at the time of the summative evaluation.

Conclusion 10

The three year funding cycle is appropriate at the project level but not for the Initiative. Overall, the uncertainty associated with the three year cycle has negatively affected the flexibility of the Initiative and aspects of its cost-effectiveness (see conclusions under Design and Delivery section).

Recommendation 7

Similarly to the Canadian Regulatory System for Biotechnology, the Genomics R&D Initiative should become an ongoing initiative with dedicated A-base funding. This will provide stability to the Initiative while ensuring an ongoing focused funding source for genomics R&D.

Conclusion 11

The benefits (sharing of information, communications with central agencies, etc.) resulting from the interdepartmental aspects of this Initiative, while limited, have outweighed the costs which have been minimal. The limited costs are, to a large extent, due to the fact that the Initiative is not structured as a truly horizontal initiative (nor was it intended to be).

Recommendation 8

In light of other recommendations, greater effort to strategically plan and to share the results of this Initiative will become important to its ongoing success. As such, horizontal management costs may increase but the benefits resulting from increased horizontal activity are expected to be greater.

  • Design and Delivery

Conclusion 12

It is appropriate to have this Initiative as a separate initiative within the larger federal government biotechnology strategy. Within departments, the Initiative is well integrated with other biotechnology programs (such as the Canadian Regulatory System for Biotechnology – CRSB, in the regulatory departments). However, there is limited integration with these programs from a horizontal perspective.

Recommendation 9

As per Recommendation 8, due consideration should be given to exploring opportunities for better horizontal integration with other biotechnology programs. As a result, horizontal management costs may increase but the benefits associated with horizontal management could be important in terms of ensuring complementarity while avoiding overlap and duplication.

Conclusion 13

The governance structure currently in place for this Initiative is of limited complexity and burden. As such, it is appropriate. However, some of its elements need improvement. The Interdepartmental Genomics R&D ADM Coordinating Committee is not providing the required level of leadership. Additionally, the working group has no documented terms of reference and could play a more active role in identifying areas for horizontal coordination or more common interdepartmental processes.

Recommendation 10

Without adding unnecessary burden to the Interdepartmental Working Group, specific terms of reference need to be defined for this group in order to ensure that, with ongoing support for this Initiative, its roles and responsibilities are clear. These terms of reference should include responsibilities for defining how funds can / should be allocated for departmental overhead costs as well as common approaches to some of the departmental processes (e.g., project selection, reporting, etc.).

Recommendation 11

The Interdepartmental Genomics R&D ADM Coordinating Committee should play a more active role in providing strategic direction for government wide genomics R&D priorities linking to other components of the Canadian Biotechnology Strategy.

Conclusion 14

Departmental processes (such as for project selection and approval) have evolved and improved over time.

Recommendation 12

Departments should continue to build on lessons learned and refine departmental processes as needed. The Interdepartmental Genomics R&D ADM Coordinating Committee should take steps to ensure that transparency and accountability continue as key elements in program proposal and approval processes, and that integrated performance reporting is formally implemented.

Conclusion 15

There is insufficient information to reliably conclude on the extent to which most departments have been able to leverage the funds provided through the Genomics R&D Initiative. There is, nonetheless, evidence of internal leveraging as well as leveraging through partnerships with other research organizations.

Recommendation 13

The summative evaluation needs to address the issue of leveraging in a way to reliably conclude on this issue. Departments will need to ensure that they put in place the required systems to meet the specific leveraging evaluation requirements which will be outlined in the revised RMAF for the Initiative.

Conclusion 16

There is currently no formal performance measurement system in place for this Initiative either horizontally or within the departments. As a result, there is limited performance information available. Recognizing that it is still fairly early to measure impacts, it is important to ensure that performance information available within departments is not limited to inputs and outputs measures.

Recommendation 14

The performance measurement system outlined in the upcoming revised horizontal RMAF for this Initiative needs to clearly define common performance measures and ensure that the appropriate tools are available to collect, analyze and report performance information without imposing undue burden or cost requirements to the departments.

Several of the conclusions presented above helped lead the evaluation team to the following series of recommendations:

Recommendation 15

The total funding for the Genomics R&D Initiative should be increased.

First, funding should be increase to compensate for inflation. It is important for departments to, at least, be able to maintain previous levels of research.

In addition, some of the additional budget should be used to re-balance departmental inequities. The funding for Phase 1 of this Initiative was initially allocated to the departments on the basis of existing capacity and it was expected that funding re-allocations would occur in later phases. This has not been the case. Nevertheless, the re-balancing cannot be done by reducing the existing funding levels of departments receiving a larger proportion of the total funding, as this could negatively affect the ability of these departments to undertake the genomics R&D required to support their departmental mandates.

Finally, some of this additional funding could be pooled for interdepartmental projects. Assuming that a pooled fund is set aside, appropriate processes will need to be put in place including approval processes as well as performance monitoring and reporting processes.

Date modified: