ARCHIVED - Evaluation of NRC Human Health Therapeutics

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

November 13, 2014

Acknowledgements

The Office of Audit and Evaluation would like to gratefully acknowledge the management and staff of the Human Health Therapeutics (HHT) Portfolio for their support and contribution to this evaluation. We would also like to acknowledge the contribution of knowledge and expertise from other branches of NRC, including Finance, Business Management Support, Communications, Knowledge Management, and Planning and Reporting Services. The evaluation team would also like to express its gratitude to members of the committee of Methodological and Subject-Matter Experts for their indispensable guidance and advice. Finally, the evaluation team would like to thank the expert reviewers, HHT clients and other stakeholders who provided rich information and insights in support of this evaluation.

Executive Summary

This report presents the results of the 2013‑14 evaluation of the National Research Council (NRC) Human Health Therapeutics (HHT) Portfolio. HHT mainly targets Canadian small and medium enterprises (SMEs) who develop therapeutics and vaccines to address unmet needs in human health. The Portfolio hosts three programs: Biologics and Subsequent Entry Biologics, Vaccines, and Therapeutics Beyond Brain Barriers (TBBB). The primary mode of interaction with these companies is the co-development of a product (therapeutic or vaccine) through R&D agreements. These co-development agreements are complemented by licensing and research services activities. Overall, these interactions are expected to de-risk the development of therapeutics and vaccines for HHT partners and accelerate the progress of candidates to clinical trials and market, thereby catalyzing the generation of wealth for industrial partners, increasing the availability of new or improved treatments and reducing health care costs.

This evaluation assessed the value-for-money of the HHT Portfolio, including relevance, performance, and resource utilization. It focused on the two-year period since the Portfolio was created (2012‑13 to 2013‑14). Where appropriate and available, the evaluation also considered earlier information relating to HHT's predecessor institutes (i.e., 2008‑09 to 2011‑12).

The key findings for each evaluation issue have been summarized and are presented below, along with the recommendations and management responses.

Key Findings: Relevance

Continued Need for the Program

HHT addresses a demonstrable need. The pharmaceutical/ biopharmaceutical industry is important to the Canadian economy and HHT activities help to address some of the needs of biopharmaceutical SMEs. HHT works with firms to accelerate the drug discovery process and de-risk the development of products, making the firms more attractive to funders.

External documentation shows that the pharmaceutical sector makes a significant contribution to the Canadian economy. The industry is also a major R&D performer, conducting around 10% of all Canadian business research and development. Still, in this area, Canada is not currently a major player on the global stage. The industry in Canada is mainly composed of early-stage research companies, many of which are years away from clinical trials and even further away from significant revenues. It is also worth noting that over the last few years, many of the leading Canadian companies have been acquired by foreign entities, further diluting Canada's presence in this area.

SMEs play a key role in driving industry innovation within the sector. In response to recent productivity and profitability challenges, MNEs are increasingly relying on outsourcing of R&D and licensing, particularly through partnerships with SMEs, to help them fill their pipelines while also externalizing risk. SMEs are also dependant on partnerships with MNEs for financial capital and commercialization expertise. HHT responds well to industry needs, as it works with firms to accelerate the drug discovery process and de-risk the products, making the firms more attractive to funders.

With respect to HHT-hosted programs, the evaluation also found that there is generally strong industry endorsement in favour of HHT's selected areas of focus. However, some external stakeholders raised concerns around the alignment of the Vaccines Program.

A number of external stakeholders identified access to certified Good Manufacturing Practice (cGMP) facilities as being an unmet need within the industry. NRC has made the decision that it will not be offering cGMP services. However, HHT is currently working with existing and potential contract manufacturing organizations (CMOs) to help strengthen biomanufacturing in Canada.

HHT fills a unique niche within the industry, given the breadth and depth of services and expertise provided, as well as its ability to be experimental and to innovate. For many clients, the only options for them to access comparable services would have been from organizations outside of Canada.

Role of Government

HHT activities and expected outcomes are consistent with federal roles and responsibilities.

HHT activities and outcomes are in line with the mandate of NRC, as stated in the NRC Act. In addition, almost all external interviewees and expert reviewers agreed that federal involvement, and specifically NRC's role, is appropriate in supporting biopharmaceutical SMEs, which they say will in turn strengthen the sector and the economy overall. A few interviewees and experts specifically noted that the external R&D model, where Canadian SMEs leverage existing NRC infrastructure and expertise, is a particularly appropriate role for NRC.

Alignment with Federal and NRC Priorities

The HHT Portfolio aligns well with federal priorities and NRC strategic outcomes, given its focus on industry-relevant R&D and its intended contribution to healthcare.

HHT's industry focus aligns well with the new NRC strategy and contributes to the achievement of NRC strategic outcomes. In the long-term, the work of HHT also seeks to contribute to alleviating growing healthcare pressures, which has been identified as a critical issue in NRC strategic documents.

At a federal level, the refocusing of NRC toward more industry-relevant R&D, with the aim of "helping the growth of innovative businesses in Canada" has been identified as a priority in both the 2012 and 2013 federal budgets. Moreover, the federal Science and Technology Strategy identifies "Health and related life sciences and technologies" as a priority area.

Key Findings: Performance

Positioning and Progress toward the Achievement of Expected Outcomes

HHT is well positioned to achieve expected program outcomes. Since its creation in 2012, HHT has made good progress in developing required competencies and has generally been successful in meeting its activity targets. By addressing certain limitations it currently faces, HHT has the potential, going forward, to achieve even greater levels of success.

HHT generally possesses the right technical and scientific competencies (people and research infrastructure) to achieve its expected outcomes. However, greater efforts are required to strengthen the leadership and business development skills of portfolio staff.

HHT has been successful in reaching its target clients, as demonstrated by a market penetration rate of 25% and the fact that the Portfolio has significantly exceeded its portfolio-level activity targets for 2013‑14. At the program level, the Biologics Program achieved $7.3M in revenues for 2013‑14, exceeding its program-level revenue target by 38%. The TBBB Program met its $1.9M revenue target, while the Vaccines Program generated a total of $3.08M in revenues, slightly below its $3.4M target.

Although the Vaccines Program was close to meeting its revenue targets for 2013‑14, the evaluation raised concerns about the Program's ability to secure important co-development research agreements with Canadian industry.

Recommendation 1: As part of the proposed "strategy refresh" for the Vaccines Program, HHT should determine the reasons why the program has had difficulty securing strong collaborative relationships with Canadian industry. The Portfolio should then identify strategies and establish an action plan to address the issue.

Management Response and Proposed Actions: Accepted. HHT will deliver a Vaccine Program ‘strategy refresh', including a root cause analysis for slower/lower uptake by Canadian industry and recommendations to address gaps and overcome difficulties. HHT will also incorporate key actions in the HHT Operational Plan for FY2016.

Despite its success, there is still an opportunity and a capacity for HHT to further expand and strengthen its current client base and, as a result, increase its annual revenues. In fact, the evaluation revealed that a large segment of the biopharmaceutical industry in Canada is not familiar with the services and competencies offered through HHT.

In addition, HHT's success with Zymeworks Inc. can serve as a model as there are opportunities to replicate this type of relationship with other companies. Internal interviewees stressed the importance of better showcasing HHT's success stories in order to demonstrate the potential impact NRC can have on SMEs and other clients.

Some internal and external interviewees also felt that IRAP Industrial Technology Advisors (ITAs) could, when appropriate, play a stronger role in making their clients aware of the services being offered through HHT. However, the evaluation found that ITAs are generally not very familiar with HHT's service offering, making it more difficult for them to identify opportunities where their clients could benefit from HHT services.

Recommendation 2: HHT should seek ways to increase portfolio visibility and awareness, for instance by: a) maximizing the use of communications channels (e.g., press releases) and b) intensifying recent efforts to engage with IRAP ITAs on a regular basis, involving senior management as appropriate.

Management Response and Proposed Actions: Accepted. HHT will increase portfolio visibility via increased support from NRC Communications Branch and closer interactions with IRAP.

HHT has exceeded its revenue targets, but with enhanced business development capacity, the potential exists to further expand its business. HHT's ability to conduct proactive business development activities, both to develop new client relationships and strengthen existing ones, is impeded by two factors: 1) the current capacity and workload of BMS staff assigned to HHT and 2) the limited business development skills of senior level researchers who are also expected to contribute to this activity.

Recommendation 3: HHT should identify and implement strategies to strengthen business development efforts.

Management Response and Proposed Actions: Accepted. In collaboration with BMS, HHT will implement key strategies to strengthen business development efforts.

Impacts on Clients and Industrial Partners

HHT has had a significant impact on the companies it has worked with, particularly with respect to the advancement and acceleration of product development and, as a result, increased company valuation and growth. In addition to this, HHT has also contributed to the strengthening of biomanufacturing capabilities in Canada.

Although the HHT Portfolio has only been in existence for a little over two years, the evaluation identified strong evidence of client impacts experienced as a result of their interactions with HHT. For most clients, HHT helped to advance and/or accelerate the development of their therapeutic candidates. In particular, HHT's strengths in the area of bioprocessing were frequently cited as having helped to increase the yield and purity of the material produced, often helping to overcome roadblocks the companies were facing in their product development. In other cases, product advancement occurred as a result of significant co-development between HHT and the client and/or through the licensing of NRC intellectual property or enabling technologies.

Of those clients who reported that work with HHT has contributed to the advancement or acceleration of new therapeutics, some succeeded in reaching critical milestones in the drug development process. For example, five companies reported that HHT played a significant role in getting their products to the clinical trial phase, or progressing through the various phases of clinical trials. For one company, work with HHT led to the commercialization of an animal vaccine to prevent the incidence of rabies in wildlife.

Projects conducted in collaboration with HHT have, in some cases, helped to make the clients' products more attractive to investors, thus translating into additional financing opportunities for the companies. While very few clients were able to attribute a specific amount to the work performed by HHT, most clients agreed that HHT's contribution played an important, and even critical, role in their ability to access investment capital in support of their growth. One client went so far as to directly attribute the survival of the company to the services provided by NRC.

In addition to the impacts noted above, the evaluation found that, through its work with PnuVax, HHT has also contributed to strengthening of Canada's biomanufacturing capabilities.

Key Findings: Resource Utilization

Economy and efficiency

HHT generally operates in an economical and efficient manner.

NRC corporate key performance indicators show a relatively high level of overhead efficiency and staff utilization. When compared to operations under the previous institute model, HHT also appears to be operating with a healthy level of productivity (as per an analysis of expenditure and revenue data).

HHT management and staff noted that the internal restructuring that has taken place over the past couple of years has led to significantly higher levels of efficiency and effectiveness. In line with the new matrix-based approach instituted at NRC, HHT has implemented its own matrix structure within the Portfolio. Expertise from across its three sites is now organized into four R&D departments, as well as an Operations department in which a project management office has been established. Staff is deployed cross-functionally to projects in its three programs. Interviewees indicated that the alignment of HHT's workforce and the streamlining of its activities have resulted in greater synergies within the Portfolio and have made it easier for management to assign resources according to project priorities. In fact, some interviewees noted that projects of the magnitude of those that are conducted for Zymeworks, for example, never could have succeeded under the previous institute model, where resources were much more siloed.

HHT clients generally reported a high level of client satisfaction and most clients felt that project costs were reasonable. Still, when asked about portfolio efficiency, over half of clients interviewed mentioned the "slow and bureaucratic" administrative processes that are associated with working with NRC. Slow contracting processes stood out as being a specific frustration among clients. In line with this, most internal interviewees and at least one client specifically noted that the BMS staff assigned to the Portfolio appear to be overwhelmed in terms of workload. From a process perspective, some internal interviewees noted that these could be further streamlined (for example, by streamlining the chain of approvals or by making better use of centralized Strategic Contract Advisors). A reduction in the level of effort required for administrative processes could also allow BMS staff to devote more time to proactive business development work.

Recommendation 4: BMS should explore opportunities to streamline contracting processes within the HHT Portfolio. As part of this, an attempt should be made to further leverage centralized BMS resources, where appropriate.

Management Response and Proposed Actions: Accepted. In consultation with HHT management, BMS will review the HHT contracting practices to ensure alignment with best corporate practices and deploy resources for improved support.

Acronyms and Abbreviations

ADC Antibody drug conjugate
BBB Blood brain barrier
BMS Business Management Support
BRI Biotechnology Research Institute
CDRD Centre for Drug Research and Development
cGMP Certified Good Manufacturing Practice
CGPA Canadian Generic Pharmaceutical Association
CMO Contract manufacturing organization
CNS Central nervous system
CRL Client Relationship Leader
CRO Contract research organization
FTE Full-time equivalent
GDP Gross domestic product
HHT Human Health Therapeutics
IBS Institute for Biological Sciences
IMB Institute for Marine Biosciences
IND Investigational new drug
IP Intellectual property
IPO Initial public offering
IRAP Industrial Research Assistance Program
ITA Industrial technology advisor
MNE Multinational enterprise
MSE Methodological and Subject-Matter Expert
NIH National Institutes of Health
NMR Nuclear magnetic resonance
NRC National Research Council
OAE Office of Audit and Evaluation
OGD Other government department
O&M Operations and maintenance
PBA Portfolio Business Advisor
PREVENT Pan-Provincial Vaccine Enterprise Inc.
R&D Research and development
RTO Research and technology organization
SEC Senior Executive Committee
SME Small and medium enterprise
TBBB Therapeutics Beyond Brain Barriers
VC Venture capital
VIDO Vaccine and Infectious Disease Organization

1. Introduction

This report presents the results of the 2013‑14 evaluation of the National Research Council (NRC) Human Health Therapeutics (HHT) Portfolio.

HHT mainly targets Canadian small and medium enterprises (SMEs) who develop therapeutics and vaccines to address unmet needs in human health. The primary mode of interaction with these companies is the co-development of a product (therapeutic or vaccine) through R&D agreements. These co-development agreements are complemented by licensing and research services activities. Overall, these interactions are expected to de-risk the development of therapeutics and vaccines for HHT partners and accelerate the progress of candidates to clinical trials and market, thereby catalyzing the generation of wealth for industrial partners, increasing the availability of new or improved treatments and reducing health care costs.

Following the evaluation overview presented below, Section 2 of this report provides a profile of the Portfolio. Sections 3, 4 and 5 present the evaluation study's findings organized by broad evaluation question (relevance, performance, resource utilization) along with associated recommendations. Section 6 presents a brief conclusion drawn from the evaluation, while Section 7 of the report lays out the management response to these recommendations and the actions that will be taken as a result.

1.1 Evaluation Overview

This evaluation assessed the value-for-money of the HHT Portfolio, including relevance, performance, and resource utilization. It focused on the two-year period since the Portfolio was created (2012‑13 to 2013‑14). Where appropriate and available, the evaluation also considered earlier information relating to HHT's predecessor institutes (i.e., 2008‑09 to 2011‑12). For example, given the long timelines associated with research and development (R&D) in life sciences, earlier data was consulted in the assessment of client impacts. As well, financial and resource data from HHT's predecessor institutes were used as a comparator for the assessment of HHT efficiency.

Evaluations are conducted at the portfolio level. The relevance and performance of HHT's three hosted programs are examined insofar as portfolio outcomes are achieved as a result of program activities. The portfolio-hosted programs will undergo a more in-depth assessment as part of planned mid-term program reviews. It is expected that information drawn from this evaluation could contribute to the mid-term reviews.

This evaluation was led by an independent evaluation team from the NRC Office of Audit and Evaluation (OAE). The work of the evaluation team was supported by Methodological and Subject-Matter Experts (MSEs) who provided advice related to the evaluation framework, approach, interpretation of findings and recommendations. These experts did not act in a decision-making fashion, but rather played an advisory role in ensuring a high-quality and useful evaluation product.

1.1.1 Evaluation Rationale

The work was carried out in accordance with NRC's approved evaluation plan. The decision to evaluate HHT at this time is largely based on the maturity of its hosted programs. Relative to other NRC portfolios, HHT is more advanced in terms of program implementation, thus allowing for a better assessment of program outputs and early outcomes. In addition, prior to this evaluation, neither HHT nor its predecessor institutes had been the subject of an evaluation within the past ten years. As per Treasury Board's Policy on Evaluation (2009), all programs benefiting from any level of direct program spending must be evaluated every five years.

1.1.2 Evaluation Design and Methodology

The approach and level of effort for the evaluation was commensurate with the low level of risk facing the Portfolio, as identified during an assessment conducted as part of the planning phase. The specific evaluation questions, outlined at the start of each section, are based on consultations with portfolio management and a review of key documents during the planning stage. The issues identified are also aligned to the requirements of the 2009 Treasury Board Policy on Evaluation. The complete evaluation matrix, which identifies the lines of evidence employed to respond to each question, was developed by the evaluation team in consultation with Methodological and Subject-Matter Experts and is presented in Appendix A: Evaluation Matrix.

To enhance the reliability and validity of the information and data collected, the methodology for this evaluation includes multiple lines of evidence and complementary research methods. The specific methods used in the study include:

  • Internal and external document review
  • Administrative and performance data review
  • Expert consultations
    • One-on-one interviews with Canadian experts in the field of biopharmaceutical R&D
    • A total of nine experts were interviewed (three for each of HHT's three hosted programs).
    • Experts were provided with portfolio and program documentation to review in advance of the interview.
  • Semi-structured interviews
    • 17 internal interviewees (including staff from HHT, NRC Business Management Support and NRC's Industrial Research Assistance Program, or IRAP)
    • 29 external interviewees (including client organizations, prospective clients, industry associations and other stakeholder organizations)
  • Case studies
    • A series of 10 case studies, based upon reviews of project and client documentation, as well as both internal and external (client) interviews Footnote 1

For interviews, the following scale is used in the text of the report to indicate the relative weight of the responses for each of the respondent groups.

Table 1: Scale used when reporting on views and opinions of respondents
All/almost all 90% or more
Many/most At least 50% but less than 90%
Some/several At least 25% but less than 50%
A few At least two respondents but less than 25%

A more detailed description of the study methodology, limitations and challenges is provided in Appendix C: Methodology.

2. Profile of the HHT Portfolio

The HHT Portfolio was created on April 1, 2012, through a merger of a subset of resources from three former NRC institutes: the Biotechnology Research Institute (BRI), the Institute for Biological Sciences (IBS) and the Institute for Marine Biosciences (IMB).

The mission of HHT is to provide innovative technologies and expertise to Canadian SMEs, enabling them to develop new treatments to prevent or treat diseases cost effectively – contributing to a healthier Canada and generating prosperity opportunities for Canadian companies. HHT's vision is to be recognized as a major contributor to the prosperity of Canada's biopharmaceutical sector. Through a strong track record of industry collaborations, the Portfolio seeks to add value to co-developed biologics, vaccines and biomarkers, de-risk their development and accelerate their progress to market. Footnote 2

To fulfill its mission, HHT works with approximately 100 Canadian and international clients in the biopharmaceutical and biotechnology industries. While HHT's focus is on SME clients, these clients also include large multinational enterprises (MNEs), universities and other research institutes, government departments and non-governmental organizations. HHT mostly works with clients to develop biologics and vaccines in the areas of cancer, infectious diseases, neurological diseases, inflammatory diseases and autoimmune diseases. The other main areas of HHT's interactions with clients include: working with Canadian biomanufacturers (who produce clinical trial and commercial grade biologics and vaccines); collaborating with leading research organizations, centres of excellence, commercialization initiatives and universities on strategic projects; and licencing its proprietary expression systems to about 100 biopharmaceutical firms around the world, including large pharmaceutical firms. A full logic model, developed as part of evaluation planning, details the activities, outputs and intended outcomes of the HHT Portfolio (Appendix B: HHT Logic Model).

2.1 Support to NRC Programs

At NRC, portfolios are the business units responsible for managing people and facilities. These resources are then deployed to various programs that have been approved by NRC's Senior Executive Committee. At this time, the large majority of HHT resources are supporting HHT-hosted programs. For example, in 2013‑14, approximately 77% of HHT's labour workforce supported HHT-hosted programs, whereas approximately 1% worked on programs for other NRC portfolios. The remainder of labour expenditures were devoted to portfolio activities that fell outside the scope of approved programs.

HHT currently hosts three NRC programs that have been approved for implementation:

  • Biologics and Subsequent Entry Biologics (approved for implementation in May 2012): This program aims to co-develop novel biologics, subsequent entry biologics, and biobetters in partnership with industry while also generating revenues from contractual activities. It focuses on the prioritization of therapeutic product candidates, bioprocessing, and performance characterization.
  • Vaccines Program (approved for implementation in July 2012): This program works with industry to co-develop a pipeline of novel and improved vaccines, offers contract services to a broad clientele, and researches novel approaches in priority disease areas (so as to ensure that the program can meet the future needs of its partners and clients).
  • Therapeutics Beyond Brain Barriers (TBBB) (approved for implementation in September 2013): This program addresses the need to further carriers for biologics across the blood brain barrier (BBB), an innovation which HHT presently leads internationally. The goal of the program is to collaborate with industrial partners to reach proof of concept in humans and, longer-term, to bring carrier-conjugate biologics to market for poorly served central nervous system (CNS) indications. The deployment of NRC's BBB carriers in co-development projects is expected to create a global competitive advantage for Canadian SMEs developing therapeutics with targets within the central nervous system.

2.2 Organizational Structure

The HHT General Manager reports to the Vice-President, NRC Life Sciences, and has overall managerial accountability for the Portfolio. The Portfolio is organized into four R&D departments, each headed by an R&D director, as well as an Operations department responsible for quality control, process improvement, project management and coordination, and portfolio administrative support. Each department comprises a number of teams that are deployed in a matrix manner to the Portfolio's three programs and the projects within those programs.

2.3 Portfolio Resources

2.3.1 Human Resources

HHT has a staff of approximately 300 full-time equivalents (FTEs), located across two sites in Ottawa (50%), one site in Montreal (47%) and another in Halifax (3%).

2.3.2 Financial Resources

As the HHT Portfolio has only existed since April 1, 2012, historic financial data before that time are not available. As well, with 2012‑13 being a year of transition, financial figures for that year are not considered to be typical.

For 2013‑14, HHT reported total expenditures (direct and indirect) of approximately $49.1M. Approximately 25% of these expenditures were offset by earned revenues ($12.3M), which were derived mainly from strategic research projects with industry (see Table 2, on the following page).

Table 2: HHT financial data ($)
2012‑13 2013‑14 2014‑15
(Budgeted)
Earned Revenues
Technical Services
16,298 31,981 -
Strategic Research
9,926,352 12,253,281 12,010,062
Other Revenue Footnote 3
10,340 46,615 -
Total earned revenues 9,952,991 12,331,876 12,010,062
NRC Program Investment Footnote 4
20,081,751 25,941,782
Total income 9,952,991 32,413,627 37,951,844
Total Direct Expenditures 26,393,406 32,529,884 36,451,843
Total Indirect Expenditures 20,491,668 16,603,954 13,884,283
Gains/Losses and other adjustments
−6 −20,723
Net income or loss −36,932,089 −16,740,934 −12,384,282
Total Capital Investment 2,847,937 5,250,015 3,428,322

Source: NRC Finance Branch (Statements of Operations)

2.3.3 Physical Resources

The Portfolio currently holds $53M worth of physical assets, including facilities for scale-up, animal resources, nuclear magnetic resonance (NMR) and mass spectroscopy among others.

2.4 Profile of HHT Clients

As shown in Figure 1 (on the following page), almost two-thirds of HHT's revenue in 2013‑14 came from Canadian industry sources, mainly SMEs developing protein therapeutics and vaccines. HHT also interacts significantly with foreign industry, mainly MNEs, and does so for three reasons: (1) the high cost of clinical trials and product commercialization usually requires the involvement of large players, who are more experienced and better able to afford the long-term costs; (2) projects carried out with these partners allow the Portfolio to stay abreast of international developments in the field and translate what is learned back to the benefit of Canadian companies; (3) complementary revenue generation from interactions with these clients ensures Portfolio sustainability. Footnote 5 Finally, HHT undertakes co-development projects in areas of common interest with some other government departments (OGDs), and with other players, such as centres of excellence, research institutes and health authorities.

Figure 1: HHT sources of revenue (2013‑14) Footnote 6

Source: NRC Finance Branch

3. Relevance

The relevance of the HHT Portfolio was examined through three evaluation issues: the continued need for the program (section 3.1); its alignment with the priorities of the federal government (section 3.2); and its alignment with the roles and responsibilities of the federal government and of NRC (section 3.3).

3.1 Continued Need for Portfolio Capabilities

Evaluation Question 1

Does HHT address a demonstrable need?

Assessment

Yes. The pharmaceutical/biopharmaceutical industry is important to the Canadian economy and HHT activities help to address some of the needs of biopharmaceutical SMEs. HHT works with firms to accelerate the drug discovery process and de-risk the development of products, making the firms more attractive to funders.

3.1.1 Size and Importance of the Canadian Pharmaceutical/Biopharmaceutical Sector

The biopharmaceutical sector is a specialized subset of the pharmaceutical industry; the biopharmaceutical industry develops therapeutic and vaccine products using biotechnology. Biopharmaceuticals currently account for approximately 16% of global pharmaceutical sales. Footnote 7

The biopharmaceutical market in Canada is characterized by approximately 200 start-ups and SMEs, about a dozen multinational pharmaceutical companies with an R&D and/or manufacturing presence in Canada, and a similar number of contract service providers (contract research organizations and contract manufacturing organizations). Major biopharmaceutical clusters are located in Montreal, Toronto and Vancouver.

Key Finding 1: External documentation demonstrates the importance of the sector to the Canadian economy.

The pharmaceutical sector Footnote 8 makes a significant contribution to the Canadian economy. According to Statistics Canada, in 2012, Canadian drug manufacturers exported $10.5 billion worth of goods, contributed almost $4.6 billion to Canada's gross domestic product (GDP), and employed approximately 27,000 Canadians, many of whom are well-paid and highly skilled. Footnote 9 A 2012 study by KPMG reports that, on a per FTE basis, the industry's direct contribution to Canada's GDP surpasses all other industrial sectors except mining oil and gas, utilities and financial services. Footnote 10

The industry is also a major R&D performer, conducting around 10% of all Canadian business research and development. Pharmaceuticals/biotechnology is the third largest industry for R&D spending in Canada (after the aerospace and communications/telecom equipment sectors) and has the highest average level of research intensity (R&D as a percentage of revenue). Twenty-two pharmaceutical and biotechnology companies are listed in Research Infosources' Top 100 Corporate R&D Spenders 2013 in Canada. Footnote 11 Still, data from the Patented Medicine Prices Review Board shows a declining trend in the ratio of R&D expenditures to sales of Canadian pharmaceutical patentees. From 2001 to 2011, this ratio fell from 9.9% to 5.6%, the lowest ratio for at least two decades. Footnote 12

Figure 2: Total R&D expenditures and R&D-to-sales ratios of Canadian pharmaceutical patentees

Source: Patented Medicine Prices Review Board

While the Canadian pharmaceutical/biopharmaceutical sector plays an important role within the Canadian economy, it is not currently a major player on the global stage. Industry Canada reports that pharmaceutical sales in Canada have a 2.5% share of the global market, making Canada the 8th largest world market. Footnote 13 In addition, a report by the Conference Board of Canada ranks Canada 14th out of 16 countries in terms of export market share for pharmaceuticals. Footnote 14 The report notes that the Canadian pharmaceutical/biopharmaceutical industry is still mainly composed of early-stage research companies, many of which are years away from clinical trials and even further away from significant revenues. It is also worth noting that over the last few years, many of the leading Canadian companies have been acquired by foreign entities, further diluting Canada's presence in this area. Footnote 15

Key Finding 2: SMEs play a key role within the sector.

Globally, it is widely acknowledged that R&D productivity within the pharmaceutical industry, and particularly that of MNEs, has seen a declining trend for a number of years. The industry's profitability and growth prospects are also under pressure from a range of factors including major losses of revenue owing to patent expirations, increasingly cost-constrained healthcare systems and more demanding regulatory requirements.

In response to these challenges, MNEs are increasingly relying on outsourcing of R&D and licensing, particularly through partnerships with SMEs, to help them fill their pipelines while also externalizing risk. This new model offers opportunities for the Canadian industry, as MNEs take on the role of investors and SMEs play an increasingly significant role in driving industry innovation.

3.1.2 Alignment with the Needs of Canadian Industry

Key Finding 3: HHT responds well to industry needs, as it works with firms to accelerate the drug discovery process and de-risk the products, making the firms more attractive to funders.

Above all else, Canadian biopharmaceutical SMEs require access to funding, which will enable them to move their products from the early stages of drug development through clinical trials. This need was emphasized repeatedly by stakeholders consulted as part of the evaluation and in the documentation reviewed.

Within the biopharmaceutical sector, barriers to entry are high as developing a new drug is costly and time-consuming. On average, it can take between 10 and 15 years, and require more than $1 billion to discover and develop a new medicine or vaccine. Footnote 16 In Canada, as in the United States and in Europe, financing for biopharmaceutical SMEs has been limited in recent years. Initial public offerings (IPOs) are very uncommon and venture capital (VC) funding for biotechnology has been declining since 2008. Footnote 17 More and more, Canadian biopharmaceutical SMEs are depending on alliances with multinationals for financial capital, as well as for expertise in clinical research, regulatory affairs, commercialization and international distribution. A 2013 survey of industry leaders from life sciences organizations across Canada, conducted by PwC for BIOTECanada, revealed that access to capital is the most significant priority for respondents, followed closely by their ability to attract a licensing or strategic partner. Footnote 18

With limited funding available, SMEs can face challenges overcoming the industry's initial "valley of death". This term has been defined by the National Institutes of Health (NIH) as the "period of transition when a developing technology is seen as promising, but it is too new to validate its commercial potential and unable to attract the necessary funding for its continued development".  Footnote 19 As part of the evaluation, client interviewees were asked what needs their companies were looking to address through their interactions with HHT. Many described specific roadblocks they were facing at the time, noting that HHT expertise was sought to provide a solution, which would help them to advance their product to the next phase of the drug development process. By accelerating the drug discovery process and de-risking the development of products, HHT helps to make the firms more attractive to funders.

Most clients and expert reviewers agreed that HHT services are aligned with industry needs and one VC company representative confirmed that the kind of progress HHT aims to achieve when collaborating with clients is attractive to funders.

Key Finding 4: HHT is generally focused on areas that are likely to have a strong impact.

As part of the creation of the Portfolio and the development of program plans, HHT sought to focus its efforts on areas that are more likely to yield a strong impact for Canada. Areas thought not to yield strong impacts were abandoned. The evaluation found that there is strong industry endorsement with respect to HHT's selected areas of focus.

Biologics

"Undeniably, the future lies in biologics. Within Canada and even on an international scale, NRC is ahead of the game in the area of monoclonal antibodies […] Focusing on this niche was a good decision."

Expert Reviewer

HHT's focus on biologics was strongly supported by most of the stakeholders interviewed and in the documentation reviewed. Two of the three expert reviewers in this area commented that HHT's focus on biologics, working with Canadian SMEs in the pre-clinical phase, is likely to lead to considerable or strong impacts. The third felt that this area of focus had the potential to lead to positive, yet moderate, impacts, given the size and structure of the industry in Canada. This expert also commented that the next few years could see changes in how the regulatory environment views biosimilars Footnote 20 and that HHT is well positioned, given the potential for future growth in this area.

Globally, the pharmaceutical industry, especially "Big Pharma", has dramatically shifted its R&D focus from its historical concentration on small molecule drugs to include a rapidly increasing number of biotechnology products. As noted by one MNE representative consulted as part of the evaluation, "There has been a rapid shift away from small molecules to biologics. In our case, we've had to make changes on a dime." Tufts Center for the Study of Drug Development found that biotechnology products, which accounted for only 7% of revenue generated by the 10 top selling pharmaceutical-biotech products worldwide in 2001, accounted for 71% of the revenue generated by the 10 top selling products in 2012. Footnote 21 An article in the OECD Journal reports that biopharmaceuticals offer greater therapeutic value than other pharmaceuticals and that the large number of experimental biopharmaceuticals, offering new modes of action, suggests that the future stream of biopharmaceuticals should provide substantial therapeutic advantages over existing therapies. Footnote 22

In Canada, the biologics market has grown from approximately $1.7 billion in 2005 to $3.5 billion in 2011, reflecting a compound annual growth rate of 12.8% during this time period. According to the Canadian Generic Pharmaceutical Association (CGPA), the biologic drugs market in Canada is expected to reach $9.1 billion by 2020, with an approximate 20% market share. Footnote 23

Figure 3: Biologics share of the Canadian pharmaceutical market

Source: GlobalData

Therapeutics Beyond Blood Brain Barriers

Global sales for central nervous system (CNS) drugs are estimated at around US$65 billion. Footnote 24 It is the fastest growing segment of the pharmaceutical market, yet treatments for most disorders are considered to be either suboptimal or not available. Hence, NRC's work in the development of blood brain carriers has the potential to open up new markets in this area.

External stakeholders generally felt that there is a strong need to find effective ways of getting therapeutics across the blood brain barrier in order to better treat CNS diseases and supported HHT's efforts in this area.

"The raison d'être of TBBB is spot on. [NRC] picked a winning game but we will see if you can perform."

Expert Reviewer

Two of the three expert reviewers for the TBBB Program stated that the Program has the potential to achieve strong impacts for the industry, but that its success will ultimately depend on program execution. The third expert reviewer felt that the Program would likely generate small to moderate impacts. This reviewer explained that the Program, as outlined in the business plan, is unrealistically ambitious and that there are significant risks in being able to deliver on planned outcomes (for instance, bringing 14 BBB-enabled candidates to investigational new drug, or IND, submission by 2019). In order to mitigate the risks involved, he stressed the importance of strengthening relationships with academic and other research organizations who are also working in this area. He also felt that the Program's focus on biologics is limiting, a sentiment that was also echoed by a second expert reviewer.

As part of the TBBB Program, HHT is currently working with a handful of MNE partners, coupling their CNS therapeutics with NRC carriers to reach proof of mechanism, thereby validating the effectiveness of the NRC carriers. Representatives from two of these MNEs were interviewed as part of the evaluation. Both were very complimentary of NRC's expertise, considering NRC to be a leader in this area, and were optimistic about the results that could potentially be achieved.

Vaccines

Key Finding 5: External stakeholders raised some concerns around the alignment of the Vaccines Program.

"NRC has been working in traditional areas for a number of years. While I feel that the scientists are strong and talented, their work [in the area of vaccines] could be more applied, with a stronger link to industry. There may be a need to review what they have and maybe modify the program so that they are more relevant to vaccine development."

Client interviewee

Some specific concerns were raised by external stakeholders with respect to alignment of the Vaccines Program with industry needs. Two multinational companies working in this area were interviewed. They were mildly interested in the services being offered by the Portfolio and both felt that HHT is somewhat aligned with the needs of their organization. Company representatives commented that while they feel that the Program is "on the right track" and that there are some areas of interest to them (examples given include biomarkers, bio-defense vaccines, predicting immune responses and alternate vaccine delivery mechanisms), there are other areas where the existing technical capabilities of the companies surpass those of NRC. In addition, one interviewee noted that the work of HHT scientists in the area of vaccines would be of greater benefit to industry if it was more applied.

Expert reviewers also raised concerns around alignment of the Vaccines Program. Whereas experts for the Biologics Program had a very positive view of that program's alignment with industry needs, the views of expert reviewers for the Vaccines Program were more mixed. Two of the three experts for Vaccines noted that the services and competencies available through the Program are meeting industry needs to a slight or moderate extent, while the third reported that the Program is addressing industry needs to a considerable extent. Issues of concern appeared to stem mostly from program execution, rather than program relevance. For instance, experts noted challenges engaging with industry and the perceived academic culture of NRC researchers. One expert also questioned whether the services being offered were the right ones, given the needs of the industry. Both of these experts also noted that there are many players in Canada with strong expertise in the area of vaccines and NRC should attempt to be more flexible in supporting and collaborating with these organizations. It should be noted that the Vaccines Program has demonstrated strong efforts to network and engage with other key players in this area. This includes, for example, its work in developing the Research, Development and Innovation Vaccine Federal Action Plan, as well as collaborative research agreements with federal, academic and not-for-profit partners.

Key Finding 6: Access to certified Good Manufacturing Practice (cGMP) facilities is currently viewed as an unmet need.

A number of external stakeholders noted that one of the major needs facing Canadian biopharmaceutical SMEs with respect to R&D is the ability to produce clinical grade material (material that is produced according to the standards required for clinical trials). The need itself was raised by many of the expert reviewers, by a few current clients and by one prospective client, as well as by one of the three IRAP Industrial technology advisors (ITAs) that were interviewed. These stakeholders noted that cGMP manufacturing services in Canada are limited and that, in many cases, companies have no choice but to procure these services from international service providers. Some stakeholders also added that addressing this need is important if NRC is to achieve the greatest possible impact.

NRC has made the decision that it will not be offering cGMP services, as the set-up and operation of such a facility would be very resource-intensive and the organization feels that it is an area that is better served by the private sector. That said, HHT does recognize the need within the sector and is currently working with existing and potential contract manufacturing organizations (CMOs) to help strengthen biomanufacturing in Canada. The Portfolio also proposes to act as a liaison, developing what it calls "cGMP-ready processes" and enabling its Canadian SME partners to successfully transfer their product manufacturing to a Canadian CMO.

3.1.3 Ability to Meet Client Needs in the Absence of the Portfolio

Key Finding 7: HHT fills a unique niche within the industry.

"In Canada, there really is nobody else who could provide as comprehensive of services. There were some overseas options but we elected to keep the work at home."

Client interviewee

External stakeholders (clients and expert reviewers) identified certain organizations that can provide some of the same services as those provided by HHT. Academic institutions and contract research organizations (CROs) were noted most often, although in almost all cases, clients noted that HHT provides certain advantages which make working with them more attractive. Specifically, the breadth and depth of services and expertise provided by HHT was frequently noted by clients and by expert reviewers. A few clients also reported that CROs generally lack the ability to be experimental, to solve problems and to innovate. Ultimately, several clients indicated that, in the absence of HHT, the only options for them to access comparable services would have been from organizations outside of Canada.

Although several expert reviewers noted that HHT has a unique service offering, particularly in the area of biologics, some identified Canadian not-for-profit organizations whose mandates may overlap slightly with that of NRC. These included the Centre for Drug Research and Development (CDRD) for the development of therapeutics more generally, as well as the Vaccine and Infectious Disease Organization (VIDO) and the Pan-Provincial Vaccine Enterprise Inc. (PREVENT) for vaccine development. However, the focus of these organizations is generally different from that of HHT. For example, CDRD is focused on small molecules, and conducts virtually no research in the areas of biologics and vaccines. PREVENT funds pre-clinical vaccine development, but does not perform any intramural research. In fact, the Vaccines Program has collaborated with PREVENT for two projects in 2013‑14, and has also been exploring opportunities for partnerships with VIDO. The expert reviewers commented that partnering between these organizations is important to help ensure that they remain complementary, rather than duplicative.

3.2 Role of Government

Evaluation Question 2

Are HHT activities and expected outcomes consistent with federal roles and responsibilities?

Assessment

Yes. HHT activities and expected outcomes are consistent with federal roles and responsibilities.

Key Finding 8: HHT's activities and outcomes are in line with the mandate of NRC, as stated in the NRC Act.

NRC's specific role in support of the biopharmaceutical industry stems from subsection 5 (1) (c) of the NRC Act which states that the Council may "undertake, assist or promote scientific and industrial research". Footnote 25 Under the Act, NRC is charged with the direction or supervision of research undertaken by or for industrial firms or other organizations [subsection 5 (1) (d)] and with carrying out experimental and developmental work with respect to the above and making the resulting processes, methods and products available for the benefit of manufacturing and other scientific purposes [subsection 5 (1) (k)].

Key Finding 9: External stakeholders were generally supportive of NRC's role within the biopharmaceutical industry.

Almost all external interviewees and expert reviewers agreed that federal involvement, and specifically NRC's role, is appropriate in supporting biopharmaceutical SMEs, which they say will in turn strengthen the sector and the economy overall. Many interviewees noted that federal government support is essential if Canada is to remain competitive with other countries whose governments are investing heavily to support their domestic sectors. Some interviewees referenced the National Institutes of Health (NIH) in the United States, which have spent more than $300 billion over the last decade ($30.9 billion in 2012 alone) on life sciences research, Footnote 26 and provide support to pharmaceutical SMEs (e.g. through grants, partnerships, and preferential license agreements). Footnote 27 In addition, a few interviewees and experts noted that the external R&D model, where Canadian SMEs leverage existing NRC infrastructure and expertise, is a particularly appropriate role for NRC. In these cases, the fact that SMEs do not need to maintain their own wet lab operations is very beneficial to SMEs and is particularly attractive to VC funders.

"Venture capitalists have been pushing companies to adopt a virtual model. They are not interested in investing in infrastructure, given the high cost and long-term nature [of these types of investments]."

Expert Reviewer

A small number of interviewees cautioned that some of the fee-for-service work conducted by HHT could potentially overlap, or compete, with CRO services. However, HHT management has expressed on multiple occasions and in program documentation that the Portfolio strives to offer innovative and differentiated R&D support, so that it is not in direct competition with CROs. In fact, some internal interviewees noted that when asked to perform technical services that would be more appropriate for CROs, HHT will frequently point clients in that direction.

3.3 Alignment with Government and NRC Priorities

Evaluation Question 3

Do the objectives and activities of HHT align with federal priorities and NRC strategic outcomes?

Assessment

Yes. The HHT Portfolio aligns well with federal priorities and NRC strategic outcomes, given its focus on industry-relevant R&D and its intended contribution to healthcare.

HHT's industry focus aligns well with the new NRC strategy and contributes to the achievement of NRC strategic outcomes. As per the logic model presented in Appendix B, HHT strives to contribute to the following two ultimate outcomes:

  • Economic growth and prosperity of the Canadian biotherapeutics industry; and
  • Improved and more affordable vaccines and biologics for the marketplace.

The first of these two portfolio-level outcomes aligns very well with NRC's first strategic outcome, that is, "Canadian businesses prosper from innovative technologies". Footnote 28 The second portfolio-level outcome demonstrates how, in the long-term, HHT seeks to contribute to alleviating growing healthcare pressures, which has been identified as a critical issue in NRC strategic documents.

It is also worth noting that, at a corporate level, mechanisms have been put in place to ensure that portfolio activities align with NRC priorities. The most significant of these mechanisms is a business planning process whereby the work of HHT, like that of all other NRC portfolios, must fall within the scope of NRC programs. Programs are approved by NRC's Senior Executive Committee (SEC) following a rigorous stage gate process. The fact that all three HHT-hosted programs have been approved for implementation by NRC's SEC demonstrates their alignment with NRC's new strategic direction and stated goals.

At a federal level, the refocusing of NRC toward more industry-relevant R&D, with the aim of "helping the growth of innovative businesses in Canada" has been identified as a priority in both the 2012 and 2013 federal budgets. Footnote 29 Moreover, the federal Science and Technology Strategy identifies "Health and related life sciences and technologies" as a priority area. Footnote 30

4. Performance

The evaluation assessed the performance of HHT in terms of a) the extent to which the Portfolio has positioned itself to achieve expected program outcomes (section 4.1); and b) the impact it has had on clients (section 4.2).

4.1 Positioning and Progress toward the Achievement of Expected Outcomes

Evaluation Question 4

To what extent has HHT positioned itself to achieve expected program outcomes?

Assessment

HHT is well positioned to achieve expected program outcomes. Since its creation in 2012, HHT has made good progress in developing required competencies and has generally been successful in meeting its activity targets. By addressing certain limitations it currently faces, HHT has the potential, going forward, to achieve even greater levels of success.

This section describes the extent to which HHT has positioned itself to achieve its expected outcomes by examining the development of its core competencies and its level of engagement with industry clients. It concludes with a discussion around the key barriers to client reach.

4.1.1 Development of Competencies

Key Finding 10: HHT possesses the right technical and scientific competencies (people and research infrastructure) to achieve its expected outcomes. However, greater efforts are required to strengthen the leadership and business development skills of portfolio staff.

As described in Section 2, portfolios are the business units responsible for managing people and facilities. These resources are then deployed to the approved NRC programs. Generally, internal interviewees believe that HHT has established the research competencies required to successfully deliver its three approved programs and, in turn, achieve its expected outcomes. Although a limited number of positions remain to be filled (i.e., 14 FTEs in specific fields of expertise) and HHT has identified $2.75M in major capital investments required to upgrade its facilities, the Portfolio appears to have taken the proper actions to address these gaps (i.e., staffing actions initiated or planned, and major capital requests submitted through the NRC investment planning process).

The quality of HHT's research competencies was confirmed by the expert reviewers consulted as part of the evaluation. In this regard, seven experts out of the nine consulted considered HHT's research competencies to be "good" or "excellent". Footnote 31 The case study findings and client interviews also revealed that many HHT clients strongly appreciate the quality and breadth of expertise offered by the Portfolio.

While HHT appears to have the right scientific and technical competencies in place, portfolio management interviewed as part of the evaluation highlighted the need for staff to develop stronger leadership and business development skills. Specifically, these interviewees referred to staff's ability to generate business and to take initiative when an opportunity or issue arises. Interviewees indicated that one of the major challenges in recent years has been the shift from a more academic culture to one that is more industry focused, in line with NRC's new strategy as a research and technology organization (RTO). Some of the actions taken so far to promote this change of culture have included: recruitment of new staff with a strong industry background; coaching of team leaders by NRC Business Management Support (BMS) staff to help strengthen business development skills; participation of team leaders in training offered corporately by NRC; and, organization of an annual team leader retreat. Although HHT management interviewees felt that a great deal of progress has been made in this area, they were all of the opinion that further efforts are still required.

4.1.2 Client Reach

Key Finding 11: HHT has been successful in reaching its target clients, as demonstrated by a market penetration rate of 25% and the fact that the Portfolio has significantly exceeded its client agreement and revenue targets for 2013‑14.

Overall, based on HHT performance data and on the opinions of expert reviewers, the evaluation found that HHT has performed well from a client reach perspective, especially when considering the significant transformation the organization experienced over the past two and a half years.

At the portfolio level, since 2012, HHT has provided services to, or collaborated with, a total of 51 unique Canadian private sector clients out of an estimated population of 206 Canadian biopharmaceutical companies, Footnote 32 thus indicating a market penetration rate of 25%. The data also shows that for 2013‑14, HHT has significantly exceeded its portfolio-level activity targets. Across its three programs, HHT signed a total of 123 client agreements, surpassing its annual target of 100 agreements, and generated total revenues of $12.3M, a figure that is 16% higher than the Portfolio's $10.6M target for the year. Footnote 33

Figure 4: Revenue by program (2013‑14)

Source: NRC Finance Branch

At the program level, the Biologics Program achieved $7.3M in revenues for 2013‑14, exceeding its program-level revenue target by 38% (see Figure 4, above). The Program's exceptional performance with respect to revenues is largely attributable to its relationship with Zymeworks Inc., which accounted for a significant proportion of program revenues. The evaluation also found that the strong client base of the Biologics Program, which is a critical factor in its ability to generate revenues, stems, to a large extent, from previous client relationships established by the former NRC Biotechnology Research Institute in Montreal.

The TBBB Program met its $1.9M revenue target, while the Vaccines Program generated a total of $3.08M in revenues, slightly below its $3.4M target.

Key Finding 12: While the Vaccines Program was close to meeting its revenue targets for 2013‑14, the evaluation raised concerns about the Program's ability to secure key collaborations with Canadian industry.

The Vaccines Program came close to meeting its revenue targets for 2013‑14 and has been successful at providing research services to Canadian clients, particularly in areas of bioprocessing and analytics. However, a few key internal interviewees and some expert reviewers raised concerns about the Program's ability to establish strong co-development research agreements with Canadian industry. While the Vaccines Program has engaged in a few strategic partnerships with Canadian SMEs to accelerate the development of their vaccine candidates, some key collaborations (both with SMEs and MNEs with a strong footprint in Canada) that had originally been anticipated by the Program have not yet materialized. It is also worth noting that the Program draws a significant proportion of its revenues from Canadian non-industry and foreign sources. These sources represented 30% and 22% of Program revenues, respectively. Revenues from SMEs make up the remaining 48%.

Documentation provided by HHT as part of the expert consultations also reveals that the Vaccines Program has undertaken several internal research projects with a strong focus on the development of novel vaccine candidates or enabling technologies, which could potentially be leveraged to the benefit of Canadian industry partners. However, many of these projects have yet to be licensed or otherwise exploited by Canadian industry players.

It became apparent through the evaluation that the Vaccines Program has been devoting significant efforts to network and engage with Canadian industry. However, it was difficult to determine why significant research collaborations stemming from these efforts have been slow to materialize. In its Operational Plan for 2014‑15, HHT is proposing a "strategy refresh" for the Vaccines Program. This exercise provides an ideal opportunity for HHT to further examine these challenges and establish appropriate measures to address them.

Recommendation 1: As part of the proposed "strategy refresh" for the Vaccines Program, HHT should determine the reasons why the program has had difficulty securing strong collaborative relationships with Canadian industry. The Portfolio should then identify strategies and establish an action plan to address the issue.

4.1.3 Industry Reach – Challenges and Opportunities

Discussions with HHT management revealed that, while HHT is currently exceeding its established targets, there is still an opportunity and a capacity for the Portfolio to further expand and strengthen its current client base and, as a result, increase its annual revenues.

Key Finding 13: The evaluation revealed that a large segment of the biopharmaceutical industry in Canada is not familiar with the services and competencies offered through the HHT Portfolio.

Many experts and interviewees (both internal and external) raised concerns about the fact that a large segment of the Canadian biopharmaceutical industry is not familiar with HHT's services. This is especially the case with changes that have taken place since the NRC transformation. For instance, representatives from the two life sciences industry associations, as well as the three prospective clients interviewed as part of the evaluation, had limited knowledge of the Portfolio. It is worth noting that even some clients commented that they are not fully aware of HHT's service offerings. In this regard, many evaluation participants strongly emphasized the need to raise the profile and degree of visibility of HHT within the biopharmaceutical industry in order to further expand its reach.

HHT management recognized that portfolio visibility could be improved. Some internal interviewees noted that while the first two years since the creation of the Portfolio have been rather inwardly focused, with the aim of properly establishing organizational structures and processes, HHT can now place a greater focus on outreach efforts. Internal interviewees provided some examples of recent initiatives that have been undertaken. For example, in March 2014, HHT co-organized the BioTransfer conference in Toronto, which allowed for networking opportunities with local companies. Other actions undertaken by the Portfolio in 2013‑14 include the updating of communications materials and of the NRC external website, as well as the publication of 20 print articles and third party press releases. With respect to the Vaccines Program, HHT organized the Canadian Adjuvant Initiative Workshop (March 2013) and continues to actively participate in the development of the Federal Research, Development and Innovation Vaccine Federal Action Plan. This action plan seeks to enhance communication among sector stakeholders (government, industry and academia) and to address identified Canadian vaccine and immunization priorities. Footnote 34 Internal and external stakeholders described these two activities as having been central to the Vaccines Program's strategic positioning within the industry, as well as to the broader vaccine innovation ecosystem.

Key Finding 14: HHT's success with Zymeworks can serve as a model for other collaborations.

As will be described in Section 4.2, HHT has established a major collaborative relationship with Zymeworks Inc., which has proven to be extremely beneficial for both parties. The case study that was conducted on NRC's relationship with Zymeworks showed that the working relationship between the two organizations, which initially began as a relatively small collaboration, has evolved and expanded into a three-year, multi-million dollar agreement. In fact, the majority of Zymeworks' wet lab operations are now outsourced to NRC.

HHT management has indicated that this success reflects a strong model of industry interactions and that there are opportunities to replicate this with other companies. A number of internal interviewees stressed the importance of better showcasing HHT's success stories in order to demonstrate the potential impact NRC can have on SMEs and other clients. In building awareness, interviewees noted the importance of not only targeting potential clients, but also the VC community in order to give investors the confidence to invest in companies that adopt this model.

Key Finding 15: Although HHT has experienced challenges issuing press releases in the past, it appears that most of the processes that led to these difficulties have since been improved.

In this regard, many interviewees and experts observed that the best mechanism for showcasing HHT/client success stories is through NRC and third party press releases. While most internal interviewees agreed on the importance of press releases, many perceived the process of getting an official NRC press release approved as being extremely complex. They highlighted past difficulties in getting press releases issued even after significant time and effort had been expended. In response to this, NRC Communications Branch staff reported that although certain difficulties may have been experienced in past years, processes with respect to the issuance of NRC press releases have recently been improved.

Key Finding 16: IRAP Industrial Technology Advisors are not very familiar with HHT's service offering, making it more difficult for them to identify opportunities where their clients could benefit from HHT services.

Some internal and external interviewees felt that IRAP Industrial Technology Advisors (ITAs) could, when appropriate, play a stronger role in making their clients aware of the services being offered through HHT. Interviews with three IRAP ITAs confirmed the perception of HHT management that this group of employees could be more familiar with the new strategy and service offering of the Portfolio. A few internal interviewees noted that this situation may result in lost business opportunities for HHT and the inability of NRC as a whole (including IRAP) to best serve Canadian companies.

Recent efforts have been undertaken to increase the synergies between HHT and IRAP, including two meetings that were held in 2014 between HHT management and IRAP management/biomedical team staff. Still, efforts to date have been somewhat regionally focused. Many internal interviewees noted that ongoing engagement and sharing of information would be beneficial in order for ITAs to be able to more easily identify opportunities where their clients could benefit from HHT services.

Recommendation 2: HHT should seek ways to increase portfolio visibility and awareness, for instance by: a) maximizing the use of communications channels (e.g., press releases) and b) intensifying recent efforts to engage with IRAP ITAs on a regular basis, involving senior management as appropriate.

Key Finding 17: HHT has exceeded its revenue targets, but with enhanced business development capacity, the potential exists to further expand its business.

"Building a relationship with external clients can be complex. It takes a long time to get to know a company and to get to know what the company's needs are. It requires a team effort [between the research directors and the Client Relationship Leaders]."

Internal interviewee

As noted above, HHT has performed very well, meeting and, in some cases, exceeding the activity targets that had been set. Notwithstanding this, the evaluation found that HHT has the potential to further expand its business by developing new client relationships and strengthening existing ones. As noted by a few interviewees and illustrated in the case studies, the development and nurturing of client relationships is something that can take a lot of time and effort, and can involve many parts of the organization. HHT's ability to conduct proactive business development activities, both to develop new client relationships and strengthen existing ones, is impeded by two factors: 1) the current capacity and workload of BMS staff assigned to HHT and 2) the limited business development skills of senior level researchers who are also expected to contribute to this activity.

NRC's Business Management Support (BMS) Branch currently has four FTEs assigned to HHT: three Client Relationship Leaders (CRLs) and one Portfolio Business Advisor (PBA). It became evident throughout the evaluation that these resources devote a significant proportion of their time to managing the Portfolio's current client base and negotiating contracts, leaving very little capacity to conduct proactive business development activities. In the past year, this situation was further exacerbated by the temporary assignment of the PBA to the Medical Devices Portfolio, with no replacement provided. As noted by one research director; "Our CRLs are just in over their heads with work. It is very difficult for them to manage the current portfolio of clients and to reach out to potential clients".

While the role of BMS staff has been found to be critical in this area, HHT management clearly expressed the need to also increase the contribution of team leaders and other senior researchers in portfolio marketing activities. HHT management indicated that "team leaders need to become better ambassadors for the Portfolio", while also recognizing that significant training and coaching will be required before this group can fully embrace this new role. Some internal interviewees noted that despite the good progress that has been made to adopt a more industry-focused approach, remnants of an academic culture persists in some parts of the Portfolio. One interviewee estimated that it could take three to four more years before a complete culture change of this magnitude is realized.

In addition to the information presented above, the evaluation found that HHT and its predecessor institutes have invested significant resources in the development of technologies and intellectual property (IP), with an aim of eventually transferring them to industry partners. However, due to time and resource constraints, few efforts have been made to actively market these technologies to Canadian companies. Some internal interviewees stated that increased business development capacity would allow HHT to further reach out to clients to explore potential co-development or licensing opportunities and therefore better exploit the commercial potential of IP developed as part of internal research projects. This is particularly important when considering the success experienced by some SMEs, such as Alethia Biotherapeutics, following the licensing of IP from NRC (see Section 4.2).

Recommendation 3: HHT should identify and implement strategies to strengthen business development efforts.

4.2 Impacts on Clients and Industrial Partners

Evaluation Question 5

What impact have the activities of HHT (building on those of its predecessor institutes) had on portfolio clients and industrial partners, and on the Canadian biopharmaceutical sector?

Assessment

HHT has had a significant impact on the companies it has worked with, particularly with respect to the advancement and acceleration of product development and, as a result, increased company valuation and growth. In addition to this, HHT has also contributed to the strengthening of biomanufacturing capabilities in Canada.

Although the HHT Portfolio has only been in existence for a little over two years, the evaluation identified strong evidence of client impacts experienced as a result of their interactions with HHT. In fact, of the 22 clients that were interviewed as part of the evaluation, 18 reported noteworthy impacts resulting from their work with HHT (of the four clients that did not report any impacts, three indicated that it was too soon or their project was ongoing). Of those clients that reported impacts, most indicated a high level of attribution of these impacts to HHT. Given the long timelines associated with life sciences R&D, the most significant impacts were reported by clients, in particular Canadian SMEs, who had also previously interacted with HHT's predecessor institutes.

Overall, HHT has helped companies de-risk the development of their biologic and vaccine candidates and/or other innovative technologies. For some of the clients consulted, these improved or new therapeutics and technologies have resulted in increased company valuation, evidenced by increased access to investment capital. However, given the length of the product development process in the biopharmaceutical industry, longer term impacts, such as human health drug commercialization, impacts on patient health and reductions in health care costs, have yet to be realized. In the opinion of experts and based on evidence from the literature, these longer term impacts are more likely to occur in a 15 to 25-year time frame.

This section presents three main categories of impacts: 1) Advanced and accelerated product development; 2) Increased company valuation and growth; and 3) Strengthening of Canada's biomanufacturing capabilities.

4.2.1 Advanced and Accelerated Product Development

Key Finding 18: For most clients, HHT helped to advance and/or accelerate the development of their therapeutic candidates.

The evaluation found that, through its collaborations with private sector companies, HHT strongly contributed to the advancement and/or acceleration of therapeutic candidates, as this was an impact that was specifically identified by most of the clients interviewed. In particular, HHT's strengths in the area of bioprocessing were frequently cited as having helped to increase the yield and purity of the material produced, often helping to overcome roadblocks the companies were facing in their product development. In other cases, product advancement occurred as a result of significant co-development between HHT and the client and/or through the licensing of NRC intellectual property or enabling technologies. The examples below present information drawn from some of the evaluation case studies and illustrate HHT's Footnote 35 contributions in these areas.

  • Oncolytics: Prior to the collaboration with HHT, the company's outsourced production process had been experiencing difficulties, which resulted in problems with the yield and purification of its product. According to the client, the work done by HHT to optimize, and demonstrate the scalability, reproducibility and robustness of the production process, allowed Oncolytics to get its product to Phase III trials in a more timely and cost-effective manner than would have otherwise been possible.
  • PnuVax: HHT was found to have played an important role in the development of PnuVax's pneumonia vaccine for vulnerable populations, particularly in terms of getting it, in the words of the client, "this far, this fast". In the absence of HHT, certain aspects of the work would have been more costly, would have taken longer, and might not have happened at all. Specifically, the client estimates that having access to NRC's nuclear magnetic resonance (NMR) facility likely saved the company close to $1M.
  • Helix Technologies: The joint efforts of Helix and HHT led to the co-development of an antibody drug conjugate (ADC) molecule for the treatment of lung cancer. The ADC developed by the partners combines NRC antibody technology with the company's candidate therapeutic. This product is now in Phase I/II trials. The longer term impacts of this collaboration will be determined by the results of the clinical trials.
  • Alethia Biotherapeutics Inc.: Alethia initially licensed the exclusive rights on clusterin Footnote 36 -specific antibodies from HHT. The company then engaged in a series of projects which aimed at selecting and validating the therapeutic and commercial value of the lead therapeutic monoclonal antibody. Footnote 37 Throughout this process, Alethia benefited from the expertise of the HHT researchers (inventors of the antibody). As a result of this research, the monoclonal antibody was found to increase response to chemotherapeutic agents and to inhibit metastatic invasion. The work has led to a strong IP position on the target and monoclonal antibodies. Alethia expects to file Clinical Trial Application (CTA) and Investigational New Drug (IND) Footnote 38 applications for a Phase I clinical trial in November 2014.

Key Finding 19: Some clients reached key milestones in the drug development process as a result of their work with HHT.

Of those clients that reported that work with HHT has contributed to the advancement or acceleration of new therapeutics, some succeeded in reaching critical milestones in the drug development process. For example, five companies reported that HHT played a significant role in getting their products to the clinical trial phase, or progressing through the various phases of clinical trials. One of these therapeutics is now in Phase III trials, one is in Phase I/II trials, and three companies are now ready to begin Phase I trials and plan to file IND submissions in 2014. For one company, the work with HHT led to product commercialization as described in the example below:

  • Artemis Technologies Inc.: Artemis produces an animal vaccine to prevent the incidence of rabies in wildlife. They had partnered with the Ontario Ministry of Natural Resources to test, as part of field trials, the efficacy of the vaccine. When the company began its collaboration with HHT, it did not possess the R&D capabilities (expertise and equipment) required to scale up the production of the virus in order to meet the production needs of the field trials. HHT's unique expertise in viral production and process scalability led to the optimization of this process. The manufacturing and testing procedures developed by HHT have since been reviewed and certified by the appropriate regulatory authorities, allowing Artemis to sell its product on the Canadian market. The company has been generating sales and moving the business forward since the collaboration with HHT began.

4.2.2 Increased Company Valuation and Growth

Key Finding 20: Some clients reported increased access to investment capital as a result of their work with HHT.

Projects conducted in collaboration with HHT have, in some cases, helped to make the clients' products more attractive to investors, thus translating into additional financing opportunities for the companies. In this regard, seven of the 22 HHT clients who were interviewed reported having been successful at securing a total of $81.8M in venture capital and other financing since their companies first began collaborating with NRC. It is worth noting that some clients were not able to provide a specific amount and were therefore not included in this calculation. This number should therefore be considered to be a conservative estimate of HHT's impact in terms of increased access to investment capital. In addition to this amount, Zymeworks reported that it established a strategic collaboration with Merck & Co. Inc. worth up to US$187M (see the example below). While very few clients were able to attribute a specific amount to the work performed by HHT, most clients agreed that HHT's contribution played an important, and even critical, role in their ability to access investment capital in support of their growth. The following example illustrates HHT's impact in this area.

  • Zymeworks Inc.: Zymeworks has entered into a major collaboration with HHT, whereby HHT helps with the ongoing research and development, in a laboratory setting, of the company's engineered antibody and protein-based therapeutics platforms as well as the new therapeutics that are developed from these platforms. The validation of Zymeworks' platform is important since the technology is the basis for the company's strategic partnerships with large pharmaceutical companies, including the aforementioned collaboration with Merck & Co. Inc. and a more recent agreement with ImClone Systems. In addition to these agreements, Zymeworks has closed three rounds of private placement financing since 2011, totaling $34.1M. Speaking about the company's partnership successes, a Zymeworks representative stated, "At the end of the day, pharmaceutical companies are partnering with us based on the solid science that we demonstrated. The bulk of that has come from work that NRC has done. NRC is an integral part of this entire process."

With regards to HHT's contribution to client growth, a few clients specifically reported that their company has increased its number of employees since the beginning of the collaboration with NRC. Moreover, another client directly attributed the survival of the company to the services provided by NRC.

4.2.3 Strengthening of Canada's Biomanufacturing Capabilities

Key Finding 21: HHT contributed to the establishment of at least one Canadian biomanufacturer.

HHT's strategic documents strongly emphasize the importance of strengthening Canada's biomanufacturing capabilities so as to ensure that a greater proportion of the biologics developed by Canadian collaborators are ultimately manufactured in Canada. The manufacturing of biologics has been identified by the Portfolio as a critical condition for the achievement of strong impacts for Canada given that, with very few exceptions, most options for accessing CMO services are currently international.

The following example, drawn from the case study on PnuVax, provides evidence of HHT's contribution to the strengthening of Canada's biomanufacturing capabilities.

  • PnuVax: HHT has provided ongoing support to the establishment of PnuVax' biomanufacturing facility. In fact, the presence of NRC was found to be a critical factor that influenced the company's decision to acquire the building adjacent to HHT's Montreal facility. According to a company representative, it is very unlikely that the company would have purchased and established the cGMP facility in Montreal had it not been for the relationship with NRC. At the time, the company had considered other facilities in the area and elsewhere around the world. In the end, this particular location in Montreal was selected because: 1) NRC was willing and very able to help in a variety of technical aspects of PnuVax' program; and 2) the expectation was that companies conducting R&D with NRC would then need cGMP scale up, clinical trial level production or ongoing licensed manufacturing, aspects that were included in PnuVax' business plan from the outset. As stated by the client, both expectations have worked out very well.

5. Resource Utilization

Evaluation Question 6

Are HHT activities being undertaken in an economical and efficient manner?

Assessment

Yes. HHT generally operates in an economical and efficient manner.

Key Finding 22: Corporate key performance indicators show a relatively high level of overhead efficiency and staff utilization.

NRC corporately tracks a number of key performance indicators, some of which monitor portfolio overhead efficiency and business efficiency. The overhead efficiency indicator calculates indirect costs (that is, any costs that are not directly charged against a project) as a percentage of total expenditures. For 2013‑14, Footnote 39 HHT's indirect expenditures represented 33.8% of total expenditures. This ratio comes close to meeting the 30% target set by NRC and was the lowest of the three life sciences portfolios.

From a business efficiency perspective, NRC also tracks staff utilization (hours worked on projects as a proportion of total paid staff hours). For 2013‑14, HHT's staff utilization rate was 59.3%, whereas the target set by NRC is 70%. Despite not meeting the corporate target, HHT management feels that portfolio staff is highly utilized and noted two main reasons why this is not reflected in its ratio. The first is that some staff activities (for example, facility maintenance) are critical to portfolio operations but are categorized as an indirect expense. Secondly, there has been a significant learning curve with respect to NRC's new time keeping and project management systems and HHT management suspects that some staff may still be entering time incorrectly. It is also worth noting that HHT's staff utilization rate was the highest of the three life sciences portfolios and is the third highest among the 12 NRC portfolios.

Finally, another dimension that is worth noting is that of program recoveries. That is, in 2013‑14, 78% of HHT's labour expenses were spent supporting NRC programs. This ratio indicates a relatively high utilization of HHT human resources as, again, HHT has the highest ratio among the three life sciences portfolios and the third highest among all NRC portfolios.

Key Finding 23: When compared to operations under the previous institute model, HHT appears to be operating with a healthy level of productivity.

In order to assess the productivity (i.e. operational efficiency) of HHT, the evaluation examined the expenditures and revenues of the Portfolio, using 2013‑14 data, as compared to operations under NRC's previous institute model, using data from 2011‑12. In performing this comparison, the evaluation team worked closely with HHT and Finance Branch staff to ensure that the two time periods considered similar expenditure and revenue types and could therefore be reasonably compared.

As shown in Table 3 on the following page, in 2013‑14, HHT succeeded in generating more than double the amount of revenues that were generated in 2011‑12 under the previous institute model. This resulted in a productivity ratio of $41,244 in revenue per FTE in 2013‑14, as compared to $16,729 per FTE in 2011‑12. In addition, revenues represented 33% of salary and operations and maintenance (O&M) in 2013‑14, as compared to 19% in 2011‑12.

Table 3: Comparison of expenditures
2011‑12
(BRI/IBS) Footnote 40
2013‑14
(HHT)
Total expenditures (excl. capital) Footnote 41 $27,316,751 $37,875,111
# of FTEs Footnote 42 311 299
Revenue Footnote 43 $5,202,849 $12,331,876
Productivity (Revenue per FTE) $16,729 $41,244
Revenues as a proportion of Salary and O&M expenditures 19% 33%

Source: NRC Finance

Key Finding 24: Internal interviewees generally felt that HHT operates in an efficient manner.

HHT management and staff who were interviewed as part of the evaluation generally felt that the Portfolio operates in an efficient manner and provided examples of measures that have been taken to reduce or eliminate duplication. Most importantly, interviewees noted that the internal restructuring that has taken place over the past couple of years has led to significantly higher levels of efficiency and effectiveness. In line with the new matrix-based approach instituted at NRC, HHT has implemented its own matrix structure within the Portfolio, organizing expertise from across its three sites into four R&D departments, as well as an Operations department. Staff is deployed cross-functionally to projects in its three programs. Within the Operations department, HHT established a project management office, with dedicated key account managers who support client communications and work with the technical teams to ensure that project deliverables are completed on time and in an efficient manner. Interviewees indicated that the alignment of HHT's workforce and the streamlining of its activities have resulted in greater synergies within the Portfolio and have made it easier for management to assign resources according to project priorities. In fact, some interviewees noted that projects of the magnitude of those that are conducted for Zymeworks, for example, never could have succeeded under the previous institute model, where resources were much more siloed.

Internal interviewees also noted that, in 2013, HHT made some strategic cuts to its workforce (21 positions were eliminated), as it opted not to pursue certain areas of research in order to be better aligned with approved programs. The Portfolio also sought to consolidate some of its administrative staff, although some internal interviewees pointed out that this has occasionally led to inefficiencies as more administrative work has been taken on by senior level staff.

Finally, other measures that were mentioned by interviewees include reduced travel and better use of videoconferencing, as well as a cross-portfolio review of specialized facilities and equipment. This review showed, in particular, that current Portfolio needs for NMR equipment could be met through the reallocation of existing equipment and/or shared use. As a result, all major NMR equipment requests were canceled, saving NRC $2.31M in capital spending.

Some areas of inefficiencies were raised by interviewees, particularly with respect to interactions with NRC common services (information technology, human resources, and property management services, for instance). It is expected that at least some of these inefficiencies will be addressed as part of NRC's Common Services Transformation, which is currently ongoing and is expected to be completed in the fall of 2015.

Key Finding 25: Clients generally had an overall positive assessment of HHT efficiency although many reported that administrative processes can sometimes be slow.

Among the clients interviewed as part of the evaluation, the level of client satisfaction was generally found to be very high. Of the 22 clients who were interviewed (including those interviewed as part of the case studies), 17 (77%) were very positive about their interactions with HHT, three were somewhat positive and two were somewhat negative. Footnote 44 With respect to project costs, most clients felt that these were reasonable. When asked to rate their level of satisfaction on a scale of 1 to 5 for a number of different elements, clients provided average scores of at least 4 out of 5 for almost all of these elements. The only exception was with respect to the timeliness of services and deliverables. While clients provided an overall average satisfaction level of 3.9 out of 5 for this element, it is worth noting the exception since a couple of clients rated it very low.

Although client satisfaction is high overall, over half of clients interviewed mentioned the "slow and bureaucratic" administrative processes that are associated with working with NRC. Slow contracting processes stood out as being a specific frustration among clients (noted by nine of the 22 clients interviewed), and problems with invoicing were also raised (noted by four clients). With respect to contracting, most internal interviewees and at least one client specifically noted that the BMS staff assigned to the Portfolio appear to be overwhelmed in terms of workload. From a process perspective, some internal interviewees noted that these could be further streamlined (for example, by streamlining the chain of approvals or by making better use of centralized Strategic Contract Advisors). Some interviewees also noted that because BMS staff are currently fully occupied with contract negotiations, they have very little capacity to perform any proactive business development work (see Section 4.1.3 for more on this). A reduction in the level of effort required for administrative processes could therefore allow BMS staff to devote more time to these kinds of activities.

Recommendation 4: BMS should explore opportunities to streamline contracting processes within the HHT Portfolio. As part of this, an attempt should be made to further leverage centralized BMS resources, where appropriate.

6. Conclusion

Overall, the evaluation found that the HHT Portfolio has performed very well and exemplifies NRC's new strategy and direction as a research and technology organization (RTO). Despite the significant transformation that occurred over the past few years and the fact that the Portfolio, in its current form, has only existed for about two years, HHT activities have had a notable impact on clients. More importantly, HHT appears to be well positioned to continue to achieve impacts in the future. Portfolio management have adopted a culture of continuous improvement and are revisiting internal processes and approaches to ensure that HHT operations are both effective and efficient.

7. Management response

Recommendation Response and Planned Action(s) Proposed
Person(s)
Responsible
Timelines Measure(s) of Achievement
Recommendation 1: As part of the proposed "strategy refresh" for the Vaccines program, HHT should determine the reasons why the program has had difficulty securing strong collaborative relationships with Canadian industry. The Portfolio should then identify strategies and establish an action plan to address the issue. Recommendation accepted.
HHT will deliver a Vaccine Program "strategy refresh", including a root cause analysis for slower/lower uptake by Canadian industry and recommendations to address gaps and overcome difficulties.
Vaccine Program Leader 12/31/2014 (draft)
03/31/2015 (final)
Vaccine Program "strategy refresh" delivered. If, as part of this, the program changes beyond the boundaries originally established in program business planning documents, then these program changes will be reviewed by SEC. If not, the "strategy refresh" will be approved by the VP Life Sciences and reported in quarterly updates to SEC.
HHT will also incorporate key actions in the HHT Operational Plan for FY2016. HHT General Manager 02/15/2015 (or when requested) Key actions incorporated in FY2016 Operational Plan.
Recommendation 2: HHT should seek ways to increase portfolio visibility and awareness, for instance by: a) maximizing the use of communications channels (e.g., press releases) and b) intensifying recent efforts to engage with IRAP ITAs on a regular basis, involving senior management as appropriate. Recommendation accepted.
HHT will increase portfolio visibility via increased support from NRC Communications Branch and closer interactions with IRAP.
HHT General Manager, in collaboration with Communications Team Leader (HHT), HHT Portfolio Business Advisor and HHT management. 03/31/2015
(ongoing)

Press releases are delivered for all NRC-HHT collaborations with Canadian companies; success stories are showcased on the web.

Data on HHT impacts on Canadian companies are incorporated into promotional material.

HHT strategy and services presented to IRAP ITAs (Biomedical Sector).

09/30/2015 IRAP and HHT-BMS progress the Advanced Biomanufacturing initiative and enable the transfer/ manufacturing of at least 1 Canadian product to a Canadian CMO.
Recommendation 3: HHT should identify and implement strategies to strengthen business development efforts. Recommendation accepted.
In collaboration with BMS, HHT will implement key strategies to strengthen business development efforts.
HHT General Manager, in collaboration with Communications Team Leader (HHT), HHT Portfolio Business Advisor and HHT management. 06/30/2015 (ongoing)

Training delivered to strengthen the business development skills and customer-service behaviour of portfolio staff.

Outreach initiatives organized to increase our interactions with Canadian Biotechs and the venture capitalist community; presence at "Investors conferences".

Key client engagement by Senior HHT management.

Recommendation 4: BMS should explore opportunities to streamline contracting processes within the HHT Portfolio. As part of this, an attempt should be made to further leverage centralized BMS resources, where appropriate. Recommendation accepted.
In consultation with HHT management, BMS will review the HHT contracting practices to ensure alignment with best corporate practices and deploy resources for improved support.
BMS Executive, Director, BMS HHT Portfolio Business Advisor, and Group Leader Contracts, in collaboration with General Manager HHT and Director of Operations. 03/31/2015 Contracting processes and performance are reviewed.
06/30/2015 BMS contracting services are deployed in a manner commensurate with the HHT need and best practices embedded in the Portfolio's processes.

Appendix A: Evaluation matrix

Questions Methods
Document & Literature Review Administrative & Performance Data Analysis Stakeholder Interviews Case Studies Expert Consultations
Relevance
R1. Continued Need
1. Does HHT address a demonstrable need?
R2. Alignment with Federal Roles and Responsibilities
2. Are HHT activities and expected outcomes consistent with federal roles and responsibilities?
R3. Alignment with Government Priorities
3. Do the objectives and activities of HHT align with federal priorities and NRC strategic outcomes?
Performance (Effectiveness, Efficiency and Economy)
P1. Achievement of Expected Outcomes
4. To what extent has HHT positioned itself to achieve expected program outcomes?
5. What impact have the activities of HHT (building on those of its predecessor institutes) had on portfolio clients and industrial partners, and on the Canadian biopharmaceutical sector?
P2. Resource Utilization
6. Are HHT activities being undertaken in an economical and efficient manner?

Appendix B: HHT logic model

Appendix C: Methodology

This appendix presents the detailed evaluation methodology, and includes a discussion on the evaluation rationale, the scope, the design and methods, as well as the challenges and limitations encountered.

Evaluation Rationale

The evaluation of HHT was carried out in accordance with NRC's approved evaluation plan. The decision to evaluate HHT at this time is largely based on the maturity of its hosted programs. Relative to other NRC portfolios, HHT is more advanced in terms of program implementation, thus allowing for a better assessment of program outputs and early outcomes. In addition, prior to this evaluation, neither HHT nor its predecessor institutes had been the subject of an evaluation within the past ten years. As per Treasury Board's Policy on Evaluation (2009), all programs benefiting from any level of direct program spending must now be evaluated every five years.

The evaluation of HHT was led by an independent evaluation team from the NRC Office of Audit and Evaluation. The work of the evaluation team was supported by Methodological and Subject-Matter Experts who provided advice related to the evaluation framework, approach, interpretation of findings and recommendations. These experts did not act in a decision-making fashion, but rather played an advisory role in ensuring a high-quality and useful evaluation product.

Evaluation Scope

The evaluation assessed the value-for-money of HHT, including relevance, performance and resource utilization. It focused on the two-year period since HHT was created (2012‑13 to 2013‑14). Where appropriate and available, the evaluation considered earlier information relating to HHT's predecessor institutes (i.e., 2008‑09 to 2011‑12). Specifically, earlier data was consulted in the assessment of client impacts, given the long timelines associated with life sciences R&D. As well, financial and resource data from HHT's predecessor institutes were used as a comparator for the assessment of HHT efficiency.

Evaluation Design and Methodology

The selection of methods was based upon the most efficient means of addressing the evaluation issues in a rigorous way, while taking into account cost, time and resource constraints, as well as other considerations, such as minimizing response burden. The evaluation approach and level of effort was commensurate with the program risk, which was assessed as low during an assessment conducted as part of the planning phase.

In order to maximize the possibility of generating useful, valid and relevant evaluation findings, mixed methods were used, allowing for triangulation (i.e., convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different facets of a complex issue). Both qualitative and quantitative methods were used, and include:

  • Internal and external document review;
  • Administrative and performance data review;
  • Consultations (one-on-one interviews) with Canadian experts in the field of biopharmaceutical R&D;
  • Semi-structured interviews (including internal interviewees and external interviewees representing client organizations, prospective clients, industry associations and other stakeholder organizations); and
  • A series of ten case studies, based upon reviews of project and client documentation, as well as both internal and external (client) interviews.

A discussion of the approach used for each of these methods is provided in the following paragraphs.

Internal and External Document Review

Internal and external documents were reviewed, synthesized and integrated into the evaluation to provide context and history, and to complement other lines of evidence in assessing relevance and performance. Internal documents reviewed included strategic and business plans for the Portfolio and the three programs, special studies, intranet articles, and the program profiles prepared as part of the expert consultations. In addition, a wide range of external documentation was also reviewed by the evaluation team. A selected list of the documents reviewed can be found in Appendix D.

Administrative and Performance Data Review

Administrative and performance data for 2012‑13 and 2013‑14 were reviewed to provide information on program outputs and client reach, as well as to contribute to the analysis of resource utilization (e.g., staff utilization rates, productivity ratios). Administrative and performance data were provided by HHT staff, as well as by NRC corporate branches including Business Management Support (BMS), Finance, and Planning and Reporting Services.

In order to assess the productivity (i.e. operational efficiency) of HHT, the evaluation examined the expenditures and revenues of the Portfolio, using 2013‑14 data, as compared to operations under NRC's previous institute model, using data from 2011‑12. In performing this comparison, the evaluation team worked closely with HHT and Finance Branch staff to ensure that the two time periods considered similar expenditure and revenue types and could therefore be reasonably be compared.

Semi-Structured Interviews

Conducting interviews with key informants is an essential element of an evaluation methodology. The information gathered through the qualitative, semi-structured interview process was based on personal experiences, opinions and expert knowledge. This information plays an important role in contextualizing performance data and other statistics.

Interviews were conducted either in-person or by telephone. Each interview lasted between 1 and 2 hours and was conducted using an interview guide. Interview guides provided the common questions to be asked of each interviewee thus ensuring that the same issues were addressed by all relevant interviewees. All interviewees received the interview guide in advance of the interview. In some cases, interviewees elected to provide their comments in writing to the evaluation team, either following the interview or in one case, instead of participating in an interview. The majority of interviews were individual. However, one group interview (with HHT Key Account Managers) was conducted for reasons of efficiency and to provide a richer discussion.

A total of 46 stakeholders were consulted through the evaluation, including 17 internal stakeholders (HHT management and staff, BMS staff and IRAP Industrial Technology Advisors), and 29 external stakeholders (see Table 4). External stakeholders included active clients (defined as those who had had a project with NRC in the past three years), prospective clients (those who the Portfolio had identified as potential clients, but had not yet worked with on a project) and other stakeholders. These other stakeholders included representatives from two industry associations, one academic, and a representative of the Public Health Agency of Canada. Interviewees were selected in consultation with Methodological and Subject-Matter Experts and portfolio management, as well as through a review of HHT's client list. To encourage interviewee participation, the HHT General Manager sent a memo to potential interviewees informing them of the evaluation and advising that they may be contacted to participate.

Table 4: Stakeholders interviewed
Interviewee Category Number of Individuals Interviewed
Internal stakeholders 17
External stakeholders
Active clients 22
Prospective clients 3
Other stakeholders 4
Total 46

The number of active clients interviewed includes those who were interviewed as part of the case studies. In order to minimize the burden on clients, participants who were asked to contribute to a case study interview were not asked to be interviewed separately as key informants. For these clients, the questions in support of the case study development were supplemented by a few additional questions (primarily related to relevance). The responses were then analyzed along with the responses of the other client interviewees.

Case Studies

A series of ten case studies were developed to illustrate the types of projects and activities, which had been conducted by HHT, to identify whether client needs had been met through these projects and to gain a better understanding of the impacts resulting from industrial projects. The case studies were selected in collaboration with HHT management and BMS staff and included projects for all three HHT-hosted programs. The case studies also sought to include many of HHT's key clients and to capture an appropriate representation of the diverse impacts that the Portfolio has had on the biopharmaceutical industry. The projects selected for study are included in Table 5, on the following page.

The case studies were developed through reviews of project documentation, available external documentation (e.g., newspaper articles discussing the project or its impacts), and interviews with both HHT project staff and external client representatives. Semi-structured interview guides were developed and tailored for each client type (i.e., MNE versus SME). The case studies were developed using a common template and drafts were shared with internal and external interviewees for factual validation prior to their completion.

Table 5: HHT clients included in case studies
Program Client Name
Biologics Zymeworks Inc.
Oncolytics Biotech Inc.
Alethia Biotherapeutics inc.
Helix BioPharma
Vaccines PnuVax SL Biopharmaceuticals Inc.
Sanofi Pasteur
Artemis Technologies Inc.
VBI Vaccines Inc.
TBBB "Multinational A" Footnote 45
"Multinational B"

Expert Consultations

Individual consultations (one-on-one interviews) with nine Canadian experts (i.e., three experts for each of the three programs) in the field of biopharmaceutical R&D were conducted in order to assess the relevance and performance of each program. These experts were selected in consultation with management and staff from the HHT Portfolio and NRC's Business Management Support branch. Three of the recruited experts were affiliated with private sector firms, two experts were affiliated with government organizations, and four experts were affiliated with universities and other research institutes.

The consultation process proceeded as follows:

  • Preparation of the program profiles and other review material by HHT staff;
  • Review of program profiles and other relevant material by the experts;
  • Delivery of a presentation to experts by the HHT General Manager to provide high level context around the NRC transformation and the HHT Portfolio, including its programs and activities (experts were also given the opportunity to ask questions); and
  • Conduct of semi-structured interviews with experts, during which the experts were asked to comment on the program's alignment with industry needs, the strategic positioning of the program, the program's ability to reach clients, the quality/quantity of the outputs produced, and the extent to which the program is positioned to achieve its expected outcomes.

The responses provided by the experts were analyzed and findings were generated at the program level. These findings were triangulated with the findings from other lines of evidence to strengthen the analysis and interpretation of results.

Challenges and Limitations

Various limitations and challenges were experienced in conducting the evaluation of HHT that need to be taken into consideration when interpreting the findings. The challenges and limitations in the current evaluation are not unique. Rather, they are common in most evaluations conducted within the federal government. In order to alleviate the effects of the limitations and challenges on the evaluation findings, various mitigation strategies have been used. The main challenges experienced by the evaluation team as part of the project include the following:

  • Interviews as a primary line of evidence and potential response bias of client interviewees: Client interviewees (as well prospective clients) were limited to those who could be contacted, using the most up-to-date contact information obtained from the Portfolio and who agreed to be interviewed. This could bias interviewees towards those who have a more positive view of the program, and therefore a greater vested interest in the results of the evaluation. As a mitigation strategy, clients were contacted at least twice. Those who were hesitant to participate were provided with thorough information about the evaluation, its importance (and the value of their participation in ensuring valid results) and about the confidentiality of responses. In the end, clients interviewed as part of the evaluation represented organizations that were the source of 64% of HHT's revenue over the past two years.
  • Case studies were not identified randomly: The Portfolio (with support from BMS staff) proposed the list of projects from which the case study projects were selected. Therefore, these projects are not intended to be representative of all client activity. While no specific mitigation strategy was employed, the case studies do include a high proportion of HHT "key account" clients. As well, findings from the case studies were found to be consistent with the results of client interviews.
  • Content of program profile (reviewed by experts prior to the consultations): One of the main challenges experienced as part of the expert consultations pertains to the fact that the level of detail provided in the program profiles was not sufficient for experts to provide an informed assessment on specific aspect of the program's performance. This was particularly problematic with respect to the assessment of the quality and quantity of output produced. Many experts expressed the need for information in order to provide a fair assessment. As a mitigation strategy, the results from this line of evidence were complemented with the case study findings.

Appendix D: Selection of Documents Reviewed

Arundel, Anthony, David Sawaya and Ioana Valeanu (2010), "Human Health Biotechnologies to 2015", OECD Journal: General Papers, Vol. 2009/3. http://dx.doi.org/10.1787/gen_papers-2009-5kmjkjtfxdg7.

Conference Board of Canada. (2013). Export Market Share: Pharmaceuticals. Retrieved from http://www.conferenceboard.ca/hcp/details/innovation/export-market-share-pharmaceuticals.aspx.

Ernst & Young. (2013). Beyond Borders: Matters of Evidence. Retrieved from http://www.ey.com/Publication/vwLUAssets/Beyond_borders/$FILE/Beyond_borders.pdf.

GlobalData. (2012, September). Pharmaceutical and Medical Device Market Overview in Canada: Includes Market size, Market Segmentation by generics, Over-the-counter (OTC) and biologics/biosimilar products and Major Players SWOT Analysis. Reference Code: GDHR4898.

Government of Canada. (2013). Budget 2013. Retrieved from http://www.budget.gc.ca/2013/doc/plan/chap3-4-eng.html.

IMS Institute of Health Care Informatics. (2012). The Global use of medicines: Outlook through 2016. Parsipanny, NJ.

Industry Canada. (2006). The Canadian Biopharmaceutical Industry Technology Roadmap. Ottawa, Ontario. (Cat. No. Iu44-31/2006-MRC; ISSN: 0-662-49455-5).

Industry Canada. (2014). Life Science Industries: Biopharmaceuticals and Pharmaceuticals. Retrieved from http://www.ic.gc.ca/eic/site/lsg-pdsv.nsf/eng/h_hn01702.html.

Industry Canada. (2012). Life Science Industries: Pharmaceutical Industry Profile. Retrieved from a http://www.ic.gc.ca/eic/site/lsg-pdsv.nsf/eng/h_hn01703.html.

Industry Canada (2007), Mobilizing Science and Technology to Canada's Advantage. Ottawa, Ontario. (Cat. No. Iu4-105/2007E-PDF; ISSN: 978-0-662-44918-8).

KPMG, SECOR. (2012). Improving the Health of Canadians: The Contribution of the Innovative Pharmaceutical Industry.

Mazzucato, Mariana. (2013). The Entrepreneurial State: Debunking Public vs. Private Sector Myths (Anthem Other Canon Economics). Anthem Press: London.

National Research Council Act (R.S., 1985, C. N-15). http://laws-lois.justice.gc.ca/eng/acts/N-15/index.html.

National Research Council Canada. (2014). Human Health Therapeutics Portfolio Operational Plan FY2015. Ottawa, Ontario.

National Research Council Canada. (2013). Human Health Therapeutics Portfolio Strategic Plan FY2015‑2019. Ottawa, Ontario.

National Research Council Canada. (2014). Report on Plans and Priorities 2014‑15. Ottawa, Ontario.

Patented Medicine Prices Review Board. (2012). Annual report 2011: Analysis of Research and Development Expenditures. Retrieved from http://www.pmprb-cepmb.gc.ca/view.asp?ccid=895#1552.

PricewaterhouseCoopers LLP. (2013). The Missing Ingredient: Canadian Life Sciences Industry Forecast 2013. Toronto, Ontario.

Research Infosource Inc. (2013). Canada's Top 100 Corporate R&D Spenders 2013: Analysis. Retrieved from http://researchinfosource.com/pdf/Top%20100%202013%20-Final%20Article.pdf.

Rx&D. (2013). Saving Lives – Transforming Care. Retrieved from http://www.canadapharma.org/view.asp?x=342.

Society for Laboratory Automation and Screening. (2011). Bridging the Valley of Death: How can Academia and Pharma Best Work Together? Retrieved from http://eln.slas.org/story/1/49-bridging-the-valley-of-death-how-can-academia-and-pharma-best-work-together.

Tufts Center for the Study of Drug Development. (2013, November 14). Biotech Products in Big Pharma Clinical Pipelines Have Grown Dramatically. Retrieved from http://csdd.tufts.edu/news/complete_story/pr_ir_nov_dec_2013.

Footnotes

Footnote 1

Client interviews conducted as part of the case studies are included in the count of semi-structured interviews, as clients were asked a consistent set of questions (internal interviews are not included since these focused solely on the details of the case).

Return to footnote 1 referrer

Footnote 2

National Research Council (2013), Human Health Therapeutics Portfolio Strategic Plan FY2015‑2019, p.1.

Return to footnote 2 referrer

Footnote 3

Other revenue includes Royalties and Licenses, Lease and Use of Property, and Sale of Goods and Information Products.

Return to footnote 3 referrer

Footnote 4

Both the NRC Program Investment and the Net Loss are covered by NRC A-based and B-base funding.

Return to footnote 4 referrer

Footnote 5

National Research Council (2013), Human Health Therapeutics Portfolio Strategic Plan FY2015‑2019, p.2.

Return to footnote 5 referrer

Footnote 6

Revenue from other Canadian sources amounted to less than 1% of revenues in 2013‑14 and is therefore not included in the graph.

Return to footnote 6 referrer

Footnote 7

IMS Institute of Health Care Informatics (2012), The Global use of medicines: Outlook through 2016, p.9.

Return to footnote 7 referrer

Footnote 8

Industry and market statistics are generally reported for the pharmaceutical industry, as opposed to biopharmaceuticals specifically.

Return to footnote 8 referrer

Footnote 9

Industry Canada (2014), Life Science Industries: Biopharmaceuticals and Pharmaceuticals.

Return to footnote 9 referrer

Footnote 10

KPMG, SECOR (2012), Improving the Health of Canadians: The Contribution of the Innovative Pharmaceutical Industry, p.14.

Return to footnote 10 referrer

Footnote 11

Research Infosource Inc (2013), Canada's Top 100 Corporate R&D Spenders 2013: Analysis, pp.1-4.

Return to footnote 11 referrer

Footnote 12

Patented Medicine Prices Review Board (2012), Annual report 2011: Analysis of Research and Development Expenditures.

Return to footnote 12 referrer

Footnote 13

Industry Canada (2012), Life Science Industries: Pharmaceutical Industry Profile.

Return to footnote 13 referrer

Footnote 14

Conference Board (2013), Export Market Share: Pharmaceuticals.

Return to footnote 14 referrer

Footnote 15

Ernst & Young (2013), Beyond Borders: Matters of Evidence, p.34.

Return to footnote 15 referrer

Footnote 16

Rx&D (2013), Saving Lives – Transforming Care, p.8.

Return to footnote 16 referrer

Footnote 17

Ernst & Young, op. cit., p.54.

Return to footnote 17 referrer

Footnote 18

PricewaterhouseCoopers LLP (2013), The Missing Ingredient: Canadian Life Sciences Industry Forecast 2013, p.6.

Return to footnote 18 referrer

Footnote 19

Society for Laboratory Automation and Screening (2011), Bridging the Valley of Death: How can Academia and Pharma Best Work Together?

Return to footnote 19 referrer

Footnote 20

The term biosimilars (or subsequent entry biologics) is used to describe officially approved subsequent versions of innovator biopharmaceutical products made by a different sponsor following patent and exclusivity expiry on the innovator product.

Return to footnote 20 referrer

Footnote 21

Tufts Center for the Study of Drug Development (2013), Biotech Products in Big Pharma Clinical Pipelines have Grown Dramatically.

Return to footnote 21 referrer

Footnote 22

Arundel, Sawaya, & Valeanu (2010), "Human Health Biotechnologies to 2015", OECD Journal, p.121.

Return to footnote 22 referrer

Footnote 23

GlobalData. (2012). Pharmaceutical and Medical Device Market Overview in Canada: Includes Market size, Market Segmentation by generics, Over-the-counter (OTC) and biologics/biosimilar products and Major Players SWOT Analysis, p.11.

Return to footnote 23 referrer

Footnote 24

Industry Canada (2006), The Canadian Biopharmaceutical Industry Technology Roadmap, p.61.

Return to footnote 24 referrer

Footnote 25

National Research Council Act (R.S.C., 1985, C. N-15)

Return to footnote 25 referrer

Footnote 26

Mazzucato, Mariana (2013), The Entrepreneurial State: Debunking Public vs. Private Sector Myths (Anthem Other Canon Economics), p. 25.

Return to footnote 26 referrer

Footnote 27

It should be noted that more than 80% of the budget is awarded to academic researchers through competitive grants (http://www.nih.gov/about/budget.htm).

Return to footnote 27 referrer

Footnote 28

National Research Council (2014), Report on Plans and Priorities 2014‑15, p.8.

Return to footnote 28 referrer

Footnote 29

Government of Canada, Budget 2013. http://www.budget.gc.ca/2013/doc/plan/chap3-4-eng.html.

Return to footnote 29 referrer

Footnote 30

Industry Canada (2007), Mobilizing Science and Technology to Canada's Advantage, p.13.

Return to footnote 30 referrer

Footnote 31

Two expert reviewers provided a "moderate" rating with respect to TBBB Program competencies. However, both commented that it was difficult for them to provide an assessment based on the level of technical detail contained in the program documentation. The very positive endorsement of TBBB Program collaborators with regards to the NRC carrier technology and expertise of HHT staff was found to balance these lower ratings.

Return to footnote 31 referrer

Footnote 32

The list of Canadian biopharmaceutical companies is based on a target market list compiled by BMS in 2013 (194 firms), which was supplemented with 12 HHT clients that are considered to be biopharmaceutical SMEs, but do not appear on the BMS list, possibly because they were more recently established.

Return to footnote 32 referrer

Footnote 33

NRC Finance Branch, NRC Statement of Operations for the period ending March 31st, 2014.

Return to footnote 33 referrer

Footnote 34

NRC HHT, Operational Plan FY2015.

Return to footnote 34 referrer

Footnote 35

References to HHT are also meant to include its predecessor institutes.

Return to footnote 35 referrer

Footnote 36

Clusterin is a secreted factor which plays an important role in tumor progression and invasion.

Return to footnote 36 referrer

Footnote 37

A monoclonal antibody is an antibody produced by a laboratory-grown cell clone.

Return to footnote 37 referrer

Footnote 38

A CTA must be filed prior to the initiation of a clinical trial in Canada. An IND application is the equivalent process for clinical trials in the United States.

Return to footnote 38 referrer

Footnote 39

Only the corporate indicators for 2013‑14 are reported since, although NRC started collecting this data in 2012‑13, the first year was one of transition and the resulting data contains many anomalies.

Return to footnote 39 referrer

Footnote 40

2011‑12 expenditures and revenues include those associated with the human health and management components of NRC-BRI as well as those from NRC-IBS. Although a very small number of HHT FTEs previously fell under NRC-IMB, these resource and associated revenue figures were not included in the 2011‑12 calculation for the sake of simplicity and given the relatively small values involved.

Return to footnote 40 referrer

Footnote 41

In order to ensure consistency across the two time periods, the following expenditures were not included for either time period: amortization and other building-related costs, expenses related to business development and IP, and IT, finance and communications-related expenses.

Return to footnote 41 referrer

Footnote 42

FTEs for 2011‑12 include those associated with the human health component of NRC-BRI, all of NRC-IBS, and one FTE from NRC-IMB, as of September 2011. FTEs for 2013‑14 include all those of HHT as of September 2013.

Return to footnote 42 referrer

Footnote 43

2011‑12 revenues include those associated with the human health component of NRC-BRI, as well as those from NRC-IBS. IP royalties and Industrial Partnership Facility (rental) income is not included.

Return to footnote 43 referrer

Footnote 44

These negative views stemmed mostly from dissatisfaction with IP provisions for collaborations, project delays, the perceived high cost of fee-for-service work, and the unsatisfactory communication/presentation of final results.

Return to footnote 44 referrer

Footnote 45

The two multinational firms that are working with the TBBB Program preferred not to be identified in the report.

Return to footnote 45 referrer

Date modified: