ARCHIVED - Portfolio Evaluation of the NRC Technology Cluster Initiatives

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Executive summary

This report presents the findings from an evaluation of the National Research Council (NRC) technology cluster initiatives (CIs). The CIs represent the contributions of NRC to the development and growth of technological clusters across Canada. The CIs, located in eleven Canadian communities, "integrate industry, government and university resources through new partnership models that create the technological and entrepreneurial advantage for Canadian businesses to innovate and compete in the global marketplace".

An evaluation of the CIs was undertaken to provide information in support of the renewal process for the initiatives in 2009-2010. The study was led by NRC's evaluation function with the Planning and Performance Management Directorate, with support from private sector evaluators, as needed. The evaluation team also secured academic support for the literature review conducted as part of the evaluation.

The selection of evaluation issues used to guide the study was based on the Treasury Board Secretariat's Policy on Evaluation (2009), as well as the Results-based Management and Accountability Framework prepared for the Technology Cluster Initiatives. The issues identified include relevance, effectiveness and performance and leverage (efficiency and cost-effectiveness).

Evaluation scope

The evaluation focuses on examining relevance and performance achieved with the resources provided to NRC from 2000-01 to 2007-08 (Round I), 2002-2003 to 2007-08 (Round II), and 2003-04 to 2007-08 (Round III). This was necessary as the evaluation was conducted in late 2008-09 and concluded in early 2009-10.

Evaluation methodology

The methodology developed for this evaluation included multiple lines of evidence, as is standard practice in evaluation. The following methods were used to address the evaluation issues:

  • administrative and performance data review;
  • document and literature reviews;
  • cluster community discussion groups (n=11);
  • internal staff discussion groups (n=11);
  • cluster initiatives leverage analysis;
  • case studies (n=6), and
  • targeted interviews (n=52).

Summary of evaluation findings

The following provides a summary of key findings stemming from the evaluation.

Relevance

The role of the federal government in clustering is supported by the approaches taken in other OECD countries. Many countries, including members of the G8 and Asia, have put in place nation-wide cluster policies. In some instances, approaches have been adopted to stimulate laggard regions, to reinforce highly performing ones, and to diversify older industrial areas into higher technology ones. Almost all of these approaches involve some type of partnership between academia and government R&D laboratories, which supports the role of NRC in clustering.

All eleven cluster initiatives were found to be aligned with one of the four priority areas outlined in the Knowledge Advantage component of the Government of Canada's Science and Technology Strategy (2007). Further, a number of sub-priorities identified by the Science, Innovation and Technology Council (STIC) were found to be closely linked to the technological focus of the CIs. In several instances, the initiatives are also aligned with provincial or municipal strategies and investments (e.g., Nanotechnology, Nutrisciences and Health, Fuel Cell and Hydrogen Technologies).

Most of the technology areas at the heart of the cluster initiatives were found to be consistent with stakeholder needs and their vision for their region. They are also often consistent with the choices made by other industrialized nations in terms of technological development and public sector investment (e.g., nanotechnology and photonics). However, a few initiatives were identified in the evaluation as having a focus or orientation that may warrant closer examination in the future. These include Fuel Cell and Hydrogen Technology, Life Sciences, and Plants for Health and Wellness.

Effectiveness of program delivery and governance

A recurrent theme raised by evaluation participants was the effect of five-year funding for what is intended to be a long-term activity - regional technology or knowledge-based cluster development. This cycle and its associated requirements (e.g., performance reporting, evaluation, Memoranda to Cabinet, etc.) were identified by both internal and external stakeholders as hindrances to optimizing the results to be achieved by the investment. Further, the short-term funding was found to hinder the attraction and retention of highly qualified personnel in the cluster initiatives, and increases the risk to the ongoing maintenance of new infrastructure funded through the CIs.

The integration of cluster initiatives activities to existing Institutes was also found to lead to program delivery issues, especially in those cases where the technological foci of the cluster initiative and the Institute are identical. In these instances, it has been difficult and time-consuming for the Institutes to report separately on cluster initiative and core activities, investments, and outcomes.

Effectiveness and performance in delivering cluster support mechanisms

Overall, stakeholders consulted for the evaluation felt that there had been a positive change in the extent to which business support services have been made available to firms since the start of cluster funding and partially attributed this change to NRC. The role of NRC-IRAP in funding organizations was identified as an important enabler to clustering. These organizations work to support the innovative capacity of firms, or of a region delivering programming related to such areas as mentorship, business planning, regional planning (roadmapping and strategy development) and networking (conferences or networking, including participation in international missions).

The activities of NRC-CISTI in the CIs have evolved in various forms over the years and are highly valued by stakeholders. Despite a lack of direct funding in Rounds II and III of the cluster initiatives, NRC-CISTI has been able to provide Information Services and Competitive Technical Intelligence (CTI) products to nine of the eleven initiatives. The CTI services, in particular, are delivered through arrangements with NRC-IRAP and focus directly on the needs of its Industrial Technology Advisors (ITAs) and their client firms. In some instances, CTI services are covered by cluster initiative funding allocated to Institutes. The complexity of these arrangements, coupled with results from NRC's recent Strategic Review, leave questions as to how NRC will proceed in the future in terms of the provision of these services.

Industry Partnership Facilities (IPFs) are one of the ways in which NRC contributes to the development of firms. These facilities are generally designed to offer firms the opportunity to co-locate with NRC staff and serve a variety of purposes, such as providing tenants with access to NRC equipment and expertise, IT infrastructure, meetings rooms, and, in some cases, business services. Seven new IPFs were established using cluster initiative funding. The key strengths of the IPFs, in the view of tenants, are that the NRC brand provides credibility to cluster firms, that tenants have access to highly specialized equipment not available elsewhere, and that firms benefit from knowledge or technology transfer from NRC. The study also noted that the programming offered by IPFs across the cluster initiatives varies substantially, and that IPFs do not only house firms that are working in a cluster-related technology area.

Effectiveness and performance in developing specialized infrastructure

One of the most significant ways in which the CIs contribute to the development of clusters is through the provision of facilities and equipment to cluster stakeholders. Six CIs have contributed to the development or expansion of new facilities owned by NRC totaling over 425,000 square feet of new space in the cluster regions. Leased space is also an important component of the NRC presence across the country, with new space leased in six regions. Overall, these are seen as a positive contribution to cluster development and in many cases, speak to the unique role of NRC within the cluster regions. The NRC facilities are often considered a focal point in cluster regions and help to generate awareness of the presence of the federal government in these areas. The use of cluster funding, in some cases, to ensure ongoing maintenance of the new buildings was raised as a concern in the study due to the added risk this poses to the organization in the long-term.

Effectiveness and performance in supporting the development of highly qualified personnel

The cluster initiatives were found to support the development of highly skilled personnel (HQP) through the attraction and retention of individuals to the cluster regions and through the contributions of NRC staff to training in other organizations. CI funding was used in several instances to hire research officers, technical officers, students, and other individuals to work on cluster-related projects. In total, 490 individuals were hired by NRC between the fiscal years 2001-02 and 2007-08. The split between continuing positions and term positions is approximately even for this period. In 2007-08, the positions funded out of the cluster initiatives represented approximately 7.5 percent of all NRC positions.

The initiatives also contributed in an important way to the training of students in the cluster regions. A total of 235 students received direct funding from the cluster initiatives to conduct research work at NRC in various capacities (summer students, graduate theses, etc). This is a conservative figure, given that many students receive grants from other organizations to pursue thesis research in NRC laboratories or may receive funding from the Institutes' core budgets. By providing students with access to its laboratories and expertise, NRC exposes students to multi-disciplinary research teams and facilitates their transition to industry once their studies are complete. In addition to the knowledge transfer that occurs when HQP participate in NRC projects, NRC researchers are also heavily involved in their respective cluster communities in terms of training the next generation of scientists through cross-appointments at universities, where they teach courses and sit on thesis committees.

Finally, one of the ways in which the cluster initiatives also have been able to attract HQP is through the Visiting Workers Program, which allows Institutes to host scientists for a pre-determined period of time. Visiting workers may be university or industry researchers on sabbatical leave, industrial collaborators engaged under the terms of a collaborative agreement, retired NRC staff, or students. Up until 2007-08, NRC has hosted 404 individuals in relation with its cluster initiatives. The consideration of Visiting Workers is particularly important when studying the impacts of cluster initiative funding, since the influx of these workers increases the research capacity of NRC in the cluster region.

Effectiveness and performance in developing leading-edge knowledge

Although scientific excellence was not specifically assessed in the evaluation due to the large breadth of science areas represented by the cluster initiatives, evidence pertaining to leading-edge scientific activities conducted within the scope of the initiatives was nonetheless identified through stakeholder consultations and other methods. Overall, the funding provided through the CIs has enabled the creation of new NRC initiatives in advanced fields such as photonics and nanotechnology, and has supported leading-edge research activities in existing areas of competence, such as plant biotechnology, ocean technology, and biodiagnostics. In many of these cases, the research conducted at NRC was found to be complementary and aligned to the research priorities of other cluster partners.

One of the ways in which NRC generates leading-edge knowledge in the clusters is through collaborative agreements with other partners, such as universities, firms, and other government organizations. In total, the cluster initiatives reported 151 collaborative agreements signed over the evaluation period. Even though the initiatives report collaborative activities with the private sector, one of the potential barriers to success raised by stakeholders is the lack of receptor capacity for the results of advanced research. Further work is required to better align the research activities that take place in many cluster Institutes with the needs of the industrial community.

Effectiveness and performance in fostering the development of innovative firms

The extent to which the cluster initiatives have contributed to the development of innovative firms and industries depends largely on the stage of development of the cluster itself. With a number of clusters assessed as being in emerging or developing stages of evolution, expectations regarding firms should remain realistic. Further, some initiatives have been launched in regions that have different sizes, economic bases, and infrastructure. In many cases, the development of firms tends to occur once the impacts and outcomes described previously have been achieved, at least in part. However, direct and longer-term support to firms is provided primarily through the collaborative research undertaken by NRC and firms, the services and access to equipment provided by NRC Institutes, as well as support provided for R&D activities by NRC-IRAP.

Despite NRC's efforts, a number of challenges remain in the sustainable development of firms located in clusters. These include a lack of investment capital, a lack of incubation space, and the high cost of technology.

Effectiveness and performance in fostering networks and alliances

The development of networks and alliances between organizations is at the very core of cluster success. In general, the evaluation findings point to an increase or maintenance in the relationships developed between cluster actors over the entire CI funding period. The nature of these relationships, as well as the forms they may take in clustering, varies between the different cluster initiatives. The evaluation found that all CIs are involved in some way in networking activities, which tend to aim for the establishment of relationships between organizations at a general level. Most initiatives are also cooperating with key partners, undertaking projects or other activities that have finite objectives and separate tasks. Some initiatives have been able to move beyond these relationships and establish collaborative relationships with other cluster actors, where resources are shared to address common issues and a formal communication system has been established. Finally, a few initiatives are looking beyond their own geographic region and are involved in "cluster twinning" activities, where relationships are developed between clusters from different jurisdictions sharing similar technological foci.

Leverage and efficient use of resources

In order to determine whether cluster initiative resources were used efficiently, the evaluation sought to determine the extent to which other parallel investments were made in support of the cluster initiatives, both in tangible (i.e., financial) and less tangible (i.e., social) areas. In particular, an attempt was made to determine the extent to which there had been ‘leverage' in that NRC investment had resulted in subsequent investments by other levels of government, the academic community and the private sector.

Overall, available data suggest that NRC's direct regional investment in the CIs from the years 2000-01 to 2007-08, totaling $342M, has resulted in the leverage of $330M additional dollars in investment over this same period. Therefore, for every dollar spent in the CIs, almost an equivalent amount has been invested by other cluster actors in everything from infrastructure to research and development.

The significant finding is that in all cases the NRC investment in technology clustering has resulted in investment from other sources in support of the chosen technology area. The greatest degree of leverage has occurred in nanotechnology in Edmonton where the largest investments from partners have been made. The CI with the lowest level of total levered funds is Life Sciences, with roughly $6M levered against a $32M investment.

Sizeable investments have been made by provincial governments in six of the initiatives with smaller investments in a number of others. Other parts of the federal government are also shown to have been major contributors, such as in the Saguenay, where $25M was provided to help build the new Aluminium Technology Centre.

Although the evaluation did not attempt to assess the full investment in R&D by all cluster firms, it was able to determine the financial and in-kind contribution that is being made by firms in projects in which they are engaging with NRC. On average, 20 percent ($50M) of current leverage activity has emanated from the private sector. The Aluminium Transformation CI has the highest amount of levered funds from firms at $16M. The fuel cells and hydrogen technologies cluster also has a high proportion of firm-invested R&D, in keeping with the stage of development of the cluster.

When looking overall at research projects with the Institutes, on average for every dollar invested by NRC, collaborators have invested $4.40. In the case of NRC-IRAP, the investments made are estimated to have a leverage effect of engaging firms in investing at least 8 times the NRC contribution.

The investments being made by NRC in technology clusters across Canada are closely intertwined, not only with its other regular A-base activities, but with investments being made by other levels of government, as well as universities, NGOs and municipalities. As firms begin or continue to interact with NRC, and early stage government support for infrastructure development tapers, the proportion of leveraged activities from firms will likely grow.

Conclusion and recommendations

Ongoing work on NRC's clustering activities is supported by the findings of the evaluation. Without exception, community-based participants in the evaluation study expressed a desire to see investments continue. However, given the complexity of clustering and the level of investment required to support cluster development and growth, support by multiple players must be consistent, focused and long-term. Failure to engage over a 15 to 20-year period, with a reasonable level of resources, will not generate growth. Early results from this portfolio evaluation have identified that the investments made by NRC in its technology cluster initiatives have served to distribute research capacity and innovation opportunity more broadly across Canada. The variety of research areas being targeted, the range of activities supported, and the size of the communities hosting such initiatives demonstrate the applicability of clustering strategies and initiatives to a wide range of conditions.

NRC's role in support of regional clusters NRC has revolved around its infrastructure, people and brand. NRC has demonstrated an ability to play a 'broker' and 'catalyst' role, while still allowing communities to self-direct. A future challenge for the organization will be to balance its clustering strategy with other national priorities.

Five recommendations were put forward as a result of the evaluation study with NRC management responses and proposed actions as follows:

Recommendation 1:In light of evolving conditions and any apparent constraints (e.g., changing regional priorities, evolving scientific priorities, regulatory environment, etc.), it is proposed that NRC review and either reaffirm or modify the focus of its initiatives in the following areas: fuel cell and hydrogen technologies; life sciences, and plants for health and wellness. Timeline: March 2009 - April 2010.

NRC Management Response Action Plan: Accepted. NRC will review the focus of the initiatives in the areas mentioned to ensure that their alignment is consistent with the needs of the cluster and the community. Timeline: March 2009 - April 2010.

Recommendation 2:In light of the five year funding underlying this substantive investment for NRC (greater than 10% of total expenditures and affecting 11 of NRC's 19 Institutes), it is recommended that NRC assess, as part of any planned funding renewal, the risk associated with this investment. Strategies proposed as a result of this assessment should attempt to position relevant CIs as long-term activities and address issues such as staffing and capital assets.

NRC Management Response Action Plan: Accepted. The risks associated with the five-year funding period are well-understood and recognized by NRC. In order to mitigate those risks, the funding renewal exercise will include every possible argument in favour of obtaining A-base funding for the clusters. For any ongoing B-base funding, I/P/Bs will be asked to include a specific view of cluster risks in their business plan. Timeline: March 2009 - April 2010.

Recommendation 3:It is suggested that the management of technology cluster initiatives by NRC be undertaken in a more holistic and integrated approach across Institutes and Programs. It is recognized that in many instances, NRC cluster initiative activities are incremental to existing activities being undertaken by NRC. This is particularly the case for NRC-IOT, NRC-IMB, NRC-IBD, NRC-PBI and NRC-IFCI, and extends practically to all other delivery Institutes or programs, including NRC-IRAP and NRC-CISTI.

An ideal state would be to integrate the strategy, planning, and oversight of any cluster development progress into regular ongoing NRC processes (e.g., NRC strategy development, business planning, evaluation plan, etc). These would continue to report on and monitor contributions in support of planned objectives.

NRC Management Response Action Plan: Accepted. NRC will work to integrate cluster activities into Institutes' and Programs' business plans and performance reports, recognizing the necessity of also reporting and monitoring the contributions specific to cluster initiatives. Timeline: Fall 2010.

Recommendation 4:Review with NRC-IRAP and NRC Institutes engaged in clustering activities strategies for addressing the ongoing need for Information and Intelligence Services (IIS) (i.e., NICs and CTI products) in support of their regional cluster objectives given the impact of decisions surrounding the Strategic Review process.

NRC Management Response Action Plan: Accepted. Given that as a result of the 2008 NRC Strategic Review decisions, the Competitive Technical Intelligence services formerly provided by NRC-CISTI will be concluded. The decision to acquire any information research and analysis services for the individual cluster initiatives will rest with the lead cluster initiative NRC Institute. Timeline: April 2010.

Recommendation 5:Provide NRC researchers with the opportunity to learn about the purposes and goals associated with clustering. Adopt strategies that recognize and place value on interactions and projects with cluster firms or firms that are engaged in activities that are relevant to the technology focus of the cluster.

NRC Management Response Action Plan: Accepted. NRC will examine ways of raising the awareness and understanding of cluster initiatives with employees. Also, when reviewing its formal incentive programs, NRC will give proper consideration to the issues surrounding awareness and understanding of clustering activities. Timeline: March 2011.

Date modified: