Evaluation of the National Institute for Nanotechnology
Evaluation of the National Institute for Nanotechnology (PDF, 1.3 MB)
This report was approved by NRC's President on February 2, 2016
Acronyms and Abbreviations
|ACAMP||Alberta Centre for Advanced Micro Nano Technology Products|
|AITF||Alberta Innovates - Technology Futures|
|ASPM||Administrative Services and Property Management|
|CFI||Canada Foundation for Innovation|
|CFIA||Canadian Food Inspection Agency|
|CIHR||Canadian Institutes of Health Research|
|CNST||Center for Nanoscale Science and Technology|
|DND||Department of National Defence|
|EAC||Evaluation Advisory Committee|
|HEMiC||Hitachi Electron Microscopy Product Centre|
|HQP||Highly Qualified Personnel|
|IPF||Industrial Partnership Facility|
|IRAP||Industrial Research Assistance Program|
|MGLPC||Management, Group Leaders and Program Coordinator Committee|
|NCE||Networks of Centres of Excellence|
|NINT||National Institute for Nanotechnology|
|NIST||National Institute of Standards and Technology|
|NRC||National Research Council Canada|
|NRCan||Natural Resources Canada|
|NSERC||Natural Sciences and Engineering Research Council|
|OECD||Organisation for Economic Co-operation and Development|
|OAE||Office of Audit and Evaluation|
|PRC||Peer Review Committee|
|RO/RCO||Research Officer and Research Council Officer|
|ROI||Return on Investment|
|RTO||Research and Technology Organization|
|S&T||Science and Technology|
|SDT||Security and Disruptive Technologies|
|SME||Small and Medium Enterprises|
|SSHRC||Social Sciences and Humanities Research Council|
|STAC||Scientific & Technology Advisory Committee|
|TRL||Technology Readiness Scale|
|UofA||University of Alberta|
|WED||Western Economic Diversification Canada|
|WIN||Waterloo Institute for Nanotechnology|
|XRCC||Xerox Research Centre of Canada|
This report presents the results of the evaluation of the National Institute for Nanotechnology (NINT) for the period 2008‑09 to 2013‑14. NINT is a nanotechnology research and development facility based in Edmonton. It operates as a joint initiative between the University of Alberta, the Government of Alberta, and the Government of Canada, through the National Research Council (NRC). The study was carried out by an independent evaluation team from NRC Office of Audit and Evaluation. The work of the evaluation team was supported by an Evaluation Advisory Committee which included representatives from the Government of Alberta and the University of Alberta as well as the NRC. The evaluation addressed the core evaluation issues of value-for-money of NINT, including relevance, performance and resource utilization. The evaluation methods included interviews, administrative and performance data review, comparison study, bibliometric study and case studies. In addition, an international peer review committee was convened to assess NINT's past performance.
The key findings have been summarized and are presented below, along with the recommendations. Management has responded to the six recommendations, which is provided in Section 7.
The evaluation found that there is a need for continued public funding to support the advancement of key knowledge in nanotechnology and the growth and competitiveness of nanotechnology-enabled products and services that could lead to significant economic benefits for Canada. Specifically, there is a need for the scientific community and industry to access the type of facilities, equipment and professional staff offered by NINT. An undefined nanotechnology market and organizational barriers made it challenging for NINT to fully meet the needs of its stakeholders. Aside from the electron microscopy center, specialised services and dedicated technical staff at NINT, other facilities in Canada and internationally offer similar services.
The stated goals of NINT are in line with the role and responsibilities of the federal government and the NRC strategy. However, the evaluation brought to light a number of challenges that have resulted in NINT not achieving many of the goals. These challenges include deficiencies in project selection processes and the overall organizational/operational model.
Reach to stakeholders: Over the last six years, NINT has not had a notable increase in industrial engagement and much of the industrial engagement the Institute did have was limited to companies in Alberta. In a similar vein, NINT had a limited number of national and international academic collaborations. While NINT provided a quality training experience to a similar number of students as other competitive academic programs in North America, opportunities exist to enrich the scientific training.
Scientific excellence: In the Canadian context, NINT's publications in its four research areas had varying degrees of success in terms of research quality and scientific impact. On the international scene, NINT did not stand out as a world leader in the quality of publications it produced and the impact of its scientific research. A limited number of NINT projects were of leading-edge status and positioned to advance key knowledge in nanotechnology.
Impact on industry: NINT supported the creation of some new companies, licensed some technologies to industry and contributed to some positive outcomes for its clients (e.g., new or improved products or services, new intellectual property, new skills and knowledge). There is, however, little evidence to suggest that NINT contributed to the growth and competitiveness of nanotechnology enabled products and services in Canada to the extent that would be expected of a national institute of its maturity and with the level of funding it received.
NINT had the scientific and technical expertise to conduct leading-edge research and it had sufficient financial resources to achieve its expected outcomes. NINT's operational efficiency was facilitated, in part, by certain aspects of the NINT organizational model, its efficient use of human resources, and the recently implemented Project Management Office.
There were, however, several areas for concern. These included:
- undefined / unadhered to performance management approach
NINT's governing council, aware of some of the challenges with NINT's performance, introduced several changes to NINT's operation over the course of the evaluation period (e.g., change in NINT leadership, more focused strategic plan on R&D and commercialization) and continues to work on changes in an effort to improve performance (e.g., identification of strategies to address challenges associated with the NINT organizational model). Some of the changes had only been implemented for a period of two years or less at the time this evaluation was conducted.
The evaluation findings led to six recommendations:
Recommendation 1: NRC should work with NINT partners to develop and implement a strategy that ensures NINT fulfills its national mandate and develops international linkages.
Recommendation 2: NRC should work with NINT participants to train graduate students and postdoctoral fellows in order to enhance their ability in taking on leadership positions associated with nanotechnology in industry, government and academia.
Recommendation 3: NRC should ensure that NINT adheres to a rigorous project review process from project initiation to completion. All research expenditures should be subject to this review process, and external experts should be brought in to evaluate larger projects on a regular basis and act as advisors as needed.
Recommendation 4: NRC should monitor the recently implemented matrix approach to ensure that it in fact leads to changes in responsibilities and operations at NINT.
Recommendation 5: NRC, in collaboration with the NINT partners, should:
- Ensure that NINT makes use of its Science and Technology Advisory Committee to critically assess programs and expand its mandate to provide advice on the decisions on the project selections to ensure that NINT's strategic vision is implemented.
Recommendation 6: NRC should work with the NINT partners to ensure that the following systems and approach are developed and implemented at NINT:
- a performance management system
This report presents the results of the evaluation of the National Institute for Nanotechnology (NINT) which was undertaken in fiscal year 2014‑15. NINT is a nanotechnology research and development facility housed on the University of Alberta (UofA) campus in Edmonton. It is operated as a joint initiative between UofA, the Government of Alberta (GoA), and the Government of Canada, through the National Research Council Canada (NRC). NINT's mission is to transform nanoscience ideas into novel, sustainable nanotechnology solutions with socioeconomic benefits for Canada and Alberta.
NINT was selected for evaluation this year based on consultations with NRC Senior Management. NINT's last evaluation dates back to 2009, when the NRC technology cluster initiatives were evaluated. This evaluation covers the period 2008‑09 to 2013‑14.
NINT's current evaluation was led by an independent evaluation team from the NRC Office of Audit and Evaluation (OAE). The work of the evaluation team was supported by an Evaluation Advisory Committee (EAC) who provided advice related to the evaluation framework, approach, instruments, interpretation of findings, and recommendations. UofA and GoA were represented on the EAC and were also invited to attend the peer review committee meeting as observers.
The evaluation assessed the core issues of the 2009 Treasury Board Policy on Evaluation (see Appendix A: Evaluation Matrix). Given that the evaluation of NINT was conducted by NRC to assess the Government of Canada's investment in NINT, relevance was assessed relative to NRC's mandate, and resource utilization was assessed mainly from NRC's point of view. As the fulfillment of NINT's objectives draws upon resources beyond those provided by NRC, specifically from UofA and GoA, the full range of NINT's activities and outcomes were considered part of the overall performance evaluation.
However, all recommendations from the evaluation are directed to NRC. During the time that the evaluation was underway, NINT's organizational model was under review. Relevant findings from the evaluation were provided to NRC senior management during the review process for their consideration in discussions around the NINT model. Changes proposed or implemented outside of the evaluation period have been noted in the report.
The evaluation methodology integrated the use of multiple lines of evidence and complementary research methods as a means to enhance the reliability and validity of the information and data collected. The specific methods used in the study include:
- Peer review (committee membership may be found in Appendix E: Peer review committee membership).
A more detailed description of the study methodology and its limitations and challenges is provided in Appendix B: Methodology.
Sections 3 through 5 present the evaluation study's findings organized by broad evaluation questions (relevance, performance and resource utilization), along with associated recommendations. Section 6 presents a brief conclusion drawn from the evaluation, while Section 7 lays out management's response to these recommendations and the actions that will result.
2.0 NINT Program Profile
2.1 NINT overview
|2001:||Governments of Canada and Alberta commit $120M to the creation of NINT.|
|2002:||NRC begins to fund NINT via its Clusters Initiative.|
|2006:||GoA releases its nanotechnology strategy.|
|2006/2007:||NINT building and Innovation Centre officially opens.|
|2011/2012:||NRC realigns its strategy to become an industry-focused research and technology organization. NINT becomes a part of NRC's SDT portfolio and enhances its focus on commercialization.|
This profile provides readers with an overview of the program and how it has been defined for the purposes of the evaluation.
NINT is a 15,000 m2 nanotechnology research and development facility housed on the UofA campus in Edmonton. It operates as a joint initiative between the UofA, GoA, and the Government of Canada, through NRC (under its Security and Disruptive Technologies [SDT] portfolio). Footnote 1
NINT's mission is to transform nanoscience ideas into novel, sustainable nanotechnology solutions with socioeconomic benefits for Canada and Alberta. NINT aims to accomplish this mission by supporting a unique interdisciplinary environment that merges the discovery-focused, knowledge creation culture of a university laboratory with the innovation and commercialization-focused, problem-solving culture of a national laboratory.
NINT is currently involved in five research programs: Hybrid NanoElectronics, Energy Generation and Storage, Nano-enabled Bio-Materials, Metabolomics Sensor Systems, and Innovation Support. To accomplish the goals of these programs, NINT supports approximately 350 researchers annually and maintains $40M worth of facilities and equipment. Due to its location and relationship with UofA, NINT accesses an extensive array of equipment located elsewhere on campus (e.g. nanoFAB).
To transition its research into applied technologies, NINT offers opportunities for R&D collaborations or partnerships with industry, targeted technical services on a fee-for-service basis, and hosts laboratory and office space for start-ups as well as established companies in its Innovation Centre (NIC). NINT identifies its core competencies as: Materials, Fabrication and Characterization of Nanostructures (MFC); Surface and Interface Science (SIS); Electron Microscopy (EM); and Enabling Facilities and Policies (EFP).
2.2 Program goals
Through its research programs and support for industry, NINT plans to achieve the following goals:
- Become internationally recognized for its research and technology
NINT aims to rank among the top five nanotechnology research institutions in the world, a prestige expected to attract world-class researchers, establish the region as an international hub for selective areas of nanoscience and technology, and position NINT in a leadership role for national and international nanotechnology research networks.
For more details about inputs, activities, outputs and outcomes, see NINT's logic model Appendix C: Logic model.
2.3 NINT organizational structure
The NINT Council provides overall governance and is considered NINT's senior strategic decision-making body. The Executive Director (ED) of NINT reports directly to the NINT Council and to the NRC Vice President (VP), Emerging Technologies Division. The NINT Council represents the interests of the Government of Canada, NRC, UofA, and GoA.
NINT's Director of Operations reports to the NINT ED and its Director of Research reports to the General Manager of Security and Disruptive Technologies at NRC, who in turn reports to the VP of Emerging Technologies at NRC Footnote 2.The Directors of Research and Operations meet with the NINT ED as part of the Management Committee and as part of the Management, Group Leaders and Program Coordinator Committee (MGLPC), to discuss and update budget, human resources actions, business development, program management, health and safety, and other strategic and operational issues.
The NINT ED also receives advice from the Scientific & Technology Advisory Committee (STAC). As the senior advisory body of the institute, STAC includes members from the academic, business and industry communities. More information on this topic is found in Section 5: Resource utilization.
2.4 Key stakeholders and beneficiaries
Key stakeholders of NINT's efforts include Canadian and international industrial clients and collaborators (from a variety of industrial sectors), NINT researchers (including students) and staff, the founding partners (Government of Canada, NRC, GoA, and UofA), and national and international nanotechnology scientific communities.
2.5 Program resources
This section presents NINT's human resources and financial resources over the evaluation period.
2.5.1 Human resources
NINT was supported by approximately 383 staff each year Footnote 3. This included approximately 74 scientific and technical staff employed by NINT annually. The core scientific staff included those employed solely by NRC (i.e., 35) and those who were cross-appointed from the UofA (i.e., 17). The UofA cross-appointees typically committed 50 percent of their time to NINT with the remainder to UofA. In addition, NINT had approximately 27 management and support staff (including the Institute's administrative staff and NRC's common service staff and 280 visiting workers, post-doctoral fellows and students annually (see Appendix G: Additional tables and figures). Footnote 4
2.5.2 Financial resources
Over its first five years (2002‑2007), the Government of Canada, through NRC, and UofA and GoA contributed $120M to NINT. The GoA and UofA contributions were used for the NINT building and equipment acquisition while NRC's contributions were used for equipment acquisition as well as salary and operations. Since then, NRC, UofA and GoA have contributed to the operational, capital, and research funding of NINT. This section presents the financial contributions to NINT from its partners as well as earned revenues from 2008‑09 to 2013‑14. More information on the Institute's resources is found in Section 5: Resource utilization.
- Earned revenues – NINT earned an average of $2.6M annually in revenue from collaborative R&D and fee-for-service work (see Figure 2, below).
Source: NINT financial data
The relevance of NINT was examined through three evaluation issues: the continued need for NINT; its alignment with the priorities of the federal government; and its alignment with the roles and responsibilities of the federal government and of NRC.
3.1 Continued need for NINT
In order to assess the continued need for the program, the evaluation examined the importance of nanotechnology for the Canadian economy, the need for a national nanostrategy, the needs of stakeholders, the challenges faced in meeting the needs of stakeholders and the ability to meet client needs in the absence of NINT.
3.1.1 Importance of nanotechnology for the Canadian economy
Nanotechnology is defined as science, engineering and technology conducted at the nanoscale. It has applications across science fields such as chemistry, biology physics, material science, and engineering. As an enabling technology it translates into several industry sectors, including energy, transport and security Footnote 5. In Canada, nanotechnology based research is carried out at the federal and provincial levels in research laboratories, industry and universities (e.g. UofA, University of British Columbia, University of Toronto, McGill University and University of Waterloo). Edmonton is identified as one of the eight Canadian nanotechnology clusters Footnote 6.
In 2011, total government funding for nanotechnology research worldwide amounted to $67.5 billion Footnote 7. It is estimated that by 2015, the global market for nanotechnology will reach between US$1.5 trillion and US$3 trillion and 2 million new jobs could result Footnote 8. The USA, Japan, and the European Union (EU) are leaders in nanotechnology R&D investments. Canada, however, is not seen as a major player in nanotechnology with regards to R&D investments, having ranked 21st in the amount per capita invested in R&D in nanotechnology and contributing to 2% of global GDP in nanotechnology. Footnote 9.
The Alberta Nanotechnology Strategy (2007) Footnote 10 and internal interviewees, including representatives from the GoA, indicated that investing in nanotechnology industries is important to stay on the leading-edge and remain competitive. As one senior official pointed out, the need for public funding in support of nanotechnology stems from industry often not willing to invest its own money in research. As is discussed in Section 3.1.3, NINT clients indicated that public funding in nanotechnology (i.e., the existence of NINT) allowed them to pursue research that they would not have conducted themselves because of their lack of capacity to do so, suggesting that there is a role for public investment in nanotechnology in Canada. According to the Return on Investment (ROI) Analysis prepared for NINT in 2011, there is great potential for investments in the areas of life sciences for reduced diagnostics costs as well as in the area of energy in terms of sales revenues for nano-micro-technology based manufacturers. Footnote 11.
3.1.2 Need for a national nanotechnology strategy
Within the scope of the evaluation, the absence of a national nanotechnology strategy was found to have affected Canada's ability to capture a greater share of the nanotechnology market. The Comparison Study demonstrated that Canada was one of the few Organisation for Economic Co-operation and Development (OECD) countries without a nanotechnology strategy/initiative. Internal and external interviewees argued that the absence of a national strategy led to the emergence of provincial strategies. For instance, Alberta has the Alberta Nanotechnology Strategy, which is an example of a provincial effort to align and coordinate nanotechnology activities and resources to a strategy. Individual provincial efforts were, however, largely uncoordinated at the national level and also resulted in duplication of efforts among the nanotechnology centres across Canada. Table 1, below, provides an overview of the key services offered through the provincial associations. Some of these services are similar to those provided by NINT (e.g. technical services).
|Technical services (through its members)||✓||✓|
|Point of contact for industry and community of practice (i.e., acts as the interface)||✓||✓||✓|
|Access to state of the art facilities and R&D expertise (through its members)||✓||✓|
|Development of research networks||✓||✓|
|Support of highly qualified people (e.g., by providing funding to professors)||✓||✓|
Source: Web-based search and document review
Recently, NINT played an important role in the creation and the launch of NanoCanada, a national nanotechnology network, which has the establishment of a national nanostrategy as one of its objectives Footnote 12. The NanoCanada Memorandum of Understanding (MoU) was signed by NINT, NanoQuebec, NanoOntario, Innovation Saskatchewan, and Advanced Materials and Process Engineering Laboratory (British Columbia), CMC Microsystems, XEROX Canada, Hitachi Canada, and the University of Toronto in 2013. Three meetings were held in 2013-2014. By the end of the evaluation period, five additional meetings took place leading to the launch of NanoCanada in March 2015.
Documentation on NanoCanada describes it as a national initiative that brings together the community to stimulate innovation, enhance R&D capacity and stimulate the development of nanotechnology applications in collaboration with industry. In its Strategic Plan for 2012-17, NINT claimed that it is well positioned to spearhead NanoCanada because it is the only national organization that performs nanotechnology R&D and is well positioned to provide leadership in working with all nanotechnology players. Findings indicate that if NINT is to be successful at spearheading the NanoCanada it will need to ensure that it has the appropriate capacity to do so.
3.1.3 Needs of NINT stakeholders
The evaluation identified the needs of NINT stakeholders, challenges faced by NINT in addressing these needs, and alternative organizations that could meet the needs of NINT stakeholders. Each of these items is discussed below.
Needs of the scientific community and industry
The evaluation found that NINT played a role in supporting the scientific community, including researchers and students, as well as industry, largely centered on the provision of facilities (i.e., labs and the Innovation Centre), equipment, and professional staff.
As was highlighted by NINT clients, the cost to acquire necessary equipment and competencies required to conduct research in nanotechnology would otherwise have been prohibitive, resulting in their need for NINT services. Excerpts from the impact case studies demonstrate how NINT has met companies' needs.
"By partnering with NRC, we can do high risk R&D and then look at commercialization potential. This allows us to spend all of our energy on product development and also it helps us focus on expanding our sales and markets."
- Client interviewee
- Jet-Lube of Canada believed that their small R&D division was not capable of undertaking the required R&D itself and the company was uncertain about its ability to acquire a sufficient amount of capital to properly conduct the required research and tests. NINT was able to conduct the research.
NINT's Innovation Centre, an industrial partnership facility that assists technology start-ups and established companies with their nanotechnology ventures, also helped meet companies' needs by providing the latest instrumentation and technical expertise. The occupancy rate in the first two years was lower at 69% but has been at 77% for the last four years. Between 2008‑09 and 2013‑14, the Centre hosted 15 companies (mostly SMEs), start-ups or university spin-offs. Between 2009‑10 and 2014‑15, the Innovation Centre generated an estimated $1.8 million in revenue for NRC. Despite a significant increase in the rate charged to tenants in 2011‑12, the occupancy rate remained relatively stable, indicating that the facility was well used by clients.
"Long term continuity in terms of the operation of the equipment and science are afforded by the NRC technical staff as opposed to graduate students who do not stay long term.”
– UofA interviewee
In terms of scientific and industrial support, NINT's approach is unique in that full-time technical officers are dedicated to the operation and maintenance of equipment, whereas these tasks are usually performed by graduate students in university labs. The long-term continuity of the technicians and their expertise was viewed by the scientific community and industry as a value-added feature at NINT. Likewise, the service provided by dedicated technical officers allowed NINT to offer improved reproducibility.
Challenges faced by NINT in meeting stakeholder needs
Despite generally supporting the needs of its stakeholders, the evaluation found that NINT faced some challenges in fully meeting the needs of the scientific community and industry. Several factors, discussed below, contributed to the challenges faced by NINT in fully meeting its stakeholder's needs.
"The (engagement) strategy is a little opportunistic and spread across sectors. It is, however, possible to better align low TRL projects with industry needs and this is not always the case for NINT's internal projects."
- Internal interviewee
- Organizational barriers, such as restrictive access to NINT's facilities (e.g., single entry point via a commissionaire), higher costs for services and other administrative issues (e.g., completion of lengthy administration procedures to access equipment) affected NINT's ability to meet some of the needs of industry and the scientific community. These challenges, which are further discussed in Section 5: Resource utilization, are related to NRC's security and visiting worker process and Treasury Board policies for operating a government building (i.e., Operational Security Standard on Physical Security).
Ability to meet needs in the absence of NINT
Canada has several university-based research clusters that produce world class research in nanotechnology in which the Canada Foundation for Innovation (CFI) has made significant investments (> many $10s millions). Clients identified other universities in Canada that could provide similar services as NINT (i.e. University of Toronto and McGill University). Interviewees noted that nanoFAB at UofA was an alternative to NINT. NanoFab is an open-access facility supporting industrial and academic research users from across Canada and it caters to academic and industrial users. NINT and nanoFAB have some different service offerings but aside from NINT's unique electron microscopy centre, several interviewees noted that they could access equipment similar to NINT's at nanoFAB.. In addition to national universities that could provide similar services as NINT, some interviewees highlighted foreign universities as well, such as the University of Rochester. According to clients and the PRC what is unique to NINT, and what adds value, is the dedicated technicians that operate and maintain the equipment.
Figure 3 , below, reveals that NINT operated in a similar space as universities, which offer industry support and nanofabrication facilities in the private, public and not-for-profit sectors. Despite there being alternatives, NINT was still chosen for the reasons discussed above (i.e., facilities, equipment and/or professional staff).
3.2 Role of the federal government
The appropriateness of NRC's role in supporting the nanotechnology industry through NINT stems mostly from subsection 5(1)(c) of the NRC Act, which states that the Council may "undertake, assist or promote scientific and industrial research Footnote 13. Under the Act, NRC is charged with the direction or supervision of research undertaken by or for industrial firms or other organizations (subsection 5[d]) and with carrying out experimental and developmental work with respect to the above, and making the resulting processes, methods and products available for the benefit of manufacturing and other scientific purposes (subsection 5[k]).
3.3 Alignment with government and NRC priorities
Although NINT is a partnership between NRC, UoA and GoA, this evaluation looked at relevance from the federal perspective.
3.3.1 Alignment with federal government priorities
NINT's nanotechnology research and increasing emphasis on industry and innovation support are both aligned well with the federal government's two most recent science and technology strategies. The previous 2007 Science and Technology (S&T) Strategy, Mobilizing Science and Technology to Canada's Advantage, emphasized the need to translate knowledge into commercial applications, promote world-class excellence and for Canada to be a magnet for highly skilled people. This alignment is demonstrated through NINT's intended outputs of new or improved nano-enabled products and technologies and its aim to rank among the top five nanotechnology research institutions in the world while attracting world-class researchers. The recently updated strategy, Seizing Canada's Moment: Moving Forward in Science, Technology and Innovation (2014), retains the core principles of the previous strategy and has explicitly identified advanced manufacturing, such as nanotechnology, as a new priority.
Further evidence of the federal government's continued support for nanotechnology is provided by its recent announcements of public support for nanotechnology. For example, in January 2015, the federal government made an investment in state-of-the art metabolomics assessment equipment at a new Metabolomics Technology Demonstration Centre (to which NINT is a contributor). As well, in June 2014, an announcement was made about the purchase of an ultra-high resolution scanning tunneling microscope at the UofA, acquired with funding provided by the Canada Foundation for Innovation (CFI) a federal funding agency, and the GoA (with additional support from NRC).
3.3.2 Alignment with NRC priorities
This section assessed NINT's alignment with NRC and from the federal perspective. NINT's planned outcomes (see Appendix C: Logic model) such as "innovation breakthroughs in nanotechnology" align with NRC's strategic outcome for Canadian businesses to prosper from innovative technologies. At the NRC corporate level, mechanisms have been put in place to ensure that program activities align with NRC priorities. Programs are approved by NRC's Senior Executive Committee (SEC) following a rigorous stage gate process. The fact that NINT was approved for implementation in June 2013 by SEC has helped to ensure that it is aligned with NRC's new strategic direction and stated goals.
Despite this process for ensuring alignment with NRC priorities, the evaluation revealed challenges in NINT's practical achievement of its stated goals and outcomes, which is discussed in greater detail in Section 4: Performance. As a result, NINT's alignment to NRC has been largely in theory and not in practice. As the Peer Review Committee (PRC) noted, this was due in part to the poor alignment between NINT's programs/projects and its strategic plan. NRC, in conjunction with the NINT partners, is currently looking at strategies to address the misalignment. The GoA has indicated that it welcomes the opportunity to explore approaches to increase alignment and collaboration on national and provincial nanotechnology developments with the federal government and other provinces.
Additionally, senior executives from GoA, UofA and NRC noted that the misalignment was, in part, due to the conflicting cultures and mandates of the NINT partners. Evaluative evidence found that NINT's corporate practices and processes have not enabled an environment that brings together the different cultures of its partners. The different cultures and mandates of the NINT partners affected the extent to which staff shared a common identity. NINT staff were treated differently depending on which organization the individual came from – NRC or UofA. This created confusion among staff with regard to their roles and responsibilities in supporting NINT's goals and objectives.
Interviewees highlighted the following differences in operating and administrative practices and processes between NRC and UofA staff at NINT:
- Access to UofA facilities and resources – UofA staff had access to UofA library and to recreational and medical facilities whereas NRC employees did not. Likewise, NRC staff were not allowed in the UofA nanoFab labs without a university access card and were charged a 15% overhead on services. Only the few NRC staff with adjunct professorship or appointment with UofA have a university access card.
Internal interviewees commented that the different operating and administrative processes were a source of frustration. In fact, internal interviewees suggested that some voluntary departures of NRC staff seeking positions at UofA stemmed from this perceived imbalance. Nevertheless, internal interviewees felt that NINT staff genuinely want to see NINT succeed but that it would require the right operating structure to function cohesively. This view is supported by the fact that some employees have taken the initiative to create a group (i.e. the MOD SQUAD) to resolve some operational challenges.
While NINT's organizational model was originally designed to capture the strengths of an academic institution and a federal lab, findings, such as those previously discussed and those discussed in subsequent sections of the evaluation report, point towards shortcomings in its success. NINT's governing council, aware of some of the challenges with NINT's performance, introduced several changes to NINT's operation over the course of the evaluation period (e.g., change in NINT leadership, more focused NINT strategy on R&D and commercialization; see Section 5.3.1) and continues to work on changes in an effort to improve performance (e.g., identification of strategies to address challenges associated with the NINT organizational model). Some of the changes introduced had only been implemented for a period of two years or less at the time this evaluation was conducted.
NINT's performance was assessed by evaluating: client reach; national and international research collaborations; training of students; scientific excellence and impact; and NINT's contribution to the commercialization of nanoenabled products and services.
4.1 Client reach
This section evaluates NINT's client reach by first describing NINT's overall clients and agreements (i.e., projects), and then type of agreement (i.e., collaborative R&D, and research, testing and technical services).
Overview of NINT's clients and agreements
Since 2008‑09, NINT has worked with 62 clients on 146 agreements. The total value of these agreements was $33M, from which NINT received around $15.7M in revenue. As shown in Figure 4 below, there has not been a notable increase in NINT's engagement with clients over the past six years.
NINT's clients were predominantly from the private sector, which accounted for approximately three quarters (46 of 62) of NINT's clientele and represented around 83% ($28M) of the overall value of agreements. Two of NINT's largest clients had agreements valued at $9.3M (in 2008‑09) and $9.5M (in 2011‑12), respectively. Of the 46 private sector clients, 16 (35%) returned for subsequent work and five (11%) signed more than two agreements with the Institute.
NINT's clients were mainly located in Alberta (i.e., 66% or 41 of 62), which amounted to 60% ($20M) of NINT's total agreement value and 61% ($9.6 M) of revenue. Of NINT's remaining clients, 8% were from Ontario, 15% were from elsewhere in Canada and 11% werefrom foreign countries.
Source: NINT client agreement database
NINT works with clients on collaborative R&D projects. Between 2008‑09 and 2013‑14, NINT had 46 collaborative R&D agreements and represented 32% of NINT's total agreements. R&D collaborations comprised 73% ($11.5M) of NINT's revenues. The annual number of collaborative R&D agreements increased in 2009‑10 and then remained relatively stable following 2010-11 (see Figure 5 below). The majority of clients with whom NINT engaged in collaborative R&D were from the private sector.
Source: NINT client agreement database
The Institute's ability to conduct collaborative R&D with private companies was significantly bolstered by the GoA's nanoWorks program. As part of the Alberta Nanotechnology Strategy, nanoWorks delivered funds jointly with industry to increase industry's access to Alberta's micro-nanotechnology (MNT) infrastructure, and to stimulate industry-institutional research and product development collaborations. Of the 18 private sector companies that worked with NINT on collaborative R&D projects, 8 were supported by nanoWorks (i.e., 44%). The proportion of revenue that NINT received from these projects accounted for 84% of NINT's revenue derived from R&D collaboration.
Conversely, revenue from private sector clients who worked with NINT on collaborative R&D projects and who did not receive nanoWorks funding accounted for a very small proportion of NINT's revenue generated from collaborative R&D projects (i.e., 7%; see Figure 6, below). The large proportion of collaborative research revenue from nanoWorks is particularly noteworthy because nanoWorks is currently under review. This may present a risk to NINT. Reinvestment in nanoWorks might be a practical way to encourage further SME commercial developments.
'Other' includes OGD, university, and foreign research centre agreements as well as one small ($30k) Alberta Heritage Foundation for Science & Engineering Research grant
Source: NINT client agreement database
Research, testing, and technical services (fee-for-service)
NINT performed fee-for-service work for clients. Between 2008‑09 and 2013‑14, NINT had 82 research, testing and technical agreements (i.e., fee-for-service), which represented 56% of NINT's total agreements. Research, testing, and technical services comprised 10% ($1.6M) of NINT's revenues. Over six years, the number of research, testing and technical services, the majority of which were provided to private sector companies (see Figure 12, in Appendix G) have remained relatively stable. NINT's large number of fee-for-service agreements compared to collaborative R&D is unexpected given that the Institute operates in an emerging technology area. Examples of fee-for-service agreements include access to equipment and clean room facility and testing and technical services.
4.2 National and international collaborations
The evaluation assessed the extent to which NINT was successful at engaging with national and international nanotechnology research organizations and researchers. NINT's collaborations with nanotechnology organizations are first discussed followed by its collaborations with researchers.
Collaborations with national and international nanotechnology research organizations
The evaluation found limited evidence to suggest that NINT engaged in much collaborative research with national or international research organizations. Below are a few examples of the research collaborations that did take place:
"We collaborate in a small degree. There is potential for more. The barrier is that there is strong collaboration at the individual level but none at the strategic level.”
- Internal interviewee
- National collaborations : NINT was involved in one project of the NRC-Natural Sciences and Engineering Research Council (NSERC)-Business Development Bank Canada (BDC) initiative, Technologies and Strategies for Assessment of Aquatic Toxicology of Manufactured Nanoparticles, in which it collaborated with other NRC institutes, industry, and universities across Canada. From 2007 until 2010, NRC had a program called NRC-Nano, coordinated by NINT, with a specific mandate to leverage NRC-wide research activities in nanotechnology. NINT did three NRC-Nano projects that were internal to NRC. NINT also worked with various former NRC institutes and current NRC portfolios Footnote 14 over the evaluation period.
Despite these examples of collaboration, there was ample evidence, including comments made by the PRC, to suggest that NINT had in fact not engaged in the number of national and international collaborations that would be expected of an institute with a national mandate and a goal of being internationally recognized.
As part of the bibliometric study, a network analysis based on co-publications was conducted to examine the relationships in the group of nanotechnology researchers to uncover the connections between NINT and these researchers. This study demonstrated that in Canada, NINT collaborated with very few organizations (mainly UofA; see Figure 11 in Appendix G: Additional tables and figures). NINT collaborations with top nanotechnology academic institutions, such as the University of Toronto, the University of British Columbia, the University of Waterloo and McMaster University, were almost non-existent. Given that these universities all performed strongly in nanotechnology research, this represents missed opportunities for NINT.
Evidence from interviews also indicated that regional collaborations did not frequently occur. Specifically, according to internal interviewees, collaboration with the Ingenuity Lab, located in the NINT building to promote collaboration with NINT researchers, was minimal.
Likewise, at the international level, collaboration with nanotechnology research centres was lacking. NINT did not appear to be involved with any of the top organizations in the world conducting research on nanotechnology. An internal interviewee at NINT highlighted that there are limited resources for NINT to collaborate with other nanotechnology research organizations.
Collaborations with national and international researchers
"Overall NINT appears to be an isolated community of researchers in the area of nanotechnology.”
- Peer review committee
In addition to collaborating with international and national research organizations, NINT worked with visiting researchers from Canada and abroad. NINT hosted an average of 103 visiting workers each year, who contributed regularly or intermittently to NINT's research programs. Despite this, the PRC concluded that NINT had supported a limited number of Canadian and international researchers. Other than direct collaborations with NINT/UofA researchers there were few Canadian and international researchers who developed strong links to NINT. The PRC concluded that NINT did not serve as a national research resource in this respect. All but one of the cross-appointees to NINT were from UofA (the single non-UofA project lead hailed from the University of Calgary). According to the PRC, the process whereby researchers become formally associated with NINT, as a cross-appointee or NINT Fellow, does not seem to encourage such relationships to develop. For example, there is no information on the NINT website on how Canadian researchers can collaborate on NINT projects. The PRC also noted that there was an absence of NINT-specific internship and visitor programs that promote Canadian and international exchanges of researchers and trainees.
4.3 Training of students
Similar to other national and international institutes considered in the Comparison Study conducted as part of the evaluation, NINT contributed to the development of Highly Qualified Personnel (HQP) by supporting the training of students. NINT supported approximately 180 students annually between 2008‑09 and 2013‑14. This included, on average, 118 undergraduate students and graduate students, and 61 post-doctoral fellows (PDFs) each year (see Table 7 in Appendix G: Additional tables and figures). The PRC concluded that the number of students supported at NINT was similar to that of competitive academic programs in North America. However, the PRC noted that there was little evidence of training HQP from universities other than UofA, beyond several visiting/exchange students. NINT indicated that while they have supported some undergraduate students from across Canada as co-op or summer students, the graduate students supported by NINT are largely from the UofA.
The PRC acknowledged that NINT trainees moved on to excellent PDF positions, good-to-excellent academic positions, and employment in a range of industry sectors, reflecting a quality training experience. However, they noted that other than access to state-of-the-art NINT facilities, equipment and professional staff, the value-added component to trainees provided by a NINT-associated degree experience, during the review period, was not apparent (e.g., special courses, training workshops, NINT seminars and conferences). Students did not gain NINT-specific credentials through NINT-specific, value-added non-curricular or academic experiences. The PRC felt that there was an opportunity for a NINT scholar program to provide enrichment for the scientific training of graduate students and PDFs, in order to enhance their ability in taking on leadership positions associated with nanotechnology in industry, government and academia. While NINT has been working on developing a NanoCertificate program at the graduate level, this had not been fully implemented at the time of the evaluation. Footnote 15
4.4 Scientific excellence
In order to assess NINT's scientific performance, the evaluation considered the research quality and scientific impact of NINT's work nationally and internationally based on its publications. The evaluation also considered the PRC's assessment of the extent to which NINT conducted leading-edge research and contributed to key knowledge in nanotechnology.
4.4.1 Research quality and scientific impact in the Canadian context
As part of the evaluation, a bibliometric study was conducted to assess NINT's research output, research quality, and scientific impact. In terms of research output, NINT published close to 900 papers between 2003-04 and 2013‑14, 71% of which were in the area of nanotechnology (majority of the other publications were either relevant to metabolomics research or material science). The number of publications in nanotechnology produced by NINT grew quickly between 2003-04 and 2009‑10, at which point the number of publications stabilized (see Figure 7, below).
The bibliometric study also demonstrated that the research quality and scientific impact of NINT's work varied depending on the program. According to average relative citations (ARC Footnote 16) , a measure of scientific impact for NINT's publications, as well as the average relative impact factor (ARIF)Footnote 17) , a measure of scientific quality, the bibliometric study demonstrated that NINT was a national leader in the scientific impact stemming from its research in metabolomics and the quality of its research in nanoelectronics. NINT, however, was not a national leader for quality or impact of its other research in nanotechnology overall or its other research areas (see Table 2, below).
Source: Bibliometric study
|NINT bibliometric statistics||NINT's ranking compared to top Canadian organizations Table 2 note 1|
|No. Publications||Research Quality (ARIF)||Scientific Impact (ARC)||No. Publications||Research Quality (ARIF)||Scientific Impact (ARC)|
|Energy generation and storage||92||1.14||1.09||15th||21st||35th|
Source: Bibliometric study
Note: Publications from nanoelectronics, energy generation and storage, and nanobiomaterials are included in the nanotechnology publications. The majority of publications in metabolomics fall outside of the nanotechnology publications.
- Table 2 note 1
Ranking is based on the top 35 to 41 Canadian organizations, depending on the research area.
Likewise, the bibliometric study indicated that when compared to Canada as whole, although NINT was much more specialized in nanotechnology overall and had higher research quality, its scientific impact was not comparably higher:
- Canada's specialization index Footnote 18 was 0.61 whereas NINT's was 6.43
- Canada's ARIF was 1.12 whereas NINT's was 1.20
- Canada's ARC was 1.14 whereas NINT's was 1.07
In Canada, out of the top 25 publishing organizations, the following five organizations had the highest scientific impact (based on publications) in the area of nanotechnology:
- University of Victoria (ARC – 1.30)
- University of Ottawa (ARC – 1.28)
- INRS Énergie, Matériaux et Télécommunications (ARC – 1.28)
- University of Toronto (ARC - 1.25)
4.4.2 Research quality and scientific impact in the international context
NINT's vision positioned it to be a global leader in nanotechnology. When measuring success based on scientific output, quality and impact, the bibliometric study found that NINT did not fall among the top 25 international organizations in nanotechnology, nanoelectronics, energy generation and storage, nanobiomaterials or metabolomics. The bibliometric study demonstrated that out of the top 25 publishing organizations internationally in nanotechnology, the top three with the highest scientific impact were: 1) Harvard University (ARC – 1.96); 2) Massachusetts Institute of Technology (ARC – 1.95); and 3) University of California, Berkeley (ARC – 1.84).
NINT's scientific performance was also compared to eight international institutes of a similar size and similar mid-ranking performance Footnote 19. As is the case when doing any comparison study, some differences exist in size, scope, and mandate of comparator institutes despite being chosen for their similarities with NINT. Compared to these institutes, NINT did not stand out as a leader (see Table 3, below).
|Number of publications||Scientific Impact (ARC)||Research Quality (ARIF)|
|Nanotechnology||4th (out of 9)||8th (out of 9)||7th (out of 9)|
|Nanoelectronics||5th (out of 9)||6th (out of 9)||6th (out of 9)|
|Energy generation and storage||4th (out of 9)||3rd (out of 4) Table 3 note 1||5th (out of 7) Table 3 note 1|
|Nanobiomaterials||3rd (out of 9)||3rd (out of 3) Table 3 note 1||5th (out of 5)Table 3 note 1|
|Metabolomics||2nd (out of 5) Table 3 note 1||N/C||N/C|
Source: Bibliometric study
Note: N/C = Statistics were not computable for all comparator institutes due to too few data points and as such, NINT could not be ranked
- Table 3 note 1
In some cases, statistics could not be computed for all institutes. NINTs ranking is relative to those that statistics could be computed.
4.4.3 Leading-edge research and key knowledge generation
Upon its review of NINT's programs, the PRC found that only a select number of NINT projects achieved leading-edge status, whereas others had only a national or regional status. The PRC concluded that isolation from and a lack of an appreciation for the international competitive scientific and pre-commercial landscape resulted in NINT's projects having varying degrees of leading-edge status grades. As a result, only a limited number of projects were said to be positioned to advance key knowledge in nanotechnology. The following paragraphs highlight the PRC's findings on the scientific performance of NINT's research areas.
- Nanoenabled biomaterials - The 2011 Strategic Plan stated that the main goal of the nanoenabled biomaterials program is to develop and commercialize a renewable platform for the efficient production of high value, carbon-based chemicals, materials and products utilizing Canada's biomass resource. Research in the area of nanoenabled biomaterials is conducted by both NINT and the Ingenuity Lab, which was created by the Province of Alberta in 2012 with $68.9M of funding over 7 years specifically as part of the Alberta nanotechnology strategy. As a result, the Director of the Ingenuity Lab was invited to participate as part of the NINT management team with the intent of aligning all activities between the two organizations in the area of bio-inspired materials. Consequently, for the last two years some of the NINT supported activities have taken place under the guidance of the Director. In return, the Ingenuity Lab makes use of NINT facilities on a cost recovery basis. Since it is the intent to continue to align and leverage these activities in the future we have included some of the Ingenuity Lab programs in the NRC evaluation. Because the information provided by NINT to the PRC included work conducted by NINT and the Ingenuity Lab during the evaluation period, it was not possible for the PRC to assess separately the NINT contribution to scientific excellence in this research area. Overall, the PRC concluded that this nanoenabled biomaterials program was not leading-edge. Still, the PRC considered that research in biomaterials itself could lead to strong opportunities for commercialisation that are not currently being tapped into by NINT.
4.5 Impact on growth and competitiveness
In order to assess the extent to which NINT contributed to the growth and competitiveness of nanotechnology enabled products and services in Canada, the evaluation looked at the technology transferred from NINT to industry, the impacts on clients who worked with NINT, as well as barriers to the commercialization of nanoenabled products and services.
4.5.1 Technology transfer
Between 2008‑09 and 2013‑14, NINT had four spin-off companies, three of which appeared to be in operation at the time of the evaluation (i.e., Intelligent Nano, Hy-Power Nano, Quantum Silicon Inc., and Carbonitum Energy Solutions Footnote 20). NINT is currently pursuing two additional spin-off ventures. The number of spin-offs from NINT was on par with the average number of spin-off companies reported by comparable institutes in the Comparison Study (i.e., six spin-off companies).
Between 2008‑09 and 2013‑14, NINT had eight licensing agreements with industrial companies (three were terminated and one expired). This is comparable to the six licences of one institute that reported on licenses in the Comparison Study. As one expert in the area of intellectual property highlighted, in an emerging technology area such as nanotechnology, it is often difficult to attract companies that want to license the technology. In this regard, the fact that NINT was able to successfully license eight technologies is commendable. Two business advisors also reinforced this message.
However, a few internal interviewees highlighted that when the number of inventions at NINT is considered (e.g., as represented by NINT's 37 patent applications between 2008‑09 and 2013‑14), there is some indication that additional opportunities to commercially exploit NINT's research may exist. The unexploited opportunities to transfer technology to industry also raise questions around the alignment and relevance of NINT's internal R&D with the needs of industry.
4.5.2 Outcomes and impacts
Interviews and case studies with NINT clients indicate that the Institute's research collaboration and fee-for-service work contributed to some benefits for the companies it worked with, including:
- New skills and knowledge (6 of 9 clients consulted cited this as a benefit).
In further support of the finding that new knowledge was generated, the bibliometric study indicated that NINT had co-published with seven companies and research organizations located in Edmonton and in Canada, including Applied Nanotools Inc., IntelligentNano Inc., Norcada Inc., the Canadian Neutron Beam Centre, Xerox, Merck and Alberta Innovates Technology Futures.
Table 4: Examples of products and services from NINT's work with industry
Source: Interviews, case studies, document review.
In some cases, products created or improved with NINT's support are currently available for sale on the market. For example, NINT reported that its collaboration with Hitachi High Technologies Canada Inc., on the design, development and initial testing of an environmental holder for Hitachi ultra-high resolution SEM resulted in this technology and expertise being transferred to Hitachi Canada. Hitachi Canada sales of the holder accessory have been in the order of approximately $1M per year (NINT Peer Review Material). It is worthwhile to note, however, that in the few instances where client interviewees reported increased sales (i.e., 2 of 9 clients interviewed) or employee growth (4 of 9 interviewed), they only attributed it in part to their work with NINT. In addition to the primary benefits, the 2011 ROI Analysis of NINT revealed that working with NINT yielded secondary benefits for clients, including: a 'seal of approval', changes to corporate strategies, and linkages and networking.
Despite the positive outcomes for some clients, there is little evidence to suggest that NINT contributed to the growth and competitiveness of nanotechnology enabled products and services in Canada to the extent that would be expected of a national institute of its maturity and with the level of funding it received. For instance, the 2011 ROI Analysis of NINT found that, at that time, most projects were in relatively early stages, more so than would be expected given NINT's stage of development, even taking into account the lengthy lead time that was required to get the Institute up and running. This study also concluded that the exact nature, size, and timing of impacts really were not yet known.
Representatives from UoA and GoA echoed the key message that NINT had not been successful at engaging with industry and had not had a significant impact on the growth and competitiveness of nanotechnology enabled products and services in Canada as would have been expected of a national institute of its maturity and with the level of funding it received. Likewise, the general conclusion from the NINT Council retreat held in 2013 was that, after 12 years, tangible results are primarily associated with the academic goals and there are very few innovation outcomes.
The general conclusions regarding NINT's overall contribution to the growth and competitiveness of nanotechnology enabled products and services in Canada may be due in part to the varying success of NINT's individual programs. To this effect, the PRC concluded that NINT's contribution to Canadian nanotechnology-enabled products and services varied greatly among the Institute's core programs. Although the translation of research developments to industry from the nanoelectronics, metabolomics and innovation support programs were not exemplary or outstanding, the PRC felt that they were very good (i.e., grade of B). This, however, was not the case for the remaining NINT programs. Contributions from the energy generation and storage program were found to be modest (i.e., grade of C), while contributions from the nanoenabled biomaterials program and the nanosensors portion of the metabolomics program were minimal to nil (i.e., grade of F).
4.5.3 Barriers to commercialization
Interviews with external clients and a review of documents revealed various barriers to NINT's ability to support the commercialization of nanoenabled products and services. These included:
- Relevance of NINT's work (e.g., low TRL research that may not always be of high interest to industry, outside of the companies that are already working with NINT)
Some of these barriers to the commercialization of nanoenabled products and services are not unique to NINT and faced by many jurisdictions. Findings from the Comparison Study conducted as part of the evaluation demonstrated that access to venture capital funding, availability of business development support, such as incubators, IP negotiations, and relevance of research to industry are common among similar institutes.
5.0 Resource Utilization
In order to assess NINT's utilization of resources, the evaluation considered: the availability and adequacy of resources; operational efficiency; and barriers to efficient operations.
5.1 Availability and adequacy of resources
The availability and adequacy of resources for NINT was examined by evaluating the extent to which it had the appropriate scientific and technical expertise, business development resources, financial resources as well as facilities and equipment to achieve its strategic objectives and expected outcomes. The evaluation also identified organizational barriers that affected NINT's optimization of its resources.
5.1.1 Scientific and technical expertise
"NINT offered a unique expertise that no one else is able to offer. My company was attracted to NINT for its scientific results and the scientific reputation of the microscopy laboratory as well as the staff's ability to transfer the research results."
-– Client interviewee
The vast majority of NINT's clients (8 of 9) were either very satisfied or satisfied with the scientific and technical expertise of NINT staff. Further evidence that NINT researchers have scientific expertise is found in the awards received by NINT researchers, such as those from the federal tri-council granting agencies (i.e., CIHR, NSERC, and SSHRC) in recognition of their work. For example, three NINT cross-appointees were awarded the prestigious Tier One Canada Research Chair. In addition to national recognition, NINT researchers received regional and international recognition. For instance, six cross-appointees were awarded the i-Core Chair, administered by the Alberta Innovates Technology Future (AITF). Examples of the international awards received by NINT researchers during the evaluation period included:
- American Physical Society Fellowship, Division of Condensed Matter Physics, American Physical Society Distinction (2012) - Bob Wolkow
The PRC concluded that while NINT technical staff operated at the quality level necessary to conduct research at the leading-edge, they had concerns whether some services, such as theory and modelling, or some projects within programs involved a sufficient number of HQP (staff and/or graduate students/PDFs) to sustain a research effort that could be competitive at international levels in terms of innovation and timeliness.
5.1.2 Business development resources
As part of NRC's restructuring to an RTO, it was evident that the business development mechanism across NRC (including NINT) was inadequate for the effective execution as an RTO. As such in late 2012, NRC restructured its business facing functions into the Business Management Support Branch which now includes client facing personnel, contract and IP management. As a result, at NRC & NINT all the Business Development Officers (BDO's) positions were eliminated in late 2012 (approximately 80 in total and including 4 at NINT). In the restructure, NINT as a sub group under the Security and Disruptive Technology Portfolio was supported by a Portfolio Business Advisor and three client relationship leaders (CRLs). One of the CRLs was based at NINT. A further CRL to support NINT was added in 2014. At this time, NINT also created the Industrial Innovation Support program. NINT staff assigned to this program worked with NRC Business Management Support (BMS) to develop industry partnerships. However, internal interviewees suggested that client engagement could be improved if the two groups were more coordinated and better integrated.
The evaluation also found evidence that clients had low satisfaction with NINT's overall business processes and the researchers' execution of the project. As is depicted below in Figure 8, NINT client interviewees reported low satisfaction with the efficiency and responsiveness of contract administration, ability to understand client needs, and timeliness of services and deliverables. At least one third (or 3 of 9) of NINT clients interviewed specifically cited challenges with business development support at NINT as the reason for their low rating and some recommended additional resources.
Source: Evaluation interviews and case studies
When compared to two other NRC portfolios, for which comparable data was available, NINT's clients were less satisfied on two dimensions:
"NINT does not appear to have dedicated enough human resources to deal with the reoccurring delays throughout the whole project. In my opinion, an exceptional business development officer needs to be present for the duration of the project to ensure its success"
– Client interviewee
- 56% of NINT clients were satisfied with the timeliness of services and deliverables compared to 86% of HHT clients and 68% of CONST clients
As noted above NINT has the potential to further expand its business by addressing issues around client satisfaction.
5.1.3 Financial resources
There is no evidence to suggest that NINT is lacking financial resources. NINT has increasingly had access to various sources of funding, with more financial resources available in 2013‑14 than in 2008‑09 (when all sources of funding are considered). As depicted below in Figure 9, NRC's financial contributions to NINT remained relatively stable from 2008‑09 to 2013‑14.
Revenue earned from collaborative R&D and fee-for-service projects provided an additional financial resource. Between $1.5M and $4M in earned revenue per year was available for NINT over the evaluation time period (see Figure 9, below). Given this significant amount of resources, the PRC concluded that NINT's outcomes were modest at best.
Source: NINT financial data
Apart from the base funding provided by NRC and earned revenues, NINT was able to draw on additional resources as a result of its partnership with UofA. However, because these resources were not directly under NINT's control, the exact amount that NINT benefited from remains estimated. Even so, NINT was able to provide an overview of the financial resources that it could have leveraged (see Figure 10 below).
Financial resources from UofA significantly increased from 2008‑09 (representing 16% total available financial resources) to 2013‑14 (representing 51% of total available financial resources). The increased funds available from UofA were due, in part, to the cross- appointees and NINT Fellows' ability to increasingly leverage awards and research grants from federal and provincial funding agents. Evidence suggests that NINT used some of these resources to support its projects. For instance, the wages of student researchers, who also worked on NINT projects, were sometimes paid with funding from research grants. Through NINT's affiliation with the university, NINT staff and researchers are able to access a variety of non-NINT nanotechnology facilities on campus.
In addition to UofA, GoA provided continued support to NINT indirectly, through the Ingenuity Lab and the nanoWorks program (although this program is now under review). The value of GoA's support increased significantly from 2012-13 to 2013‑14 as shown in Figure 10.
5.1.4 Facilities and equipment
"The quality of the equipment was one of the main reasons why [my company] chose to work with NINT. We would not have been able to access the same expertise and equipment anywhere else in Canada."
– Industry client interviewee
Between 2007 and 2010, UofA and GoA made a significant investment of $7.6M to acquire key infrastructure for nanofabrication, the electron microscopy centre and high performance computing at NINT. The PRC judged that the NINT equipment was at the level necessary to conduct leading-edge research. In particular, the electron microscopy centre was deemed to be world class. As well, both case studies and interviews with industry clients revealed that the quality and type of equipment/facilities satisfied their needs. Some of the equipment was even identified as leading-edge. Eighty-nine percent of client interviewees (or 8 of 9) gave a rating of four out of five or higher for their satisfaction with access to NINT's world class infrastructure and equipment. Case study clients, in particular, were very satisfied with the quality of infrastructure and equipment, with all three clients rating their satisfaction as five out of five.
While facilities and equipment were viewed as adequate by clients, NRC has invested relatively little in capital to replace NINT facilities and equipment in recent years. NRC's limited investment is noteworthy given that evaluation evidence suggests that some of the equipment may be aging. The PRC, for example, noted that for NINT to continue to operate at the leading-edge, it would require continual renewal, upgrading, and/or replacement. They expressed concerns given the size of the NINT's annual capital budget. In the event that infrastructure upgrades are considered, creating an environment that would allow for a large collaboration base should be a priority. Notwithstanding, the PRC considered that NINT had clearly benefited from the almost seamless interchange of equipment between NINT and UofA, which appeared to have partially compensated for the problem of aging instrumentation.
As discussed below, internal and external stakeholders indicated that access and pricing at NINT were problematic because of different policies at UofA nanoFab. Since NRC currently operates the building, NINT falls under access requirements established in the Treasury Board Operational Security Standard on Physical Security Footnote 22 for the federal government. The Standard indicates that departments must control access to restricted-access areas, which includes standard workplace operation zones, using safeguards that will only grant access to authorized personnel (e.g. locked doors accessible by card access; single entry point via a commissionaire) Interviews with internal and external stakeholders revealed that access requirements to NINT were perceived to be numerous, onerous and restrictive for UofA faculty and students, as well as NINT clients. Training on health and safety and equipment use, although necessary due to the valuable equipment at NINT, was also perceived as long and cumbersome by stakeholders. Furthermore, because NINT is a government building, users must obtain a federal government security clearance, which can take a long time to complete. Industry clients perceived the administrative requirements at NINT as more stringent or cumbersome than those at universities such as UofA nanoFAB.
Pricing has also limited client and researcher access to NINT's equipment. The costing models for the use of equipment differed between UofA (nanoFAB) and NRC. Only NINT included the recapitalization cost of the equipment in their rates to clients as part of a sustainable infrastructure initiative. This may explain why the rates charged by NINT were perceived by researchers and clients to be high.
5.2 Operational efficiency
In assessing operational efficiency, NINT's organizational model, use of resources, and newly implemented Project Management Office were considered.
All of NINT's partners benefited from the organizational model by leveraging financial and human resources, as well as facilities and equipment. For NRC specifically, some of these resources would not otherwise be available. For instance, NRC scientists are not eligible to apply for some research and infrastructure grants without adjunct professor status at a university. Without the partnership, NRC (a federal institution) would also not be able to leverage provincial government funding.
NRC has also had the benefit of accessing a large pool of human resources (scientists, students, PDFs from UoA) who are not only experts but also have the creativity commonly found among academics. Likewise, UofA and GoA benefit from the partnership with NRC because NRC brings experienced scientists and technicians, a strategic focus, the ability to employ larger scale teams, and provides centralized administrative and business management support. NRC has also benefitted from leveraging existing facilities and equipment at UofA. NRC has not had to invest money at NINT for major capital purchases because UofA has been able to purchase much of the equipment located at NINT using funds leveraged from grants and awards. However, interviewees commented that recently some of the new equipment acquired had been located at UofA and not at NINT. This may be a cause for concern if the equipment is not also made available to NINT researchers.
Use of resources
The evaluation found that NINT made efficient use of its human resources. For example, NINT's efficiency (indirect costs as a proportion of total NRC expenditures Footnote 23 ) was 44% in 2013‑14, which was below the NRC and NINT target of 47% Footnote 24 . In terms of productivity, it appears that NINT generated more revenue with less people in 2013‑14 as compared to 2012‑13 Footnote 25 .
However, the evaluation concluded that NINT's ability to generate revenue was lower than expected. NINT did not meet its revenue target of $3.5 M in 2012-13, or its target of $3.4 M in 2013‑14 Footnote 26 . However, the latter amount was adjusted to $2M to reflect current circumstances. The revised $2M goal for 2013‑14 was met. The suspension of the nanoWorks program, as discussed in Section 4.1, represents a risk to NINT's ability to generate revenue in the future.
Project management office
NINT has also made efforts to improve its operations. Specifically, in September 2013, NINT set up a Project Management Office (PMO) to provide processes and structure for managing projects through their lifecycle. The PMO is complemented by a new and improved project approval process that allows for better planning of how resources are used during the year. In addition, NRC recently provided NINT with some necessary tools, such as SAP, which is expected to further enhance the PMO's ability to track the use of resources. Although it is too early to tell what impact it may have on efficiency, the implementation of a PMO has been identified by NRC senior management as a best practice for NRC portfolios.
While NINT has an internal project review and approval process, the PRC noted that the process in place was not rigorous. The PRC further noted that research expenditures did not appear to have been aligned to a review process, and that external experts were not brought in to evaluate larger projects.
5.3 Barriers to Efficient Operations
Findings from the evaluation identified various barriers to efficient operations. These included challenges with: the clarity around the roles, responsibilities and accountabilities of key positions in the NINT governance structure; oversight from governing bodies; and mechanisms to ensure accountability. Each of these challenges is discussed below.
5.3.1 Roles, responsibilities and accountabilities in governance structure
"While good governance does not guarantee success, a confusing one will enable failure."
– Internal interviewee
A study of NINT's governance structure conducted internally in 2013 is consistent with the evaluation's observations that the institute has an appropriate governance framework (e.g. NINT Council) embodied in the current NINT governance structure Footnote 27. However, the evaluation found that the specific role of the NINT ED was unclear and the reporting relationship between the NINT research staff and the NINT ED was not optimal. During the evaluation period, the Director of Operations reported to the NINT ED and the Director of Research reported to the GM of the SDTech Portfolio Footnote 28. So while the NINT ED was accountable to and reported directly to the NINT Council on the performance and strategic direction of NINT, this position did not have direct oversight on the scientific research staff or the research direction of NINT. Interviewees and documents agree that this lack of clarity and the reporting relationship affected NINT's operational efficiency. The organizational chart has been changed as of April 2015.
Additionally, from 2008 to 2011 cross-appointees led scientific programs at NINT, aligned with their own research and supervised NRC staff assigned to their projects. In 2011, a matrix management structure was introduced and was fully implemented as of April 2015. This new structure allowed NRC to take on more of a leadership role for scientific programs as the leadership positions are associated with NRC personnel, except for the bio-materials program, given the overlap with the Ingenuity Lab. Group Leaders have an administrative role and the Project Coordinator's role is to manage the teams that work in a program. The PRC strongly endorsed the new structure.
The structure is a considerable shift from previous years and has led to some challenges with reporting and accountability procedures. For example, NRC Program Coordinators are accountable for the work conducted by cross-appointees, however, cross-appointees are UofA employees who are also accountable to UofA for their performance and may not strictly adhere to NRC policies and directives. Internal interviewees expressed frustration with regard to their limited amount of influence over the projects and people for which they are responsible for.
In addition to changes in the roles, responsibilities and accountabilities in the governance structure at NINT, there was a change in leadership at NINT over the evaluation time period. In 2012, a new ED was appointed to NINT by NINT Council in an effort to address challenges identified with the Institute's performance. The new ED was tasked with better defining NINT's strategy to ensure that it reflected the importance of R&D and engagement with industry.
5.3.2 Oversight from governance bodies
NINT's governance structure includes both the NINT Council and STAC. The NINT Council is the overarching governance body with equal representation of all partners. Although the Terms of Reference for NINT Council require that it meets at least three times a year, it decided not to meet in 2014 while discussions on the appropriateness of the NINT organizational model were held. While the Council meetings were on hiatus, bilateral meetings between UofA and NRC took place as per the decision of the NINT Council. The goal of these bilateral meetings was to address the challenges of the different objectives and cultures of the partners. The Council meetings have resumed as of 2015.
In addition to the NINT Council, NINT has a scientific advisory board in place – STAC. The role of STAC is to provide strategic advice on the R&D programs at NINT, be it from a scientific or business perspective (e.g. Industrial need and potential impact). STAC's Terms of Reference highlight that it could, for example, evaluate current competencies and suggest what gaps need to be filled to be competitive in the field. The Terms of Reference also state that the members could undertake an evaluation of one of the programs in greater depth and make recommendations on objectives and outcomes. While STAC did convene twice for the first time in 2013, it did not meet in 2014, despite the requirement to meet biannually. The evaluation found that NINT did not use STAC effectively, and as such, STAC did not deliver on its Terms of Reference. The evaluation also identified opportunities to expand the mandate of STAC. The PRC, for example, encouraged the participation of external members to NINT in the selection of projects, indicating that members of STAC may be appropriate for this task.
5.3.3 Mechanisms to ensure accountability
The lack of comprehensive accounting practices at NINT, originally highlighted in the 2011 NINT Strategic Plan, continues to be a major challenge today. The evaluation team and the PRC found it very difficult to develop a comprehensive understanding of NINT's resources over the reporting period from the information that was provided. It was not possible to determine NINT's total inputs from UofA and GoA leveraged funds. These funds were not under NINT's direct control nor were they tracked by NINT. In addition, much of the financial information provided to both the evaluation team and the PRC often differed depending on the source, making it difficult to reconcile. Likewise, NINT's ability to provide complete data on its visiting workers and students, including the amount of time they spent working at NINT, highlighted challenges with the approach/systems in place.
While NRC provided NINT with access to corporate financial and project management software, there were challenges with fully capturing all financial and human resource information within these systems, such as funds from grants and awards used on specific projects or time spent on in-kind projects and by visiting workers. Such contributions can sometimes represent up to half of the resources of a project, resulting in a system that lacks the required features for NINT's situation. The limitations of these systems further support the need for a more comprehensive accounting system.
In addition to challenges with accounting for NINT's financial and human resources, assessing NINT's value-for-money proved difficult due to limited and/or fragmented performance data for each of NINT's programs. The PRC specifically commented that they found it very challenging to develop a comprehensive understanding of NINT activities and outcomes over the reporting period from the information that was provided about the programs in advance of the site-visit and during the site-visit.
While NINT has recently taken steps to improve its performance reporting by implementing an Annual Report, the PRC recommended that a performance measurement approach be strictly adhered to and that all of the information and data required for a comprehensive evaluation should be routinely collected, collated and mapped to NINT's objectives.
Overall, the evaluation found that there is a need for clients to access the type of services and equipment offered by NINT. However, aside from NINT's electron microscopy facility, specialised services and dedicated technicians, there are other facilities in Canada that offer similar services and facilities. The stated goals of NINT are in line with the role and responsibilities of the federal government and the NRC strategy. However, the evaluation brought to light a number of challenges that have resulted in NINT not achieving many of the goals. These challenges include deficiencies in project selection processes and the overall organizational/operational model.
Despite NINT's national mandate, most of NINT's industrial engagement was limited to Alberta. The return on investment in NINT is at best modest when considering NINT's scientific output and impact, the extent of industry engagement and support, and national and international leadership position. Still, NINT has the scientific and technical expertise as well as the financial resources to conduct leading-edge research. While NINT's organizational model was originally designed to capture the strengths of an academic institution and a federal lab, findings from the evaluation point towards shortcomings in its success. Both NINT and the NINT governing council have taken steps to address challenges with NINT's performance and some of the changes introduced had only been implemented for two years or less at the time this evaluation was conducted. A number of areas for further improvement were identified in the evaluation. Six recommendations resulted from the evaluation, all of which were directed to NRC to work with its partners to address the issues found in the evaluation.
7.0 Management Response
|Recommendation||Response and planned action(s)||Proposed person(s) responsibilities||Timelines||Measure(s) of achievement|
The NINT Council has recognized the need to reconsider the NINT model and has taken action to define a new model to ensure NINT will be more national and international in reach, engaging with NRC and university researchers from across Canada and with international collaborators.
Vice-President Emerging Technologies
March 31, 2016
|New NINT model developed and ready to be implemented, ensuring NINT establishes research collaborations with national and international partners.|
March 31, 2017
NINT has started to develop national and international research collaborations.
As part of the new NINT model, NRC will work with NINT participants to train research and technology innovators and entrepreneurs in order to grow a larger pool of nano-trained HQP for Canadian companies and universities.
Vice-President Emerging Technologies
March 31 2016
New NINT model developed and ready to be implemented, including a training component for research and technology innovators and entrepreneurs.
March 31 2017
A training component has been development and is starting to be implemented.
NRC will continue to build on the recent actions undertaken to set up a Project Management Office, define a new project approval process and use the tools necessary to track the use of resources in line with the best practices already implemented across NRC.
Projects undertaken in the context of the new NINT model will be conducted under standardized ‘'terms of reference'', including a defined scope, milestones, deliverables, and duration of the project. A rigorous peer and business review process will be put in place to evaluate progress of research projects against the pre-defined terms of reference.
Vice-President Emerging Technologies
March 31 2017
A rigorous peer and business review process is developed and implemented as part of the new NINT
NRC will monitor the effectiveness of the matrix approach to ensure operational efficiency during the transition to the new NINT model.
As part of defining the governance model for the new NINT, NRC will work with the NINT partners to ensure clarity of roles, responsibilities and accountabilities of key positions.
Vice-President Emerging Technologies
March 31 2016
New NINT model implemented with a governance structure clearly defining roles and responsibilities of key positions.
Recommendation partially accepted
The NINT Council will continue to meet during the NINT transition.
As part of the new governance model for NINT, NRC and the NINT partners will ensure that the frequency of meetings for each governance elements is clearly defined.
Vice-President Emerging Technologies
March 31 2016
New NINT governance clearly defines the frequency of meetings for each of the governance components.
The governance for the new NINT will provide for the creation of an independent committee mainly composed of members from the nanotechnology community to evaluate progress of research projects against the planned scope, milestones and deliverables and perform periodic reviews of research projects.
Vice-President Emerging Technologies
March 31 2017
Committee has been established as part of the new NINT governance with specific roles regarding projects and programs selection and assessment of projects progress against plans.
NRC will work with the NINT partners to develop the required systems and approach for the new NINT.
Independent operating systems will be developed and implemented for the new NINT.
Vice-President Emerging Technologies
New NINT Management
March 31 2017
The new NINT has operating systems in place providing comprehensive accounting information.
The Performance management system will be developed according to Treasury Board guidelines and be established in the NRC contribution agreement to the new NINT.
Vice-President Emerging Technologies in collaboration with the NINT partners.
March 31 2016
Performance management system developed and included in the Treasury Board submission.
March 31 2017
NINT track the performance measures and use them for performance management.
Appendix A: Evaluation matrix
|Document & literature review||Performance data review and analysis||Key informant interviews||Bibliometric study||Peer-review||Impact case studies||Comparison study|
|R1. Continued need for Program|
|1. Does NINT continue to address a demonstrable need in support of Canadian industries with respect to nanotechnology enabled products and services? What is the role of NINT in supporting Canadian and International scientists and students?||✓||✓||✓||-||✓||✓||✓|
|R2. Alignment with Government priorities and R3. Alignment with federal roles and responsibilities|
|2. Are the strategic objectives and activities of NINT aligned with the expectations and roles of NRC?||✓||✓||✓||-||-||✓||-|
|P.1 Achievement of Program outcomes|
|3. Has NINT been successful at engaging with R&D performers and stakeholders nationally and internationally?||✓||✓||✓||✓||✓||-||✓|
|4. To what extent has NINT conducted research at the leading edge in the field and has had an impact on advancing key knowledge on nanotechnology?||✓||✓||✓||✓||✓||✓||✓|
|5. To what extent has NINT contributed to the development of HQP in Canada?||✓||✓||✓||-||✓||✓||-|
|6. To what extent has NINT contributed to the growth and competitiveness of nanotechnology enabled products and services in Canada?||✓||✓||✓||-||-||✓||✓|
|P2. Resource utilization|
|7. To what extent has NINT acquired the appropriate resources to achieve its strategic objectives and expected outcomes? In what ways could NINT's resources be optimized?||✓||✓||✓||-||✓||✓||-|
8. Is NINT administered in an efficient manner?
Appendix B: Methodology
The evaluation approach and selection of methods was based upon the information needs of NRC Senior Management to support timely decision making as well as the complexity of the NINT model (i.e., partnership between NRC, UofA and GoA). In order to maximize the possibility of generating useful, valid and relevant evaluation findings, mixed methods were used, allowing for triangulation (i.e., convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different facets of a complex issue).
Both qualitative and quantitative methods were used, and included:
- Peer review
A discussion of the approach used for each of these methods, including any limitations and challenges, is provided in the following paragraphs.
Internal and external document review
Internal and external documents were reviewed, synthesized and integrated into the evaluation to provide context and history, and to complement other lines of evidence in assessing relevance and performance. Internal documents reviewed included strategic and business plans for the Institute, special studies, and the program profiles prepared by NINT for the Peer Review Committee. In addition, a wide range of external documentation was also reviewed by the evaluation team. A selected list of the documents reviewed can be found in Appendix E.
Administrative and performance data review
Administrative and performance data for 2008‑09 and 2013 14 were reviewed to provide information on program outputs and client reach, as well as to contribute to the analysis of resource utilization (e.g., staff utilization rates). Administrative and performance data were provided by NINT, as well as by NRC corporate branches including Business Management Support (BMS), Finance, Human Resources, and Planning and Reporting Services.
Challenges were faced in obtaining administrative and performance data from NINT in a timely fashion and in cases where the data were provided they were difficult to understand and reconcile with other sources of information. A notable amount of time was spent working with NINT to transform the data so that they were valid and reliable to use in the evaluation. The difficulty encountered acquiring financial data that reflected all partner contributions (i.e., NRC, UofA and GoA) resulted in an incomplete representation of NINT's financial resources (and consequently lead to a recommendation to improve accounting practices). The unavailability of certain data (e.g., equipment usage, visiting worker's time spent at NINT, current status of students trained at NINT) meant that the evaluation team was not able to assess certain issues as in-depth as originally planned.
Conducting interviews with key informants is an essential element of an evaluation methodology. The information gathered through the qualitative, semi-structured interview process was based on personal experiences, opinions and expert knowledge. This information plays an important role in contextualizing performance data and other statistics.
Interviewees were selected in consultation with NINT management and NINT's client list. Interviews were conducted either in-person or by telephone. Each interview lasted between one and two hours and was conducted using an interview guide.
A total of 42 stakeholders were consulted through the evaluation, including 20 internal stakeholders (e.g., NINT management and staff, BMS staff, IRAP Industrial Technology Advisors), 12 external stakeholders and 10 stakeholders from the NINT partners (e.g., UofA and GoA). External stakeholders included active clients (defined as those who had had a project with NINT over the evaluation time period) and stakeholders from other organizations (e.g., Western Economic Diversification of Canada; Alberta Centre for Advanced MNT Products; Corning West Technology Centre Science & Technology).
|Interviewee category||Number of individuals interviewed|
Three case studies were conducted to illustrate NINT's work with clients. The case studies assessed the extent to which client needs had been met and the impacts resulting from the project (s). The case studies were selected through a review of NINT's client list and in consultation with NINT management. The three case studies selected for the evaluation profiled NINT's relationships with:
- Hitachi High Technologies Canada Inc.
These three case studies were selected because they represented NINT's largest and most successful collaborative R&D projects.
The case studies were developed through reviews of project documentation, available external documentation (e.g., newspaper articles discussing the project or its impacts), and interviews with both NINT project staff and external client representatives. Semi-structured interview guides were developed. The NINT Executive Director contacted the companies selected for the case studies to inform them of the evaluation and to advise them that they may be contacted to participate. The case studies were developed using a common template and drafts were shared with internal and external interviewees for factual validation prior to their completion.
The OAE commissioned Science-Metrix to assess the scientific output of NINT in research relevant to nanotechnology and a group of subsets aligned with the four programs at NINT (i.e., energy storage and production, nanobiomaterials, nanoelectronics and metabolomics). As part of the bibliometric study, NINT's performance was also compared to a selection of comparable organizations, which were selected in consultation with the NINT Executive Director and the EAC. The comparable organizations included:
- Waterloo Institute of Nanotechnology (University of Waterloo, Canada)
The list of journals and key word searches used to compile the publication database for the bibliometric study were informed by an expert panel of researchers. The experts validated the lists of journals and key words for nanotechnology and each of the four subsets, as well as suggested new additions that were incorporated when deemed relevant. The NINT Executive Director was also given the opportunity to comment on the list of journals and key words used in the study.
A comparison study was conducted to provide insight on how comparable national and international institutes bring nanotechnologies to market by looking at their commercialization strategies as well as their barriers and enablers to commercialization. Tombstone data were also collected on the institutes, as well as performance data and details on their respective national nanostrategies (if applicable).
The sample of institutes was chosen for their similarity to NINT in terms of resources and mandate and was done in consultation with the NINT Executive Director and the EAC. Of nine institutes that were contacted to participate in the study, four agreed (44%). These four institutes are:
- The Nanometer Structure Consortium (Sweden)
Web-based searches were used to first gather publicly available information on each of the comparators. Participating institutes were then asked to validate the information and/or to fill in missing information. This was followed up with a semi-structured interview.
In order to elicit participation in the study, representatives from each of the comparator institutes were contacted by the NINT Executive Director via email informing them of the evaluation and the Comparison Study. Despite this strategy, there was limited participation (i.e., 4 out of 9). Given that common themes emerged with the sample of four institutes, findings from this study were deemed reliable. Findings from this method were also triangulated with findings from other lines of evidence.
An International Peer Review Committee (PRC) was convened in Edmonton, Alberta at NINT April 13-15, 2015 to assess NINT's past performance. The Peer Review Committee was composed of eight Canadian and international experts, with expertise in each of NINT's main research areas. Potential Committee members were identified and validated through a variety of sources (e.g., NRC, NINT, other subject matter experts) to ensure they had the breadth of knowledge to comment on each of NINT's areas, as well as identify potential biases or conflicts of interest. The Committee membership is listed in Appendix E. The NRC Office of Audit and Evaluation (OAE) invited the Committee members and acted as secretariat to the Peer Review Committee throughout the peer review process.
The PRC was provided a selection of key documents to review (e.g., NINT's strategic plans, program profiles prepared by NINT for each of its four research programs, preliminary findings of the evaluation, findings from the Bibliometric Study conducted as part of the evaluation). These were during a conference call prior to the site visit. The peer review site visit took place over the course of one day and included a series of presentations by NINT staff to provide the Committee with the necessary information in order to respond to the questions posed to them (listed in Table 6, below). Committee members had the opportunity to ask for additional information throughout the site visit, and the Chair debriefed the NINT Executive Director on the last day.
Following the site visit, the PRC produced a report of their conclusions and recommendations, which was reviewed and endorsed by all members of the Committee and reviewed for factual accuracy by NINT management. It was then integrated into the evaluation material by OAE to produce the final evaluation report.
One of the challenges faced in conducting the peer review was securing a Chair, which required the evaluation timeline to be extended. Challenges were also faced in obtaining the necessary information for the PRC to conduct their assessment of NINT at the program level (due in part to NINT's reporting practices, where information is not tracked by program). In some cases, information was not available for the Peer Review Committee and resulted in their inability to render an assessment of NINT's performance.
Table 6: NINT peer review questions
- To what extent has NINT conducted research that is leading edge?
To what extent has NINT conducted research that has had an impact on advancing key knowledge in nanotechnology?
- To what extent does NINT have the appropriate HQP to conduct leading edge research?
- To what extent has NINT contributed to the growth and competitiveness of nanotechnology enabled products and services in Canada?
Appendix C: Logic model
Appendix D: Stakeholders consulted
- National Research Council (NRC)
International Nanotechnology Research Facilities
- Waterloo Institute for Nanotechnology (WIN)
- NIST – CNST
- Nanometer Structure Consortium – Lund, Sweden
- Molecular Foundry
External Canadian Organizations
- ACAMP (Alberta Center for MNT products)
- Corning West Technology Centre Science & Technology
- Western Economic Diversification of Canada
- Government of Alberta
- Alberta Research Chemicals Inc.
- Kemira Chemicals
- University of Alberta
- Norcada Inc.
- Jet lube of Canada
- Exciton Technologies Inc.
- Xerox Canada
- Hitachi Canada
- Canmet - NRCan
Appendix E: Peer review committee membership
The NINT Peer Review Committee was comprised of the following members:
Dr. Bruce Lennox
Tomlinson Professor, Chemistry
Dr. Mark Reed (NINT program area: Nanoelectronics)
Professor, Electrical Engineering & Applied Science
Dr. Ulrich Krull (NINT program area: Nanosensors and Metabolomics)
Professor, Chemistry - Biological & Bioanalytical Chemistry
University of Toronto
Dr. Warren Chan (NINT program area: Nanosensors and Metabolomics)
Professor and CRC Tier I in Biomaterials & Biomedical Engineering
University of Toronto
Dr. Marya Lieberman (NINT program area: Nanoelectronics)
Associate Professor, Chemistry
University of Notre Dame
Dr. Guojun Liu (NINT program area: Nanoenabled biomaterials and Innovation Support)
Professor & CRC Tier I in Materials Science
Dr. David Ginley (NINT program area: Energy generation and storage)
Senior Researcher and Manager in Energy
National Renewable Energy Laboratory,-Colorado
Dr. George Demopoulos (NINT program area: Energy generation and storage)
Professor, Materials Engineering
Appendix F: Selection of documents reviewed
- Statistics Canada. (2007). Overview and Discussion of the Results of the Pilot Survey on Nanotechnology in Canada Science, Innovation and Electronic Information Division.
Appendix G: Additional tables and figures
|Core staff||Administrative staff||25||28||28||29||27Table 7 note 1||27|
|Scientific staff||NRC scientific staff (RO/RCO classification)||33||37||37||37||34||30|
|U of A cross appointees||18||16||17||17||17||19|
|Technical staff (TO classification)||17||22||24||26||23||21|
|Students (undergraduate and graduate)||94||98||114||133||136||131|
Source: NINT Human resources data
- Table 7 note 1
Administrative services began to be centralized at NRC in 2012‑13. These numbers include the common services staff at NRC corporate offices
Note: The number of papers is reflected by the size of the bubble, the number of collaboration is reflected by the width of the link and a community indicator identifying communities is reflected by the colour of the bubble.
- Footnote 1
NRC plays a unique role in this initiative because NINT is not a stand-alone legal entity. NRC acts as the operator of the NINT joint initiative and, as a result, is responsible for entering into all contractual, employee and banking arrangements for its operation.
- Footnote 2
The reporting structure was changed and as of April 2015 the Director of Research now reports to the NINT ED.
- Footnote 3
Human resource figures are based on head counts.
- Footnote 4
The visiting workers category includes secondments, research assistants and research associates from Canada and international countries. Visiting worker summary statistics were provided by NINT. Based on available data, it is not known the extent to which visiting workers were involved in projects at NINT and the breakdown of which countries they were from.
- Footnote 5
http://ec.europa.eu/environment/chemicals/nanotech/index_en.htm; retrieved on June 1, 2015.
- Footnote 6
Schiffauerova and Beaudry. (2009). Canadian Nanotechnology Innovation Networks: Intra-cluster, Intercluster and Foreign Collaboration
- Footnote 7
http://cientifica.com/wp-content/uploads/downloads/2011/07/Global-Nanotechnology-Funding-Report-2011.pdf; retrieved on June 1, 2015.
- Footnote 8
OECD. (2009). Nanotechnology: an Overview Based on Indicators and Statistics.
- Footnote 9
OECD. (2009). Nanotechnology: an Overview Based on Indicators and Statistics.
- Footnote 10
The 2007 Alberta Nanotechnology Strategy was the most up to date publicly available version at the time of the evaluation.
- Footnote 11
Dennis Rank and Associates and J.E. Halliwell Associates Inc. (2011). Return on Investment Analysis of NINT
- Footnote 12
Memorandum of Understanding Regarding an Action Plan to Establish a National Nanotechnology Network "NanoCanada" (September 15, 2013).
- Footnote 13
National Research Council Act (R.S.C.,1985, C.N-15)
- Footnote 14
NINT was involved with the following current portfolios and former NRC institutes: Automotive and Surface Transportation (AST), Measurement Science and Standards (MSS) and Institute for Microstructural Sciences (IMS), Institute for Chemical Process and Environmental Technology (ICPET), Biotechnology Research Institute (BRI), Institute for Biological Sciences (IBS) and Steacie Institute for Molecular Sciences (SIMS).
- Footnote 15
The core course, NANO 500, was approved by the Executive committee of General Faculties Council for the 2014-2015 calendar and the Certificate program is pending approval by a UofA committee (NINT Annual Report: June 2014)
- Footnote 16
The ARC indicates the observed scientific impact of research conducted by an entity, based on an average of the number of citations that each of its papers received, relative to the average number of citations received by world papers published the same year, in the same specialty. An ARC > 1 means that the entity's research is more cited than the average world research.
- Footnote 17
The ARIF indicates the quality of research conducted by an entity, based on an average of the impact factors of the journals in which its papers are published, relative to the average of impact factors by world papers published the same year and in the same specialty. An ARIF > 1 means that the entity's research is published in journals cited more frequently than the average journal.
- Footnote 18
The specialization index measures the intensity of research of an entity in a given field relative to the intensity of the world in the same field.
- Footnote 19
The comparator institutes, which were chosen for their similarity to NINT in terms of resources and mandate and was done in consultation with the NINT Executive Director and the EAC, included: University of California Santa Barbara (USA); Institute of Nanotechnology (at Karlsruhe Institute of Technology; Germany) California Nanosystems Institute (USA); Colleges of Nanoscale Science and Engineering (USA); Molecular Foundry (USA); Centre for Nanoscale Science and Technology (USA); Nanometer Structure Consortium (Sweden); and Waterloo Institute of Nanotechnology (Canada).
- Footnote 20
Carbonitum Energy Solutions does not appear to be currently in business.
- Footnote 21
Note both of these portfolios are further along the TRL scale than NINT. Comparable client satisfaction data was not available for portfolios within the NRC Emerging Technologies Division.
- Footnote 22
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12329#Restricted; retrieved September 1, 2015
- Footnote 23
Based on data from the NRC Statement of Operations
- Footnote 24
Target for NINT is for 2014‑15, as previously NINT financials were not reported separate from the SDTech Portfolio. It is important to note that NINT would generally be expected to have low overhead, as they do not pay of O&M, rent, or administrative support. Administrative support is provided by NRC from its centralized services budget.
- Footnote 25
Productivity is based on revenue per core scientific staff, including NRC RO/RCOs, UofA cross appointees, and technical officers (administration, students, visiting workers, and post-doctoral fellows were excluded).
- Footnote 26
Achievement of revenue targets could not be calculated prior to 2012‑13 because NRC financial statements (Utilization of Resources Reports) did not track revenue targets.
- Footnote 27
The NINT governance structure was previously described in more detail under the Profile section of this report.
- Footnote 28
The reporting structure was changed and as of April 2015 the Director of Research now reports to the NINT ED.
Report a problem or mistake on this page
- Date modified: