ARCHIVED - Impact Evaluation of the NRC Industrial Research Assistance Program (NRC-IRAP)
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
The following is a summary of the findings stemming from the evaluation of the National Research Council's Industrial Research Assistance Program (NRC-IRAP). NRC-IRAP is a federally funded contribution program designed to assist small and medium-sized enterprises (SMEs) in improving their innovation capacity. It provides both advisory services and non-repayable contributions to Canadian firms, and supports the broader innovation system by funding relevant organizations.
An evaluation of NRC-IRAP takes place every five years and is undertaken as a requirement under the recently introduced Federal Accountability Act. As a federally funded grant and contribution program, it must undergo review every five years.
In October 2006, NRC's Senior Executive Committee (SEC) approved Terms of Reference for the evaluation of the Program. The document outlined that the evaluation would seek to:
- understand and assess NRC-IRAP's current relevance and impact, as well as its value for money;
- provide input to decisions related to the renewal of the Program's terms and conditions (e.g. funding renewal); and,
- to provide NRC-IRAP management with information that may contribute to improved results achievement in the future.
The evaluation examined NRC-IRAP's activity and impact from 2002-03 to 2006-07. The evaluation considered NRC-IRAP activity only and excluded the evaluation of NRC-IRAP's delivery of Technology Partnerships Canada (TPC), and the Youth Employment Strategy (YES).
Planning and Performance Management Directorate (PPM) of the Strategy and Development Branch (SDB), Corporate Services at NRC both managed and conducted the evaluation. However, the firm Goss Gilroy Inc., an Ottawa-based consultancy specializing in evaluation, participated in fieldwork and the analysis of results.
The evaluation was conducted over a 12-month period. Information to address questions posed by the evaluation was gathered using: internal and external document review; administrative and performance data review; in-depth interviews (approximately 120); focus groups (a total of 14 held with SME clients or NRC-IRAP Industrial Technology Advisors (ITAs)); and electronic surveys with SME clients, both funded and non-funded, and NRC-IRAP ITAs.
In addition to the methods employed above, the evaluation team undertook two additional sub-projects including a pilot of Treasury Board's Value for Money Profile tool and, socio-economic impact analyses including: cost-benefit analysis (CBA); input-output analysis; econometric analysis; risk and sensitivity analysis; and, comparative analysis.
All evaluative work was overseen by an eleven member Expert Review Panel convened for the evaluation.
Summary of Evaluation Findings
The following provides a summary of key findings stemming from the evaluation.
The Program continues to experience a decline in its client reach, both in terms of number of firms receiving funding and number of projects, a trend that it has experienced for the last 15 years. The decline is, however counter-balanced by an increase in project values as the Program is supporting fewer firms with larger contribution amounts. If the intent of the Program is to have as great an economic impact in the most effective manner, potentially by providing firms with higher amounts of money, or by funding more than a single project, a reduction in the number of SMEs that the Program interacts with should not be viewed negatively.
Potential factors influencing client reach, either in the past five years or potentially into the future, include:
- a decision by the Program that it would focus on building relationships with clients over time. Therefore, the Program moved to supporting firms, rather than simply supporting projects within firms.
- any systemic change among SMEs in Canada in general (i.e., decrease in the number of manufacturing firms, changes in the competitiveness of primary industries such as forestry and fisheries, increasing competitive pressures from countries such as China and India);
- a reduction in the number of ITAs delivering the Program in the field, which may affect the ability to identify new opportunities, and general program access;
- increasing demands in the areas of public accountability, due diligence or project monitoring and reporting, which have diminished the time available to ITAs to develop new relationships or new projects; and,
- a view by clients that accessing the Program is now more difficult and burdensome.
SME Innovation Capacity
The evaluation surveyed and questioned firms about the impact of the Program on their innovation capacity. It found that the R&D capacity and capabilities of NRC-IRAP clients grew over the evaluation period and NRC-IRAP clients have on average greater capabilities and capacity than non-client SMEs. Further, the infusion of NRC-IRAP funds has the complimentary effect of inducing firms to increase their spending on R&D.
Of the support provided by the Program, management, marketing and financial advisory services were the most accessed by clients. On average, the capabilities of firms grew in all three of these areas following receipt of NRC-IRAP assistance.
In terms of firm growth, analysis suggests that NRC-IRAP client firms have experienced growth, both in real and comparative terms. Over the five year evaluation period client firm sales grew on average by just over 28%, overall firm employment grew by 30%, and company assets grew on average by 15%.
The extent to which NRC-IRAP stimulates wealth creation within Canada is illustrated in the overall net socio-economic benefits that it generates. By aggregating the impacts resulting from individual client firm sales growth and cost reductions as well as the value of the advisory services provided to Canadian SMEs, it was estimated that, between 2002 and 2007, the total wealth creation benefits of the Program were in the range of $2.3 and $6.5 billion.
By contributing to the development of research and development (R&D) capacity, the overall growth of SMEs, the commercialization of new products, services and process, as well as the creation of new knowledge and intellectual property, NRC-IRAP has positively stimulated overall innovation in Canadian SMEs and in Canada as a whole.
Community Innovation Capacity
A review of projects supported by NRC-IRAP regions identified three primary types of services offered as part of any given regional orientation, including: services addressing the shared needs of firms or a group of firms; direct services to individual SMEs; and, the provision of access to foreign knowledge and networks.
The majority of activities funded through contribution agreements with organizations has been in support of networking and business capacity development. The investments made by NRC-IRAP, as a publicly funded program, when combined with the resources of other sources, both government and non, appear to support an increase in the range and number of services available for SMEs at a lower cost than if the Program were to attempt to deliver these on its own. Moreover, the importance of Innovation Network Advisors (INA) and the ITA in connecting innovation players was identified.
According to the views of funded organizations, ITAs and other NRC-IRAP staff, the Program has a measured impact on the innovation system. The logic that underpins support to organizations is that an increase in capacity within the SME environment will have an impact on the performance of firms.
Although evaluation participants were positive about the contributions of the Program to the innovation system, the evaluation was not able to substantiate, in quantitative terms, the impact of these investments on the performance of SMEs.
Competitive Technical Intelligence and Strategic Information
The use of competitive technical intelligence (CTI) has increased as NRC-IRAP has now expanded its support to NRC-CISTI to deliver specific CTI services in three of NRC-IRAP's five regions. Funded organizations, in some instances, may also provide similar types of information. Firms who reported receiving CTI and related 'strategic information' support generally rated it as having provided high value. SMEs are using CTI delivered by NRC-CISTI and other types of strategic information to support business and technical decision-making. They identified reports and information received in such products as being of real relevance.
Targeted Value for Money
The evaluation examined the economy, efficiency and cost-effectiveness of the Program. It identified that there are opportunities for the Program to improve its performance in the area of resource utilization or economy. Program costs (direct, indirect and corporate overhead) represented just over a third of total program expenditures. Direct program costs, although they have remained relatively constant since 2003-04, in 2006-07 were at 17.2% of total program costs. These costs are those associated with delivery of funding and advice by ITAs.
In 2006-07, the ratio of field staff was at approximately one half non-field person for every field staff. If analysis is done using ITAs only the ratio is higher at approximately two thirds of a management or support person for every ITA.
Given that the Program's administrative costs are high and that the ratio of program delivery staff to administrative staff is also high, some consideration may be given by the Program to reviewing the delivery approaches used by other comparable programs. It may also look to determine whether there are means to reduce the current administrative pressures on program staff. This would be done with the expectation that the number of ITAs could once again be increased to previous levels without the need to change substantially the level of regional support resources needed to facilitate their work.
In terms of program efficiency, a number of output metrics for the Program show decreases, such as the number of actively funded firms and the number of new projects. These have been counter-balanced by an increase in project values. However, the number of firms who receive advice only has also decreased.
Ratios of ITA to client reach activity show decreases with ITAs handling fewer clients and fewer projects, although again projects are of greater value. Reasons for these drops include: fewer ITAs delivering the Program's resources resulting in fewer client relationships; ITAs are having to absorb changes such as a generalized increase in project complexity and size; and, the need to manage larger investment risks.
Over the past five years, a variety of audits has required that the Program make a number of process changes to deal with elements of risk. There is evidence to suggest that these increased accountability, monitoring and performance requirements are having an impact on the Program's ability to deliver funding and advice in an efficient manner, which may be contributing to the drop in program reach as well as the ability of the Program to serve particular types of clients (e.g., start up firms, small firms) or projects of relative risk.
The Program's efficiency has also been influenced by changes in available budgets from one fiscal year to the next and the fact that NRC-IRAP has been subject to numerous changes in available resources over the past five years. Although the total amount of available resources given out by the Program to firms has on average not changed during the evaluation period, any change on an annual basis in budgets can influence the Program's ability to expend all moneys in that year. Overall, the evaluation identified that uneven levels of funding, or clarity about the level of contribution dollars available for SME projects, can influence the general efficiency of the Program, the level of program reach and the satisfaction level of both staff and firms with the Program.
In the area of cost-effectiveness, the evaluation examined, in a limited manner, two questions including whether advisory services could be provided in a more cost-effective manner if, for example, they were delivered by the not-for-profit or the private sectors, and whether the delivery of contribution agreements and the delivery of advisory services using a network of advisors rather than some other type of science and technology funding structure or framework is most appropriate. These questions had been raised during the course of the evaluation.
With respect to the first point, delivery of advisory services, the analysis conducted suggests that when compared to prices for comparative services in the marketplace, the Program can be considered cost-effective. With respect to the second question, it was found that with no benchmarks available, it was difficult to properly address the issues surrounding whether NRC-IRAP is minimizing the unit costs of its outputs and outcomes. However, decreases in program efficiency would suggest slightly diminished cost-effectiveness overall.
The evaluation found that NRC-IRAP continues to be consistent and aligned with stated government priorities on science, technology and innovation. Key stakeholder organizations, such as Industry Canada and Regional Economic Development Organizations, continue to see a role for the Program. Input into the evaluation confirms a strong view by many in the innovation system that it is appropriate for the Canadian government to support SMEs in their innovation needs as NRC-IRAP helps to address issues of market failure and underinvestment in R&D in Canadian SMEs. The program helps to "level the international playing field" and to match the efforts and commitments that other governments are making towards increasing the innovative capacity of their SME communities.
Based on the response of almost all individuals who participated in the evaluation or who participated in the focus group sessions, the ongoing need of the Program is dictated primarily by the ongoing needs of SMEs. SMEs were clear that NRC-IRAP has made a difference. They have been able to proceed with projects of greater scope, and generally have done so more quickly, as a result of NRC-IRAP.
Subsequent to the evaluation, the following recommendations were presented to the Program. The responses to each recommendation by NRC-IRAP management follow.
Recommendation 1: That the Program make every effort to simplify due diligence and monitoring processes in order to allow ITAs to pursue new client relationships, and that consideration be given to simplifying due diligence and reporting requirements for contributions of lower amounts (potentially under $25,000) or projects of lower risk.
Response: NRC-IRAP agrees with this recommendation. Over the past two years, the Program has responded to the demands from program audits and the Federal Accountability Act to put more administrative controls in place. With our experience in implementing these new controls and their attendant administrative costs for our SME clients and our staff, we intend to shift to customizing our controls requirements to be commensurate to the level of risks associated with NRC-IRAP activities. NRC-IRAP is currently developing Standard Operating Procedures (SOPs) to identify the proper level of controls required to adequately manage its various transfer payments. Through this exercise, NRC-IRAP will focus on the impacts the administrative process has on both staff and clients.
As an active member of Treasury Board Secretariat working groups stemming from Blue Ribbon Panel (BRP) findings, we are aware that process simplification is a government priority. A number of BRP-related activities suggest the need to shift the administrative culture towards delivering results in a sensible risk management framework. NRC-IRAP started to explore this "risk management" in Spring 2007 by simplifying the administration related to the Youth Program we deliver on behalf of Human Resources and Social Development Canada. We have tailored our financial monitoring requirements to the level of financial risks associated with clients/projects.
Recommendation 2: That the Program seek means to increase its level of contribution funding available to firms so that it may maintain an acceptable level of client reach while increasing the value of investments made to firms.
Response: NRC-IRAP agrees with this recommendation and has already initiated actions to do so. In April of 2007 NRC-IRAP was successful in obtaining expenditure authority to direct $15 million per year originally provided to NRC-IRAP in 1998 to co-fund industrial R&D projects with the now sunset Industry Canada TPC program ($13.5 million of this will be dedicated to contributions to firms).
IRAP management has instituted strong central financial management and financial forecasting. The Program will continue to seek various avenues and mechanisms to increase the level of contributions.
Recommendation 3: Senior management should develop a nationally-coordinated approach to the collection, analysis and reporting of performance information. This should involve close consultation with the NRC-IRAP regions to ensure consistency of information for national roll-up. As well, NRC Planning and Performance Management Directorate should be consulted to ensure congruence with federal accountability requirements.
Response: NRC-IRAP agrees with this recommendation. In fact NRC-IRAP executive included the need to have enhanced performance measures as one of seven key program commitments in its 2006-07 planning documents, and that commitment for national metrics continues to be part of key Senior Leadership Team objectives.
In collaboration with NRC-IRAP regions, the Program is currently piloting indicators related to the technical and commercial benefits of its funded projects to firms. Once the pilot is complete it will be launched across all regions. Measurements and processes related to our advisory services and contributions to organizations should be in place by December 2008. We have already had preliminary discussions with NRC.
We will consult Planning and Performance Management Directorate staff as we implement this action. We look forward to benefiting from the wealth of knowledge they have gained as a result of the NRC Evaluation of NRC-IRAP, as well as their experience with developing performance indicators. A balance must be struck, however, between resources expended, both PYs and operating funds, to collect performance information. The primary goal will be to collect data to enhance strategic planning and program management.
Recommendation 4: That NRC-IRAP continues to explore collaborative opportunities to work with NRC institutes, recognizing that the researchers, laboratories and facilities managed by these institutes are increasingly focussed on supporting Canadian firms' innovative capabilities.
Response: We agree that the working relationships between NRC-IRAP and Institute staff have been enhanced in the past two years – no doubt due to the inclusion of NRC-IRAP staff on the many working groups and committees related to NRC's renewal journey. These forums were excellent opportunities for NRC to better appreciate the potential asset of the ITA network industry experiences and knowledge of SMEs and the industrial innovation process. ITA contributions to NRC's fuller understanding of the impact of various science areas on industry during the selection of national program areas was also an excellent vehicle for building understanding and mutual respect.
We also believe the advent of NRC key sectors and closer collaboration with NRC clusters are practical, results oriented ways for NRC-IRAP to work collaboratively with institutes, and will continue to strengthen institute capacity to transfer its knowledge to industry.
Institute staff are invited to participate in NRC-IRAP regional meetings to work with NRC-IRAP staff in seeking areas for collaboration.
Many of the 'barriers' cited in the Evaluation Report are outside the ability of NRC-IRAP to tackle, such as NRC's Intellectual Property Policy, researcher recognition and time available to work with SME clients, and institute revenue generation focus. While NRC-IRAP can inform, advise and share knowledge and experience related to SMEs, these are systemic issues that more appropriately would be addressed by NRC senior management, who can set reasonable expectations and clearly communicate these to all staff.
Report a problem or mistake on this page
- Date modified: