ARCHIVED - Audit of the National Research Council-Industrial Research Assistance Program (NRC-IRAP)

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

ARCHIVED - Audit of the National Research Council-Industrial Research Assistance Program (NRC-IRAP) (PDF, 682 KB)

1.0 Executive Summary

Background

This audit report presents the findings of the NRC internal audit of the National Research Council's Industrial Research Assistance Program (NRC-IRAP). This program provides Canadian small and medium-sized enterprises (SMEs) with innovation support including advisory services, technological assistance and financial assistance through limited cost-shared contributions. NRC-IRAP also collaborates in the development and maintenance of the organizational infrastructure that exists to support innovation in Canada. Expenditures were $132 million in 2006-07.

NRC-IRAP is a decentralized program. It is delivered through staff in five regions which serve some 100 communities across the country. As reported by program management, they work directly with more than 8,400 clients.

Audit Objectives, Scope and Methodology

The broad objective of the audit was to assess the extent to which NRC-IRAP is managed according to sound management principles. Specific objectives were to assess NRC-IRAP's management control and accountability frameworks; measure compliance with applicable legislation, policies and guidelines; assess the status of management's Program Improvement Plan; and follow up on NRC-IRAP-related recommendations in Industry Canada's September 2003 Audit of Technology Partnerships Canada.

The scope of the audit covered funding decisions from fiscal years 2003-04 and 2004-05 for which audit work was completed between June 1, 2005 and December 31, 2005 at NRC-IRAP National Office in Ottawa, Ontario, and at NRC-IRAP regional offices in Boucherville, Québec and in Winnipeg, Manitoba. The audit team also did limited work in spring 2007 at the National Office and in three of the regions: Atlantic & Nunavut, Québec and Ontario.

Audit Opinion and Statement of Assurance

We found that NRC-IRAP's management control and accountability frameworks were generally adequate. This conclusion is based, in part, on the improved management controls that were implemented and observed during the course of the audit. However, weaknesses are present which demand management's attention. These include financial monitoring and documentation to support payment approvals, performance information and strategic and operational planning.

While problems existed in the past relating to compliance with the Financial Administration Act (FAA) and Treasury Board Transfer Payment Policy, we found evidence of improvement for 2006-07 – specifically with respect to missing project proposals and lack of documentation to support recommendation decisions for organizations. However, as noted above in relation to the control framework for financial monitoring, significant problems continue with a lack of appropriate documentation to support claims for payment. Taking into account the improvements that were observed for 2006-07 in conjunction with the ongoing implementation of the detailed management action plan to address financial monitoring and its subsequent impact on FAA Section 34 certifications, we can conclude that NRC-IRAP is compliant overall with government legislation and policies

Performance information at the national level is problematic. Much of the available information on performance remains in the regions, and it is not routinely collected, analyzed and reported for management's use at the national level. Strategic and operational plans have not been developed, applied, reviewed and updated with appropriate frequency.

Finally, with respect to the implementation of previous recommendations for improvement, we found that NRC-IRAP management has implemented the majority of the recommendations in the May 2004 Program Improvement Plan as well as those emanating from the Industry Canada 2003 Audit of the Technology Partnerships Canada program.

In my professional judgement as Chief Audit Executive, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The conclusions were based on a comparison of the situations as they existed at the time against the audit criteria. The evidence was gathered in accordance with the Treasury Board Policy, directives and standards on Internal Audit, and the procedures used to meet the professional standards of the Institute of Internal Auditors.

Conclusions and recommendations

We found that NRC-IRAP's management control and accountability frameworks were generally adequate. This conclusion is based, in part, on the improved management controls that were implemented and observed during the course of the audit. However, weaknesses are present which demand management's attention. These include financial monitoring and documentation to support payment approvals, performance information and business planning.

First and foremost, financial monitoring continues to be weak. While some improvements have been noted from earlier years, Industrial Technology Advisors (ITAs) and Innovation Network Advisors (INAs) Footnote1 for contributions to firms and organizations are not verifying with suitable frequency that claims from recipients are in accordance with their contribution agreements. An audit of recipients for 2004-05 contributions completed in December 2006 by NRC Finance Branch indicated that 36 percent of agreement projects examined had claimed ineligible costs resulting in overpayments of $859,210. NRC-IRAP has responded by putting in place a detailed management action plan that was developed in conjunction with NRC Finance Branch.

While problems existed in the past relating to compliance with the Financial Administration Act and Treasury Board Transfer Payment Policy, we found evidence of improvement for 2006-07 – specifically with respect to missing project proposals and lack of documentation to support recommendation decisions for organizations. However, as noted above, significant problems continue with a lack of appropriate documentation to support claims for payment. Taking into account the improvements that were observed for 2006-07 in conjunction with the ongoing implementation of the detailed management action plan to address financial monitoring and its subsequent impact on FAA Section 34 certifications, we can conclude that NRC-IRAP overall is compliant with government legislation and policies.

Second, performance information at the national level is problematic. Much of the available information on performance remains in the regions, and it is not routinely collected, analyzed and reported for management's use at the national level. As a result NRC-IRAP does not show sufficient evidence of using performance information in its strategic or management decision making processes. Third, strategic and operational plans have not been developed, applied, reviewed and updated with appropriate frequency – linked possibly to the lack of readily available performance information at the national level and the period of significant change in organizational structure and senior management that has characterized NRC-IRAP these past two years.

We found that NRC-IRAP management has implemented the majority of the recommendations in the May 2004 Program Improvement Plan. As of March 2006, we verified through management reports that 64 percent or 50 of the 76 Task Force recommendations had been implemented or were close to full implementation. NRC-IRAP Management informed us that all essential tasks had been completed and the remainder were no longer valid. While this appears to be slow given the time lapse of two years, it should be viewed in terms of the sheer number of recommendations that had to be delivered during a time of significant organizational change and changes in senior leadership.

Finally, we found that most of the recommendations from the 2003 Industry Canada Audit of the Technology Partnerships Canada program had been implemented or are nearing completion. Of note is that although the program ended on March 31, 2006, NRC Finance Branch continues to work with NRC-IRAP on collecting the outstanding debt due to the Crown.

Key recommendations

  1. Senior management should complete a comprehensive NRC-IRAP business plan for 2007-08 without delay. Subsequent plans for future years should be developed and approved in a timely manner.
  2. Senior management should develop and implement a monitoring control framework that includes:
    • developing and communicating guidelines and instructions for ITAs and INAs that both describe what they should look at during on-site visits to recipients, and indicate the minimum requirements for documenting the results of these visits; and
    • developing and implementing mechanisms for verifying that ITAs and INAs have carried out site visits in accordance with the guidelines developed in (a) in order to provide greater assurance that amounts claimed by firms and organizations have been incurred for specific costs in accordance with the contribution agreements.
  3. Senior management should develop a nationally-coordinated approach to the collection, analysis and reporting of performance information. This should involve close consultation with the NRC-IRAP regions to ensure consistency of information for national roll-up. As well, NRC's Planning and Performance Management Directorate should be consulted to ensure congruence with federal accountability requirements.
  4. Senior management should develop and adopt simple paper-based and / or electronic system generated tools as part of its update or replacement of SONAR that will assist ITAs and INAs in exacting and demonstrating due diligence with respect to appropriate project approvals and amendments. A requirement that INAs also use the same centralized management information system for administering contributions for organizations, as it is presently the case for ITAs for firms, should be adopted.

Jayne Hinchliff-Milne, CMA, Chief Audit Executive

NRC Audit Team Members Footnote2:
Jayne Hinchliff-Milne, CMA, Chief Audit Executive

2.0 Introduction

2.1 Background and Context

The National Research Council's Senior Executive Committee, at its July 26, 2004 meeting, approved the 2003-2007 Internal Audit Plan, which included an internal audit of the National Research Council's Industrial Research Assistance Program (NRC-IRAP). The Committee recognized the importance of conducting this audit because of the public nature of the program and the high level of scrutiny that exists for transfer payments. In addition, NRC-IRAP's most recent Risk-Based Audit Framework (RBAF) provided to the Treasury Board Secretariat required that a value-for-money audit be conducted in fiscal year 2004-05.

The National Research Council Act (1985) provides the legislative authority for NRC-IRAP under section 5(1) (c): "undertake, assist or promote scientific and industrial research, including, without restricting the generality of the foregoing (ii) researches with the object of improving the technical processes and methods used in the industries of Canada, and of discovering processes and methods that may promote the expansion of existing or the development of new industries... ".

NRC-IRAP provides Canadian small and medium-sized enterprises (SMEs) with technological and business advice, financial assistance and a range of other innovation assistance aimed at helping SMEs develop, adopt or adapt technology in their quest to develop new or improved products, services or processes. This in turn is expected to lead to increased economic SME growth (including jobs and/or sales) and ultimately result in wealth creation for Canada.

At the time of the audit, NRC-IRAP was also responsible for delivering programs for other government departments, including Industry Canada's Technology Partnership Program (TPC) which ended March 2006 and Human Resources and Social Development Canada's (HRSDC) Youth Employment Strategy (YES) Program.

There are two classes of recipients of contribution funds: "firms" and "organizations". The Terms and Conditions for firms define two classes of eligible recipients as any Canadian firm or any other for-profit legal entity carrying on business in Canada. The potential recipient must also demonstrate willingness to accept NRC-IRAP's conditions of contribution as defined in the agreement covering contributions to firms.

The Terms and Conditions for organizations define the two classes of contributions for which organizations may be eligible. The first is contributions to organizations to support the costs of building and/or integrating the innovation capacity in their community for the benefit of Canadian firms, primarily SMEs. The second is contributions to organizations to support the costs of providing innovation services to Canadian SMEs.

Exhibit 1 provides an overview of NRC-IRAP expenditures for the past four fiscal years.

Exhibit 1
NRC-IRAP Expenditures from fiscal year 2003-04 to fiscal year 2006-07 Table 1 note 1
(Millions of $)

Fiscal Year
2006
2007
2005
2006
2004
2005
2003
2004
Contributions
Projects with Firms (SMEs) 61.7 68.3 60.6 62.4
Projects with Organizations 10.5 10.7 18.2 14.5
IRAP-TPC Projects 11.6 16.2 14.9 20.2
YES Projects 4.4 5.0 4.9 4.8
Sub-total Contributions 88.2 100.2 98.6 101.9
Operations
O&M – Excluding IRAP-TPC 40.0 42.9 43.4 42.3
Financial arrangements with OGDs
and IRAP-TPC operations & salaries
3.4 2.9 3.1 2.8
Sub-Total Operations 43.4 45.8 46.5 45.1
Total 131.6 146.0 145.1 147.0

Table Notes

Note 1

Of note, regional development agencies, the Atlantic Canada Opportunities Agency, the Canada Economic Development for Quebec Region Agency, the Federal Economic Development Initiative in Northern Ontario, and Western Economic Diversification Canada provided funds for IRAP contributions totaling less than $12 million annually for IRAP in 2003-04 and 2005-06.

Return to table 1 note 1 referrer

Source: NRC-IRAP Performance Reports

2.2 Management of NRC-IRAP

NRC-IRAP functions under the Treasury Board Policy on Transfer Payments. It administers contributions in accordance with two sets of Terms and Conditions approved by Treasury Board: Contributions to Firms, and Contributions to Organizations.

NRC-IRAP is a decentralized program and has an integrated network of approximately 220 technical and business experts located in 100 communities across the country. As reported by program management, they work directly with more than 8,400 clients annually. The program is supported by a National Office located in Ottawa, and delivered through five regions in Ontario, Québec, Atlantic / Nunavut, West and Pacific.

As shown in Exhibit 2, the Director General of NRC-IRAP reports directly to NRC's Vice-President of Technology and Industry Support and is responsible for managing the national delivery of the program and demonstrating the program's alignment with its strategic objectives. The position is supported by six Executive Directors located at the National Office and in each of the five regions. The Executive Directors are supported by Directors who are responsible for directing the Industrial Technology Advisors (ITAs) in their responsibilities for providing technological and funding assistance to small and medium size firms.

Innovation Network Advisors (INAs) in the region who report to the Executive Directors, focus on building effective regional innovation system relationships and, where warranted, work with innovation support organizations to provide expanded or new innovation assistance to SMEs. They also work closely with the Directors and the ITAs in each region and help identify gaps in regional innovation assistance available to respond to SME needs and work with organizations to provide resources to address these unmet needs.

Business Analysts (BAs) who are located in all regions except Québec, manage a portfolio of clients in collaboration with ITAs, and conduct due diligence on client companies and their projects to ensure that the basic business functions are planned for and provide support and advice to clients throughout the process.

Each region has either a Regional Operations Manager (ROM) or a Finance and Operations Manager (FOM) reporting to the Executive Director, responsible for ensuring day-to-day operations of the program in the areas of resource management, quality assurance and performance management. Finally, Regional Contribution Agreement Officers (RCAOs) who work under the supervision of the ROM / FOM are responsible for a wide range of activities undertaken in support of the delivery of the program with regard to financial contributions. More specifically they administer contribution agreements with firms and organizations, including assisting the ITA with preparing and reviewing agreements and amendments, reviewing claims and processing payments in accordance with the terms of the contribution agreement, and advising ITAs, INAs, signing authorities and clients on appropriate modifications to agreement clauses.

Exhibit 2
NRC-IRAP Organization Chart

Exhibit 2 - NRC-IRAP Organization Chart

Source: NRC-IRAP

2.3 About the audit

Objectives

The broad objective of this audit was to assess the extent to which NRC-IRAP is managed according to sound management principles. The specific objectives were:

  • To provide an independent assessment of NRC-IRAP's management control and accountability frameworks;
  • To measure compliance with the Financial Administration Act (FAA) and with Treasury Board policies and guidelines regarding transfer payments administration;
  • To assess the status of the Program Improvement Plan prepared by NRC-IRAP management; and
  • To follow-up on the recommendations pertaining to NRC-IRAP contained in Industry Canada's September 2003 Audit of Technology Partnerships Canada (TPC).

Scope

The scope of the audit emerged into two distinct phases. The first phase of the audit covered funding decisions in fiscal years 2003-04 and 2004-05 in four of the five IRAP regions: Atlantic & Nunavut, Quebec, Ontario and West. The Pacific Region was excluded due to other ongoing audit work being conducted at the time. For this phase, a sample of 46 contribution agreement files comprising firms, organizations and IRAP-TPC were examined in detail. The audit did not review contributions made under HRSDC's Youth Employment Strategy (YES) Program. These contributions represented less than five percent of total 2004-05 contributions and were considered to be of lower risk.

During this time, IRAP was undergoing a significant period of organizational change and administrative adjustments. When management was presented with the audit's preliminary findings of this portion of the audit, their comments were overwhelmingly consistent that much had changed in NRC-IRAP and that the results were therefore misleading. Subsequently, a second phase was conducted in spring 2007 to substantiate management's assertions of improved performance for contributions to organizations as well as other areas audited. Testing for an additional 16 agreements was limited to three regions – Atlantic & Nunavut, Québec and Ontario – and concentrated on the deficiencies found in the first sample of 46 files. Because the rates of non-compliance for IRAP-TPC contributions and contributions to firms were low, they were not revisited during phase two.

Exhibit three summarizes the files selected for review by region, fiscal year and type of contribution:

Exhibit 3
Summary of Files Selected for Review

2003
2004
2004
2005
2006
2007
Region Total Firms Org. IRAP-TPC Firms Org. IRAP-TPC Org.
Atlantic & Nunavut 20 3 3 1 2 6 0 5
Quebec 13 1 2 1 0 2 2 5
Ontario 20 2 3 1 3 4 1 6
West 9 3 2 0 0 2 2 0
Total 62 9 10 3 5 14 5 16

Source: NRC Internal Audit

Approach and methodology

Interviews were conducted with key personnel in order to examine program processes, procedures, and practices. These included managers and staff in Corporate Services Branch, Finance Branch, Human Resources Branch and NRC-IRAP National Office in Ottawa. Interviews were also held with the regional Executive Directors, Directors, ITAs, INAs and operational staff at two regional offices NRC-IRAP West in Winnipeg and NRC-IRAP Québec in Boucherville. Interview guides and questionnaires were developed for the interviews.

We reviewed relevant program documentation, which included, but was not limited to, Treasury Board Secretariat submission documents, NRC evaluation and departmental performance reports, IRAP's electronic management information system SONAR and NRC Finance Branch recipient audit reports. A risk assessment of key program activities and processes was completed at the outset to concentrate on the areas of greatest concern. Finally, as noted above, we reviewed a sample of 62 IRAP recipient files in detail.

The audit was conducted using a series of detailed audit criteria that addressed the audit objectives, against which we drew our observations, assessments and conclusions. These audit criteria (see Appendix A) were derived primarily from the TB Policy on Transfer Payments and the Financial Administration Act.

3.0 Audit Findings

3.1 Audit Objective One: To provide an independent assessment of NRC-IRAP's management control and accountability frameworks

Overall Conclusion

While some areas are in need of improvement, we found that NRC-IRAP has overall an adequate management control and accountability framework. This conclusion is based, in part, on the improved management controls that were implemented and observed over the course of the audit where we noted that problems existed with the framework in various areas between 2003-04 and 2005-06. However we found significant improvement in all areas audited in 2006-07, except for the following:

  • Financial monitoring;
  • Strategic and operational planning; and
  • Performance measurement and reporting.

Findings

Direction, leadership and organizational structure

We expected to find that the direction and leadership of NRC-IRAP have been clear and communicated and that an organizational structure has been put in place that supports the goals and objectives of the program. We found that over the period from 2003 to 2006, there have been major changes in leadership and organizational structure within NRC-IRAP. However, the organizational structure as shown in Exhibit 2 and leadership have emerged from this period and now appears to be stable.

Following the intake of 160 new ITA staff in April 2003, seven regions were collapsed into five and the National Office in Ottawa was restructured which entailed entirely new reporting relationships. However, in our view the most critical challenge that NRC-IRAP was facing in that period was a large turnover in senior management. For example, in the past two years, two people have filled the key position of the NRC-IRAP Director General on an acting basis. The Director General is responsible for both managing the delivery of the program across Canada, and demonstrating the program's alignment with its strategic objectives. Finally, in July 2006, the serving Director General was confirmed on a permanent basis. At that time, he acknowledged in a communications to staff that the lack of this appointment along with added challenges in firming up subsequent management positions had led to a certain degree of instability.

The position of Vice-President, Technology and Industry Support was filled on an acting basis by three individuals between June 2003 and November 2006. In November 2006, the serving Vice-President was confirmed on a permanent basis. In December 2006, three of five serving regional Executive Director positions were confirmed on a permanent basis. In July 2006 the Executive Director for the National Office had also been confirmed.

As of May 2007, the situation was more stable in that most management positions have been filled. Only four Director positions of 17 in the regions were filled by actors. From our interviews with senior managers, we found that senior management recognized the need to build a financial management framework within NRC-IRAP. Accordingly, new finance positions were created and staffed in May – June 2007. While the first, Manager NRC-IRAP Finance, reports to the Executive Director National Office, three Finance Officer positions report to NRC Finance Branch.

Strategic and operational planning

We expected that appropriate strategic and operational plans would have been developed for NRC-IRAP and applied, reviewed and updated with suitable frequency. We also expected that these plans would specify program objectives for the planning period; specific strategies or actions for attaining them; measurable outcomes or targets to assess planned results during the planning period; and the financial and people resources to be used to put the plan into effect.

In December 2001, NRC-IRAP produced an IRAP Planning Outlook 2002-2005 document which identified the program's strategic objectives and seven priorities and key planned results. While planned results were identified, measures or targets to assess their success were not identified. Resources required to implement the plan were identified only as "to be carried out within the existing resources".

Following the outlook document, NRC-IRAP produced a comprehensive business plan Investing in Innovation: 2003-2008 Strategy to Stimulate Canadian SME Success. This plan clearly articulated specific program objectives and strategies for attaining them. It also provided for most strategies key results areas and quantified targets for measuring their success. Finally, financial resources required for the implementation of each strategy were specified but not the human resources.

Program documents reveal that following the development of the 2003-2008 strategic plan, the 2004 program reorganization resulted in NRC-IRAP management refocusing its strategic direction with an emphasis on improving its program delivery framework and the alignment of roles and responsibilities in the regions. Subsequently senior management approved in 2005 a NRC-IRAP Directional Document 2005-2006 that was viewed as being responsive to the needs of NRC-IRAP clients and their marketplace. It is generally referred to as the "Strategic Intent" document. While this document describes in general terms the program's two primary objectives for the next three years and the three measures for assessing their success, it does not provide the specific actions that will be used to attain them nor the specific targets by which to measure their success or the required resources to implement them.

Since that time, NRC-IRAP management has engaged in several comprehensive exercises to identify priorities for the program. These included a two-day strategic planning meeting of the NRC-IRAP National Executive Committee which resulted in the two-page document NRC-IRAP Strategic Priorities 2006-07 and 2007-08 which identified two ongoing commitments and seven key commitments. Working documents have been prepared for each of the commitments largely consisting of how each priority may be addressed or measured. Of particular interest is the commitment to "develop an IRAP business plan". This same commitment to develop a business plan for 2008-09 was later reiterated in the most recent one page planning document NRC-IRAP Senior Leadership Team Business Plan Priorities 2007-2008. Neither of these documents identifies specific strategies or actions for attaining the priorities, measurable outcomes or targets to assess their success nor the financial resources to implement them.

This recent lack of comprehensive strategic and operational plans which address the generally accepted components identified above may be the result of the significant changes in management and the lack of stability, as noted earlier. It may also be due to the fact that, as noted below under "performance measurement and reporting", there is a lack of easily accessible performance information at the national level.

Recommendation 1:

Senior management should complete a comprehensive NRC-IRAP business plan for 2007-08 without delay. Subsequent plans for future years should also be developed and approved in a timely manner.

NRC Management Response:

We agree partially with this recommendation. We agree that the program should have a comprehensive business plan and that these should be developed and approved in a timely manner. NRC-IRAP also believes that business planning for 2008/09 – 2010/11 should be aligned with NRC's new integrated approach to business planning and we are currently working on this. We do not, however, agree that NRC-IRAP should devote resources to the exercise of developing a formal 2007/08 plan at this point, as recommended by the auditors, given that we are six months into the fiscal year and a business plan for 2008/09 to 2010/11 is due October 2007.

Background:

Although IRAP does not have a formal "business plan" for 2007/08, the program has taken steps to articulate its strategic direction and priorities, and these have been communicated to staff and reflected in managers' MAAs.

For example, in 2005 NRC-IRAP prepared a strategic Directional Document. In 2006/07, NRC-IRAP voluntarily began to apply the NRC integrated planning approach that is now required for all I/B/Ps. In consultation with NRC Planning and Performance Management staff, NRC-IRAP's Director General and senior management team identified structured, priority strategies that were focused, action oriented, and integrated with the NRC Management Accountability Agreements (MAA). In June 2007, NRC-IRAP management reviewed and updated its program and key commitments, and yearly activities for a three year business cycle. Work is now well underway on the 2008-2011 business plan as part of the NRC wide business planning exercise.

Both in 2006 and again in 2007 NRC-IRAP's strategic directions and priorities were communicated to NRC-IRAP management and staff. Work has proceeded on these commitments and the Director General and Executive Directors are held accountable for progress in their individual MAAs.

Control Framework for managing contribution files

We expected to find that information contained in recipient files is complete and up to date, contain required approval documents, and takes advantage of office automation to help ensure compliance and financial integrity.

Management Information System

NRC-IRAP's main management information system for managing contribution agreements for firms, SONAR, is a customer relation management tool which has been extensively customized for NRC-IRAP. It was introduced in 1998 to support the collection, monitoring and reporting of information, including performance information for firms. Some INAs use SONAR for managing agreements with organizations; however, there is no requirement to do so.

At the time of our examination, NRC-IRAP was facing significant costs to adapt SONAR to technology changes made by the firm that supports SONAR. A group has been formed to study the possibility of adopting a new management information system that takes advantage of new technologies. In our opinion, a centralized management information system for both firms and organizations would be an important management tool for assuring compliance – especially if it includes system generated checks, as described later in this report under "the approval process – due diligence".

Definition of roles and responsibilities and segregation of duties

We expected to find that roles and responsibilities have been well defined and that an adequate segregation of duties exists between individuals responsible for assessing the eligibility of firms and organizations requesting funding, and those individuals responsible for approving funding.

We found that NRC-IRAP follows a team approach comprising several ITAs when recommending to the Director funding decisions for firms, and that segregation of duties effectively exists for these decisions in that distinct documents are prepared by different individuals such as the technical assessment, the business assessment and the recommendation document. The approach taken for recommending funding and authorizing subsequent expenditures for organizations is different which blurs somewhat the segregation of duties. Innovation Network Advisors appear to work on a one-on-one basis with the respective regional Executive Directors in making funding decisions and authorizing expenditures. While we were informed that others in addition to the INA and Executive Director are involved in the initiation and approval of organization agreements, there is a single Due Diligence Document which includes the recommendation. As well, the Executive Director has sole discretion for the approval of allowable costs. The steps preceding the preparation of the Due Diligence Document seems to be informal and are not documented in the files.

The financial monitoring control framework

We expected to find that an active regime has been put in place for monitoring contribution agreements and project results.

We examined the project monitoring activity of Industrial Technology Advisors and Innovation Network Advisors who are assisted in that function by the Regional Contribution Agreement Officer. ITAs and INAs receive project status reports and claims for costs, as required for the basis of payment for the contribution agreements. According to the program's existing Risk-Based Audit Framework developed for Treasury Board, ITAs and INAs are responsible for conducting periodic on-site visits with contribution recipients to ensure that work performed is in accordance with the project work outlined in the contribution agreements. However, we found that neither guidelines nor instructions have been provided to them on how to document and track their site visits.

We also found in our review of 62 contribution agreements for firms, organizations and NRC IRAP-TPC, that ITAs and INAs are not adequately verifying the specific details of the claims. We reviewed documentation for on-site visits contained either in SONAR and/or the hard copy files. The results varied from a complete absence of recorded on-site visits to detailed notes in a few cases explaining the results of the visit. This finding is consistent with the earlier finding identified in 2004 in the Report on the NRC-IRAP Control and Risk Management Framework Relevant to Contributions to Recipients. The 2004 report noted that ITAs rarely visited clients' premises for monitoring purposes after an agreement had been signed and that the focus of ITAs appeared, based on the files reviewed, to have been almost entirely on developing new contribution agreements. It's important to note, however, this does not necessarily mean that site visits did not occur. Rather it could mean they were simply not documented. Having said that, further evidence is provided below that at a minimum site visits are inadequate either in terms of having actually occurred or in terms of their quality.

These observations are corroborated by the work undertaken on behalf of NRC-IRAP by NRC Finance Branch's Transfer Payments Advisory Services (TPAS). This office has a rigorous recipient audit review process in place to carry out desk audits on contributions of $25,000 or less, and regional field audits for contributions greater than $25,000. The objectives of the TPAS audits are to verify, among other things, that amounts claimed by firms and organizations have been incurred for specific costs as per the contribution agreement. Such audits have been conducted for contribution agreements between 1999 and 2000, 2001 and 2003, and for fiscal year 2004-2005.

For contribution agreements in place in 2004-05, the audits were completed in 2006 by a chartered accounting firm. The review was based on a sample of 96 contribution agreements considered to be statically valid at a 90% confidence level with an accepted error rate of 7.5 percent. While an improvement over previous years, the results of this most recent review continue to demonstrate a lack of financial monitoring activity. Approximately 36 percent (36 out of 96) of the agreement projects examined had claimed ineligible costs resulting in overpayments of $859,210. For 22 percent of the contribution agreements where claims were examined, it was not possible to arrive at an opinion about the extent to which firms or organizations were complying with the agreements because they did not have the appropriate records on hand to verify the claims. The review concluded that NRC-IRAP staff are not doing enough financial verification of agreements at the firms' premises throughout the life of these projects to prevent over-claims from occurring. It noted as well that management controls were inadequate to ensure that NRC-IRAP staff perform the on-site work needed to prevent weaknesses leading to over-claims.

Of note is that on-site monitoring plays a central role in enabling staff to sign off under Section 34 of the Financial Administrative Act. Other findings of the audit relating to verification work under Section 34 are discussed later in this report under "Financial Administration Act (FAA) – Section 34".

Recommendation 2:

Senior management should develop and implement a monitoring control framework that includes:

  • developing and communicating guidelines and instructions for ITAs and INAs that both describe what they should look at during on-site visits to recipients, and indicate the minimum requirements for documenting the results of these visits; and
  • developing and implementing mechanisms for verifying that ITAs and INAs have carried out site visits in accordance with the guidelines developed in (a) in order to provide greater assurance that amounts claimed by firms and organizations have been incurred for specific costs in accordance with the contribution agreements.
NRC Management Response:

We agree with the recommendation that an effective level of monitoring is required and indeed NRC-IRAP has already identified a need for improvement in this area. In December 2006, NRC-IRAP implemented its current Financial Monitoring Requirements (FMR) process. The requirement is documented and face to face training has been carried out across the country, and is briefly summarized below:

All clients and projects are subjected to a risk assessment that determines the minimum level of financial monitoring required for each funded project (based on a low, medium or high risk level). A post-payment validation is mandatory on the first claim of every client/project and thereafter as dictated by the assigned risk level. There is also the requirement to have an obligatory meeting (either on-site (preferred) or by telephone (where on-site is not easy or timely) with the client to explain the Conditions of the Contribution Agreement and the claiming process, and the evidence required by NRC-IRAP to enable the Section 34 sign-off on claims. This new FMR is in line with the current direction of Treasury Board in light of the Blue Ribbon Panel Report on Gs and Cs.

NRC-IRAP's Operational Policy Unit and Finance Manager will institute a schedule, within one year, for random desk audits to ensure the new FMR is understood and is being practiced. Follow-up action in the form of training and communication will be taken should it be found that the FMR is not being properly implemented.

Further, NRC-IRAP will undertake a study to determine the best method for recording the outcomes of site visits by November 2007, and also determine the minimum requirements for such visit notations. Introduction and training related to the new visit recording requirement will be completed in all regions by March 2008.

The above mentioned desks audits will also serve as a mechanism for ensuring site visit requirements are being met.

 

 

Performance measurement and reporting

We expected to find that ongoing performance information is identified, collected, analyzed and appropriately reported for the NRC-IRAP program at the regional and national levels. Internal Audit looked to the expertise and findings of the NRC Planning and Performance Management Directorate's evaluation team that was conducting an evaluation of NRC-IRAP in 2006-07 to comment on these expectations. They found that NRC-IRAP's capacity in these areas appears to be inadequate. The regionalized nature of NRC-IRAP is leading to the development of regionally based processes for collecting, analyzing and reporting information on performance – information that is not readily available for management at the national level.

Identifying performance information

NRC-IRAP has identified the performance information it needs to report against program objectives as outlined in various performance documents. These include the Results-based Management Accountability Framework, October 2002; the NRC-IRAP Logic Model and Performance Indicators, June 2004; and the NRC-IRAP Logic Model, (included in the NRC-IRAP Directional Document 2005-2006), September 2005. All have been developed since its last evaluation in 2002. However, because the program has identified many different strategic objectives or "program logic" models over time, the indicators used to assess performance have varied over the last four years. As a result, clear communication of specific indicators upon which all regions of the program should collect information could be improved.

Collecting, reporting and analyzing performance information – national

NRC-IRAP can collect performance information at the output level for firms through databases such as SONAR, the program's client relations management information system. For example, NRC-IRAP can collect summary information on the program's outputs such as the number of SME contribution agreements and their value. An SME Client Profile report is available for 2002-03, and annual program performance reports include information on outputs. However this information – largely comprising program outputs – is not routinely collected, reported and analyzed at the national level to be used, for example, to identify possible trends. As a result NRC-IRAP does not show sufficient evidence of using performance information in its strategic or management decision-making processes.

Output information on funded organizations is difficult to capture at the national level. There is no requirement to collect data on contribution agreements with Network Members for input into SONAR. Although information on the number and total value of contribution agreements to organizations can be extracted through another system, SIGMA, it is necessary to contact each individual region of NRC-IRAP to obtain a complete picture of the purpose and nature of the contribution agreements in place.

Information on the actual performance of the program, i.e., immediate and intermediate results of contribution agreements, is not readily available. A performance "snapshot" menu of performance information exists in SONAR, which enables NRC-IRAP to collect data on certain outcome-oriented indicators for contribution agreements. These include total sales, employment and profit figures, and the number of products and new processes associated with NRC-IRAP investments. However, the evaluators found that the collection and analysis of this information tend to drop off considerably after the initial year in which a project is first launched and initial data entered. Because many contribution agreements with firms span multiple years, this results in a fair degree of difficulty in clearly understanding the impact of these investments over time.

NRC-IRAP does report on "Success Stories", i.e., cases where investments made in companies result in marked outcomes. These "Success Stories" have been used as a reasonable means for the program to demonstrate impacts in select cases.

Collecting, reporting and analyzing performance information – regional

The evaluators found that at least one region, Quebec, has developed a process for reporting on the benefits or outcomes of contribution agreements. Since 2003, through a direct client/ITA survey process, the region has been collecting data on results from firms anywhere from six to 18 months after projects have been completed. Ontario launched a similar process about 18 months ago. British Columbia is also considering launching a web-based tool that would facilitate some data collection. The BC Benefits to Canada (BtC) system has recently received approval for piloting, and the region hopes to have it implemented in the next few months.

Performance information on the reach and impact of funding to organizations is available in reports completed at the regional level (e.g., in final reports on funded projects). These reports provide performance information that includes, for example, concrete results of funded activities and feedback from participants. However, again, this information is not available at the national level. This information appears to exist in paper-based form in the regions only.

The various approaches that the regions have either adopted or are piloting may make it difficult for NRC-IRAP to report as a whole on its activities and achievements. Recognizing that some regions are implementing their own systems for capturing and reporting performance information, the NRC-IRAP program has recently emphasized the need for improved performance reporting, as outlined in its 2005 Strategic Priorities document.

The evaluators noted that a system for collecting data from clients on a periodic basis for use at the national level would have to be developed to avoid over-burdening NRC-IRAP staff and clients with duplicate requests for performance information.

Recommendation 3:

Senior management should develop a nationally-coordinated approach to the collection, analysis and reporting of performance information. This should involve close consultation with the NRC-IRAP regions to ensure consistency of information for national roll-up. As well, NRC's Planning and Performance Management Directorate should be consulted to ensure congruence with federal accountability requirements.

NRC Management Response:

We agree with this recommendation. In fact NRC-IRAP executive included the need to have enhanced performance measures as one of seven key program commitments in its 2006-07 planning documents and that commitment for national metrics continues to be part of Senior Leadership Team objectives.

In collaboration with our regions, we are currently piloting indicators related to the technical and commercial benefits of our funded projects to firms and once the pilot is complete it will be launched across all regions.

Measurements and processes related to our advisory services and contributions to organizations should be in place by December 2008. We have already had preliminary discussions with NRC Planning and Performance Management Directorate staff and will continue to do so as we implement this action. We look forward to benefiting from the wealth of knowledge they have gained as a result of the NRC Evaluation of NRC-IRAP, as well as their experience with developing performance indicators.

3.2 Audit Objective Two: To measure compliance with the Financial Administration Act (FAA) and with Treasury Board policies and guidelines regarding transfer payments administration

Overall Conclusion

While problems existed in the past relating to compliance, we found evidence of improvement for 2006-2007 – specifically with respect to missing project proposals and a lack of documentation to support recommendation decisions for organizations. However, significant problems continue with a lack of appropriate documentation to substantiate FAA Section 34 certifications – a consequence of the poor financial monitoring control framework as noted earlier in this report. NRC-IRAP has responded by putting in place a detailed management action plan that was developed in conjunction with NRC Finance Branch. Taking into account the improvements that were observed for 2006-07 in conjunction with the ongoing implementation of the detailed management action plan to address financial monitoring and its subsequent impact on FAA Section 34 certifications, we can conclude that NRC-IRAP overall is compliant with government legislation and policies.

Findings

We expected to find that NRC-IRAP had in place sufficient financial and management controls to ensure compliance with Program Terms and Conditions, the FAA and the TB Policy on Transfer Payments.

Guidelines have been developed as part of NRC-IRAP's business process to ensure that NRC-IRAP staff understand the requirements of the two sets of Terms and Conditions – one set for firms and another for organizations.

Eligibility for funding

All of the 62 contribution agreements that we examined met eligibility requirements. We found that a series of guidelines have been developed to assist the Industrial Technology Advisors and Innovation Network Advisors in determining eligibility and the level of funding to be recommended. Technological assessment guidelines, business guidelines and project costing guidelines form part of the contribution business process and are included on the NRC-IRAP intranet site. Project costing guidelines have been developed to assist the ITAs and INAs in understanding the various components which make up the project-costing process.

Content of contribution agreements

We found that NRC-IRAP has a standard, comprehensive contribution agreement. It includes all Treasury Board and departmental policy and program requirements relating to NRC (the contributor) and the recipients.

The approval process: due diligence

We expected to find evidence that decisions concerning the approval of recipients and projects were based on due diligence, i.e., a sound justification, reasonable analysis and accountability. Hence, we reviewed the files to ensure that project proposals, assessments and recommendations to proceed were on file.

Over the course of the audit, we reviewed 46 contribution agreements relating to the period between 2003 and 2005 consisting of 24 organizations, 14 firms, and 8 IRAP-TPCs . An additional sample of 16 organization network agreements was reviewed for 2006-07 in order to identify improvements. We found gaps in documentation such as missing proposals, assessments and recommendations on whether to proceed with projects. We also found minor problems with respect to delegated authority, as noted below. However, as shown in Exhibit 4, we found significant improvement in the contribution agreements for organizations from 2006-2007.

Exhibit 4
Summary of Missing Documents in Files by Type and by Period

Missing Documents Contribution Type
2003-04
2004-05
2006-07
Proposals Firms 0/14 N/A Exhibit 4 note *
Organizations 6/24 3/16
IRAP-TPC 0/8 N/A
Assesments Firms 0/14 N/A
Organizations 14/24 1/16
IRAP-TPC 1/8 N/A
Recommendations Firms 0/14 N/A
Organizations 13/24 1/16
IRAP-TPC 0/8 N/A
Total   34/46 (74%) 5/16 (31%)

Source: NRC Internal Audit

Table Notes

Note 1

N/A - not applicable as files were not examined as part of phase two of the audit to identify improvements.

Return to Exhibit 4 note * referrer

We found that project proposals were missing in nine files out of 62 contribution agreements examined. All were contributions to organizations. It is important to note that six out of the nine were from 2003-04 and 2004-05. The remaining three (all from one region) were from the 16 additional files that we reviewed for fiscal year 2006-07. The local office in this region indicated that they do not require a new proposal from a proponent with whom they had had a long-standing relationship. In our opinion this practice does not respect transfer payment policy requirements. However, these findings overall indicate that the situation with respect to missing proposals has improved over time for most regions and local offices.

In our opinion, many of the problems that we found could have been avoided if NRC-IRAP had used a simple checklist to make certain that all documentation requirements had been met and were on file. Using such a checklist could translate into a 100 percent compliance rate – especially if it is built into planned changes to SONAR. These changes could include system generated checks designed to ensure that ITAs and INAs complete each step in the process before being able to proceed electronically to the next. For example, the system will have to have recorded a proposal on file, before allowing the ITAs and INAs to move to the next screen to complete the assessment, and so forth. We noted that some regions have started to develop their own paper-based checklists to help ensure that they comply with documentation requirements. While we acknowledge the merits of these checklists, we encourage NRC-IRAP to consider adopting system checks as part of its efforts to replace SONAR with more up to date technology.

In reviewing the contribution files selected in our sample for 2003-04 and 2004-05, we noted a problem with a lack of documentation to support decisions to recommend funding to organizations. However, the situation had improved for the 2006-07 files in our sample.

As part of the business process, ITAs and INAs are required to complete a recommendation document. We found during our file review of 62 contribution agreements that completed recommendation documents were not available either in SONAR or in the regional paper files for 23 percent (14 out of 62) of the files reviewed. All of these files were for contributions to organizations. Most exceptions related to fiscal-year 2003-04 and fiscal 2004-05. For fiscal-year 2006-07 only one recommendation document of 16 files reviewed was missing. Therefore we have concluded that the management of files for this area has improved over time.

Amendments to contribution agreements

We expected to see that amendments to contribution agreements respected all necessary Treasury Board and NRC requirements for compliance and financial integrity.

We looked at whether staff who had signed off under Section 32 of the FAA had the delegated authority to do so. In 60 out of 62 cases, we found that they did.

One of the two instances in which evidence of authority was lacking involved a case in 2003 whereby a written record could not be located that the individual who was acting at that time had been given the authority to do so. The second instance was a case in which an agreement had been amended ten times, over a three-year period, from $50,000 to $832,000. For this agreement, we noted that the person who had signed off on the last three amendments did not have the authority to do so. This situation arose because the increases in value were not tracked accurately, and the higher dollar amounts crossed the threshold of that person's signing authority.

This problem could have been avoided if each NRC-IRAP contribution file included a cover sheet that tracks amendments indicating, in highly summarized form, any change to an agreement, the date of the change and the cumulative value. This procedure is an important one because many contribution agreements span a number of fiscal years and tend to increase in value. Discussions with Regional Office Managers confirmed that none of the regions audited was keeping track in a formal way of the cumulative balance of contribution agreements.

The results of our review of the files for 2003-2004 and 2004-2005 indicate that in 38 (83 percent) of the 46 files reviewed, contribution agreements were amended shortly after the original agreement had been signed, i.e., within one month. The frequency of amendments ranged anywhere from one amendment, to 14 and 21 amendments in two separate cases.

Amendments can dramatically change financial materiality. For example, the 21 amendments for one firm increased the value of the contribution agreement from $18,500 to $412,585 over a five-year period. As well, the scope of work facilitated through these amendments was not the same as the work originally proposed. Due diligence in the form of new project assessments should have been completed when the scope of the project changed.

Of note is that justifications for amendments were based primarily on rationales provided by firms relating to project time delays and shortage of funds.

Recommendation 4:

Senior management should develop and adopt simple paper-based and / or electronic system generated tools as part of its update or replacement of SONAR that will assist ITAs and INAs in exacting and demonstrating due diligence with respect to appropriate project approvals and amendments. A requirement that INAs also use the same centralized management information system for administering contributions for organizations, as it is presently the case for ITAs for firms, should be adopted.

NRC Management Response:

We agree with the recommendations that these are areas of opportunity for improvement. In order to appropriately address the subject matter, NRC‑IRAP has recently established a financial unit to develop, implement and assess the ongoing financial policies and procedures related to contribution agreement management for firms and organizations in addition to all other financial related matters faced by NRC-IRAP. The financial unit, in conjunction with the Operational Policy Unit (OPU) has been tasked to immediately (0-3 months) review the amendment process and provide an assessment on the nature of amendments. Subsequent to the assessment, NRC-IRAP will develop (3-6 months) and implement (6-9 months) a set of standard procedures to improve records management of all relevant amendment information for firms and organizations. Following the implementation, NRC-IRAP will monitor its standard procedures to ensure their sufficiency and compliance. Simultaneously NRC-IRAP will also undertake a review of its System User Requirements (3-6 months) and the update or replacement of SONAR will be determined (within the subsequent 12 months). Full conversion will be achieved by 2009. However, the expected date is dependent on meeting targets for selecting developing and implementing the SONAR upgrade or replacement.

Financial Administration Act (FAA) – Section 34

A number of financial controls are in place to ensure compliance with the FAA, including: financial signing authorities that have been established for the various levels of management.

The lead ITA in the case of contribution to firms, and either the INA, the ROM or the Executive Director in the case of contributions to organizations, certify that invoices comply with Section 34 of the FAA. In signing the performance certification, the NRC signing authority is certifying that:

  • the work performed was in accordance with the conditions of the contribution agreement;
  • the costs charged or claimed for are reasonable and in accordance with the terms of the agreement; and
  • all conditions of the agreement have been met.

Contribution agreements require that recipients (organizations and firms) maintain adequate financial records and accounts relating to the performance of the work under the agreement. Agreements also require recipients to make these records available to NRC with reasonable notice if requested to do so.

The appropriate authority must certify that recipients have met these requirements. The only way that they can do this is to visit a recipient's premises at least once during the course of the agreement either to observe results/activity directly and by assessing the accounting systems which produce these records – records which are critical to certification under Section 34.

As discussed earlier under Objective One, in the section on NRC-IRAP's financial monitoring control framework, in reviewing the 62 contribution agreements in our sample, we found that the documentation of the results of these site visits ranged from a complete absence of documentation to detailed notes in only a few instances. This variation could be explained by the lack of guidelines covering what ITAs and INAs should look for when visiting a recipient's premises, and what they should be documenting.

One of the objectives of NRC Finance Branch's Protocol for the Quality Assurance Review of NRC-IRAP Contributions is to provide assurance of the adequacy of FAA Section 34 account verification and to be able to state that a process for managing contributions is in place and is being properly and conscientiously followed. While the most recent audit of recipients completed in 2006 for 2004-05 agreements identified improvements in compliance, compliance rates remained significantly deficient. In March 2004, NRC Finance Branch reported to NRC's management that 52 percent of projects had overpayments associated with them in fiscal years 2001-2002 and 2002-2003. This figure declined to 36 percent in 2004-2005 or 6.5 percent of the total dollar value audited down from 15.7 percent. The 2006 report therefore made recommendations relating to FAA Section 34 sign offs, specifically that "expectations be clarified with regards to the requirements associated with exercising... authority under Section 34 of the FAA where [staff] are certifying that the work was performed as per the contribution agreement, that the costs were properly incurred and that all conditions of the contribution agreement have been met". NRC-IRAP is presently undertaking cost recoveries for these over payments.

In response to this report, NRC-IRAP has implemented a detailed action plan that was developed in conjunction with NRC Finance Branch. Planned actions include, but are not limited to, heightened financial monitoring requirements based on established risk assessment criteria and including 100 percent verification of first claims for all recipients in addition to randomly selected reviews for subsequent claims; training and communications with ITAs, INAs and recipients regarding requirements; and the development of electronic tools as part of its planned replacement of SONAR. We observed during the period from October to December 2006 that each NRC-IRAP region delivered an extensive training program to its staff on the regional implementation of these financial monitoring requirements. NRC Finance Branch was also involved in this training to varying degrees for each region.

 

3.3 Audit Objective Three: To assess the status of the Program Improvement Plan prepared by NRC-IRAP management

Overall Conclusion

We found that NRC-IRAP management has implemented the majority of the recommendations in the May 2004 Program Improvement Plan. As of March 2006, we verified through management reports that 64 percent or 50 of the 78 of the Task Force recommendations had been implemented or were close to implementation. While this appears to be slow given the time lapse of two years, it should be viewed in terms of the sheer number of recommendations that had to be delivered during a time of significant organizational change and changes in senior leadership.

Findings

In April and May 2004, an assessment of NRC-IRAP was carried out to determine the extent to which the Program's current practices for delivering contribution funding had addressed the concerns raised in current and past audit findings. This assessment was carried out as a result of a specific mandate established by an NRC-IRAP Steering Committee composed of the Directors General for NRC-IRAP and NRC Finance Branch and the Executive Director for the National Office. This assessment was delivered to the Task Force on Audit Findings which was responsible for developing a control management framework for regional and national action plans and proposing specific measures to better mitigate risk in the NRC-IRAP contribution program.

The NRC-IRAP Control Management Framework (May 2004) prepared by the Audit Task Force included the following observations:

"The task force... found that NRC-IRAP had not clearly defined roles and responsibilities and had not set-up a clear accountability framework for management and staff directly responsible for program delivery. It was also found that there was a lack of clear direction as to what the Program required from its clients seeking funding under NRC-IRAP's programs."

The Framework included some 78 detailed recommendations for improving NRC-IRAP touching on three primary areas of interest: adequate accounting records; costs properly claimed; and recipients contributing their share of total project costs.

During the course of this audit, we observed that NRC-IRAP had developed an extensive array of spreadsheets to list and track the status of implementation of the recommendations. Recommendations were bundled according to workload type, i.e., due diligence, communication, project costing, etc. and individual Operations Policy Unit leads had been assigned to address issues. For this audit, we completed a cross-walk between the document identifying all recommendations that was prepared by the Task Force in 2004 and an NRC-IRAP management progress mapping report. It demonstrated that 64 percent of the Task Force recommendations had been implemented as of March 2006. NRC-IRAP management informed us that all essential tasks were addressed over the two-year period and those no longer valid were closed; and that it is not expected that 100 percent of the recommendations would be implemented because of the evolution of the program over time and new priorities, e.g., management action plan to address the 2006 audit of 2004-05 recipients. Our review of the tracking document reveals those areas not yet addressed include some aspects of claim management, amendments, related SONAR functions and quality assurance processes. Management informed us that the outstanding requirement to define roles and responsibilities will be incorporated into the work undertaken through NRC-IRAP Business Plan commitments.

 

 

3.4 Audit Objective Four: To follow up on the recommendations pertaining to NRC-IRAP contained in Industry Canada's September 2003 Audit of Technology Partnerships Canada (TPC)

Overall Conclusion

We found that NRC-IRAP management addressed five of the eight audit recommendations as identified in their management action plan in response to the September 2003 Audit of the Technology Partnerships Canada. As of July 2007 management action plans for two of the three outstanding recommendations were partially completed and the remaining recommendation was not addressed as planned as it was felt that after some attempts to do so it would be more useful to respond by other means.

Findings

Included in the scope of this audit was a follow-up on the status of the implementation of recommendations contained in the 2003 Industry Canada Audit of Technology Partnerships Canada (TPC) program. At the time our audit commenced, NRC-IRAP worked in partnership with Industry Canada to deliver TPC which provided Canadian SME's with repayable financial assistance for projects at the pre-commercialization stage.

A Memorandum of Understanding (MOU) existed between Industry Canada and NRC-IRAP outlining the Technology Partnership Canada program commitments and Terms and Conditions. Although the MOU was to expire in December 2005, it was extended for an additional year. In September 2005 the Minister for Industry Canada announced that the TPC program would be wound down and sunset in March 2006. Hence, the program continues only for those contributions agreements that were already in place.

Our review of program documents revealed that problems in collecting repayable contributions were experienced by all regions in 2004-05. However, as of March 31, 2006, NRC Finance Branch asserted that the entire portfolio had been brought up to date.

As noted in Appendix B, we found that NRC-IRAP management addressed five of the eight audit recommendations as identified in their management action plan in response to the September 2003 Audit of the Technology Partnerships Canada. As of July 2007 management action plans for two of the three outstanding recommendations were partially completed and the remaining recommendation was not addressed as planned as it was felt that after some attempts to do so it would be more useful to respond by other means.

The two recommendations only partially completed pertain to standardizing file documentation through the development of checklists and the implementation of a quality assurance process to ensure completeness and monitoring of project risks. Both are expected to be addressed by a new comprehensive set of program guidelines as noted below.

At the time of this audit, we observed there was no overarching management-approved complete set of guidelines. Rather partial guidelines for TPC contribution agreements have been developed as needed and circulated to others as part of NRC-IRAP's business process. A new guide, "IRAP-TPC Repayment Operational Policies and Procedures Guide" has been under development and expected to be approved by Fall 2007. It is intended to be a single source of information with regard to IRAP-TPC administration. This guide once completed and approved by management should adequately address the two outstanding recommendations from the 2003 audit.

 

 

4.0 Conclusion

We found that NRC-IRAP's management control and accountability frameworks were generally adequate. This conclusion is based, in part, on the improved management controls that were implemented and observed during the course of the audit. However, weaknesses are present which demand management's attention. These include financial monitoring and documentation to support payment approvals, performance information and strategic and operational planning.

First and foremost, financial monitoring continues to be weak. While some improvements have been noted from earlier years, Industrial Technology Advisors (ITAs) and Innovation Network Advisors (INAs) Footnote 1 for contributions to firms and organizations are not verifying with suitable frequency that claims from recipients are in accordance with their contribution agreements. A recent audit of recipients performed by NRC Finance Branch in 2006, indicated that 36 percent of the agreement projects examined had claimed ineligible costs resulting in overpayments of $859,210. NRC-IRAP has responded by putting in place a detailed management action plan that was developed in conjunction with NRC Finance Branch.

While problems existed in the past relating to compliance with the Financial Administration Act and Treasury Board Transfer Payment Policy, we found evidence of improvement for 2006-07 – specifically with respect to missing project proposals and lack of documentation to support recommendation decisions for organizations. However, significant problems continue with a lack of appropriate documentation to support claims for payment. Taking into account the improvements that were observed for 2006-07 in conjunction with the ongoing implementation of the detailed management action plan to address financial monitoring and its subsequent impact on FAA Section 34 approvals, we can conclude that NRC-IRAP overall is compliant with government legislation and policies.

Second, performance information at the national level is problematic. Much of the available information on performance remains in the regions, and it is not routinely collected, analyzed and reported for management's use at the national level. Third, strategic and operational plans have not been developed, applied, reviewed and updated with appropriate frequency.

We found that NRC-IRAP management has implemented the majority of the recommendations in the May 2004 Program Improvement Plan. As of March 2006, we verified through management reports that 64 percent or 50 of the 78 of the Task Force recommendations had been implemented or were close to complete implementation. NRC-IRAP management informed us that all essential tasks were addressed and those no longer valid were closed. While this appears to be slow given the time lapse of two years, it should be viewed in terms of the sheer number of recommendations that had to be delivered during a time of significant organizational change and changes in senior leadership.

Finally, we found that most of the recommendations from the 2003 Industry Canada Audit of Technology Partnerships Canada program had been implemented or are nearing completion. Of note is that although the program ended on March 31, 2006, NRC Finance Branch continues to work with NRC-IRAP on collecting the outstanding debt due to the Crown.

Key recommendations

  1. Senior management should complete a comprehensive NRC-IRAP business plan without delay. Subsequent plans for future years should also be developed and approved in a timely manner.
  2. Senior management should develop and implement a monitoring control framework that includes:
    • developing and communicating guidelines and instructions for ITAs and INAs that both describe what they should look at during on-site visits to recipients, and indicate the minimum requirements for documenting the results of these visits; and
    • developing and implementing mechanisms for verifying that ITAs and INAs have carried out site visits in accordance with the guidelines developed in (a) in order to provide greater assurance that amounts claimed by firms and organizations have been incurred for specific costs in accordance with the contribution agreements.
  3. Senior management should develop a nationally-coordinated approach to the collection, analysis and reporting of performance information. This should involve close consultation with the NRC-IRAP regions to ensure consistency of information for national roll-up. As well, NRC's Planning and Performance Management Directorate should be consulted to ensure congruence with federal accountability requirements.
  4. Senior management should develop and adopt simple paper-based and / or electronic system generated tools as part of its update or replacement of SONAR that will assist ITAs and INAs in exacting and demonstrating due diligence with respect to appropriate project approvals and amendments. A requirement that INAs also use the same centralized management information system for administering contributions for organizations, as it is presently the case for ITAs for firms, should be adopted.

See Appendix C for the detailed management action plans that will address the recommendations.

Appendix A: NRC-IRAP Audit Criteria

No. Audit Criterion
Audit Objective 1: To provide an independent assessment of NRC's NRC-IRAP's management control and accountability frameworks
1. Project proposals contain clearly stated objectives and expected results, which are directly related to program objectives and approved, program Terms and Conditions.
2. The Program has the processes and the administrative capacity in place to review and recommend projects for funding.
3. On-going performance information is identified, collected, analyzed and appropriately reported.
4. Direction and leadership of IRAP is clear and communicated.
5. An organizational structure is in place that supports the goals and objectives of the Program.
6. Appropriate strategic and operational plans are developed, applied, reviewed and updated with suitable frequency.
7. Roles, responsibilities and segregation of duties have been defined.
8. A capacity to actively monitor, on an ongoing basis, management practices and controls have been established.
9. A process is in place for the monitoring of contribution agreements and project results (e.g. adequacy and timeliness of reporting).
10. There are mechanisms in place to ensure that quality management processes are monitored.
11. Information contained in the recipient files is provided in a way that is conducive to their use:
  • Files are complete and up-to-date;
  • Required approval documents (administrative and financial) are contained in files;
  • Correspondence related to the agreements are on files;
  • Duplication of information is avoided; and
  • The organization takes advantage of office automation.
Audit Objective 2: To measure compliance with the Financial Administration Act (FAA) and with TB policies and guidelines regarding transfer payments administration
12. Program officers understand who is eligible for funding, under what conditions, for what purposes, and in what amounts.
13. Funding is used for the purposes agreed.
14. The Program complies with appropriate Acts, regulations, terms and conditions, policies and appropriate agreements, more specifically with the TPP, FAA and Program Terms and Conditions.
15. A comprehensive contribution agreement exists that specifies all necessary Treasury Board and departmental policy and program requirements of the contributor and the recipients.
16. Systems, procedures, controls and resources are adequate to ensure compliance and financial integrity.
17. The decisions concerning the approval of recipients and projects respect the concepts of due diligence, namely a sound justification, a reasonable analysis and accountability.
Audit Objective 3: To assess the status of the Program Improvement Plan prepared by NRC-IRAP management
18. Problems with project and program performance are resolved quickly.
Objective 4: To follow up on the recommendation pertaining to IRAP in Industry Canada's September 2003 Audit of Technology Partnerships Canada
19. Follow-up on the status of the implementation of the recommendations of the 2003 Industry Canada audit of TPC

Appendix B: 2003 Industry Canada Audit of Technology Partnerships Canada Recommendations and 2007 NRC Follow-up Audit Findings

Recommendation from 2003 Audit Table 2 note 1 Recommendation Implemented
1. (a) Innovation Network Advisors (ITA) are not required to have designated business backgrounds. We recommend IRAP-TPC put greater emphasis on business and financial knowledge/experience in the ITA screening process given the degree of business background that is required in order to complete the necessary due diligence. A business background will also assist ITAs to appropriately monitor the financial and operational status of the projects and the Client's business. Yes; Business Analysts (BAs), as shown in Exhibit 2: NRC-IRAP Organization Chart, have been staffed in four of the five program delivery regions to provide expertise in monitoring the financial status of repayments. Prior to the sunset of the program as of March 31, 2006, new proposals had the benefit of IRAP-TPC Business Assessment Guidelines developed and disseminated to the regions in 2004 and a separate business assessment screen in SONAR that was developed for IRAP-TPC proposals. It should be noted however that the 2006 financial audit of 2004-05 recipients identified that the greatest number of overpayments (50 percent) were related to IRAP-TPC projects.
1.b) Repayments of contributions differ from estimates and forecasts.
We understand that IRAP-TPC has identified several factors explaining the large variance between forecasted and actual repayments. IRAP-TPC is in the process of reviewing their current practice to take these factors into consideration to assist them in determining forecasted repayments. We recommend that IRAP-TPC prioritize this process review in order to modify their current due diligence phase and funding agreement repayment terms to capitalize on past experience.
No; following initial efforts undertaken in 2004 and 2005 to implement better processes in forecasting repayments, NRC-IRAP management took the decision to develop Repayment Guidelines which were considered to be more useful. Management informed us that following extensive consultation with the regions in Spring 2006, work was undertaken to develop a national IRAP-TPC Repayment Operational Policies and Procedures Guide. This guide will include in one place all standard operating policies pertaining to IRAP-TPC administration. As of July 2007, the guidelines were in draft form with final approval expected in Fall 2007.
3.a) Project file documentation is not standardized and/or complete. Table 2 note 2
3.a) i) We recommend standardized forms be developed for all sectors to assist in documenting the names, positions, area of expertise, risk analyses, findings and recommendations of all due diligence team members. The standardized forms should include specific areas for each team member to document opportunities, risks, weaknesses and strengths of the projects, and resolution regarding concerns raised and a section for final recommendation by the reviewer. Such a form will ensure information requested from the reviewers during the due diligence is included in the file and the position of the reviewers regarding the project submitted by the applicant is documented. This documentation will also facilitate management of the file by the ITA or the review of the files by the Director.
Yes; prior to the sunset of the program as of March 31, 2006, new proposals had the benefit of IRAP-TPC Business Assessment Guidelines developed and disseminated to the regions in 2004 and a separate business assessment screen in SONAR that was developed for IRAP-TPC proposals. The 2007 NRC audit of eight TPC files demonstrated that IRAP-TPC files are generally complete. As shown in Exhibit 4 Summary of Missing Documents in Files by Type and by Period, for each agreement we found on file the appropriate project proposals and recommendation documents; for one agreement the assessment document was missing.
3.a) Project file documentation is not standardized and/or complete.
3.a) ii) We recommend meetings between Clients and IRAP-TPC be documented with minutes to maintain appropriate records of discussions and to support the annual risk assessment of the project.
Yes; the 2007 NRC audit's review of 8 TPC files demonstrated that site visits while well documented with respect to the technical aspects of project development, overall they did not capture well all necessary aspects of financial monitoring. As such, it is not surprising that the 2006 financial audit of 2004-05 recipients identified that 50 percent of overpayments were related to IRAP-TPC projects.
3.a) Project file documentation is not standardized and/or complete.
3.a) iii) We recommend IRAP-TPC introduce a standardized policy for documentation retention. A checklist should also be developed, identifying all documents for inclusion in an applicant file. A Quality Assurance process should be developed to ensure completeness of documentation. These recommendations will ensure files are properly supported and mitigate the risk that documents are lost or misplaced.
In progress; management informed us that limited progress has been in developing a policy for standardized document retention and accompanying checklist. For the future, these elements will be included as part of the standard operating procedures that are being developed for the national IRAP-TPC Repayment Operational Policies and Procedures Guide. This guide will include in one place all standard operating policies pertaining to IRAP-TPC repayment administration. As of July 2007, the guidelines were in draft form with final approval expected in Fall 2007.
3.b) Liaison position between TPC & IRAP-TPC remains vacant. IRAP-TPC and TPC [Industry Canada] to meet regularly to share information and best practices. We also recommend the liaison position be filled to facilitate communication and synergies between the two programs. Yes; the NRC IRAP-TPC liaison position has been staffed since October 2003. Regular communication between NRC-IRAP and TPC (Industry Canada) took place until such time Industry Canada's equivalent liaison departed May 2004. Industry Canada subsequently abolished the position. Although there were a few meetings between NRC-IRAP and Industry Canada at various working levels, there was a significant reduction in the frequency of interaction / communication on issues relating to program delivery until fall 2004. Management informed us that Industry Canada continues to be forwarded NRC-IRAP activity reports on an ongoing basis.
3.c) Current human and financial resources place strain on the program. IRAP-TPC to review its budget requirements to ensure all IRAP-TPC business processes are appropriately resourced, including the monitoring of project risks. In progress; Business Analysts (BAs), as shown in Exhibit 2: NRC-IRAP Organization Chart, have been staffed in four of the five program delivery regions to provide expertise in monitoring the financial status of repayments. As part of the development of the national IRAP-TPC Repayment Operational Policies and Procedures Guide, management has informed us that the roles and responsibilities for the administration of repayments will be clarified. As of July 2007, the guidelines were in draft form with final approval expected in Fall 2007. With respect to a review of budget requirements, as it was noted in the formal response to this audit recommendation, "NRC-IRAP does not anticipate substantially increasing the human resources involved in the due diligence process for IRAP-TPC beyond what is already discussed earlier in the NRC response to Recommendation 5.1 a)."
4. Compliance with TPC Terms and Conditions relating to audits. As dictated under Section 5.3 of the agreement template, IRAP-TPC is required to obtain for, Client's in the repayment phase an annual audited report of gross revenues. We recommend IRAP-TPC regularly follow-up with Clients to ensure that these annual audited reports of gross revenues are obtained on a timely basis. Yes; NRC Finance Branch informed us that the IRAP‑TPC portfolio is up-to-date including the fact that gross revenue reports are audited. Annual gross revenue reports are being collected except in instances where an exemption has been requested and given. The number of such exemptions is rising due to a shortage of auditors (relative to demand) and corresponding increase in audit costs to the clients.

Table Notes

Note 1

The original recommendations can be found in section 5 of the 2003 report.

Return to table 2 note 1 referrer

Note 2

Recommendations are numbered as they appear in the 2003 Industry Canada Audit Report of Technology Partnerships Canada ; there was no recommendation 2.0

Return to table 2 note 2 referrer

Appendix C: Management Action Plans

Audit
Recommendations
Corrective Management
Action Plan
Expected Completion Date Responsible
NRC Contact
1. Senior management should complete a comprehensive business plan for 2007-08 without delay. Subsequent plans for future years should be developed and approved in a timely manner. We agree partially with this recommendation. We agree that the program should have a comprehensive business plan and that these should be developed and approved in a timely manner. NRC-IRAP also believes that business planning for 2008/09 – 2010/11 should be aligned with NRC's new integrated approach to business planning and we are currently working on this. We do not, however, agree that NRC-IRAP should devote resources to the exercise of developing a formal 2007/08 plan at this point, as recommended by the auditors, given that we are six months into the fiscal year and a business plan for 2008/09 to 2010/11 is due October 2007. Background:
Although IRAP does not have a formal "business plan" for 2007/08, the program has taken steps to articulate its strategic direction and priorities, and these have been communicated to staff and reflected in manager's MAAs.
For example, in 2005 NRC-IRAP prepared a strategic Directional Document. In 2006/07, NRC-IRAP voluntarily began to apply the NRC integrated planning approach that is now required for all I/B/Ps. In consultation with NRC Planning and Performance Management staff, NRC-IRAP's Director General and senior management team identified structured, priority strategies that were focused, action oriented, and integrated with the NRC Management Accountability Agreements (MAA). In June 2007, NRC-IRAP management reviewed and updated its program and key commitments, and yearly activities for a three year business cycle. Work is now well underway on the 2008-2011 business plan as part of the NRC wide business planning exercise. Both in 2006 and again in 2007 NRC-IRAP's strategic directions and priorities were communicated to NRC-IRAP management and staff. Work has proceeded on these commitments and the Director General and Executive Directors are held accountable for progress in their individual MAAs.
October 2007 DG NRC-IRAP
2. Senior management should develop and implement a monitoring control framework that includes: a. developing and communicating guidelines and instructions for ITAs and INAs that both describe what they should look at during on-site visits to recipients, and indicate the minimum requirements for documenting the results of these visits; and b. developing and implementing mechanisms for verifying that ITAs and INAs have carried out site visits in accordance with the guidelines developed in (a) in order to provide greater assurance that amounts claimed by firms and organizations have been incurred for specific costs in accordance with the contribution agreements. We agree with the recommendation that an effective level of monitoring is required and indeed NRC-IRAP has already identified a need for improvement in this area. In December 2006, NRC-IRAP implemented its current Financial Monitoring Requirements (FMR) process. The requirement is documented and face to face training has been carried out across the country, and is briefly summarized below: All clients and projects are subjected to a risk assessment that determines the minimum level of financial monitoring required for each funded project (based on a low, medium or high risk level). A post-payment validation is mandatory on the first claim of every client/project and thereafter as dictated by the assigned risk level. There is also the requirement to have an obligatory meeting either on-site (preferred) or by telephone (where on-site is not easy or timely) with the client to explain the Conditions of the Contribution Agreement and the claiming process, and the evidence required by NRC-IRAP to enable the Section 34 sign-off on claims. This new FMR is in line with the current direction of Treasury Board in light of the Blue Ribbon Panel Report on Gs and Cs. Immediate DG NRC-IRAP
NRC-IRAP's Operational Policy Unit and Finance Manager will institute a schedule, within one year, for random desk audits to ensure the new FMR is understood and is being practiced. Follow-up action in the form of training and communication will be taken should it be found that the FMR is not being properly implemented.
August 2008 Executive Director National Office
Further, NRC-IRAP will undertake a study to determine the best method for recording the outcomes of site visits by November 2007, and also determine the minimum requirements for such visit notations. November 2007 Executive Director National Office
Introduction and training related to the new visit recording requirement will be completed in all regions by March 2008. The above mentioned desks audits will also serve as a mechanism for ensuring site visit requirements are being met. March 2008 DG NRC-IRAP
3. Senior management should develop a nationally-coordinated approach to the collection, analysis and reporting of performance information. This should involve close consultation with the NRC-IRAP regions to ensure consistency of information for national roll-up. As well, NRC's Planning and Performance Management Directorate should be consulted to ensure congruence with federal accountability requirements. We agree with this recommendation. In fact NRC-IRAP executive included the need to have enhanced performance measures as one of seven key program commitments in its 2006-07 planning documents and that commitment for national metrics continues to be part of Senior Leadership Team objectives.
In collaboration with our regions, we are currently piloting indicators related to the technical and commercial benefits of our funded projects to firms and once the pilot is complete it will be launched across all regions. November 2007 DG NRC-IRAP
Measurements and processes related to our advisory services and contributions to organizations should be in place by December 2008. We have already had preliminary discussions with NRC Planning and Performance Management Directorate staff and will continue to do so as we implement this action. We look forward to benefiting from the wealth of knowledge they have gained as a result of the NRC Evaluation of NRC-IRAP, as well as their experience with developing performance indicators. December 2008 DG NRC-IRAP
4. Senior management should develop and adopt simple paper-based and / or electronic system generated tools as part of its update or replacement of SONAR that will assist ITAs and INAs in exacting and demonstrating due diligence with respect to appropriate project approvals and amendments. A requirement that INAs also use the same centralized management information system for administering contributions for organizations, as it is presently the case for ITAs for firms, should be adopted. We agree with the recommendations that these are areas of opportunity for improvement. In order to appropriately address the subject matter, NRC‑IRAP has recently established a financial unit to develop, implement and assess the ongoing financial policies and procedures related to contribution agreement management for firms and organizations in addition to all other financial related matters faced by NRC-IRAP.
The financial unit, in conjunction with the Operational Policy Unit (OPU) has been tasked to immediately (0-3 months) review the amendment process and provide an assessment on the nature of amendments. October 2007 Executive Director National Office
Subsequent to the assessment, NRC-IRAP will develop (3-6 months) January 2008 Executive Director National Office
and implement (6-9 months) a set of standard procedures to improve records management of all relevant amendment information for firms and organizations. Following the implementation, NRC-IRAP will monitor its standard procedures to ensure their sufficiency and compliance. March 2008 Executive Director National Office
Simultaneously NRC-IRAP will also undertake a review of its System User Requirements (3-6 months) and January 2008 Executive Director National Office
the update or replacement of SONAR will be determined (within the subsequent 12 months). June 2008 DG NRC-IRAP
Full conversion will be achieved by 2009. However, the expected date is dependent on meeting targets for selecting, developing and implementing the SONAR upgrade or replacement. December 2009 DG NRC-IRAP

Appendix D: Glossary

List of Abbreviations

BA – Business Analyst
BtC – Benefits to Canada
CA – Chartered Accountant
CIA – Certified Internal Auditor
CMA – Certified Management Accountant
FAA – Financial Administration Act
FMR - Financial Monitoring Requirements
FOM – Finance and Operations Manager
HRSDC – Human Resources and Social Development Canada
I/B/Ps – Institutes, Branches and Programs
INA – Innovation Network Advisor
ITA – Industrial Technology Advisor
MAA – Management Accountability Agreement
MOU – Memorandum of Understanding
NRC-IRAP – National Research Council - Industrial Research Assistance Program
O&M – Operations and Management
OGD – Other Government Department
OPU – Operational Policy Unit
RBAF – Risk-Based Audit Framework
RCAO – Regional Contribution Agreement Officer
ROM – Regional Operations Manager
SME – Small and medium-sized enterprises
TB – Treasury Board
TPAS – Transfer Payments Advisory Services
TPC – Technology Partnerships Canada
YES – Youth Employment Strategy

Footnotes

Footnote 1

This can include Regional Operations Managers and Directors who also have FAA Section 34 authority.

Return to footnote 1 referrer

Footnote 2

The NRC Audit team was supplemented by a team of experienced auditors that were contracted to carry out the first phase of the audit work.

Return to footnote 2 referrer

Date modified: