Evaluation of NRC Grants to International Affiliations (IA)

Evaluation of NRC Grants to International Affiliations (IA) (PDF, 688 KB)

Executive summary

National Research Council's (NRC) Grants to International Affiliations program (IA) is a transfer payment program that funds Canada's membership dues in selected international scientific organizations as well as some travel costs. NRC's role in administering this program also includes acting as Canada's national adhering member to the International Council for Science (ICSU).

As was the case with the previous evaluation (2010), the international scientific bodies supported by NRC through the IA program appear to serve a demonstrable need, are aligned well with federal priorities, and perform well. NRC's role in administering the IA program has been conducted efficiently by NRC's Government and International Relations – International Relations Office (GIR-IRO) and has generated strong value for money despite a number of challenges. The two main issues identified by this evaluation are: 1) limited alignment with NRC priorities and 2) challenges related to the fundamental role of representing Canada's scientific, engineering, and industrial innovation communities.

A low level of alignment was identified in the previous evaluation of IA and some changes were subsequently made to address this. However, NRC's organizational transformation to a research and technology organization (RTO) model in 2013 has affected IA's alignment with NRC priorities and has shifted NRC further away from both its historical role as the national adhering member at ICSU, which began in 1931, and its associated role as a steward for managing international affiliation grants on behalf of Canada's scientific and engineering communities. As a result, it appears as though NRC may no longer be the most appropriate organization to administer IA.

Based on the findings of this evaluation, the following recommendations are proposed:

Recommendation 1: NRC should exhaust its options for transferring the IA program and ICSU national adhering member responsibilities to a more aligned organization within the next year.

Meanwhile GIR-IRO should continue to manage the program to the best of its ability and pursue the following recommendations.

Recommendation 2: GIR-IRO should, if appropriate, partner with pre-existing committees and organizations that represent Canada's scientific and engineering communities so as to bolster its representation of these communities.

Recommendation 3: Once a new CISET-type committee is established, NRC should task it with re-assessing the portfolio of IA-supported international scientific bodies to ensure that it adequately represents the needs of Canada's scientific and engineering communities.

Recommendation 4: GIR-IRO should continue with its efforts to reduce the burden of the annual performance review (APR) process and include within these efforts a reduction in the frequency of reporting.

1.0 Introduction

NRC's Grants to International Affiliations program (IA) is a low-materiality (<$600K) and low-risk program. IA underwent an extensive evaluation in 2010 which resulted in a solid foundation of evaluative information. In consideration of these factors, this current evaluation has been streamlined and includes methods that have been tailored to an appropriate level, including a review of selected documents, a review of available administrative and performance data, and seven interviews with key informants Footnote 1 . Although streamlined, this evaluation assesses the core issues of the TBS Policy on Evaluation (i.e. continued need, alignment with government priorities, alignment with federal roles and responsibilities, achievement of expected outcomes, and demonstration of efficiency and economy) over the past five years (2010-11 to 2014-15).

Since the previous evaluation, two major changes have occurred at IA (both occurring in 2012):

  1. The program's terms and conditions were revised. This revision resulted in an expansion of the program's list of stakeholder communities to include industrial innovation communities as well as the transfer of BIPM (Bureau International des Poids et Mesures) membership responsibilities to a new, separate terms and conditions (to be administered by NRC's Measurement Science and Standards (MSS) portfolio) Footnote 2 .
  2. IA's external advisory body, Committee on International Science, Engineering and Technology (CISET), was put on hiatus.

This evaluation focuses on IA's support of its remaining memberships, namely ICSU, other affiliations related to ICSU (18 ICSU unions, five ICSU interdisciplinary bodies/joint initiatives, and one ICSU scientific associate), and three other international scientific bodies; hereafter referred to as 'IA-supported international scientific bodies'. Subsidized travel costs for non-governmental Canadian delegates who attend general assemblies associated with these bodies are also included within the scope of this evaluation.

2.0 Program profile

On behalf of Canada's scientific, engineering, and industrial innovation communities, NRC is responsible for the payment of membership dues in international affiliations as well as certain travel costs for non-governmental scientists. Supporting these memberships and travel costs ensures that Canada has the opportunity to participate in international endeavours, collaborate on R&D projects, benefit from the exchange and dissemination of knowledge, host international scientific conferences and general assemblies, and, ultimately, remain competitive in the advancement of scientific knowledge and the development of technology.

NRC currently supports memberships in 28 international scientific bodies through IA. See Appendix B for a detailed list.

NRC's role in administering these grant monies is undertaken by its Government and International Relations - International Relations Office (GIR-IRO) and accountability for this function is the responsibility of NRC's Vice President, Business and Professional Services (VP-BPS). To support this responsibility GIR-IRO undertakes the following main activities:

  • Determines which international bodies will be supported by grant money and at what membership level
  • Maintains partnership agreements with collaborative groups (comprised of Canadian learned societies, federal departments/agencies, and/or universities) Footnote 3 who are responsible for establishing and maintaining the Canadian National Committees (CNCs) that coordinate the participation of Canadians in the supported international body
  • Collects reports on the activities and monitors performance of the supported international bodies via annual performance review questionnaires (APRs) that are completed by CNCs.

To help ensure that NRC administers IA funding on behalf of the relevant Canadian communities, GIR-IRO had, from 1995 to 2012, relied on the input of the Committee on International Science, Engineering and Technology (CISET). The main responsibilities of this committee were to represent the interests of various scientific disciplines, advise NRC on the composition of its IA investments in international affiliations, annually assess the performance of each supported international body, and act as the CNC for NRC's national adhering member role in ICSU. Beginning in 2012, all activities related to CISET have been undertaken by GIR-IRO. It is expected that a new committee, which is currently being planned, will resume these activities.

In addition to its role as administrator of the IA funds, NRC has, since the founding of ICSU in 1931, been Canada's national adhering member (also known as National Scientific Member) for ICSU. According to ICSU, this role involves providing input into ICSU on their strategic programs, engaging domestic stakeholders on options for nominations or ICSU related programs/policies, participating in ICSU board activities, and facilitating links with national governments and science agencies.

For a visual overview of the interactions of the parties involved in this program, see Figure 1.

Figure 1: Summary diagram of roles and interactions related to the IA program

Figure 1: Summary diagram. Long description follows.

Note: CISET was disbanded in 2012. A replacement committee is currently being planned. GIR-IRO has undertaken this committee's duties in the meantime.

Long description of the summary diagram of roles and interactions related to the IA program

This figure illustrates the roles and interactions related to the IA program. The diagram shows that NRC GIR-IRO provides money for membership dues to IA-supported international scientific bodies (which includes ICSU, ICSU related affiliations, and other international scientific bodies) as well as travel costs for non-government scientists and engineers for IA approved events. The diagram shows that leading Canadian scientists and engineers voluntarily participate in IA-supported international scientific bodies, Canadian national committees (related to IA), NRC partners (learned societies or government departments that partner with NRC, through IA, and support relevant Canadian national committees), and CISET. The diagram also shows a chain of linkages from these leading Canadian scientists and engineers to NRC partners, then to Canadian national committees, then finally to IA-supported international scientific bodies. This chain indicates that each level participates and supports the level above each. In relation to the Canadian scientists and engineers voluntary participation in CISET, this is done mostly for the purpose of providing input and advice to CISET, which is then analyzed by CISET members and delivered to NRC through NRC's VP of Business and Professional Services. This VP has responsibility for NRC GIR-IRO. The final other two relationships captured in the diagram are NRC GIR-IRO's partnership agreements with NRC partners and NRC GIR-IRO's support and coordination for CISET.

Finances

The annual average amount associated with the IA grants to international scientific bodies was $410K per annum. The vast majority of this amount (96%) goes towards membership dues and the remaining amount pays for eligible travel costs for non-federal government scientists who attend associated general assemblies.

Not included in this amount are the cost of administering the program and the value of the in-kind contributions provided by the Canadian scientists and engineers who volunteer with the CNCs and act as delegates. In regards to the former, the annual average cost was $70K from 2012-13 – 2014-15. In regards to the latter, the 2010 evaluation estimated (very conservatively) that this in-kind amount was approximately $140K per year.

3.0 Evaluation findings

3.1.1 Relevance

Finding 1: Participation in international scientific organizations addresses a demonstrable need for Canadian scientific and engineering communities and is aligned well with federal government responsibilities and priorities. However, since there is a lack of alignment between the program and NRC's priorities NRC may no longer be the most appropriate organization to administer IA.

3.1.1.1 Continued need

The 2010 evaluation identified five broad categories of need related to partnerships with international scientific organizations:

  1. Access to knowledge and expertise
  2. Knowledge mobilization
  3. Competitiveness and efficiency of Canadian research
  4. International influence and reputation of Canada
  5. Coordination of research worldwide

The 2010 evaluation found that NRC's participation in ICSU and NRC's support of international scientific bodies fully met its criteria for continued need in each of the above areas. Evaluative evidence confirms that these needs continue to exist and are being met by the bodies supported by the IA program. However, some criticisms were raised by interviewees regarding the limited amount of funds that were domestically available to support membership fees in additional international bodies that address these needs.

3.1.1.2 Alignment

Federal alignment

The 2010 evaluation found that supporting international S&T networks and partnerships were deemed to be consistent with the 2007 federal government S&T strategy, Mobilizing Science and Technology to Canada's Advantage.

An update to the 2007 federal S&T strategy was published in 2014, entitled Seizing Canada's Moment: Moving Forwards in Science, Technology and Innovation 2014. This revised strategy promotes the importance of international connections that help Canada tap into the strength of other countries and recognizes the requirement to conduct international research collaborations across many disciplines so as to address complex global challenges. Given the activities and objectives of the international scientific bodies supported by NRC, it is clear that these investments remain aligned to federal priorities.

NRC alignment

The 2010 evaluation of IA noted that NRC's role as Canada's national adhering member to ICSU and it maintenance of ICSU's memberships was due more to history than to NRC's strategic goals or interests. While NRC had historically played roles in representing Canada's scientific and engineering communities and in promoting and strengthening international research activities, the 2010 evaluation found that NRC had largely shifted away from these roles. As a result, the program represented only a limited amount of alignment and the evaluation recommended that NRC re-confirm the appropriateness of its role as the national adhering member to ICSU along with its associated role related to the administration of IA. If this could not be re-confirmed, the evaluation recommended that NRC should seek to transfer the program to another organization.

Immediately following the release of the 2010 evaluation, NRC's Senior Executive Committee (SEC) discussed the appropriateness of this role and noted that NRC had previously tried, without success, to transfer the role to another organization. For this reason, SEC agreed that NRC should take a more proactive role in the program to leverage greater value for NRC and Canada. Subsequent to this decision however was the release of NRC's 2013-2018 strategy, which announced NRC's organizational transformation to a research and technology organization (RTO). This shift resulted in an increased focus on industry support and on bolstering NRC's role in addressing the innovation gap between university-based discovery and industrial commercialization. As a result, NRC has shifted even further away from the historical roles that the 2010 evaluation identified. Therefore, despite some changes during the evaluation period, a significant portion of the IA program remains unaligned with NRC priorities. The extent of this misalignment is such that NRC should consider whether it is the most appropriate body for administering IA.

Appropriateness of federal government role

Authority over international relations and activities which represent Canada is typically conferred upon the federal government, despite not being designated officially under any constitutional provision. In practice however, this federal responsibility does not exclude the participation of other groups, particularly with regards to international science. Relevant examples of this include: 1) at least one international scientific umbrella organizations is supported by a non-federal government organization in Canada, i.e. Royal Society of Canada's national adhering member role with the InterAcademy Partnership (IAP); and 2) internationally, it is more common for a national scientific academy to play the role of national scientific member to ICSU than a federal government organization.

Beyond the convention of federal responsibility for international activities, the appropriateness of NRC involvement in IA activities is somewhat confirmed with federal legislation, namely the National Research Council Act, which mandates that NRC "undertake, assist or promote scientific and industrial research."

3.1.2 Performance

Finding 2: For the most part, the performance of IA-supported international scientific bodies is strong. The ICSU organization itself has recently received some strong criticism but the performance of IA's 27 other affiliations remains satisfactory.

3.1.2.1 Achievement of outcomes by IA-supported international scientific bodies

IA's list of intermediate outcomes in its 2012 terms and conditions fall largely within the three overarching categories listed as headers below.

Canada's leading scientists demonstrate influence, enhance Canada's reputation, and promote Canada's interests within international scientific bodies

The 2010 evaluation found that Canada's ability to promote its interests and contribute to decision-making within IA-supported international scientific bodies was strong; owing to the many instances of Canadian representation on executive bodies, working groups and other committees of these international organizations. Specifically, the evaluation found that among its peer group, Canada tied for fifth in terms of the number of Canadian researchers holding executive positions. When this was normalized according to GDP, Canada was placed first among its peers. This performance was found to both enhance Canada's reputation and promote Canadian, as well as global, interests.

To assess the performance of IA-supported international scientific bodies this evaluation focused on APR scores Footnote 4 , a selected document review, and qualitative evidence from interviews. Analysis of the APR scores revealed:

  • 68% of scores were ranked as 'high' and only 7% were ranked as 'low' regarding the question of whether the NRC partner/CNC was able to ensure that Canadian delegates were able to participate effectively and strategically in the conduct of the international scientific bodies' general assemblies
  • 81% of scores were ranked as 'high' and only 6% were ranked as 'low' regarding the questions of whether NRC partners/CNCs' were able to ensure that Canadian scientists rise to leadership positions in its associated international affiliation, promote Canadian contributions, and take active roles in international conferences, symposia, and workshops. These questions were the highest ranked of all questions asked.

Examples of influence, enhanced reputation, and promotion of Canadian interests were also discussed in interviews and selected APRs. The most notable of which were: the attainment of president position by Dr. Gordon McBean at ICSU; the strong historic performance in executive position attainment at International Union of Geodesy and Geophysics (IUGG) (Canada is third, behind USA and Russia); the fact that more than 20 projects are chaired by a Canadian at International Union of Pure and Applied Chemistry (IUPAC); and the leadership of NRC's Dr. Barry Wood in establishing a task force and permanent working group at Committee on Data for Science and Technology (CODATA).

However, while the reputation and influence of Canadian participants within IA-supported international scientific bodies were deemed as strong, it is important to consider the findings of a 2014 external review of the ICSU organization itself. This review found that the organization has lost its position as the preeminent organization that speaks for the international scientific community. The reasons stated for this loss of status were 1) a growing multiplicity of international scientific bodies with overlapping agendas; 2) an unclear vision; 3) an ineffective voice in raising awareness in the international science community and the public at large; and 4) governance challenges that have prevented ICSU's ability to speak for its whole membership and sufficiently engage its members. The review warned that if ICSU does not address these issues "there is a serious risk that it will wither on the vine and become irrelevant over the next few years." Footnote 5 While this is indeed a cause for concern for ICSU, it should be noted that the activities of ICSU affiliations, particularly its unions, are largely independent of the activities of ICSU itself and are separate from the assessment of that report.

While the performance of most of IA-supported international scientific bodies was deemed as strong, it is important to note that these bodies only represent a limited portion of Canada's overall involvement in international science, and as such, should not be seen as an assessment of Canada's performance in international science as a whole. The specific proportion that this program represents was not quantified for this evaluation. Qualitative evidence from interviews and the document review revealed a large amount of international scientific bodies that Canada participates in outside of IA-supported international scientific bodies.

Canada's leading scientists network, exchange knowledge, collaborate, and partner with other leading global scientists

The 2010 evaluation found some qualitative evidence to suggest that the opportunities for networking, exchanging knowledge, and collaborating within IA-supported international scientific bodies were of a quality that was expected. Similarly this current evaluation also found that all interviewees associated with the CNCs for IA-supported international scientific bodies indicated that the opportunities to network and exchange knowledge at general assemblies and conferences hosted by their international scientific body were either the best in their discipline or nearly the best. In addition, APR scores for questions regarding the dissemination of important scientific knowledge and the encouragement of Canadian scientists to take advantage of networking opportunities were strong, with approximately 74% of the scores ranked at the 'high' level and only 7% at the 'low' level.

Canada's involvement in IA-supported international scientific bodies contributes to Canadian economic prosperity

The 2010 evaluation found that the role that IA-supported international scientific bodies play in strengthening Canadian research is important and that this role has a link to the economic competitiveness of Canada. For its assessment of IA's contribution to Canadian economic prosperity, the evaluation assessed the economic impact of hosting four actual ICSU union general assemblies in Canada. The resulting impact included over $47M in economic output, $6M in exports, $16M in wages, and 550 direct and indirect employment opportunities.

While this type of economic analysis was not conducted for this evaluation, it was found that at least three general assemblies from IA-supported international scientific bodies occurred in Canada during the evaluation period. Given the large domestic and local economic returns that result from these events, the importance of this economic impact should not be overlooked.

3.1.2.2 Representation of relevant communities

Finding 3: GIR-IRO's ability to administer the IA program on behalf of Canada's scientific, engineering, and industrial innovation communities has been hampered by issues related to resource constraints, the composition of CISET membership, as well as a perception that CISET's disbandment has resulted in reduced representation.

In the administration of IA, NRC has the stewardship role of financially supporting certain membership dues and travel costs on behalf of Canada's scientific, engineering and industrial innovation communities. The 2010 evaluation of this program revealed some criticism of NRC's performance in this regard by noting that NRC's investments in international scientific bodies appeared to reflect historical commitments rather than commitments based on assessments that adequately reflected performance and the desires of the affected communities. As a result, the program was described as appearing "all but closed to new possibilities" for supporting other or different international scientific bodies. Since the 2010 evaluation, the composition of the international scientific organizations supported by IA has not changed and the impression of interviewees has remained largely consistent with this assessment.

The selected document review and key informant interviews revealed four likely reasons for this lack of change as well as the perception of an insufficient amount of representation:

  1. IA has insufficient resources to spread its investments across all communities and has consequently maintained the core group of organizations
  2. From 2010 to 2012, CISET membership contained limited representation from the affected communities, as only half of the external membership positions were filled
  3. The CISET committee was difficult to adequately populate because of the program's low amount of resources, its limited ability to support new investments, and its relatively low profile
  4. Following 2012, CISET was disbanded and this has resulted in a perception among some stakeholders that their research communities are less represented
    1. It should be noted however that this perception has persisted despite an increase in consultations with CNCs that GIR-IRO had instigated as a mitigation strategy following the disbandment of CISET
  5. NRC's shift in strategy in 2013 has reduced the priority placed on IA and a minimal amount of resources have been committed.

Given the fundamental importance of administering this program on behalf of these communities, improvements to this area should be considered.

3.1.2.3 Resource utilization

Finding 4: The utilization of IA's resources was efficient and contributed to a number of positive outcomes for Canada's scientific and engineering communities. However, given the lack of alignment with NRC, transferring the IA program to another organization may allow for greater synergies and a more effective means of supporting Canada's performance in international science.

Value for money

The average annual amount of $410K invested in membership dues and travel costs related to IA-supported international scientific bodies was universally seen by interviewees as a very strong investment. The main reasons for this included: strong opportunities for showcasing Canadian achievements internationally; unique opportunities for young scientists to build their reputations and network with world-leading scientists; the international perception that Canada is seen as doing its part for international science and global issues; and the economic return that results from hosting international events in Canada. However, it is important to note two aspects concerning value for money here. First, the amount of value derived from these investments is due largely to the volunteering of time by Canada's leading scientists and engineers. Second, it is not entirely clear whether or, to what extent, these activities would have continued in the absence of investments from IA, as Canadian scientists and engineers participate in almost all ICSU unions despite the fact that only around half of the unions are supported by IA.

While Canadian scientists and engineers have been able to participate in international scientific bodies without IA support, both the 2010 evaluation and this evaluation found qualitative evidence that this support should continue. For example, a number of interviewees from CNCs explained the reality of the strained finances and limited budgets of national scientific associations (which are mainly supported by membership fees). As well, other interviewees noted that Canada provides a relatively limited amount of support when compared to its peers, so reductions could lead to effects on Canada's reputation in supporting international science. These comments are supported by a 2010 evaluation finding that, when controlling for GDP, Canada ranked second to last among a group of peer countries in terms of the amount of funding provided for ICSU and related affiliations.

Efficiency

The 2010 evaluation undertook a costing exercise to determine the costs of administering IA. The cost was estimated to be $66K per year, which included costs related to in-person meetings of CISET, the costs associated with attending one ICSU general assembly (GA), and other related labour, operational, and travel costs.

The average annual administration cost over the past three years (which includes the cost of supporting one ICSU general assembly, the cost of acting as CISET in its absence, and other related labour, operational, and travel costs) is $70K. This amount represents a slight cost increase from the previous evaluation period but it is not clear how much of this increase can be attributed to the disbandment of CISET, as $5.7K was annually saved by not hosting CISET meetings but, conversely, the disbandment of CISET required that GIR-IRO take on more activities that otherwise would have been undertaken on an in-kind basis by the committee.

One important aspect with regards to the efficiency of this program is managing the cyclical requirements of IA. Figure 2 shows the annual costs of administering IA for the years where data was readily available. The relatively high cost in 2014-15 ($108K), which is more than double the average of the previous two years, is due mostly to the responsibilities associated with an ICSU general assembly (which occurs every three years) and, to a lesser degree, a more accurate tracking of labour, which particularly affects 2014-15. This cyclical ICSU GA requirement is routinely a strain on the limited labour resources of the International Relations team of GIR-IRO, which includes only 9 FTEs and has other ongoing responsibilities.

Figure 2: NRC annual costs for administering IA

Figure 2: NRC annual costs for administering IA. Long description follows.

Source: NRC corporate data
Note: Operational and travel costs includes the travel cost of the required participation of NRC's President at ICSU's GA in 2014-15, which occurs every three years, as well as the cost of some consulting services

Long description of NRC annual costs for administering IA
2012-13 2013-14 2014-15
Labour costs $28K $38K $94K
Operational and travel costs $38K $21K $59K
Total $44K $59K $107K
Economy

IA has provided an annual amount of $410K to NRC for its support of international scientific bodies since 2007-08. This amount has annually supported membership dues and some eligible travel costs for 28 affiliations. Over the years, GIR-IRO's ability to continue to support these bodies has been affected by inflation, increases in membership due costs, and annual fluctuations in currency.

The effect of inflation and the regular increases to membership dues has resulted in a diminishing ability for GIR-IRO to support its portfolio of 28 affiliations. Since 2007-08, inflation has reduced the purchasing power of the $410K budget by 12%. This cost pressure is compounded by regular increases in costs for memberships, which in some cases has increased greater than 50% over the evaluation period.

In addition, there is also the challenge associated with currency fluctuations. In years where the Canadian dollar is low compared to the European euro (i.e. 2013-14 and 2014-15) GIR-IRO faces challenges in affording the costs of all 28 affiliations.

In consideration of the challenges mentioned above, GIR-IRO should be commended for its ability to consistently support its portfolio of international scientific bodies. Inflationary pressures will continue to impact IA and, without relief from an increased Canadian dollar, the program may soon need to make decisions about the amount of memberships that the program can afford.

Another aspect of economy is the ability to achieve the goals of the program with the resource available. As mentioned above, Canada does not rank highly in its financial commitments to international scientific bodies. A couple of external interviewees noted that this relatively low amount of financial commitment has led to a number of missed opportunities to capitalize on beneficial circumstances. For example, a lack of dedicated resources for pursuing general assemblies has led to missed opportunities to host some international events in Canada. As well, interviewees noted squandered opportunities to fully reap the benefits from Canadians achieving prominent positions in international scientific bodies. In addition, some interviewees noted that the limited amount of funds both prevented an adequate breadth of coverage in international scientific bodies that are important to Canadian scientists. All of the above have, according to some interviewees, led to negative reputational effects for Canada as a contributor to international science.

These impacts and opportunities are worthy of attention and consideration but the transformation of NRC in 2013 has reopened the question of whether NRC remains the most appropriate Canadian organization to support these opportunities. Another organization that is more aligned with IA's objectives and goals may be more likely, willing, and able to seize these opportunities and may, in general, be in a better position to support the program. A short list of organizations identified by this evaluation includes Natural Sciences and Engineering Research Council (NSERC), Royal Society of Canada (RSC), Council of Canadian Academies (CCA), Industry Canada (IC), Canadian Institute for Advanced Research (CIFAR), and Department of Foreign Affairs, Trade and Development (DFATD).

Other options for economy were explored by the evaluation but the only notable additional option was the reduction in the administrative burden of the APRs, which was unanimously supported by interviewees.

4.0 Recommendations

In consideration of the evidence and findings of this evaluation, the following recommendations are put forth by NRC OAE.

Recommendation 1: NRC should exhaust its options for transferring the IA program and ICSU national adhering member responsibilities to a more aligned organization within the next year.

Meanwhile GIR-IRO should continue to manage the program to the best of its ability and pursue the following recommendations.

Recommendation 2: GIR-IRO should, if appropriate, partner with pre-existing committees and organizations that represent Canada's scientific and engineering communities so as to bolster its representation of these communities.

Recommendation 3: Once a new CISET-type committee is established, NRC should task it with re-assessing the portfolio of IA-supported international scientific bodies to ensure that it adequately represents the needs of Canada's scientific and engineering communities.

Recommendation 4: GIR-IRO should continue with its efforts to reduce the burden of the annual performance review (APR) process and include within these efforts a reduction in the frequency of reporting.

5.0 Management response

Recommendation Response and Planned Action(s) Proposed Person(s) Responsibilities Timelines Measure(s) of Achievement

Recommendation 1: NRC should exhaust its options for transferring the IA program and ICSU national adhering member responsibilities to a more aligned organization within the next year.

NRC agrees with the recommendation and will:

  1. review options
  2. explore feasibility of transfer
  3. decide on whether to transfer the program

VP, Business and Professional Services (Executive Champion)

Director General – GIR-IRO

  1. January 2016 – complete review of options
  2. June 2016 – feasibility explored
  3. August 2016 - decision on transfer (if feasible).
  1. Options summary created.
  2. Feasibility of options to transfer the program explored with top organizations identified.
  3. Decision made on whether to transfer the program.

Recommendation 2: GIR-IRO should, if appropriate, partner with pre-existing committees and organizations that represent Canada's scientific and engineering communities so as to bolster its representation of these communities.

NRC agrees with the recommendation and will explore partnerships as it reviews options for program transfer (recommendation 1).

VP, Business and Professional Services (Executive Champion)

Director General – GIR-IRO

  1. March 2016 – complete review of options
  1. Options summary created and partnership strategy defined.

Recommendation 3: Once a new CISET-type committee is established, NRC should task it with re-assessing the portfolio of IA-supported international scientific bodies to ensure that it adequately represents the needs of Canada's scientific and engineering communities.

NRC agrees with the recommendation and will task the new CISET-type committee to re-assess the portfolio.

VP, Business and Professional Services (Executive Champion)

Director General – GIR-IRO

January 2017

Review of IA-supported international scientific bodies and recommended actions.

Recommendation 4: GIR-IRO should continue with its efforts to reduce the burden of the annual performance review (APR) process and include within these efforts a reduction in the frequency of reporting.

NRC agrees with the recommendation and will ask the new CISET-type committee to review the APR process.

VP, Business and Professional Services (Executive Champion)

Director General – GIR-IRO

January 2017

Updated strategy and action plan for review.

Appendix A: Evaluation matrix

Questions Methods
Selected Document Review Administrative and Performance Data Analysis Stakeholder Interviews

Relevance

R1. Continued Need for the Program

  • To what extent is there a continued need for affiliation in international scientific bodies?

 

R2. Alignment with Government Priorities

  • To what extent is membership in international scientific bodies consistent with federal priorities?

 

  • To what extent is membership in international scientific bodies consistent with NRC priorities?

 

  • Is NRC the most appropriate administrator for the IA program?

 

R3. Alignment with Federal Roles and Responsibilities

  • Is a federal government role in this area appropriate and required?

 

Performance (Effectiveness, Efficiency and Economy)

P1. Achievement of Expected Outcomes

  • To what extent has Canada's participation in IA-supported international scientific bodies enhanced Canada's reputation and influence?
  • To what extent has Canada's participation in IA-supported international scientific bodies allowed Canadian scientists to collaborate, network, and partner with world leaders in their field?
  • To what extent has Canada's participation in IA-supported international scientific bodies supported Canada's competitiveness in S&T?

P2. Demonstration of Efficiency and Economy

  • Is the IA program delivered in an efficient manner?

 

  • Is the level of resources expended on Annual Performance Reviews (APRs) appropriate given the value of the resulting data and the level of risk of the program?

 

 

  • Are there feasible options for the program to be delivered more economically?

 

Appendix B: List of international bodies and networks supported by IA

Table 1: List of ICSU unions and partners supported by IA

Organization name Partner
International Astronomical Union (IAU) Canadian Astronomical Society (CASCA)
International Geographical Union (IGU) Canadian Association of Geographers (CAG)
International Mathematical Union (IMU) Canadian Mathematical Society (CMS)
International Union for Pure and Applied Biophysics (IUPAB) Biophysical Society of Canada (BSC)
International Union for Quaternary Research (INQUA) Canadian Quaternary Association (CANQUA)
International Union of Basic and Clinical Pharmacology (IUPHAR) Pharmacological Society of Canada (PSC)
International Union of Biochemistry and Molecular Biology (IUBMB)Table 1 note 1 Canadian Society for Molecular Biosciences (CSMB)
International Union of Crystallography (IUCr) NRC - Human Health Therapeutics
International Union of Geodesy and Geophysics (IUGG) Canadian Geophysical Union (CGU)
International Union of Geological Sciences (IUGS) Canadian Federation of Earth Sciences (CFES)
International Union of History and Philosophy of Science (IUHPS) – Division of History of Science (DHST) Canadian Society for the History and Philosophy of Science (CSHPS)
International Union of History and Philosophy of Science (IUHPS) – Division of Logic, Methodology and Philosophy of Science (DLMPS) Canadian Society for the History and Philosophy of Science (CSHPS)
International Union of Nutritional Sciences (IUNS) Canadian Nutrition Society (CNS)
International Union of Physiological Sciences (IUPS) Canadian Physiological Society (CPS)
International Union of Psychological Science (IUPsyS) Canadian Psychological Association (CPA)
International Union of Pure and Applied Chemistry (IUPAC) NRC - Energy, Mining and Environment
International Union of Pure and Applied Physics (IUPAP) NRC - Security and Disruptive Technologies
International Union of Radio Science (URSI) NRC - Herzberg Astronomy and Astrophysics
International Union of Theoretical and Applied Mechanics (IUTAM) Canadian Society for Mechanical Engineering (CSME)

Table notes

Table 1 note 1

In 2015, IUBMB voluntarily left ICSU and is no longer considered an ICSU union. Considering that IUBMB was an ICSU union during the majority of the evaluation period it has been counted as an ICSU union for this evaluation.

Return to table 1 note 1 referrer

Table 2: List of ICSU interdisciplinary bodies/joint initiatives and partners supported by IA

Organization name Partner
Committee on Data for Science and Technology (CODATA) NRC - National Science Library
Committee On Space Research (COSPAR) Canadian Space Agency (CSA)
Scientific Committee on Oceanic Research (SCOR) Fisheries and Oceans Canada (DFO)
Scientific Committee On Solar-Terrestrial Physics (SCOSTEP) Canadian Space Agency (CSA)
WMO-ICSU-IOC World Climate Research Programme (WCRP) Canadian Foundation for Climate and Atmospheric Sciences (CFCAS)

Table 3: List of ICSU scientific associates and partners supported by IA

Organization name Partner
International Commission on Illumination (CIE) NRC - Measurement Science and Standards

Table 4: List of other international scientific bodies and partners supported by IA

Organization name Partner
International Association for the Properties of Water and Steam (IAPWS) CANDU Owners Group
International Commission on Illumination (CIE) NRC - Measurement Science and Standards
International Council for Science (ICSU) NRC, with CISET serving as the CNC
International Permafrost Association (IPA) Geological Survey of Canada

Appendix C: Evaluation methodology

Scope

The time period covered by this evaluation is FY 2010-11 to FY 2014-15. IA's involvement with BIPM during the first two fiscal years of this period has been excluded from this evaluation's scope due to coverage of its performance in a 2014 evaluation of MSS. IA's involvement in EUREKA has also been excluded because of its recent implementation and a pending review of its performance in 2017. The focus of this evaluation is NRC's involvement in ICSU and the 28 international scientific bodies that were supported by IA over the evaluation period.

Design

This evaluation was designed to answer the five evaluation issues prescribed by TBS Directive on Evaluation in a way that was calibrated to the low-level of risk and low materiality of the IA program. These five issues and the associated high-level evaluation questions are presented in Appendix A. These evaluation questions were developed following some preliminary data gathering and interviews with program representatives.

The methods selected to answers these questions included a review of selected documents, analysis of selected administrative and performance data, and key informant interviews. Details about these methods are included below.

Review of selected documents

This evaluation reviewed close to 70 documents. The majority of these documents were internal to NRC and were selected by an NRC evaluation officer who had access to program files. A review of external documents was largely limited to a pre-identified list of reports and strategies but also included a few documents that were identified during the data collection phase. Qualitative data analysis software, QDA Miner, was used to assist with data collection and analysis.

Analysis of selected administrative and performance data

The program provided some administrative data regarding membership dues for IA-supported international bodies, the composition of IA-supported international bodies, the timing and location of general assemblies, and the travel costs expended by IA to support non-federal government scientists. This data was supplemented by SAP data, accessed by the evaluation team, regarding the cost of labour for administering IA.

Performance data was exclusively comprised of data from IA's annual performance reviews (APRs).

The data consisted of scores generated by APR reviewers, who annually (2010-2013) assessed each IA-supported international body's responses to APR questions (which largely followed the framework of the five core issues required by the TBS Directive on Evaluation). This data was analyzed according to the frequency of scores (high, medium, low) by question.

Key informant interviews

A total of seven key informants were interviewed during the data collection phase of the evaluation. These informants were selected by the evaluation team and consisted of a mix of ICSU unions, interdisciplinary bodies, ICSU itself, and a non-ICSU body. Two of the ICSU related bodies listed an NRC portfolio as their CNC. Interview results were analyzed qualitatively, as ability to generalize was limited with the small sample size.

Limitations

The calibrated design of this evaluation has resulted in a streamlined evaluation that is not as comprehensive as the previous evaluation; which generally found positive results and did not identify significant risks. The methods and scope of this evaluation were adjusted according to this as well as the generally acknowledged low-risk of the program and its low materiality (~$410K). This tailored method and scope served the purpose of providing a reasonable assessment of IA's relevance and performance within this context. However, it is acknowledged that this strategy was not without its limitations, which are listed below.

Review of selected documents

The somewhat limited and prescribed list of external documents did not allow for a full literature review and exploration of all research avenues. This was however somewhat mitigated by the inclusion and review of a few documents that were recommended by interviewees or found via a Google Alert that was set-up for various keywords (so as to avoid missing relevant developments that occurred while the evaluation was being conducted).

Analysis of selected administrative and performance data

Analyzing administrative data provided directly by the program can be problematic because of the possibility of manipulation. To mitigate this risk selected pieces of data were reconfirmed by accessing available data in a corporate database, SAP, and were confirmed to be accurate.

Key informant interviews

The number of key informants interviewed was limited to seven. Excluding internal interviewees, this number was six. This amount cannot be viewed as sufficiently numerous to represent the views of Canada's broad scientific, engineering, and industrial innovation communities. However, the six selected interviewees did represent a minimum cross section of groups that was identified by the evaluation team in the planning stages.

Appendix D: Acronyms and abbreviations

APR Annual performance review
BIPM Bureau International des Poids et Mesures
CCA Council of Canadian Academies
CIFAR Canadian Institute for Advanced Research
CISET Committee on International Science, Engineering and Technology
CNC Canadian National Committee
CODATA Committee on Data for Science and Technology
DFATD Department of Foreign Affairs, Trade and Development
DG Director general
GDP Gross domestic product
GIR-IRO Government and International Relations – International Relations Office
IA Grants to International Affiliations Program
IAP InterAcademy Partnership
ICSU International Council for Science
IUGG International Union of Geodesy and Geophysics
IUPAC International Union of Pure and Applied Chemistry
MSS Measurement Science and Standards
NRC National Research Council
NSERC Natural Sciences and Engineering Research Council
R&D Research and development
RSC Royal Society of Canada
RTO Research and technology organization
SEC Senior Executive Council
STIC Science, Technology and Innovation Council
TBS Treasury Board Secretariat
VP-BPS Vice President, Business and Professional Services

Footnotes

Footnote 1

Appendix C: Evaluation methodology contains more detail about the selected evaluation methods.

Return to footnote 1 referrer

Footnote 2

Given that the performance of BIPM was recently covered within the 2014 evaluation of NRC's Measurement Science and Standards (MSS) portfolio, it has been excluded from the scope of this evaluation.

Return to footnote 2 referrer

Footnote 3

These groups will hereafter be referred to as NRC partners

Return to footnote 3 referrer

Footnote 4

Annual performance reviews (APRs) are reports prepared by NRC partners (who are responsible for Canada's involvement in IA-supported international affiliations). These reports were, until 2013, evaluated and scored by members of CISET and NRC and were used to inform GIR-IRO's decisions regarding its support of each affiliation. In 2013, GIR-IRO established an external peer review process for this review.

Return to footnote 4 referrer

Footnote 5

ICSU. (2014). Report of the External Review Panel of the International Council for Science. p. 4-5.

Return to footnote 5 referrer

Date modified: