Evaluation of the NRC program of Class Grants to international affiliations

Final Summary Report, August 2010

Prepared by: Planning and Performance Management Directorate
Strategy and Development Branch

This Report was approved by NRC's President on October 15, 2010.

Evaluation Summary

Background

The National Research Council (NRC) maintains, on behalf of Canada and the Canadian scientific and engineering community, memberships in a number of international scientific bodies (see Appendix A). Adherence to these bodies is paid for through transfer payments, in accordance with the terms and conditions for Class Grants to International Affiliations. This report presents a summary of the evaluation of the NRC program of Class Grants to International Affiliations, which considered both the investments made through the program and the activities undertaken by NRC in support of these investments. The Terms of Reference for the study were approved by NRC's Senior Executive Committee (SEC) in September 2009. A full, detailed report is available on request.

Program Context

The development of Canada's science and technology (S&T) infrastructure and capabilities is critical to the country's ability to compete in the global knowledge-based economy. The 2007 Federal S&T Strategy – Mobilizing Science and Technology to Canada's Advantage – positions S&T as a source of national competitive advantage, and speaks to the importance of supporting excellence and performance on the international stage. It also highlights the critical importance of building links to the knowledge and resources of others in order to both maximize Canada's own strengths and address issues of a global nature Footnote1. One important mechanism that allows researchers to gain access to their international colleagues is the vast network of professional and scholarly organizations that researchers around the world are part of. In addition, one of the ways in which Canada is able to mobilize local expertise to address global challenges is through the many international scientific bodies that have taken on the task of coordinating scientific and technological efforts worldwide either through direct contact with individuals or via the organizations that they are affiliated to.

Program Overview

As part of the Government of Canada's role in fostering national and international networks and scientific partnerships, and its role in fostering a competitive and dynamic business environment Footnote2, the NRC program of Class Grants to International Affiliations supports Canada's membership in a total of 30 international science organizations, including: the Bureau International des Poids et Mesures (BIPM); the International Council for Science (ICSU); 19 international scientific unions and five interdisciplinary bodies / joint initiatives that fall under the umbrella of ICSU; and four other international scientific and technological organizations that have similar characteristics to those that fall under ICSU. Further details are provided in Table 1.

Table 1. Overview of Affiliations Held by NRC
Organization Mandate / Mission Funded by Canadian membership held since Footnote3
Bureau International des Poids et Mesures (BIPM) Footnote 4 The BIPM is an intergovernmental organization that represents the permanent financial, management and organizational structure for member governments to act in common accord on all matters relating to units of measurement.
Its mandate is to provide the basis for a single, coherent system of measurements throughout the world, traceable to the International System of Units (SI).
Established by the Metre Convention, an international treaty signed in 1875.
Financed jointly by 54 Member States.
Canada became a Member State of the BIPM by acceding to the Metre Convention in 1907.
International Council for Science (ICSUFootnote 5 The ICSU is a non-governmental organization with a global membership of 121 national scientific bodies, 30 international scientific unions and 21 scientific associates. Thousands of scientists are affiliated with ICSU via their professional organizations.
Its mission is to strengthen international science for the benefit of society by mobilizing the knowledge and resources of the international science community to:
  • Identify and address major issues of importance to science and society;
  • Facilitate interaction amongst scientists across all disciplines and from all countries;
  • Promote the participation of all scientists in the international scientific endeavour; and
  • Provide independent, authoritative advice to stimulate constructive dialogue between the scientific community and governments, civil society, and the private sector.
Membership dues of national scientific bodies and international scientific unions, as well as grants from the United Nations Educational, Scientific and Cultural Organization (UNESCO), the United States National Science Foundation (NSF), the Government of France, and other foundations. NRC has been the national adhering organization to ICSU since it was founded in 1931.
28 other international scientific organizations Footnote6 Various – however, in general:
Unions seek to foster, coordinate, strengthen, promote and safeguard science through international cooperation at the disciplinary level.
Interdisciplinary bodies / joint initiatives seek to address and advise on broad issues of an interdisciplinary nature and international scope through cooperation between unions, organizations, experts, etc.
Various – however, in general, funded through membership dues (Country, organization and/or individual level), activity-based dues and/or other sources. Various – Canada has been a member of some of the unions since their inception.

Accountability and Governance The Secretary General of NRC holds accountability for the program. The BIPM portion of the program is administered by the NRC Institute for National Measurement Standards (NRC-INMS). NRC is directly involved in the governance of the BIPM through the Director General (DG) of NRC-INMS who participates in meetings of the governing council of the BIPM (the Conférence Générale des Poids et Mesures [CGPM]) as a delegate of the Government of Canada, and was elected Vice-President (one of two) of the management board of the BIPM (the Comité International des Poids et Mesures [CIPM]). The ICSU and related affiliations portion of the grant is administered by the NRC International Relations Office (NRC-IRO). NRC is directly involved in the governance of ICSU through participation in meetings of the organization's governing council (the ICSU General Assembly [GA]). For the 28 other affiliations, NRC has entered into a partner agreement with a learned society, government agency and / or one of its own Institutes in order to share responsibility for the relationship with the international organization. In particular, partners are responsible for creating and maintaining a Canadian National Committee (CNC) composed of leading Canadian researchers to support Canada's affiliation with each international organization.

NRC is indirectly involved in the governance of other affiliations through the participation of Canadian delegates in the general assemblies of these organizations and through the CNCs who act at the interface between the international scientific bodies and the Canadian science and engineering community. NRC reserves the right to determine the level of adherence that it will hold in individual affiliations, and to terminate any affiliation.

Policy and program advice related to ICSU and other affiliations is provided by the Committee on International Science, Engineering and Technology (CISET) which is mandated toFootnote7:

  • Ensure appropriate Canadian participation in the activities of international scientific bodies to which NRC adheres on behalf of the Canadian S&T community;
  • Advise and assist NRC-IRO in developing and implementing NRC policy with respect to international scientific bodies;
  • Ensure that the NRC partners operate the CNCs corresponding to these international scientific bodies in an effective way;
  • Ensure that there are appropriate channels of communication between international scientific bodies and the Canadian interested agencies which are NRC partners for these international affiliations; and
  • Act as the CNC for ICSU.

Members of the Committee are appointed by the NRC Secretary General after consultation with program partners and other organizations. While the term of office for CISET members is three years, renewable once, there are members that have held office for more than two terms. Membership includes a Chair, a Secretary (NRC-IRO), and members representing the biological sciences; earth and space sciences; physical, chemical and mathematical sciences (currently vacant); environmental S&T; and engineering, as well as two members at large (one currently vacant). The NRC Secretary General is an ex-officio member of CISET.

Resource

Over the past five years, NRC has expended over $5.0 million in support of these affiliations, including $48.3 thousand for payment of travel costs of non-governmental Canadian delegates to international meetings. Historically, the program has faced ongoing financial pressure due to fee increases and currency fluctuations, resulting in an important amount of senior level time and effort being devoted to navigating these difficulties. In a few instances, this pressure has resulted in the need to defer payments and more recently, in advice to drop affiliations in ICSU unions or interdisciplinary bodies / joint initiatives in order to pay for the BIPM affiliation in the futureFootnote8.

The costs of NRC-INMS' participation in international activities stemming from Canada's status as a Member State of the BIPM are estimated at $1.4 million per year on average (or approximately 11 percent of the Institute's overall salary and operating expenditures). The cost to NRC-IRO is estimated at $65,898 per year on average (or approximately 6 percent of the Directorate's overall salary and operating expenditures), and includes an average $5,724 per year in costs related to the in-person meeting of the CISET. The in-kind contribution of CISET members is estimated at $17,524 per year on average. NRC-IRO also pays the costs associated with attending the ICSU GA once every three years. As previously mentioned, a number of organizations have entered into partner agreements with NRC in order to support these affiliations and create / maintain a CNC. Partners / CNCs are expected to undertake a number of activities in support of these affiliations. The majority (45 percent) of partners surveyed estimated that the cost of undertaking such activities was greater than $10,000 per year, and the validity of the methods used to arrive at these estimates was confirmed during follow-up interviews. Assuming a very conservative estimate of $5,000 per year per affiliation, the Program would be drawing on in-kind resources of $140,000 per year in order to achieve expected results.

Expected Results

Through the affiliations that are held with the organizations presented above, the NRC program of Class Grants to International Affiliations is expected to contribute to the achievement of the following results Footnote9:

  • exchange of experience, expertise, knowledge and technology between Canadian and international scientific communities;
  • development and sustaining of networks, collaborations and partnerships;
  • strengthening and positioning of basic and applied Canadian research for international competitiveness;
  • representation, promotion and protection of Canadian interests and priorities within international organizations;
  • influence and contribution of Canada to international S&T efforts, action plans, policy development and decision-making;
  • recognition of world-class Canadian S&T capacity worldwide;
  • economic spin-offs of hosting international events in Canada; and,
  • reduction of measurement-related technical barriers to trade (BIPM-specific).

The logic model included in Appendix B provides an overview of how investments made and activities undertaken feed into the achievement of these expected results. Overall, the logic model illustrates that NRC's influence in the achievement of expected results stems from the activities that it undertakes directly and the mechanisms that it has put into place for others to take action in support of these affiliations. The investments made by Canada through NRC set the stage for a number of other resources to be mobilized and for many activities to take place in support of expected results.

Evaluation Objectives, Scope and Methodology

The study was led by an independent evaluation team within NRC's Planning and Performance Management Directorate. The evaluation team's work was supported by Methodological and Subject Matter Experts who provided advice related to the evaluation framework, approach, instruments, interpretation of findings and recommendations.

The objectives of the evaluation were two-fold:

  • Decision-making Support: providing credible, timely and neutral information on the relevance and performance of Class Grants to International Affiliations in order to support decision-making around the renewal of the program's Ts&Cs (which expire on March 31, 2011); and
  • Accountability: responding to the requirement that all ongoing programs of grants and contributions be evaluated every five years (as per section 42.1 of the Financial Administration Act).

This study is the first formal, independent evaluation of NRC Class Grants to International Affiliations, and covered the period of FY 2005-06 to FY 2009-10 inclusive. The evaluation scope did not include the detailed examination of the relevance and performance of each of the 30 organizations in which NRC holds affiliation on behalf of Canada and the Canadian science and engineering community. Rather, the study was designed to look at the relevance and performance of the program as a whole, as per the core evaluation issues stipulated in the Treasury Board Policy on Evaluation. Table 2 identifies the specific evaluation questions that were addressed.

The study used a risk-based, mixed-methods design, involving multiple lines of evidence. The methods used include: a review of internal and external documents (including relevant literature); a review of administrative, financial and performance data; interviews with internal and external key informants (n=23 individuals); a review of similar programs in seven comparison countries; and a survey of a sample of international affiliations (including those held and not held by NRC) (n=24).

The main limitations of the study include: difficulty in obtaining reliable comparative data from similar programs internationally; difficulty in obtaining sufficient survey data to test the statistical significance of differences between affiliations held and not held by NRC; and some gaps in corporate records.

In addition, it is important for the reader to acknowledge that the program is part of a larger system that aims to contribute to achieving very broad economic, societal and environmental goals that are global in nature. As such, this study did not seek to establish links of attribution between the program and results observed, but rather aimed to assess its contribution to the achievement of these results.

Table 2. Evaluation Issues and Questions
Issue Question
Relevance
R1. Continued Need for Program
R1.1. To what extent do the Class Grants to International Affiliations and associated activities (collectively referred to as the program) continue to address a demonstrable need?
R1.2. To what extent is the program responsive to the needs of Canadians? What other mechanisms exist to address these needs?
R2. Alignment with Government Priorities
R2.1. To what extent is the program aligned with federal government priorities and expectations (e.g., S&T Strategy) and NRC's strategic outcomes (e.g., NRC Strategy, PAA)?
R3 Alignment with Federal Roles and Responsibilities
R3.1. To what extent is the delivery of the program appropriate to the federal government (i.e., NRC, Canadian Government), and a core federal role?
Performance
P1. Achievement of Expected Outcomes
P1.1. To what extent is the program achieving the immediate and intermediate outcomes that are expected? To what extent is it contributing to the achievement of the expected ultimate outcomes? What unintended outcomes are being achieved?
P2. Demonstration of Efficiency and Economy
P2.1. To what extent is the design, delivery (approach and capacity) and governance of the program conducive to achieving expected outcomes(immediate, intermediate and ultimate)?
P2.2. To what extent are the resources allocated to the program being utilized in an efficient and economical manner in producing outputs and progressing towards expected outcomes?
P2.3. Are there alternative modes of program delivery that would be more efficient and as effective?

Overview of Evaluation Findings

For each evaluation question, judgment criteria establishing the minimum expected level of performance were developed with program management. The tables below, organized by core evaluation issue, show the degree to which each of these criteria was deemed to be met, including the rationale for ratings provided.

Relevance

R1. Continued Need for the Program
Judgment Criteria Degree to which Criteria is Met Footnote10 Rationale
BIPM ICSU and related affiliations
The needs that the program is expected to meet are still present to at least the same degree as they were five years ago. Fully Met Fully Met BIPM / ICSU: Key informants and survey respondents believe that needs are still present to at least the same degree, with examples of areas of increased need.
ICSU: Although funding requirements have increased, it should be noted the rationale for specific adherence levels in ICSU and related affiliations has not been systematically examined.
The program's investment in travel is yielding expected results. N/A Fully Met ICSU: Survey results show that Canadian National Committees (CNCs) undertake activities in support of the role of Canadian delegates more frequently than do coordinating bodies of affiliations not held by NRC. No issues were identified in terms of the ability of delegates to play their role in international organizations.
Alternative funding [for affiliations and travel] does not exist. Fully Met Not Met BIPM: Alternative funding mechanisms do not exist for payment of BIPM.
ICSU: Data from comparison countries and survey results show that alternative funding mechanisms exist for paying for ICSU and related affiliations. Although CNCs express need for increased travel support, survey results show that alternative funding mechanisms exist for paying for travel.
There are benefits to having Canadians attend international events in person that could not be realized otherwise. Fully Met Fully Met BIPM / ICSU: Key informants and survey respondents were unanimous in their assessment of the ongoing value of attending international events in person, highlighting important benefits that could not be realized otherwise.
Affiliations held by Canada are similar to its comparison countries. Fully Met Partially Met BIPM: Affiliation in the BIPM is held by all Member States that have acceded to the Metre Convention.
ICSU: For ICSU and related affiliations, although comparison countries do not hold identical affiliations to Canada, the number of affiliations held by type is quite similar. In relation to comparison countries, Canada ranks second to last in terms of the amount of funding provided for ICSU and related affiliations in relation to Gross Domestic Product (GDP).

 

R2. Alignment with Government Priorities
Judgment Criteria Degree to which Criteria is Met Rationale
BIPM ICSU and related affiliations
There is logical alignment between the program and both government priorities and NRC's Strategy / objectives. Fully Met Partially Met BIPM: There is logical alignment with government priorities and NRC strategy / objectives.
ICSU: Although logical alignment with government priorities exists, NRC strategy / objectives related to international collaborations support actions related to NRC's R&D efforts in chosen critical fields rather than actions in support of the broad range of activities under the ICSU umbrella.
The alignment is recognized and made explicit. Fully Met Partially Met BIPM: The alignment is part of NRC-INMS' mandate in fulfilling the NRC Act.
ICSU: These relationships are mentioned but are not seen as mechanisms for strategy execution. Key informants are concerned about the alignment of ICSU and related affiliations with NRC's strategy / objectives.
The international affiliations held by NRC on behalf of Canada show better alignment with both government priorities and NRC strategy / objectives than the ones not held by NRC. N/A Not Met ICSU: The degree to which NRC has taken ownership of this program as a mechanism through which S&T priorities can be achieved has been criticized. Affiliations held reflect historical commitments, and the program is all but closed to new possibilities. CISET advice to strategically invest in unions vs. interdisciplinary bodies / joint initiatives is based on funding constraints, not on evidence of achievement of expected results.

 

R3. Alignment with Federal Roles and Responsibilities
Judgment Criteria Degree to which Criteria is Met Rationale
BIPM ICSU and related affiliations
The federal government is the only one that can administer payments and undertake activities associated with the program. Fully Met Partially Met BIPM: The BIPM is an intergovernmental organization, and National Metrology Institutes (NMIs) represent their countries governments in activities related to the Metre Convention.
ICSU: It is not necessary that a government organization take on this role for ICSU and related affiliations. However, it may be necessary and/or strategic for government to hold affiliation in bodies involved in standard-setting and in interdisciplinary bodies/joint initiatives.
Other countries administer such programs through a government body. Fully Met Partially Met BIPM: Given that this is a treaty obligation, the BIPM dotation is paid by the governments of Member States.
ICSU: More than half of the comparison countries adhere to ICSU through a non-governmental organization.

 

Performance

P1. Achievement of Expected Outcomes
Promote Canadian interests and influence international science decision-making
Judgment Criteria Degree to which Criteria is Met Rationale
BIPM ICSU and related affiliations
There is a difference between affiliations held and not held by NRC in terms of Canadian interests being promoted, and contributions to decision-making. N/A Partially Met ICSU: Although there is a difference between affiliations held and not held by NRC, this difference is small and could not be tested statistically.
Canada's level of influence and success in contributing to international S&T decision-making is at least consistent with its ranking with respect to the comparison countries. Fully Met Partially Met BIPM / ICSU: Canada ranks second in terms of participation in BIPM committees and working groups, and first in terms of executive positions in a sample of international organizations in relation to GDP. There is evidence that this participation is having an impact.
ICSU: There is room to improve Canada's level of influence and success in decision-making at the level of ICSU.
Strengthen Canadian Science through Knowledge Mobilization and Networking
Judgment Criteria Degree to which Criteria is Met Rationale
BIPM ICSU and related affiliations
There is a difference between affiliations held and not held by NRC in terms of the existence of factors that facilitate / demonstrate the achievement of expected outcomes (i.e., knowledge mobilization, collaboration, recognition, etc.). N/A Partially Met ICSU: Although survey results show consistent differences between affiliations held and not held by NRC, these differences are small and could not be tested statistically.
There is qualitative evidence to support the fact that international affiliations set the stage for these expected outcomes to occur. Fully Met Fully Met BIPM / ICSU: There is qualitative evidence to support the achievement of these expected outcomes.
There is qualitative evidence to support the fact that the program has facilitated opportunities that would not have been possible otherwise. Partially Met Partially Met BIPM / ICSU: Although interviews, survey results and annual performance data provide evidence of achievement of expected outcomes, there is insufficient evidence to assess whether or not these outcomes would have occurred in the absence of the program.
Contribute to Canadian Economic Prosperity
Judgment Criteria Degree to which Criteria is Met Rationale
BIPM ICSU and related affiliations
The BIPM affiliation and related activities are critical to protecting Canada's trade interests. Fully Met N/A BIPM: Canada's high reliance on exports makes this affiliation critical to the country's economic prosperity.
The number of events hosted by Canada is at least consistent with its ranking with respect to comparison countries (based on affiliation due structure). N/A Fully Met ICSU: For the sample of affiliations examined, Canada ranked second in terms of number of GAs held in Canada.
There is qualitative evidence to support the fact that hosting these events in Canada would not have been possible if Canada did not hold affiliations with the relevant international organizations. N/A Fully Met ICSU: Holding affiliation in these international organizations is required in order to bid on hosting events in Canada. Expenditure impacts of four Canadian events of international affiliations held by NRC were estimated at $29 million including over $47 million in economic output, over $6 million in exports, close to $16 million in wages and salaries and over 550 direct and indirect employment opportunities. The fact that NRC holds the affiliation does not appear to result in a distinct advantage.

 

P2. Delivery, Efficiency and Economy
Program Design, Delivery, and Governance
Judgment Criteria Degree to which Criteria is Met Rationale
BIPM ICSU and related affiliations
Design, delivery and governance meet or exceed Treasury Board Secretariat (TBS) expectations. Partially Met Partially Met BIPM / ICSU: Requirements under the 2000 Transfer Payment Policy were fully met. Elements of the 2008 Policy will need to be taken into consideration in the upcoming renewal of Terms and Conditions. Issues related to the Governance of the program were identified.
Performance data is of sufficient quality to make use of in evaluation and / or performance reporting. Partially Met Partially Met BIPM: Although Canada's affiliation in the BIPM is reported through NRC-INMS' annual performance report, the information provided is somewhat limited.
ICSU: The validation of annual performance data showed limited usability of data for the purposes of evaluation. Canada's affiliation in ICSU and related affiliations is not reported in any NRC corporate documents.
Decisions to support or not support specific affiliations are based on evidence. N/A Not Met ICSU: Performance information has yet to be used to reduce / cut funding based on performance. Existing affiliations are primarily based on historical commitments. Over the past five years, only one additional application for support was accepted. Other requests were denied due to lack of resources.
Program Efficiency and Economy
Judgment Criteria Degree to which Criteria is Met Rationale
BIPM ICSU and related affiliations
NRC has in place mechanisms to ensure that the most efficient and economical means are being used to administer the program. Partially Met Partially Met BIPM / ICSU: Although mechanisms are in place, cost savings are possible for both the BIPM and ICSU portions of the grant. However, such options need to be explored in light of their consequences.
Canada is using the most efficient and economical means of delivering the program. Partially Met Partially Met BIPM / ICSU: Although no other comparison countries administer the BIPM and ICSU portions of the grant under the same program, there is little evidence that cost savings would result from the grant being split.
ICSU: France and the UK have adopted means of delivery the ICSU portion of the grant that are worth exploring.
NRC is the most effective and efficient delivery agent for the program. N/A Insufficient Information ICSU: Although key informants highlighted other possible organizations for administering the ICSU and related affiliations portion of the program, no one organization emerged as being clearly the most appropriate organization for administering these affiliations. Information on whether these organizations could play this role more efficiently and economically could not be gathered.
Program management has considered and continues to explore alternative modes of delivery. N/A Fully Met ICSU: While formal discussions regarding moving the ICSU portion of the grant to the Council of Canadian Academies (CCA) were held in 2006, ongoing financial constraints have resulted in the need for management to continuously consider advice from CISET regarding cost saving options.

Conclusions

Program Relevance

The evaluation confirms the continued need for the BIPM, the ICSU and the related affiliations portions of the program. However, the rationale for NRC's affiliation in ICSU unions, ICSU interdisciplinary bodies / joint initiatives, and other affiliations reflect historical commitments rather than strategic intent. This is exacerbated by the financial constraints that have been faced by the program over many years owing to fee increases and currency fluctuations, resulting in policy advice aimed at coping with an ever shrinking margin of manoeuvre rather than advice aimed at maximizing the value of this program. Over the past five years, the program has been all but closed to new possibilities. At the same time, the evaluation highlights that other Canadian organizations hold affiliation or are involved in organizations in which NRC does not hold affiliation, implying that Canadian organizations and researchers to whom these organizations are important will find a way to take part.

The evaluation found that the program is aligned with government priorities. In terms of alignment with NRC's own goals and strategic outcomes, in the case of the BIPM, the alignment is part of NRC-INMS' mandate in fulfilling the NRC Act. In the case of ICSU and related affiliations, the evaluation highlights concerns raised by key informants regarding the role of NRC in administering this portion of the grant and the degree to which the organization has maximized the value of this program as a mechanism in support of the Federal S&T Strategy and the broader Canadian science and engineering community. At the same time, key informants spoke of the value of NRC's credibility and positioning as well as the historical role that it has played in this program, and although other organizations such as the Council of Canadian Academies (CCA), the Royal Society of Canada (RSC), Industry Canada (IC), and the Natural Sciences and Engineering Research Council of Canada (NSERC) were highlighted as possible candidates to take on this role, no one organization (including NRC) emerged as being clearly the most appropriate for administering these affiliations. However, recommendations from a recent evaluation of the CCAregarding its contributions to bodies such as ICSU and its support for a unified Canadian voice in international science policy may represent an opportunity for further exploration.

Program Performance

The evaluation presents evidence in support of the program's contributions towards effecting change in international science organizations, towards strengthening Canadian science and towards the Canadian economy. The direct financial costs to achieving these benefits for Canada are modest, and the program benefits from the financial and in-kind contributions of partner organizations and individual Canadian scientists (both academic and governmental). While the study cannot be conclusive as to the significance of the differences observed between affiliations held and not held by NRC, these may be evidence that the activities undertaken by CNCs in support of these affiliations are yielding expected results, which speaks to the effectiveness of changes made to the mechanism in the early 1990's. The evaluation also highlights Canada's superior rank as compared to other countries in terms of participation in BIPM committees and working groups, and in terms of executive positions held in a sample of ICSU unions, interdisciplinary bodies / joint initiatives and other affiliations.

At the same time, the evaluation notes a number of areas that impact upon the administration of this program and its ability to achieve its objectives:

  • A conflict exists between the DG of NRC-INMS' dual roles within the BIPM;
  • There is room to improve NRC's contribution to Canada's level of influence and success in decision-making at the level of ICSU;
  • Canada ranks second to last in terms of the amount of funding provided for affiliations to ICSU and related organizations, and many key informants felt that NRC would require additional resources to play this role properly; at the same time, given overall organizational funding constraints, costs savings have been identified which should be explored in light of their consequences;
  • Design, delivery and governance issues need to be addressed (use of APRs in decision-making and reporting; membership of CISET; creating two sets of Terms and Conditions, etc.)

Recommendations, Management Response and Action Plan

The evaluation recommendations, based on the study's findings and conclusions, are presented in the table below. The table also provides the Management Response and Action Plan proposed by NRC's Secretary General in response to these recommendations.

Recommendation Response and Planned Action(s) Responsibilities Expected Date of Completion Measure(s) of Achievement
Recommendation 1: NRC should consider the appropriateness of its role as the National Adhering Member to ICSU and as the organization responsible for administering the ICSU and related affiliations portion of this program on behalf of Canada. If it is decided that NRC's role is no longer appropriate, NRC should draw on the knowledge and expertise it has developed over many years to support the transfer of responsibility to another organization. Accepted
NRC will review the appropriateness of its role with ICSU in the context of the new NRC strategy. If it is determined that its role is no longer appropriate, NRC will undertake consultations to identify and support the transfer of responsibility to another organization.
NRC will seek the renewal of the Terms and Conditions for both programs to allow for the review to take place and the transfer of responsibility as deemed appropriate.
Secretary General April 1, 2011 Review completed.
Consultations completed.
Transfer of responsibility starts.
Terms and Conditions renewed for one year.
Recommendation 2: Contingent upon Recommendation 1, NRC should seek to maximize the strategic value of its affiliations in ICSU, its unions and interdisciplinary bodies / joint initiatives, and other related organization in support of the Federal S&T Strategy, and the broader Canadian science and engineering community. This would include: 1) ensuring an ongoing capacity with NRC-IRO related to these affiliations; 2) revisiting the mandate and renewing the membership of CISET to ensure that the Committee is representative of key actors from the Canadian science, science policy and science advisory communities, and is able to play a more strategic role within and outside the organization; and 3) ensuring better communication and promotion of these affiliations via existing channels. Accepted in principle
Contingent upon the results of the review in response to recommendation 1, should NRC confirm its role it will seek to maximize the strategic value of its affiliations in the context of its new Strategy and priorities, its emerging international strategy and available resources.
Secretary General To be determined To be determined
Recommendation 3: Contingent upon Recommendation 1, NRC should streamline administration and reporting for this program, in the spirit of the 2008 Transfer Payment Policy. This would include: 1) developing an ongoing set of Terms and Conditions that include both portions of the program (with BIPM as an assessed contribution), and 2) implementing refinements to reporting forms to improve the quality and usability of APR data as recommended in the APR Validation Study. Accepted in principle
Contingent upon the results of the review in response to recommendation 1, should NRC confirm its role it will seek to use the flexibility provided by the 2008 Transfer payment Policy for ongoing set of Terms and Conditions.
However, to allow for appropriate focus and facilitate management, NRC will seek to develop two sets of Terms and Conditions. NRC will also look at opportunities to improve the APR.
Secretary General To be determined To be determined
Recommendation 4: Contingent upon Recommendation 1, and consistent with the need to reduce administrative burden, NRC should seek to resolve the ongoing financial constraints faced by this program. Options should be explored with careful consideration of the likelihood and impact of risks associated with each. Options include: 1) seeking renewed financial commitment from NRC Senior Executive Committee to ensure a base transfer from NRC's operations vote to this program that is sufficient to cover expected due increases and currency fluctuations for the next five years (acknowledging that the BIPM operates on a four year budgetary planning cycle); 2) exploring possible cost savings identified in this Report. Accepted
NRC will explore opportunities to seek renewed financial commitment to the BIPM that clarifies the Government's intentions regarding the mandatory and discretionary contributions and will ensure that these intentions are incorporated in the renewed Terms and Conditions.
NRC will consult with TBS and/or Foreign Affairs and International Trade Canada DFAIT regarding best practices for managing the financial uncertainty that arises from the fact that the annual contributions are expressed in Euros rather than Canadian dollars.
NRC will explore opportunities for cost savings.
Director General, NRC-INMS and Director General, NRC Strategy and Development Branch (NRC-SDB) December 1, 2010 Revised TBS Terms and Conditions that specify the Canadian Government's intentions with respect to support of the activities of the BIPM, including the CIPM Mutual Recognition Arrangement, for the period 2013-2016 and eliminate the financial uncertainty related to currency fluctuations.
Recommendation 5: Given the conflict of interests that exists in the roles of the DG NRC-INMS with respect to the BIPM, it is recommended that an alternative Canadian Government representative be identified to attend future meetings of the Conférence Générale des Poids et Mesures (CGPM). This representative should have the necessary voting authority, be fully informed of the constraints set by the Terms and Conditions for Class Grants to International Affiliations in terms of budget increases, and should have sufficient knowledge of the mandate and business of the BIPM to successfully represent Canada at the CGPM. Accepted
NRC will explore opportunities to eliminate the apparent conflict of interest by having someone other than the Director General of NRC-INMS (who is a member of the CIPM) appointed as Head and voting member of the Canadian delegation at the 2011 Conférence générale des poids et mesures (CGPM). Options include: a senior official from NRC (e.g., the Secretary General or the DG, NRC-SDB); a senior official from DFAIT; or a senior official from Industry Canada.
Seeking representation from Industry Canada or DFAIT to lead the Canadian delegation to CGPM meetings and vote on the BIPM's budget.
Development of written instructions to the Head of the Canadian delegation to the 2011 meeting of the CGPM that clearly set out the financial constraints for the purposes of voting on the BIPM dotation for the period 2013-2016.
NRC Secretary General and/or Director General, Strategy and Development Branch December 1, 2010 Transparency in Canada's commitment to the BIPM program.


Appendix A – List of Affiliations

Table 3 provides the full list of current affiliations held by NRC on behalf of Canada and the Canadian science and engineering community, as well as the organizations that have entered into partner agreements with NRC in order to support Canada's affiliation through the creation and maintenance of a Canadian National Committee (CNC) made up of leading researchers in the fields of relevance to each affiliation. The responsibilities of the partner organizations and the CNCs are laid out in the partner agreement, and include such things as: ensuring the promotion of Canadian contributions to international decision-making; disseminating important scientific knowledge and information to Canadian stakeholders; and nominating Canadian delegates to attend meetings of the international organization.

Table 3: List of Affiliations by Type, Organization and Canadian Partner
Type International Organization Canadian Partner
1 Intergovernmental Organization Bureau International des Poids et Mesures (BIPM) N/A [NRC Institute for National Measurement Standards (NRC-INMS) has responsibility for this affiliation]
2 Non-governmental Organization International Council for Science (ICSU) NRC, with the Committee on International Science, Engineering and Technology (CISET) serving as the Canadian National Committee for ICSU
3 ICSU Union International Astronomical Union (IAU) Canadian Astronomical Society (CASCA)
4 ICSU Union International Geographical Union (IGU) Canadian Association of Geographers (CAG)
5 ICSU Union International Mathematical Union (IMU) Canadian Mathematical Society (CMS)
6 ICSU Union International Union for Quaternary Research (INQUA) Canadian Quaternary Association (CANQUA)
7 ICSU Union International Union of Biochemistry and Molecular Biology (IUBMB) Canadian Society of Biochemistry and Molecular and Cellular Biology (CSBMCB)
8 ICSU Union International Union of Crystallography (IUCr) NRC Biotechnology Research Institute (NRC-BRI)
9 ICSU Union International Union of Geodesy and Geophysics (IUGG) Canadian Geophysical Union (CGU)
10

11
ICSU Union International Union of History and Philosophy of Science (IUHPS) – Division of History of Science (DHST) & Division of Logic, Methodology and Philosophy of Science (DLMPS) Canadian Society for the History and Philosophy of Science (CSHPS)
12 ICSU Union International Union of Nutritional Sciences (IUNS) Canadian Society for Nutritional Sciences (CSNS)
13 ICSU Union International Union for Pure and Applied Biophysics (IUPAB) Biophysical Society of Canada (BSC)
14 ICSU Union International Union of Pure and Applied Chemistry (IUPAC) NRC Steacie Institute for Molecular Sciences (NRC-SIMS) [From 1993 to November 2009]; NRC Institute for Chemical Process and Environmental Technology (NRC-ICPET) [currently]
15 ICSU Union International Union of Pure and Applied Physics (IUPAP) NRC Institute for Microstructural Sciences (NRC-IMS)
16 ICSU Union International Union of Basic and Clinical Pharmacology (IUPHAR) The Pharmacological Society of Canada (PSC)
17 ICSU Union International Union of Physiological Sciences (IUPS) Canadian Physiological Society (CPS)
18 ICSU Union International Union of Psychological Science (IUPsyS) The Canadian Psychological Association (CPA)
19 ICSU Union International Union of Theoretical and Applied Mechanics (IUTAM) The Canadian Society for Mechanical Engineering (CSME)
20 ICSU Union International Union of Radio Science (URSI) NRC Herzberg Institute of Astrophysics (NRC-HIA)
21 ICSU Union International Union of Geological Sciences (IUGS) Canadian Federation of Earth Sciences (note: IUGS payments only started in 2008; application for support captured in minutes of June 2008 CISET meeting; NRCan had paid before and declined to continue to pay)
22 ICSU Interdisciplinary Body / Joint Initiative Committee on Data for Science and Technology (CODATA) NRC Canada Institute for Scientific and Technical Information (NRC-CISTI)
23 ICSU Interdisciplinary Body / Joint Initiative Committee On Space Research (COSPAR) Canadian Space Agency (CSA)
24 ICSU Interdisciplinary Body / Joint Initiative Scientific Committee on Oceanic Research (SCOR) Fisheries and Oceans Canada (DFO)
25 ICSU Interdisciplinary Body / Joint Initiative Scientific Committee On Solar-Terrestrial Physics (SCOSTEP) Canadian Space Agency (CSA)
26 ICSU Interdisciplinary Body / Joint Initiative World Meteorological Organization (WMO)-ICSU-Intergovernmental Oceanographic Commission (IOC) World Climate Research Programme (WCRP) Canadian Foundation for Climate and Atmospheric Sciences (CFCAS)
27 Independent, non-profit, professional organization International Commission on Illumination (CIE) NRC Institute for National Measurement Standards (NRC-INMS)
28 International non-profit association of national organizations International Association for the Properties of Water and Steam (IAPWS) Candu Owners Group
29 International Association of National Adhering Bodies and Associate Members International Permafrost Association (IPA) Geological Survey of Canada; Natural Resources Canada (GSC / NRCan)
30 International Professional Body of National Member Societies and Individual Members International Society for Soil Mechanics and Geotechnical Engineering (ISSMGE) Canadian Geotechnical Society (CGS)


Appendix B – Logic Model

Appendix B - Logic template

Appendix B - Logic Model

Footnotes

Footnote 1

Mobilizing Science and Technology to Canada's Advantage (accessed 7 June 2010)

Return to footnote 1 referrer

Footnote 2

Ibid.

Return to footnote 2 referrer

Footnote 3

NRC Corporate Records

Return to footnote 3 referrer

Footnote 4

BIPM website (accessed June 4, 2010).

Return to footnote 4 referrer

Footnote 5

ICSU website (accessed June 4, 2010).

Return to footnote 5 referrer

Footnote 6

Review of websites of international scientific organizations (accessed between March and June 2010).

Return to footnote 6 referrer

Footnote 7

NRC (2002). Committee on International Science, Engineering and Technology (CISET): Terms of Reference

Return to footnote 7 referrer

Footnote 8

NRC Corporate Records, Financial Data and Key Informant Interviews.

Return to footnote 8 referrer

Footnote 9

Adapted from NRC (2005). Integrated Results-Based Management Accountability Framework (RMAF) and Risk-Based Audit Framework (RBAF) for International Affiliations; following logic model update exercise.

Return to footnote 9 referrer

Footnote 10

Note: Rating of "Fully Met", "Partially Met" or "Not Met" is based on the overall analysis of data presented in support of each judgment criteria.

Return to footnote 10 referrer