ARCHIVED - Evaluation of the Indoor Air Quality Theme of the Clean Air Agenda

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Executive Summary

This report presents the findings, conclusions and recommendations of an evaluation of the Indoor Air Quality (IAQ) Theme of the Clean Air Agenda (CAA). The evaluation examines the IAQ Theme’s relevance, effectiveness, and efficiency/economy during the period from 2007/2008 to 2009/2010. The research for this evaluation was conducted between November 2009 and February 2010.  

Description of the IAQ theme 

The IAQ theme is one of eight themes found under the CAA, an interdepartmental initiative with $1.9 billion in funding over the 2007/2008 to 2010/2011 timeframe. The IAQ theme has been allocated a total of $23 million over this four-year time frame. The IAQ theme is a horizontal initiative between Health Canada (HC) and the National Research Council (NRC), including two programs: the Indoor Air Research and Development (R&D) Initiative (led by NRC; receiving $8 million over the four-year timeframe) and the Radon Program (led by HC; receiving $15 million over the four-year timeframe). It is important to note that the radon program has components in more than one CAA theme. Some radon work is carried out under the Clean Air Regulatory Agenda (CARA) – led by EC. Only the IAQ-related radon work will be presented in this evaluation report. The work conducted under the CARA is not reported on in this evaluation. 

The objective of the IAQ theme is to develop and apply knowledge for reducing or eliminating risks from natural and man-made indoor-air pollutants. This objective is expected to be achieved through a focus on the research, evaluation and testing of measures to improve indoor air quality and to identify and address specific risks to human health presented by radon gas in indoor environments.Footnote 1

Methodology

This evaluation was conducted in accordance with the 2009 Government of Canada Policy on Evaluation and addresses its core evaluation issues: continued need for the program, alignment with Government priorities, consistency with federal roles and responsibilities, achievement of expected outcomes, and demonstration of efficiency and economy. 

The evaluation includes the use of multiple lines of evidence and complementary research methods as a means to ensure the reliability of the information and data collected.  Five main lines of inquiry were used for this evaluation:

  • document and literature review;
  • interviews;
  • survey;
  • analysis of program data; and
  • review of financial data.

Relevance 

The IA R&D Initiative and the Radon Program were aligned with federal government priorities when the IAQ theme was developed and continue to be aligned with Health Canada, National Research Council and Environment Canada priorities. Both the IA R&D Initiative and the Radon Program continue to reflect the needs of Canadians. Provinces, non-governmental organizations (NGOs) and at least one Canadian university are conducting work that complements the Radon Program; however there does not appear to be any duplication of either the IA R&D Initiative or the Radon Program. 

Effectiveness 

Because this evaluation is being conducted during the third year of a four-year program and many of the outcomes are not anticipated until the end of the four-year period (particularly for the IA R&D Initiative), the evaluation focuses on examining progress made to date. 

Progress on Achievement of IA R&D Initiative Outcomes 

Funding was provided under the IA R&D Initiative for three primary components: 1) Assessment of the Impacts of Ventilation on Health ($4.8 million over four years), 2) Evaluation of IAQ Technologies and Solutions ($1.6 million over four years), and 3) Establishment of a National Advisory Committee Focusing on Indoor Air, now called the Canadian Committee on IAQ and Buildings ($1.6 million over four years). 

Components 1 and 3 are expected to lead to increased knowledge of risks, health impacts and mitigation strategies related to indoor air quality. Component 2 is expected to lead to increased knowledge and support for the development of technological solutions for improved IAQ. 

The IA R&D Initiative appears to be generally on track towards achieving expected outcomes. The majority of targets for Components 1 and 2 are expected to be met, with a few minor delays.  More challenges have been experienced in Component 3 and these challenges may have an impact on committee members’ ability to achieve expected objectives. The committee members come from diverse sectors and are not necessarily familiar with federal government administrative processes (e.g., government contracting process, getting public opinion research approval). 

Despite the minor delays, parties outside of NRC (including CMHC and an international expert) have already started to express an interest in the knowledge that is expected to be generated by the IA R&D Initiative. A variety of lessons learned were noted by interviewees, particularly related to improving the effectiveness of the National Advisory Committee (e.g., need for an administrative support function for the committee). This multi-sector collaborative approach that permeates the IA R&D Initiative is viewed by interviewees as a best practice. 

 Progress on Achievement of Radon Program Outcomes 

Funding was provided under the Radon Program for three primary components: 1) Mapping of Radon Zones and Affected Populations ($8.4 million for four years), 2) Testing and Remediation of Radon in Federal Buildings Located in High Risk Radon-Prone Areas ($3.6 million for four years) and 3) Radon Education and Awareness Program ($3 million for four years). 

Components 1 and 2 are expected to lead to increased knowledge of risks, health impacts and mitigation strategies related to radon while Component 3 is expected to lead to improved awareness of health risks and causes of elevated radon and strategies to improve it.

The mapping of radon zones and affected populations is mostly being implemented as planned. Targets for this component have been, for the most part, met or exceeded. Component 2 (testing and remediation of federal buildings) is off track due to a number of barriers and external factors. It was targeted that 60 percent of federal buildings would be tested for radon levels by the end of fiscal year 2009/10 but it is anticipated that 17 percent of buildings will actually be tested. Barriers encountered include: getting buy-in from departments at the senior level to conduct Federal Building Testing (FBT) in their buildings, a lack of interest by those at the operational level even if FBT was approved, fewer buildings meeting testing criteria than initially anticipated, and differing expectations on who should be responsible for distributing and collecting radon detectors in federal buildings. Program managers also experienced a number of barriers implementing Component 3 (radon education and awareness program). Program managers developed a broad based radon communication strategy/plan which was approved by senior management for dissemination.  However, approval was subsequently revoked and the plan could not be rolled out. As a result, program managers used alternative methods to disseminate the information. The limited performance data available suggests that awareness levels have increased but that alternative methods used have not resulted in the expected level of increased awareness.

 Lessons learned were found by comparing Canada’s Radon Program to those in other countries. The key difference that should be noted in comparing Canada’s program to those in other countries is that Canada is in its third year of implementation, whereas comparison programs operating in other countries have been in place for, in many instances, twenty years.  In sum, the key lessons that Canada can learn from are that comparison countries have radon specific public awareness campaigns, access to data on when a house is tested for radon, the results of the test,  and whether or not remedial action has been taken). 

Efficiency and Economy

Efficiency and Economy of the IA R&D Initiative 

The IA R&D Initiative appears, for the most part, to be on track towards achieving objectives and has remained close to budget at the Initiative level. One of the components (Assessment of the Impacts of Ventilation on Health) of the Initiative has drawn resources away from the other two components, suggesting that it is not being delivered as efficiently as planned. Program staff of the remaining two components are attempting to meet objectives with fewer resources than originally planned. The reduced resources for the evaluation of IAQ Technologies and Solutions component does not seem to be affecting the achievement of targets but the lack of resources for the Canadian Committee on IAQ and Buildings is inhibiting the committee’s ability to meet targets. However, the concentration of resources has been allocated to activities that will play the greatest role in generating new knowledge, particularly with respect to the risk, health impacts and mitigation strategies related to IAQ. 

There is a perception that efficiency of the IA R&D Initiative cannot be improved as NRC is undertaking a large volume of tasks for the amount of funding received and there are no viable alternatives to the current delivery model. 

Efficiency and Economy of the Radon Program 

Fewer than anticipated resources (financial and human) have been directly expended on Radon Program components. Radon Program funds were used to support lab operational and overhead costs, the regions, the Director General’s (DG’s) office, accommodation (PWGSC), and corporate service costs.  Interim targets for Component 1 have been met or exceeded, for the most part, with fewer resources (financial and human) than anticipated suggesting that the mapping component is being done efficiently. Targets for Component 2 are far from being met and more resources than anticipated have been expended. This suggests that federal building testing has not been efficient. Fewer than budgeted resources were directly allocated to Component 3 (Education and Awareness) and many of the output targets have been met or exceeded meaning that certain aspects of this component are being delivered efficiently. However, resources were expended developing a radon specific public awareness campaign that was approved but never granted permission to be rolled out. Resources were therefore expended but were not used to help affect the intermediate outcome of improving public awareness of health risks and causes of elevated radon and strategies to improve it. The limited awareness data that is available suggests that interim awareness targets have not been met. The reduction of dedicated resources (human and financial) and barriers encountered by Radon Program managers has had an impact on the economy of the awareness building component. 

Perceptions of the economy of the Radon Program vary, but just over half of interviewees felt that the program was economical. Suggestions on how to improve the efficiency of the Radon Program vary but tended to concentrate on increasing education and awareness and federal building testing. 

Recommendations

The following recommendation should be addressed at the Theme level: 

Recommendation 1: Bring theme level partners together to start planning for the future beyond 2010/11, ensuring that 'lessons learned' identified in this evaluation are discussed.  Determine how Theme partners would like to proceed and then broaden the discussion to include CAA operational managers and other CAA stakeholders to discuss whether there is a rationale for going forward under one agenda, with certain components of the CAA or not at all. 

Management Response and Proposed Actions: Accepted.
NRC has been meeting with two branches of Health Canada (the Water, Air and Climate Change Bureau (WACCB) and Radiation Protection Bureau (RPB)) regularly and are working on a common plan for future funding. Synergies and leverages between the proposed activities of the 3 branches are being discussed. 

NRC and Health Canada are also working with Environment Canada to best position the Theme collaborators within the overall Clean Air Agenda. 

Lessons learned noted in the evaluation will be considered in these discussions, particularly as they relate to next steps for the Canadian Committee on IAQ and Buildings.

The following recommendations for the IA R&D Initiative should be addressed: 

Recommendation 1: Ensure Intellectual Property issues with INSPQ are resolved and MOU with provincial partner is signed. 

Management Response and Proposed Actions: Accepted.
The IP issues have been resolved, and the MOU with the provincial health partner, INSPQ, has now been signed 

Recommendation 2: Put in place an NRC staff member who is dedicated to supporting the Canadian Committee on IAQ and Buildings (e.g., coordination, administration, follow-up, assistance with government processes) and ensure funds are available to conduct planned research activities. 

Management Response and Proposed Actions: Partially Accepted.
The level of effort to support the CCIAQB is not constant enough to require a staff member dedicated to support the Committee. The resource currently acting as the Secretary to the Committee will be able to access additional NRC-IRC staff on an as-needed basis when more support is required, including during the procurement process. 

In 2010/11 more funding has been allocated to this activity and future funding applications will reflect the additional resources required. 

The next CCIAQB meeting will include an agenda item as to whether the committee members feel the committee is now sufficiently supported. 

Recommendation 3: Revisit program plans and determine if there are alternate methods of collecting information that has been delayed by Public Opinion Research restrictions, or if other information could be used. 

Management Response and Proposed Actions: Accepted.
NRC-IRC and NRC Corporate Communications Branch have revisited definition of public opinion research and determined that the desired collection of information (Awareness Survey of Building Operators) is no longer considered public opinion research. Request for quotes has been sent to potential contractors. 

The following recommendations for the Radon Program should be addressed:

Recommendation 1: Review communication practices with delivery partners to ensure clear communication channels are in place (e.g., clear points of contact at HC) and that messages are consistent (e.g., regarding the assistance that HC desires from stakeholders).

Recommendation 2: Ensure the performance data is being tracked against targets outlined in the IAQ RMAF including IAQ funded regional and provincial outputs.


Recommendation 3: If there is a continued absence of approval to conduct POR for determining awareness levels; investigate whether alternate methods could be used to capture this change


Recommendation 4: Ensure that comprehensive financial data is being tracked against the budget outlined in the IAQ RMAF and assign responsibility for this tracking.

Footnotes (1)

Footnote 1

Indoor Air Quality Theme Results-based Management and Accountability Framework (HC/NRC, 2008)

Return to footnote 1 referrer

Date modified: