ARCHIVED - Evaluation of the NRC Herzberg Institute of Astrophysics' Implementation of the Long Range Plan for Canadian Astronomy and Astrophysics
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
The National Research Council (NRC) Act mandates NRC with the operation and administration of astronomical observatories established or maintained by the Government of Canada. This is accomplished through the NRC Herzberg Institute of Astrophysics (NRC-HIA), which not only operates and manages national observatories for ground-based astronomy (the Dominion Astrophysical Observatory (DAO) in Victoria, British Columbia and the Dominion Radio Astrophysical Observatory (DRAO) in Penticton, British Columbia) but manages Canada's contribution payments to major international observatories to ensure access to these facilities by the Canadian scientific community.
In 1998, NRC, along with the Natural Sciences and Engineering Research Council (NSERC), convened a panel of seven astronomical experts from Canada and the United States to develop a comprehensive view and decadal program for astronomy and astrophysics in Canada. The resulting document, completed in 1999 and referred to as: the Long Range Plan for Canadian Astronomy and Astrophysics (LRP), recommends projects in which Canada should participate over the course of 2001-2015. The LRP notes that responsibility for its successful implementation rests with three federal government organizations: NRC, NSERC and the Canadian Space Agency (CSA).
The LRP set out the justification for Canada to invest a total of $362.3M in astronomy and astrophysics between 2001-2015 by allocating funds to these three organizations. The federal government did not however, provide the full requested funding as laid out in the LRP document. Funds received by NRC in 2002-2003 and 2003-2004 represent 75% of the LRP recommended expenditures by NRC in the first five years of implementation.
In fiscal years 2005-2006 and 2006-2007, the elements of the LRP supported by NRC-HIA in the first five years of implementation were evaluated. The primary reasons for conducting the evaluation at this time were to collect information on the progress of the implementation of the LRP at NRC-HIA to date, including lessons learned and best practices and to support the Institute's future planning and on-going management. In addition, the evaluation provided information to support decisions around funding renewal.
The Terms of Reference for the evaluation were approved by NRC Senior Executive Committee (SEC) in March 2005. The evaluation covered the period 2002-2003 to mid-2006-2007 inclusive. The evaluation focused on gathering information on NRC-HIA's implementation of the first four and a half years of the LRP; assessing whether efforts are unfolding as planned; uncovering obstacles, barriers or unexpected opportunities; and identifying mid-course adjustments and corrections through the recommendations. The evaluation also examined whether any early outcomes were evident and identified impacts to date. The key methodologies used to address the evaluation issues included review of documents, review of administrative/performance data and key informant interviews.
Based on the evaluation study conducted from October 2005 to June 2006, the following key evaluation findings and recommendations were identified.
Summary of Findings and Recommendations – Relevance
NRC-HIA's involvement with the LRP is consistent with the role ascribed in the NRC Act. There is a legitimate and necessary role for NRC-HIA to play in Canadian astronomy. There was consensus amongst interviewees that it should be the federal government that takes the lead in large scale projects, such as astronomy, on behalf of the country.
Interviewees identified astronomy's need for large, multi-disciplinary teams of scientists, engineers and technicians to support these new facilities and felt that NRC-HIA has access to these types of resources. In addition, the opinion of those consulted is that astronomy, and by extension NRC-HIA's work, is in the public interest.
NRC-HIA's implementation of the LRP is relevant to industrial and commercial development in Canada. The technologies and know-how developed for and through astronomy can find many important applications in industry.
NRC-HIA's implementation of the LRP is relevant to both the Canadian and international astronomy communities. This relevance derives from the process by which NRC-HIA activities were planned in relation to the LRP. Within the LRP, the roles of each stakeholder are well defined and mutually supportive. The LRP was developed by and for the Canadian astronomy community and emphasizes Canada's role within the larger international community.
Summary of Findings and Recommendations – Success
The examination of success in this evaluation focused on early outcomes, recognizing that the LRP is still in its early years of delivery and will take 15 years to fully implement. NRC-HIA is delivering on its commitments within the LRP, given its resource constraints. Concerns were expressed within the astronomy community as well as at NRC-HIA about the funding situation. Stakeholders from within and outside of the organization commented that not all activities outlined in the LRP were funded to the levels requested and therefore expectations of success should be modified accordingly.
NRC-HIA, on behalf of the federal government, not only operates and manages national observatories for ground-based astronomy but manages Canada's contribution payments to major international observatories to ensure access to these facilities by the Canadian scientific community. The LRP recommended that funds be allocated to augment existing Institute activities and therefore it was difficult for the evaluation to separate NRC-HIA LRP-related activities and performance from other Institute activities and performance.
Research activities and capacity have improved at NRC-HIA since the initiation of the LRP, notably in terms of assured participation in existing international projects, the design of future state-of-the art projects (e.g., ALMA, TMT, SKA), the number of researchers hired directly by the Institute and the research they have undertaken. However, research capacity at NRC-HIA in both instrumentation and observation may be threatened in the near future by the retirement of top researchers. In addition to staff hirings, LRP funds have provided for training and hiring of additional students and post-doctoral fellows at NRC-HIA.
LRP funds have allowed NRC-HIA to provide increased support to the astronomy community both in terms of access to research infrastructure with respect to increased telescope access and increased data and data management services.
NRC-HIA's involvement in LRP projects has positioned it to ensure Canadian expertise is recognized for its experience and capabilities to supply technology, particularly in terms of instrumentation, to the international telescope projects. A net economic benefit of $4.31 million and a 67% return on investment could be attributed to Canada's participation in international telescope projects since the LRP was initiated.
LRP funds were not specifically allocated for public outreach, however, interviewees nonetheless highlighted some of NRC-HIA's outreach activities.
Recommendation 1: NRC-HIA should develop a succession plan to ensure that research capacity at the Institute is not weakened with upcoming potential retirements of staff.
Management Response and Proposed Actions: Aging demographics and global competition for scientific talent are challenges for NRC research institutes, including NRC-HIA. As noted in the evaluation, the Institute took advantage of LRP funding to recruit new staff, including two outstanding scientists funded through the NRC New Horizons Program, and to provide a research experience to a number of students and post-doctoral fellows.
In response to the recommendation, institute groups are creating staffing outlooks for the next five to ten years which will be integrated into an Institute hiring plan. The plan will emphasize global recruitment at the entry level, an area where NRC has a slight competitive advantage.
Recommendation 2: There should be increased emphasis and efforts at NRC-HIA to systematically identify Canadian companies to be included in the work undertaken by the Institute for large international telescope project funded through the LRP. Progress made in Penticton, through NRC-HIA's relationship with the Okanagan Research and Innovation Centre (ORIC) should be formally monitored and if deemed successful, NRC-HIA should consider a similar approach in Victoria.
Management Response and Proposed Actions: This recommendation concerns two separate areas where NRC-HIA works with companies: in the identifying, notifying and working with Canadian companies on LRP-related opportunities, and in transferring technologies to the private sector for other applications.
With respect to the first part, the Institute faces a number of challenges: building industrial capacity and interest, addressing communication and legal impediments, and meeting stringent international performance standards. The considerable efforts that were made to identify Canadian expertise are documented in this report. As a result, Canadian companies that were successful in winning high technology contracts associated with the EVLA and ALMA projects demonstrated the impact of LRP funding and the effect of exacting requirements of international astronomy projects on pushing a company's expertise to the leading edge.
The Institute intends to continue pursuing avenues for industrial outreach related to international projects. Bid packages are being prepared for the ALMA Band 3 receiver production, the major deliverable to ALMA from NRC-HIA. NRC-HIA will position itself as an integrator and component contractor to Canadian companies while taking on the more specialized testing and quality control responsibilities. NRC-HIA will make an effort to win bids on components of projects that have more potential volume and market applications (such as the Band 1 receivers of ALMA) and that might attract more interest from companies. The institute will continue to work with, and seek the advice of, the Canadian SKA Consortium, the organization that is directing SKA development, and which includes industry representation.
With respect to non-astronomy applications of associated technologies, NRC-HIA initiated the ORIC "experiment" in Penticton, which has achieved some success in a relatively short period of time. The institute recently commissioned an assessment of HIA-Victoria technology with precisely this aim. Formal monitoring of ORIC is present to some extent by virtue of the Agreements in place. As well, NRC acts as the secretary to the ORIC Board of Directors. NRC HIA will also continue to build on its relationship with the NRC Industrial Research Assistance Program (NRC – IRAP) in identifying technology transfer opportunities for SMEs.
Summary of Findings and Recommendations – Scientific Excellence
NRC-HIA's research and development of astronomical tools, instruments, and equipment is considered to be excellent from the perspective of both national and international peers and stakeholders. NRC-HIA's participation in and contribution to future facilities is valued by the international scientific community, however, NRC-HIA's expertise in astronomical research is perceived, by external stakeholders, to be weakening.
Recommendation 3: NRC-HIA should take into consideration the perceived concerns regarding the weakening of astronomical research at the Institute and, if determined to be valid, should take action to remedy the situation.
Management Response and Proposed Actions: It is the Institute's view that this perception may no longer be valid: NRC-HIA has made a number of excellent junior recruitments, whose visibility is increasing as their research publications become more numerous and more cited. The two New Horizons candidates are considered to be at the level of Canada Research Chair Tier I recruitments at Canadian Universities.
NRC will, however, attempt to quantify the relative performance of its Research staff to a representative external sample through the implementation of a system for monitoring citation rates and/or the impact factor of the journals accepting NRC-HIA publications. The results will be communicated to the astronomical community. An integrated hiring plan is to be developed (see Recommendation 1).
Summary of Findings and Recommendations – Delivery and Cost Effectiveness
NRC-HIA appears to have made effective use of the resources provided to implement its responsibilities laid out in the LRP, although not all LRP funds were tracked separately from NRC-HIA A-base funds. Additional resources are needed to complete the obligations set out for NRC-HIA in the LRP in future years. The opinion of external interviewees is that NRC-HIA has done an excellent job with its limited resources.
Overwhelmingly, NRC-HIA activities are seen as complementary to other astronomy funding and research programs in Canada. This is not surprising as the Institute has closely followed the plans set out for it (as allocated resources would allow) in the LRP which was developed to be a unified vision for astronomical research in Canada. The plan represented the consensus of the Canadian astronomical community and set out plans to ensure complementarity amongst the various players.
NRC-HIA's internal and external communications of LRP related projects is considered good. Improvements could be made if more resources were available. Some industry interviewees felt that communication with NRC-HIA regarding potential commercial opportunities could be improved.
Recommendation 4: NRC should secure funds for the remainder of the work assigned to NRC-HIA as outlined in the LRP and Mid-term Review documents to ensure that positive impacts continue. Significant impacts of NRC-HIA's implementation of the LRP to date include strengthened research capacity at the Institute and increased opportunities for Canadian firms to participate in providing technology to international astronomical facilities.
Management Response and Proposed Actions: Action toward this goal has been ongoing for more than one year, and is a high priority for NRC. The Director General of HIA is working with the Director General of Strategy and Development Branch and the NRC Senior Executive Committee on developing a case for continued funding, and continues to interface with the Coalition for Canadian Astronomy to ensure its support.
Recommendation 5: The next round of funding received to implement the LRP should be tracked separately in NRC-HIA's financial system.
Management Response and Proposed Actions: The LRP is a complex array of international projects, each with its own reporting and cash management requirements, combined with initiatives related to improving the research capability of the Institute. As noted in the report, NRC-HIA aligned its own efforts with those of the LRP and leveraged the LRP funding with its own A-base as well as other funds from within NRC. Where full LRP funding was not available, certain activities were augmented by NRC-HIA A-base funds (chiefly salaries). All finances for individual projects were tracked in detail within the Institute and that will continue to be the case.
However, in the future, the cash flow from any new funding received for the advancement of the LRP will be tracked at the top level to the extent allowed by the NRC Financial system. This will provide a more transparent picture of the interplay between NRC-HIA A-base funding and the LRP funding to better quantify the impact of the LRP. The LRP projects will continue to be tracked.
Summary of Findings and Recommendations – Opportunities and Future Directions
The final area examined by the evaluation was opportunities and key lessons learned that could be adopted by NRC-HIA and the federal government at large in its next phase of LRP implementation.
There was an overwhelming sentiment amongst interviewees, particularly from external stakeholders, that there is a need for more funding and more forthcoming commitment by the Canadian federal government regarding implementation of the remainder of the LRP.
The funding mechanisms set up by the federal government to allocate monies to large scale projects was seen as problematic. Another aspect of funding that was highlighted as a challenge was the fact that project timelines and funding timelines are usually not in alignment. More often than not the required project spending profile does not fit with the federal government's fiscal year causing strain on NRC and NRC-HIA's budget at different times throughout the year.
Some ideas were provided by interviewees for consideration to improve results in the area of fostering industry involvement. These included finding an effective way of identifying potential commercial partners and companies capable and willing to accept to shoulder some of the costs and efforts required to successfully have technologies and know-how transferred to them. NRC-HIA obtaining a better understanding of the markets for potential applications and a high level review of the NRC contracting process for the control of intellectual property and technology transfer.
To properly understand the economic and industrial impacts of the LRP, a return on investment analysis and benefit-cost analysis of the implementation of the entire plan should be considered sometime in the future.
There is a desire within Canada, among Canadian university researchers interviewed, to have more formal communication mechanisms between NRC-HIA staff and their university colleagues.
Recommendation 6: The difficulties of financially managing LRP long-term "Big Science" projects commitments within a five year planning cycle should be reviewed in light of a recommended framework by the Office of the National Science Advisor (ONSA)Footnote 1. This proposed framework should be reviewed and discussed with the ONSA to determine how LRP projects might benefit and how it could be improved to better address some of the lessons learned during this evaluation and the needs of all Canadian "Big Science" projects in the future.
Management Response and Proposed Actions: With its many national facilities, NRC has a strong interest in seeing the "Big Science" issue for Canada resolved. NRC provided input to the original framework, and participated in a subsequent workshop with 40 distinguished community leaders. As well, it sought input from its governing council on the proposed framework, and participated in a workshop at the Canadian Light Source devoted to this issue.
NRC, with all relevant agencies and organizations, is also participating in a working group sponsored by the Coalition for Canadian Astronomy to examine the barriers that were impeding progress toward implementing the LRP. The Working Group issued a report and circulated it to the participants and relevant stakeholders for further feedback.
NRC intends to continue the dialog with all sectors of the astronomical community, including the ONSA, to address this issue and seek solutions.
Recommendation 7: If a second round of LRP funding is awarded, NRC should consider conducting a return on investment analysis and benefit-cost analysis, in coordination with other LRP-funded federal government organizations. If undertaken, these studies should be completed in advance of the next evaluation.
Management Response and Proposed Actions: NRC-HIA will examine whether it is opportune to initiate a follow-on study similar to that done in 1999 by the ARA group under contract to KPMG. This study provides a methodology template and sets a benchmark.
Keeping in mind that the timeline required to assess benefits beyond the immediate economic impacts might prevent a full review before the next evaluation, NRC will look at the opportunity to conduct a return on investment analysis and benefit-cost analysis. Consultation with other Agencies will be undertaken if a decision to proceed is taken.
- Footnote 1
A Framework for the Evaluation, Funding and Oversight of Canadian Major Science Investments, Office of the National Science Advisor, January 2005
Report a problem or mistake on this page
- Date modified: