ARCHIVED - Evaluation of the Interactive Language Technologies (ILT) Group - NRC Institute for Information Technology (NRC-IIT)

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Executive Summary

The following summarizes the findings of the evaluation of the Interactive Language Technologies (ILT) Group, a research entity of the NRC Institute for Information Technology (NRC-IIT). The evaluation addresses, among other things, the Group's contribution to the creation and operation of the Language Technologies Research Centre (LTRC), whose objective is to support Canadian research in the language industry and the development of new solutions in language technology.

The ILT Group is an initiative of the Action Plan for Official Languages (APOL) released in March 2003 and is one of the main components of the LTRC. The APOL aims to strengthen Canada's linguistic duality by addressing three priority areas: education, community development and an exemplary public service. The APOL was meant to be based in part on a strong language industry consisting of the following three sectors: translation, including localization and interpretation; language training (language learning); and language technologies. However, a lack of research and development (R&D) capacity in the language industry identified prior to the APOL's launch proved to be a significant barrier to the development of the industry and thus implementation of the Plan.

Under the Action Plan, NRC received a total of $10 million ($2 million per year for the period from 2003-2004 to 2007-2008) to:

  • create a research program in language technologies; and
  • create, in partnership with the Translation Bureau of Canada (TBC) and the Université du Québec en Outaouais (UQO), a Language Technologies Research Centre (LTRC).

The funding was granted recurrently by Treasury Board Secretariat to ensure the long-term stability of NRC research activities.

The ILT Group is one of eight entities within NRC-IIT, which itself is one of 19 NRC institutes and programs. NRC-IIT manages research programs in New Brunswick and the National Capital Region (Ottawa and Gatineau). By the time of the evaluation, the ILT Group had reached its maximum capacity in terms of resources with nine researchers and developers as well as three support staff (administration, commercial development and information systems support) for a total of 12 full-time employees, in addition to a variable number of individuals working part time (co-op and graduate students, postdoctoral fellows and visiting researchers).

In conjunction with the implementation of the research program by NRC, the three partners endeavoured to create the LTRC. From 2003 to 2005, the LTRC operated in the form of a consortium of the three founding partners (NRC, UQO and TBC) under the terms of a memorandum of understanding between the three parties. In spring 2005, the initial LTRC concept acquired independent legal status through the creation of a non-profit organization (NPO) serving as a co-ordinating body for all of the LTRC's partners.

Objectives of the Evaluation and Methodology

The evaluation of the ILT Group was designed to meet the following two objectives:

  • Fulfil the commitment made to Treasury Board, specifically that NRC would evaluate the research program, i.e., the ILT Group, either through a formative evaluation after three years of operation or through a summary evaluation after four or five years; and
  • Fulfil the evaluation requirements under the APOL, on the basis of which funding for the ILT Group was allocated and which provides for a contribution to the LTRC prior to the withdrawal or renewal of the Plan.

This evaluation is concerned primarily with the activities generated by the $10 million allocated by Treasury Board Secretariat under the APOL. It is therefore limited to the ILT Group's activities and outcomes. However, since the program's background documents specified that the objectives of the funding included the creation and operation of the LTRC, and the latter relies not only on the partnership between the three founding organizations (NRC, TBC and UQO) but also on the role now played by the LTRC-NPO, one part of the study is devoted to the LTRC as a whole, specifically the sections on the implementation and relevance of the program. The evaluation mandate did not include the LTRC concept or the LTRC-NPO, however. Such an evaluation would have been conducted from a different optic. Consequently, aspects such as the contribution of the other partners to the creation and operation of the LTRC were not assessed.

This study focused on the following three evaluation elements: outcomes achieved; design, implementation and cost-effectiveness; and relevance. The data forming the basis for the evaluation were gathered using the following three primary methods: a review of relevant documents, interviews with key representatives (n=23) and a review by a panel of five experts in language technologies consisting of three members from Canada and two from abroad, in order to assess the level of quality and excellence of the ILT Group's R&D activities.

Conclusions and Recommendations – Level of Achievement of Outcomes

The level of achievement of outcomes was assessed primarily on the basis of the logic model and performance measurement plan for the ILT Group. It was noted first of all that the range and quality of the ILT Group's research activities and outputs are very satisfactory, considering the few years the Group has been in existence. Projects such as PORTAGE, Barçah and TerminoWeb testify to the Group's renown. Achievement of immediate outcomes was deemed to be satisfactory, save for knowledge and technology transfer which, despite efforts made and numerous collaborations with industry, was strongly influenced by respondents' perceptions. Some felt that the Group has achieved a satisfactory transfer level, whereas others were of the opinion that more effort should have been made in that regard. This last outcome needs to become the focus of greater attention, as specified in one of the recommendations. Lastly, it was noted that the intermediate outcomes and the strategic outcome cannot be achieved without co-operation between the LTRC's main partners and identification of adequate performance indicators, which is supported by another recommendation.

Recommendations:

That NRC- IIT, in co-operation with the LTRC's key partners and industry, identify technology transfer needs and objectives and that it contribute to the achievement of these objectives through the identification of appropriate actions.

Management response: Recommendation accepted. This recommendation cannot be implemented until the actions in recommendation 6 concerning the Technology Roadmap (TR) have been completed. The industry's R&D needs will have to be ascertained before its technology transfer needs and objectives can be determined. In the meanwhile, however, NRC will continue to fine-tune its own technology transfer objectives on the basis of its current interaction with its key partners and industry.

That NRC validate the expected outcomes (intermediate and strategic) that are common to two or more of the LTRC's key partners and that it identify appropriate performance indicators in conjunction with the latter.

Management response: Recommendation accepted. NRC-IIT will spearhead an exercise with the LTRC's key partners to review objectives, responsibilities and appropriate performance indicators.

Conclusions and Recommendations – Design, Implementation and Cost-effectiveness

It was noted first of all that the design for the ILT program was adequate overall. In contrast, the design for the LTRC-NPO program was said to have improved, although there was disagreement among the key partners in that regard. The level of co-ordination and co-operation between the partner organizations associated with the LTRC has also improved significantly since its creation, but there is still room for improvement in the years ahead. NRC's management process for the ILT Group is efficient; however, there is the need for some improvement with respect to management of the LTRC-NPO. Two recommendations were made in that regard.

With respect to performance management, the ILT Group has a good system for the level of outputs and immediate outcomes. However, it has to rely on extensive co-operation and co-ordination with its LTRC partners in order to monitor the intermediate and strategic outcomes for which it is responsible. Consequently, the ILT Group cannot be wholly accountable for these outcomes. Lastly, it is important to note that the resources allocated to NRC under the APOL were used for the creation and operation of a research group and that the lack of allocated funding to the LTRC resulted in financial challenges for the LTRC-NPO. The issue of funding for the LTRC, which is addressed in a recommendation, could have a negative impact on the ability of both the ILT Group and the LTRC-NPO to achieve the objectives set out under the APOL.

Recommendations:

That the ILT Group prepare more formal annual work plans and multi-year plans to ensure greater transparency in communications on R&D activities and strategic directions while maintaining confidentiality of certain information, and to increase opportunities for internal and external collaboration at the LTRC.

Management response: Recommendation accepted. NRC-IIT is currently developing its activity plan for the next three years as part of an NRC-wide planning exercise and will disclose to its partners non-confidential aspects of its plan concerning the ILT Group. NRC-IIT will also enhance promotion of outcomes obtained and will seek feedback from other R&D groups, institutional users and the industry to identify new opportunities for collaboration.

Sharing of key partners' work plans will be promoted to increase opportunities for internal collaboration.

That NRC-IIT clearly communicate its policy on intellectual property and licensing to LTRC partners and companies concerned in the industry.

Management response: Recommendation accepted. NRC-IIT will develop an expanded communications plan explaining its intellectual property policy and its management of technology transfer. NRC's intellectual property policy, currently under review, will be disseminated to LTRC partners and to the companies concerned.

That NRC, in co-operation with its partners, continue to help look for a permanent solution with respect to the LTRC-NPO's financial self-sufficiency.

Management response: Recommendation accepted. However, NRC wishes first of all to review the LTRC management model as well as the roles and responsibilities of the three founding partners, as discussed in the evaluation, in order to determine the best approach for NRC and its partners in future.

Conclusions and Recommendations – Relevance

A review of the third element addressed by this evaluation, namely the relevance of the ILT Group, shows that certain important points will have to be considered to ensure the Group's existence over the long term, both within NRC and from the optic of the LTRC-NPO. It was noted first of all that the ILT Group's research focus is aligned with NRC and NRC-IIT objectives but is not a national priority area for NRC. In that regard, the report titled Science at Work for Canada, A Strategy for the National Research Council, 2006-2011, does not explicitly outline the strategic importance of the ILT Group to NRC. However, the ILT Group does align with the three NRC-IIT strategic directions contained in the NRC Report on Plans and Priorities.

On the other hand, even though the ILT Group is well aligned with the R&D environment in the language industry, it was said that the Group's alignment with the APOL's objectives will depend on its ability to clearly identify the language industry's needs in terms of R&D in language technologies and to adjust its R&D strategy accordingly.

Achievement of the strategic outcome common to both the ILT Group and the LTRC-NPO, namely to contribute to the growth of the Canadian language industry through R&D, clearly sets out the need for an in-depth comprehension of the actual needs of the industry in terms of innovation and problem issues. Many stakeholders agree that the Technology Roadmap (TR) released in 2005 was an excellent starting point because it flowed from an extensive national consultation with industry held in 2003. However, it needs to be reviewed and expanded upon.

Recommendation:

That NRC-IIT, with the help of the LTRC's key partners, craft a strategy to develop a broader understanding of R&D needs in the language industry by using the Technology Roadmap as a starting point. For this to be achieved, the ILT Group will require a significant contribution of targeted resources.

Management response: Recommendation accepted. The first phase of the Technology Roadmap (TR) for the language industry was completed by the Language Industry Association (AILIA) in March 2005. Its findings were then promoted by the LTRC-NPO. A second phase of the TR is needed in order to obtain relevant information for implementation of this recommendation.

NRC undertakes to work with its key partners and the industry in order to identify the best way to initiate the second phase of the TR.

NRC undertakes to contribute time and money along with its key partners and other stakeholders in order to carry out the second phase of the TR.

Date modified: