ARCHIVED - Evaluation of the Interactive Language Technologies Group (ILTG) - NRC Institute for Information Technology

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Table of contents

EXECUTIVE SUMMARY

The Interactive Language Technologies (ILT) group of the National Research Council Institute for Information technology (NRC-IIT) is a research group that focuses on developing computer-based tools for processing oral and written information in multiples languages. The group's core research activity revolves primarily around the development of Statistical Machine Translation (SMT) systems based on Machine Learning (ML) techniques in computer science. This NRC initiative, which was launched in 2003 in response to the Action Plan for Official Languages (APOL), supports the Roadmap for Canada's Linguistic Duality 2008-2013, which constitutes a key component of the current federal government official languages strategy.

Evaluation Scope and Methodology

This report presents the findings of an evaluation that was conducted by the NRC Office of Audit and Evaluation from September 2011 to January 2012. The evaluation was designed to address the core evaluation issues, which fall within two broad categories of relevance and performance, and covers the period from 2008-09 to 2011-12 inclusive.

A mixed methods approach was used for this evaluation. The approach allowed for triangulation and complementarity through the use of a restricted set of both qualitative and quantitative methods. The specific methods used in the study include a review of internal and external documentation, an analysis of performance and financial data in addition to interviews with key informants.

Relevance - Program Need

The ILT group was created to respond to the needs of two types of clients that constitute two different segments of the broader language industry; the end users of language technologies (e.g., translation firms, freelance translators, users of language technologies) and the language technology companies. The evidence collected as part of the evaluation confirmed that the language technology and translation industries lack the R&D capabilities and financial resources to make a significant investment in the development of innovative language tools and technologies. This is particularly the case for the development of large and complex technologies such as SMT systems.

Relevance - Role of Federal Government

The intervention of the federal government in the area of language technologies was found to be justified by three key elements. First, the role of government in support of Canada's linguistic duality is mandated in the Official Languages Act. Second, the critical mass and world-class expertise of the ILT group makes NRC the best federal government organization to conduct this type of research. Third, it was found that no alternative exists in Canada to the SMT system developed by the ILT group. In fact, other foreign systems available to Canadian users present serious limitations that justify the intervention of the government with regards to the development of such technology.

Relevance - Alignment with Government and NRC Priorities

The ILT group was found to be aligned with the Federal Science and Technology (S& T) strategy and the priorities of the Quebec provincial government. However, opportunities for improvement were identified with regards to its alignment with the programs and priorities developed by NRC as part of its new strategy. Although it was not possible for NRC executives to provide a precise description of the role that the group will play under the new program structure at the time of the evaluation, there was a consensus among the internal evaluation participants that opportunities exist with regards to the application of the ILT group skills, expertise and technologies to the field of security.

The realignment of the group's activities in the field of security can however appear problematic as long as the group remains accountable for the strategic objectives of the Roadmap, which aim at strengthening the Canadian language industry. In this context, the key question becomes whether both mandates are compatible. Based on the evaluation evidence, it appears that research in security can benefit the Canadian language industry as long as civil applications are explored and that efforts are made to transfer these technologies to the private sector.

  • Recommendation 1: As long as the ILT group is considered an initiative under the Roadmap for Canada's Linguistic Duality, NRC must balance the contribution of the group's activities to the new NRC Portfolio/Program structure with its contribution to the strategic objective of the Roadmap (i.e., to strengthen the Canadian language industry).

Achievement of expected outcomes - Strengthening of Canada's Position in Language Technologies

Overall, the evaluation found that the creation of innovative knowledge and technologies by the group has positioned Canada as a world leader in language technologies, especially in the field of SMT systems. More specifically, the quality of the publications developed by the ILT group, its involvement in the scientific community, and its impact on the quality of the research conducted in Canadian universities has helped position Canada as a world leader in this field.

In addition to the creation of innovative knowledge, the ILT group was also found to have contributed to the creation of innovative technologies and tools in the field of translation aid, document categorization/management and terminology. While the quality of these technologies was confirmed by all of the evaluation participants consulted, the PORTAGE system was usually referenced as the key accomplishment of the group thus far. The evaluation also found that the ILT group leveraged the research skills and competencies of NRC-IIT by involving researchers from other groups, which contributed to the achievement of this expected outcome.

The evaluation also noted that the achievement of this strategic objective was realized with a limited contribution from Université du Québec en Outaouais (UQO) and Translation Bureau of Canada (BtB), which were expected, based on the original Treasury Board Secretariat (TBS) submission, to contribute to the creation of knowledge and the development of language technologies in collaboration with NRC.

Achievement of expected outcomes - Strengthening of the Canadian Language Industry

Although the group was successful at creating innovative technologies, the evaluation found that the group had a small impact on the growth of the language technology industry.

The impacts reported by the clients consulted varied depending on the technology that was licensed. In the case of technologies other than PORTAGE (i.e., PORTalign with Multicorpora and Webitext with Terminotix), the clients expressed a high level of satisfaction with the technologies in addition to an increased competitive advantage and increased sales. The two clients that licensed PORTAGE, CLS-Lexitech and BtB, could not identify any impact as both organizations were in the process of implementing and assessing the performance of the system when the interviews were conducted. Nonetheless, both organizations noted that preliminary results stemming from internal assessments of the system indicated a potential for increased productivity.

The transfer of PORTAGE to two of the biggest players in the Canadian translation industry, CLS-Lexitech and BtB, represents a significant achievement for the group considering the number of translators that will now have access to the technology. Moreover, the transfer of PORTAGE to the BtB will help the organization to achieve its mandate to offer translation services to federal departments who are required to use both official languages as required by the Official Languages Act. This represents a significant contribution to the strategic objective of the Roadmap for Canadian Linguistic Duality.

Notwithstanding the transfer of PORTAGE to these two large organizations, many evaluation participants highlighted opportunities for the commercialization of the technologies in foreign markets.

  • Recommendation 2: NRC must develop and implement a strategy to commercialize the PORTAGE technology and associated IP, and thus, increase its impact on the language industry. NRC should also explore the business opportunities available in other industry sectors because potential use/applications for this technology were identified as part of the evaluation (e.g., security and defense sector).

The evidence shows that the limited impacts of the group with regard to this expected outcome are explained by the numerous barriers faced by the group when transferring technology. The following barriers to technology transfer were identified as a result of the evaluation:

  • Strong competitive environment in the market of language technologies for translation aid;
  • Limited receptor capabilities in the case of translation companies;
  • Financial capacity of language technology companies;
  • Resistance of translators toward Machine Translation (MT) systems;
  • Perceived complexity of doing business with NRC within the community; and,
  • Existence of alternative MT systems (i.e., open-source technologies and commercial products).

Achievement of expected outcomes - Contribution to the strengthening of Canada's innovation system

NRC, including the ILT group and NRC-IRAP, was found to be a key player in the local innovation system. In addition to its contribution to the development and transfer of innovative technologies, NRC has supported the growth of this local innovation system in the following manner:

  • Organization of events and seminars that bring together the scientific community;
  • Attraction of innovative firms and investments to the business incubator due to the physical proximity of researchers;
  • Sharing of formal and informal knowledge and expertise with university researchers and entrepreneurs; and,
  • Training of HQP.

In addition to these elements, it was found that NRC is perceived by a vast majority of stakeholders as one of the main pillars that support the growth and ensure the sustainability of the Language Technology Research Centre (LTRC), a not for profit organization created by NRC, BtB and UQO in collaboration with the language industry. Footnote 1

The evaluation evidence also indicates that the LTRC has significantly improved its overall situation since the last evaluation was conducted. In fact, it appears that the LTRC now possesses all the tools required to contribute to the growth of the language technology industry.

Economy and Efficiency

The ILT group, which follows the administrative processes and practices established by NRC Corporate Branches and NRC-IIT, was found to be operating in an efficient manner. In addition to the fact that NRC management expressed a strong level of satisfaction with the group's achievements, it was demonstrated as part of this report that the group used its resources to produce innovative technologies that are considered among the best in the world.

Moreover, the group was found to manage its resources in an economical manner. However, the evaluation identified two opportunities for improvement in terms of economy, which both pertain to the leasing agreement with UQO.

First, it was mentioned by internal evaluation participants that the cost of the lease with UQO is more expensive than the cost for NRC to locate the group in the empty space available in its Montreal Road facilities in Ottawa. However, interviewees raised the concern that the move could negatively impact the LTRC partners because the ILT group constitutes the core of the research activity of the centre. It should also be noted that the BtB left the facilities in 2011 to reduce its operating costs within the context of government budget reductions. Under these circumstances, the departure of another key founding partner would certainly have an impact on the credibility and sustainability of the LTRC.

Second, it was found that the group is renting more space than it needs. Based on discussions with evaluation participants, approximately 25% of the space rented by the group was not being used when the evaluation was conducted.

  • Recommendation 3: NRC should ensure that the space rented reflects the group's needs and adjust future leasing agreements accordingly if the ILT group remains in its current facilities.

Conclusion

While the ILT group has demonstrated good value for money (i.e., relevance and performance) after eight years of contribution to the field of language technologies, the ILT group is currently at a crossroads in terms of its future research activities as internal and external elements have created the needed conditions to change direction.

On one hand, the development of the PORTAGE system, which constitutes the bulk of the group's research activities, has reached a sufficient level of maturity and stability and thus, does not require the same level of investment as it did in the past. As such, the ILT group has the opportunity to undertake research in new areas or to explore the potential applications of the PORTAGE system to other sectors. On the other hand, the development of the new NRC strategy has created the need for all research groups to realign their activities with the new programs and priorities of the organization. In this context, opportunities were identified by internal and external evaluation participants in the field of security, which would demonstrate alignment with the newly created Security and Disruptive Technologies portfolio. However, the success of this niche area will depend on the ability of the group to establish formal research collaboration agreements with key partners from the defense and security sectors, including partners from other government departments and the private sector.

Despite the need for the ILT group to realign its activities with the new NRC program and priorities, the group remains accountable until fiscal year 2012-13 for some of the strategic objectives of the Roadmap for Canada's Linguistic Duality. Therefore, it will be critical that NRC maintain a balance between these dual mandates in order to remain relevant.

1.0 INTRODUCTION

The Interactive Language Technologies (LTR) group of the National Research Council Institute for Information technology (NRC-IIT) is a research group that focuses on developing computer-based tools for processing oral and written information in multiples languages. The group's core research activity revolves primarily around the development of Statistical Machine Translation (SMT) systems based on Machine Learning (ML) techniques in computer science. This NRC initiative supports the Roadmap for Canada's Linguistic Duality 2008-2013, which constitutes a key component of the federal government official languages strategy.

Senior Executive Committee (SEC) approved the conduct of an evaluation of the ILT group in October 2011. This report presents the findings of an evaluation that was conducted in the winter of 2011-12. It covers exclusively the activities of NRC in support of the language industry. This clarification is essential as the ILT group and the Language Technologies Research Centre (LTRC), a non-profit organization that supports the language industry in Gatineau, are still perceived as a unique entity among key stakeholders. Footnote 2 The evaluation will therefore focus on the key achievements of the ILT group over the last three years, including its support of the growth and success of the LTRC. Other partners such as l'Université du Québec en Outaouais (UQO) and the Translation Bureau of Canada (BtB) share different responsibilities toward the success of the LTRC. The contribution of these partners will not be covered by this evaluation.

This evaluation is part of a broader evaluation of the Roadmap which is led by Department of Canadian Heritage (PCH). This evaluation, as well as the evaluations conducted by each departments that was funded under the Roadmap, is expected to feed the horizontal evaluation that will be completed in the summer of 2012.

Following an overview of the evaluation approach used for this project, Section 2.0 presents a brief profile of the ILT group as well as the general ecosystem in which it operates. While section 3.0 and section 4.0 examine the evaluation findings organized by broad evaluation issues (relevance and performance), section 5.0 concludes by synthesizing the key messages and observations that can be drawn from the evaluation results. The last section presents a management response to the evaluation recommendations.

2.0 EVALUATION OVERVIEW

This evaluation was carried out in accordance with NRC's approved Evaluation Plan for 2011-12 to 2015-16. It meets the requirements of the Treasury Board Secretariat (TBS) Policy on Evaluation (2009) and supporting Directive (2009). The evaluation was conducted by the NRC Office of Audit and Evaluation from September 2011 to January 2012, following a planning process that included consultations with the Acting Director General of NRC-IIT and the Vice President, Frontier Science.

2.1. Evaluation Objectives

The evaluation's objectives were to:

  • respond to a commitment made to the Department of Canadian Heritage (PCH) and TBS that NRC would contribute to the horizontal evaluation of the Roadmap for Canada's Linguistic Duality 2008-2013 (herein referred to as the Roadmap) by conducting a summative evaluation of the ILT group; and
  • provide SEC with strategic information in support of decision-making in the context of the new NRC strategy.

It is expected that both the 2008 evaluation report and the current report will be used by PCH as a primary source of information for the horizontal evaluation of the Roadmap, which will be presented to an inter-departmental ADM-level steering committee as part of the renewal of the Roadmap.

2.2. Evaluation Scope

The evaluation was designed to address the core evaluation issues, which fall within two broad categories of relevance and performance, and covers the period from 2008-09 to 2011-12 inclusive. The evaluation questions that guided the methodology and research tools are presented in Appendix 1.

2.3 Evaluation Approach

The evaluation approach used for this project was selected by considering two key factors. First, an evaluation of the ILT group had been completed in early 2008 and a preliminary analysis of the program context conducted as part of the planning phase revealed that no significant change has affected the group's activities and results over the last three years. Second, it was determined as part of the planning process that the evaluation of the ILT group could be considered a low-risk endeavor. Under these circumstances, the level of effort (i.e., the evaluation resources deployed) for this project was calibrated to the level of risk identified.

In order to maintain a balance between the production of meaningful evaluation findings and the limited resources available (i.e., human and financial), the evaluation focused on the following priorities:

  • Update the key results identified in the 2008 evaluation by focusing on the intermediate and longer term results given that the group had only been fully operational for three years when the previous evaluation took place;
  • Assess the progress made by the group with regards to the key challenges and issues raised in 2008;
  • Identify the key issues and challenges currently faced by the group and attempt to explain their causes and consequences;
  • Ensure that all of the evaluation issues identified in the TBS Evaluation Policy (2009) or any other specific requirements are covered in the report as the Policy came into effect after the 2008 evaluation was conducted.

Although this evaluation fully covers the issues of relevance and performance, very limited information is included with regards to the program background, the context of the language industry, the key partners of the LTRC, or the program theory (i.e., what are the rationale and expected results that underpin the government intervention). Any reader who wishes to expand his or her knowledge on those background elements may consult the 2008 evaluation report.

2.4 Evaluation Methodology

A mixed methods approach was used for this evaluation. The approach allowed for triangulation (i.e., convergence of results across lines of evidence) and complementarity (i.e., developing better understanding by exploring different facets of a complex issue) through the use of a restricted set of both qualitative and quantitative methods. The following methods were used to generate the evaluation findings:

  • Internal interviews (n=6; including staff from the group, NRC-IIT and the VP Frontier Science office);
  • External interviews (n=10; including LTRC partners, universities and companies);
  • Review of key internal and external documents; and
  • Analysis of performance and financial data.

Although the evaluation Terms of Reference planned for a case study on the SMT system developed by the group (i.e., PORTAGE technology) with the intent of highlighting the impact of the technology on its collaborators, it was decided that using such a method was inappropriate in the context of this evaluation. In fact, it was found that almost 80 percent of the group's activities support this technology and, thus, evaluating the ILT group is the equivalent of conducting a case study on PORTAGE. Moreover, including additional information about the relevance and impacts of the other technologies developed by the group required minimal effort.

2.5 Limitations

Although it is less complex to evaluate a research group than an entire institute or a large research program, a certain number of limitations to the evaluation design and results can be identified. The main limitations and associated mitigation strategies are summarized in Table 1, below.

Evaluation limitations and Mitigation Strategies
Limitations Mitigation strategies
Interviews as the primary source of evaluation evidence.
  • Consult a vast diversity of respondents such as companies, LTRC partners, Other Government Departments (OGDs), universities, and NRC staff to obtain a balanced perspective on the different evaluation issues/questions.
  • All of the key clients and collaborators of the group were consulted.
  • Triangulate findings with other lines of evidence, particularly performance data.
  • Validate key findings with relevant internal stakeholders.
Limited information about the language technology industry and research environment given the decision to rely on the findings of the 2008 evaluation.
  • Identify any changes to the industry context and confirm the validity of the previous evaluation findings as part of interviews.
  • Collect and review specific strategic external and internal documents when necessary.
  • Assign an internal evaluator with an extensive knowledge of the file. The lead evaluator was involved in the 2008 evaluation.
Uncertainties regarding the future positioning of the group as part of the new NRC strategy/program structure.
  • Validate the preliminary findings with internal stakeholders who hold a strategic position within the Portfolio.

3.0 PROFILE OF THE INTERACTIVE LANGUAGE TECHNOLOGIES GROUP

This section briefly describes the history and profile of the group, including key financial data, the number of Highly Qualified Personnel (HQP) and the group's main research area. Additional and complementary information about the creation of the group and its ecosystem, including NRC-IIT, the key partners and the LTRC are available in Section 2.0 of the 2008 evaluation report. Footnote 3

3.1 History and Mandate of the ILT Group

The ILT group was created in 2003 as part of the Federal Government Action Plan for Official Languages (APOL). Under the APOL, NRC received a total of $10 million ($2 million per year from 2003-04 to 2007-08) to Footnote 4:

  • Create a research program in language technologies; and
  • Create, in partnership with the BtB and UQO, a research center in the field of language technologies. Footnote 5

In 2007, on-going funding was granted by TBS to ensure the long-term stability of NRC research activities.Footnote 6 In 2008, the funding granted under the APOL was renewed and increased ($1.1. billion over five years) with the creation of the Roadmap for Canada's Linguistic Duality 2008-13. The level of funding attributed to NRC under the Roadmap remained the same as it was under the APOL. The value of these investments represents a small proportion of both the Roadmap and NRC investments. More specifically:

  • The funds allocated to NRC under the Roadmap represent less than 1% of the total investment made; and,
  • The research activities of the ILT group represent approximately 9% of the total NRC-IIT budget and approximately 0.2% of the total NRC budget for 2010. Footnote 7

3.2 Research Focus of the ILT Group

Many research opportunities exist in the field of language technologies, from the development of writing correctors such as those included in basic office software to the voice recognition systems used by telecommunication distributors. The ILT group made the decision to focus on the development of SMT systems for three reasons:

  • The need to improve the competitiveness of the Canadian translation industry in a globalized market;
  • The previous experience and research competencies of the ILT group researchers; and
  • The need to find a niche area of impact given the limited resources of the group.

In addition to its activities in SMT, the research team has also worked on the development of tools and technologies in the fields of document categorization/management and terminology.

3.3 Profile of Highly Qualified Personnel

As of December 2011, the ILT group employed seven Research Officers (ROs), two Research Council Officers (RCOs), two Computer Systems staff (CSs) and one administrative assistant for a total of 12 Full Time Employees (FTEs). Since 2008-09, the total number of HQP has decreased following the departure of two researchers. The positions that were left vacant were not renewed given NRC's plan to stabilize its financial situation in the context of the new organizational strategy. The reduction of the group's staff has resulted in reduced capabilities in the field of document management/categorization and the complete loss of its expertise in terminology technologies. As of December 2011, the bulk of the ILT group was dedicated to the development and ongoing improvement of the PORTAGE system with two researchers focusing on document management/categorization.

In addition to its core staff, the group has also benefited from the expertise of researchers from the Interactive Information group of NRC-IIT. Footnote8 In fact, both groups are conducting research based on Machine Learning (ML) techniques, which facilitate the collaboration and synergies between the groups. The potential for collaboration in the development of innovative technologies in the field of security was identified on many occasions by external and internal evaluation participants.

The research capabilities of the group have also been increased through the hosting of students and visiting researchers. A total of 12 students and 11 individual visiting workers have been hosted by the group during the period covered by the evaluation.

3.4 Profile of Financial Resources

The group's financial resources have remained relatively stable since 2007-08 at approximately $2M per year, as can be seen in Figure 1, below. Over the last three years, NRC has generated revenues from collaborative research projects, pilot projects with companies and commercial licenses of an approximate value of $855K. Footnote 9 Almost half of these revenues stem from collaborative research contracts with two American security agencies; the Defence Advanced Research Projects Agency (DARPA) and the National Virtual Translation Center (NVTC). It should be noted that the revenues generated were not entirely invested in the group's activities given NRC practices with regard to the management of revenues.

Figure 1: ILT Group Expenses from 2003-04 to 2010-11

ILT Group Expenses from 2003-04 to 2010-11

Gatineau Expenses Per Year
Expense 2003/2004 2004/2005 2005/2006 2006/2007 2007/2008 2008/2009 2009/2010 2010/2011
Operations $213,736 $330,159 $213,859 $829,812 $608,292 $554,749 $367,847 $280,906
Capital $146,854 $221,553 $293,001 $346,971 $14,465 $77,073 $115,462 $0
Wage $349,771 $1,074,668 $1,134,403 $1,552,122 $1,286,013 $1,582,006 $1,725,878 $1,734,219
Total $710,361 $1,626,380 $1,641,263 $2,728,905 $1,908,770 $2,213,828 $2,209,187 $2,015,125

4.0 RELEVANCE OF THE INTERACTIVE LANGUAGE TECHNOLOGIES GROUP

Canada is facing different societal and economic challenges that call for the intervention of the federal government. However, in a federal system where the government has limited resources to respond to the various needs of Canadians, it is imperative to assess, on a regular basis, whether the intervention (in this case the ILT group) is responding to a demonstrable need, is aligned with government priorities and correspond to federal responsibilities. This section addresses these three elements, as part of a broader discussion on the group's continued relevance.

4.1 Relevance - Program Need

This section outlines the Research and Development (R&D) investment gap faced by language technology and translation firms in Canada. The importance of language technologies and tools for the growth and performance of the language industry will also be discussed within the context of the Roadmap for Canada's linguistic duality.

4.1.1 Lack of R&D Investment and Capabilities of Canadian Companies

The language technology and translation industries lack the resources and R&D capabilities to support the development of large and/or complex technologies. The ILT group was created to develop and transfer technologies that will support the growth of the Canadian language industry.

The ILT group was created to respond to the needs of two types of clients that constitute two different segments of the broader language industry; the end users of language technologies and the language technology companies. The former include Canadian translation firms and freelance translators. The technologies developed by the group can also be transferred to international organizations with large translation needs such as the United Nations. Language technology companies, on the other hand, are mostly SMEs in the Information and Communication Technology (ICT) sector that can license the technologies of the group with the intention of developing commercial products that will be, in turn, offered to the end users. Both types of companies have the potential to adopt the technologies developed by the ILT group, as illustrated in Figure 2.

Figure 2: Companies Supported by the ILT Group

Figure 2 illustrates the technology transfer process of the Interactive Language Technologies group with regards to two types of clients it serves; language technology companies and end-users of language technologies. The figure shows that the first type of clients, who are primarily companies involved in the development of language technologies, transforms the knowledge and technologies transferred by the ILT group in commercial products which are, in turn, sold to the end-users of language technologies. The figure also shows that the Interactive Language Technologies group transfers its technology directly to the end-users, who are, for the most part, translation firms and freelance translators.

Companies Supported by the ILT Group

The evidence collected as part of the evaluation confirmed that the language technology and translation industries lack the R&D capabilities and financial resources to make a significant investment in the development of innovative language tools and technologies.

The fragmentation of the translation industry into a large number of very small companies and freelance translators explains the lack of R&D capabilities of the industry. While it is obvious that small companies and freelance translators are not in a position to conduct R&D, it was found that large translation companies, despite their volume of business and financial capabilities, rarely conduct R&D. In most cases, these companies purchase tools and technology solutions from language technology firms. As such, only a very small number of translation companies have the financial and technical skills to develop or absorb large and/or complex technologies.

For their part, the language technology companies, which are closer to the ICT sector, usually have some R&D capabilities but are not able to invest in large and complex technologies such as SMT systems because of their small size. Moreover, Machine Translation tools have not reached the level of maturity or demonstrated their performance sufficiently to justify the financial risk associated with these large scale projects. Around the world, only large companies like Google or SYSTRAN possess both the capabilities and resources to conduct such large R&D projects. However, according to the documentation consulted, the systems developed by these firms are not available for transfer to other companies.

4.1.2 The Role of the Language Industry in the Context of Canada's Official Languages

Language technologies are perceived as a key solution to support the growth of the language industry which is essential to preserve Canada's Linguistic Duality.

According to the official documents for both the Action Plan for Official Languages (APOL) and the Roadmap, the presence in Canada of a strong language industry is an essential factor in preserving Canada's linguistic duality. For this reason, both of these horizontal programs invested in the language industry, including the creation of the ILT group.

More specifically, the logic that underpins the existence of the ILT research group is based on the notion that language technologies are a key solution to maintaining the competitiveness of the Canadian language industry. Footnote 10 This logical link was confirmed by all the stakeholders consulted, including the BtB, the biggest translation bureau in the country, as well as other large translation firms. In this regard, an evaluation participant outlined the fact that the reduction of translation costs could significantly increase the capacity of the government and private sector to work in both official languages.

Finally, it should also be noted that the language industry faces other challenges that the development of technology alone will not be able to resolve. In fact, challenges such as the fragmentation of the industry, the lack of available HQP and the visibility of the sector in international market, are addressed by other programs funded under the APOL and the Roadmap. Footnote 11

4.2 Relevance - Role of the Federal Government

The intervention of the federal government in the area of language technologies was found to be justified by three key elements. First, the role of government in support of Canada's linguistic duality is mandated in the Official Languages Act. Second, the critical mass and world-class expertise of the ILT group makes NRC the best organization positioned in the federal government to conduct this type of research. Third, it was found that no alternative exists in Canada to the SMT system developed by the ILT group. In fact, other foreign systems available to Canadian users present serious limitations that justify the intervention of the government with regards to the development of such technology.

4.2.1 Legal Framework and the Role of the Federal Government in Support of Official Languages

Canada's linguistic duality is the responsibility of the federal government as stated in the Canadian Charter of Rights and Freedoms as well as the Official Languages Act.

The linguistic duality of Canada is a key element of the Canadian Charter of Rights and Freedoms Footnote 12. Another component of the legislative framework that justifies government intervention in this area is the Official Languages Act, which states the duties and responsibilities of the federal government with regards to official languages. Footnote 13 As such, the creation of the APOL and the Roadmap is supported by a strong and clear legislative framework.

NRC's mandate to develop and transfer language technologies to the industry stems directly from the APOL and the Roadmap. As a federal R&D laboratory, NRC was best positioned to conduct research in this area given the expertise, infrastructure and reputation of the organization. Under the latter horizontal program, NRC shares the responsibility of strengthening the language industry with the BtB, which was mandated to create a different but complementary program.

4.2.2 NRC's Role as Catalyst for Research in the Field of Language Technologies

No other actors in Canada, whether the private sector or universities, have the ability to develop and transfer complex SMT systems like PORTAGE. NRC, as a federal government laboratory, is best positioned to conduct this type of research.

The evaluation participants consulted in 2011 as well as the results of the expert review that was conducted as part of the 2008 evaluation mentioned the quality and world-class expertise of the researchers of the ILT group. These evaluation results also showed that no equivalent group exists in Canada. Moreover, the evaluation results show that universities do not have the industrial focus and critical mass of the group, and the private sector lacks the expertise and resources required to develop complex technologies.

4.2.3 Alternative Technologies for Canadian Companies

Despite the fact that alternative SMT systems are available to Canadian companies, the ILT group offers a system with better performance and provides better services to its collaborators.

An environmental scan of the alternative SMT systems available for use by Canadian companies as well as discussions with experts showed that although a certain number of options exist, the PORTAGE system presents several advantages over these alternative systems.

One of the only comparable commercial products available in the market is the SDL Language Weaver, an SMT system that was originally developed by a Californian research laboratory (University of Southern California) and commercialized through the creation of a spin-off company that was later acquired by the SDL Corporation. Footnote 14 When consulted about this product, the Canadian companies that were in a position to compare technologies and/or products noted that the PORTAGE system performs better in terms of the quality of the translations produced. The clients consulted as part of the evaluation also identified the responsiveness of the group in case of problems and the ongoing collaboration between NRC and their company as key differences with the other service providers. One of the ILT group's clients that holds a commercial license for PORTAGE insisted on the fact that: "We are more than clients, we are partners."

The companies and organizations interested in adopting an SMT system also have the option of accessing Moses, an open-source technology (or software) that was developed by the academic community for research purposes. This system is currently used by international organizations (e.g., United Nations) with large translation needs. In the opinion of the experts consulted as part of the evaluation, Moses provides an acceptable level of performance although it presents several limitations when compared to PORTAGE. The main limitations identified by the experts include:

  • The ongoing development and improvement of the system depends on the enthusiasm of the research community that supports Moses. An expert pointed out the fact that the technology has been neglected in recent years and that this constitutes a serious risk for organizations wishing to invest resources and efforts in this open-source technology. In contrast, the NRC researchers are paid to develop, improve and transfer PORTAGE to Canadian companies, which ensures the on-going improvement of the technology;
  • The academic community does not provide any form of support for the implementation of the Moses technology. As such, any company or organization with limited IT capabilities will not be in a position to access Moses. Few companies or organizations in Canada have the ability to use the technology despite the fact that there is no charge to access it. Contrary to this, NRC offers training and support to its clients as part of any commercial transfer; and,
  • The installation and integration of Moses within the IT system/network of a company appears to be more complex than the installation and integration of the PORTAGE technology. In fact, one of the evaluation participants implemented both systems and was therefore in a position to provide a comparative perspective on this aspect. In their case, the implementation of PORTAGE took three days whereas the implementation of Moses took more than a month. Once the technology has been adopted by a company or an organization, the availability and responsiveness of the academic community in the case of problems with the system is not comparable to what the ILT group offers. This is explained by the fact that the group has contractual obligations toward its clients.

4.3 Relevance - Alignment with Government and NRC Priorities

Although the ILT group was found to be aligned with the Federal Science and Technology (S&T) strategy and the priorities of the Quebec provincial government, opportunities for improvement were identified with regards to its alignment with the programs and priorities developed by NRC as part of its new strategy.

4.3.1 Alignment with the Federal Science and Technology Strategy

The group is aligned with the three advantages of the Federal Science and Technology (S&T) strategy.

Overall, the group's activities were found to be aligned with the key elements of the Federal S&T strategy. Footnote 15 First, the ILT group is aligned to the Entrepreneurial Advantage of the strategy by establishing key partnerships with the private sector. The collaboration of the ILT group with the LTRC, the academic community and other partners to increase its impact on industry reflects the spirit of this Advantage. Second, the research activities of the ILT group support the ICT sector, which is identified in the Strategy under the Knowledge Advantage as one of the priority sectors of the federal government. Finally, ILT group staff members contributed to the People Advantage by training students in computer sciences through internships and by teaching a course at UQO on the use of language technologies to students in translation studies.

4.3.2 Alignment with the Priorities of Other Levels of Government

The ILT group operates in an ecosystem where the local and provincial governments are active through various types of investments.

The presence in Gatineau of the ILT group and the LTRC has stimulated local investments in support of the local innovation system in the field of language technologies. The city of Gatineau was the first partner to support the initiative by investing $250K to help the LTRC attract firms in its industrial incubator, which is located in the LTRC building on the UQO campus. A regional association of elected officials, the Conférence régionale des élus de l'Outaouais (CREO), also invested $200K to support the activities of the LTRC.

The provincial government is investing in the research centre via the University and through the ACCORD program. Footnote16. It was also noted, however, that the Quebec government could increase its participation in the development of the local innovation system through the Programme de soutien à la valorisation et au transfert - Soutien aux regroupements sectoriels de recherche industrielle (PSVT). Despite several attempts made by the LTRC to access this program over the last years, no funding was obtained from this source. The LTRC has submitted a proposal as part of the new round of funding that is currently being allocated.

4.3.3 Current Alignment with NRC Programs and Priorities

The ILT group, like all of the other NRC institutes, is currently in the process of realigning its research activities with the NRC new research portfolios and programs. It was challenging, in this context, to determine the level of alignment of the group with NRC programs and priorities.

NRC is currently implementing a new strategy that is based on a new organizational structure, towards which the organization is currently transitioning. As such, limited information was available on the programs that will be created as a result of the transition and that will make up NRC's research portfolios. Consequently, it was not possible to clearly ascertain the level of alignment of the ILT group with NRC's programs and portfolios. However, the information shared by NRC's management as part of the evaluation indicated that regardless of the transition currently taking place, the ILT group's research activities will remain consistent with the objectives of the Roadmap until March 2013, when the funding for the horizontal program expires. After this date, if the ILT group remain an initiative under the renewed Roadmap, it will be critical for NRC to ensure that the activities of the ILT group support both the NRC portfolio/program objectives and the objectives of the new Roadmap.

Although it was not possible for NRC executives to provide a precise description of the role that the group will play under the new program structure at the time of the evaluation, there was a consensus among the internal evaluation participants that opportunities exist with regards to the application of the ILT group skills, expertise and technologies to the field of security. Footnote17. The significant revenues generated by the group through research contracts in the field of security in recent years support this statement (see section 3.4). As such, a decision to reorient the group's activities in the area of security will ensure the alignment of the group with the Security and Disruptive Technologies and Information and Communication Technologies portfolios.

In the case where the ILT group remains an initiative under the new Roadmap beyond 2012-13, the key question becomes whether research in language technologies for security applications can help grow the language technology industry. In fact, based on the evaluation evidence, it appears that research in security can benefit the Canadian language industry as long as civil applications are explored and that efforts are made to transfer these technologies to the private sector. The development of the PORTAGE system is one example of a generic technology that can have civil or military/security applications. In fact, the research that was done on the PORTAGE system with the funds provided by DARPA helped improve the technology which is now used by translation firms.

  • Recommendation 1: As long as the ILT group is considered an initiative under the Roadmap for Canada's Linguistic Duality, NRC must balance the contribution of the group's activities to the new NRC Portfolio/Program structure with its contribution to the strategic objective of the Roadmap (i.e., to strengthen the Canadian language industry).

5.0 EFFECTIVENESS AND PERFORMANCE OF THE INTERACTIVE LANGUAGE TECHNOLOGIES GROUP

The performance framework of the ILT group that was developed as a result of the 2008 evaluation identified three intermediate expected outcomes. The performance of the group was determined by examining the progress made toward the achievement of these outcomes. More specifically, the three expected outcomes pertain to the group's ability to create innovative knowledge and technologies, to transfer these technologies to Canadian companies and, finally, to support the growth of the innovation system specialized in the field of language technology research in Gatineau. These three outcomes aim at strengthening the Canadian language industry, which corresponds to the expected contribution of the ILG group under the Roadmap.

5.1 Strengthening of Canada's Position in Language Technologies

Overall, the evaluation found that the creation of innovative knowledge and technologies by the ILT group has positioned Canada as a world leader in language technologies, especially in the field of SMT systems. However, the achievement of this strategic objective was realized with a limited contribution from UQO and BtB, which were expected, based on the original TBS submission, to contribute to the development of language technologies in collaboration with NRC.

5.1.1 Contribution to the Creation and Diffusion of Innovative Knowledge

The presence of the ILT group was found to have a significant impact on the creation of innovative knowledge in Canada and on its position as world leader in this field.

The quality of the publications developed by the ILT group, its involvement in the scientific community, and its impact on the quality of the research conducted in Canadian universities have strengthened Canada's position in the field of language technology on the world stage.

5.1.1.1 Publications, Peer Recognition and Involvement in the Scientific/Academic Community

The performance data reviewed as part of the evaluation indicate that the ILT group is well positioned to exceed its performance target (i.e., 50 papers to be published for the period between 2008-09 to 2012-13) with regards to the number of publications. In addition, the data also reveal that these papers have being strongly referenced by the academic community, which speaks to the quality of the research. In total, the group published 49 papers in peer-reviewed journals and conferences. These papers have been referenced 649 times by the research community during the period covered by the evaluation. In 2010, one of the group's researchers received an award for the best paper presented at the Traitement automatique des langues naturelles conference (TAN), an international conference on automatic natural language processing held in Montreal. Footnote 18

The research results produced by the ILT group have been disseminated as part of international and national conferences and presentations to the scientific and business communities, which have contributed to the positioning of the ILT group and, thus Canada, as a world leader in language technologies.

The quality of the research produced by the ILT group was also ascertained by examining its involvement in the scientific/academic community. In this regard, the performance data show that ILT group members were involved in 16 editorial committees of scientific journals and in directing six program committees of international conferences. In addition to this, the fact that the group leader is representing Canada on the North Atlantic Treaty Organization (NATO) Task Group on Speech and Language Technology is indicative of the skills and expertise of the group and, thus, of the quality of its research.

5.1.1.2 Availability of a Technology Platform that Facilitates Fundamental Research

Several Canadian universities currently benefit from the PORTAGE system. In total, eight Canadian universities, including McGill University, Simon Fraser University, Université de Montréal and UQO accessed PORTAGEshare, a license that was established specifically to facilitate the transfer of PORTAGE to the academic community for research purposes.

The development of the PORTAGE technology, a stable SMT system on which fundamental research can be performed, was found to have an impact on the quality of the research produced by the Canadian scientific community. In this regard, one external interviewee mentioned that he strongly benefited from the system because PORTAGE allowed him and his team to validate theoretical models without having to develop their own SMT system. Developing such a system for a research group of their size would have been almost impossible given the time and resources required. The ILT group and this Canadian research group co-authored papers as a result of their collaborations. The collaboration was found to be mutually beneficial as the research conducted by the university helped improve the PORTAGE system.

5.1.1.3 Creation of Innovative Knowledge Through Research Collaborations/Contracts

The ILT group was involved in strategic collaborative research projects that helped improve its technologies in addition to generate revenues for NRC. The fact that international clients have established research collaborations with the ILT group speaks to the quality of its research. Two of the three projects, the GALE program and the project with the National Virtual Translation Center (NVTC), were funded by American defense and security agencies for application of the group's technologies in the field of security. These two projects generated a total of approximately $1.5M for NRC. The group was also involved in the SMART consortium, a European research consortium led by Xerox. In the latter case, the PORTAGE system was used by the collaborators as a reference system for the evaluation of the technologies developed by the different European research laboratories. As a result of these three collaborations, NRC researchers had access to some of the best scientists and state-of-the-art technologies in the world, which in turn contributed to the improvement of the PORTAGE system.

Another key observation pertains to the limited research collaboration between NRC and its strategic partners, UQO and BtB. It was expected, based on the original TB submission, that collaborative research projects would be undertaken between these partners. In this regard, it was found that over the eight years of existence of the ILT group, only two research projects (Barça and LOPLT) has been developed in collaboration with UQO and that no collaborative research was undertaken with BtB.

With regard to UQO, the limited collaboration can be explained by the lack of strong capabilities of the institution in terms of language technologies research. In other words, a very small number of professors (approximately two) are conducting research in this field. For its part, the BtB was expected to be a test-bed for the development of the ILT group technologies. However, the public organization was reluctant to contribute to the development of a technology system that may benefit their competitors (i.e., private sector translation companies). In sum, it was found that the absence of synergy between the partners stems from the original program design since the research capabilities of the UQO were overestimated and the particular position of the BtB with regards to the private sector was not anticipated. Although the UQO expressed its desire to increase its research capabilities in this field and despite the fact that the PORTAGE technology was transferred to the BtB, there is little evidence that collaborative projects will be undertaken in the near future.

5.1.2 Contribution to the Creation of Innovative Technologies

The research activities of the ILT group has resulted in increased access by Canadian companies to innovative technologies and tools that are considered among the best in the world.

The group has created innovative technologies in the field of translation aid, document categorization/management and terminology. While the quality of these technologies was confirmed by all of the evaluation participants consulted, the PORTAGE system was usually referenced as the key accomplishment of the group so far. The strong performance of this system was also demonstrated by the results obtained by the group in international evaluations where the SMT systems developed by different research laboratories around the world are compared to each other.

5.1.2.1 Innovative Technology Created

Over the last three years, the group's main research activities have been dedicated to the development or improvement of six core technologies or tools. While some of these technologies are mature enough to be transferred to industry, some were still under development when the evaluation was conducted. These technologies and tools are briefly described in Table 2. During the same period, the ILT group also filed four patent applications to protect the Intellectual Property (IP) generated by these research projects.

Technologies and Tools of ILT Group by Field of Research (2008 to 2012)
Technologies and Tools Description Field of Research Status of Project Footnote 19
PORTAGE Machine translation system Footnote 20 that produces translations for any language pair provided that a collection of translated texts is available for the language pair of interest, and from which PORTAGE will extract the necessary translation knowledge. Translation Aid Completed
Licensed
WeBiText Online tool that allows translators to quickly search the Web for multilingual sites containing terminology of interest in the user-selected source and target languages; the user enters a word or expression in the source language, and WeBiText displays a series of current bilingual web pages that contain both the word or expression in the source language and their equivalent in the target language. Translation Aid Completed
Licensed
Project Integrated Tools for Translators (ITT) Development of tools which will help translators make more effective use of existing translation aid technologies. Two current priority topics are: 1) the use of confidence scores on the output of machine translation and translation memory systems in order to show only the most reliable translations to the translator; and 2) the use of techniques that can automatically select the translation references that are best adapted to the needs of a specific user; the latter topic is relevant to different translation aid technologies: machine translation, translation memory systems, and terminology tools. Translation Aid In development
PROFACT Technology toolbox for the automated management of electronic documents using different approaches adapted to the needs of users, and exploiting various characteristics of the documents of interest (content, format, labels, etc.). The two main types of applications of this technology toolbox are either to establish a classification of documents where none existed beforehand, or to dispatch new documents into a known set of document categories. Content Management In development
Improving Translation Through Corpus Exploration (ITTCE) This project applies PROFACT technologies in a collaboration with the U.S. National Virtual Translation Centre (NVTC), a translation service that caters to the needs of the U.S. Defence and Intelligence communities; it helps determine groups of documents that are better candidates for the use of particular NVTC translation workflows which may use technologies such as machine translation or translation memories. Translation Aid and Content Management In development
TerminoWeb 2.0 Online tool that seeks Web content according to a simple user input (e.g. one word only, such as "vitamin") and that selects the most pertinent and information-rich texts found on the Web that correspond to the user request; once this first step is accomplished, the user can use other functions of TerminoWeb to explore the terminology of the texts obtained as well as the semantic relationships expressed in these. Terminology Completed
5.1.2.2 Quality of the Technologies Developed by the ILT Group

There was a consensus among the external interviewees that all of the technologies and tools developed by the ILT group are highly innovative and have a strong potential for commercialization. The PORTAGE system was particularly praised by the evaluation participants who recognized real opportunities for the development of a commercial product that would help to move the technology closer to the market. In this regard, an expert in the field of translation technologies reported that: "PORTAGE is a brilliant technology".

One of the best ways for a research laboratory to determine the performance of its Machine Translation (MT) system is by participating in international benchmarking competitions. Based on the information provided by the ILT group, it appears that the PORTAGE technology performed well in all of the eight competitions in which it participated since 2005. This is the case for the NIST MT Evaluation as well as for other evaluations of MT systems organized by the University of Edinburgh. Footnote 21 When asked to comment about the quality of the technology developed by the group, the evaluation participants usually referred to the performance of PORTAGE in these events as a key indicator of the quality of the system.

5.2 Strengthening of the Canadian Language Industry

The evaluation found that the ILT group has undertaken significant efforts to transfer its technologies to its targeted clientele. At the same time, these efforts yielded limited results as only four commercial licenses have been established with clients (i.e., public organizations or companies). It was also found that the impact on the companies and organizations that licensed the technologies depends on the type of technology transferred. The evaluation findings show that the limited impacts of the group with regard to this expected outcome are explained by the numerous barriers faced by the group when transferring technology.

5.2.1 Licensing of Technologies and Impacts on Clients

Over the last three years, four commercial licenses have been granted to clients. While two clients reported an increased competitive advantage and increased sales, the clients that licensed PORTAGE could not identify any impact as it was too early in the technology transfer process when the evaluation was conducted. These latter clients however recognized a strong potential for improved productivity of quality translation products.

Although the group was successful at creating innovative technologies, the evaluation found that the group had a small impact on the growth of the language technology industry. In fact, only four clients, three private sector clients and the Translation Bureau of Canada (BtB), licensed technologies from the ILT group. Footnote 22

The impacts reported by the clients consulted varied depending on the technology that was licensed. In the case of technologies other than PORTAGE, the clients expressed a high level of satisfaction with the technologies in addition to an increased competitive advantage and increased sales.

The two clients that licensed PORTAGE, CLS-Lexitech and BtB, could not identify any impact as both organizations were in the process of implementing and assessing the performance of the system when the interviews were conducted. Nonetheless, both organizations noted that preliminary results stemming from internal assessments of the system indicated a potential for increased productivity. In this regard, one client reported that the expected productivity gain will have an impact on the competitiveness of the company in a context where the use of MT systems is becoming an inevitable trend in the industry. Footnote 23 These early results are consistent with studies that show that the use of MT systems can result in increased productivity for a company. Footnote 24 These studies are also consistent with the result of a survey that was commissioned by the LTRC in 2011. More specifically, 97% of survey respondents reported that language technologies accelerated translation work and 90% noted that these tools contributed to increased quality of translations. Footnote 25

The transfer of PORTAGE to two of the biggest players in the Canadian translation industry, CLS-Lexitech and BtB, represents a significant achievement for the group considering the number of translators that will now have access to the technology. Moreover, the transfer of PORTAGE to the BtB will help the organization to achieve its mandate to offer translation services to federal departments who are required to use both official languages as required by the Official Languages Act. This represents a significant contribution to the strategic objective of the Roadmap for Canadian Linguistic Duality.

An external interviewee, when talking about the PORTAGE system, summarized the situation of the group in terms of business development in the following manner: "NRC is sitting on a gold mine but doesn't know what to do with it. The business aspect is missing. They need a game plan to commercialize the tool." In this regard, many evaluation participants highlighted opportunities for the commercialization of the technologies in foreign markets.

  • Recommendation 2: NRC must develop and implement a strategy to commercialize the PORTAGE technology and associated IP, and thus, increase its impact on the language industry. NRC should also explore the business opportunities available in other industry sectors because potential use/applications for this technology were identified as part of the evaluation (e.g., security and defense sector).

5.2.2 Pre-Transfer Through Collaborative Projects

The ILT group has adopted flexible technology transfer strategies to overcome one of the key obstacles to technology transfer: the general skepticism and resistance of end users toward language technologies.

One of the key themes that emerged from the discussions with experts of the language technology industry pertains to the significant resistance of the end-users, mainly translators, toward the use of technologies. In order to address this challenge, the ILT group has established pilot projects and facilitated access to its technology by developing low-cost evaluation licenses for companies who wish to assess their performance and commercialization potential. Footnote 26 The performance data provided by the group show that 11 pilot projects and evaluation licenses were established over the last three years for a total value of $772K. As an example, the group has worked with the Government of Ontario to implement a terminology tool to help translate job offers on the government's website. In the opinion of an internal interviewee, another benefit of evaluation licenses and pilot projects is the fact that it helps the group assess the absorption capacity of the industry.

5.2.3 Technology Transfer Challenges

Despite the existence of several obstacles to the transfer of technologies, the interviewees consulted felt that the impacts of the ILT group on the industry could be increased with the proper commercialization strategy.

The numerous and significant challenges faced by the ILT group explain the mixed results with respect to the transfer of technology and thus, its impact on the industry. In fact, there was a consensus among the interviewees that doing business with NRC is complex and time-consuming. In contrast, the evaluation participants showed a paradoxical interest in the group's technologies.

The business ecosystem in which the group operates is summarized in Figure 3, below. It provides an outline of the key barriers to technology transfer.

Figure 3 - Summary of key barriers to technology transfer

The main barriers that the Interactive Language Technologies group is facing as part of the technology transfer process are illustrated in figure 3, which essentially follows the transfer model described in figure 2. Figure 3 presents obstacles pertaining to each individual actor (i.e., the Interactive Language Technologies group, language technology companies, end-users) in the context of the technology transfer process. These barriers are described in details in the document.

Summary of key barriers to technology transfer

The key barriers identified as part of the evaluation can be described as follows Footnote 27:

  • Strong competitive environment - The language technology industry counts a very limited number of companies (i.e., less than ten companies) that are competing in a small market. This is particularly the case for translation aid tools; in this area, companies are protective of any competitive advantage that they can obtain from doing business with NRC. Although this is the case for any company in any sector of the economy, it appears to be exacerbated in the context of the language Technology industry where companies are looking for exclusive licenses. The evaluation found that this had a strong impact on the negotiation process between NRC and its collaborators.
  • Limited receptor capabilities in the case of translation companies - As previously discussed, the PORTAGE technology is complex and requires the receptor company to have strong IT capabilities. This reduces the number of translation companies that can acquire and implement such a complex system. In addition to this, translation firms need to have a large body of translated text in order to be able to train the system properly. Once trained, the system performs better for work done with clients that have significant translation needs.
  • Financial capacity of language technology companies - The PORTAGE system in its current format cannot be offered as a commercial product. The development of the PORTAGE technology into a market-ready solution for the translation market would require a significant investment from a language technology company. The evaluation evidence shows that a limited number of companies have the financial capability to undertake a project of this magnitude. Also, any company that would be interested in doing this would certainly need a guarantee that these financial risks are protected through an exclusive license.
  • Resistance of translators toward Machine Translation systems - Many evaluation participants pointed out the strong resistance of a large number of translators who are skeptical of the ability of MT systems to increase their productivity and/or produce quality translations. In fact, it appears that translators rarely learn how to use language technologies as part of their university training, which may contribute to this culture of resistance.
  • Perceived complexity of doing business with NRC within the community - There was a shared perception among the evaluation participants, including those that did not collaborate with NRC in recent years, that doing business with NRC is a lengthy and complex process. Some of the reasons provided by internal and external stakeholders to explain this perception include: the importance for NRC to ensure a return on investment that is consistent with the objectives of the new strategy, the fact that companies tend to insist on obtaining long-term exclusivity clauses in their license agreements to protect their competitive advantage and the approval process established by NRC-IIT for all of its projects, including non-commercial agreements.
  • Existence of alternative MT systems - The existence of alternative systems is an additional hindrance for NRC when negotiations are taking place with clients. In fact, although the PORTAGE system presents several advantages over Moses, clients may be inclined to use the latter system if the cost of the PORTAGE license is too expensive.

5.2.4 NRC-IRAP Support to the Language Industry

NRC-IRAP has supported the growth of the language technology industry through the provision of funding and advisory services delivered by the ITA located on the premises of the CRTL.

The NRC Industrial Research Assistance Program (NRC-IRAP) was also found to play a key role in supporting the language technology industry. More specifically, the advisory and referral services provided by the Industrial Technology Advisor (ITA) as well as the funding granted by NRC-IRAP for R&D projects were strongly valued by the external participants. The performance data provided by NRC-IRAP indicate that eight companies benefited from advisory and/or funding services. One of the interviewees that received services from NRC-IRAP noted that: "NRC-IRAP is a good partner for us. It's a good vehicle for technology transfer."

The total value of the funding provided by NRC-IRAP to language technology companies since 2005, including the Youth Employment Program and the R&D project funding, was a little over $2.5M. It should also be noted that this amount does not include the salary of the ITA or any other type of support provided by the Program such as the hiring of an expert consultant for the assessment/development of strong R&D project proposals.

5.3 Contribution to the Strengthening of Canada's Innovation System

When the APOL was launched in 2003, one of the key elements of the government's strategy in support of the language industry involved the creation of a research centre in the field of language technologies. The responsibility to establish this research centre was given to NRC, UQO and BtB.Footnote 28 In 2004, the three partners, in collaboration with the industry, were successful at creating a non-profit organization, the LTRC.

One of the key observations that stem from the evaluation pertains to the significant changes experienced by the LTRC since the last evaluation was conducted. The LTRC is now an independent non-profit organization that is supported by various partners, including the three founding partners.

5.3.1 NRC's Contribution to the Development of the Local Innovation System in the field of Language Technologies

NRC's contribution to the growth of the local innovation system is valued by all key stakeholders consulted.

NRC, including the ILT group and NRC-IRAP, was found to be a key player in the local innovation system. In addition to its contribution to the development and transfer of innovative technologies, NRC has supported the growth of this local innovation system in the following manner:

  • Organization of events and seminars that bring together the scientific community;
  • Attraction of innovative firms and investments to the business incubator due to the physical proximity of researchers;
  • Sharing of formal and informal knowledge and expertise with university researchers and entrepreneurs; and,
  • Training of HQP.

In addition to these elements, it was found that NRC is perceived by a vast majority of stakeholders as one of the main pillars that support the growth and ensure the sustainability of the LTRC. The other pillars identified by the interviewees are the presence and involvement of a strong industry sector and the training of HQP by the UQO.

5.3.2 Evolution of the LTRC as an independent organization

The overall situation of the LTRC has significantly improved since 2007 and the organization is now positioned to support the NRC in the development of the language technology industry.

The LTRC has significantly improved its overall situation since the last evaluation was conducted. In 2007, the LTRC had no Director General, industrial involvement in its governance structure was limited, its financial situation was uncertain, the business incubator was not operating at full capacity and few projects had been undertaken by the organization itself. In contrast, the LTRC now possesses all the tools required to contribute to the growth of the language technology industry. More specifically, the new Director General, in collaboration with his partners, was successful at stabilizing the financial situation of the organization. As a result of this stability, the LTRC has:

  • Established a research network that involved universities from all over Canada;
  • Attracted language technology firms to its incubator; and,
  • Developed a large scale project, Linguistech, to increase the access of language technologies by the end-users.Footnote 29

Over the last three years, the governance structure of the LTRC has evolved to allow for an increased role of the industry in the decision making process of the not-for-profit organization. Recent amendments to the regulations concerning the participation of federal departments on the boards of directors of non-profit organizations have been a key driver for this change. NRC now acts as an observer and is therefore not involved in the decision-making process of the LTRC. In addition to the changes made to the board membership, all of the previous supporting ad hoc committees, such as the scientific committee, no longer exist and have been instead replaced with informal meetings between NRC and the LTRC. This change was not identified as a major concern by any of the evaluation participants consulted.

6.0 ECONOMY AND EFFICIENCY OF THE INTERACTIVE LANGUAGE TECHNOLOGIES GROUP

The manner in which the ILT group has used its resources to achieve its expected outcomes was also examined as part of the evaluation. While the issue of efficiency examines whether the level of output produced by the group is acceptable given the level of resources invested, the issue of economy verifies whether efforts were made to reduce the cost of the input used by the group to achieve its results.

6.1 Efficiency

The ILT group has made good use of its resources by developing innovative technologies that are considered some of the best in the world and are valued by its collaborators.

The ILT group, which follows the administrative processes and practices established by NRC Corporate Branches and NRC-IIT, was found to be operating in an efficient manner. In addition to the fact that NRC management expressed a strong level of satisfaction with the group's achievements, it was demonstrated as part of this report that the group used its resources to produce innovative technologies that are considered among the best in the world. Considering the limited resources allocated to the group, this represents an important achievement.

Only one key risk was identified as a result of the evaluation. More specifically, the management of NRC Information Technology (IT) services by Shared Services Canada was found to present a risk with regard to the ability of the group to achieve its expected outcomes in an efficient manner. Because of the intensive use of IT in the ILT group's research activities, some internal interviewees expressed a concern about the impact of the transfer of NRC IT resources to Shared Services Canada. In fact, it is essential for the group to receive a minimum level of service and to access certain specialized/tailored IT services in order to maintain the same level of performance. According to evaluation participants, a backlog of service requests has already been reported by the researchers. While this situation presents a potential risk for the ILT group, the evaluation could not identify strong evidence that this risk had materialized as of January 2012.

6.2 Economy

Opportunities for increased economies were identified with regard to the space leased by the ILT group.

Overall, the group was found to manage its resources in an economical manner. This statement, which stems primarily from discussions with evaluation participants, is also supported by the fact that the group's operating costs have decreased over the last three years (See Figure 1 earlier in this report). The evaluation only identified two opportunities for improvement in terms of economy, which both pertain to the leasing agreement with UQO. More specifically, it was mentioned by internal evaluation participants that the cost of the lease with UQO is more expensive than the cost for NRC to locate the group in the empty space available in its Montreal Road facilities in Ottawa.Footnote 30 However, external interviewees raised the concern that the move could negatively impact the LTRC partners because the ILT group constitutes the core of the research activity of the centre. It should also be noted that the BtB left the facilities in 2011 to reduce its operating costs within the context of government budget reductions. Under these circumstances, the departure of another key founding partner would certainly have an impact on the credibility and sustainability of the LTRC.

It was also found that the group is renting more space than it needs. Based on discussions with evaluation participants, approximately 25% of the space rented by the group was not being used when the evaluation was conducted.

  • Recommendation 3: NRC should ensure that the space rented reflects the group's needs and adjust future leasing agreements accordingly if the ILT group remains in its current facilities.

7.0 CONCLUSION

The creation of the ILT group has resulted in the development of innovative knowledge, technologies and tools that are valued by the academic community, the language industry, and all the key collaborators involved in the LTRC. The ILT group's technologies, which were found to be among the best in the world, have been transferred to a limited number of Canadian companies. Although two of the biggest translation organizations in Canada are now using PORTAGE, which should reduce the resistance of translators toward SMT systems, several barriers are still hindering the group's ability to strengthen the Canadian language industry. In this regard, the evaluation found that with the appropriate commercialization strategy, the ILT group could have a significant impact on the growth of the language industry. The establishment of the proper partnerships and strategy will ensure the value for money of NRC investments in the field of language technologies in support of Canada's official languages.

After eight years of contribution to the field of language technologies, the ILT group is currently at a crossroads in terms of its future research activities as internal and external elements have created the needed conditions to change direction. On one hand, the development of the PORTAGE system, which constitutes the bulk of the group's research activities, has reached a sufficient level of maturity and stability and thus, does not require the same level of investment as it did in the past. As such, the ILT group has the opportunity to undertake research in new areas or to explore the potential applications of the PORTAGE system to other sectors. From a technology transfer perspective, the evaluation found that given the absence of strong receptor capacity in the translation industry, very limited business opportunities remain in this sector in Canada. However, the commercialization of the PORTAGE technology for foreign markets or other applications has a significant business potential for language technologies companies and NRC.

On the other hand, the development of the new NRC strategy has created the need for all research groups to realign their activities with the new programs and priorities of the organization. In this context, opportunities were identified by internal and external evaluation participants in the field of security, which is consistent with the objectives of the Security and Disruptive Technologies and Information and Communications Technologies portfolios. Although the program development process had not been completed when the evaluation was conducted, the evaluation findings presented in this report confirm that the use of the PORTAGE technology for security and defense applications present real opportunities for the organization. However, the success of this niche area will depend on the ability of the group to attract key partners from the defense and security sectors, including partners from other government departments and the private sector.

Despite the need for the ILT group to realign its activities with the new NRC program and priorities, the group remains accountable for some of the strategic objectives of the Roadmap for Canada's Linguistic Duality. Therefore, it will be critical that NRC maintain a balance between these dual mandates in order to remain relevant.

While the group has demonstrated good value for money (i.e., relevance and performance), opportunities for improvement were identified as a result of the evaluation. More specifically, three recommendations are proposed with regards to the strategic alignment of the group as part of the NRC strategy, its contribution to the transfer of technology and the use of its resources. These recommendations are presented in the following section.

8.0 MANAGEMENT RESPONSE

Recommendation 1:

As long as the ILT group is considered an initiative under the Roadmap for Canada's Linguistic Duality, NRC must balance the contribution of the group's activities to the new NRC Portfolio/Program structure with its contribution to the strategic objective of the Roadmap (i.e., to strengthen the Canadian language industry).

Response:

Accepted

Planned Action:

The ILT group focuses on developing computer-based tools for processing oral and written information in multiple languages in accordance with the mandate received through the Action Plan for Official Languages and the Roadmap for Linguistic Duality. Language technologies are expected to continue to play a significant role in the NRC-Information and Communications Technologies (NRC-ICT) and the Security and Disruptive Technologies (NRC-SDT) strategies, including applications such as business intelligence and security. NRC intends to continue to support Canadian language industries while balancing its contribution according to industry and national needs and its strategic directions.

Proposed Person:

Vice-President, Emerging Technologies Division

Timelines:

March 2013

Measure of Achievement #1:

NRC met its 5 year commitment to the Roadmap for Canada's Linguistic Duality 2008-13.

Measure of Achievement #2:

NRC-ICT and SDT portfolio strategies are in place reflecting the role of language technologies and the capabilities of the ILT group are effectively deployed into programs.

Recommendation 2:

NRC must develop and implement a strategy to commercialize the PORTAGE technology and associated IP, and thus, increase its impact on the language industry. NRC should also explore the business opportunities available in other industry sectors because potential use/applications for this technology were identified as part of the evaluation (e.g., security and defense sector).

Response:

Partially accepted

Planned Action:

A formal commercialization plan already exists for PORTAGE. As NRC develops its ICT portfolio strategy, the commercialization plan will be updated in the context of NRC's new business approach, the receptor capacity in the Canadian language technology industry and opportunities in other industry sectors such as security.

Proposed Person:

ICT Portfolio General Manager

Timelines:

March 2013 for measure of achievement #1

March 2015 for measure of achievement #2

Measure of Achievement #1:

An updated commercialization plan is in place as part of the ICT portfolio strategy.

Measure of Achievement #2:

Two additional commercial licences for PORTAGE.

Recommendation 3:

NRC should ensure that the space rented reflects the group's needs and adjust future leasing agreements accordingly if the ILT group remains in its current facilities.

Response:

accepted

Planned Action:

Space needs will be reviewed in light of NRC requirements, operational efficiencies, ICT portfolio strategy, and Treasury Board directives.

Proposed Persons:

ICT Portfolio General Manager

Director General ASPM

Timelines:

March 2013

Measure of Achievement:

A review of space requirements conducted.

The space occupied by the ILT Group is optimized to achieve business goals and operational efficiency, in accord with new Treasury Board directives and enable the ICT Portfolio strategy.

APPENDIX 1 - EVALUATION ISSUES AND QUESTIONS

Relevance

R1. Continued Need for the Program

Does the IIT-ILTG continue to address a demonstrable need?

R2. Alignment with Government Priorities

Is the program aligned to federal government priorities and to NRC's new strategy and priorities?

R3. Alignment with Federal Roles and Responsibilities

Is NRC's current role in delivering the program appropriate, and consistent with federal roles and responsibilities?

Performance

P1. Achievement of Expected Outcomes

To what extent has the IIT-ILTG helped strengthened the capacity of the Canadian language industry?

To what extent has NRC helped strengthened the Canadian innovation system through its participation in the development of a research center in the field of language technologies (i.e., LTRC)?

To what extent has the IIT-ILTG helped strengthened Canada's position in the field of language technologies at the international scale?

P2. Demonstration of Economy and Efficiency

Have the IIT-ILTG resources been used in an economic manner?

Is the IIT-ILTG administered in an efficient manner?

What are the factors (internal or external) that have an impact on the cost-effectiveness of the IIT-ILTG?

BIBLIOGRAPHY

National Research Council (2008), Evaluation of the Interactive Language Technologies Group - Final Report

National Research Council (2010), Project: PORTAGE - Technology Transfer Plan

National Research Council (2010), Departmental Performance Report - Section III Supplementary Information

Government of Canada (2003), Action Plan for Official Languages

Government of Canada (2007), Mobilizing Science and Technology to Canada's Advantage

Government of Canada (2008), Roadmap for Canada's Linguistic Duality 2008-13: Acting for the Future

Department of Justice, Canadian Charter of Rights and Freedoms, Site consulted on January 10, 2012

Department of Justice, Official Languages Act, Site consulted on January 10, 2012

Masselot F. & M., Plitt (2010). A Productivity Test of Statistical Machine Translation Post-Editing in a Typical Localisation Context, The Prague Bulletin of Mathematical Linguistics, No 93, p. 7-16

Language Technology Research Center (2011), Preliminary Summary Report on the Results of the Survey Conducted among Users of Language Technologies

Footnotes

Footnote 1

Unless indicated otherwise, any reference to the LTRC in this report refers to the LTRC not for profit organization.

Return to footnote 1 referrer

Footnote 2

The public document about the Roadmap and Canadian Heritage's internal documents are referring to the LTRC when describing NRC investments in support of language technology research.

Return to footnote 2 referrer

Footnote 3

National Research Council (2008), Evaluation of the Interactive Language Technologies Group - Final Report, p. 9-14

Return to footnote 3 referrer

Footnote 4

Government of Canada (2008), Roadmap for Canada's Linguistic Duality 2008-13: Acting for the Future, p. 13

Return to footnote 4 referrer

Footnote 5

Additional information about the LTRC is available.

Return to footnote 5 referrer

Footnote 6

The funds, which were originally granted as sunsetting funds (B-base funds), were converted into permanent funding for NRC (A-base funding).

Return to footnote 6 referrer

Footnote 7

National Research Council (2010), Departmental Performance Report - Section III Supplementary Information, p. 28

Return to footnote 7 referrer

Footnote 8

The value of the contribution of the Interactive Information group of NRC-IIT is reflected in Figure 1.

Return to footnote 8 referrer

Footnote 9

This total does not include NRC in-kind contribution to the total project or agreement value.

Return to footnote 9 referrer

Footnote 10

Government of Canada (2003), The Action Plan for Official Languages, p. 59

Return to footnote 10 referrer

Footnote 11

Government of Canada (2008), Roadmap for Canada's Linguistic Duality 2008-13: Acting for the future, p. 13

Return to footnote 11 referrer

Footnote 12

Department of Justice, Canadian Charter of Rights and Freedoms, Site consulted on January 10, 2012

Return to footnote 12 referrer

Footnote 13

Department of Justice, Official Languages Act , Site consulted on January 10, 2012

Return to footnote 13 referrer

Footnote 14

Information about SDL Language Weaver is available at this location, Site consulted on January 12, 2012

Return to footnote 14 referrer

Footnote 15

Government of Canada (2007), Mobilizing Science and Technology to Canada's Advantage, p. 12

Return to footnote 15 referrer

Footnote 16

The ACCORD program provides funding for the development of strategic industry sectors in a specific region. The funds invested as part of this program can be used for a variety of purposes including marketing campaign or strategic planning. Additional information about the ACCORD program in support of the language industry in the Outaouais region is available at this location:

Return to footnote 16 referrer

Footnote 17

The Information and Security program was in the process of being developed when the evaluation was conducted.

As such, it was not possible for NRC management to clearly describe the future research orientation of the group.

Return to footnote 17 referrer

Footnote 18

NRC, NRC-IIT Researcher Wins Best Paper Award at TALN Automatic Natural Language Processing Conference, Site consulted on January 11, 2012

Return to footnote 18 referrer

Footnote 19

The status of the project was provided by the ILT group and indicates whether internal resources are used to pursue the development of the technology and/or if the technology is mature enough to be transferred to the industry.

Return to footnote 19 referrer

Footnote 20

Additional information as well as the rudiments of SMT systems are discussed at the following locations, Sites consulted on January 11, 2012

Return to footnote 20 referrer

Footnote 21

For a complete description of the results of the PORTAGE system in benchmarking competitions of MT System, consult the following website

Return to footnote 21 referrer

Footnote 22

Specific information about the licensing agreements could not be included in the report for confidentiality reasons.

Return to footnote 22 referrer

Footnote 23

NRC, NRC-IIT Announcement, Site consulted on January 12, 2012

Return to footnote 23 referrer

Footnote 24

Masselot F. & M., Plitt (2010). A Productivity Test of Statistical Machine Translation Post-Editing in a Typical Localisation Context, The Prague Bulletin of Mathematical Linguistics, No 93, p. 7-16

Return to footnote 24 referrer

Footnote 25

Language Technologies Research Centre, Idem, p. 10

Return to footnote 25 referrer

Footnote 26

Language Technologies Research Centre (2011), Preliminary Summary Report on the Results of the Survey Conducted among Users of Language Technologies, p. 10

Return to footnote 26 referrer

Footnote 27

Most of these limitations are specific to the PORTAGE technology.

Return to footnote 27 referrer

Footnote 28

The TBS submission details the responsibilities of the three partners in support of the language industry.

Return to footnote 28 referrer

Footnote 29

More information about the Linguistech project is available at this location

Return to footnote 29 referrer

Footnote 30

It has to be noted that this information was not validated through the conduct of a thorough cost analysis.

It is therefore difficult to estimate the potential savings that the Institute would incur by moving the group.

Return to footnote 30 referrer

>
Date modified: