publications

2020

1. When and Why is Unsupervised Neural Machine Translation Useless?

   Yunsu Kim, Miguel Graça, and Hermann Ney

   Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT 2020)

   Abs PDF Slides Video

   This paper studies the practicality of the current state-of-the-art unsupervised methods in neural machine translation (NMT). In ten translation tasks with various data settings, we analyze the conditions under which the unsupervised methods fail to produce reasonable translations. We show that their performance is severely affected by linguistic dissimilarity and domain mismatch between source and target monolingual data. Such conditions are common for low-resource language pairs, where unsupervised learning works poorly. In all of our experiments, supervised and semi-supervised baselines with 50k-sentence bilingual data outperform the best unsupervised results. Our analyses pinpoint the limits of the current unsupervised NMT and also suggest immediate research directions.

2019

1. Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages

   Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney

   Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019)

   Abs PDF Poster

   We present effective pre-training strategies for neural machine translation (NMT) using parallel corpora involving a pivot language, i.e., source-pivot and pivot-target, leading to a significant improvement in source-target translation. We propose three methods to increase the relation among source, pivot, and target languages in the pre-training: 1) step-wise training of a single model for different language pairs, 2) additional adapter component to smoothly connect pre-trained encoder and decoder, and 3) cross-lingual encoder training via autoencoding of the pivot language. Our methods greatly outperform multilingual models up to +2.6% BLEU in WMT 2019 French-German and German-Czech tasks. We show that our improvements are valid also in zero-shot/zero-resource scenarios.

2. When and Why is Document-level Context Useful in Neural Machine Translation?

   Yunsu Kim, Duc Thanh Tran, and Hermann Ney

   Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)

   Abs PDF Code Poster Slides

   Document-level context has received lots of attention for compensating neural machine translation (NMT) of isolated sentences. However, recent advances in document-level NMT focus on sophisticated integration of the context, explaining its improvement with only a few selected examples or targeted test sets. We extensively quantify the causes of improvements by a document-level model in general test sets, clarifying the limit of the usefulness of document-level context in NMT. We show that most of the improvements are not interpretable as utilizing the context. We also show that a minimal encoding is sufficient for the context modeling and very long context is not helpful for NMT.

3. Generalizing Back-Translation in Neural Machine Translation

   Miguel Graça, Yunsu Kim, Julian Schamper, Shahram Khadivi, and Hermann Ney

   Proceedings of the Fourth Conference on Machine Translation (WMT 2019)

   Abs PDF Slides

   Back-translation — data augmentation by translating target monolingual data — is a crucial component in modern neural machine translation (NMT). In this work, we reformulate back-translation in the scope of cross-entropy optimization of an NMT model, clarifying its underlying mathematical assumptions and approximations beyond its heuristic usage. Our formulation covers broader synthetic data generation schemes, including sampling from a target-to-source NMT model. With this formulation, we point out fundamental problems of the sampling-based approaches and propose to remedy them by (i) disabling label smoothing for the target-to-source model and (ii) sampling from a restricted search space. Our statements are investigated on the WMT 2018 German <-> English news translation task.

4. The RWTH Aachen University Machine Translation Systems for WMT 2019

   Jan Rosendahl, Christian Herold, Yunsu Kim, Miguel Graça, Weiyue Wang, Parnia Bahar, Yingbo Gao, and Hermann Ney

   Proceedings of the Fourth Conference on Machine Translation (WMT 2019)

   Abs PDF

   This paper describes the neural machine translation systems developed at the RWTH Aachen University for the German-English, Chinese-English and Kazakh-English news translation tasks of the Fourth Conference on Machine Translation (WMT19). For all tasks, the final submitted system is based on the Transformer architecture. We focus on improving data filtering and fine-tuning as well as systematically evaluating interesting approaches like unigram language model segmentation and transfer learning. For the De-En task, none of the tested methods gave a significant improvement over last years winning system and we end up with the same performance, resulting in 39.6% BLEU on newstest2019. In the Zh-En task, we show 1.3% BLEU improvement over our last year’s submission, which we mostly attribute to the splitting of long sentences during translation. We further report results on the Kazakh-English task where we gain improvements of 11.1% BLEU over our baseline system. On the same task we present a recent transfer learning approach, which uses half of the free parameters of our submission system and performs on par with it.

5. Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron

   Yunsu Kim, Hendrik Rosendahl, Nick Rossenbach, Jan Rosendahl, Shahram Khadivi, and Hermann Ney

   Proceedings of the Fourth Workshop on Representation Learning for NLP (RepL4NLP 2019)

   Abs PDF Poster

   We propose a novel model architecture and training algorithm to learn bilingual sentence embeddings from a combination of parallel and monolingual data. Our method connects autoencoding and neural machine translation to force the source and target sentence embeddings to share the same space without the help of a pivot language or an additional transformation. We train a multilayer perceptron on top of the sentence embeddings to extract good bilingual sentence pairs from nonparallel or noisy parallel data. Our approach shows promising performance on sentence alignment recovery and the WMT 2018 parallel corpus filtering tasks with only a single model.

6. Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies

   Yunsu Kim, Yingbo Gao, and Hermann Ney

   Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019)

   Abs PDF Code Poster

   Transfer learning or multilingual model is essential for low-resource neural machine translation (NMT), but the applicability is limited to cognate languages by sharing their vocabularies. This paper shows effective techniques to transfer a pretrained NMT model to a new, unrelated language without shared vocabularies. We relieve the vocabulary mismatch by using cross-lingual word embedding, train a more language-agnostic encoder by injecting artificial noises, and generate synthetic data easily from the pretraining data without back-translation. Our methods do not require restructuring the vocabulary or retraining the model. We improve plain NMT transfer by up to +5.1% BLEU in five low-resource translation tasks, outperforming multilingual joint training by a large margin. We also provide extensive ablation studies on pretrained embedding, synthetic data, vocabulary size, and parameter freezing for a better understanding of NMT transfer.

2018

1. Improving Unsupervised Word-by-Word Translation with Language Model and Denoising Autoencoder

   Yunsu Kim, Jiahui Geng, and Hermann Ney

   Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018)

   Abs PDF Code Slides

   Unsupervised learning of cross-lingual word embedding offers elegant matching of words across languages, but has fundamental limitations in translating sentences. In this paper, we propose simple yet effective methods to improve word-by-word translation of cross-lingual embeddings, using only monolingual corpora but without any back-translation. We integrate a language model for context-aware search, and use a novel denoising autoencoder to handle reordering. Our system surpasses state-of-the-art unsupervised translation systems without costly iterative training. We also analyze the effect of vocabulary size and denoising type on the translation performance, which provides better understanding of learning the cross-lingual word embedding and its usage in translation.

2. The RWTH Aachen University Supervised Machine Translation Systems for WMT 2018

   Julian Schamper, Jan Rosendahl, Parnia Bahar, Yunsu Kim, Arne Nix, and Hermann Ney

   Proceedings of the Third Conference on Machine Translation (WMT 2018)

   Abs PDF

   This paper describes the statistical machine translation systems developed at RWTH Aachen University for the German→English, English→Turkish and Chinese→English translation tasks of the EMNLP 2018 Third Conference on Machine Translation (WMT 2018). We use ensembles of neural machine translation systems based on the Transformer architecture. Our main focus is on the German→English task where we to all automatic scored first with respect metrics provided by the organizers. We identify data selection, fine-tuning, batch size and model dimension as important hyperparameters. In total we improve by 6.8% BLEU over our last year’s submission and by 4.8% BLEU over the winning system of the 2017 German→English task. In English→Turkish task, we show 3.6% BLEU improvement over the last year’s winning system. We further report results on the Chinese→English task where we improve 2.2% BLEU on average over our baseline systems but stay behind the 2018 winning systems.

3. The RWTH Aachen University English-German and German-English Unsupervised Neural Machine Translation Systems for WMT 2018

   Miguel Graça, Yunsu Kim, Julian Schamper, Jiahui Geng, and Hermann Ney

   Proceedings of the Third Conference on Machine Translation (WMT 2018)

   Ranked 1st in English→German and German→English tasks

   Abs PDF Poster

   This paper describes the unsupervised neural machine translation (NMT) systems of the RWTH Aachen University developed for the English ↔ German news translation task of the EMNLP 2018 Third Conference on Machine Translation (WMT 2018). Our work is based on iterative back-translation using a shared encoder-decoder NMT model. We extensively compare different vocabulary types, word embedding initialization schemes and optimization methods for our model. We also investigate gating and weight normalization for the word embedding layer.

4. The RWTH Aachen University Filtering System for the WMT 2018 Parallel Corpus Filtering Task

   Nick Rossenbach, Jan Rosendahl, Yunsu Kim, Miguel Graça, Aman Gokrani, and Hermann Ney

   Proceedings of the Third Conference on Machine Translation (WMT 2018)

   Ranked 1st, 2nd, 5th, 5th in four tracks

   Abs PDF

   This paper describes the submission of RWTH Aachen University for the De→En parallel corpus filtering task of the EMNLP 2018 Third Conference on Machine Translation (WMT 2018). We use several rule-based, heuristic methods to preselect sentence pairs. These sentence pairs are scored with count-based and neural systems as language and translation models. In addition to single sentence-pair scoring, we further implement a simple redundancy removing heuristic. Our best performing corpus filtering system relies on recurrent neural language models and translation models based on the transformer architecture. A model trained on 10M randomly sampled tokens reaches a performance of 9.2% BLEU on newstest2018. Using our filtering and ranking techniques we achieve 34.8% BLEU.

2017

1. Unsupervised Training for Large Vocabulary Translation Using Sparse Lexicon and Word Classes

   Yunsu Kim, Julian Schamper, and Hermann Ney

   Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017)

   Abs PDF Poster

   We address for the first time unsupervised training for a translation task with hundreds of thousands of vocabulary words. We scale up the expectation-maximization (EM) algorithm to learn a large translation table without any parallel text or seed lexicon. First, we solve the memory bottleneck and enforce the sparsity with a simple thresholding scheme for the lexicon. Second, we initialize the lexicon training with word classes, which efficiently boosts the performance. Our methods produced promising results on two large-scale unsupervised translation tasks.

2016

1. A Comparative Study on Vocabulary Reduction for Phrase Table Smoothing

   Yunsu Kim, Andreas Guta, Joern Wuebker, and Hermann Ney

   Proceedings of the First Conference on Machine Translation (WMT 2016)

   Abs PDF Slides

   This work systematically analyzes the smoothing effect of vocabulary reduction for phrase translation models. We extensively compare various word-level vocabularies to show that the performance of smoothing is not significantly affected by the choice of vocabulary. This result provides empirical evidence that the standard phrase translation model is extremely sparse. Our experiments also reveal that vocabulary reduction is more effective for smoothing large-scale phrase tables.

2015

1. Subspace Clustering of Data Streams: New Algorithms and Effective Evaluation Measures

   Marwan Hassani, Yunsu Kim, Seungjin Choi, and Thomas Seidl

   Journal of Intelligent Information Systems 45 (3)

   Abs HTML

   Nowadays, most streaming data sources are becoming high dimensional. Accordingly, subspace stream clustering, which aims at finding evolving clusters within subgroups of dimensions, has gained a significant importance. However, in spite of the rich literature of subspace and projected clustering algorithms on static data, only three stream projected algorithms are available. Additionally, existing subspace clustering evaluation measures are mainly designed for static data, and cannot reflect the quality of the evolving nature of data streams. On the other hand, available stream clustering evaluation measures care only about the errors of the full-space clustering but not the quality of subspace clustering. In this article we present a method for designing new stream subspace and projected algorithms. We propose also, to the first of our knowledge, the first subspace clustering measure that is designed for streaming data, called SubCMM: Subspace Cluster Mapping Measure. SubCMM is an effective evaluation measure for stream subspace clustering that is able to handle errors caused by emerging, moving, or splitting subspace clusters. Additionally, we propose a novel method for using available offline subspace clustering measures for data streams over the suggested new algorithms within the Subspace MOA framework.

2. Phrase Table Smoothing with Word Classes

   Yunsu Kim

   Master's Thesis, Fakultät für Mathematik, Informatik und Naturwissenschaften, RWTH Aachen University

   Abs PDF Slides

   Phrase-based statistical machine translation (SMT) has a severe data sparsity prob- lem in computing translation probabilities. In this thesis, we present and analyze various usages of word classes to alleviate the sparsity and improve the translation quality of a phrase-based SMT system. First of all, we propose a novel smoothing formulation of phrase translation mod- els, using word classes on the word level within a phrase. Secondly, we modify the standard phrase-based decoder to utilize word class phrases as additional transla- tion options. Finally, we investigate the word alignments trained from word classes. The performance of our proposed approaches is measured on three different trans- lation tasks to prove their broad applicability. The experiments show that our smoothed translation model is comparable to the state-of-the-art word class models with a smaller number of features. In addition, our modified decoder significantly reduces the out-of-vocabulary rate and enhances the overall translation quality in both automatic metrics and manual evaluation. We also make an extensive comparison among different word class mappings in terms of their performance in phrase-based SMT. Our results reveal that only the number of classes affects the translation quality of our proposed methods, regard- less of the type of clustering algorithms and other parameters for estimating word classes.

2013

1. Subspace MOA: Subspace Stream Clustering Evaluation Using the MOA Framework

   Marwan Hassani, Yunsu Kim, and Thomas Seidl

   Proceedings of the 18th International Conference on Database Systems for Advanced Applications (DASFAA 2013)

   Best Demo Award Runner-up

   Abs HTML Code

   Most available static data are becoming more and more high-dimensional. Therefore, subspace clustering, which aims at finding clusters not only within the full dimension but also within subgroups of dimensions, has gained a significant importance. Recently, OpenSubspace framework was proposed to evaluate and explorate subspace clustering algorithms in WEKA with a rich body of most state of the art subspace clustering algorithms and measures. Parallel to it, MOA (Massive Online Analysis) framework was developed also above WEKA to provide algorithms and evaluation methods for mining tasks on evolving data streams over the full space only. Similar to static data, most streaming data sources are becoming high-dimensional, and tracking their evolving clusters is also becoming important and challenging. In this demonstrator, we present, to the best of our knowledge, the first subspace clustering evaluation framework over data streams called Subspace MOA. Our demonstrator follows the online-offline model which is used in most data stream clustering algorithms. In the online phase, users have the possibility to select one of three most famous summarization techniques to form the microclusters. In the offline phase, one of five subspace clustering algorithms can be selected. The framework is supported with a subspace stream generator, a visualization interface to present the evolving clusters over different subspaces, and various subspace clustering evaluation measures.

2. Effective Evaluation Measures for Subspace Clustering of Data Streams

   Marwan Hassani, Yunsu Kim, Seungjin Choi, and Thomas Seidl

   Proceedings of the Second Workshop on Quality Issues, Measures of Interestingness and Evaluation of Data Mining Models (QIMIE 2013)

   Abs HTML

   Nowadays, most streaming data sources are becoming high-dimensional. Accordingly, subspace stream clustering, which aims at finding evolving clusters within subgroups of dimensions, has gained a significant importance. However, existing subspace clustering evaluation measures are mainly designed for static data, and cannot reflect the quality of the evolving nature of data streams. On the other hand, available stream clustering evaluation measures care only about the errors of the full-space clustering but not the quality of subspace clustering. In this paper we propose, to the first of our knowledge, the first subspace clustering measure that is designed for streaming data, called SubCMM: Subspace Cluster Mapping Measure. SubCMM is an effective evaluation measure for stream subspace clustering that is able to handle errors caused by emerging, moving, or splitting subspace clusters. Additionally, we propose a novel method for using available offline subspace clustering measures for data streams within the Subspace MOA framework.