책읽는공대생

While the nal goal of a statistical machine translation system is to create a model of the target sentence E given the source sentence F, P(E j F), in this chapter we will take a step back, and attempt to create a language model of only the target sentence P(E). Basically, this model allows us to do two things that are of practical use.

Assess naturalness: Given a sentence E, this can tell us, does this look like an actual, natural sentence in the target language? If we can learn a model to tell us this, we can use it to assess the  uency of sentences generated by an automated system to improve its results. It could also be used to evaluate sentences generated by a human for purposes of grammar checking or error correction.

Generate text: Language models can also be used to randomly generate text by sampling a sentence E0 from the target distribution: E0 P(E).2 Randomly generating samples from a language model can be interesting in itself { we can see what the model \thinks" is a natural-looking sentences { but it will be more practically useful in the context of the neural translation models described in the following chapters.

In the following sections, we'll cover a few methods used to calculate this probability P(E).

3.1 Word-by-word Computation of Probabilities

As mentioned above, we are interested in calculating the probability of a sentence E = eT1 . Formally, this can be expressed as

P(E) = P(jEj = T; eT1 ); (3)

the joint probability that the length of the sentence is (jEj = T), that the identity of the rst word in the sentence is e1, the identity of the second word in the sentence is e2, up until the last word in the sentence being eT . Unfortunately, directly creating a model of this probability distribution is not straightforward,3 as the length of the sequence T is not determined in advance, and there are a large number of possible combinations of words.

As a way to make things easier, it is common to re-write the probability of the full sentence as the product of single-word probabilities. This takes advantage of the fact that a joint probability { for example P(e1; e2; e3) { can be calculated by multiplying together conditional probabilities for each of its elements. In the example, this means that P(e1; e2; e3) =P(e1)P(e2 j e1)P(e3 j e1; e2).

Figure 2 shows an example of this incremental calculation of probabilities for the sentence \she went home". Here, in addition to the actual words in the sentence, we have introduced an implicit sentence end (\h/si") symbol, which we will indicate when we have terminated the sentence. Stepping through the equation in order, this means we rst calculate the probability of \she" coming at the beginning of the sentence, then the probability of \went" coming next in a sentence starting with \she", the probability of \home" coming after the sentence prex \she went", and then nally the sentence end symbol \h/si" after \she went home". More generally, we can express this as the following equation:

P(E) = TY+1t=1P(et j et􀀀1 1 ) (4) 

where eT+1 = h/si. So coming back to the sentence end symbol h/si, the reason why we introduce this symbol is because it allows us to know when the sentence ends. In other words, by examining the position of the h/si symbol, we can determine the jEj = T term in our original LM joint probability in Equation 3. In this example, when we have h/si as the 4th word in the sentence, we know we're done and our nal sentence length is 3.

Once we have the formulation in Equation 4, the problem of language modeling now becomes a problem of calculating the next word given the previous words P(et j et􀀀1 1 ). This is much more manageable than calculating the probability for the whole sentence, as we now have a xed set of items that we are looking to calculate probabilities for. The next couple of sections will show a few ways to do so.

3.2 Count-based n-gram Language Models

The rst way to calculate probabilities is simple: prepare a set of training data from which we can count word strings, count up the number of times we have seen a particular string of words, and divide it by the number of times we have seen the context. This simple method, can be expressed by the equation below, with an example shown in Figure 3.

2 Statistical MT Preliminaries

First, before talking about any specic models, this chapter describes the overall framework of statistical machine translation (SMT) [16] more formally.

First, we dene our task of machine translation as translating a source sentence F =

f1; : : : ; fJ = fjFj1 into a target sentence E = e1; : : : ; eI = ejEj1 .Thus, any type of translation system can be dened as a function

^E = mt(F); (1)

which returns a translation hypothesis ^E given a source sentence F as input.

Statistical machine translation systems are systems that perform translation by creating a probabilistic model for the probability of E given F, P(E j F; ), and nding the target sentence that maximizes this probability:

^E= argmaxEP(E j F; ); (2)

where are the parameters of the model specifying the probability distribution. The parameters are learned from data consisting of aligned sentences in the source and target languages, which are called parallel corpora in technical terminology.Within this framework, there are three major problems that we need to handle appropriately in order to create a good translation system:

Modeling: First, we need to decide what our model P(E j F; ) will look like. What parameters will it have, and how will the parameters specify a probability distribution?

Learning: Next, we need a method to learn appropriate values for parameters from training data.

Search: Finally, we need to solve the problem of nding the most probable sentence (solving \argmax"). This process of searching for the best hypothesis and is often called decoding.1

The remainder of the material here will focus on solving these problems.

번역

Neural Machine Translation and Sequence-to-sequence Models:

A Tutorial

Graham Neubig

Language Technologies Institute, Carnegie Mellon University

1 소개

This tutorial introduces a new and powerful set of techniques variously called “neural machine translation” or “neural sequence-to-sequence models”. These techniques have been used in a number of tasks regarding the handling of human language, and can be a powerful tool in the toolbox of anyone who wants to model sequential data of some sort. The tutorial assumes that the reader knows the basics of math and programming, but does not assume any particular experience with neural networks or natural language processing. It attempts to explain the intuition behind the various methods covered, then delves into them with enough mathematical detail to understand them concretely, and culiminates with a suggestion for an implementation exercise, where readers can test that they understood the content in practice.

소개에서는 인공신경망 기계 번역(neural machine translation)과 인공신경망 시퀀스-투-시퀀스 모델(neural sequence-to-sequence models)이라는 새롭고 훌륭한 기술을 알아본다. 이 기술은 언어를 다루는 부분과 관련된 다양한 작업에 사용되며, 연속으로 정렬된 모델링이 필요한 사람에게 유용한 도구가 될 수 있다. 이 튜토리얼에서 독자는 기본적인 수학과 프로그래밍에 대한 지식이 있는 것을 가정하지만, 신경망 및 자연어처리와 같은 경험은 제외한다. 이 논문에서는 다양한 방법 이면에 담긴 직관을 살펴보고, 이를 구체적으로 이해할 수 있는 수학적인 부분가지 알아본 후, 여러분들이 제대로 이해했는지 연습해본다.

1.1 배경

Before getting into the details, it might be worth describing each of the terms that appear in the title \Neural Machine Translation and Sequence-to-sequence Models". Machine translation is the technology used to translate between human language. Think of the universal translation device showing up in sci- movies to allow you to communicate eortlessly with those that speak a dierent language, or any of the plethora of online translation web sites that you can use to assimilate content that is not in your native language. This ability to remove language barriers, needless to say, has the potential to be very useful, and thus machine translation technology has been researched from shortly after the advent of digital computing. We call the language input to the machine translation system the source language, and call the output language the target language. Thus, machine translation can be described as the task of converting a sequence of words in the source, and converting into a sequence of words in the target. The goal of the machine translation practitioner is to come up with an eective model that allows us to perform this conversion accurately over a broad variety of languages and content.

The second part of the title, sequence-to-sequence models, refers to the broader class of models that include all models that map one sequence to another. This, of course, includes machine translation, but it also covers a broad spectrum of other methods used to handle other tasks as shown in Figure 1. In fact, if we think of a computer program as something that takes in a sequence of input bits, then outputs a sequence of output bits, we could say that every single program is a sequence-to-sequence model expressing some behavior (although of course in many cases this is not the most natural or intuitive way to express things).

자세히 들어가기 전, Neural Machine Translation and Sequence-to-sequence Models: A Tutorial 논문에 등장하는 용어에 대해 정리해보자. 기계 번역이란 언어간 번역을 위해 사용되는 기술이다. 서로 다른 언어를 사용하는 사람들끼리 대화하는 공상 과학 영화에 등장하는 번역기나, 모국어로 작성되지 않은 컨텐츠를 이해할 때 사용되는 온라인 번역 사이트등을 생각해보자. 이렇게 언어 장벽을 무너뜨리는 이 기술은 당연히 많은 잠재력을 가졌으며, 따라서 디지털 컴퓨터이 시작된 후부터 해당 기술이 연구돼 왔다.

기계 번역 시스템에 입력되는 언어를 원시 언어(source language), 출력되는 언어를 대상 언어(target language)라고 한다. 따라서 기계 번역은 원시 언어의 단어 시퀀스를 대상 언어의 단어 시퀀스로 변환하는 과장이다. 기계 번역 시스템의 목표는 이러한 변환이 다양한 컨텐츠와 언어간에 효과정으로 수행하도록 모델을 구성하는 것이다.

두번째 부분인 시퀀스-투-시퀀스 모델은 하나의 시퀀스에서 다른 시퀀스로 맵핑되는 모든 모델을 포괄적으로 가리킨다. 물론, 기계 번역도 포함되나, 그림 1과 같이 다양한 작업을 다루는 광범위한 범위도 포함된다. 사실, 입력 비트 시퀀스를 받아 출력 비트 시퀀스를 내보내는 컴퓨터 프로그램을 생각할 수 있는데, 결국 모든 싱글 프로그램이 이러한 시퀀스-투-시퀀스 모델임을 설명할 수 있다(하지만 많은 경우에 있어 일반적이거나 직관적인 형태는 아니다).

그림 1: 시퀀스-투-시퀀스 모델링 작업의 예

The motivation for using machine translation as a representative of this larger class ofsequence-to-sequence models is many-fold:

1. Machine translation is a widely-recognized and useful instance of sequence-to-sequence models, and allows us to use many intuitive examples demonstrating the diculties encountered when trying to tackle these problems.

2. Machine translation is often one of the main driving tasks behind the development of new models, and thus these models tend to be tailored to MT rst, then applied to other tasks.

3. However, there are also cases where MT has learned from other tasks as well, and introducing these tasks helps explain the techniques used in MT as well.

이렇게 방대한 시퀀스-투-시퀀스 모델을 대표하는 것으로 기계 번역을 사용하는 이유는 여러 가지가 있다.

1. 기계 번역은 널리 알려져있고, 시퀀스-투-시퀀스 모델의 유용한 예이며, 이 같은 문제를 해결할때 마주하는 어려움을 잘 설명한다.

2. 기계 번역 새로운 모델 개발이면에 핵심적인 역할을 종종하며, 먼저 기계 번역에 최적화된 모델을 생성하며, 그 후 다른 영역에 적용된다.

3. 하지만, 다른 영역에서도 배워오는 경우가 있으며, 기계 번역에 사용되는 기술 또한 잘 설명한다.

1.2 목차 구성

This tutorial first starts out with a general mathematical denition of statistical techniques for machine translation in Section 2. The rest of this tutorial will sequentially describe techniques of increasing complexity, leading up to attentional models, which represent the current state-of-the-art in the eld.

First, Sections 3-6 focus on language models, which calculate the probability of a target sequence of interest. These models are not capable of performing translation or sequence transduction, but will provide useful preliminaries to understand sequence-to-sequence models.

• - Section 3 describes n-gram language models, simple models that calculate the probability of words based on their counts in a set of data. It also describes how we evaluate how well these models are doing using measures such as perplexity.

• 
  

• -Section 4 describes log-linear language models, models that instead calculate the probability of the next word based on features of the context. It describes how we can learn the parameters of the models through stochastic gradient descent { calculating derivatives and gradually updating the parameters to increase the likelihood of the observed data.

• -Section 5 introduces the concept of neural networks, which allow us to combine together multiple pieces of information more easily than log-linear models, resulting in increased modeling accuracy. It gives an example of feed-forward neural language models, which calculate the probability of the next word based on a few previous words using neural networks.

• 
  

• -Section 6 introduces recurrent neural networks, a variety of neural networks that have mechanisms to allow them to remember information over multiple time steps. These lead to recurrent neural network language models, which allow for the handling of long-term dependencies that are useful when modeling language or other sequential data.

• 
  

Finally, Sections 7 and 8 describe actual sequence-to-sequence models capable of performing

machine translation or other tasks.

• -Section 7 describes encoder-decoder models, which use a recurrent neural network to encode the target sequence into a vector of numbers, and another network to decode this vector of numbers into an output sentence. It also describes search algorithms to generate output sequences based on this model.

• 
  

• -Section 8 describes attention, a method that allows the model to focus on dierent partsof the input sentence while generating translations. This allows for a more efficient and intuitive method of representing sentences, and is often more eective than its simpler encoder-decoder counterpart.

섹션 2에서는 기계 번역과 관련된 통계적인 기술에 대한 일반적인 수학적 정의부터 시작한다. 그 후 이 분야의 최신 모델들을 알아보며 점차 복잡한 기술을 하나하나씩 배워본다.

먼저, 섹션3-6에서는 대상 언어의 시퀀스 확률을 계산하는 언어 모델을 알아본다. 이 모델은 번역 및 시퀀스 변환은 불가하지만, 시퀀스 모델을 이해할 수 있는 좋은 예제를 제공한다.

• - 섹션 3는 데이터셋내 단어의 개수를 기반으로 해당 단어의 확률을 계산하는 n-gram 언어 모델에 대해 다룬다. 해당 모델이 얼마나 측정치를 잘 사용하고 있는 평가하는 방법(Perplexity)도 알아본다.

• 
  

• - 섹션 4는 문맥의 특성을 기반으로 다음 단어의 확률을 계산하는 모델인 로그-선형 언어 모델(log-linear language model)에 대해 알아본다. 확률적 경사 하강법(stochastic gradient descent, 관찰된 데이터의 우도(likelihood)를 증가시켜 파라미터를 계속 미분해 계산하는 방법)을 통해 모델의 파라미터가 잘 학습됐는지 알아본다.

• 
  

• - 섹션 5는 로그-선형 모델보다 정확도를 높여 좀 더 쉽게 여러 정보를 조합할 수 있는 신경망에 대해 알아본다. 신경망을 이용해 이전 단어를 기반으로 다음 단어의 확률을 계산하는 피드-포워드 신경망 언어 모델의 예제도 살펴본다.

• 
  

• - 섹션 6는 여러 번 정보를 계속 기억하는 메커니즘 형태의 신경망인재귀신경망(recurrent neural networks)에 살펴본다. 이는 결국 재귀 신경망 언어 모델인데, 언어나 순차 데이터를 모델링할 때 유용한 장기 의존성(long-term dependencies)를 다룰 수 있다.

마지막으로, 섹션 7, 8은 기계 번역 및 다른 작업을 실제로 수행하는 시퀀스-투-시퀀스 모델에 대해 살펴본다.

• -섹션 7은 대상 언어를 숫자 벡터로 인코딩하며, 반대로 벡터를 출력 문자으로 디코딩하는 재귀신경망을 사용하는 인코더-디코더(encoder-decode) 모델에 대해 알아본다. 이 모델을 기반으로 출력 문장을 생성하는 검색 알고리즘 또한 알아본다.

• 
  

• - 섹션 8은 번역시 입력 문장의 다른 부분을 살펴보는 모델링이 가능한 방법인 어텐션(attention)에 대해 알아본다. 이는 문장으로 좀 더 효율적이고 직관적인 방법으로 나타낼 수 있게하며, 인코더-디코더와 비교해 간단해 효과적이다.