UH Biocomputation Group - machine learninhttp://biocomputation.herts.ac.uk/2021-12-08T11:05:28+00:00Attention Is All You Need2021-12-08T11:05:28+00:002021-12-08T11:05:28+00:00Harpreet Singhtag:biocomputation.herts.ac.uk,2021-12-08:/2021/12/08/attention-is-all-you-need.html<p class="first last">Harpreet Singh's Journal Club session where he will talk about a paper "Attention Is All You Need"</p>
<p>This week on Journal Club session Harpreet Singh has planned an introductory presentation on the research developments in Natural Language processing NLP.
The main neural architecture he will discuss is the Transformers network, introduced in the paper "Attention Is All You Need", which forms the backbone of many states of the art NLP models.</p>
<hr class="docutils" />
<p>The dominant sequence transduction models are based on complex
recurrent or convolutional neural networks that include an encoder and
a decoder. The best performing models also connect the encoder and
decoder through an attention mechanism. We propose a new simple
network architecture, the Transformer, based solely on attention
mechanisms, dispensing with recurrence and convolutions entirely.
Experiments on two machine translation tasks show these models to be
superior in quality while being more parallelizable and requiring
significantly less time to train. Our model achieves 28.4 BLEU on the
WMT 2014 Englishto-German translation task, improving over the
existing best results, including ensembles, by over 2 BLEU. On the WMT
2014 English-to-French translation task, our model establishes a new
single-model state-of-the-art BLEU score of 41.8 after training for
3.5 days on eight GPUs, a small fraction of the training costs of the
best models from the literature. We show that the Transformer
generalizes well to other tasks by applying it successfully to English
constituency parsing both with large and limited training data.</p>
<div class="line-block">
<div class="line"><br /></div>
</div>
<p>Papers:</p>
<ul class="simple">
<li>A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, I. Polosukhin, <a class="reference external" href="https://arxiv.org/abs/1706.03762">"Attention Is All You Need"</a>, 2017, arXiv:1706.03762 [cs],</li>
</ul>
<p><strong>Date:</strong> 2021/12/10 <br />
<strong>Time:</strong> 14:00 <br />
<strong>Location</strong>: online</p>