Recurrent Neural Networks and Transformer Models

Front

What are recurrent neural networks commonly used for?

Back

Sequence modeling and transduction problems such as language modeling and machine translation.

Front

What are the two types of recurrent neural networks mentioned?

Back

Long short-term memory (LSTM) and gated recurrent neural networks.

Front

What is a key limitation of recurrent models in training?

Back

The inherently sequential nature precludes parallelization within training examples.

Front

What recent advancements have improved computational efficiency in recurrent models?

Back

Factorization tricks and conditional computation.

Front

What role do attention mechanisms play in sequence modeling?

Back

They allow modeling of dependencies without regard to their distance in the input or output sequences.

Front

What is the main innovation of the Transformer model?

Back

It relies entirely on an attention mechanism to draw global dependencies between input and output, eschewing recurrence.

Front

How does the Transformer model improve parallelization?

Back

By not relying on recurrence, it allows for significantly more parallelization.

Front

What is the training time mentioned for achieving a new state of the art in translation quality with the Transformer?

Back

As little as twelve hours on eight P100 GPUs.

Front

What is the primary advantage of using attention mechanisms over recurrent layers?

Back

Attention mechanisms allow for capturing long-range dependencies more effectively without the vanishing gradient problem.

Front

What is a common application of the Transformer model in natural language processing?

Back

Machine translation, where it has set new benchmarks for translation quality.