Transformers

Edit

Natural Language Processing • Language Models • 93 methods

Transformers are a type of neural network architecture that have several properties that make them effective for modeling data with long-range dependencies. They generally feature a combination of multi-headed attention mechanisms, residual connections, layer normalization, feedforward connections, and positional embeddings.

Subcategories

1 Autoencoding Transformers

2 Autoregressive Transformers

3 Rendezvous

Methods

Add a Method

Method	Year	Papers
Transformer Attention Is All You Need	2017	9608
Focus Focus Your Attention (with Adaptive IIR Filters)	2023	6122
BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	2018	5116
GPT-3 Language Models are Few-Shot Learners	2020	1379
RoBERTa RoBERTa: A Robustly Optimized BERT Pretraining Approach	2019	757
GPT Improving Language Understanding by Generative Pre-Training	2018	691
T5 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer	2019	561
GPT-2 Language Models are Unsupervised Multitask Learners	2019	554
BART BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	2019	477
RAG Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks	2020	175
XLNet XLNet: Generalized Autoregressive Pretraining for Language Understanding	2019	156
ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations	2019	155
DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	2019	128
Electric Pre-Training Transformers as Energy-Based Cloze Models	2020	120
PaLM PaLM: Scaling Language Modeling with Pathways	2022	113
ELECTRA ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators	2020	108
mT5 mT5: A massively multilingual pre-trained text-to-text transformer	2020	83
Longformer Longformer: The Long-Document Transformer	2020	79
Performer Rethinking Attention with Performers	2020	73
DeBERTa DeBERTa: Decoding-enhanced BERT with Disentangled Attention	2020	66
Transformer-XL Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	2019	61
mBART Multilingual Denoising Pre-training for Neural Machine Translation	2020	59
XLM Cross-lingual Language Model Pretraining	2019	55
CodeBERT CodeBERT: A Pre-Trained Model for Programming and Natural Languages	2020	50
ERNIE ERNIE: Enhanced Representation through Knowledge Integration	2019	48
PEGASUS PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization	2019	45
Sparse Transformer Generating Long Sequences with Sparse Transformers	2019	36
ViLBERT ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	2019	30
ETC ETC: Encoding Long and Structured Inputs in Transformers	2020	29
GPT-Neo	2000	29
CTRL CTRL: A Conditional Transformer Language Model for Controllable Generation	2019	22
CodeT5 CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation	2021	19
Universal Transformer Universal Transformers	2018	18
Switch Transformer Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	2021	16
Reformer Reformer: The Efficient Transformer	2020	15
BigBird Big Bird: Transformers for Longer Sequences	2020	15
Linformer Linformer: Self-Attention with Linear Complexity	2020	13
Levenshtein Transformer Levenshtein Transformer	2019	12
Primer Primer: Searching for Efficient Transformers for Language Modeling	2021	10
ProphetNet ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training	2020	10
TNT Transformer in Transformer	2021	9
E-Branchformer E-Branchformer: Branchformer with Enhanced merging for speech recognition	2022	8
PLATO-2 PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning	2020	6
MATE MATE: Multi-view Attention for Table Transformer Efficiency	2021	6
Bort Optimal Subarchitecture Extraction For BERT	2020	5
Charformer Charformer: Fast Character Transformers via Gradient-based Subword Tokenization	2021	5
EGT Global Self-Attention as a Replacement for Graph Convolution	2021	5
MobileBERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	5
Subformer Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers	2021	3
Adaptive Span Transformer Adaptive Attention Span in Transformers	2019	3
GANformer Generative Adversarial Transformers	2021	3
VideoBERT VideoBERT: A Joint Model for Video and Language Representation Learning	2019	3
TAPEX TAPEX: Table Pre-training via Learning a Neural SQL Executor	2021	3
Compressive Transformer Compressive Transformers for Long-Range Sequence Modelling	2019	3
ConvBERT ConvBERT: Improving BERT with Span-based Dynamic Convolution	2020	3
Routing Transformer Efficient Content-Based Sparse Attention with Routing Transformers	2020	3
DeeBERT DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference	2020	3
Feedback Transformer Addressing Some Limitations of Transformers with Feedback Memory	2020	2
CuBERT Learning and Evaluating Contextual Embedding of Source Code	2019	2
Nyströmformer Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention	2021	2
I-BERT I-BERT: Integer-only BERT Quantization	2021	2
DynaBERT DynaBERT: Dynamic BERT with Adaptive Width and Depth	2020	2
AutoTinyBERT AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	2021	2
TernaryBERT TernaryBERT: Distillation-aware Ultra-low Bit BERT	2020	2
Fastformer Fastformer: Additive Attention Can Be All You Need	2021	2
Parallel Layers PaLM: Scaling Language Modeling with Pathways	2022	2
ESACL Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization	2021	2
DeLighT DeLighT: Deep and Light-weight Transformer	2020	1
PAR Transformer Pay Attention when Required	2020	1
PermuteFormer PermuteFormer: Efficient Relative Position Encoding for Long Sequences	2021	1
NormFormer NormFormer: Improved Transformer Pretraining with Extra Normalization	2021	1
BP-Transformer BP-Transformer: Modelling Long-Range Context via Binary Partitioning	2019	1
IB-BERT MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices	2020	1
MacBERT Revisiting Pre-Trained Models for Chinese Natural Language Processing	2020	1
MHMA Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos	2021	1
RealFormer RealFormer: Transformer Likes Residual Attention	2020	1
Sinkhorn Transformer Sparse Sinkhorn Attention	2020	1
SongNet SongNet: Rigid Formats Controlled Text Generation	2020	1
Funnel Transformer Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing	2020	1
T-D Generating Wikipedia by Summarizing Long Sequences	2018	1
SC-GPT Few-shot Natural Language Generation for Task-Oriented Dialog	2020	1
SMITH Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching	2020	1
Chinese Pre-trained Unbalanced Transformer CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation	2021	1
ClipBERT Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling	2021	1
BinaryBERT BinaryBERT: Pushing the Limit of BERT Quantization	2020	1
Adaptively Sparse Transformer Adaptively Sparse Transformers	2019	1
SqueezeBERT SqueezeBERT: What can computer vision teach NLP about efficient neural networks?	2020	1
Sandwich Transformer Improving Transformer Models by Reordering their Sublayers	2019	1

Transformers Edit

Methods Add a Method

Transformers

Edit

Methods

Add a Method