Electra
The ELECTRA model is a new pre-training approach for text encoders.
December 29th, 2024
About Electra
The ELECTRA model was proposed in the paper ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. ELECTRA is a new pretraining approach which trains two transformer models: the generator and the discriminator. The generator's role is to replace tokens in a sequence, and is therefore trained as a masked language model.
Key Features
- Pretraining approach for text encoders.
- Trains generator and discriminator models.
- Generator replaces tokens in a sequence.
- Trained as a masked language model.
Use Cases
- Text generation.
- Language understanding.
- Text classification.
Loading reviews...
