Transformer (deep learning)
Algorithm for modelling sequential data
📊 Rating
1 news mentions · 👍 0 likes · 👎 0 dislikes
📌 Topics
- Artificial Intelligence (1)
- Machine Learning (1)
- Neural Networks (1)
🏷️ Keywords
Free Energy Mixer (1) · Transformer architecture (1) · Attention mechanism (1) · Log-sum-exp (1) · Channel-wise selection (1) · arXiv (1) · Deep learning (1)
📖 Key Information
In deep learning, the transformer is an artificial neural network architecture based on the multi-head attention mechanism, in which text is converted to numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished.
Transformers have the advantage of having no recurrent units, therefore requiring less training time than earlier recurrent neural architectures (RNNs) such as long short-term memory (LSTM).
📰 Related News (1)
-
🇺🇸 Free Energy Mixer
arXiv:2602.07160v1 Announce Type: cross Abstract: Standard attention stores keys/values losslessly but reads them via a per-head convex average, bloc...
🔗 Entity Intersection Graph
People and organizations frequently mentioned alongside Transformer (deep learning):
- 🌐 Deep learning (1 shared articles)
- 🌐 Attention (machine learning) (1 shared articles)