The batch primary components transformer and auto-plasticity learning linear units architecture: synthetic image generation case
Affiliation
University of BedfordshireIssue Date
2024-01-02Subjects
transformerfeature selection
cosine distance
ANN plasticity
catastrophic forgetting
learning ReLU
Metadata
Show full item recordAbstract
Context tokenizing, which is popular in Large Language and Foundation Models (LLM, FM), leads to their excessive dimensionality inflation. Traditional Transformer models strive to reduce intractable excessive dimensionality at the among-token attention level, while we propose additional between-dimensions attention mechanism for dimensionality reduction. A novel Transformer-based architecture is presented, which aims at the individual dimension attention and, by doing so, performs the implicit relevant primary components' feature selection in artificial neural networks (ANN). As an additional mechanism allowing adaptive plasticity learning in ANN, a neuron-specific Learning Rectified Linear Unit layer is proposed for further feature selection via weight decay. The performance of the presented layers is tested on the encoder-decoder architecture applied for the synthetic image generation task for the benchmark MNIST data set.Citation
Selitskiy S, Inoue C, Schetinin V, Jakaite L (2024) 'The batch primary components transformer and auto-plasticity learning linear units architecture: synthetic image generation case', 2023 Tenth International Conference on Social Networks Analysis, Management and Security (SNAMS) - Abu Dhabi, IEEE.Publisher
IEEEAdditional Links
https://ieeexplore.ieee.org/document/10375471Type
Conference papers, meetings and proceedingsLanguage
enae974a485f413a2113503eed53cd6c53
10.1109/SNAMS60348.2023.10375471