The batch primary components transformer and auto-plasticity learning linear units architecture: synthetic image generation case

Abstract

Context tokenizing, which is popular in Large Language and Foundation Models (LLM, FM), leads to their excessive dimensionality inflation. Traditional Transformer models strive to reduce intractable excessive dimensionality at the among-token attention level, while we propose additional between-dimensions attention mechanism for dimensionality reduction. A novel Transformer-based architecture is presented, which aims at the individual dimension attention and, by doing so, performs the implicit relevant primary components' feature selection in artificial neural networks (ANN). As an additional mechanism allowing adaptive plasticity learning in ANN, a neuron-specific Learning Rectified Linear Unit layer is proposed for further feature selection via weight decay. The performance of the presented layers is tested on the encoder-decoder architecture applied for the synthetic image generation task for the benchmark MNIST data set.