#VisionTransformersViTs

[govindhtech]

Vision Transformers: NLP-Inspired Image Analysis Revolution

Vision Transformers are revolutionising Edge video analytics.

Vision Transformer (ViT) AI models conduct semantic image segmentation, object detection, and image categorisation using transformer design. Transformer architecture has dominated Natural Language Processing (NLP) since its birth, especially in models like ChatGPT and other chatbots' GPT design.

Transformer models are now the industry standard in natural language processing (NLP), although their earliest CV applications were limited and often included combining or replacing convolutional neural networks. However, ViTs show how a pure transformer applied directly to picture patch sequences can perform well on image classification tasks.

How Vision Transformers Work

ViTs process images differently than CNNs. Instead of using convolutional layers and a structured grid of pixels, a ViT model presents an input image as fixed-size image patches. Text transformers employ word embeddings in a similar sequence to patches.

The general architecture includes these steps:

Cutting a picture into predetermined blocks.

The picture patches are flattening.

Creating lower-dimensional linear embeddings from flattened patches.

We incorporate positional embeddings. Learning the relative positioning of picture patches allows the model to reconstruct the visual structure.

delivering transformer encoders with these embeddings.

For image classification, the last transformer block output is passed to a classification head, often a fully linked layer. This classification head may use one hidden layer for pre-training and one linear layer for fine-tuning.

Key mechanism: self-attention

The ViT design relies on the NLP-inspired self-attention mechanism. This approach is necessary for contextual and long-range dependencies in input data. It allows the ViT model to prioritise input data regions based on task relevance.

Self-attention computes a weighted sum of incoming data based on feature similarity. This weighting helps the model capture more meaningful representations by weighting relevant information. It evaluates pairwise interactions between entities (image patches) to establish data hierarchy and alignment. Visual networks become stronger during this process.

Transformer encoders process patches using transformer blocks. Each block usually has a feed-forward layer (MLP) and a multi-head self-attention layer. Multi-head attention lets the model focus on multiple input sequence segments by extending self-attention. Before each block, Layer Normalisation is often applied, and residual connections are added thereafter to improve training.

ViTs can incorporate global visual information to the self-attention layer. This differs from CNNs, which focus on local connectivity and develop global knowledge hierarchically. ViTs can semantically correlate visual information using this global method.

Attention Maps:

Attention maps show the attention weights between each patch and the others. These maps indicate how crucial picture features are to model representations. Visualising these maps, sometimes as heatmaps, helps identify critical image locations for a particular activity.

Vision Transformers vs. CNNs

ViTs are sometimes compared to CNNs, which have long been the SOTA for computer vision applications like image categorisation.

Processors and architecture

Convolutional layers and pooling procedures help CNNs extract localised features and build hierarchical global knowledge. They group photos in grids. In contrast, ViTs process images as patches via self-attention mechanisms, eliminating convolutions.

Attention/connection:

CNNs require hierarchical generalisation and localisation. ViTs use self-attention, a global method that considers all picture data. Long-term dependencies are now better represented by ViTs.

Inductive bias:

ViTs can reduce inductive bias compared to CNNs. CNNs naturally use locality and translation invariance. This must be learnt from data by ViTs.

Efficient computation:

ViT models may be more computationally efficient than CNNs and require less pre-training. They achieve equivalent or greater accuracy with four times fewer computational resources as SOTA CNNs. The global self-attention technique also works with GPUs and other parallel processing architectures.

Dependence on data

ViTs use enormous amounts of data for large-scale training to achieve great performance due to their lower inductive bias. Train ViTs on more than 14 million pictures to outperform CNNs. They may nonetheless perform poorly than comparable-sized CNN alternatives like ResNet when trained from scratch on mid-sized datasets like ImageNet. Training on smaller datasets often requires model regularisation or data augmentation.

Optimisation:

CNNs are easier to optimise than ViTs.

History, Performance

Modern computer vision breakthroughs were made possible by ViTs' high accuracy and efficiency. Their performance is competitive across applications. In ImageNet-1K, COCO detection, and ADE20K semantic segmentation benchmarks, the ViT CSWin Transformer outperformed older SOTA approaches like the Swin Transformer.

In an ICLR 2021 publication, the Google Research Brain Team revealed the Vision Transformer model architecture. Since the 2017 NLP transformer design proposal, vision transformer developments have led to its creation. DETR, iGPT, the original ViT, job applications (2020), and ViT versions like DeiT, PVT, TNT, Swin, and CSWin that have arisen since 2021 are major steps.

Research teams often post pre-trained ViT models and fine-tuning code on GitHub. ImageNet and ImageNet-21k are often used to pre-train these models.

Applications and use cases

Vision transformers are used in many computer vision applications. These include:

Action recognition, segmentation, object detection, and image categorisation are image recognition.

Generative modelling and multi-model activities include visual grounding, question responding, and reasoning.

Video processing includes activity detection and predictions.

Image enhancement comprises colourization and super-resolution.

3D Analysis: Point cloud segmentation and classification.

Healthcare (diagnosing medical photos), smart cities, manufacturing, crucial infrastructure, retail (object identification), and picture captioning for the blind and visually impaired are examples. CrossViT is a good medical imaging cross-attention vision transformer for picture classification.

ViTs could be a versatile learning method that works with various data. Their promise resides in recognising hidden rules and contextual linkages, like transformers revolutionised NLP.

Challenges

ViTs have many challenges despite their potential:

Architectural Design:

Focus on ViT architecture excellence.

Data Dependence, Generalisation:

They use huge datasets for training because they have smaller inductive biases than CNNs. Data quality substantially affects generalisation and robustness.

Robustness:

Several studies show that picture classification can preserve privacy and resist attacks, although robustness is difficult to generalise.

Interpretability:

Why transformers excel visually is still unclear.

Efficiency:

Transformer models that work on low-resource devices are tough to develop.

Performance on Specific Tasks:

Using the pure ViT backbone for object detection has not always outperformed CNN.

Tech skills and tools:

Since ViTs are new, integrating them may require more technical skill than with more established CNNs. Libraries and tools supporting it are also evolving.

Tune Hyperparameters:

Architectural and hyperparameter adjustments are being studied to compare CNN accuracy and efficiency.

Since ViTs are new, research is being done to fully understand how they work and how to use them.