Introduction

Welcome to Advanced Transformer Architectures!

In the ever-evolving landscape of artificial intelligence, staying at the forefront of technology means continually adapting and learning about the most cutting-edge developments. One of the most significant breakthroughs in recent years has been the advent of Transformer architectures. Originally designed for tasks in natural language processing (NLP), Transformers have revolutionized how machines understand and generate human language. But their influence doesn't stop there; these architectures have paved the way for advancements in numerous other domains of AI, making an understanding of them not just beneficial but essential for any serious AI practitioner.

Why Advanced Transformer Architectures?

Transformers are behind many of the modern AI wonders, from chatbots that can sustain coherent and context-aware conversations to systems that can write like Shakespeare! As we dive deeper into more complex applications, the basic Transformer model evolves, leading to Advanced Transformer Architectures. These sophisticated models enhance performance, efficiency, and adaptability across various tasks and large datasets. By mastering these advanced models, you are not just keeping up with the AI world; you are actively pushing its boundaries.

What Will You Learn?

This tutorial is meticulously crafted to take you through the intricacies of Advanced Transformer Architectures. Whether it's understanding the mechanics of self-attention mechanisms or dissecting the latest variants like GPT-3, T5, or BERT, this guide has it all. You will learn:

• The foundational concepts of Transformer architectures.
• How these architectures have evolved over time.
• Detailed exploration of specific advanced models and their applications.
• Practical insights into training these models effectively.

Prerequisites

Before diving into this tutorial, a solid understanding of basic machine learning concepts and familiarity with standard neural network architectures is recommended. Knowledge of Python and experience with libraries such as TensorFlow or PyTorch will be crucial for following along with code examples and practical implementations.

Tutorial Overview

Throughout this tutorial, we'll cover several key topics:

1. Introduction to Basic Transformers: Refreshing the core ideas that paved the way for advanced architectures.
2. Evolution of Transformer Architectures: How we moved from basic to advanced, including discussions on efficiency improvements and capability enhancements.
3. Deep Dive into Select Advanced Models: Each model’s architecture, intended use case, strengths, and limitations.
4. Hands-on Implementation: Practical coding sessions implementing some of the advanced transformers using popular AI frameworks.
5. Future Trends and Applications: Where are transformers heading, and how can you stay ahead in the game?

Ready to embark on this advanced journey into the world of AI and Transformers? Let's unleash the full potential of these powerful architectures together!

Understanding the Core Concepts of Transformers

Understanding the Core Concepts of Transformers

Transformers have revolutionized the way we handle sequential data, particularly in the fields of natural language processing (NLP) and beyond. This section delves into the foundational concepts that make transformers a powerful tool in advanced transformer architectures.

1. The Basics of Attention Mechanisms

The attention mechanism is a critical component that allows transformers to weigh the importance of different words in a sentence, regardless of their positional distance from each other. At its core, the attention mechanism can be thought of as a query-key-value (QKV) model:

• Query: Represents the word for which we are trying to compute attention.
• Key: Represents all possible words that interact with the query.
• Value: Contains the actual information of each word that the model processes.

A practical example is when the model processes the sentence "The cat sat on the mat." If "cat" is the query, attention helps determine how much focus or 'attention' "cat" should give to other words like "sat" or "mat" for better understanding and processing.

Here is a simple code snippet showing how attention weights might be computed:

import numpy as np

def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)

query = np.array([1, 0, 0])
key = np.array([[0, 1, 0], [1, 0, 1], [0, 1, 1]])
value = np.array([[1, 2], [2, 3], [3, 4]])

# Compute attention scores
attention_scores = softmax(np.dot(query, key.T))

# Apply scores to values
attention_output = np.dot(attention_scores, value)

2. Self-Attention and Multi-Head Attention

Self-attention allows each word in the input sequence to attend to all other words. This is crucial in understanding context within sentences. For instance, in the sentence "The bank he deposited money into was on the river," self-attention helps distinguish whether "bank" refers to a financial institution or the side of a river.

Multi-head attention extends this concept by having multiple 'heads' of attention. Each head looks at different parts of the sentence, providing diverse perspectives before integrating these insights. This approach enhances the model's ability to focus on various aspects of linguistic data.

3. Positional Encoding and Layer Normalization

Since transformers do not inherently process sequential data as sequences (like RNNs do), positional encoding is added to give some sense of word order. For example:

def positional_encoding(position, d_model):
    angle_rates = 1 / np.power(10000, (2 * (np.arange(d_model)[np.newaxis, :] // 2)) / np.float32(d_model))
    angle_rads = position * angle_rates
    # apply sin to even indices in the array; 2i
    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
    # apply cos to odd indices in the array; 2i+1
    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])
    return angle_rads

pos_encoding = positional_encoding(50, 512)

Layer normalization is another key aspect that helps stabilize deep neural networks by normalizing the inputs across the features instead of batch-wise normalization. This is beneficial in training deeper transformer models effectively.

4. Transformer Blocks: Encoders and Decoders

The transformer model is typically composed of encoder and decoder blocks. The encoder processes the input text by passing it through a series of self-attention and feed-forward layers, preparing a context-rich representation. The decoder, on the other hand, uses this representation along with previous outputs to generate predictions step-by-step.

For instance, in machine translation, the encoder might process an English sentence to understand its context fully, while the decoder would generate a translation in French.

Here’s a high-level overview of how encoder and decoder layers might be structured programmatically:

def encoder_layer(inputs):
    # Self-attention and feed-forward network
    pass

def decoder_layer(encoder_output, prev_output):
    # Masked self-attention, encoder-decoder attention, and feed-forward network
    pass

Transitioning Between Concepts

Understanding each of these components—attention mechanisms, self-attention with multi-head attention, positional encoding with layer normalization, and the roles of encoder and decoder blocks—is essential for mastering advanced transformer architectures. Each element plays a crucial role in enhancing the model's ability to process and understand vast amounts of sequential data effectively.

By grasping these core concepts, developers can better design and implement sophisticated models tailored to specific tasks like translation, summarization, or even generative tasks beyond text processing.

Exploring Advanced Transformer Architectures

Exploring Advanced Transformer Architectures

Transformers have revolutionized the way we handle sequential data in machine learning. These architectures are particularly effective because they simultaneously process data points, allowing for more contextual understanding compared to traditional models like RNNs or LSTMs. In this section, we will explore several advanced transformer architectures that build on the original transformer concept to address specific challenges and use cases.

1. BERT: Bidirectional Encoder Representations from Transformers

BERT (Bidirectional Encoder Representations from Transformers) is a groundbreaking model in the realm of natural language processing (NLP). Developed by Google, BERT's key innovation is its bidirectional training of transformers. Unlike previous models that read text data in one direction, BERT reads the entire sequence of words at once. This allows it to capture a richer understanding of context.

Practical Example:

from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

input_text = "Transformers are powerful models."
encoded_input = tokenizer(input_text, return_tensors='pt')
output = model(**encoded_input)

Best Practices:

• Fine-tune BERT on task-specific data to improve performance.
• Use a pre-trained BERT model as a starting point to leverage its powerful contextual embeddings.

2. GPT: Generative Pre-trained Transformer Series

The GPT series, developed by OpenAI, stands out with its ability to generate human-like text based on the input it receives. Each version of GPT has been an improvement over its predecessor, enhancing everything from the size of the model and its training data to its overall architecture. The most recent, GPT-3, uses 175 billion parameters.

Practical Example:

from transformers import GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

input_text = "Advanced Transformer Architectures like GPT"
inputs = tokenizer.encode(input_text, return_tensors='pt')
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

Best Practices:

• When using GPT for applications like chatbots, fine-tuning on domain-specific data can significantly enhance relevance and coherence.

3. Transformer-XL: Extending Transformer Memory

Transformer-XL was developed to overcome one of the limitations of standard Transformer models: their fixed-length context window. Transformer-XL introduces a recurrence mechanism to transformers, which allows it to remember information from much earlier in the dataset, thereby making it efficient for handling longer texts.

Practical Example:

from transformers import TransfoXLModel, TransfoXLTokenizer

tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLModel.from_pretrained('transfo-xl-wt103')

input_text = "With extended memory, Transformer-XL models can handle long texts effectively."
encoded_input = tokenizer(input_text, return_tensors='pt')
output = model(**encoded_input)

Best Practices:

• Utilize Transformer-XL for tasks involving large texts or documents where long-term dependency is crucial.

4. Reformer: The Efficient Transformer

The Reformer addresses the scalability issues of standard Transformers by introducing two key innovations: locality-sensitive hashing (LSH) attention and reversible residual layers. These modifications reduce the memory usage and computation time significantly, making the Reformer suitable for processing extremely long sequences.

Practical Example:

from reformer_pytorch import ReformerLM

model = ReformerLM(
    num_tokens=20000,
    dim=512,
    depth=12,
    max_seq_len=8192,
    heads=8,
    lsh_dropout=0.1,
)

input_ids = torch.randint(0, 20000, (1, 8192))
output = model(input_ids)

Best Practices:

• The Reformer is ideal for tasks that involve very long sequences where traditional transformer models would be computationally prohibitive.

Conclusion

Advanced Transformer Architectures such as BERT, GPT, Transformer-XL, and Reformer each tackle unique challenges in processing sequential data. By understanding their strengths and applications, you can choose the right model based on your specific needs in tasks ranging from simple text classification to complex sequence generation scenarios.

Applications and Practical Examples

Applications and Practical Examples of Advanced Transformer Architectures

Advanced Transformer architectures have revolutionized various fields beyond their initial foray into natural language processing (NLP). These models, known for their ability to handle sequential data, are now pivotal in image recognition, time-series analysis, and even in the domain of reinforcement learning. This section explores practical applications and examples across these areas, demonstrating the versatility and power of Transformer models.

1. Natural Language Processing Applications

Transformers were born out of the need to improve NLP tasks, and they have certainly delivered. A prime example is BERT (Bidirectional Encoder Representations from Transformers) which radically improved benchmarks on a variety of NLP tasks such as sentiment analysis, named entity recognition, and question answering.

Example: Sentiment Analysis with BERT

Consider a scenario where a company wants to analyze customer reviews automatically to determine sentiments expressed about their products. Using BERT, one can fine-tune the model with review data to classify sentiments as positive, negative, or neutral. Here’s a simplified code snippet using Hugging Face’s transformers library:

from transformers import BertTokenizer, BertForSequenceClassification
from torch import nn, optim

# Load pre-trained model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Example review
review = "The product was great!"
inputs = tokenizer(review, return_tensors="pt")

# Predict sentiment
with torch.no_grad():
    logits = model(**inputs).logits
predicted_class = logits.argmax().item()
print("Predicted sentiment class:", predicted_class)

This snippet loads a pre-trained BERT model, tokenizes an example review, and predicts the sentiment class.

2. Image Recognition and Vision Transformers (ViTs)

Vision Transformers (ViTs) apply the self-attention mechanism of Transformers directly to sequences of image patches, treating them similarly to tokens (words) in NLP. This allows ViTs to understand contextual relationships in image data.

Example: Image Classification with ViT

Google’s ViT model, for instance, achieves state-of-the-art results on image classification tasks. Below is an example using ViT for classifying images in the CIFAR-10 dataset:

from transformers import ViTFeatureExtractor, ViTForImageClassification
from PIL import Image
import requests

# Load pre-trained ViT model
model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224-in21k')
feature_extractor = ViTFeatureExtractor.from_pretrained('google/vit-base-patch16-224-in21k')

# Load an image
url = 'http://example.com/image.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# Prepare image for the model
inputs = feature_extractor(images=image, return_tensors="pt")

# Classify image
outputs = model(**inputs)
predictions = outputs.logits.argmax(-1)
print("Predicted class:", predictions.item())

This code demonstrates how to use a pre-trained Vision Transformer for image classification, highlighting its ability to handle complex visual data.

3. Time-Series Analysis Using Transformers

Transformers can also be adapted for time-series forecasting, which is crucial for financial markets, weather forecasting, and more. The key adaptation here involves treating time steps as sequential tokens.

Example: Stock Price Prediction

One can use a Transformer architecture to predict future stock prices based on historical data. This involves training the model on sequences of past price data and using it to forecast future prices.

# Pseudocode for a Transformer-based time-series model
model = TimeSeriesTransformer(num_layers=4, d_model=128, num_heads=8)
train(model, historical_stock_data)
future_prices = predict(model, recent_stock_data)

This simplified pseudocode outlines how one might set up a Transformer model tailored for time-series data.

4. Transformer Models in Reinforcement Learning

Lastly, Transformers are making strides in reinforcement learning (RL). They can be used to process sequences of states and actions in an environment, helping agents learn optimal policies.

Example: Game Playing with Transformers

In environments like chess or Go, where understanding the sequence of moves is crucial, Transformers can significantly enhance an agent’s performance.

# Pseudocode for a Transformer in RL
model = TransformerRLModel()
train(model, game_data)
policy = model.derive_policy()

This pseudocode implies how a Transformer model could be trained on game data to derive strategies or policies.

In conclusion, the adaptability of advanced Transformer architectures across such diverse fields showcases their robustness and efficiency. Whether it's understanding human languages, recognizing objects in images, forecasting future events, or strategizing in complex games, Transformers are at the forefront of AI research and application.

Implementation Techniques and Code Samples

Implementation Techniques and Code Samples for Advanced Transformer Architectures

Setting up the Development Environment

Before diving into Transformer model implementation, it's essential to set up a robust development environment. Whether you choose PyTorch or TensorFlow, both frameworks support the implementation of advanced transformer architectures due to their flexibility and comprehensive libraries.

Prerequisites:

1. Python: Install Python (version 3.6 or newer). Anaconda is recommended for managing Python and dependencies.
2. PyTorch/TensorFlow: Install the latest version of PyTorch or TensorFlow. Both frameworks provide extensive support for transformers.
3. IDE: Use an IDE or a code editor such as VSCode, PyCharm, or Jupyter Notebook for writing and testing your code.
4. Additional Libraries: Install libraries like transformers by Hugging Face, which provides pre-trained models and utilities for working with them.

pip install torch torchvision torchaudio  # For PyTorch
pip install tensorflow  # For TensorFlow
pip install transformers

Implementing a Basic Transformer Model in PyTorch/TensorFlow

Let's start by implementing a basic Transformer model using both PyTorch and TensorFlow. This will give you a practical understanding of how transformers are structured and executed.

PyTorch Example:

import torch
from torch.nn import Transformer
import torch.nn.functional as F

# Model Parameters
input_size = 512  # Input size (vocab size)
model_dim = 512   # Dimension of the model
num_heads = 8     # Number of heads in the multi-head attention models
num_encoder_layers = 6  # Number of sub-encoder-layers in the encoder
num_decoder_layers = 6  # Number of sub-decoder-layers in the decoder

# Initialize model
transformer_model = Transformer(d_model=model_dim, nhead=num_heads,
                                num_encoder_layers=num_encoder_layers,
                                num_decoder_layers=num_decoder_layers)

# Dummy input (batch size, sequence length, model dimension)
src = torch.rand((10, 32, model_dim))
tgt = torch.rand((10, 32, model_dim))
out = transformer_model(src, tgt)

print(out.shape)  # Output shape

TensorFlow Example:

import tensorflow as tf
from tensorflow.keras.layers import TransformerLayer

# Model Parameters
model_dim = 512  # Dimension of the model

# Initialize model
transformer_layer = TransformerLayer(num_heads=8, key_dim=model_dim//8, dropout=0.1)

# Dummy input (batch size, sequence length, feature size)
input_tensor = tf.random.normal([10, 32, model_dim])
target_tensor = tf.random.normal([10, 32, model_dim])
output_tensor = transformer_layer([input_tensor, target_tensor])

print(output_tensor.shape)  # Output shape

Advanced Techniques: Fine-Tuning BERT for Specific Tasks

Fine-tuning BERT (Bidirectional Encoder Representations from Transformers) involves customizing this powerful pre-trained model to perform specific tasks like sentiment analysis or question-answering. Here's how you can do it using the Hugging Face transformers library:

Example Code:

from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# Sample text for classification
texts = ["This is an amazing product!", "I did not enjoy the movie."]
encoded_input = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')
labels = torch.tensor([1,0])  # 1 for positive sentiment, 0 for negative

# Fine-tuning the model
training_args = TrainingArguments(output_dir='./results', num_train_epochs=3, per_device_train_batch_size=4)
trainer = Trainer(model=model, args=training_args, train_dataset=encoded_input['input_ids'], eval_dataset=labels)
trainer.train()

Debugging and Optimization Tips for Transformer Models

Debugging and optimizing transformer models can be challenging due to their complexity. Here are some tips to make this process smoother:

1. Batch Size: Start with a smaller batch size to ensure that the model fits into your GPU memory without causing out-of-memory errors.
2. Gradient Checking: Use gradient checking during early stages of development to ensure that the implementation is correct.
3. Monitor Overfitting: Keep an eye on the training loss and validation loss. Implement early stopping or increase dropout rates if you notice overfitting.
4. Use Mixed Precision Training: This can help in speeding up training times and reducing memory usage significantly.

By following these guidelines and exploring both theoretical and practical aspects of transformer models, you can enhance your understanding and application of these powerful architectures in various AI tasks.

Best Practices, Challenges, and Common Pitfalls

Best Practices, Challenges, and Common Pitfalls in Advanced Transformer Architectures

Transformers have revolutionized the field of deep learning, especially in tasks like natural language processing, image recognition, and beyond. As we delve deeper into the complex landscape of these models, it's crucial to understand not just their capabilities but also the challenges and pitfalls they present. This guide will navigate through some of the best practices and common challenges when dealing with advanced transformer architectures.

1. Data Preparation and Preprocessing for Transformer Models

Data is the foundation on which transformer models are built. Proper data preparation and preprocessing are critical for achieving optimal performance.

Best Practices:

• Tokenization: Convert text into tokens since transformers process sequential data. Tools like Hugging Face’s transformers library provide robust tokenizers for this purpose.

  from transformers import AutoTokenizer
  tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
  tokens = tokenizer.encode("Example sentence for encoding", add_special_tokens=True)
  

• Handling Various Data Lengths: Use padding or truncation to handle texts of varying lengths. Ensure that the model input has a uniform size.
• Data Augmentation: Although less common in NLP, techniques like back-translation or synonym replacement can enrich your dataset and improve model robustness.

Common Pitfalls:

• Neglecting to clean and preprocess data adequately can lead to suboptimal model training. Issues like missing data, inconsistent formats, or noise can significantly impact performance.

2. Handling Overfitting in Large Transformer Models

Overfitting is particularly challenging in large transformer models due to their vast number of parameters.

Best Practices:

• Regularization Techniques: Implement dropout or weight decay. These methods help prevent the model from relying too heavily on any single aspect of the data.

  from transformers import BertConfig, BertModel
  configuration = BertConfig(hidden_dropout_prob=0.2, attention_probs_dropout_prob=0.2)
  model = BertModel(configuration)
  

• Data Augmentation: As mentioned, increasing the diversity of your training data can help mitigate overfitting.
• Early Stopping: Monitor validation loss during training and stop when it begins to deteriorate, even if training loss continues to improve.

Common Pitfalls:

• Overcomplicating the model architecture unnecessarily increases the risk of overfitting, especially with insufficient data.

3. Efficiency Considerations: Training Time and Model Size

Efficiency is a key concern with advanced transformer architectures given their computational demands.

Best Practices:

• Model Pruning: Reducing the number of trainable parameters in a model can lead to significant improvements in efficiency without a substantial loss in performance.
• Knowledge Distillation: Train a smaller model (the "student") to replicate the behavior of a larger, pre-trained model (the "teacher").

  # Pseudo code for knowledge distillation
  teacher_outputs = teacher_model(input_ids)
  student_loss = loss_fn(student_outputs, teacher_outputs)
  student_loss.backward()
  update(student_model.parameters())
  

• Quantization: Implementing lower precision arithmetic during inference can reduce model size and speed up computation.

Common Pitfalls:

• Sacrificing too much accuracy for efficiency without thorough testing can lead to models that are fast but underperforming in practical applications.

4. Future Directions and Emerging Trends in Transformer Technology

The landscape of transformer technology is continually evolving, with several exciting trends on the horizon.

Emerging Trends:

• Attention Variants: Techniques like sparse, global, and local attention are being explored to reduce complexity and improve performance.
• Cross-modal Applications: Transformers are increasingly being applied across different types of data, such as combining text and image data for richer understanding.
• Energy-Efficient Architectures: There is a growing emphasis on creating transformers that require less energy, aligning with the broader goals of sustainable AI.

Challenges:

• Keeping up with rapidly changing technology and continuously integrating new findings into existing models can be daunting but is necessary for staying at the cutting edge.

In conclusion, while advanced transformer architectures offer remarkable capabilities, they come with their own set of challenges and pitfalls. By adhering to best practices in data preparation, model tuning, efficiency optimization, and staying informed on future trends, one can effectively leverage the power of transformers in various AI applications.

Conclusion

Conclusion

As we conclude our journey through the intricate world of advanced transformer architectures, let's recap the essential insights and knowledge you've gathered from this tutorial. Starting with the core concepts of Transformers, you have built a foundational understanding crucial for grasping how these powerful models operate. From self-attention mechanisms to positional encoding, each component plays a vital role in the functionality of Transformers.

We then ventured into exploring advanced transformer architectures, where you were introduced to innovations like BERT, GPT-3, and T5, each representing significant milestones in the AI field. By dissecting these models, you gained a deeper appreciation of how transformers continue to evolve and address diverse challenges across various domains.

Through applications and practical examples, you saw firsthand the versatility of transformers in tasks such as natural language processing, image recognition, and beyond. The implementation techniques and code samples provided you with hands-on experience, equipping you with the tools to build and tweak transformer models yourself.

The section on best practices, challenges, and common pitfalls prepared you to navigate the complexities of working with these architectures effectively, highlighting the importance of a meticulous approach to training, fine-tuning, and deploying transformer models.

Moving forward, I encourage you to deepen your understanding by engaging with community forums, contributing to open-source projects, and staying updated with the latest research papers. Websites like arXiv.org and conferences such as NeurIPS can provide invaluable resources for staying at the cutting edge of transformer technology.

Lastly, remember that the field of AI is rapidly evolving, and continuous learning is key. Apply what you've learned here in real-world applications and experiments. Challenge yourself to solve new problems and innovate using the advanced concepts discussed. Your journey into the transformative world of transformers is just beginning, and the possibilities are limitless. Embrace them with curiosity and enthusiasm!

Code Examples

```json
[
    {
        "title": "Implementing a Basic Transformer Block using PyTorch",
        "description": "This example demonstrates how to implement a basic transformer block using PyTorch. It includes the self-attention mechanism and position-wise feed-forward network, which are core components of the transformer architecture.",
        "language": "Python",
        "code": `
import torch
import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, embed_size, heads, dropout, forward_expansion):
        super(TransformerBlock, self).__init__()
        self.attention = nn.MultiheadAttention(embed_dim=embed_size, num_heads=heads, dropout=dropout)
        self.norm1 = nn.LayerNorm(embed_size)
        self.norm2 = nn.LayerNorm(embed_size)

        self.feed_forward = nn.Sequential(
            nn.Linear(embed_size, forward_expansion * embed_size),
            nn.ReLU(),
            nn.Linear(forward_expansion * embed_size, embed_size)
        )
        self.dropout = nn.Dropout(dropout)

    def forward(self, value, key, query, mask):
        attention = self.attention(query, key, value, attn_mask=mask)[0]
        x = self.dropout(self.norm1(attention + query))
        forward = self.feed_forward(x)
        out = self.dropout(self.norm2(forward + x))
        return out

# Example usage
if __name__ == "__main__":
    embed_size = 256
    heads = 8
    dropout = 0.1
    forward_expansion = 4
    batch_size = 64
    seq_length = 50

    transformer_block = TransformerBlock(embed_size, heads, dropout, forward_expansion)
    src = torch.rand((seq_length, batch_size, embed_size))
    out = transformer_block(src, src, src, None)
    print(out.shape)  # Expected Shape: [seq_length, batch_size, embed_size]
`,
        "explanation": "To run this code, ensure you have PyTorch installed. You can install it via pip (pip install torch). The expected output is the shape of the tensor output by the transformer block, which should match the input dimensions. This shows how data flows through a transformer block."
    },
    {
        "title": "Building a Vision Transformer (ViT) Model for Image Classification",
        "description": "This code example shows how to build a Vision Transformer (ViT) model from scratch using TensorFlow and Keras. The ViT model applies transformer techniques to image classification tasks.",
        "language": "Python",
        "code": `
import tensorflow as tf
from tensorflow.keras.layers import LayerNormalization, Dense, Dropout
from tensorflow.keras.models import Sequential

class Patches(tf.keras.layers.Layer):
    def __init__(self, patch_size):
        super(Patches, self).__init__()
        self.patch_size = patch_size

    def call(self, images):
        batch_size = tf.shape(images)[0]
        patches = tf.image.extract_patches(
            images=images,
            sizes=[1, self.patch_size, self.patch_size, 1],
            strides=[1, self.patch_size, self.patch_size, 1],
            rates=[1, 1, 1, 1],
            padding='VALID',
        )
        patch_dims = patches.shape[-1]
        patches = tf.reshape(patches, [batch_size, -1, patch_dims])
        return patches

def create_vit_classifier():
    input_shape = (32, 32, 3)  # Example input shape for CIFAR-10
    num_patches = (input_shape[0] // 8) * (input_shape[1] // 8)
    projection_dim = 64

    inputs = tf.keras.Input(shape=input_shape)
    patches = Patches(8)(inputs)
    encoded_patches = Dense(projection_dim)(patches)

    for _ in range(6):  # Stack of Transformer blocks
        x1 = LayerNormalization(epsilon=1e-6)(encoded_patches)
        attention_output = tf.keras.layers.MultiHeadAttention(
            num_heads=8,
            key_dim=projection_dim,
            dropout=0.1
        )(x1, x1)
        x2 = LayerNormalization(epsilon=1e-6)(attention_output + encoded_patches)
        x3 = Dense(projection_dim * 2, activation='relu')(x2)
        encoded_patches = Dense(projection_dim)(x3)

    representation = LayerNormalization(epsilon=1e-6)(encoded_patches)
    representation = tf.reduce_mean(representation, axis=1)
    logits = Dense(10)(representation)

    model = tf.keras.Model(inputs=inputs, outputs=logits)
    return model

# Example usage
if __name__ == "__main__":
    model = create_vit_classifier()
    model.summary()
`,
        "explanation": "To run this example, TensorFlow must be installed (`pip install tensorflow`). This script builds a Vision Transformer and outputs its summary. The model processes image patches through multiple transformer layers and outputs class logits for image classification."
    }
]
```