Analog-AI Hardware Accelerators for Low-Latency Transformer-Based Language Models (Invited)

Analog Non-Volatile Memory-based accelerators offer highthroughput and energy-efficient Multiply-Accumulate operations for the large Fully-Connected layers that dominate Transformer-based Large Language Models (LLMs). We describe recent chip-demo and architectural efforts, quantify the unique benefits of Fully- (rather than Partially-) Weight-Stationary systems, and discuss factors affecting latency of token-processing and generation.