LightningRMS: a high-throughput mixed-precision RMSNorm accelerator for transformer inference

Official URL: https://dx.doi.org/10.1145/3787109.3815212

Root mean square layer normalization (RMSNorm) has become the dominant normalization operation in modern transformer architectures such as LLaMA, Mistral, and Gemma, yet dedicated hardware accelerators for this operation remain scarce. Although RMSNorm is computationally simple, its low arithmetic intensity makes it memory-bound, and existing implementations either rely on HLS-generated pipelines with limited throughput or target algorithmic approximation without complete system integration. We present LightningRMS, the first pure RTL RMSNorm accelerator with complete AXI4 system integration. The design employs a dual finite state machine (FSM) ping-pong architecture that decouples input accumulation from output normalization, enabling back-to-back vector processing without pipeline stalls. A custom pipelined floating-point library replaces the costly square root and reciprocal with a fast inverse square root approximation, while the mixed-precision datapath supports configurable INT32/INT8/BF16 input, FP32 internal accumulation, and BF16/INT8 output. Implemented on a Kintex UltraScale+ FPGA at 385 MHz, LightningRMS delivers a sustained throughput of 13.8 elements per cycle, achieving 2.7 × higher throughput than the leading FPGA-based RMSNorm kernel and 67 × over a Jetson Orin Nano GPU baseline, while occupying only 13.95% of LUTs and 5.59% of DSP slices with an energy cost of 0.48 nJ per element. Hardware outputs maintain ≤ 1 ULP BF16 agreement with an FP32 software reference across all tested dimensions up to N = 12288, confirming suitability for transformer inference.