FP8 Mixed-Precision Training

FP8 Mixed-Precision Training is a framework that uses 8-bit floating point operations to boost training throughput and reduce memory requirements for large Transformer models. It aims to achieve comparable accuracy to BF16 standards and enables efficient post-training quantization with up to 36% throughput gain. This approach is challenging due to potential numerical instability but is being addressed by frameworks that systematically suppress activation outliers.