AMD has unveiled performance benchmarks for low-latency CPU inference on its EPYC processors, leveraging optimizations in its ZenDNN library and zentorch PyTorch plugin. The tests, conducted in September 2026, focused on BERT-Base uncased inference using Hugging Face's implementation. The results highlight significant latency gains, particularly when using BF16 precision, with improvements reaching up to 58.1% compared to ONNX Runtime FP32.

Performance Gains Across Configurations

AMD's testing spanned two configurations: a 96-core EPYC 9654 bare-metal system with NPS4 and an 80-core EPYC 9V74 virtual machine with NPS5. Across 42 configurations, zentorch outperformed ONNX Runtime in 83% of cases, tied in 10%, and trailed in 7% on the bare-metal setup. In the virtual machine, zentorch achieved similar results, leading in 83%, tying in 7%, and trailing in 10% of configurations.

For FP32 precision, the average latency gain was 9.7% on bare metal and 9.1% in the virtual machine. BF16 precision delivered more substantial improvements, with average gains of 47.3% and 46.1%, respectively. The highest reported latency gain occurred in the BF16 configuration, achieving 58.1% at four instances, sequence length 128, and 16 cores.

Key Optimization Techniques

AMD attributed the performance improvements to several key optimizations in ZenDNN and zentorch:

  • Fusion of query, key, and value projections into a single matrix multiplication, improving single-instance performance by approximately 7% for FP32 and 9% for BF16.
  • Tiling scaled-dot-product attention around the processor's cache hierarchy.
  • Low-overhead operator API (LowOHA) with AOCL-DLP kernels to reduce repeated primitive-creation overhead.
  • Pinning inference instances to complete Core Complex Dies (CCDs) while isolating the load generator outside those dies.

These interventions narrowed the multi-instance performance deficit at shorter sequence lengths. For example, at sequence length 32, the deficit was reduced from 21% to 7% below ONNX Runtime.

Implications for AI Workloads

The benchmarks demonstrate AMD's focus on optimizing CPU-based inference for AI workloads such as search ranking, classification, and retrieval-augmented generation reranking. While the results are vendor-reported and based on median-run methodology, they provide valuable insights into the potential of EPYC processors for small-batch AI inference tasks.