Lila Sciences and Anyscale have successfully trained the GLM-5.3-Flash model on sequences of up to 1 million tokens, a significant milestone in long-context AI training. The collaboration utilized SkyRL, an open-source training stack, running on 64 NVIDIA B200 GPUs. This achievement represents a 64-fold increase in sequence length compared to the initial training capacity and addresses the growing need for AI models capable of handling extensive scientific datasets.

Technical Innovations and Optimizations

The GLM-5.3-Flash model incorporates a hybrid architecture with 34 layers of Kimi Delta Attention (linear attention) and 11 layers of DeepSeek Sparse Attention (DSA). To achieve the 1M-token context, the team implemented several critical optimizations:

  • Sparse Attention Improvements: Adjustments to tensor shapes enabled sparse kernels, halving memory usage and ensuring compatibility with DSA layers.
  • Memory Management: Techniques such as activation recomputation, CPU offloading, and context parallelism were selectively applied to balance memory and throughput. For example, offloading reduced peak memory at 256k tokens from 142 GiB to 98 GiB, albeit with a 21% increase in step time.
  • Mixture-of-Experts (MoE) Adjustments: Real-data routing revealed uneven token distribution across experts, with the busiest GPU handling up to four times the average token load. Chunked processing of MoE layers resolved memory bottlenecks at 512k tokens.
  • Host Memory Planning: At 1M tokens, offloaded layer inputs required approximately 180 GB of CPU memory per GPU, necessitating pinned memory for faster data transfers and careful memory release after each step.

These optimizations enabled a peak memory footprint of 144.5 GiB per GPU and a training throughput of 13,700 tokens per second at 1M tokens, with a peak throughput of 24,900 tokens per second achieved at 256k tokens.

Scientific Impact and Validation

The model was fine-tuned on the human genome, using chromosomes 1 and 2 for training and chromosome 22 for evaluation. Post-training at 512k tokens per sequence, the held-out loss improved by 4.8% in 80 steps, with the largest gains observed on tokens with extensive preceding context. This demonstrates the model's ability to leverage long-range dependencies, a critical capability for analyzing structured scientific data such as DNA sequences.

Validation included reinforcement learning runs and comparisons with vLLM, achieving a log-probability gap of 0.04–0.07 across context lengths from 128k to 1M tokens. These results confirm the correctness of the training process while highlighting the practical challenges of scaling long-context models.

Open-Source Contributions

All changes made during this project are open source and integrated into SkyRL, with several upstream patches submitted to Megatron-LM. The implementation tracker, opened on October 14, 2026, documents the full training process, including supervised fine-tuning, reinforcement learning, and memory benchmarks.

This collaboration between Lila Sciences and Anyscale underscores the potential of open-source frameworks to advance AI research and address complex scientific challenges.