Model Card for Panda-72M

This is a scaled-up version of the checkpoint originally presented in our preprint. We will include results with this checkpoint in the appendix of our next preprint update. This model has 12 layers with 12 attention heads each.

Trained with larger dataset of multiple initial conditions per system, with mixed periods as well. Specifically, using 8 out of the 16 initial conditions (ICs) per system that we provide in our skew-mixedp-ic16 dataset. We trained this model for 800k iterations, with per-device batch size 384, across 6 AMD MI100X GPUs. Panda: Patched Attention for Nonlinear Dynamics.

Paper abstract:

Chaotic systems are intrinsically sensitive to small errors, challenging efforts to construct predictive data-driven models of real-world dynamical systems such as fluid flows or neuronal activity. Prior efforts comprise either specialized models trained separately on individual time series, or foundation models trained on vast time series databases with little underlying dynamical structure. Motivated by dynamical systems theory, we present Panda, Patched Attention for Nonlinear DynAmics. We train Panda on a novel synthetic, extensible dataset of 2 \times 10^4 chaotic dynamical systems that we discover using an evolutionary algorithm. Trained purely on simulated data, Panda exhibits emergent properties: zero-shot forecasting of unseen real world chaotic systems, and nonlinear resonance patterns in cross-channel attention heads. Despite having been trained only on low-dimensional ordinary differential equations, Panda spontaneously develops the ability to predict partial differential equations without retraining. We demonstrate a neural scaling law for differential equations, underscoring the potential of pretrained models for probing abstract mathematical domains like nonlinear dynamics.

• Preprint: arXiv:2505.13755
• Repository: https://github.com/abao1999/panda

Citation

BibTeX:

If you find our work valuable for your research, please cite us:

@misc{lai2025panda,
      title={Panda: A pretrained forecast model for universal representation of chaotic dynamics}, 
      author={Jeffrey Lai and Anthony Bao and William Gilpin},
      year={2025},
      eprint={2505.13755},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.13755}, 
}