Fine-Tuned MLP-Mixers as data-driven Numerical Surrogates?

Abstract

Scientific Machine Learning has significantly advanced climate science by enabling precise forecasting of complex dynamical systems. While state-of-the-art models excel in domain-specific tasks, recent advancements in time series-based foundation models seek to replicate the success seen in natural language processing and computer vision. This study investigates whether a "small" MLP-Mixer-based foundation models, Tiny Time Mixers (TTMs), can be fine-tuned to forecast complex real-world dynamical systems accurately while adhering to practical resource and cost constraints. Our findings reveal that TTMs are sensitive to the dynamical characteristics present in the training data, particularly in terms of amplitude and periodicity, yet significant variations in forecast accuracy were observed within the same training distribution. These results highlight the need for further adaptation of TTMs to enhance their robustness in specialized SciML forecasting tasks.