In an article recently submitted to the ArXiv* server, researchers examined the role of AI in understanding and forecasting weather patterns by advancing the growing interest from multiple sectors in creating digital Earth twins. They discussed recent advances in transformers, physics-informed machine learning, and graph neural networks by moving toward generalizable AI foundation models (FM). While noting the early stage of development, this paper explored criteria for success and potential applications, which suggested that AI methodologies were now evolved enough for a weather foundation model.
Study: AI Breaks New Ground in Weather Forecasting. Image credit: IgorZh/Shutterstock
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
Background
The 2010s marked significant progress in deep learning that was driven by large datasets and Graphics Processing Unit (GPU) computing. This era introduced FM and self-supervised models with millions or billions of parameters, primarily in natural language processing (NLP). While NLP benefited initially, the weather and climate domains have also embraced these techniques. Numerical weather prediction (NWP) has succeeded, but there is a long history of using data-driven methods to post-process NWP outputs. Meteorological and climate research organizations continue to generate massive datasets, such as ECMWF's 450 petabytes of archived data and daily 300 terabyte additions.
Proposed Method
Developing an FM for weather and climate requires careful consideration of the relevant spatiotemporal scales within the Earth system, which range from seconds to centuries and encompass a diverse spatial range. It is vital to account for external cycles linked to the solar system that affect Earth's weather. Identifying areas where machine learning (ML) can provide the most value through foundational modeling and selecting feasible downstream tasks with societal and business implications are critical. The availability of ample pre-training data is essential for the success of FMs. This data should showcase a rich variety of phenomena, often spanning decades. The exploration of having individual FM systems for Earth's various subsystems, including the atmosphere, hydrosphere, cryosphere, biosphere, and geosphere, is a future possibility. The challenges related to obtaining high spatiotemporal resolution datasets, handling variables across multiple sources, and modeling specific temporal scales are discussed.
Several innovative AI approaches have been introduced for weather forecasting and atmospheric dynamics modeling. For instance, Pangu-Weather utilizes a 3D Earth-specific transformer to improve global weather forecasting while reducing computational costs. Another approach leverages temporal dynamics and diffusion models for multi-step and long-range forecasting. Geometric Clifford Algebra Networks (GCANs) explore symmetry group transformations and geometric algebras to improve the modeling of three-dimensional rigid body transformations and large-scale fluid dynamics simulations.
These self-supervised learning (SSL) and joint embedding methods provide efficient alternatives to numerical solvers. Transformer-based architectures, specifically three-dimensional self-attention and two-dimensional cross-attention, have not only reduced computational complications but have also proven to be highly efficient in weather forecasting. The Spherical Fourier Neural Operators (SFNOs) broaden the capacity of Fourier Neural Operators (FNOs) to design atmospheric dynamics on spherical geometries. Meanwhile, the Convolutional Neural Operators (CNOs) alter Convolutional Neural Network (CNN) architectures to determine solution operators for partial differential equations. These ground-breaking approaches are promising in enhancing weather forecasting and climate modeling, opening doors to more precise predictions and a deeper comprehension of climate change's consequences. Nevertheless, comprehensive research and rigorous benchmarking are essential to fully assess their potential advantages and drawbacks within operational systems.
Prerequisites, Trade-offs, and Data Representation in ImplementingWeatherFM
The evaluation of specific prerequisites for implementing a WeatherFM involves an assessment of trade-offs associated with various model components, including transformer blocks, data representations, and neural operators. Developing a weather model must be guided by two critical objectives: multi-scale robustness and long-term stability. Multi-scale robustness equips the model to effectively handle diverse data sources by accommodating variations in both distance and time. Simultaneously, long-term stability ensures the reliability of the model even when predictions extend beyond its initial design considerations. Additionally, the imperative task is to address any code gaps that could impact computational performance, particularly when dealing with the immense volume of data inherent in weather and climate studies.
The foundation of any ML solution lies in its data representation, and weather and climate data introduce unique challenges distinct from conventional image and video data. Although some similarities exist, the distinctive characteristics of weather and climate data necessitate careful considerations beyond off-the-shelf solutions. This paper meticulously examines key considerations about the representation of weather and climate data, setting the stage for the subsequent discussions.
Conclusion
In summary, the adoption of FM in weather and climate is still in its early stages. Key considerations include defining the scope, handling varying data resolutions, and ensuring flexibility in spatial coverage using transformers. While forecasting is a primary application, challenges related to rollout stability and fidelity to extremes must be addressed in this evolving field.
Furthermore, as this field progresses, it is essential to establish design principles for foundation models that consider both meteorological requirements and computational efficiency. Utilizing a wide range of meteorological applications beyond forecasting, including climate simulations and downscaling. This has the potential to increase the impact of these models. Continuous research and collaboration are vital to refine and expand the capabilities of foundation models for weather and climate.
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
Boost Machine Learning Trust With HEX's Human-in-the-Loop Explainability