In the News

Highly advanced Generative Artificial Intelligence (AI) language models, like ChatGPT, use deep machine learning algorithms to generate human-like, conversational text responses to prompts. The technology is incredibly useful for tasks like writing code, composing emails or essays, or engaging in conversations, but these models have limitations when it comes to more complicated tasks. Imagine, instead, a machine learning technology that can respond to scientific inquiries with clear mathematical expressions, augmenting existing knowledge, understanding, and applicability. 

Such technology could be a “holy grail for engineers and scientists,” according to Pierre Lermusiaux, the Nam Pyo Suh Professor of Mechanical Engineering and Professor of Ocean Science and Engineering. Applications could include better weather, ocean, earth system, and climate forecasts, improved energy systems, new control and robotics systems, better financial and economic market forecasts, and even new human health models.

Pierre Lermusiaux (Right), the Nam Pyo Suh Professor of Mechanical Engineering and Professor of Ocean Science and Engineering, and Abhinav Gupta PhD '22

An approach called generalized neural closure models (gnCMs), created by Abhinav Gupta PhD '22 and Lermusiaux combines machine learning with fundamental physics to discover and improve predictive models. By incorporating neural networks into existing mathematical models, the researchers have achieved better interpretability and generalization across different scenarios – and, unlike general deep learning models, “the mathematical answer is not hidden in the machine learning black box,” Lermusiaux says. “It is laid bare for humans to use and to learn from.”

Using machine learning from scratch works well in situations where existing models aren’t available. Generating images, for example, is something machines can do really well working from big data, but in physics-based systems or biological systems—where governing laws already exist but observations are limited—it doesn’t make much sense to start from scratch.

“Over centuries, people have built models – which are based on physical, biological laws – and they are inherently generalizable,” says Gupta. “The idea of our approach is to leverage the generalizability of the existing models, then piggyback on that to make sure machine learning models that build on top of that are also generalizable in the same sense.”

The field of Scientific Machine Learning has seen the introduction of several innovative methods that combine machine learning with existing scientifically derived differential equation models and computational physics. This is because all models are approximate. In the new application, machine learning is used to learn and represent neglected and unresolved terms to express missing dynamics as functions of modeled state variables and parameters, which is referred to as neural closure modeling.

Take Newton's second law of motion. “If I drop my watch, it’s going to fall,” says Lermusiaux. “There’s a force, gravity, and mass, so there’s acceleration. If I kick a soccer ball, that’s also F = ma, but the full picture is more complicated because you have to simulate the fluid of the air around the ball, the structure of the ball, and such.”

To really understand and explain the motion of the ball, factors like the drag created by the surrounding air must be considered. Without viscous effects, the ball continues to accelerate; closure models then account for these viscous effects. 

The gnCMs framework provides full autonomy for the design of the unknown closure terms, including, for the first time, capturing delays in the system. It learns new closure models for existing known-physics and low-fidelity models – using sparse real, high-fidelity data – to ensure that learned closure models are generalizable and interpretable, and it makes this closure modeling more accessible. 

Once a simple soccer ball model exists, the gnCMs approach can augment and interpret the knowledge gained to allow reuse for other applications—a researcher trying to understand and predict curveballs in baseball, for example, could benefit from the understanding of fluid dynamics gained developing the soccer ball model—a task that most machine-learned models traditionally lack.

Closure modeling is necessary for a variety of reasons. It can decrease computational costs and is beneficial in situations where simple models are preferred over complicated models or when scientific understanding of the processes and variables involved is lacking, but overdependence on too simple models can have consequences. In general, it is a significant scientific undertaking to find such closure models even for very simple systems. 

Simple or known models (low-fidelity models) may be computationally cheaper or faster to use than high-fidelity models (models drawn from real-world data), but low-fidelity models can quickly accumulate errors due to unaccounted or missing, neglected, or unresolved terms. This can lead to incomplete representations of data or incorrect approximations which, in research on dynamical systems, can have real-world, global-scale impacts.

The gnCMs method builds on learned information for dynamical systems and then successfully discriminates among existing models, discovers missing physics, identifies numerical errors, selects optimal functional forms, identifies generalized models outside of their sparse training data, and compensates for the lack of complexity in simpler models without compromising computational speeds – and the framework works for many dynamical systems.

“We wanted to create a framework which would enable people, in an agnostic fashion, to augment existing models and make them more accurate without sacrificing too much on computational costs,” Gupta says. “Or, if a researcher doesn’t have a model, this approach can help them discover a model from scratch, if needed.” 

This is particularly beneficial when attempting to collect data in real, physical systems – and from natural systems in particular – which tends to be difficult and expensive. “For example, collecting data from a system as vast as the ocean is really resource intensive,” says Gupta. “The kind of machine learning models we’ve developed reduces the data requirement.”

For ocean processes, the team has been able to augment existing physical, biogeochemical, and ecosystem models – machines correct missing terms and numerical errors in dynamical models of nonlinear waves and shocks, and discover mathematical formulations that better explain real impacts on ocean acidification and zooplankton mortality. 

In one experiment, the researchers started with low-fidelity models used to study and predict essential carbonate chemistry and biological production cycles and their interplay with global warming. They then added high-fidelity data to resolve biological variables, nitrogen, depth, and other more complex parameters like the maximum growth rate of phytoplankton, saturation and light measures, and known data for photosynthetically active radiation at the sea surface.

In another experiment, for the low-fidelity model, the researchers assumed only prior knowledge about the existence of a linear zooplankton mortality term. For the high-fidelity model, however, the true zooplankton mortality contained an additional quadratic dependence. Their framework not only ensured that the learned closure kept errors low throughout the observation period, but also discovered the exact functional form of the Zooplankton mortality in the high-fidelity model. 

In these and other experiments, they found that training on a limited set of data is sufficient to ensure that the learned closures are generalizable over large ranges of grid resolution and initial and boundary conditions and problem-specific parameters, and to outperform popular closure models. 

“Usually, the way science and engineering progress is that you start from what you know and what you’ve built, then you create new thoughts, new ideas, then build new things,” says Lermusiaux. “The machine learning we’ve created fills in more complex knowledge gaps, and reduces the time and expense to help get to the answers you’re seeking.” 

“Generalized neural closure models with interpretability” appeared in the journal Scientific Reports. Gupta, A., Lermusiaux, P.F.J. Generalized neural closure models with interpretability. Sci Rep 13, 10634 (2023).

This article appeared in the Spring 2024 edition of our magazine, MechE Connects. For more stories, and for past issues, please visit meche.mit.edu/magazine.