Analysis of Complex Dynamical Systems by Combining Recurrent Neural Networks and Mechanistic Models

Abstract

The aim of this study is to analyze the time series data in an innovative manner, which combined the deep learning model with the mechanistic model to form a hybrid model. In order to demonstrate the feasibility and theoretical background of the hybrid model, the Lorenz System is used as an example to explore the underlying mathematical mechanism. Through several simple exams, it is observed that the Bidirectional Long Short Term Memory (BDLSTM) in the hybrid model could be trained to learn the dynamical behavior of the complex dynamical system properly and the mechanistic model in the hybrid model could adapt to different situations flexibly. In practical exploration, a real hybrid model, namely the Argiculture-informed Neural Network (AINN) model, that consists of deep neural networks (DNN), including but not limited to: Long Short Term Memory (LSTM) , Convolutional Neural Network (CNN), and Transformer, and Dynamic Land Ecosystem Model (DLEM), are proposed to predict nitrous oxide emissions in agricultural fields. The model is trained using data collected from smart farm, named Area X.O, in Ottawa, Ontario, Canada, during the 2021 growing season and tested using data from the 2022 growing season. During the data preprocessing stage, we utilized the Robust Scaler (or MinMax Scaler) to scale the data into a narrow range. Our statistical analysis revealed that temperature and humidity are closely related and share similar time series patterns, suggesting that they contain the same information for predicting nitrous oxide (N$_2$O) emissions. To evaluate the model's performance, several metrics were used, including mean absolute error (MAE), mean absolute percentage error (MAPE), root mean square error (RMSE), and coefficient of determination (R$^2$). Our results indicate that the AINN outperforms the pure DNN and DLEM in both the training and testing datasets, because in the hybrid model, the Recurrent Neural Network (RNN) part could catch the dynamical behavior of the emission of N$_2$O, and the DLEM part could regulate the training path and lead the entire model to converge in a limited number of basins of attraction. Additionally, since it is costly to collected the data from the field, it is better for us to do time series data augmentation using Generative Adversarial Network (GAN), with the aim of closely matching the original data distribution while also preserving the dynamic behavior of the original data. However, even state-of-the-art GAN models like TimeGAN fall short in preserving the temporal dynamics present in the original time series due to the absence of first-order difference information. To address this limitation, this study proposes a novel process for generating multivariate time series data. The proposed process comprises four essential modules: a) the GAN module for generating multivariate time series data, b) the sampling module for preserving the first-order difference distribution, c) the smoothing module for refining the generated data, and d) an evaluation module using the Kolmogorov-Smirnov Test (KS-test) and Hilbert-Schmidt Independence Criterion (HSIC), along with other metrics to test the synthetic time series data. This comprehensive approach ensures that the synthetic time series data maintains both the distribution and the dynamic behavior of the original data. With the advent of quantum computing, we transitioned from the conventional LSTM to quantum LSTM and formulated the Quantum Long Short Term Memory-Dynamical Land Ecosystem Model (QLSTM-DLEM) model, showcasing enhanced generalization capability and stability. Experimental results indicated that QLSTM-DLEM achieved comparable performance to Long Short Term Memory-Dynamical Land Ecosystem Model (LSTM-DLEM) using several quantum bits.