System Identification

System Identification is a set of methods to build a mathematical model based on the system’s input/output data.
One of the common techniques in identifying Time-Series is data-driven modeling. In this approach, The goal is to identify a reduced model that indicates mathematical relations between data without going into details. Different models can be used based on:

System complexity
Amount of available data
Model application
Quality of the data

The following sections will describe some of the well-known models in this field.

ARX model

ARX model is a system identification model that uses the linear correlation between previous and future values. ARX model consists of previous output terms (y) and previous input (u) terms multiplied by the linear coefficients (a, b). The linear combination of past input and output value computes the one-step-ahead prediction of output value(y_k+1). Multiple iterations of the one-step-ahead prediction return the multi-step prediction (from y_k+1 to y_k+n). The time window of the past input and output terms is shifted toward next segment for every iteration for multi-step prediction. Here is an example formular with $n_a=3$ and $n_b=2$ for the single input and single output system,

$y_{k + 1}=a_{0} y_{k} + a_{1} y_{k-1} + a_{2} y_{k-2} + b_{0} u_{k} + b_{1} u_{k-1}$

$ARX$ : Auto-Regressive with eXogenous input (“Auto” indicates the past ‘y’ values, and “Exogenous” indicates the past ‘u’ values serving as model inputs)
$a$ : ARX coefficient for the past output value
$b$ : ARX coefficient for the past input value
$n_a$ : Number of terms for the past output value
$n_b$ : Number of terms for the past input value
$n_c$ : Delay between input and output
$k$ : present time step

Here is another example formular with delay $(n_a=3, n_b=2, n_c=2)$ ,

$y_{k + 1}=a_{1} y_{k} + a_{2} y_{k-1} + a_{3} y_{k-2} + b_{1} u_{k-2-2} + b_{2} u_{k-1-2}$

Although the ARX model can be more detailed by increasing the number of terms $(n_a, n_b)$ , it could result in an overfit. Thus, it could be an essential step to compare the training and validation set, ensuring the prediction for the validation set is as good as the training set. The model fitting can be quantified using different statistical methods such as MSE (Mean Squared Error) or SSE (Sum of Squared Error). Read more about ARX models.

State Space model

The State Space model is one of the linear representations of dynamic systems. Converting a model to a State Space structure is routine for process control and optimization applications. These models can be used for both continuous and discrete forms.

$\dot{x} = Ax + Bu$

$y = Cx + Du$

Figure 2. Continuous State Space model structure.

Which $x$ is the array of states, $u$ is the array of inputs, and $y$ is the array of outputs. $A$ is the transition matrix, $B$ is the input matrix, $C$ is the output matrix, and $D$ is the direct transition matrix or feedthrough matrix. Considering n states, m inputs, and p outputs:

$A\in \mathbb{R}^{n \times n}$

$B\in \mathbb{R}^{n \times m}$

$C\in \mathbb{R}^{p \times n}$

$D\in \mathbb{R}^{p \times m}$

that $\mathbb{R}$ indicates set of real numbers. Read more about State Space models.

Transfer Function

Transfer functions are the linearized representations of dynamic systems. The first step to create a transfer function is to calculate the Laplace transform of a differential equation. After applying Laplace transform differential equation converts to an algebraic equation. These algebraic equations can be transformed back into the time domain.

In the industry, transfer functions are base of many prediction and control tuning applications. For the sake of simplicity first, order and second-order transfer functions are more common than others:

First Order Plus Deadtime (FOPDT)

The first order plus deadtime is the simplest form of the transfer function. The structure and unit step response of a FOPDT has shown below. in this equation $\tau_p$ is the process time constant, $K_p$ is the process gain, $\theta_p$ is the process deadtime, $y$ is the process output and $u$ is the process input.

$\tau_p \frac{dy(t)}{dt} = -y(t)+ K_p u(t-\theta_p) \ \Large \xrightarrow[\hspace{1cm}]\mathscr{L} \ \frac{y(s)}{u(s)} = \frac{K_p e^{-\theta s}}{\tau_p \ s + 1}$

Second Order Plus Deadtime (SOPDT)

The step response of SOPDT is usually S-shaped. In the SOPDT equation $\zeta$ is the relative damping factor. Overshoot, Settling time and Rise time are other characterestics of SOPDT step response.

$\tau_p^2 \frac{d^2y}{dt^2} + 2 \tau_p \zeta \frac{dy(t)}{dt} = -y(t)+ K_p u(t-\theta_p) \ \Large \xrightarrow[\hspace{1cm}]\mathscr{L} \ \frac{y(s)}{u(s)} = \frac{K_p e^{-\theta s}}{(\tau_p \ s)^2 + 2\tau_p\zeta + 1}$

Neural Network model

Neural networks are a set of neurons contacting each other using corresponding weight coefficients. There are many kinds of neural networks, according to their application. Multilayer Perceptron is commonly used for control applications because of its simplicity and fast response. This neural network is made up of three sections: the input layer, the hidden layer(s), and the output layer. An input layer, where the input data comes in. It is conventional to normalize data before calculations to avoid neuron saturation. Neuron saturation is a state that which a neuron gives a constant value for each entering data. The output layer receives the output data and denormalizes the results. The layers between the input layer and the output layer call the hidden layer(s) where. The number of these layers and their neurons can vary depending on the complexity of the system. Increasing hidden layers and neurons require more data and time to train the network.

simple_nn — Figure 3. A simple neural network with one hidden layer

Inputs of the network are input(s), delayed input(s), output(s), and delayed output(s) of the system. The number of delays given to the inputs and outputs of the system depends on its complexity. After specifying the network’s structure (number of hidden layers, number of neurons in each layer, inputs, and output of the network), weights and biases will be initialized randomly. Normalized data will be passed through the network, identification error will be calculated, the error backpropagates through the network, and weights and biases will be updated till the Mean of Square Error (MSE) meats the specified criteria then the training procedure will be terminated. Now the network is ready to mimic the system’s behavior and can be used as a model which correlates the input(s) and output(s) of the system.

neuralnetwork_flowchart — Figure 4. Neural network identification flowchart