SpectraTr: A novel deep learning model for qualitative analysis of drug spectroscopy based on transformer structure

1. Introduction

The active ingredients of the same type of drug produced by different manufacturers differ due to distinctions in raw materials, manufacturing processes, resulting in price differences. Unscrupulous merchants take the similarity of active ingredients in drugs to package low-cost drugs into high-priced drugs from well-known manufacturers. This happens from time to time, seriously damaging consumers’ rights and interests as well as the reputations of well-known manufacturers. Therefore, it is critical to identify drug manufacturers during drug supervision.

Traditional methods for the identification of drug manufacturers include chemical methods, HPLC authorization methods, etc. But the detection process is cumbersome, the analysis speed is slow and destructive. As one of the mainstream nondestructive testing techniques, near-infrared spectroscopy has been widely used in many fields due to its fast detection speed and low cost.^1,2,3,4 However, the drug supervision methods based on spectral analysis are heavily dependent on the chemometrics model which characterizes the relationship between spectral data and drug categories. The resolution of spectroscopic measuring devices has grown in tandem with the advancement of manufacturing technology. While more information about the analysis target can be obtained, the high-dimensional and easily disturbed spectra also make it difficult for traditional chemometrics to directly extract effective features. Therefore, pre-processing and wavelength selection algorithms are needed prior to modeling.^5,6,7 At the same time, due to the wide variety of existing drugs and the continuous research and development of new drugs, the cumulative drug data also poses challenges to methods such as PLS and SVM.⁸

The application of advanced methods represented by deep learning in spectroscopy analysis promotes the development of chemometrics. Autoencoder networks (AEs) and convolutional neural networks (CNNs) are the most common deep learning-based spectral analysis approaches currently available. The former uses an autoencoder network to reconstruct features and then classifies them with traditional classifiers, while the latter extracts feature based on convolutional layers and use fully connected layers for classification.^9,10,11,12 Although both can automatically learn and extract features from a large volume of spectral data, with low dependency on preprocessing and wavelength selection engineering, some studies have shown that convolution neural networks perform better in applications.^13,14,15 However, CNNs have many hyperparameters, and their prediction performance depends on appropriate hyperparameter selection, which leads to difficult network construction.^16,17,18,19 According to the most recent deep learning research, the transformer model based on the attention mechanism has the feature extraction potential of the CNN model and is easier to build network parameters. It is one of the current research hotspots of deep learning.^20,21,22 Some studies have applied transformers to various fields such as drug design,²³ chemical synthesis,²⁴ protein three-dimensional structure prediction,²⁵ medical imaging,²⁶ and hyperspectral analysis,²⁷ and demonstrated excellent results. But its application potential in molecular spectroscopy has not been studied yet.

This paper proposes a novel deep learning model for drug spectral identification, in order to build an analytical model with the better predictive ability and lower hyperparameter dependence. The approach is named SpectraTr based on the transformer structure. The performance of the model was verified based on the spectral data sets of our drugs from multiple manufacturers, and then the public drug data sets were used to further prove the advanced nature of the model. In the following paper, we will introduce our experiment materials and the mechanism of the transformer network structure in Sec. 2. In Sec. 3, the experiment procedure and the classification prediction results in different datasets will be discussed. Section 4 will summarize the whole paper. The source code is available at https://github.com/FuSiry/Transformer-for-Nirs for academic use only.

2. Materials and Methods

2.1. Datasets

Dataset A is provided by the National Institute for Food and Drug Control and all samples pass the statutory method test. It comes from 333 cefixime drugs produced by Guangzhou Baiyunshan Pharmaceutical General Factory and 716 phenytoin sodium drugs produced by 11 pharmaceutical companies including Southwest Pharmaceutical Co., Ltd. The raw spectra of both drugs are shown in Fig. 1 and are measured by the Bruker Matrix spectrometer. Each spectral wavelength ranges from 4000 to 11995cm${}^{-1}$, with intervals of 4cm${}^{-1}$ and a total of 2074 absorption points. It can be seen from the figure that the spectra of similar drugs from different manufacturers are similar.

Dataset B is an open-source drug tablet dataset with a wavelength range of 7400–10507cm${}^{-1}$ and 404 absorbances points. The samples were classified into four categories according to their active ingredients. The raw spectra are shown in Fig. 2 and can be obtained from http://www.models.life.ku.dk/Tablets.

2.2. SpectraTr architecture

As one of the most groundbreaking works in the deep learning field, the transformer has been widely discussed. We propose SpectraTr, as shown in Fig. 3, based on its work.²⁸ SpectraTr is similar to a classifier based on AE network and consists of an encoder and MLP layers. The raw spectrum is divided into several spectra patches. After the position coding, the spectral features are extracted from the Spectra encoder composed of multi-head attention layers and weighted by MLP layer, which obtains the qualitative analysis results of the drug spectrum.

2.2.1. Position embedding

SpectraTr splits the full spectrum into multiple spectra patch sequences and extracts the features, which makes it compatible with 1d spectrum and 2d image data. Previous researchers have made similar attempts in the CNN network by splitting the raw spectrum into multiple segments and then entering the model, with better results.¹⁶ However, the position information of the spectra patch in the raw spectrum is destroyed because of the splitting of the spectrum into multi-segment spectra patches. In order to retain the position information, the position encoding is performed on the spectra patch, as shown in the following equation :

For each spectral patch, create a positional encoding of the Gaussian distribution of the $1\times D$ dimension at random, and then combine all positional encodings of spectral patches to get the entire positional encoding information.

2.2.2. Multi-head attention mechanism

The self-attention mechanism is the core innovation of the transformer model. It can automatically focus on the important information in the data, increase the weight of important information and decrease the weight of unimportant information in the input data. The implementation of the attention mechanism in chemometrics is similar to automatic wavelength selection.²⁹ The computational procedure of the self-attention mechanism is shown in Fig. 4.

The matrices $q_i$, $k_i$, and $v_i$ are obtained by linear transforming the input sequence $a_i$. Then, the elements of matrix $q_i$ from $a_i$ are dot-product with the corresponding elements of the matrix $k_j$ from $a_j$ to obtain $a_{ij}$, which is the correlation between $a_i$ and $a_j$, as shown in the following equation :

where $d_k$ is the vector dimension of the matrix $k$ to keep the dot-product with $q_i$ and $k_j$ from being too large. The correlation obtained from the vectors $a_i$ and $a_j$ is input into the softmax function to obtain the probability distribution of $a_{i,j}$, as shown in the following equation :

Finally, the relationship degree of each input vector with other vectors $a_{i,j}^{\prime }$ is multiplied by $v_j$, to obtain the attention score of the input, as shown in Eq. (4). If $a_i$ is strongly correlated with $a_j$, the calculated value of $a_{i,j}^{\prime }$ will be larger. Eventually, $o_{i,j}$ will be higher, representing the information is more critical.

Multiple self-attention layers are used to form a multi-head attention block to capture richer feature information. This is similar to the principle of using multiple filters simultaneously in CNN. The schematic is shown in Fig. 5.

A single input $a_1$ is sequentially stacked through the output matrix of $n$ self-attention layer. Then, the result is linear matrix transformed to obtain the final attention matrix $o_1$. Multiple self-attention layers form multiple subspaces, which allow the model to focus on different aspects of information.

2.2.3. Layersnorm and residual connection

A layersnorm (LN layer) is added before and after the multi-head attention block to eliminate the undesirable effects caused by odd sample data, which is similar to the principle of using Batchnorm (BN) in CNN, as shown in the following equation :

where $\gamma$ and $\beta$ are the scaling and translation variables of the LN layer, which are the parameters that can be learned to ensure that the original learned features are retained after each normalization. $\epsilon$ is a minimal value, which aims to ensure that the target is not 0. $\mu$ and $\sigma$ are, respectively, the mean and variance of the LN layer. Specific calculation formulas are shown in the following equations : where $H$ is the number of hidden nodes in the layer. Overall, the LN makes the same layer neuron inputs have the same mean and variance by normalizing all neurons in each layer.

Since the spectral encoder also uses a stacked structure similar to CNN, an over-deep network structure may cause the gradient disappearance problem. Therefore, drawing on the idea of Resnet, a residual mechanism is introduced between the multi-head attention block and the MLP block, as shown in the following equation³⁰ :

where $x$ is the input. $F(x)$ is the output of $x$ after the convolutional layer. $H(x)$ is the sum of $x$ and $F(x)$. In the backpropagation process, $H(x)$ can receive the gradient information transmitted by $x$ and $F(x)$ at the same time. Even if the network is too deep, the updated signal can still be obtained through branch $x$. Therefore, the gradient disappearance can be effectively suppressed.

2.3. Modeling

SpectraTr is implemented based on Python 3.8 and the open-source deep learning framework Pytorch 1.8.0. It runs on workstations equipped with GeForce MP40GPU, Intel$\circledR$Corei7 4770K CPU @ 3.5GHz and ubuntu operating system. The network structure of SpectraTr is described in Sec. 2.2. The initial parameters are shown in Table 1.

On both datasets, cross-entropy is used as the loss function and the Adam optimizer is used for training. The total epoch is set to 200. In order to balance the training time and training effect, the learning rate decline strategy is introduced. If the loss of the training set does not decline within 10 epochs, the learning rate becomes half of the current one. The early_stop strategy is also introduced to prevent over-fitting. If the loss of the test set does not drop within 30 epochs, the training stops. The final selected parameters are shown in Table 2.

2.4. Baseline comparison

The classical chemometrics model SVM, PLS_DA, and the mainstream deep learning model SAE and CNN network are used as baseline models, respectively and compared with the proposed SpectraTr. SVM and PLS_DA are implemented by the machine learning library scikit-learn, where SVM parameter C is set to 1.0, gamma is set to 0.001, PLS_DA component is set to 22, and the rest is the default. The SAE network consists of three encoder stacks of input-1024, 1024-256, 256-output, as in the reference study.⁸ The CNN network consists of three convolution layers and three fully connected layers, as in the reference study.¹³ The above model is the same as the operating environment for SpectraTr.

2.5. Model evaluation

Models are evaluated using the classical qualitative analysis metrics: accuracy, specificity, recall, and F1-Score. Accuracy is the number of samples predicted correctly as a percentage of the total. Precision represents the percentage of positive samples predicted by the model that is actually positive. The recall represents the percentage of samples actually positive that are predicted to be positive. F1-score is the result of reconciling precision and recall. Their mathematical expressions are shown in the following equation :

where TP is true positive, which indicates both the real label and the prediction are positive. FP is false positive, indicating that the true label is negative, but the prediction is positive. FN is false negative, representing the true label is positive, but the prediction is negative. FP is true negative, which presents both real label and the prediction are negative.

3. Result and Discussion

3.1. Result of dataset A

3.1.1. Performance of spetralTr in dataset A

In this section, based on the proposed SpectraTr model, the near-infrared spectroscopy of cefixime and phenytoin sodium produced by multiple manufacturers are predicted and compared with other baseline methods. For comparison with previous research, we use the same dataset as in Ref. 13 and follow the arrangement of experiments described in the text. First, a qualitative analysis of cefixime tablets from seven manufacturers was carried out. Then, the mixture of cefixime tablets and phenytoin sodium tablets was used for qualitative analysis of 18 manufacturers in two drug classes. In experiments, 80% of the samples were randomly selected as the train set, and the remaining 20% were used as the test set. Both train and test sets use standardized pre-processing to speed up the model training, and all experimental results come from the test set. The prediction performance of SpectraTr on cefixime and mixed drugs is shown in Table 3.

The proposed SpectraTr model has excellent manufacturer identification capabilities for both cefixime and mixed drug spectra. The prediction accuracy on the cefixime data set of seven manufacturers achieves 100%, and prediction accuracy on the spectrum data set of 18 manufacturers of mixed drugs also achieves 99.5%.

3.1.2. Model performance under different training sets

The usual spectrum data is less due to the expensive cost of gathering spectral data and labeling it. However, deep learning is driven by the data. It is easy to overfit and leads to poor prediction performance with fewer samples. Therefore, the training set is randomly selected from the drug spectroscopy data set at a ratio of 0.8 to 0.2 to construct training sets and testing at different scales. Comparing SpectraTr with the baseline model, the accuracy is taken as the evaluation index, and the result on cefixime drugs is shown in Table 4.

The prediction accuracy of the deep learning model is better than that of the classic chemometric method when the pre-processing method is not precisely chosen, and there is no notable performance erosion when the sample size is reduced. The proposed SpectraTr model performs best on every different scale train set. The prediction accuracy even reaches 100% when the train set is 266 and 233. After the train set is reduced to 66, the accuracy is still 97.06%, which is slightly better than CNN and SAE, and significantly better than PLS_DA and SVM. Construct train sets with various proportions and conduct studies on mixed drugs in the same way. The result is shown in Table 5.

The results of deep learning methods are still better, similar to the results on cefixime. In most cases, SpectraTr is the best model in the majority of scenarios. It achieves a prediction accuracy of 99.68% when the training sample is 734. When the training sample is reduced to 209, the result is still 93.81%, which is better than CNN and SAE. However, when the train set is reduced, the performance loss is more noticeable than on cefixime, indicating that deep learning performance in small samples is also restricted by the data set’s classification difficulty.

3.1.3. Influence of pre-processing algorithm on model prediction ability

The drug is pre-processed with the methods of standardization, MSC, SG (window is 15), and SNV, respectively, to compare the effects of different pre-processing algorithms on the model. Under the 7:3 train set and test set division, the experimental results of the model on the data set A are shown in Table 6. SVM, PLS_DA are greatly affected by the pre-processing algorithm, under the suitable pre-processing can have more than 90% accuracy rates. But in most cases, the model does not have usability. Deep learning shows a low dependence on pre-processing. Although different pre-processing methods have different predictive performances, they are all acceptable, which is one of the main reasons for the gradual application of deep learning in spectral analysis.

In this experiment, we observed that the SpectraTr model is less dependent on pre-processing than the CNN model. On cefixime, the accuracy of the original spectra-based SpectraTr model was 99%, with a difference of less than 1% from the prediction of the qualitative analysis model based on the pre-processing spectra. The raw spectral-based SpectraTr model achieves 96.42%, while the Stander, MSC, SG and SNV pretreatment models achieve 99.68%, 96.30%, 99.15%, and 94.06% on cefixime and phenytoin sodium, respectively. This demonstrates that the SpectraTr model can extract features from spectral data with various noise components automatically.

3.2. Result of dataset B

To compare with previous research effectively, we used the same open-source dataset as in Ref. 10 and conducted experiments with the arrangement described in the text. 201 samples were randomly selected to create the train set, and the remaining 99 samples were utilized as the test set. The prediction performance of the SpectraTr model on the original spectrum is shown in Table 7.

The SpectraTr model achieve 96.97%, 97.03%, 96.97%, and 96.98% of accuracy, specificity, sensitivity, and F1-score, respectively, better than the model built based on CNN. The prediction performance of SpectraTr model and other baseline models under different pre-processing are shown in Table 8.

SpectraTr reaches 99.21%, 89.36%, 96.09%, and 88.85%, respectively after the pre-processing of standardization, MSC, SG, and SNV, which are better than other models in most cases. But pre-processing has a negative gain on SpectraTr in the vast majority of situations. We think that the deep learning method does not rely on pre-processing or only needs standardized pre-processing, because the deep learning method has good spatial invariance. The extensive application of the CNN algorithm in the image shows that image rotation, translation, and clipping have little effect on the prediction results of the deep learning model. In our previous studies, we have also found that the deep learning method has good adaptability to the slight rotation of spectra.³¹ Therefore, deep learning does not rely too much on preprocessing algorithms to remove noise such as baseline drift in spectra but needs to be standardized to allow faster gradient convergence.

3.2.1. The effect of hyperparameters on model performance

The above experiments have shown the qualitative analysis performance of the SpectraTr model and the ability of automatic feature extraction under different preprocessing. But the research shows that the network of the deep learning model is large and sensitive to the setting of network parameters. Therefore, on our drug and open-source drug data sets, the depth of the spectral encoder, the number of multi-heads, the number of patches, and the dimension of Mlp_dim are used as objects to study the impact of the above parameter settings on the spectralTr. The results are shown in Fig. 6.

The effect of different patch numbers on the model is shown in Fig. 6(a). The difference in dataset A is not large. It is relatively better when the patch number is set to 40. As for dataset B, the performance is relatively best when the patch number is set to 10. When the patch number is set to 25, the prediction accuracy drops below 95%. However, due to the different spectral dimensions of dataset A and dataset B, all the results perform better when there are about 50 spectral points in the patch. Figure 6(b) shows the influence of Spectra encoder depth on the model. On the two data sets, the effects are similar. A network that is too shallow is not enough to extract features. If it is too deep, it is easy to overfit. From Fig. 6(c), the parameter setting of the number of attention heads has little effect on the model, and the model prediction accuracy fluctuates around 3% under different parameter settings. The influence of different parameters of Mlp_dim on the model performance is shown in Fig. 6(d), and the effect is better when it is around 512 or 1024. In summary, the four groups of main parameters can make the model have better predictive performance in a larger range, which shows that the SpectraTr model has low sensitivity to the parameter settings of the network layer. Therefore, when setting up the SpectraTr model, there is no need to search for the optimal parameter combination one by one from the massive parameter space. This has an advantage over the CNN model in modeling.

3.3. The result of training time

The training time of different models on datasets A and B is shown in Fig. 7. Different numbers of training samples will affect the model training time. The experiment of the table dataset was carried out when the number of training samples was 211, and the rest of the dataset was tested under the 8:2 training test set.

The training period for deep learning is substantially longer than for SVM and PLS_DA algorithms since it takes several training sessions. The need for greater hardware resources and longer training time is the drawback of deep learning methods. The SpectraTr model has the longest training time among the deep learning models. The SpectraTr model’s training time is affected by both the number of samples and the spectral dimension. The training time increases as the number of samples and spectral dimensions increases.

4. Conclusion

A novel and advanced deep learning-based model is proposed called SpectraTr for qualitative analysis of near-infrared spectroscopy of drugs. Experimental results on multiple drug classification datasets show that SpectraTr outperforms the classical PLS, SVM, and existing deep learning methods AE and CNN in most cases. In addition, the model can automatically extract features from spectral data and does not rely on complex pre-processing and model parameter selection engineering. The initial application of SpectraTr in spectroscopy has shown potential and is expected to become a major class of deep learning methods in addition to AE networks and CNN networks.

Acknowledgments

The authors would like to express our gratitude to anonymous reviewers and editors for their helpful comments. This work was supported by the National Natural Science Foundation of China (61906050 and 21365008); Guangxi Technology R&D Program (2018AD11018), and Innovation Project of GUET Graduate Education (2021YCXS050).