Recognition of Similar Weather Scenarios in Terminal Area Based on Contrastive Learning

，，，

1.College of Computer Science and Technology，Nanjing University of Aeronautics and Astronautics，Nanjing 211106，P.R.China；

2.College of Civil Aviation，Nanjing University of Aeronautics and Astronautics，Nanjing 211106，P.R.China；

3.Collaborative Innovation Center of Novel Software Technology and Industrialization，Nanjing 210023，P.R.China

Abstract: In order to improve the recognition accuracy of similar weather scenarios（SWSs）in terminal area，a recognition model for SWS based on contrastive learning（SWS-CL）is proposed.Firstly，a data augmentation method is designed to improve the number and quality of weather scenarios samples according to the characteristics of convective weather images.Secondly，in the pre-trained recognition model of SWS?CL，a loss function is formulated to minimize the distance between the anchor and positive samples，and maximize the distance between the anchor and the negative samples in the latent space.Finally，the pre-trained SWS-CL model is fine-tuned with labeled samples to improve the recognition accuracy of SWS.The comparative experiments on the weather images of Guangzhou terminal area show that the proposed data augmentation method can effectively improve the quality of weather image dataset，and the proposed SWS-CL model can achieve satisfactory recognition accuracy.It is also verified that the fine-tuned SWS-CL model has obvious advantages in datasets with sparse labels.

Key words：air traffic control；terminal area；similar weather scenarios（SWSs）；image recognition；contrastive learning

0 Introduction

In recent years，with the rapid development of the domestic air transport industry，the limited air?space resources and traffic surges in the airspace have made the air traffic control more complicated.Weather is the main factor affecting the air traffic and different weather will have different degrees of influence on the traffic control in terminal area.With long-term experience，some rules for formulating or selecting control strategies can be discovered and summarized.For experienced controllers，they usu?ally can release operation strategies quickly based on their experience.However，for most controllers，due to the lack of experience in dealing with some weather scenarios，it is difficult for them to make ap?propriate control strategies.Therefore，it is impor?tant to recognize the similar weather scenarios（SWSs）in terminal area so as to help the control?lers clearly understand the real-time status and make strategies according to the historical similar scenari?os and strategies.

To recognize the similar scenarios in air traffic，in recent years，scholars have carried out a lot of re?searches and achieved some results［1-4］.The existing recognition methods of similar traffic scenarios are mostly based on numerical weather data，rather than the existing weather images.Therefore，the recognition accuracies of these methods are mainly dependent on the features of the numerical weather data extracted from weather radar monitoring data.With the development of deep learning technology and its great success in image recognition，it pro?vides us a new idea for the recognition of similar weather scenarios in terminal area with the abun?dance of weather images.Considering the weather images have no labels，in this paper，we attempt to take advantage of the contrastive learning in learning feature representation and semantic embedding，and propose a recognition model for similar weather sce?nario based on contrastive learning（SWS-CL）.In this method，the unlabeled convective weather im?ages are the input，and a data augmentation method is designed to improve the number and quality of the images.Then，the pre-trained SWS-CL model is obtained by calculating the loss function，which is designed to minimize the distance between the an?chor and positive samples，and maximize the dis?tance between the anchor and the negative samples at the same time in the latent space.After that，the pre-trained SWS-CL model is fine-tuned with a few labeled samples to improve the recognition accura?cy.Finally，the performance of the proposed SWSCL method is validated on the real convective weather image dataset.The contributions of this pa?per can be summarized as follows：

（1）A data augmentation is designed for con?vective weather images and its effectiveness is veri?fied.

（2）SWS-CL is proposed to improve the recog?nition accuracy of similar weather scenarios in termi?nal area.

（3）The proposed SWS-CL model is verified to achieve a better recognition accuracy on the real weather image dataset and has good performance on low-labeled images.

1 Related Works

In recent years，scholars have carried out a lot of researches on similar traffic scenarios and ob?tained some meaningful results.Liu et al.［1］used a semi-supervised learning algorithm to measure the similarity of each weather variable in the airport weather data based on some similarity relationships given in advance，analyzed airport runway accep?tance rates and runway configuration results under similar weather conditions，and then made control decision for a given date according to the decisions made on the similar historic days.Chen et al.［2］per?formed cluster analysis on the traffic scenarios under convective weather，and analyzed the similarity of the airport arrival rates，departure rates and control strategies with unbalanced capacity and flow under different weather.Xie et al.［3］measured the similari?ty of the air traffic data and selected similar spatialtemporal data to search the similar scenarios.Schelling et al.［4］used the dynamic time warping al?gorithm to calculate 13 similarity matrices，complet?ed the division of flight trajectories under the differ?ent weather conditions，and then gave the route se?lection strategies under similar weather.Hu et al.［5］used Euclidean distance and DTW to measure the similarity of the discrete features and time series fea?tures of busy sector，and input it to the spectral clus?tering model to identify the similar operation scenes.Chen et al.［6］proposed an active support vector ma?chine metric learning algorithm to measure and iden?tify the similar traffic scenes，in which a semi-super?vised method can be used in cases without sample la?bels.

The above methods are all applied to the nu?merical weather data converted from the original convective weather images with large information loss，so the accuracies of these methods are limited.So far，research on the recognition of similar traffic scenarios directly based on convective weather imag?es is rare.Deep learning technology can get deep feature extraction on images and has made great suc?cess in the field of computer vision，which inspires us to solve the recognition of SWSs with the images directly.However，the success of deep learning al?gorithms is very dependent on the labels of images，which is scarce in our convective weather image da?taset.To avoid the requirement for labels，a com?mon practice is to use data augmentation methods，such as rotation，cropping，etc.，to obtain augment?ed samples of anchors as positive samples.In the same batch of training data，other samples are re?gard as negative samples.Contrastive learning is proposed to minimize the distance between the an?chor and positive samples，and maximize the dis?tance between the anchor and the negative samples in the latent space by its loss function.By this way，contrastive learning can implement some machine learning tasks by using the characteristics of the im?ages without label information.

A simple framework for contrastive learning of visual representations（SimCLRS）［7］was proposed in 2020 and then widely used in computer vision.SimCLR learns visual representations by maximiz?ing the agreement between differently augmented views of the same example via a contrastive loss in the latent space.It proves that data augmentation plays a critical role in prediction task and learnable nonlinear transformations can improve the quality of samples.Later，Cai et al.［8］used SimCLR to create an multi-modality fusion framework for glaucoma grading.She et al.［9］applied SimCLR to create a multi-taskself-supervised framework to extract se?mantic information from synthetic data and make vi?sual representations.Ayush et al.［10］provided a selfsupervised learning framework based on SimCLR for remote sensing data，which can shorten the gap between self-supervised and supervised learning on image classification，object detection and semantic segmentation.Agastya et al.［11］applied SimCLR to optical remote sensing data and proved that the de?tection precision was nine times better than the tradi?tional supervised learning methods.Vu et al.［12］in?troduced SimCLR to classify diseases based on Xray images.

Inspired by existing works，in this paper，we try to build a SWS recognition model based on the contrastive learning framework SimCLR to make full use of the unlabeled data and improve the recog?nition accuracy.

2 Method

The recognition of SWS is essentially a classifi?cation task based on convective weather images.Fig.1 illustrates the framework of SWS?CL.We first design a data augmentation method for the con?vective weather image.Then，the augmented con?vective weather images are used as the input of the contrastive learning process to extract high-quality feature representations to complete the pre-training SWS recognition model.Finally，a few labeled con?vective weather images are used to supervise the fine-tuning of the obtained pre-training model to im?prove the accuracy of SWS recognition.

Fig.1 Framework of SWS-CL

2.1 Augmentation of convective weather images

The weather images are collected in real time by authoritative weather center with a certain fre?quency.Each image accurately depicts the intensity and distribution of convective weather in a certain airspace，and different convective weather affects the traffic flow in the terminal area at different de?grees.Usually，severe convective weather will have a lethal effect on traffic flow which is the scenario needs to be focused.However，since severe convec?tive weather only occurs occasionally，there are not enough typical images in the samples and most sam?ples are of no convective weather or mild convective weather.Therefore，it is necessary to design a suit?able data augmentation method to expand the train?ing dataset artificially by generating more equivalent samples from the limited samples.Furthermore，since the core idea of contrastive learning is to short?en the distance between positive samples and anchor samples，and reduce the distance between negative samples and anchor samples in the embedding space，the quality of the selected positive and nega?tive samples directly determines the performance of the whole model.However，the samples have no la?bels，and it is difficult to build positive and negative sample pairs with unlabeled samples.

Common image transformations including color distortion，cropping and resizing，vertical flipping and horizontal flipping［13］，as shown in Fig.2.

Fig.2 Weather image transformations

In the original image（Fig.2（a）），red，yellow and green represent severe，moderate and mild con?vective weather correspondingly.Color distortion（Jitter）（Fig.2（b））may change the contrast and brightness of the images，which is meaningless for similar weather recognition.Cropping and resizing（Fig.2（c））may cut off a part of the severe convec?tive weather which may significantly affect the rec?ognition result.Flip operations（Figs.2（d，e））will change the direction of convective weather，which is also important for air traffic control.Color distortion（Fig.2（f））coverts the color image to grayscale，which may weaken the weather information.

After several trials，we find that the image we get after rotating the original image with a slight ran?dom angle between 1° and 20° is similar to the origi?nal image，which is helpful for SWS recognition.Since only severe convective weather will affect air traffic，we can only keep the red areas in the image and convert it to Hue saturation value（HSV）for?mat with Hue=［0—10］∪［156—180］，Saturation=［43—255］，Value=［46—255］，which can enable the model to learn more patterns of severe convec?tive weather.Thus，we design a data augmentation method based on random rotation and keep severe convective weather for our recognition task，as shown in Fig.3.

Fig.3 Augmentation of weather images

From Fig.3 we can see that the batch size of an augmentation is 2，and for each image in a batch，a pair of augmented images are generated by the two transformations.One is a rotation of the original im?age with a random angle from 1° to 20°，and the oth?er is keeping the red parts of the original image and converting it to some HSV values.Specifically，since we consider each sample in a batch belongs to an unknown class，for the original Sample 1，the two samples generated from itself in Pair 1 are its positive sample pairs and the two samples generated from Sample 2 in Pair 2 is its negative sample pairs.So as to original Sample 2.By this way，we cannot only double the sample size in a batch，but also get the positive sample pairs and negative sample pairs for each original sample.It should be noted that for the two samples in Pair 1，if one is set as the anchor point，the other is its positive sample，and the two samples in Pair 2 are its negative samples.There?fore，data augmentation prepares the dataset for comparative learning.

2.2 Definition of contrastive loss

SWS-CL attempts to learn a mapping func?tion，f：x→z∈Rd，in an unsupervised way.Then the augmented weather imagesandcan get their semantic representationzi=Net(xi) andzj=Net(xj) through the neural network.Here，is the anchor sample andis the positive sample.In multi-classification problem，ziandzjare both onehot vectors and the cosine similarity is used to mea?sure their distance as

It can be concluded that the closerziis tozjare，the closer sim(zi，zj) is to 1.For a given sam?ple，the contrastive loss function is defined as

where sim(zi，zj) is the similarity of positive sample pair and sim(zi，zj) the similarity of negative sample pair.Specifically，we require that the similarity sim(zi，zj) of positive samples should be as large as possible，and the similarity sim(zi，zk) of negative samples should be as small as possible.The temper?ature coefficientτis defined to control the penalty strength on hard negative samples.Specifically，contrastive loss with small temperature coefficient tends to penalize much more on the hardest negative samples so that the local structure of each sample tends to be more separated，and the embedding dis?tribution is likely to be more uniform.By calculating the contrastive loss for all pairs of positive samples includingin a batch，key fea?tures with discriminative representation capability are extracted from the images with the aim to mini?mize the distance between the anchorand positive sampleand maximize the distance between the anchor and the negative samplesin the latent space.

The total loss for a batch is defined as

whereNis the batch size.By calculating the loss in each batch，the pre-trained SWS-CL model can be obtained on the unlabeled image dataset.

2.3 Pre?training of SWS?CL

The main process of the pre-training of SWSCL model is described as in Algorithm 1.

Algorithm 1Pre?training of SWS?CL

Input：DatasetX；Batch sizeN；Constantτ

Output：Encoder networkm(?)

（1）While SWS?CL is not converged do

（4）Extract representation with Base encod?erm(?)

（5）Getz2k-1，z2kwith Projection headn(?)

（6）For alli∈{1，…，2N}andj∈{1，…，2N}

Calculatel()based on Eq.（1）

CalculateLbased on Eq.（2）

Update networksm(?)andn(?)

（7）End while

2.4 Supervised fine?tuning of SWS?CL

In this section，the pre-trained SWS-CL model is fine-tuned by using a few labeled convective weather images to re-train the model.As shown in Fig.4，in the supervised fine-tuning step，we trans?fer parameters from the pre-trained model and add an output layer to realize the network structure and parameter sharing between unlabeled data and la?beled data.The pre-trained SWS-CL model can au?tomatically extract deep features，but the weights of the output layer of SWS-CL model are randomly ini?tialized.If these random values are allowed to be backpropagated to the entire network through gradi?ent updates，this may destroy previously trained weights.To avoid this problem，we freeze all layers of the main body of the network and only train the output layer on the labeled dataset.The training da?ta is forward propagated to the network as before，but the backpropagation stops in the output layer.It can significantly reduce the risk of overfitting in downstream model training stage，reduce the work?load of parameter tuning and obtain recognition re?sults more quickly.

3 Experiments and Discussion

In this section，the performance of SWS-CL method is verified by the comparative experiments.We introduce the image dataset and experiments set?ting in detail at first.The experiments have two parts.The first part mainly discovers the impact of the proposed data augmentation method on the mod?el.In the second part，the impacts of the hyperpa?rameters of SWS-CL are studied.

3.1 Dataset

The dataset we use in our experiments is the weather avoidance field（WAF）data of the Guang?zhou Baiyun International Airport terminal area col?lected from 2018 to 2019 with 10 min interval.There are 80 000 unlabeled and 4 000 labeled con?vective weather images of the terminal area.80 000 unlabeled and 2 000 labeled images are used for training.2 000 labeled images are used for testing.The size of the image is 1 250×1 250.There are three types of labels，SWS-1，SWS-2，SWS-3，which are evaluated by the experts of air traffic con?trol，as shown in Fig.5.It can be seen that label SWS-1，SWS-2 and SWS-3 represent the mild，moderately and severe convective weather scenarios correspondingly.

Fig.5 Labels of convective weather images

3.2 Setting of experiments

Five experiments are carried out to verify effec?tiveness of the proposed data augmentation method and SWS-CL model.In all the experiments，ResNet-50［14］is used as the embedding network for contrastive learning.

ResNet?50［14］：Training ResNet-50 on the orig?inal labeled images to get a recognition accuracy as the baseline.

ResNet?50+our augmentation：Training ResNet-50 on the labeled images augmented by our augmentation method.

Pre?trained SWS?CL+common augmenta?tions：Training the pre-trained SWS-CL model（ResNet-50+CL）on the unlabeled images aug?mented by common augmentation methods.

Pre?trained SWS?CL+our augmentation：Training the pre-trained SWS-CL model on the un?labeled images augmented by our augmentation method.

Fine?tuned SWS?CL+our augmentation：Retraining the pre-trained SWS-CL model on a few la?beled images augmented by our augmentation meth?od.

The experiments are performed on Python and TensorFlow 2.0 framework.The batch size of the model is 525，the epoch is 300，and the tempera?ture coefficient is 0.05.The Adam optimizer is used and the size of the input image is 224×224.

3.3 Experimental results and discussion

3.3.1 Performance comparison

The recognition accuracies of all five experi?ments are shown in Table 1.

Table 1 Results of comparative experiments

It can be seen that the accuracy of ResNet-50 is 71.45%，and the accuracy of ResNet-50+our aug?mentation is increased by 4.11%，which proves that our proposed data augmentation method has a posi?tive effect on the model.In addition，pre-trained SWS-CL+our augmentation improves the perfor?mance by 7.74% compared with the common aug?mentation，which also shows that our augmentation method still has better performance in pre-trained SWS-CL.Fine-tuned SWS-CL+our augmenta?tion improves the performance of ResNet-50 by 12.31%.Compared with pre-trained SWS-CL +our augmentation，the supervised fine-tuning of SWS-CL with a few labeled images improves the ac?curacy by 18.33%，which verifies that our method has obvious advantages on the dataset has few labels.

3.3.2 Impacts of parameters

There are three key hyperparameters，epoch，batch size and temperature coefficient，in the pro?posed SWS-CL model.In this section，we will study their impacts on the performance of pre-trained SWS-CL with all the settings the same as those in Section 3.2，except the focused hyperparameter.

3.3.3 Epoch

Epoch means the iteration times of training a deep learning algorithm.Fig.6 and Fig.7 show the effect of different epoch on the performance of the pre-trained SWS-CL model.From Fig.6，we can find that the accuracy of pretrained SWS-CL model increases with the epoch size.In Fig.7，as the epoch increases，the loss becomes lower.More epochs mean more negative samples can be used to improve model performance.If there are 20 consecutive in?creases in the loss value，the training can be stopped.

Fig.6 Impact of epoch size on accuracy

Fig.7 Impact of epoch size on loss

3.3.4 Batch size

Batch size means the number of images used in each iteration of training.It can be seen from Fig.8 that the larger the batch size，the higher the model accuracy.When the batch size is 525，the accuracy reaches the highest 66.44%.A larger batch size means that more negative samples can be used to ac?celerate the convergence and improve the model per?formance.At a given batch size，when the accuracy no longer improves after 20 consecutive iterations，it can be considered that the highest accuracy has been reached，and the training at this batch size can be stopped.

Fig.8 Impact of batch size on accuracy

3.3.5 Temperature coefficient

The temperature coefficient means how much the contrastive loss penalizes the hard negative sam?ples.The larger the temperature coefficient，the smaller the penalty for hard negative samples.The smaller the temperature coefficient，the larger gradi?ent the hard negative samples will get to separate them from the positive samples.The impact of tem?perature coefficient on pre-trained SWS-CL model is shown in Table 2.It can be found that the best temperature coefficient is 0.05.

Table 2 Impact of temperature coefficient on accuracy

4 Conclusions

The contrastive learning technology is applied for the recognition of SWS in terminal area.A data augmentation method for convective weather imag?es is designed to generate more useful images to be used as the input of the following learning.A con?trastive loss is defined to extract high-quality fea?tures to train the pre-trained SWS-CL model on the augmented and unlabeled weather images.Finally，the pre-trained SWS-CL model is supervised finetuned with a few labeled images to improve the rec?ognition accuracy of SWS.The results of five com?parative experiments demonstrate that the proposed convective weather image augmentation method can effectively improve the quality of the dataset，and the proposed supervised fine-tuned SWS-CL model can achieve satisfactory recognition accuracy of SWS with few labeled samples.Based on our pro?posed SWS-CL model，more similar meteorologi?cal scenarios can be classified，and then control strategies under similar scenarios can be extracted to help controllers quickly make control decisions for the current scenario.