Feature Selection with Optimal Stacked Sparse Autoencoder for Data Mining

Manar Ahmed Hamza,Siwar Ben Haj Hassine,Ibrahim Abunadi,Fahd N.Al-Wesabi,4,Hadeel Alsolai,Anwer Mustafa Hilal,Ishfaq Yaseen and Abdelwahed Motwakel

1Department of Computer and Self Development,Preparatory Year Deanship,Prince Sattam bin Abdulaziz University,Al-Kharj,16278,Saudi Arabia

2Department of Computer Science,College of Science and Arts at Mahayil,King Khalid University,Muhayel Aseer,62529,Saudi Arabia

3Department of Information Systems,Prince Sultan University,Riyadh,11586,Saudi Arabia

4Faculty of Computer and IT,Sana’a University,Sana’a,61101,Yemen

5Department of Information Systems,College of Computer and Information Sciences,Princess Nourah Bint Abdulrahman University,Saudi Arabia

Abstract: Data mining in the educational field can be used to optimize the teaching and learning performance among the students.The recently developed machine learning(ML)and deep learning(DL)approaches can be utilized to mine the data effectively.This study proposes an Improved Sailfish Optimizer-based Feature Selection with Optimal Stacked Sparse Autoencoder(ISOFS-OSSAE)for data mining and pattern recognition in the educational sector.The proposed ISOFS-OSSAE model aims to mine the educational data and derive decisions based on the feature selection and classification process.Moreover,the ISOFS-OSSAE model involves the design of the ISOFS technique to choose an optimal subset of features.Moreover,the swallow swarm optimization (SSO) with the SSAE model is derived to perform the classification process.To showcase the enhanced outcomes of the ISOFSOSSAE model,a wide range of experiments were taken place on a benchmark dataset from the University of California Irvine (UCI) Machine Learning Repository.The simulation results pointed out the improved classification performance of the ISOFS-OSSAE model over the recent state of art approaches interms of different performance measures.

Keywords: Data mining;pattern recognition;feature selection;data classification;SSAE model

1 Introduction

Data mining(DM)is an approach of extracting hidden prediction data from the larger database;also,it is one of the modern technologies having greater potential to assist institutions/Universities to focus on the fundamental data in their data warehouses[1].The DM tool predicts behaviors and future trends,enabling institutions to make knowledge-driven,proactive decisions.The automatic,prospective analysis provided by the DM tool develops the analysis of historical events offered by retrospective tool typical of decision support system.The DM tool is capable of answering institution questions that usually take a lot of time to solve[2].Nowadays,several industries utilize the DM tool to prepare marketing strategies and to make decisions towards target segmented customers for achieving their goals.However,the various university ignored practicing DM methods.The DM Application in the field of education is a new tendency in the globally competitive business.Understanding the DM application,terms,tasks,and techniques are the basis for the development of DM tools in educational sectors.Hence,it is necessary to examine the function of DM in educational sectors[3].

Educational Data Mining(EDM)is the application of the DM system on education information.EDM aims to study this information and solve education research problems.EDM acts toward the development of modern technology to examine the education information,and utilize DM tools for better understanding student learning environments [4-6].The EDM method converts transform raw data coming from the education system into helpful data that might have a greater impact on education practice and research.The ever-growing technologies in the education system have made a great number of data available.EDM offers a clearer picture of learners and their learning processes[7]and provides a significant amount of relevant information.It employs the DM technique to examine education information and resolve education problems.Like other DM methodologies extraction process,EDM extract novel,interesting,interpretable,and useful data from the education system.But EDM is specially designed for using different kinds of information in the education system[8].Then,this technique is utilized for enhancing the knowledge about the settings,educational phenomena,and students where they can learn[9].The development of computational methods that integrate data and theory would assist to improve the quality of teaching and learning(T&L)procedures.

EDM research studies several fields,involving specific learning strategies from computer-adoptive testing (as well as testing at a larger scale),education software,computer-assisted collaborative learning,and the factors i.e.,related to the non-retention/student failure in courses [10].Several critical areas include the development of the student system;applications of the EDM method have been in examining pedagogical supports(in learning software,and other fields,namely collaborative learning behavior) and improving/determining the method of a domain knowledge structure.There has been growing interest in the field of an EDM system.This newly emerging field,named EDM,is a concern with the development of techniques that determine knowledge from data originating from the education environment.EDM system employs several methods,for example,K-nearest neighbor (NN),Decision Trees (DT),Na?ve Bayes (NB),Neural Networks (NN),etc.analysis,and the Prediction of student performances is an important building block in education environments.academic performance of students is a critical factor in creating their future.Student Academic performance isn’t a result of only one determining factor as well it depends heavily on socio-economic,personal,and psychological factors.

Serik et al.[11]suggest the incorporation of techniques like data analysis and artificial intelligence(AI)methods,with learning management systems to enhance learning.These goals are defined in novel normality which tries to find a strong education system,where particular activity can be performed in an online system,amongst technique allows the student to have a virtual assistant to assist them in their learning.In[12],a novel predictive method to evaluate student performances in academics has been designed based on clustering and classification systems and tested in a real-time manner with the student’s database of several academic fields of high education institutions in Kerala,India.The results show that the hybrid method integrating classification and clustering methods yields outcomes i.e.,much greater interms of attaining accuracy in prediction of student academic performances.

Francis et al.[13]aimed at finding fast,slow and average learners amongst students and displays it by prediction DM tool with classification-based models.Mhetre et al.[14]determine the student performances with various classification methods and discover the better one that produces optimum outcomes.Education Database is gathered from a Saudi University database.The database is preprocessed for filtering duplicate records;missing areas are recognized and filled with certain information.DL methods such as DNN and DM methods like random forest(RF),support vector machine(SVM),DT,and NB are applied to the dataset with Rapid Miner and Weka methods.

Aslam et al.[15]separate the significance of 2 blooming techniques,ML and Blockchain,in the education domain.Blockchain technique,with data immutability as its major benefits,was employed in the miscellaneous field for security factors.It is utilized for storing securely the achievement/degree certificates.This data will be included by the university/college to the blockchain technology,that is shared/accessed by the students via online resume with employers.This method is highly secured since there is no need to worry about the loss of data/modification to the institution.Shah et al.[16]gathered several samples of distinct kinds of student attributes with the survey forms which are related to academic performances.Then,select a few significant features with distinct feature extraction methods.Next,employed a few ML methods to that pre-processed dataset.Malhotra et al.[17]examine the educational quality that is tightened closely with the sustainable developmental objectives.The performance of the method has generated excessive information that should be suitably processed to attain useful data which is highly beneficial for future planning and development.Student grade and mark predictions from their historic academic data are useful and popular applications in the EDM system,hence it become a useful data provider that is employed in various forms to enhance the educational quality in the country.The classification method would forecast the grades whereas the regression method would forecast the mark,lastly,the outcomes are attained from both the models were investigated.

This study proposes an Improved Sailfish Optimizer-based Feature Selection with Optimal Stacked Sparse Autoencoder (ISOFS-OSSAE) for data mining and pattern recognition in the educational sector.The proposed ISOFS-OSSAE model aims to mine the educational data and derive decisions based on the feature selection and classification process.Moreover,the ISOFS-OSSAE model involves the design of the ISOFS technique to choose an optimal subset of features.Moreover,the swallow swarm optimization(SSO)with the SSAE model is derived to perform the classification process.To showcase the enhanced outcomes of the ISOFS-OSSAE model,a wide range of experiments were taken place on a benchmark dataset from the UCI repository.

2 The Proposed Model

In this work,a novel ISOFS-OSSAE technique is derived aims to mine the educational data and derive decisions based on the feature selection and classification process.The proposed ISOFS-OSSAE model involves the design of the ISOFS technique to choose an optimal subset of features.Followed by,the OSSAE model-based classifier is derived in which the parameter tuning of the SSAE model is done by the use of the SSO algorithm.

2.1 Design of ISOFS Technique

The SFO approach is depending upon the behavior of sailfishes.The sailfish population can be determined by a candidate solution in the SFO method [18].The population in solution space is arbitrarily created.The location matrix of initiated sailfish has been demonstrated by:

In whichmsignifies the quantity of sailfish in sailfish population,ddenotes the dimensional parameter,characterizes the value ofjthvariable ofithsailfish,tindicates the number of existing iterations.The matrix afterward the initiation of the sardine location has been demonstrated by:

whereasnindicates the number of sardines in the sardine population,means the value ofjthparameter ofithsardine.

To estimate the quality of sardines and sailfish,the fitness of sardine and sailfish solution can be resolved andFrepresent fitness function as well as stored in matrix format.The fitness matrix of sailfish can be expressed as:

Now,signifies the fitness matrix of sailfish,mimplies the amount of sailfish,F(t)irepresent the fitness value of theithsailfish.It can be formulated by:

Here,F(t)Sdenotes the sardine fitness matrix,nsignifies the quantity of sardines,andF(τ)iindicates the fitness value ofithsardine.The hunting nature of sailfish alters its location based on the location of other sailfish near the sardines.The equation to update the location of sailfish is given in the following:

whereindicates the elite sailfish in the t-l iteration,represent the sardine i.e.,severely injured in t-l iteration,shows theithsailfish in the t-l iteration,Xi(t)refers to theithsailfish intiteration,rand denotes an arbitrary number within[0,1],λiindicates the upgrade coefficient:

Letbe the amount of sailfish intiteration andsignifies the amount of sardines intiteration.

To prevent from sailfish attack,the sardine would consider the location of the elite sailfish and the attack power of sailfish in all the iterations and upgrade the location.The location upgrade equation of sardines is expressed as:

whereasindicates theithsardine in t-l iteration,represent theithsardine intiteration,Q(t-1)signifies the attack power of sailfish in t-l iteration,A&ξdenotes attack power coefficient,within 4 and 0.001,tindicates the number of present iterations,the amount of upgrade sardinesγand the amount of parameterηbased on the attack power of the sailfish can be represented by:

NowMs(t)signifies the quantity of sardines in=titeration anddi(t)characterizes the amount ofithsardine parameter in thetiteration.

Once the fitness of sardine is minimum when compared to the sailfish,the location of the captured sardine is substituted with the location of the sailfish:

LetXi(t)be the location of sailfish intiteration andXi(t)represent the location of sardine intiteration.

Theuweight inertia is presented into the alternate attack and pursuit procedure of sailfish[19],and the local searching capacity of sardines and sailfish is improved using weight inertiau.The location upgrade equation can be given below:

In the equation,rand denotes an arbitrary number within [0,1]The weight inertiaucan be expressed by:

Nowumin&umaxrepresents the lower and upper bounds of weight inertiau.umin=0.4,umax=0.9.tas well asTrepresents the present number of iterations and the maximal amount of iterations.The steps involved in the SFO algorithm are given in Fig.1.

Figure 1:Steps involved in SFO algorithm

The ISOFS technique is applied to discover the feature area effectively and generate a proper set of features.The feature selection can be considered as a multiobjective optimization problem as it requires fulfilling distinct aims to get optimum solutions that reduce the features and improve the classification performance.Therefore,a fitness function(FF)is derived to obtain solutions for attaining a balance amongst two objectives as defined below:

ΔR(D)denotes the error level of the classifier,|Y| represents the number of features,and |T|denotes the total feature count that exists in the dataset.Here,αindicates a parameter ∈[0,1]related to the weight of error rate of classifier,andβ=1-αcharacterizes the standing of reduced features.

2.2 Design of OSSAE Based Classification Model

The reduced feature subsets are fed into the OSSAE model to carry out the classification process.The auto encoder (AE) approach is an asymmetrical NN that extracts the features with minimal recreation error.Li et al.[20]projected a perspective,termed ‘pre-train,that separates a complex network to stack sub-networks.The training failure could be prevented since the network parameter of all the layers are allocated certain values,instead of arbitrary initiation.Nevertheless,the stacked subnetwork provides generalization capacity and low training efficacy because of the easiness of the singlehidden neuron and the complexity in the selection of variables.To resolve these above-mentioned problems,the SSAE method is presented based on 2-phase networks with 5 hidden layers in all the networks.The overall framework of the SSAE method is demonstrated in Fig.2.Initially,theXinput layer is mapped to ah3hidden nueron(termed low-level hidden layer).Next,h3is mapped back to the recreated layer,Xrec.Then,theh3hidden feature is converted into the input parameter for acquiringh6(high-level hidden layer)[21].

Figure 2:Structure of SSAE

In the SSAE,the feature learning method follows a sequence of processes,like convolution,denoising,pooling,activation,and batch normalization (BN).A summary of this operation can be thoroughly explained in the following:Denoising to attain the strong and illustrative learning feature of the flame image,a denoising AE learning method is utilized to add distinct noises with the input signal.The white Gaussian noises are taken into account,such as the corrupted versionXnis attained by a fixed corruption ratio to the inputX:

A rectified linear unit (ReLU) is utilized as an activation function of the hiddenneuron γ,determined by:

The ReLU is an unsaturated piecewise linear function,i.e.,quicker when compared to the saturated non-linear functions,like TanH and Sigmoid.Particularly,the Sigmoid function is employed in the 3rd decoders for ensuring the intensity range of the recreated layerXrecis reliable with the input layer X,as:

Pooling and upsampling:The pooling function is performed for reducing the parameter of the network.Here,P(r×r+t)represents the pooling function which condenses the feature map through choosing a maximal value with a step oftand ar×rconversion kernel.The pooling function helps improve the translation invariance as:

In which&represents the value of position(i,j)in thekthfeature map of outputs and inputs[22].Considersij(i∈(1,E),j∈(1,F))signifies the activation of hidden layerj,in whichEdenotes the number of images in the trained data set andFdenotes the number of layers in the hidden neuron.Next,the average activation of all the hidden neuronsjare evaluated as:

whereaspjprojected to be closer to 0,which implies the neuron of the hidden neuron is generally inactive”.APpenaltythe penalty term is added to the loss function,that penalizepjonce it significantly deviates from theptargetsparse target.ThePpenaltypenalty term can be determined by:

LetKL(ptarget||pj)be the KullbackLeibler divergence(KL divergence)that performs as a sparsity limitation:

Whenptarget=pj,KL(ptarget||pj)=0.Or else,theKL(ptarget||pj)monotonically increased aspjdeviate fromptarget.

For optimally adjusting the parameters involved in the SSAE model,the SSO algorithm is utilized.

SSO approach was inspired by the collaboration of swallow and the interface between flock members has accomplished optimal results[22,23].This method was proposed a meta-heuristic model based on unique features of swallow comprised of intelligent social relation,fast flight,and hunting skill.Now,this approach is similar to the particle swarm optimization(PSO)method however it could be particular characteristics that couldn’t be initiated in common methods which contain:Leader Particles(li),Explorer Particles(ej),and Aimless Particles(oi),which has certain responsibilities in the group.The particleejis accountable for searching the problem space.It performed these exploring performances in the control of the number of variables.It can be expressed by

Eq.(20)exhibits the velocity vector parameter in the path of the global leader.

Eqs.(21)-(12) evaluates the acceleration coefficient parameter(αHL)which straightforwardly affects individual knowledge of each particle.

Eqs.(23)and(24)calculates the acceleration coefficient parameter(βHL)which straightforwardly affects the integrated knowledge of each particle.Both acceleration coefficients are quantified allows placing each particle with global leaders and optimal individual knowledge.Theoiparticle utilizes the succeeding Eq.(25)for arbitrary motions:

In the SSO method,there are 2 types of leaders:global and local leaders.The particle is separated into groups.The particle in each group is often similar.Next,an optimal particles in each group is chosen and termed as a local leader.After,an optimal particle amongst the local leaders is selected and termed as a global leader.The particle modification converges and way depends on the place of this article.

3 Results and Discussion

The experimental results analysis of the ISOFS-OSSAE technique takes place using the benchmark dataset from the UCI repository[24].It includes a total of 649 samples with 33 attributes and 2 class labels.The results are investigated under varying dimensions.

Tab.1 and Fig.3 demonstrate the best cost and number of chosen features offered by the ISOFS with other FS methods.The results portrayed that the Information gain and CFSSubsetEval techniques have attained poor performance with the maximum best cost of 0.386920 and 0.366000 respectively.Followed by,the genetic algorithm(GA)and PSO models have obtained moderate best cost of 0.165283 and 0.183638 respectively.In line with this,the ant colony optimization (ACO)algorithm has accomplished near-optimal best cost of 0.030509.However,the proposed ISOFS technique has accomplished superior outcomes with the least good cost of 0.02156.

Table 1:Best cost analysis of different FS models

Table 1:Continued

Figure 3:Best cost analysis of ISOFS with other techniques

Fig.4 portrays the confusion matrices generated by the ISOFS-OSSAE technique under five test runs.On test run-1,the ISOFS-OSSAE technique has identified 543 instances into class 0 and 94 instances into class 1.Besides,on test run-2,the ISOFS-OSSAE technique has identified 541 instances into class 0 and 91 instances into class 1.Moreover,on test run-3,the ISOFS-OSSAE technique has identified 542 instances into class 0 and 90 instances into class 1.Likewise,on test run-4,the ISOFSOSSAE technique has identified 543 instances into class 0 and 94 instances into class 1.At last,on test run-5,the ISOFS-OSSAE technique has identified 541 instances into class 0 and 93 instances into class 1.The values present in the confusion matrix are transformed in terms of TP,TN,FP,and FN in Tab.2.

Tab.3 provides a detailed overall classification results analysis of the ISOFS-OSSAE technique under five test runs.The table values denoted that the ISOFS-OSSAE technique has gained effective classifier results.For instance,with run-1,the ISOFS-OSSAE technique has attainedprecn,recal,Fmeasure,accy,andkappaof 0.9891,0.9891,0.9891,0.9815,and 0.8258 respectively.In line with,with run-2,the ISOFS-OSSAE technique has offeredprecn,recal,Fmeasure,accy,andkappaof 0.9836,0.9854,0.9845,0.9738,and 0.8122 respectively.Followed by,with run-3,the ISOFS-OSSAE technique has achievedprecn,recal,Fmeasure,accy,andkappaof 0.9819,0.9872,0.9846,0.9738,and 0.8223.Along with that,with run-4,the ISOFS-OSSAE technique has gainedprecn,recal,Fmeasure,accy,andkappaof 0.9891,0.9891,0.9891,0.9815,and 0.8258 respectively.Lastly,with run-5,the ISOFSOSSAE technique has attainedprecn,recal,Fmeasure,accy,andkappaof 0.9891,0.9891,0.9891,0.9815,and 0.8258 respectively.

Figure 4:Confusion matrices of ISOFS-OSSAE technique under five test runs

Table 2:Confusion matrix of ISOFS-OSSAE technique

Table 3:Overall classification results analysis of ISOFS-OSSAE technique

Fig.5 demonstrates the analysis of the average results of the ISOFS-OSSAE technique on the test dataset.The figure reported that the ISOFS-OSSAE technique has resulted in maximum averageprecn,recal,Fmeasure,accy,andkappaof 0.9862,0.9872,0.9867,0.9775,and 0.8207.

Figure 5:Average classification results analysis of ISOFS-OSSAE technique

Tab.4 offers a comprehensive comparative analysis of the ISOFS-OSSAE with recent techniques[25].Fig.6 demonstrates theprecn,recal,andFmeasureanalysis of the ISOFS-OSSAE technique on the test dataset.The figure reported that the random tree (RT) technique has accomplished least performance with the minimal values ofprecn,recal,andFmeasure.Followed by,the radial basis function network(RBFN)technique has attained slightly increased values ofprecn,recal,andFmeasure.Next to that,the multilayer perceptron (MLP) model has accomplished moderate values ofprecn,recal,andFmeasure.In line with,the ACO with logistic regression(ACO-LR),LR,RF,and DT models have tried to obtain reasonable values ofprecn,recal,andFmeasure.However,the proposed ISOFS-OSSAE technique has resulted in maximum classification performance with theprecn,recal,andFmeasureof 0.986,0.987,and 0.987.

Table 4:A comparison study of ISOFS-OSSAE technique with recent methods

Figure 6:Comparative precn,recal,and Fmeasure analysis of ISOFS-OSSAE technique

Fig.7 showcases theaccyanalysis of the ISOFS-OSSAE technique with recent approaches.The figure reported that the RBFN,MLP,and RT models have attained loweraccyof 0.895,0.897,and 0.877 respectively.Followed by,the DT and RF models have reached slightly improvedaccyof 0.914 and 0.926 respectively.Eventually,the LR and ACO-LR models have achieved reasonableaccyof 0.948 and 0.949 respectively.However,the proposed ISOFS-OSSAE technique has resulted in maximum outcome with theaccyof 0.978.

Figure 7:Comparative accy analysis of ISOFS-OSSAE technique

Fig.8 displays thekappaanalysis of the ISOFS-OSSAE technique with recent approaches.The figure stated that the RBFN,MLP,and RT models have reached inferiorkappaof 0.614,0.592,and 0.542 respectively.Also,the DT and RF models have got slightly enhancedkappaof 0.658 and 0.685 respectively.Ultimately,the LR and ACO-LR models have realized sensiblekappaof 0.792 and 0.769 respectively.However,the proposed ISOFS-OSSAE technique has resulted in supreme outcome with theaccyof 0.821.From the aforementioned figures and tables,it is ensured that the ISOFS-OSSAE technique is an effective tool for data mining and pattern recognition.

Figure 8:Comparative kappa analysis of ISOFS-OSSAE technique

4 Conclusion

In this study,a novel ISOFS-OSSAE technique is derived aims to mine the educational data and derive decisions based on the feature selection and classification process.The proposed ISOFS-OSSAE model involves the design of the ISOFS technique to choose an optimal subset of features.Followed by,the OSSAE model-based classifier is derived in which the parameter tuning of the SSAE model is done by the use of the SSO algorithm.To showcase the enhanced outcomes of the ISOFS-OSSAE model,a wide range of experiments were taken place on a benchmark dataset from the UCI repository.The simulation results pointed out the improved classification performance of the ISOFS-OSSAE model over the recent state of art approaches with the higher accuracy of 0.978.Therefore,the ISOFSOSSAE model can be utilized as an effective tool to mine data and recognize patterns.In the future,the classification performance can be improved by the utilization of clustering approaches.

Funding Statement:The authors extend their appreciation to the Deanship of Scientific Research at King Khalid University for funding this work under grant number(RGP 1/279/42).www.kku.edu.sa.The authors would like to acknowledge the support of Prince Sultan University for paying the Article Processing Charges(APC)of this publication.

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding the present study.