Relating brain structure images to personality characteristics using 3D convolution neural network

Lixian Cao| Yanchun Liang,2 | Wei Lv | Kaechang Park | Yasuhiro Miura |Yuki Shinomiya | Shinichi Yoshida

1Zhuhai Laboratory of Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education,Zhuhai College of Jilin University, Zhuhai,China

2Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education,College of Computer Science and Technology,Jilin University,Changchun,China

3School of Information &Research Institute,Kochi University of Technology,Kochi,Japan

Abstract The Keras deep learning framework is employed to study MRI brain data in a preliminary analysis of brain structure using a convolutional neural network.The results obtained are matched with the content of personality questionnaires. The Big Five personality traits provide easy differentiation for dividing personalities into different groups.Until now,the highest accuracy obtained from the results of personality prediction from the analysis of brain structure is about 70%.Although there is still no effective evidence to prove a clear relationship between brain structure and personality, the obtained results could prove helpful in understanding the basic relationship between brain structure and personality characteristics.

1 | INTRODUCTION

In recent years, the world's population has aged rapidly. MRI‐derived information about the brain is often used in health diagnoses. The spread of such medical examinations has allowed medical institutions to become more flexible. Many proposals for diagnosis support using image recognition technology and machine learning methods to perform the task of disease identification—for example, identifying tumors and bleeding. The purpose of health examinations is to detect and recognize risks and symptoms, but it is difficult to use conventional image recognition methods as an effective detector.On the other hand, it can be seen generally that small differences in health status, nature, and physique, as well as future predictions from those characteristics,have long‐term impacts on the brain. Although the shape is difficult to see, it is also reflected in the anatomical shape and structure of the brain such as size of a specific region in cortex or portion of a white matter of brain. Therefore, this study uses a machine learning method to construct a brain image as a discriminator of human personality characteristics as well as a discriminator of MRI image predictions about brain structure with experimental objects of different personalities.

In applications of machine learning and brain MRI,studies can be separated into two research areas, medical image recognition and neuroscience. In the medical research area,brain hemorrhage [1] is predicted using linear regression, decision trees, neural networks, and kernel spectral regression.Alzheimer disease is also a major research topic in classifying brain MRI images using machine learning such as support vector machines with low density separation [2]. To use the conventional machine learning algorithm to discriminate brain data, feature extraction is important. Although studies of natural image recognition can utilize conventional image features,such as SIFT local features or global histogram features,extracting effective features from brain MRI images is a difficult task because MRI lacks tools used in optical imaging such as charge coupled‐device image sensors (e.g. CCD imaging cameras).This fact motivates researchers to use learning‐based features that incorporate enormous amounts of data.The most widely used learning‐based feature is the deep convolutional neural network (CNN), and recent MRIresearchers have employed the three‐dimensional convolutional neural network (3DCNN) to predict disease using MRI images [3, 4]. Against this background, we employ the standard 3DCNN method to classify MRI images of brain structure based on human characteristics (the Big Five personality traits).

TA B L E 1 Big Five personality questionnaire

Because information provided by anatomical brain images is substantial as well as complex [5], employing ordinary feature‐engineering techniques is not effective,and the results are unsatisfactory. With the recent development of a neural network on the Keras framework, we can find and calculate complex information and relationships among two or more objects and use algorithms such as the 3DCNN to analyze brain structure more accurately.This experiment is conducted to explore the relationship between brain structure and human personality. Whether human character is present at birth or is acquired has always been a controversial topic. This experiment was set up to explore this point of view.

In past experiments,the computational cost of MRI image analysis was too large—ordinary computers were unable to perform such analysis. The advantage of the CNN is that neural networks can be built to effectively use ordinary computers for analysis. With its own computing power, this method allows us to use a simpler and faster way to obtain results when conducting experiments that use the same content. We select the CNN [6] because of its low cost and high speed.

This study's results can be used to provide technical support for the study of human personality in medical and psychology fields. Difference in gender should be considered [7],such as the higher number of non‐amygdaloid synapses on dendritic spines in the preoptic area of normal females versus the number for normal males [8]. If neural correlates of personality differ between males and females, then according to the results of [9], brain structure–personality relationships are highly dependent on gender. If this experiment becomes established, it could be used as a technical support in the future to investigate potential criminals and anti‐social personality disorders [10].

2 | RELATED WORKS

We employ the Big Five personality traits,which are often used to study human personality in psychology. We provide details in this section.

2.1 | The Big Five personality traits

In recent years, researchers have found that five traits cover nearly all aspects of personality descriptions and can be divided into the following five areas: openness, conscientiousness, extroversion, agreeableness, and neuroticism [11].The questionnaire divided the five personality traits into 10 items, as shown in Table 1. The personality characteristics of the experimenter object were obtained from the above fivepairs, each of which involves a single‐choice question with an answer value ranging from 1 to 7. Then, a calculation was performed to sort the 10 data in the questionnaire into corresponding values in the five major personality characteristics and obtain the experimenter's tendency for each of the five personality characteristics[11],as shown in Table 2.To reduce the relevance between questionnaire questions and reduce user orientation,we design the J,H,F,B,and I questions so they are the opposites of the target personality characteristics. In the final calculation, we subtract the score of the question from a constant to achieve a positive result.Since 8 is the mode in the experimental data, the constant is set to 8.

TA B L E 2 Conversion formulas for questionnaire and Big Five personality traits

2.2 | Prediction of personal property from brain structure

In the experiment, we first need to build a neural network to study MRI brain structure. The MRI image is analysed based on the method in reference[12].We also must load MRI brain images and select suitable parameter settings to analyze them.Based on reference[12],we obtain the parameters already used in the study about personal property using the brain structure.The CNN parameter settings greatly influence the results.Likewise, the analysis of different objects causes the parameters to change.Reference[13]provides excellent data about the selected CNN.

3 | PROPOSED METHOD

3.1 | Data acquisition

The original data set used in this research was obtained from a health clinic located in Japan (Kochi Kenshin Clinic). From 2016 to 2018, we asked patients who were receiving brain MRIs to check their health to attend our study, and 1933 voluntary participants submitted written informed consent.They completed the Big Five test after their MRI brain scans.

3.2 | MRI scan

We used a Hitachi ECHELON MRI scanner(1.5 T)placed in Kochi Kenshin Clinic and used 3DGEIR_SAG (three‐dimensional gradient echo with inversion recovery pulse sequence, sagittal slicing, T1‐weighted imaging).

The scan parameters were as follows:

Slice thickness: 1.2 mm

Spacing between slices: 1.2 mm

TE: 0.004 s

TR: 0.0092 s

TA B L E 3 Pearson correlation coefficient

Flip Angle: 8

Acquisition matrix: 192

3.3 | Preprocessing of MRI images

We converted the original images from the Digital Imaging and Communications in Medicine (DICOM) format to the Neuroimaging Informatics Technology Initiative (NIfTI)format. We did not employ other tasks such as coregistration,anatomical normalization, or smoothing, as we believed that brain detail information would be lost in the processes.

3.4 | Judgment of personality and MRI‐related standards

The standard employed to judge the two parameters is the Pearson correlation coefficient [14], which is used to measure the correlation between two variablesXandY(linear correlation), with a value that ranges between ?1 and 1. The value ranges typically used to define the strength of correlation between the two parameters is shown in Table 3.

3.5 | Task

What we are mainly studying is whether a connection exists between brain structure and personality. Our approach is to use MRI brain images to predict the experimenter's personality and then determine the accuracy of the prediction.If there is a strong connection between the two objects, the predicted result can be considered quite exact.

3.6 | Convolutional neural network model

F I G U R E 1 Convolutional neural network of this study

F I G U R E 2 Conscientiousness data distribution

Good prospects have been demonstrated for the CNN model in the medical field[15]. As studied here,the CNN comprises three blocks. The advantage of the CNN is that not every neuron in the CNN needs to connect with neurons in previous layers.In addition,by removing unimportant samples from the feature map, the number of parameters is reduced further,which is important because of the immense size of the experimental data set. The blocks are prepared in the Keras framework. For this case, we rebuild the full connection (FC)block. This model is trained only in the FC block. The three main blocks are used for loading MRI images. The model structure can be seen in Figure 1.The activation function used is RELU[16],and the optimizer used is Adam[17].In the early stage of our experiments, we applied the VGG16 structure to the program and found that the experimental time was severely lengthened and occupied more memory,resulting in extremely unsatisfactory efficiency. The structure used here is a CNN that is optimised, using the initial analysis of MRI brain data,for gender correlation as mentioned in the Introduction.

We have chosen to use the 3DCNN because the 3D model is widely used in the medical field, and 3D images provide more information than 2D images can provide. In fact, we conducted the two‐dimensional and the three‐dimensional experiments concurrently;because the MRI model of the brain is three‐dimensional, the two‐dimensional CNN models must be cut from the three‐dimensional model by slicing. Because that process is ineffective, we adopted the 3DCNN as our main research method, which we consider more efficient, on the whole,for the research[18]. The CNN structure is shown in Figure 1.

3.7 | Preparations for MRI data

Based on the study of references[19,20],we understand that it is possible to extract some relevant information from imperfect and noisy brain images in the standard way; however, we converted data only from the DICOM to the NIfTI format to maintain the original information from the brain MRI images.

Normally, brain data takes up too much memory. To save the computer's running memory and avoid memory leaks, the operation of putting the data set into theXtraining set is written into the judgment statement so that the data will be loaded only when the condition is true. That is, the program first judges the data in the personality traits questionnaire. If the condition is true, the program loads the corresponding encoded brain data and then puts it into training setX. A good explanation of the CNN's classification function is given in [21]. Here, we mainly compare conscientiousness and agreeableness. To make those specific features more obvious, we set them in the model only when the experimenter's conscientiousness score is more than 9 and agreeableness is less than 8, in which case the experimenter is judged to be conscientiousness; or when the experimenter's conscientiousness score is less than 8 and agreeableness is more than 9, in which case the experimenter is judged to be agreeable.

Therefore, a conditional judgment statement needs to be set up to distinguish the two situations. The reason for such data screening is that from the data distribution diagrams(Figures 2 and 3),it can be seen that a score of 8 accounts for the vast majority of the data and thus represents mostly data indicating no particular preference. Because personality characteristics are analyzed in this study's experiments, the ideal data become more obvious as the bias increases, so we finally decided to discard the data with a score of 8 and only process those data more specific to our chosen characteristics.Since characteristics are processed for each of the five major personality types,personalities should be independent;i.e.,the same person may have a positive tendency for five personality types at the same time or a negative tendency for five personality types at the same time.So only one of them is greater than 8, and the other is less than 8. This approach makes the characteristics of the data more obvious and easier to analyze.This way of analyzing the relationship between brain structure and neurohormones is based on reference [8].

F I G U R E 3 Agreeableness data distribution

F I G U R E 4 Conscientiousness data distribution

4 | EXPERIMENT

4.1 | Data set

4.1.1 | Introduction of data set

A good way to deal with MRI brain images was found in[22, 23]. In addition, because of the excessively high dimensionality of MRI data, we consider the method used in Reference[24].We used 1933 T1‐weighted MRI images in this study, with the image size reshaped to [195, 256, 256] using linear interpolation to adapt the 3DCNN classifier. Each side of conscientiousness and agreeableness is 195.If we do not set equal numbers here, the result will become unbalanced and prefer a larger number of parties.Better than 20%of the total data set is established as the test set,and the remaining 80%of the data set is used as the training set. We used whole‐brain image data that has been shown to be very useful [25]. We would also like to try using the same data type to obtain more information from the whole‐brain image data.A sample of the conscientiousness data distribution is shown in Figure 4.

4.1.2 | Data normalization approach

We must first deal with the data before we can input it as training data. Data normalization is often used in comparison and evaluation index processing. By removing the unit limitation of the data and converting it to a pure value, it is convenient to compare and weight indicators of different units or magnitudes.

F I G U R E 5 Accuracy results of the first experiment

TA B L E 4 Program environment

Here we use z‐score normalization,also known as standard deviation normalization. This method uses the mean and standard deviation of the original data to normalize the data.The processed data conforms to the standard normal distribution—that is, the distribution of the processed data has a mean of 0 and a standard deviation of 1. The formula is

whereμis the mean of all sample data andσis the standard deviation of all sample data [26].

4.2 | Evaluation method

In this experiment, we judge accuracy by the following formula:

where A means accuracy,P(positive)means conscientiousness,and N(negative)means agreeableness.TR means true positive,TN means true negative, FN means false negative, and FP means false positive.

4.3 | Experimental environment

Based on the various tests performed on the system during the foregoing process, it is recommended that the following configuration is adopted for the computer system when performing the experiment to ensure that the experiment can be run stably and efficiently. Due to the special nature of the program (the brain data is too large), this program should ideally be run on a workstation‐level computer. The program environment used in this study is shown in Table 4.

5 | RESULTS

In the experimental results, the training accuracy for training reached 1.0, and the loss dropped to 0. However, in actual operation, the result did not reach the expected index. The peak value of accuracy was 0.7662 and finally stabilized at about 0.6; the peak value of loss was 3.5 and finally stabilized near 3.0.

F I G U R E 6 Loss results of the first experiment

F I G U R E 7 Accuracy results of the second experiment

At present,the validation comes from the‘train_test_split'function,which is randomly selected from the data set.As the result of the validation set,val_acc is maintained at about 0.6;it is impossible to speculate about an inevitable connection between personality characteristics and brain structure. The actual experimental results are not ideal.It is presumed that no better parameter settings have been made for the neural network.In the future,program updates can be made from this perspective. We tried other optimizers and loss functions, but none of those results proved satisfactory.

The experimental results from three independent tests are shown in Figures 5–10.

6 | ANALYSIS OF EXPERIMENTS

Of the three experiments,the accuracy of the first experiment,whether for training or testing, is relatively sharp in the previous period,and the same is true.In the first experiment,the loss in the test is constantly rising, which leads to great instability factors in the experimental results.Thus,the CNN in this case does not generate program results that become increasingly better through training but instead makes the results worse. After finishing the first experiment, we analyzed the results. The poor results occurred because the parameters of the neural network model were not properly adjusted during the process.

F I G U R E 8 Loss results of the second experiment

F I G U R E 9 Accuracy results of the third experiment

F I G U R E 1 0 Loss results of the third experiment

The second experiment was performed after the parameters were modified.Because the parameters had been modified,the results of the second experiment were smoother than the results of the first experiment. It can be concluded that parameter modification has a very important impact on CNN training. However, in the actual test process for the second experiment, the loss continues to rise, and the experimental results are still not ideal.

In the third experiment, changes were made to the brain data,and gray and white matter pictures were used in place of the whole‐brain pictures.During the third experiment,the loss initially showed a downward trend but later rebounded; not only did the loss in the third experiment show a normal attenuation trend,but also its accuracy rate increased from 0.6 to 0.7.It can be inferred that the amount of information in the brain data is too large, resulting in poor CNN training results.In later work, the training data should be improved to enable training results that are more stable.

7 | CONCLUSIONS AND DISCUSSION

This paper makes a preliminary attempt to discuss the inevitability between the three‐dimensional structural model of the brain and personality traits, that is, to discuss whether it is possible to analyse the brain model to make personality predictions and to conduct a basis study on the exploration of potential psychological problems.The specific implementation is through deep learning analysis of personality traits that uses Big Five personality data to explore the correlation between two traits.In the course of the experiment,we use the z‐score to standardize the data and facilitate deep learning operations.By discarding the data with obscure features and obtaining data with more obvious tendencies, we draw a graph of the deep learning results and determine whether there is a clear relationship between the two by Pearson's correlation coefficient.

In the experiments, we applied deep learning and used brain data to conduct a preliminary analysis of brain structure through a CNN. The results obtained were matched with the content of the personality questionnaire,but no valid evidence has been drawn to prove the existence of the two issues. We need to clarify the relationship and make improvements in subsequent experiments.

The main innovations of this study are first, that the main body of the program uses a deep learning framework to train the neural network based on supervised learning.The program must be provided with an input and told what the expected output should be. If the neural network is generated in this process and the output result is incorrect,we should adjust the program's calculations. Overall, this method is more efficient,faster and easier to use than traditional analysis methods. The second innovation is the data‐loading method. Due to the uncertainty of the data size, when it is necessary to process a large volume of data with a larger memory requirement, the data cannot be processed normally, that is, all data is directly read and loaded. Therefore, loading data into the conditional filtering statement can tell the program to only load data that satisfy the conditions,which greatly reduces computer memory requirements when processing data and shortens waiting time when loading data.

It should be pointed out that the experimental results at this stage of the paper are not ideal, for two main reasons.First, the parameter setting of the neural network model may be unsatisfactory, which leads to insufficient training of the CNN and insufficient analysis of the brain MRI data map.Secondly,it may be that in the whole‐brain MRI data currently used, the brain parts related to personality are not shown or not obvious enough.We also find some related references like[27], which conducted experiments about the relationship between Alzheimer disease and hippocampus. The results show some significant evidence of correlation,which gives us a hint that some information may be hidden in specific areas.Therefore, a related study should be made that is deeper or more specific.

Additionally, a specific result was obtained for the relationship between brain gray matter MRI images and age [28],which was shown in detail by studying both white matter and gray matter[29].Combining the point of[30],it is promising to find a relationship for learning gray matter, and also sulcal morphology, that differs between middle years and older ages for healthy individuals [31]; meanwhile, the study also shows that the correlation between gray matter volume and gray matter density of the brain and the autism‐spectrum quotient was significant[32].We believe that in the future,it is necessary to consider reconstruction of the neural network model that uses multiple sets of brain data to analyze white matter and gray matter and reevaluate the results.

ACKNOWLEDGEMENTS

This work was supported in part by the National Natural Science Foundation of China (61972174), the Science and Technology Planning Project of Guangdong Province(2020A0505100018), Guangdong Key‐Project for Applied Fundamental Research(2018KZDXM076),and Grants‐in‐Aid for Scientific Research (JP17K00312, JP17H03326) of Japan Society for the Promotion of Science (JSPS).

ORCID

Lixian Caohttps://orcid.org/0000-0002-9610-7683