Texture analysis of the developing human brain using customization of a knowledge-based system

Research background and goals

The aim of this research was to find a customizable software framework (KBS) to mathematically determine an optimal combination of texture analysis methods to differentiate anatomical structures in the developing fetal brain (i.e., regions of interest [ROI]). Why did this experiment focus on normal development of the central nervous system (CNS) rather than common conditions affecting fetal organogenesis? When considering fetal intervention to correct an anomaly in utero, the first ethical priority is actually mother safety. Prenatal well-being is clinically second, after evaluation of the benefit-risk ratio^22–26. In the 1960s, fetal surgery was conceived and introduced in clinical practice²⁷. In spite of improvements in surgical technology, the numbers of successfully treated cases of congenital defects and life expectancy of survivors are still limited^28–31. The theoretical procedures are troublesome and have not been investigated enough³² . Hence, they are considered as experimental treatments³². Whether invasive or minimally invasive, fetal intervention is not the ultimate solution of this societal conundrum. It is usually reserved for cases of severe fetal anomalies^28–32. In recent years, prenatal therapy is gaining popularity in religiously conservative territories particularly where abortion is prohibited^33,34. Fetal intervention is recommended for fetuses with mild and non-lethal defects, in order to discourage abortion and continue childbearing. In the country where this research was carried out (i.e., Poland), abortion is, once again, nationally banned―except in cases of rape, life-threatening pregnancy and childbirth, and grave malformations^33–36. Pressures are due to growing pro-life supporters demanding total anti-abortion, child-bearing at all costs, and capital punishment for illegal termination of pregnancy^33–36. Nevertheless, Poland neither sanctions nor executes the outlaws of illegal abortion. Despite the absence of penalty, abortion is respectfully performed as permitted by local authority. The aforementioned dilemma justified the usage of fetal magnetic resonance imaging (fMRI) in this research. In theory, it was previously hypothesized and documented that the high electromagnetic fields (EMF) used for magnetic resonance (MR) procedures can disrupt the early stages of organogenesis. To this date, the embryotoxic, fetotoxic, and teratogenic effects of MRI are not well known. Normally, fMRI is not recommended during the first and second trimester, unless it is absolutely necessary to confirm and/or supplement the diagnosis of fetal anomalies. The legal decision to prescribe abortion to a patient sometimes requires advanced clinical investigation, accompanied by psychological counseling before and after the operations^37–41. To this date, ultrasound (US) devices are preferred for obstetrical examinations. With 2D, 3D, and 4D US scans, physicians can effectively and efficiently diagnose the majority of life-threatening conditions affecting mother and fetus^42–46. Therefore, ultrasonography (USG) is sufficient for diagnosis of severe malformations affecting abdominal organs. Why was fMRI prescribed? The fetal brain is where US devices struggle to produce desirable results. MRI was subsequently performed to rule out severe abnormalities in brain development, which are not visible on sonogram. MR samples came from fetuses with suspected heart and kidney defects. Disruption of organogenesis in the latter, depending on severity, might affect normal development of the brain. Specialists, structure by structure, visually investigated MR samples used in this experiment. No severe malformations were observed. In these cases, MRI studies did not add any further indication to legally fulfill the criteria to terminate pregnancy. The human visual apparatus has its limit. Unfortunately, missed diagnoses do occur. Malformations may not be apparent prior to birth. If fetal defects are suspected, bureaucracy may also restrict access to more advanced testing and healthcare. Consequently, pregnant women are deliberately forced to bear and deliver malformed babies. At the end of the day, physicians may still have to deal with the legal liability for failure to terminate pregnancy⁴¹. Unwanted fetuses may become neglected, and foundling is also a growing problem in society^47–49. Congenital brain defects and its impacts on physical and cognitive development may not be detectable until after birth. Fetal outcome and mental retardation can be difficult for a physician to predict. The process requires access to better medical examination, development of more advanced tools, and further investigation. That is why the ultimate goal of this feasibility study was to gather knowledge for the practicality of a proposed project seeking to improve diagnostic accuracy and precision, by extending HCI usage in medicine. There are many CAD tools on the commercial shelves (e.g., Radiomics,^50–52, Definiens Tissue Phenomics®⁵³, and CAD4TB Diagnostic Software^20,54). Sadly, they are primarily designed for pre-loaded applications but not much else.

Ethics and informed consent

This feasibility study was approved by the Bioethics Research Board, which regulates experiments carried out at MUL and affiliated research hospitals in Poland (permit number: RNN/213/13/KE). Due to administrative and logistic delay as well as finding volunteers and the expensive cost and availability of MRI^55,56, it took us nearly five years to collect the MR samples. The nature of the study was explained to patients in Polish by attending physicians, and individual written informed consent was obtained for this research and its publication. Agreement number: 3/2011 concluded on December 6, 2011, between the experimenter (Hugues Gentillon), research supervisor (Ludomir Stefańczyk), rector of MUL (Radzisław Kordek), and the Faculty of Biomedical Science-Post-Graduate Research in Diagnostic Imaging and Radiotherapy. Consent for publication of these data was obtained. Furthermore, all data from human participants were anonymized as per consent agreement. Informed Bioethics Committee Approval Number: RNN/213/13/KE, JULY 16, 2013. If you have any questions regarding the decision please include the above number and date in your letter. Send correspondence to: THE BIOETHICS COMMITTEE OF THE MEDICAL UNIVERSITY OF LODZ Al. Kościuszki 4, 90-419 Łódź, tel. 0 785 911 596, 42 272-59-05, fax 42 272-59-07.

Clinical trial registration

Though this research shares similarities with a clinical trial, it is “virtual” (i.e., it is non-interventional). Such a medical study does not meet criteria for clinical trial registration^57–61. Furthermore, the investigational tools were merely used for technical exploration and to measure their feasibility in medical practice by using simulation settings. Lastly, the results were not used to alter patients’ therapeutic care and outcome^57–61.

Advance notice to readers. Readers should not expect us to teach the entire science of an artificial neural network (ANN) (feedforward neural networks, recurrent neural networks, etc.) in just a manuscript. It is not possible. A full introduction is not even possible. Unfamiliar readers are expected to make an effort on their own to read and learn the basic principles of an ANN and know the basic terminology. Like a human brain, an ANN can store memories. An ANN can also judge based on stored memories and logical rules. To execute a naive trial run, it is ideal to have the ANN in a condition like “permanent global amnesia” (i.e., a phenomenon where a brain is in a state of total blackout and, thus, cannot judge based on prior memories). The KBS used in this experiment is lacking an automatic memory cleaner and optimizer. Hence, stored memories were manually deleted in the ANN for every trial run.

Computer vision. Under a reciprocal cooperation agreement between MUL and TUL, we obtained permission to test a KBS consisting of a custom version of MaZda software package^62,63. It contains algorithms for data classification and visualization. How did it come to us? In 1998, a group of medical scientists, engineers, physicists, mathematicians, and others initiated the B11 Project at the European Cooperation in Science and Technology (COST). The purpose was to develop software frameworks for quantitative analysis of MR scans to improve medical diagnosis. MaZda, a Delphi/C++ computer program, was originally built in 1996 for applications in mammography. It was later used by COST to complement its MRI software modules (e.g., B11, B21). MaZda 4.6 package was the last official release (B11 version 3.3 included). In this research, we collaborated with modular programmers to further upgrade, separate, and amalgamate the functionality and compatibility of MaZda (version 5 RC HG) with other modules. In the MaZda 5 version used in this research, the new feature names were introduced to maintain compatibility with WEKA software (www.cs.waikato.ac.nz/ml/weka/; a popular package for data mining, classification, and analysis). As a result, data generated by MaZda may be used as the input to the WEKA. Also, 3D deformable models that are used for 3D volume-of-interest (VOI) generating was introduced (for details see the publication by: Piotr M. Szczypiński, Ram D. Sriram, Parupudi V.J. Sriram, D. Nageshwar Reddy, A model of deformable rings for interpretation of wireless capsule endoscopic videos, Medical Image Analysis, volume 13, issue 2, April 2009, pages 312–324). Moreover, to speed up and automate the process, some commands regarding 3D analysis were introduced. This way one can build a script (that contains commands for image loading, analysis option loading, performing analyses, and storing results) for automatic analysis of a number of 3D data. Finally, this MaZda 5 version was compiled with an Embarcadero software environment, which was not an ideal framework. Thus, the inventors decided to return to Builder C++. Currently, the latest version of MaZda being built is qMaZda (described and available at www.eletel.p.lodz.pl/pms/SoftwareQmazda.html).

Figure 1 shows the main steps of texture analysis with MaZda and B11. A key change in this custom version is the integration of MaZda with a variety of computational geometric algorithms from Qhull (e.g., VSCH_1, VSCH_2, VSCH_3, VSCH_4, and VSCH_5). For details about 1D, 2D, 3D, and 4D features see Dataset 7. VSCH network algorithms are located in the “Convex Border” menu. VSCH_1, for example, can be used to identify the strongest discriminant parameter in a large dataset.

Figure 1. Main steps of texture analysis with MaZda and B11.

Our trial and error experiments and texture analysis software development spanned over five years of research. All the findings shared common denominators. Quantitative brain tissue segmentation was affected by several factors, such as characteristics of fetuses (e.g., gestational age, shape, and normal/abnormal development) and the quality of fMR images (e.g., 1.5/3T, resolution, slice thickness, rotational planes, artifacts, etc.). Automatic segmentation of newborn brain MRIs have been documented in the literature⁶⁴. The algorithmic contributions reported so far have achieved limited success, unfortunately. Automatic segmentation of prenatal brains is even more challenging and time consuming.

Unsupervised segmentation. Once an image was acquired in a readable format (bitmap format [BMP]), the first step was texture segmentation (i.e., partitioning an image into ROI). B11 can perform unsupervised segmentation and cluster analysis. In some instances, B11 achieved accuracy close to that of clinicians. However, unsupervised segmentation was not reliable enough for therapeutic use. Fetal brain segmentation with B11 still requires extensive expert interaction. In our observations, the key problems were maternal factors, environmental effects, growth variability, randomness of fetal movements, and its detrimental effects on image quality. Therefore, automatic segmentation was used for new insight into the possibility of improving supervised segmentation. The steps of unsupervised segmentation are relatively simple: image acquisition and run analysis. ROI and segment numbers can be manually adjusted. The drawback with B11 segmentation was its limitation to 8-bit grayscale BMP. 16-bit DICOM was converted to visually lossless BMP, by dropping the least significant bits. Note that B11 identified textures not anatomical structures. The information collected from the unsupervised trials was later used as guidance to manually estimate boundaries of anatomical structures (ROI) for the supervised trials. The preliminary trials were single-blinded (i.e., the user knew the characteristics of the ROI), and the KBS received no hints (no ROI selection). Further information was gathered on semi-automatic (unblended) segmentation by defining 4 classes (ROI): thalamus, ventricles, gray matter, and white matter. In the unsupervised mode, the KBS performed quite well when brain images were from MR examination of the same subjects and same sequences but performed poorly when they came from different subjects. The findings were likewise for the same sequences of the same patient taken at different times and MR scanner settings. The challenge was, “How do we match macroscopic characteristics with electronic recognition, regardless of MR image shadings? MaZda and B11 are not yet designed to allow the user to define semantic rules and/or import plug-ins for fully electronic recognition of anatomical structures. Object-based image analysis tools such as eCognition work consistently well for geospatial applications (e.g., identification of a river in an image)^65,66. In fetal radiology, it is still a challenge to achieve consistent results with automated pattern recognition of prenatal anatomy. Programming a reliably effective system for such highly sensitive application is feasible but also time consuming. Such a task would require taking into account all known variations due to pregnancy chronology and fetal developmental, to minimize segmentation errors. MaZda and B11 were originally built for HCI rather than fully automated applications. Therefore, the best practical methodology, in this research, was for the operator to at least have prior knowledge of human embryogenesis, in order to manually and correctly identify and select fetal ROI. Often, the macroscopic appearances of many brain structures are not well differentiated in the first trimester. Hence, the selected samples were at least 20 weeks of maternal age.

Supervised segmentation. 3-tesla (3T) and 1.5-tesla (1.5T) MR sequences of fetal brain were manually segmented into 3456 ROI. The categories were predefined as follows: ventricles (class 1), thalamus (class 2), white matter (class 3), and gray matter (class 4). The selected samples did not have any brain malformations. As previously mentioned, the anomalies were in the cardiovascular and/or renal systems. The focus of this research was on normal anatomy of fetal brain. Forward processing method, also known as “supervised segmentation”⁶⁷, was performed as delineated: (1) image acquisition from MR scanner, (2) selection of ROI with MaZda, (3) image normalization with MaZda, (4) feature extraction with MaZda, (5) data preprocessing with B11, and (6) texture classification with B11^62,63,67. The first four steps were done with MaZda and last two steps with B11 (Figure 1). After the preliminary trials, we became interested in learning what needs to be adjusted in order to reduce misclassification of MR images. The unsupervised segmentation revealed that the KBS was very sensitive to grayscale shading, artifacts, and image thickness as well as resolution quality. Thus, we trained the KBS accordingly. 1.5T images were originally encoded as 12-bit lossless JPEG (Joint Photographic Experts Group) format and wrapped in DICOM. 1.5T images were transcoded from lossless JPEG to uncompressed DICOM. 3T images were natively uncompressed. Parameters (n = 360) were extracted with MaZda (histogram: 9; co-occurrence matrix: 220; run-length matrix: 20; gradient matrix: 5; auto-regression: 5; Haar wavelet: 28; and geometry: 73). The names of the parameters are provided in the appendix at the end of this manuscript. We assessed three automated techniques of parameter selection (i.e., Fisher selection, POE + ACC [classification error probability and average correlation coefficients], MI + PA + F [combination of mutual information, pair analysis, and Fisher selection). Further details about the mechanics and functionality of these techniques are provided in the manufacturers’ user manuals and tutorial guides⁶⁸. For each technique, we also used the VSCH module in MaZda to enhance computation and visualization of geometric structures: 1) prereduction (before selection) to rule out insignificant parameters and 2) post-reduction (after selection) to identify the strongest relevant parameters. The KBS without any manual interference computed all aforementioned parts of the process automatically. Data imported (SEL [Schweitzer Engineering Laboratories] to CSV [comma-separated values]) into B11 were preprocessed with PCA, LDA, and NDA and classified by means of 1-nearest neighbor (1-NN) and ANN. In machine learning and cognitive science, 1-NN is also known as k-NN, where n = 1. It is a “lazy-learning” algorithm, in which new ROI are locally classified by getting interweaved into the closest cluster in the training set. The rest of the computation is delayed until the end of the classification process. 1-NN is implemented in B11 classifiers as well as in the preprocessing procedures in MaZda⁶⁸. On the other hand, ANN is a self-organizing algorithm with hidden layers and adjustable numbers of neurons^68–71. It can be used for both supervised and unsupervised classification. Neural classification algorithms are implemented in MaZda/B11. ANN training, for example, is standardized for NDA analysis^68,69 (i.e., a type of feedforward ANN based on multilayer Perceptron [MLP])⁶⁸. Nonlinear procedures and classifiers in B11 are MPL algorithms^68,69. For optimal performance, two sets of samples are needed: one for training and another one for validation⁶⁹. The pitfall with this algorithm is its sensitivity to overtraining (too strong memorization)^68,69. ANN training time is shorter with standardization. For continuation, training without standardization was carried out, in spite of long processing times. ANN (one-class/n-class) and 1-NN training runs were conducted with different sequences of MR images: T2-weighted, T1-weighted, and proton density (PD)-weighted sequences. N-class training was discontinued due to repeated problems with overtraining and lack of reproducibility in F values and misclassification errors.

Customization

Despite the usage of multilevel automated selection/reduction techniques, some extracted values still did not match the controlled ROI values. Differentiating the thalamus from other thalamic nuclei and gray matter was the key problem. As a result, we manually intervened to customize and improve the extracted data. First, ROI surface areas were manually increased, in order to limit the number of parameters reporting zero and infinity values. Parameters that couldn’t be correctly computed were manually omitted in the report file. Some pre-processing procedures in both MaZda and B11 couldn’t be performed when the report file contained erroneous values. We accessed MaZda generated report files by changing the extension format from SEL to CSV and then imported them into Excel 2013 for adjustment. Parameters measured with other CAD tools can also be entered in the report files by simply using Microsoft Excel. The edited file can then be imported into B11 to perform texture classification.

Regions of interest

Additional tests were carried out with the same ROI (i.e., thalamus, ventricle, gray matter, and white matter) to dramatically improve accuracy and precision of the KBS. It was done with a customized dataset derived from MaZda algorithms using semi-manual reduction and nearest-neighbor feature selection (see Data availability: Dataset 1–Dataset 2). The training data were used to orient the KBS to recognize what ROI had the same tissue characteristics, in spite of originating from different patients or different sequences of the same patients. The training was conducted with a combination of two built-in classification tools (i.e., NN and ANN) and four data processing techniques (i.e., RAW [read as written], PCA, LDA, and NDA). To measure the KBS sensitivity and specificity, we defined “normal” as ROI with identical tissue characteristics and defined “abnormal” as those with different tissue characteristics (Figure 2–Figure 5). Apart from noise and artifacts, we found out that the preliminary results were also affected by the planes (axial, coronal, and sagittal), which refer to the rotational planes of the spinning MR scanner in relation to the mother not the fetus. There flows the reason for the classification by rotational planes. In learning mode, we observed a consistent scoring for all the ROI. Thus the logical and semantic information provided to the KBS was effective. Statistical binary tests (also known as classification function tests) were computed in STATISTICA version 10 to assess the performance of each procedure (combination of preprocessing techniques and classifiers). In medicine, binary scores (TP, FP, TN, FN, etc.) are used to determine not just normal and abnormal characteristics but also classification property of an examination.

Figure 2. White matter ROI selection.

a) Specificity scoring: ROI 1 = white matter control, ROI 2 = white matter and b) Sensitivity scoring: ROI 1: white matter control, ROI 2: thalamic nucleus other than thalamus.

Figure 3. Thalamus ROI selection.

a) Specificity scoring: ROI 1 = thalamus control, ROI 2 = thalamus and b) Sensitivity scoring: ROI 1: thalamus control, ROI 2: thalamic nucleus other than thalamus.

Figure 4. Ventricle ROI selection.

a) Specificity scoring: ROI 1 = ventricle control, ROI 2 = ventricle and b) Sensitivity scoring: ROI 1: ventricle control, ROI 2: white matter.

Figure 5. Gray matter ROI selection.

a) Specificity scoring: ROI 1 = gray matter control; ROI 2 = gray matter and b) Sensitivity scoring: ROI 1: gray matter control, ROI 2: white matter.

Selecting KBS tools

The majority of the KBS we came across were designed for technical use and not easily customizable. Such programs required paying for marketing company maintenance and for in-house developed customization service, in addition to the annual license fee. Thus, this option was not feasible for application in real-world settings where resources are often sparse (e.g., eCognition,^65,66, Media Cybernetics^76,77, Radiomics^50–52, Definiens Tissue Phenomics®⁵³, CAD4TB Diagnostic Software^20,54, etc.).

Logic and reasoning behind the research design

Previous medical studies done with MaZda include inflammation, brain cancer detection, multiple sclerosis, electrophoresis, etc^78–81. Herein, we defined some test samples as “abnormal” ROI. However, they were, in reality, normal tissue. Not testing directly for a common anomaly doesn’t necessarily mean that there is no real medical application. Though the tests were simulated, the research design was conceived for real-world medical applications^82–84. For example, this research design could be used to detect ectopic tissue migration, neurogenesis and neuronal migration (brain function migration as a result of natural process or after injury), metaplasia, and interference with brain development. Last but not least, this simulation research followed standards used in clinical trials⁸⁵.

Statistical test and interpretation

The choice of binary classification (sensitivity/specificity) was favored over frequentist inference (p-value), because it provides more information in terms of statistical relevance to medical diagnosis, prognosis, and disease prevalence^86–88. One key difference between Fisher, POE + ACC, and MI + PA + F is the number of parameters. To perform MI + PA + F, the dataset must contain at least 30 parameters strongly matching its selection-reduction criteria⁶⁸. Otherwise, the KBS reported an error. In our study, it was a common occurrence when the surface area of a ROI was insufficient to extract 30 parameters meeting the MI + PA + F semantics. RAW and PCA were not so affected by the training process and, thus, remained very sensitive to minute differences in grayscale shading.

Recommendations

A solution to high misclassification (M) was to exclude some parameters that were very sensitive to postediting sharpness. In this research, the images were, however, processed without postediting sharpness because high M was not regarded as a problem. Instead, we used RAW and PCA as reference tests (results before the training of the KBS). On the other hand, LDA and NDA responded well to the training, and M was consistently zero.

KBS memory clearing

During the study, we had to obviously clear the KBS memory several times for every trial run. We hope that the software developers will soon implement a more effective and efficient way (e.g., one click) to clear specific random access memory (RAM) without closing module(s) or without manually dumping the entire RAM or restarting the computer.

Constraints, limitations, and assumptions

Patients gave consent to perform MRI examination, use of images in research, and for the manuscript publication. Nevertheless, this authorization was not enough, as ownership and copyright of medical records are not always exclusively attributed to patients, and such rights may not be assignable^89–91. For the sake of prudence, we had to also seek institutional/research hospitals’ clearance and approval, which in turn were then subject to different administrative and logistic factors and regulations. Consequently, it took nearly five years to collect sufficient MRI samples to make this research possible.