Multimodal Machine Learning Approach for Diagnosing Atopic Dermatitis

Image

The dataset used in this study consists of images of skin lesions from patients diagnosed with AD as well as patients suffering from other skin conditions whose lesions visually resemble those found in AD cases. To perform a supervised classification task, we categorized the images into two distinct classes: AD (representing AD lesions) and Non-AD (representing non-AD lesions). In order to enhance model generalization and robustness, we applied extensive data augmentation prior to training. Each original image was augmented five times using the Albumentations library to produce image sets with various transformations. These included horizontal and vertical flipping, random adjustments to brightness and contrast, rotations of up to 30 degrees, mild Gaussian blurring, and the addition of Gaussian noise. After augmentation, all images were resized to a standardized dimension of 224 × 224 pixels. Furthermore, pixel values were normalized using the ImageNet mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] to match the input expectations of the pretrained models. For the classification task, we evaluated two different DL architectures: ResNet50 and Vision Transformer (ViT). Each model’s final classification layer was adapted to produce outputs corresponding to the two target classes, AD and Non-AD.

Training was conducted using the PyTorch DL framework, which is widely used in the field of computer vision research due to its flexibility and high performance. Since the dataset used in this study is relatively small in scale, which is a common situation in the medical field where obtaining labeled images can be challenging, it becomes crucial to adopt an evaluation strategy that maximizes the use of available data while still providing a reliable estimate of the model performance. To address this, we employed the use of K-fold Cross Validation.

Rather than splitting the dataset into a single training and testing group, the cross-validation approach divides the dataset into ten equal parts, or “folds.” In each round of training, the model is trained using nine of these folds and evaluated on the remaining one. This process is repeated ten times, with each fold taking a turn as the validation set. At the end of this procedure, the results from all ten rounds are averaged to provide a comprehensive assessment of the model’s performance. This technique reduces the risk of the evaluation being biased by the specific choice of training or testing data and is particularly important when dealing with medical datasets, where the number of samples can be limited. It offers a favorable balance between model accuracy and computational efficiency.

During training, we also implemented an optimization algorithm called Adam, which automatically adjusts the learning rate during training to improve the model’s ability to learn from data. To further refine the training process, we applied a strategy called learning rate scheduling, where the learning rate was reduced gradually over time to allow the model to converge more smoothly. In addition, we implemented early stopping, a technique where the training process is halted if the model’s performance on validation data does not improve after a few consecutive training rounds. This prevents the model from overfitting, or memorizing the training data too much, and helps ensure that it learns patterns that are generalizable to unseen data.

Throughout the training and evaluation process, we monitored several key metrics to assess model performance. These included accuracy, which measures how often the model’s predictions matched the correct labels; precision, which indicates how many of the predicted DA cases were actually correct; recall, which measures how many actual DA cases the model successfully identified; and F1-score, which balances precision and recall into a single number to give an overall sense of the model’s effectiveness. By considering these multiple aspects, we aimed to evaluate not just whether the model made correct predictions overall, but also whether it was good at correctly identifying cases of AD without missing too many or falsely predicting AD where there was none.

After completing the full training and evaluation pipeline, ResNet50 emerged as the best-performing model among the three architectures tested, achieving an accuracy score of 0.8750, compared to 0.6013 for the ViT. ResNet50 consistently demonstrated superior performance across all key metrics, including higher accuracy, precision, recall, and F1-score when compared to the other model ( Table 2). In particular, ResNet50 exhibited strong stability during training and converged more rapidly. These results suggest that convolutional architectures with residual connections, such as ResNet, are particularly well-suited for medical image classification tasks such as skin lesion analysis, where fine-grained texture analysis is the highlight.

Text

The dataset used in this study consists of anamnesis text of patients whose diagnosed with AD as well as patients suffering from other skin conditions whose lesions visually resemble those found in AD cases. Clinical text inputs were derived from dermatologist notes within medical records during Phase 1, and from patient anamnesis recorded in structured forms during Phase 2. The text in the dataset are written in Bahasa Indonesia. To perform a supervised classification task, we categorized the anamnesis into two classes, the AD and Non-AD. The anamnesis contains several parameters such as patient age, gender, symptoms, contact history, etc. Those parameters were then configured to follows a template as follows:

Pasien dengan jenis kelamin: <gender>

usia: <age>

klasifikasi usia: <age_classification>

Keluhan utama dan onset: <main_onset_symptoms>

Riwayat kontak dengan bahan alergen atau iritan: <irritant_allergent_contact_hisotry>

Sumber infeksi: <infection_source>

Faktor pencetus penyakit saat ini: <current_disease_cause>

Lama sakit: <disease_duration>

Lokasi lesi: <lesion_location>

Kriteria mayor: <major_criterion>

Kriteria minor: <minor_criterion>

Riwayat penyakit dahulu: <old_historical_disease>

Riwayat penyakit keluarga: <familiy_historical_disease>

By generating text following this template, we obtained a string as the model input to be classified by the proposed method. After we obtain the full text as model input, we need to do pre-processing to the text before feeding it to the model. In this text processing task, the preprocessing part was done by using a tokenizer. A tokenizer is a tool or process used in natural language processing (NLP) to break down a text into smaller units, called tokens. These tokens can be words, subwords, or characters, depending on the tokenization method used. The purpose of tokenization is to transform raw text into a more manageable and structured form that can be analyzed or fed into ML models.

In this study we used a pre-trained model, MPNet which is a state-of-the-art transformer model that was introduced as a variant of BERT (Bidirectional Encoder Representations from Transformers) and other transformer-based models like RoBERTa and ALBERT. MPNet improves upon BERT’s pretraining strategy by using a more sophisticated approach for handling masked tokens and learning better dependencies between tokens in a sequence. The tokenizer used in MPNet is based on the WordPiece tokenization method, which is also used in BERT and similar transformer models.

The output of this MPNet pre-trained model is 768 sized vectors. To do classification task, we added some dense networks downstream the MPNet architecture and formed a single output classification. The model was created using PyTorch DL framework, which is widely used in the field of computer vision research due to its flexibility and high performance. For the training process, we splitted the dataset into training and validation parts with a ratio of 8:2. After the training process, the fine-tuned MPNet model obtained an accuracy of 100% in both validation and training data.

Multimodal

The multimodal approach in this study integrates two powerful models: ResNet50 for visual data and MPNet for textual data, to enhance classification performance by leveraging both image and text information. ResNet50, a deep CNN, is employed to process the images of skin lesions. These images are first preprocessed through augmentation, resizing, and normalization, ensuring compatibility with the input requirements of ResNet50. The model then generates a feature vector from the image, capturing the essential visual characteristics of the skin lesion, which represents the image data for subsequent processing.

In parallel, MPNet, a transformer-based model, is applied to the textual data consisting of patients’ anamnesis information. MPNet tokenizes the text and generates a fixed-size vector (768-dimensional), which encodes the relationships and context between the words. This representation is essential for understanding the nuanced medical history of the patients, such as symptoms, age, and other relevant factors. MPNet’s ability to model contextual dependencies and generate more accurate token representations provides an edge over traditional models like BERT, especially when dealing with complex and diverse medical texts.

Once both ResNet50 and MPNet have processed their respective modalities and produced their vectorized outputs, the next step is to combine these two vectors into a single unified feature representation. This fusion of image and text data ensures that the model is utilizing all available information, combining the fine-grained visual analysis from the images with the rich, contextual information from the text. The feature vectors from both modalities are concatenated, creating a comprehensive representation that encapsulates the full scope of the data.

Finally, to refine the combined feature vector and prepare it for classification, several dense layers are applied downstream. These dense layers help the model learn the most relevant patterns from the multimodal data. The final output from the dense layers is a classification label that distinguishes between the two target classes: AD and Non-AD. This multimodal approach effectively leverages both image and text data, improving the model’s robustness and accuracy by considering multiple facets of the data simultaneously, which is crucial in medical classification tasks.

Clinical implications, strengths, and limitations of the multimodal AI model

The image-only model achieved an accuracy of 87.50% on the training set and 90.13% on the test set ( Figure 1). Key performance metrics, including accuracy, precision, recall, and F1 score are presented in Figure 2 and Table 2. The text-only model achieved 100% accuracy on both the training and validation sets. Following the fusion of both modalities, the overall accuracy increased significantly to 98.28% ( Figure 3).

The multimodal AI approach employed in this study offers several advantages. First, the AI model can consistently analyze hundreds of microscopic visual features and textual patterns without fatigue, enabling the potential for high accuracy and sensitivity across diverse cases. In fact, numerous studies have demonstrated that AI performance in dermatological diagnosis can match or even surpass that of dermatology specialists for certain specific tasks.²⁷ For instance, a recent meta-analysis by Salinas et al. (2024) on skin cancer diagnosis reported AI algorithms achieving a sensitivity of 87% and specificity of 77%, comparable to expert dermatologists (sensitivity 84%, specificity 74%).²⁸ In the context of AD, the ResNet50 model, when specifically trained, has outperformed conventional models and approached expert-level performance in assessing AD severity.²⁹ By integrating anamnesis data, AI can also account for factors typically gathered through patient interviews, thereby allowing for more clinically informed decision-making.

AI systems can also process multiple cases simultaneously, supporting mass screening or triage workflows, thereby allowing physicians to prioritize complex cases. Moreover, multimodal AI reduces subjectivity: decisions are derived from patterns learned across hundreds of examples, rather than individual intuition, which can vary among physicians. With a reported accuracy of 98.28%, this system has strong potential as a reliable diagnostic aid, offering second opinions to clinicians and enhancing general practitioners’ confidence in correctly identifying AD.³⁰

Nevertheless, several important considerations must be addressed. First, the model’s outstanding performance in a controlled setting may not necessarily reflect its effectiveness in real-world clinical practice. Generalizability remains a common challenge in AI models, as they may learn patterns specific to a particular hospital’s dataset, potentially resulting in reduced accuracy when applied to different patient populations or images captured with other camera devices. While 10-fold cross-validation provides a comprehensive internal evaluation, external validation using data from independent clinics is essential to ensure the model is not overfitting. The high accuracy of 98.28% raises a concern that the model may have been overtrained on a limited dataset. In this study, external validation was conducted using datasets from multiple hospitals to ensure that the model maintains reliable and consistent performance. In future work, explainable AI (XAI) techniques could be employed to enhance interpretability, allowing the model’s decision-making process to be visually explained and verified against clinical reasoning.^31,32

Another key issue is interpretability. Although AI can achieve high levels of accuracy, it remains limited in its ability to explain the rationale behind a given prediction. Physicians, by contrast, can justify a diagnosis based on observable clinical features (e.g., “there is excoriation due to intense itching, a typical distribution pattern on the infant’s cheeks, and a positive family history of atopy, consistent with AD”). AI models require XAI techniques to offer similar justifications. Without such interpretability, trust in AI-generated recommendations—both among physicians and patients—may be diminished. Moreover, broader clinical context is often necessary for final decision-making. Physicians may rely on direct physical examination (e.g., palpating the skin texture), additional diagnostic tests, or further clinical history. AI models, on the other hand, can only process the limited input they are provided (i.e., images and structured textual data), making their reasoning less flexible than that of physicians. Clinicians can pose follow-up questions, revise diagnostic hypotheses in real time, and manage atypical or complex cases not represented in the training data—for example, a patient presenting with two concurrent skin conditions.

Finally, physicians possess critical strengths in empathy and clinical ethics—for example, delivering a diagnosis and treatment plan in a psychologically appropriate and compassionate manner. This human element lies beyond the scope of AI models. Therefore, AI should be viewed as a complementary tool that supports physicians, rather than as a full replacement. AI is intended to serve as a decision support system, not a substitute for clinical judgment. Collaborative use of AI and physicians has been shown to improve diagnostic accuracy compared to either working alone.³³ In practice, AI can provide a rapid second opinion, while the final decision and any necessary contextual adjustments remain the responsibility of the physician.

Accuracy of 98.28% and its relevance in current literature

The final accuracy of 98.28% achieved by the multimodal ResNet50 and MPNet model in distinguishing between AD and non-AD cases represents an exceptionally high performance. To assess its relevance, this result must be compared with recent studies in the field of computer-aided dermatological diagnosis. In general, AI models for skin lesion classification typically achieve accuracies above 80–90%. For example, Wu et al., (2020) developed and validated an image-based DL model, AIDDA, using the EfficientNet-b4 convolutional neural network architecture to automate the diagnosis of common inflammatory skin diseases, including psoriasis, eczema, and AD. Trained on a dataset of 4,740 expert-labeled clinical images, the model achieved high diagnostic performance, with an overall accuracy of 95.8%, sensitivity of 94.4%, and specificity of 97.2%.

Therefore, the 98.28% accuracy achieved in this study warrants attention as a highly promising result. It highlights the potential advantage of a multimodal approach. By incorporating additional clinical data, the model may resolve ambiguities that remain when relying on image analysis alone, allowing for near-perfect classification across samples. Other multimodal studies have also reported strong performance. Adebiyi et al. (2024), for instance, integrated lesion images with patient metadata and achieved an area under the curve (AUC) of 0.94 (94%) on the HAM10000 dataset—an improvement over image-only models.³⁴ In addition, recent developments in transformer-based fusion approaches, such as TFormer, have been specifically designed to deeply integrate multimodal information and have demonstrated enhanced diagnostic accuracy across various types of skin lesions.¹⁷

Nevertheless, the scientific community emphasizes the necessity of validating AI models in real-world clinical scenarios. Burlando et al. (2024), in a recent meta-analysis, stressed the importance of external validation across diverse populations and collaborative decision-making between AI and clinicians.²⁸ Near-perfect accuracy may indicate model excellence, but also raises the concern of potential data leakage or bias. For instance, if structured anamnesis texts contain inadvertently predictive features (e.g., specific keywords unique to AD cases), the model might overfit to non-clinical artifacts. Thus, transparent methodological reporting—such as details on 10-fold cross-validation and feature importance analyses—is essential for interpreting these results responsibly.

The 98.28% accuracy is consistent with previous studies reporting that advanced DL models, particularly those utilizing multimodal inputs, can achieve diagnostic performance comparable to or exceeding that of experienced clinicians in specific tasks.¹⁹ This parallels recent innovations in digital dermatology, such as the PanDerm foundation model, trained on millions of multimodal images, which outperformed dermatologists in early melanoma detection.³⁵ Similarly, generative models like SkinGPT-4 (2024) now integrate ViT with large language models to generate diagnostic reports that approximate physician reasoning.³⁶ In this context, our findings contribute compelling evidence that multimodal AI has the potential to reshape dermatological diagnostics, provided that it is demonstrated to be consistent in larger-scale studies.

Several limitations warrant consideration. First, the model was trained on anamnesis written in Bahasa Indonesia using a language-specific version of MPNet. Although MPNet has multilingual variants, the one applied here may not generalize effectively to English or other languages. Moreover, differences in documentation style (e.g., brief vs. narrative notes) could also impact model performance. Future implementations should consider language adaptation or employ inherently multilingual models for broader applicability.³⁷

Second, the differential diagnosis in the present study was restricted to four conditions: lichen simplex chronicus, psoriasis vulgaris, nummular dermatitis, and contact dermatitis. In clinical reality, AD mimickers may also include conditions such as scabies, fungal infections, or seborrheic dermatitis, which were not represented in the training data. This raises concerns about generalizability when the model is applied to unfamiliar cases. Expanding the dataset to encompass a broader spectrum of dermatoses, or transitioning to a multiclass classification model, could improve both utility and clinical relevance.

Third, real-world deployment of this system would require regulatory approval, robust documentation of safety and reliability, and integration with electronic health records to ensure seamless clinical workflow. This would necessitate interdisciplinary collaboration among clinicians, engineers, and policy makers. However, to successfully implement AI, key barriers need to be addressed. Efforts should focus on ensuring algorithm transparency and adequate regulation of algorithms. Simultaneously, improving knowledge about AI could reduce the fear of replacement.

Ethical considerations

Ethical approval for the study was obtained from the Health Research Ethics Committee, Faculty of Medicine, Universitas Indonesia–Cipto Mangunkusumo Hospital (certificate number KET-1477/UN2.F1/ETIK/PPM.00.02/2024). All participants provided written informed consent prior to data collection. For minor participants, written consent was obtained from their legal guardians. Ethical approval was obtained from the Ethics Committee of the Faculty of Medicine, Universitas Indonesia–Cipto Mangunkusumo Hospital. Patient data and images were anonymized before use, and no identifiable information was included in the analysis. This diagnostic study did not involve any interventional procedures.