Video quality assessment (VQA) in the medical field is an important task to achieve satisfactory conditions for medical imaging modalities like magnetic resonance imaging (MRI), computed tomography (CT) scans, and laparoscopy. VQA is composed of two stages: distortion classification and quality score evaluation. Laparoscopic surgery videos are prone to distortions that affect a surgeon’s visibility and degrade the vision quality for robot-assisted surgery. ¹

Laparoscopic videos are often affected by various types of distortions like noise, smoke, uneven illumination, and blur, which are all concomitant artifacts that arise from operating the laparoscopic surgical equipment. ² To enhance the distorted laparoscopic videos, most studies propose solutions that require troubleshooting the equipment. ^{2, 3} However, such solutions are time consuming and cannot guarantee high-quality laparoscopy every time.

Recent studies have suggested the use of image or video enhancement methods like de-smoking for laparoscopic surgery, ^{4– 6} and joint wavelet decomposition and binocular combination for endoscopic image enhancement. ⁷ In this case, real-time detection of the types of distortion is important to decide which enhancement methods are appropriate to apply. Real-time distortion classification is a challenging task and few recent studies have addressed it using hand-crafted features. ^{8– 12} These existing image quality assessment methods, such as BIQI, ¹¹ DIIVINE ¹² and BRISQUE, ¹⁰ were based on non-generic classification and are considered domain-dependent tasks. In addition, a distortion-specific classification approach has been demonstrated. ⁸ This approach used a separate traditional feature method for each type of distortion. ⁸

On the other hand, convolutional neural networks (CNNs) overcome the previous limitations and learn features automatically with the same CNN architecture to detect all types of distortions. This paper aims to address the challenge of distortion detection and produce a generic method for distortion classification in laparoscopic videos.

Artificial neural networks (ANNs) have shown significant capability in overcoming the issue of distortion classification by extracting informative features from all kinds of distortions. CNNs are powerful and efficient in several image tasks including classification, ¹³ segmentation, ¹⁴ enhancement, ¹⁵ and retrieval. ¹⁶ Recently, CNNs have also been used in several studies on image distortion classification for various applications. ^{17, 18} However, recurrent neural networks (RNNs), and specifically, long short-term memory (LSTM) ¹⁹ have not yet been investigated for distortion classification in video datasets. This paper aims to highlight the use of CNN-LSTM ²⁰ to improve classification accuracy.

In the context of distortion classification in laparoscopic surgery videos, a recent study has proposed the use of deep CNNs, such as ResNet for distortion ranking. ²¹ Its method achieved ranking accuracies of 83.3%, 84.7%, and 87.3% using Resnet18, Resnet34, and Resnet50, respectively. However, the previous work focused only on spatial features extracted from a collection of 20,000 images for image-level distortion ranking.

Another very recent work was found to transfer learning from pre-trained ResNet50 CNN to laparoscopic video frames. ²² The spatial features extracted from ResNet50 were applied to four support vector machine classifiers (three binary and one 5-class) utilizing decision fusion to produce the final distortion lists. ²² Hence, this paper proposes to extract spatiotemporal features using CNN-LSTM for video-level distortion classification.

The key contributions of this paper are: •

Utilization of a RNN model such as LSTM with time series of CNN-based features extracted from the frames. To the best of our knowledge, this is the first paper that uses CNN-LSTM for non-reference distortion classification in laparoscopic videos.

This paper is structured as follows: Methods describes the proposed method and the experiments including the dataset and the experimental setup. In Results and discussion, the results of the proposed solution and the comparison with existing methods are presented and discussed. Conclusions summarizes the significance of this work and opens doors for further improvement.

To the best of our knowledge, no other papers have utilized this extended version of the laparoscopic video dataset challenge dashboard for distortion classification. For this reason, we compared our approach with the best solutions presented in the ICIP2020 challenge as shown in Table 1.

The description of the baseline solutions was given by winners in the ICIP2020 challenge presentation event. One of the solutions was based on using a VGG16 CNN ²⁶ to extract features. The feature vector was applied to the fully connected neural network that included two hidden layers with 4096 nodes, two batch normalization layers, and two dropout layers. On the other hand, another solution used a deep multi-task learning model. It included one shared VGG-based feature extraction block and five independent binary classifiers (one for each distortion type). Each classifier had two fully connected layers with 512 nodes and one node in the output layer with a sigmoid activation function ICIP2020 challenge presentation event. The description of other baseline solutions was not presented, but the results were shown in the challenge dashboard.

The performance of the proposed methodology was evaluated in terms of classification accuracy, F1-score of single distortion, and F1-score of single and multiple distortions as shown in Table 1. It can be observed that the proposed ResNet50-LSTM leads to the best accuracy of 85.0%, while baseline methods yielded accuracies of between 57% and 81.5%. Additionally, ResNet50-LSTM yielded the best F1-score of single and multiple distortions (94.2%), while baseline methods yielded F1-score between 83.2% and 94.1%. Furthermore, it is clear that the performance of our method for multiple distortions outperforms that for single distortion, which still has room for improvement.

Figure 2 shows the confusion matrix for each distortion category produced from each LSTM. The LSTMs were able to correctly classify 58 videos out of 60, 46 videos out of 50, 94 out of 95, 88 out of 95 for AWGN, defocus blur, smoke, and uneven illumination, respectively. On the other hand, motion blur LSTM gave the worst classification performance with 29 correct videos out of 45. The reason for this drop was that the videos with motion blur have the minimum number of samples, which is only 80 videos. The performance of the motion LSTM can be improved significantly by having more samples affected by motion blur distortion. The performance metrics of the proposed method for each class are shown in Table 2.

The proposed ResNet50-LSTM was able to run considering real-time conditions. The inference time was 0.05 seconds to extract features from one frame using ResNet50. The features extracted from one frame were added to the features of other frames to be applied to the LSTM. The inference time for five LSTMs to produce the five distortion classes was 0.1 seconds. In summary, the proposed model updates the distortion categories every 0.15 seconds and achieves high speed performance.

In this paper, a novel strategy of distortion classification was proposed. A multi-label spatiotemporal deep model, including a pre-trained deep CNN of ResNet50 and five LSTMs, was used to address the problem of single and multiple distortion classification. The proposed model was tested with a laparoscopic video dataset and the results were promising. It was found that our model outperformed existing solutions in terms of accuracy by 4.5% and yielded the best F1-score for single and multiple distortions. Hence, we intend to enhance the performance by tuning more layers of pre-trained CNN with laparoscopic images affected by distortions to learn more informative features. The last step requires collecting a large number of images to achieve promising improvements. Additionally, more recent CNN architectures such as EfficientNet ²⁷ and DeiT (Data-efficient Image Transformers) ²⁸ models are good candidates for extracting informative features. In this paper, the proposed solution only classifies the laparoscopic distortions into five categories. Hence, in future work, we plan to rank each category of distortion in terms of distortion intensity, which is a more challenging matter.