Quantized Convolutional Neural Networks Robustness under Perturbation

2.1 Quantization

In this work we adhere to common quantization schemes with neural networks and map the higher-resolution set of possible FP32 numbers to a lower-resolution subset of INT8 numbers.^23,35 In accordance with typical approaches to network quantization, we are quantizing the weights and outputs (activations) through the network^23,35,37 We also consider strictly post-training quantization (PTQ) rather than quantization-aware training (QAT), as it does not require a complete re-training step but a simple calibration step on unlabeled data. While generally QAT yields higher compression rates and lower error rates,²³ as this paper will show, PTQ still performs well and comes with the benefit of faster and less resource-intensive implementation.

PTQ necessitates first defining several key parameters: the bit-width $b$ , the step-size or scale factor, $s$ , and the zero-point $z$ _.²³ The bit-width defines the number of possible levels in the quantization grid. The scale factor sets the step-size or difference between each level. The zero-point is an integer chosen such that the actual zero is quantized without error and is important to ensure activation functions like ReLU do not introduce additional quantization errors.²³

Using symmetric or affine quantization involves mapping parameters to the quantization grid depending on the symmetry of the scheme. For the unsymmetric (affine) case, we have

For the symmetric about $z$ case

The notation $\lfloor \cdot \rceil$ is the round-to-nearest integer operator while the clamping function is defined as

For the symmetric case, $z$ is constrained to 0, and the range is bounded by

As mentioned, we select a bit width of 8-bits, the most commonly used scheme to balance compression and error rates.³⁵ Scale and zero point are determined using optimization algorithms packaged with PyTorch’s²⁶ quantization library. We quantize weights symmetrically and the activations in an affine fashion. These choices reflect that weights are approximately symmetrically distributed around 0, whereas activations are positively skewed owing to their activation function, e.g., ReLU.

2.2 Dataset

To study FGVC, we use the fine-grained visual classification aircraft (FGVCA) dataset.²² FGVCA was first introduced as part of the 2013 International Conference on Computer Vision (ICCV), and since then has seen frequent use in the literature.^34,5 FGVCA is comprised of 10000 1-2 Mpixel images and provides several classification tasks and labels, including aircraft model (most specific), variant, family, and manufacturer (least specific).²² The family classification task is considered here and comprises 70 family labels. Examples of families include Boeing-737, which includes variants like 737-200 or 737-300.²² Figure 1 illustrates several dataset samples. Dataset images have varying resolutions and aspect ratios and are altered to a consistent input size as part of the pre-processing transformation.

Figure 1. Sample imagery from the FGVC-A dataset.

Other FGVC datasets are available, including natural species,⁸ birds,¹⁸ or flowers.²⁴ FGVCA has important subclass properties where aircraft features are strongly dependent on size (hobbyist aircraft to large transport aircraft), purpose (commercial, pleasure, or military), and technology (turbine propulsion, propeller, glider, etc.). Each of these designs have unique structural features such as the wing shape and size, fuselage style, landing gear/wheels, and engine mounting. Another interesting feature of FGVCA is that different organizations such as airlines and military often have slight modifications such as branding and camouflage, meaning classes can have the same rigid structure, but distinctly different "looks". Cumulatively, these features result in rigid variability across classes compared to tasks such as birds or flowers, where non-trivial variations can exist within the same class due to mutations, climate, etc. This rigidity is important for our study as it ensures that errors owing to ambiguity in classes are minimized, allowing us to focus exclusively on errors owing to each model’s capacity for spatial embedding.

2.3 Perturbations

To model real-world phenomena, we selected additive white Gaussian Noise (AWGN), spatially correlated Brownian noise, and vertical and horizontal occlusions. AWGN is commonly used as a model for thermal noise experienced in circuits and random photo-sensor noise.^14,6 Additionally, AWGN is commonly used in adversarial attacks against deep learning-based systems.^30,38 Brownian noise approximates correlated noise like smoke, fog, clouds, and underwater distortions. Vertical and horizontal occlusions are a model of structured image corruption owing to losses from sensor malfunctions, data corruption during transmission, or solar radiation resulting in rows or columns of dead pixels.

Given an image, x:

Pixels are indexed by $i,j,$ and $k$ respectively denoting row, column and channel yielding our $C$ channel additive noise mask $n_g(i,j,k)$ :

The perturbed input follows from a simple addition of the noise mask to the original input:

Brownian noise differs from AWGN in that it is not independent and identically distributed, but rather spatially correlated. Moreover, unlike the flat power spectral density of AWGN, Brownian noise has a power spectral density proportional to the inverse square of the frequency, $\frac{1}{f^2}$ , and lower frequencies dominate the noise spectrum. Thus, lower frequencies are amplified while higher frequencies are attenuated, giving long-range, smoothly-varying spatial correlations. Brownian noise adds blotches and blur-like artifacts to an image as illustrated in Figure 2.

Figure 2. Sample imagery from the FGVC-A dataset under various perturbations.

To generate Brownian noise, we start with white noise, $n_g(i,j,k)$ , sampled from (9). These noises are fast Fourier transformed (FFT) for each of the three channels and yield the frequency components $u,v,z$ , or $W(u,v,z)$ :

We then scale each frequency component by $\frac{1}{f^2}$ , where $f$ is the frequency magnitude, $\sqrt{u^2+v^2+z^2}$ :

Finally, we apply the inverse FFT (IFFT) to convert to the spatial domain and add the resulting noise to the image:

Lastly, we examine highly structured noise in vertical and horizontal black-out occlusions, or streaks, applied within a pre-defined bounding box of the classification target within each image, as shown in Figure 2. We generate these as having constant width such that increasing the number of streaks increases the image coverage. We control the intensity or degree of perturbation induced by such occlusions by adjusting the number of streaks present in the class’ bounding box region. Edges introduced by occlusions are expected to significantly degrade performance since CNNs extract features such as edges from images, and the occlusions may cause irrelevant feature activation and false edge identification. Streaks may also cover important or distinguishing features of a target class, such as its engine or wing structures, and reduce network performance.

2.4 Kullback-Liebler divergence

To extend our analysis of the performance of quantized networks to the softmax probability distribution, we study the KL divergence between the output probabilities of each FP32 and INT8 model. KL divergence, introduced in,¹ quantifies the difference between two distributions $P$ (true) and $Q$ (baseline).³²

If we observe similar accuracy with high KL divergence, the quantized model maintains accuracy at the cost of what is termed here confidence. For example, consider an FP32 model that yields the following output probabilities: class 1: 0.7, class 2: 0.2, class 3: 0.1, and an INT8 model that yields class 1: 0.5, class 2: 0.4, class 3: 0.1, where the actual label is class 1. Both models correctly classified class 1 as the actual label. However, they have markedly different degrees of confidence. Of course, KL divergence alone does not grant insight into class-by-class probabilities, but it does quantify a macro-level model similarity, which may serve as a precursor to specific class probability studies. However, it is important to recognize that this metric assumes the FP32 model is the true baseline distribution. We contend that this is a fair characterization as, in reality, it is the theoretical best we can do regarding capacity for information embedding per parameter. In this case, the baseline is indeed dynamic in the sense that the distribution of the FP32 model will change under a perturbation. Given that both the FP32 and INT8 models are tested on identical inputs with identical perturbation, we are then answering the question under perturbation level X, how does our quantized model deviate in its probabilities from what would otherwise be outputted given no quantization?

3.1 Models and training

We examine three state of the art models including VGG-16,³³ ResNet-18,¹⁷ and SqueezeNet1_1.¹⁹ These models were selected on the basis of their varying size and ubiquity in the current body of computer vision research. We compare the size of each network in terms of parameter count and floating point operations (FLOPs) in Table 1. Models were downloaded from PyTorch’s model zoo,²⁶ with pre-trained weights for ImageNet.¹⁰ Each model was adjusted to have 70 output neurons, in accordance with the FGVCA families classification task. Models were trained for 250 epochs on a training set of shuffled 3333 images, using the cross-entropy loss function to measure the prediction error. The stochastic gradient descent optimizer was employed with a learning rate of 0.001 and a momentum of 0.9 to update the model parameters during training. Additionally, a learning rate scheduler was applied to decrease the learning rate by a factor of 0.1 every 50 epochs.

3.2 Perturbation intensities

We examine the performance of each model under each of the perturbations discussed above. As noted, each of these perturbations has its own respective intensity parameter. For AWGN and Brownian noise it is standard deviation, and the number of occlusions for vertical and horizontal occlusions. In each case, the intensity range was experimentally selected to best capture the full spectrum of degradation, as is shown in the accuracy/F1 plots in Section 3. For AWGN we study standard deviations ranging from 0 to 1 with a step size of 0.1 from 0.1 to 0.6, and then 0.2 from 0.6 to 1.0 as beyond 0.6 model performance plateaus. For Brownian noise we study standard deviations ranging from 10 to 80 with a step size of 10. For vertical and horizontal occlusions we select occlusions ranging from 1 to 6 with step sizes of 1. Note that the inclusion of even 1 streak yielded significant degradation across all metrics, and as more occlusions were added, performance quickly degraded to near 0 across all metrics. Also, as discussed in Section 1.3 the Brownian perturbation regime required comparatively higher standard deviations than AWGN to yield any noticeable degradation. This is expected, as Brownian noise is smoother and more structured than AWGN, owing to its $1/f^2$ spectrum, making it less disruptive to the image’s overall structure and features when compared to AWGN’s direct pixel-by-pixel variations. For each perturbation scheme each model’s FP32/INT8 pair were tested on the full test set of 3333 images.

3.3 Metrics computation

We are interested in computing 4 primary metrics, top-1 accuracy, top-5 accuracy, F1-score, and KL divergence. Top 1 accuracy, top 5 accuracy, and F1 score are computed using Scikit-learn.²⁷ To compute KL divergence we store the 70 class output probabilities for each model pair and compute the KL divergence between them, on a per image basis, for each perturbation level. We then compute the average KL divergence across the entire test set of 3333 images, for each perturbation level, resulting a in a single scalar averaged measure for each model pair.

4.1 Unperturbed results

We begin by examining the unperturbed results to establish a baseline. Table 2 presents F1, top-1 and top-5 accuracies for each model pair with VGG-16 showing the best performance across categories and SqueezeNet1_1 the worst. This result is expected result and matches the general trend presented in Table 1 which supports the notion that increased parameter count driven by layer depth yields higher accuracy. As expected, we can see that top-1 accuracy is comparatively worse than top-5 accuracy across all models - SqueezeNet1_1 exhibits a 14% top-1 accuracy drop compared to VGG-16 but only a 6% drop in top-5 accuracy. This is indicative of the nature of FGVC where it is likely that smaller models like SqueezeNet1_1 lack the expressive power to capture the subtleties between similar classes but can make a near correct prediction. We also see that F1 - the average of a model’s ability to avoid false positives (precision) and false negatives (recall) - follows the same trend as top-1 accuracy. Most notable, however, is that we observe very little degeneration in model performance across all three metrics post-quantization. This result is congruent with^23,35,37 and confirms that our quantization scheme is implemented and performing as expected. Table 3 presents KL divergences for each model pair. Interestingly, ResNet-18 exhibits the lowest KL divergence by an order of magnitude, and VGG-16 the highest. While all relatively low, these results may reflect each model’s architecture. For instance, ResNet-18’s skip connections, may aid in mitigating quantization error propagation where VGG-16’s depth without such connections may exacerbate quantization error propagation.

4.2 Perturbed results

We begin by looking at the impacts of AWGN, presented in Figure 3, and Table 4. Across all models and metrics, we see steep degeneration beginning at $\sigma$ = 0.2 and plateauing at $\sigma$ = 0.8, indicating a point of potential ill-conditioning in this particular noise regime. As expected, top-5 accuracy is consistently higher than top-1 by a 20% - 30% margin, with the gap between them widening as noise levels increase. This result suggests that while the exact classification becomes more challenging, the correct class will likely remain among the top 5 predictions. Moreover, in line with our baseline observations, VGG-16 generally outperforms ResNet-18 and SqueezeNet1_1, and ResNet-18 typically outperforms SqueezeNet1_1, with the gap narrowing at higher noise levels. F1 scores follow similar trends to accuracy metrics but show a steeper decline with respect to noise, suggesting that both precision and recall are possibly more severely impacted by noise than accuracy metrics. Most important to this study, however, is that in all cases, we see no significant divergence in performance curves between each FP32/INT8 model pair, directly supporting the notion that quantized models, under AWGN, are just as performant, according to accuracy and F1 metrics, irrespective of overall losses. Figure 5 and Table 6 present the KL divergences under AWGN. Here, we witness interesting non-linear behavior - under AWGN, each model’s INT8 representation exhibits peak divergence in its class output probabilities in the $\sigma$ = 0.4-0.6 range. Also, in line with the trend in baseline KL divergences, VGG-16 demonstrates the highest divergence despite being less sensitive in its accuracy and F1 response curves. This suggests that while VGG-16 in its INT8 form is demonstrably robust according to accuracy and F1, its output distribution shifts from its FP32 form, potentially indicating decreased overall confidence or misplaced confidence. This result is particularly relevant in threshold-based systems where decisions are made based on some output probability threshold. The drop in KL divergence after $\sigma$ = 0.6 is most likely due to the model pairs reaching a saturation point in error, where, in effect, each model pair is equally weak and unable to make any meaningful predictions, which is evidenced in the accuracy plots where we can see sub-20% top-1 accuracy and sub-40% top-5 accuracy for each quantized and full-precision model.

Figure 3. Performance of each model pair under varying levels of AWGN.

Figure 4. Performance of each model pair under varying levels of Brownian noise.

Figure 5. KL divergences for each model pair under various levels of AWGN.

Next, we examine the impacts of varying levels of Brownian noise. Figure 4 and Table 5 present the F1 scores, top-1 and top-5 accuracies where we can see very different response curves compared to the AWGN results. It is clear that, as expected, the effects of Brownian noise are much more gradual compared to AWGN, as the spatial correlation of Brownian noise yields a much smoother noise pattern. Like AWGN, we see VGG-16 outperform all models as it maintains top-1 accuracy above 50% and top-5 accuracy above 70% until $\sigma$ = 50. Interestingly, we can see that Brownian noise seems to induce a relatively large error in the quantized model from the FP32 model compared to AWGN, especially in VGG-16 in the $\sigma$ = 10-50 range. In terms of F1, we also observe more significant discrepancies between the quantized and full-precision models; for instance, VGG-16 exhibits roughly 2% error in F1 at $\sigma$ = 20. Also, as expected, the top-5 accuracy is consistently higher than the top-1 accuracy in all three models. KL divergence, as shown in Figure 6 and Table 7, is much higher than AWGN and does not have the same peaking behavior shown in Figure 5 - instead we see a steady increase with noise across all three models. Like AWGN, VGG-16 exhibits the highest divergence, again suggesting a comparatively higher discrepancy in the model’s confidence among output classes when making predictions. ResNet-18, in contrast, yields the lowest divergence. Overall, we can see that the impacts of Brownian are much more gradual; however, they induce significantly higher KL divergences when compared to AWGN.

Figure 6. KL divergences for each model pair under various levels of Brownian noise.

We now look at the results under vertical occlusions, given in Figure 7 and Table 8. Here, we see performance degradation in accuracy and F1 gradually dropping off, similar to Brownian noise, which is an expected result as occlusions are another form of highly correlated noise. However, while the decrease is gradual, the immediate drop off at just 1 occlusion is significant. For example, the inclusion of 2 vertical occlusions yielded sub-50% top-1 accuracy across all models. With respect to quantization, we see the same general trend as other perturbation regimes, in that there is minimal divergence between each FP32/INT8 model pair, again suggesting that performance degradation is a macroscopic phenomenon - a function of model architecture and overall size, not parameter-wise information capacity. Indeed, each model exhibits relatively low KL divergence, with the max being 0.0636 for VGG-16 at two vertical occlusions, as shown in Figure 9 and Table 10. Moreover, the lack of divergence is relatively constant, with the most significant delta being just 0.038 for VGG-16 between 1 and 3 occlusions. However, it is notable that ResNet-18 exhibits the lowest KL divergence, which, paired with its performance in top-1, top-5, and F1, indicates it is comparatively robust against highly structured perturbation.

Figure 7. Performance of each model pair under various amounts of vertical occlusion.

Figure 8. Performance of each model pair under various amounts of horizontal occlusion.

Figure 9. KL divergences for each model pair under various amounts of vertical occlusion.

Lastly, we look at performance under horizontal occlusions with results shown in Figure 8 and Table 11 while Figure 10 shows the KL divergence. Similar to the results under vertical occlusion, we see an immediate drop-off at just one occlusion, again showing that, generally, models are much more sensitive to highly structured occlusions than AWGN and Brownian noise. Also of note is that VGG-16 performs the best out of all three models across accuracy and F1, unlike the vertical occlusions. F1 is similar to the vertical occlusion results, with a gradual decay congruent with the top-1 accuracy curve. Another interesting note is that we generally observe less degradation for the same number of streaks than vertical occlusions. This may be because the distinguishing features learned during training are dominated by horizontal edges such as the plane’s fuselage, and introducing points of high contrast in the form of structured noise is much more disruptive, as they are guaranteed to intersect the plane’s structure. Further, we see a widened discrepancy between top-1 and top-5 accuracies with horizontal occlusions compared to vertical, suggesting it is likely that generally, models are more likely to have the correct class in their top 5 predictions but struggle with exact classification. This result is likely exacerbated by the dataset’s nature, in that such occlusion severely impairs the model’s ability to distinguish between subtleties in the different classes. We can also see the same behavior as all other studied perturbations with respect to quantization in that there is minimal divergence between the INT8 and FP32 models across all metrics. Most unique to horizontal occlusions is that the KL divergence for all model pairs is near zero and shows no clear trend. This indicates that the output distributions are similar under such perturbation and lack any notable discrepancy.

Figure 10. KL divergences for each model pair under various amounts of horizontal occlusion.