Probabilistic or possibilistic expert knowledge modeling? Dunning-Kruger curve helps to choose!

Introduction

Are narrower interval estimates more accurate? Most people would confidently say YES based on their basic intuitions.¹ However, the incorrectness of this claim has already been proven experimentally.² Besides, the Dunning-Kruger effect also draws attention to the fact that greater confidence does not necessarily result from greater expertise.³ Moreover, the uncertainty of estimates can also be affected by other factors such as the pressure of giving informative judgments even in case of ignorance.⁴

To model and aggregate expert-based interval estimates, both probability and possibility theories are applicable. Considering interval estimates as a kind of (uncertain) measurement, their precision and accuracy can be characterized by the interval width and the estimation error, respectively.⁵ Probabilistic aggregation monotonically reduces the variance of low-precision estimates by the number of available judgments. On the other hand, it may keep the bias of high-precision estimates if they have low accuracy. In this case, possibilistic modeling and aggregation can be more beneficial as it ensures that the aggregated distribution would cover the true value, e.g., by using the union operator. However, when there are both low- and high-precision estimates, the decision between the probabilistic and possibilistic approaches is not so straightforward.

In case of estimates with varying precision, its relationship with accuracy may not be negligible when choosing the modeling technique. If higher precision does not mean greater expertise, the simplest and most commonly used probabilistic modeling technique (which favors narrower estimates) does not seem to be an acceptable solution. Possibility theory appears to be preferable, which not only assigns equal importance to judgments with different interval widths, but also makes it possible to investigate the consensus of estimates by simply analyzing the overlap of the resulted possibility distributions.⁶

Some previous works have compared the probabilistic and possibilistic approaches in different fields. When creating a measurement model, the preference strongly depends on the available a priori knowledge.⁷ Besides, a structural engineering example showed that problem complexity also matters in case of uncertainty propagation.⁸ As for expert knowledge modeling, the most significant work in recent years was conducted by Rohmer and Chojnacki.⁹ They performed extensive analysis involving many datasets to compare probabilistic and possibilistic approaches regarding how well they represent the aggregated opinions of multiple experts, using accuracy- and informativeness-based measures. They were unable to show significant differences between the performances; however, they investigated the average scores of multiple different estimation tasks, and did not consider the potentially different relationship between precision and accuracy in certain cases.

In this study, we are taking a deeper look into the modeling of expert-based interval estimates by investigating their distribution on the Dunning-Kruger curve. It illustrates the ambiguous relationship between confidence and expertise,³ which are equivalent to precision and accuracy in the case of interval estimates, respectively. Starting from the key differences caused by the different mathematical logic, we examine how the distribution of judgments on the Dunning-Kruger curve, i.e., the correlation trend of precision and accuracy affects the relative performance of probabilistic and possibilistic approaches, thus facilitating the decision between them.

Methods

Expert-based interval estimates are ambiguous. If we have an $[x_L^e,x_U^e]$ estimate about variable $x$ from expert $e$ , the width of the interval generally represents the uncertainty of the knowledge about $x$ . However, we usually do not have any information about the prioritization of values inside the interval: the expert may have thought that every value in the interval is equally probable, but (s)he might not have meant to give such additional information by his/her estimate. Therefore, the commonly used probabilistic modeling technique that assumes certain probability values over the interval does not always represent correctly the real information content of the estimations.

Probabilistic and possibilistic modeling

The given interval can be represented probabilistically by a uniform distribution as:

(1)

$$p_e\left(x\right)=\{\begin{array}{ll}\frac{1}{x_U^e-x_L^e}& \text{if}{x}_L^e\le x\le x_U^e\\ 0& \text{otherwise}\end{array}$$

which means that the probability is positive and inversely proportional to the interval length if the value of $x$ is in the given interval, and zero otherwise. In this case, probability represents the degree of uncertainty expressed by the interval length: narrow estimates will get a higher probability than wider ones, as they are less uncertain.

Otherwise, interval estimates can be modeled by fuzzy numbers in the possibilistic case, showing the degree of being member of the set, defined by the tuple:

(2)

$${\pi }_e\left(x\right)=(x_L^e-\alpha ,x_L^e,x_U^e,x_U^e+\alpha )$$

where the first and last members define the support, and the second and third members define the core of the fuzzy number, as shown in Figure 1. Inside the core, the fuzzy number takes one, and outside the support (defined by tunable $\alpha$ bandwidth) zero.

Figure 1. Definition of the trapezoidal fuzzy number representing the $[x_L^e,x_U^e]$ interval estimate.

$\alpha$ marks the user-defined bandwidth.

Notice that in this case, without restriction on the integral, the given interval defines the core of the fuzzy number, so the $x$ values inside $[x_L^e,x_U^e]$ take one in all the cases regardless the interval width.

Aggregation of interval estimates

The anomaly mentioned above, namely, that the probabilistic approach prioritizes narrow interval estimates but the possibilistic approach assigns all equal importance, increasingly prevails when the judgments of multiple experts are aggregated.

The aim of the aggregation is to summarize the judgments of multiple experts in one probability/possibility distribution. In case of the probabilistic approach, averaging the distribution functions representing individual opinions is a conventional aggregation technique:

(3)

$$p_{\mathit{\text{aggr}}}\left(x\right)=\frac{1}{N_e}\sum _e{p}_e\left(x\right)$$

where $N_e$ marks the number of experts.

In this work, we use the average operator in case of the possibilistic approach as well, which avoids giving zero possibility everywhere along the domain in case of conflicting estimates (e.g., min operator), and also covering uninformatively large area by possibility value one if the estimates are well distributed, without showing preferences (e.g., max operator):

(4)

$${\pi }_{\mathit{\text{aggr}}}\left(x\right)=\frac{1}{N_e}\sum _e{\pi }_e\left(x\right)$$

In the case of averaging possibility distributions, all expert judgments are considered with equal importance, as values within an interval estimate of each expert are considered with a possibility equal to one during the aggregation. Consequently, the mean curve (disregarding bandwidth) roughly represents the voting ratio for certain $x$ values as shown on the right in Figure 2. On the other hand, the $x$ values falling within narrow intervals are overemphasized in the probabilistic case, as they take higher probability than the values belonging to wide-interval responses as can be seen on the left in Figure 2.

Figure 2. Illustrative example of the comparison of probabilistic and possibilistic representation and aggregation of interval estimates.

Three interval estimates are considered here: $[1,3]$ , $[1,4]$ and $[3.5,4.5]$ . They are represented by uniform probability distributions on the left, and trapezoidal fuzzy number with zero bandwidth ( $\alpha =0$ ) on the right. The aggregated distributions are illustrated by red dashed lines, and the function values belonging to $x=2$ and $x=4.25$ by black points.

Three expert judgments with different interval widths were modeled and aggregated in Figure 2. It can be noticed that $x=4.25$ takes higher a probability but a lower possibility value than $x=2$ . This is a key consequence of the narrow-interval prioritization effect of the probabilistic approach.

Additionally, it has to be mentioned that the aggregated curve in the probabilistic case is still a probability distribution (with integral equal to one). Meanwhile, the averaging in the possibilistic case results in a subnormal possibility distribution (with a maximum below one). If needed, it should be normalized for further calculations.¹⁰

Results

The available interval estimates about the nine variables ( $x_i,i=1,\dots ,9$ ) given by the ten experts ( $N_e=10$ ) are summarized in Figure 3.

Figure 3. Interval responses of the ten experts for the nine key variables ( $x_1-x_9$ ).

The correct value is illustrated by yellow vertical lines. The correlation coefficients between interval width and estimation error ( $r_1-r_9$ ) are also depicted.

It can be seen that the correlation is positive in only about half of the cases, thus the falsity of the claim that narrower estimates necessarily imply higher level of expertise is verified. A strong correlation was found only in one case ( $x_9$ ) and a moderate correlation four times (two negative ( $x_1$ , $x_8$ ) and two positive ( $x_3$ , $x_7$ )). Estimates of four variables did not show a significant relationship between interval width and accuracy.

The aggregated probability and possibility distributions ( $\alpha =2\%$ ) with their means and the related advantage scores can be seen in Figure 4. There are some cases where the probabilistic approach outperforms the possibilistic ( $x_2$ , $x_3$ and $x_9$ ) and there are examples where the reverse is true ( $x_1$ and $x_8$ ).

Figure 4. Comparison of the performance of the probabilistic and possibilistic approaches.

The aggregated probability (blue) and possibility (red) distributions are shown. Their means are plotted by dotted lines in the corresponding color, and the correct values are illustrated by yellow vertical lines. The advantage scores ( $s_{\mathit{adv}}$ ) are also depicted.

Finally, the relationship between $r$ and $s_{\mathit{adv}}\left(x\right)$ values was investigated, as illustrated in Figure 5. A strong linear correlation ( $R=0.73$ ) was detected between these two factors, which is an encouraging result in terms of making a decision between probabilistic and possibilistic approaches based on the tendency characterized by $r$ . Although absolute errors ( $e^e$ ) are not available in practical cases to calculate $r$ as the correct value is not known, we can have a guess about the correlation trend, e.g, knowing the interval widths and the relative competences of experts.

Figure 5. Relationship between the confidence-accuracy correlation ( $r$ ) and the advantage score of the probabilistic approach ( $s_{\mathit{adv}}$ ).

The data points are labeled based on the variables ( $x_1-x_9$ ) they belong to, and the line fitted to them are depicted by red dashed line. The correlation coefficient of $r$ and $s_{\mathit{adv}}$ is $R=0.73$ .

Conclusions

Our results demonstrate that the performance difference between probabilistic and possibilistic modeling approaches strongly depends on the nature of the relationship between confidence (precision) and accuracy. If the distribution of answers on the Dunning-Kruger curve is known, the preferable approach can be defined derivatively as illustrated in Figure 6, where the red and blue points represent two different sets of estimates (e.g., about different variables).

Figure 6. Dunning-Kruger curve of expert-based interval estimates.

If the judgments show a negative correlation between accuracy and confidence, the possibilistic approach is preferable (red). However, if higher accuracy relates to narrower estimates, the probabilistic approach is recommended (blue).

Unfortunately, the type of confidence-accuracy correlation related to a certain task and a certain group of experts cannot be determined or predicted directly. It depends on several factors such as the nature of the task, the pressure of time and demand for informativeness, or the composition of the expert group. For now, it has not yet been clearly described as a function of measurable factors, which defines a research gap in the psychological field, and also provides a future research direction that has great practical relevance.

Although the dependence of the distribution of answers on the Dunning-Kruger curve, i.e., confidence-accuracy space, has not yet been explored completely, once this is done, users will be able to choose with great certainty between probabilistic and possibilistic approaches for modeling expert-based interval estimates. Until then, users are able to apply indirect solutions to deduce the nature of precision-accuracy trend, e.g., involving some a priori knowledge about the relative competence of experts in the group and comparing it with the precision of their estimates.

Ethical considerations

Ethical approval for this study was obtained from the Institutional Research Ethics Committee of the University of Pannonia (Approval number: KEB 2/2024. (12.03.)). Written and signed informed consent was obtained from all participants.

Code availability

Archived source code at time of publication: https://doi.org/ 10.5281/zenodo.16680269.¹¹

License: CC BY 4.0