Chapter 8 RSM fitting

8.1 How much finished 95%

8.2 Introduction

The Radiological Search Model (RSM) is detailed in my FROC paradigm book. Early efforts at estimating RSM parameters revealed that an FROC curve based estimation method patterned after (Edwards et al. 2002) worked only for designer level CAD data and not for human observer data. Subsequent effort focused on ROC curve-based fitting, and this proved successful at fitting human observer datasets. A preliminary account of this work is in (DP Chakraborty and Svahn 2011).

It may be surprising that the ROC curve based fitting works, which implies that one does not even need FROC data to estimate RSM parameters. I have previously stated that the ROC paradigm ignores search so how can one estimate search-model parameters from ROC data? The reason is that the shape of the RSM-predicted ROC curve and the location of the end-point depend on the RSM parameters.

8.3 ROC Likelihood function

In Chapter TempComment \@ref(rsm-predictions) expressions were derived for the coordinates \((x,y)\) of the ROC curve predicted by the RSM, see Eqn. TempComment \@ref(eq:rsm-predictions-fpf) and Eqn. TempComment \@ref(eq:rsm-predictions-tpf2), where \(x\equiv x(\zeta,\lambda)\) and \(y \equiv y(\zeta , \mu, \lambda, \nu, \overrightarrow{f_L})\).

Let \((F_r,T_r)\) denote the number of false positives and true positives in the ROC rating bin \(r\) defined by thresholds \([\zeta_r, \zeta_{r+1})\) where \(r = 0, 1, ..., R_{FROC}\). \((F_0,T_0)\) represent the numbers of non-diseased and diseased cases respectively with no marks, \((F_1,T_1)\) represent the corresponding numbers with highest rating equal to one, etc.

The probability \(P_{1r}\) of a count in non-diseased ROC bin \(r\) is:

\[\begin{equation} P_{1r} = x\left ( \zeta_r \right ) - x\left ( \zeta_{r+1} \right )\\ \tag{8.1} \end{equation}\]

The probability \(P_{2r}\) of a count in diseased ROC bin \(r\) is:

\[\begin{equation} P_{2r} = y\left ( \zeta_r \right ) - y\left ( \zeta_{r+1} \right )\\ \tag{8.2} \end{equation}\]

Ignoring combinatorial factors that do not depend on parameters the likelihood function is:

\[\left ( P_{1r} \right )^{F_r} \left ( P_{2r} \right )^{T_r}\]

The log-likelihood function is:

\[\begin{equation} LL_{ROC} \left ( \mu, \lambda, \nu, \overrightarrow{f_L} \right )= \sum_{r=0}^{R_{FROC}} \left [F_r log \left (P_{1r} \right ) + T_r log \left (P_{2r} \right ) \right ] \\ \tag{8.3} \end{equation}\]

The total number of parameters to be estimated, including thresholds, is \(3+R_{FROC}\). Maximizing the likelihood function yields parameter estimates.

The Broyden–Fletcher–Goldfarb–Shanno (BFGS) (Shanno and Kettler 1970; Shanno 1970; Goldfarb 1970; Fletcher 1970, 2013; Broyden 1970) minimization algorithm, as implemented as function mle2() in R-package (Bolker and R Development Core Team 2023) was used to minimize the negative of the likelihood function. Since the BFGS-algorithm varies each parameter in an unrestricted range \((-\infty, \infty)\), which would cause problems (e.g., RSM parameters cannot be negative and thresholds are subject to an ordering constraint) appropriate transformations (“forward” and “inverse”) were used so that, irrespective of values chosen by the BFGS-algorithm, the values supplied to the log-likelihood function were always valid.

8.4 Implementation

Function FitRsmROC() fits an RSM-predicted ROC curve to a binned single-modality single-reader ROC dataset. It is called by ret <- FitRsmRoc(binnedRocData, lesDistr$Freq, trt = 1, rdr = 1), where binnedRocData is a binned multi-treatment multi-reader ROC dataset, lesDistr is the lesion distribution vector \(\overrightarrow{f_L}\) for the dataset and trt and rdr are the desired treatment and reader to extract from the dataset, each of which defaults to one.

The return value ret is a list with the following elements:

  • ret$mu The RSM \(\mu\) parameter.
  • ret$lambda The RSM \(\lambda\) parameter.
  • ret$nu The RSM \(\nu\) parameter.
  • ret$zetas The RSM \(\zeta\) parameters.
  • ret$AUC The RSM fitted ROC-AUC.
  • ret$StdAUC The standard deviation of AUC.
  • ret$NLLIni The initial value of negative log-likelihood.
  • ret$NLLFin The final value of negative log-likelihood.
  • ret$ChisqrFitStats The chisquare goodness of fit (if it can be calculated).
  • ret$covMat The covariance matrix of the parameters (if it can be calculated).
  • ret$fittedPlot A ggplot object containing the fitted plots along with the empirical operating points and error bars.

8.5 FitRsmROC usage example

  • The following example uses the first treatment in dataset04; this is a 5 treatment 4 radiologist FROC dataset (Zanca et al. 2009) consisting of 200 cases acquired on a 5-point integer scale, i.e., it is already binned. If not one needs to bin the dataset using DfBinDataset(). The number of parameters to be estimated increases with the number of bins since for each additional bin one needs to estimate an additional cutoff parameter.
rocData <- DfFroc2Roc(dataset04)
lesDistr <- UtilLesDistr(dataset04)
ret <- FitRsmRoc(rocData, lesDistr = lesDistr$Freq)

The lesion distribution vector is 0.69, 0.2, 0.11. This means that fraction 0.69 of diseased cases contain one lesion, fraction 0.2 contain two lesions and fraction 0.11 contain three lesions.

The fitted parameter values are as follows (not shown are cutoffs excepting \(\zeta_1\), the chi-square statistic and the covariance matrix):

  • \(\mu\) = 3.642
  • \(\lambda\) = 9.529
  • \(\nu\) = 0.796
  • \(\zeta_1\) = 1.482
  • \(\text{AUC}\) = 0.9065
  • \(\sigma (\text{AUC})\) = 0.023
  • \(\text{NLLIni}\) = 281.4
  • \(\text{NLLFin}\) = 267.27

The fitting program decreased the negative log-likelihood (NLL) from 281.4 to 267.27. A decrease in negative log-likelihood is expected since one is maximizing the log-likelihood. Because the RSM contains 3 parameters, which is one more than conventional ROC models, the chi-square goodness of fit statistic usually cannot be calculated except for large datasets - the criterion of at least 5 counts in each bin for true positives and false positives is usually hard to meet. Using the method described in Section 6.12.5 the degrees of freedom is \(\text{df} = R_{FROC} - 3\)13.

Shown next is the fitted plot and exact 95% confidence intervals for the lowest and highest operating points.

RSM-fitted ROC curve for treatment one and reader one of `dataset04`.

FIGURE 8.1: RSM-fitted ROC curve for treatment one and reader one of dataset04.

The fitted ROC curve in Fig. 8.1 is proper. The area under the proper ROC (calculated by numerical integration of the RSM-predicted curve) is 0.907 which will be shown in Chapter 9 to be close to that yielded by CBM and PROPROC and higher than the binormal model fitted value.

8.6 Discussion / Summary

Over the years, there have been several attempts at fitting FROC data. Prior to the RSM-based ROC curve approach described in this chapter all methods aimed at fitting FROC curves in the mistaken belief that this approach was using all the data. The earliest was (Dev P. Chakraborty 1989). This was followed by (Swensson 1996), subsequently shown to be equivalent to my earlier work as far as predicting the FROC curve was concerned (DP Chakraborty and Yoon 2008). In the meantime CAD developers, who relied heavily on the FROC curve to evaluate their algorithms, developed an empirical approach that was subsequently put on a formal basis in (Edwards et al. 2002). While this method works with CAD data it fails with radiologist data – the reason is that it assumes \(\zeta_1 = -\infty\) (i.e., the algorithm marks every suspcioius region) which is not true for radiologists as discussed in my 2017 print book.

This chapter describes an approach to fitting ROC curves using the RSM. On the face of it fitting the ROC curve seems to be ignoring much of the data (since the ROC rating on a case is the rating of the highest-rated mark on that case). If the case has several marks only the highest rated one is used. In fact the highest rated mark contains information about the other marks on the case, namely that they were all rated lower. For example the highest of a number of samples from a uniform distribution is a sufficient statistic, i.e., it contains all the information contained in the observed samples. Neglect of the marks rated lower is not as bad as might seem at first.

The next chapter describes application of the RSM and other available proper ROC fitting methods to a number of datasets.

REFERENCES

Bolker, Ben, and R Development Core Team. 2023. Bbmle: Tools for General Maximum Likelihood Estimation. https://github.com/bbolker/bbmle.
Broyden, Charles George. 1970. “The Convergence of a Class of Double-Rank Minimization Algorithms 1. General Considerations.” IMA Journal of Applied Mathematics 6 (1): 76–90.
Chakraborty, Dev P. 1989. “Maximum Likelihood Analysis of Free-Response Receiver Operating Characteristic (FROC) Data.” Medical Physics 16 (4): 561–68.
Chakraborty, DP, and Tony Svahn. 2011. “Estimating the Parameters of a Model of Visual Search from ROC Data: An Alternate Method for Fitting Proper ROC Curves.” In Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment, 7966:189–97. SPIE.
Chakraborty, DP, and Hong-Jun Yoon. 2008. “Operating Characteristics Predicted by Models for Diagnostic Tasks Involving Lesion Localization.” Medical Physics 35 (2): 435–45.
Edwards, Darrin C, Matthew A Kupinski, Charles E Metz, and Robert M Nishikawa. 2002. “Maximum Likelihood Fitting of FROC Curves Under an Initial-Detection-and-Candidate-Analysis Model.” Medical Physics 29 (12): 2861–70.
Fletcher, Roger. 1970. “A New Approach to Variable Metric Algorithms.” The Computer Journal 13 (3): 317–22.
———. 2013. Practical Methods of Optimization. John Wiley & Sons.
Goldfarb, Donald. 1970. “A Family of Variable-Metric Methods Derived by Variational Means.” Mathematics of Computation 24 (109): 23–26.
Shanno, David F. 1970. “Conditioning of Quasi-Newton Methods for Function Minimization.” Mathematics of Computation 24 (111): 647–56.
Shanno, David F, and Paul C Kettler. 1970. “Optimal Conditioning of Quasi-Newton Methods.” Mathematics of Computation 24 (111): 657–64.
Swensson, Richard G. 1996. “Unified Measurement of Observer Performance in Detecting and Localizing Target Objects on Images.” Medical Physics 23 (10): 1709–25.
Zanca, Federica, Jurgen Jacobs, Chantal Van Ongeval, Filip Claus, Valerie Celis, Catherine Geniets, Veerle Provost, Herman Pauwels, Guy Marchal, and Hilde Bosmans. 2009. “Evaluation of Clinical Image Processing Algorithms Used in Digital Mammography.” Medical Physics 36 (3): 765–75.

  1. The expansive form of the relevant equation is \(R_{FROC} + R_{FROC} - (3 + R_{FROC}) = R_{FROC} - 3\)↩︎