Chapter 9 Three proper ROC fits

9.1 How much finished 99%

9.2 TBA Introduction

A proper ROC curve is one whose slope decreases monotonically as the operating point moves up the curve, a consequence of which is that a proper ROC does not display a chance line crossing followed by a sharp upward turn, i.e., a “hook”, usually near the (1,1) upper right corner.

There are three currently available methods for fitting proper curves to ROC datasets:

The PROPROC (proper ROC) model described in Chapter 7.
The CBM (contaminated binormal model) also described in Chapter 7.
The RSM (radiological search model) described in Chapter 8,

This chapter compares these methods by fitting them to 14 datasets described in Chapter 11. Comparing the RSM to the binormal model would be inappropriate as the latter does not predict proper ROCs.

The motivation for this work was a serendipitous finding (DP Chakraborty and Svahn 2011) that PROPROC fitted ROC AUCs and RSM fitted ROC AUCs were identical for some datasets. This led to extending that work to include CBM-fitting and more datasets.

9.3 Application to datasets

Both RSM and CBM are implemented in R-package RJafroc (Dev Chakraborty and Zhai 2022). PROPROC is implemented in Windows software OR DBM-MRMC 2.5 ¹⁴ that was available here, last accessed 1/4/21. The pre-analyzed PROPROC results-file locations are shown in Appendix 9.10.1.

The RSM, PROPROC and CBM algorithms were applied to datasets described in Chapter 11. These are named as follows:

datasetNames <-  
  c("TONY", "VD", "FR", 
  "FED", "JT", "MAG", 
  "OPT", "PEN", "NICO",
  "RUS", "DOB1", "DOB2", 
  "DOB3", "FZR")

The datasetNames array contains abbreviations for the contributors: “Dr. Tony Svahn”, “Dr. Van Dyke”, “Dr. Franken”, “Dr. Federica Zanca”, “Dr. John Thompson”, “Dr. Magnus Bath”, “Dr. Lucy Warren”, “Dr. Monica Penedo”, “Dr. Nico Karssemeijer”, “Dr. Mark Ruschin”, “Dr. James Dobbins-1”, “Dr. James Dobbins-2”, “Dr. James Dobbins-3”, “Dr. Federica Zanc a real ROC”. These are included with the RJafroc package and the corresponding objects are named datasetXX, where XX is an integer ranging from 1 to 14.

In the following we focus, for now, on just two ROC datasets: the Dr. Van Dyke (VD) and the Dr. E. Franken (FR) datasets. Fits are shown for treatment 1 and reader 2 for the Van Dyke dataset and for treatment 2 and reader 3 for the Franken dataset. Plots for all treatment-reader combinations for these two datasets are in Appendix 9.10.3 for the Van Dyke dataset and Appendix 9.10.4 for the Franken dataset.

# VD dataset
ret <- Compare3ProperRocFits(
  datasetNames, 
  which(datasetNames == "VD"))
resultsVD <- ret$allResults
plotsVD <- ret$allPlots

# FR dataset
ret <- Compare3ProperRocFits(
  datasetNames, 
  which(datasetNames == "FR"))
resultsFR <- ret$allResults
plotsFR <- ret$allPlots

The supporting code is in function Compare3ProperRocFits() located at R/compare-3-fits/.
The results file locations are shown in Section 9.10.2. Since the ML algorithm is time consuming, the cited locations contain pre-analyzed results.
The fitted parameters are contained in resultsVD and resultsFR; the composite plots (i.e., 3 overlaid plots corresponding to the three proper ROC fitting algorithms) for each treatment and reader are contained in plotsVD and plotsFR.

9.4 Composite plots

The Van Dyke dataset yields $I \times J = 2 \times 5 = 10$ composite plots.
The Franken dataset yields $I \times J = 2 \times 4 = 8$ composite plots.

The following code shows how to display the composite plot for the Van Dyke dataset (labeled D2 in the plot title as it is the second dataset in datasetNames) for treatment 1 and reader 2, i.e., i = 1 and j = 2.

# i = 1 and j = 2 
plotsVD[[1,2]]

FIGURE 9.1: Composite plots for Van Dyke dataset for treatment = 1, reader 2.

It contains 3 fitted curves:

The RSM fitted curve is in black – this is the only fit that includes a dashed line extending to (1,1).
The PROPROC fitted curve is in red.
The CBM fitted curve is in blue.

Three operating points from the binned data are shown as well as exact 95% confidence intervals for the lowest and uppermost operating points.

The next example shows composite plots for the Franken dataset (labeled D3) for treatment = 2 and reader = 3.

# i = 2 and j = 3 
plotsFR[[2,3]]

FIGURE 9.2: Composite plots for Franken dataset for treatment = 2, reader 3.

9.5 Accessing RSM parameters VD dataset

The parameters corresponding to the RSM plots for the Van Dyke dataset are accessed as follows:

resultsVD[[i,j]]$retRsm$mu is the RSM $\mu$ parameter for the Van Dyke dataset for treatment i and reader j;
resultsVD[[i,j]]$retRsm$lambda is the RSM $\lambda$ parameter;
resultsVD[[i,j]]$retRsm$nu is the RSM $\nu$ parameter;
resultsVD[[i,j]]$retRsm$zeta1 is the RSM $\zeta_1$ parameter.

For the Franken dataset one replaces resultsVD[[i,j]] with resultsFR[[i,j]].

9.5.1 RSM parameters Van Dyke dataset i = 1, j= 2

The following displays RSM parameters for the Van Dyke dataset, treatment 1 and reader 2:

## RSM parameters, Van Dyke Dataset, i=1, j=2: 
## mu =  2.201413 
## lambda =  0.2569453 
## nu =  0.7524016 
## zeta_1 =  -0.1097901 
## AUC =  0.8653694 
## sigma_AUC =  0.04740562 
## NLLini =  96.48516 
## NLLfin =  85.86244

9.5.2 RSM parameters Franken dataset i = 2, j= 3

Displayed next are RSM parameters for the Franken dataset, treatment 2 and reader 3:

## RSM parameters, Franken dataset, i=2, j=3: 
## mu =  2.641412 
## lambda =  2.137379 
## nu =  0.784759 
## zeta_1 =  -1.858565 
## AUC =  0.8552573 
## sigma_AUC =  0.03809136 
## NLLini =  132.6265 
## NLLfin =  127.9418

9.6 Accessing CBM parameters VD dataset

The parameters of the CBM plots are accessed as follows:

resultsVD[[i,j]]$retCbm$mu is the CBM $\mu$ parameter for treatment i and reader j;
resultsVD[[i,j]]$retCbm$alpha is the CBM $\alpha$ parameter;
as.numeric(resultsVD[[i,j]]$retCbm$zetas[1]) is the CBM $\zeta_1$ parameter, the threshold corresponding to the highest non-trivial operating point;
resultsVD[[i,j]]$retCbm$AUC is the CBM AUC;
as.numeric(resultsVD[[i,j]]$retCbm$StdAUC) is the standard deviation of the CBM AUC;
resultsVD[[i,j]]$retCbm$NLLIni is the initial value of negative log-likelihood;
resultsVD[[i,j]]$retCbm$NLLFin) is the final value of negative log-likelihood.

As before, for the Franken dataset one replaces resultsVD[[i,j]] with resultsFR[[i,j]].

The next example displays CBM parameters and AUC etc. for the Van Dyke dataset, treatment 1 and reader 2:

## CBM parameters, Van Dyke Dataset, i=1, j=2: 
## mu =  2.745791 
## alpha =  0.7931264 
## zeta_1 =  1.125028 
## AUC =  0.8758668 
## sigma_AUC =  0.03964492 
## NLLini =  86.23289 
## NLLfin =  85.88459

The next example displays CBM parameters for the Franken dataset, treatment 2 and reader 3:

## CBM parameters, Franken dataset, i=2, j=3: 
## mu =  2.324116 
## alpha =  0.8796571 
## zeta_1 =  -0.5599662 
## AUC =  0.8957135 
## sigma_AUC =  0.03223745 
## NLLini =  98.30823 
## NLLfin =  97.82786

The first three values are the fitted values for the CBM parameters $\mu$, $\alpha$ and $\zeta_1$. The next value is the AUC under the fitted CBM curve followed by its standard error. The last two values are the initial and final values of negative log-likelihood.

9.7 PROPROC parameters

For the VD dataset the PROPROC displayed parameters are accessed as follows:

resultsVD[[i,j]]$c1 is the PROPROC $c$ parameter for treatment i and reader j;
resultsVD[[i,j]]$da is the PROPROC $d_a$ parameter;
resultsVD[[i,j]]$aucProp is the PROPROC AUC;

Other statistics, such as standard error of AUC, are not provided by PROPROC software.

The next example displays PROPROC parameters for the Van Dyke dataset, treatment 1 and reader 2:

## PROPROC parameters, Van Dyke Dataset, i=1, j=2: 
## c =  -0.2809004 
## d_a =  1.731472 
## AUC =  0.8910714

The values are identical to those listed for treatment 1 and reader 2 in Fig. 9.8.

The next example displays PROPROC parameters for the Franken dataset, treatment 2 and reader 3:

## PROPROC parameters, Franken dataset, i=2, j=3: 
## c =  -0.3299822 
## d_a =  2.078543 
## AUC =  0.9304317

The next section provides an overview of the most salient findings from analyzing the datasets.

9.8 Overview of findings

With 14 datasets the total number of individual modality-reader combinations is 236 to each of which the three fitting algorithms were applied. It is easy to be overwhelmed by the numbers so this section summarizes an important conclusion:

The three fitting algorithms are consistent with a single algorithm-independent AUC.

If the AUCs of the three methods are identical the following relations hold with each slope $\text{m}_{PR}$ and $\text{m}_{CR}$ equal to unity:

\[\begin{equation} \left. \begin{aligned} \text{AUC}_{\text{PRO}} =& \text{m}_{PR} \text{AUC}_{\text{RSM}} \\ \text{AUC}_{\text{CBM}} =& \text{m}_{CR} \text{AUC}_{\text{RSM}} \end{aligned} \right \} \tag{9.1} \end{equation}\]

The abbreviations are as follows ($\text{AUC}_\text{PRO}$ = PROPROC AUC, $\text{AUC}_\text{CBM}$ = CBM AUC, $\text{AUC}_\text{RSM}$ = RSM AUC):

PR = PROPROC vs. RSM slope;
CR = CBM vs. RSM slope.

For each dataset the plot of PROPROC AUC vs. RSM AUC should be linear with zero intercept and slope $\text{m}_{PR}$, and likewise for the plots of CBM AUC vs. RSM AUC. The reason for the zero intercept is that if the AUCs are identical one cannot have an offset (i.e., intercept) term.

9.8.1 Slopes

Denote PROPROC AUC for dataset $f$, where $f=1,2,...,14$, treatment $i$ and reader $j$ by $\text{AUC}^{\text{PRO}}_{fij}$. The corresponding RSM and CBM values are denoted by $\text{AUC}^{\text{RSM}}_{fij}$ and $\text{AUC}^{\text{CBM}}_{fij}$, respectively.
For a given dataset the slope of the PROPROC AUC values vs. the RSM values is denoted $\text{m}_{\text{PR},f}$ and the grand average over all datasets is denoted $\text{m}^{\text{PR}}_\bullet$.
Likewise, the slope of the CBM AUC values vs. the RSM values is denoted $\text{m}_{\text{CR},f}$ and the grand average is denoted $\text{m}^{\text{CR}}_\bullet$.

A bootstrap analysis was conducted to determine, for each dataset, the slope of the constrained line fit (over all treatments and readers) and the corresponding confidence intervals. The code for calculating the slopes is in R/compare-3-fits/slopesConvVsRsm.R and that for the bootstrap confidence intervals is in R/compare-3-fits/slopesAucsConvVsRsmCI.R.

source(here::here("R/compare-3-fits/loadDataFile.R"))
source(here::here("R/compare-3-fits/slopesConvVsRsm.R"))
source(here::here("R/compare-3-fits/slopesAucsConvVsRsmCI.R"))

ret <- slopesConvVsRsm(datasetNames)
slopeCI <- slopesAucsConvVsRsmCI(datasetNames)

The call to function slopesConvVsRsm() returns ret, which contains, for each of 14 datasets, four lists: two plots and two slopes. For example:

PRO vs. RSM: ret$p1[[2]] is the slope plot for $\text{AUC}^{\text{PRO}}_2$ vs. $\text{AUC}^\text{RSM}_2$ for all treatments and readers in the Van Dyke dataset (f = 2).
CBM vs. RSM: ret$p2[[2]] is the slope plot for $\text{AUC}^{\text{CBM}}_2$ vs. $\text{AUC}^\text{RSM}_2$ for for all treatments and readers in the Van Dyke dataset (f = 2).
PRO vs. RSM: ret$m_pro_rsm has two length 14 columns: the slopes $\text{m}_{\text{PR},f}$ for the constrained linear fits of the PROPROC vs. RSM AUC values for each dataset and the corresponding $R^2$ values, where $R^2$ is the fraction of variance explained by the fit. The first column is ret$m_pro_rsm[[1]] and the second column is ret$m_pro_rsm[[2]].
CBM vs. RSM: ret$m_cbm_rsm has two two length 14 columns: the slopes $\text{m}_{\text{CR},f}$ for the constrained linear fits of the CBM vs. RSM AUC values and the corresponding $R^2$ values.

As an example, for the Van Dyke dataset, ret$p1[[2]] which is shown in the left in Fig. 9.3, is the plot of $\text{AUC}^{\text{PRO}}_2$ vs. $\text{AUC}^\text{RSM}_2$. Shown in the right is ret$p2[[2]], the plot of $\text{AUC}^{\text{CBM}}_2$ vs. $\text{AUC}^\text{RSM}_2$. Each plot has the zero-intercept linear fit superposed on the $i\times j = 10$ points where each point represents a distinct modality-reader combination.

Van Dyke dataset: Left plot is PROPROC-AUC vs. RSM-AUC with the superposed zero-intercept linear fit. The number of data points is `nPts` = 10. Right plot is CBM-AUC vs. RSM-AUC.

FIGURE 9.3: Van Dyke dataset: Left plot is PROPROC-AUC vs. RSM-AUC with the superposed zero-intercept linear fit. The number of data points is nPts = 10. Right plot is CBM-AUC vs. RSM-AUC.

The next plot shows corresponding plots for the Franken dataset in which there are $2\times 4 = 8$ points in each plot. The left plot results from ret$p1[[3]] (since f = 3 for this dataset) and the right plot from ret$p2[[3]].

FIGURE 9.4: Similar to previous plot, for Franken dataset.

9.8.2 Confidence intervals

The call to slopesAucsConvVsRsmCI returns slopeCI, containing the results of the bootstrap analysis (the bullet symbols $\bullet$ denote grand averages over 14 datasets):

slopeCI$cislopeProRsm 95-percent confidence interval for $\text{m}_{\text{PR} \bullet}$
slopeCI$cislopeCbmRsm 95-percent confidence interval for $\text{m}_{\text{CR} \bullet}$
slopeCI$histSlopeProRsm histogram of 200 bootstrap values of $\text{m}_{\text{PR} \bullet}$
slopeCI$histSlopeCbmRsm histogram of 200 bootstrap values of $\text{m}_{\text{CR} \bullet}$
slopeCI$ciAvgAucRsm confidence interval from 200 bootstrap values of $\text{AUC}^{\text{RSM}}_\bullet$
slopeCI$ciAvgAucPro confidence interval for 200 bootstrap values of $\text{AUC}^{\text{PRO}}_\bullet$
slopeCI$ciAvgAucCbm confidence interval for 200 bootstrap values of $\text{AUC}^{\text{CBM}}_\bullet$

As an example the following code displays in the first column the bootstrap CI for $\text{m}_{\text{PR} \bullet}$ (the slope, averaged over 14 datasets, of the PROPROC vs. RSM AUC values) and in the second column the CI for $\text{m}_{\text{CR} \bullet}$ (the averaged slope of the CBM vs. RSM AUC values):

df <- as.data.frame(slopeCI$cislopeProRsm)
df <- cbind(df, slopeCI$cislopeCbmRsm)
colnames(df) <- c("m-PR", "m-CR")
print(df)

##           m-PR      m-CR
## 2.5%  1.005092 0.9919886
## 97.5% 1.012285 0.9966149

The CI for $\text{m}_{\text{PR} \bullet}$ is slightly above unity, while that for $\text{m}_{\text{CR} \bullet}$ is slightly below. Note how close the CIs are to unity, implying $\text{m}_{\text{PR}} = \text{m}_{\text{CR}} = 1$.

Shown next is the histogram of 200 bootstrap value of $\text{m}_{\text{PR} \bullet}$ (left plot) and $\text{m}_{\text{CR} \bullet}$ (right plot). These histograms were used to compute the previously cited confidence intervals.

FIGURE 9.5: Histograms of slope PROPROC AUC vs. RSM AUC (left) and slope CBM AUC vs. RSM AUC (right).

9.8.3 Summary of slopes and confidence intervals

Table 9.1 summarizes the slopes and $R^2$ values for all 14 datasets.

TABLE 9.1: Summary of slopes and correlations for the two zero-intercept fits: PROPROC AUC vs. RSM AUC and CBM AUC vs. RSM AUC. The average of each slope equals unity to within 0.6 percent.
	$\text{m}_{PR}$	$R^2_{PR}$	$\text{m}_{CR}$	$R^2_{CR}$
TONY	1.0002	0.9997	0.9933	0.9997
VD	1.0061	0.9998	1.0007	1
FR	0.9995	1	0.9977	1
FED	1.0146	0.9998	0.9999	0.9999
JT	0.9964	0.9995	0.9972	1
MAG	1.036	0.9983	0.9953	1
OPT	1.0184	0.9997	1.0059	0.9997
PEN	1.0081	0.9996	0.9976	1
NICO	0.9843	0.9998	0.997	1
RUS	0.9989	0.9999	0.9921	0.9999
DOB1	1.0262	0.9963	0.9886	0.9962
DOB2	1.0056	0.9987	0.971	0.9978
DOB3	1.0211	0.998	0.9847	0.9986
FZR	1.0027	0.9999	0.9996	1

AVG	1.0084	0.9992	0.9943	0.9994
CI	(1.005, 1.012)	NA	(0.992, 0.997)	NA

The second column, labeled $\text{m}_{PR}$, lists the slopes of the straight line zero-intercept fits to PROPROC vs. RSM AUC values, for each of the 14 datasets, as labeled in the first column. The third column, labeled $R^2_{PR}$, lists the square of the correlation coefficient for each straight line zero-intercept fit.

The fourth and fifth columns list the corresponding values for the CBM AUC vs. RSM AUC fits.

The second last row lists the grand averages (AVG) and the last row lists the 95 percent confidence intervals.

9.9 Reason for equal AUCs / Summary

We find that all three proper ROC methods yield almost the same AUC. The reason is that the proper ROC is a consequence of using a decision variable that is equivalent (in the arbitrary monotonic increasing transformations sense, see below) to a likelihood ratio. Proper ROC fitting is discussed in Chapter 7. (Barrett and Myers 2013) show that an observer who uses the likelihood ratio, or any monotone increasing transformation of it, as the decision variable, has optimal performance, i.e., maximum ROC-AUC. An observer using the likelihood ratio as the decision variable is an ideal observer (Section 13.2.6 ibid). However, different ideal observers must yield the same AUC as otherwise one observer would be “more ideal” than another. Different models of fitting proper ROCs represent different approaches to modeling the decision variable and the likelihood ratio, but while the curves can have different shapes (the slope of the ROC curve at a given point equals the likelihood ratio calculated at that point) their AUCs must agree. This explains the empirical observation of this chapter that RSM, PROPROC and CBM all yield the same AUCs, as summarized in Table 9.1.

9.10 Appendices

9.10.1 Location of pre-analyzed PROPROC files

For each dataset PROPROC parameters were obtained by running the Windows software with PROPROC selected as the curve-fitting method. The results are saved to files that end with proprocnormareapooled.csv ¹⁵ contained in “R/compare-3-fits/MRMCRuns/C/”, where C denotes the name of the dataset (for example, for the Van Dyke dataset, C = “VD”). Examples are shown in the next two screen-shots.

FIGURE 9.6: Screen shot (1 of 2) of R/compare-3-fits/MRMCRuns showing the folders containing the results of PROPROC analysis on 14 datasets.

FIGURE 9.7: Screen shot (2 of 2) of R/compare-3-fits/MRMCRuns/VD showing files containing the results of PROPROC analysis for the Van Dyke dataset.

The contents of R/compare-3-fits/MRMCRuns/VD/VDproprocnormareapooled.csv are shown next, see Fig. 9.8. ¹⁶ The PROPROC parameters $c$ and $d_a$ are in the last two columns. The column names are T = treatment; R = reader; return-code = undocumented value, area = PROPROC AUC; numCAT = number of ROC bins; adjPMean = undocumented value; c = $c$ and d_a = $d_a$, are the PROPROC parameters defined in (Charles E. Metz and Pan 1999).

PROPROC output for the Van Dyke ROC data set. The first column is the treatment, the second is the reader, the fourth is the AUC and the last two columns are the c and $d_a$ parameters.

FIGURE 9.8: PROPROC output for the Van Dyke ROC data set. The first column is the treatment, the second is the reader, the fourth is the AUC and the last two columns are the c and $d_a$ parameters.

9.10.2 Location of pre-analyzed results

The following screen shot shows the pre-analyzed files created by the function Compare3ProperRocFits() described below. Each file is named allResultsC, where C is the abbreviated name of the dataset (uppercase C denotes one or more uppercase characters; for example, C = VD denotes the Van Dyke dataset.).

FIGURE 9.9: Screen shot of R/compare-3-fits/RSM6 showing the results files created by Compare3ProperRocFits().

9.10.3 Plots for the Van Dyke dataset

The following plots are arranged in pairs, with the left plot corresponding to treatment 1 and the right to treatment 2.

FIGURE 9.10: Composite plots in both treatments for the Van Dyke dataset, reader 1.

FIGURE 9.11: Composite plots in both treatments for the Van Dyke dataset, reader 2. For treatment 2 the RSM and PROPROC fits are indistinguishable.

The RSM parameter values for the treatment 2 plot are: $\mu$ = 5.767237, $\lambda$ = 2.7212621, $\nu$ = 0.8021718, $\zeta_1$ = -1.5717303. The corresponding CBM values are $\mu$ = 5.4464738, $\alpha$ = 0.8023609, $\zeta_1$ = -1.4253826. The RSM and CBM $\mu$ parameters are close and likewise the RSM $\nu$ and CBM $\alpha$ parameters are close - this is because they have similar physical meanings, which is investigated later in this chapter TBA. [The CBM does not have a parameter analogous to the RSM $\lambda$ parameter.]

FIGURE 9.12: Composite plots in both treatments for the Van Dyke dataset, reader 3.

The RSM parameters for the treatment 1 plot are: $\mu$ = 3.1527627, $\lambda$ = 9.9986154, $\nu$ = 0.9899933, $\zeta_1$ = 1.1733988. The corresponding CBM values are $\mu$ = 2.1927712, $\alpha$ = 0.98, $\zeta_1$ = -0.5168848.

Composite plots in both treatments for the Van Dyke dataset, reader 4. For treatment 2 the 3 plots are indistinguishable and each one has AUC = 1. The degeneracy is due to all operating points being on the axes of the unit square.

FIGURE 9.13: Composite plots in both treatments for the Van Dyke dataset, reader 4. For treatment 2 the 3 plots are indistinguishable and each one has AUC = 1. The degeneracy is due to all operating points being on the axes of the unit square.

FIGURE 9.14: Composite plots in both treatments for the Van Dyke dataset, reader 5.

9.10.4 Plots for the Franken dataset

The following plots are arranged in pairs, with the left plot corresponding to treatment 1 and the right to treatment 2. These plots apply to the Franken dataset.

FIGURE 9.15: Composite plots in both treatments for the Franken dataset, reader 1.

FIGURE 9.16: Composite plots in both treatments for the Franken dataset, reader 2.

FIGURE 9.17: Composite plots in both treatments for the Franken dataset, reader 3.

FIGURE 9.18: Composite plots in both treatments for the Franken dataset, reader 4.

REFERENCES

Barrett, Harrison H, and Kyle J Myers. 2013. Foundations of Image Science. John Wiley & Sons.

Chakraborty, Dev, and Xuetong Zhai. 2022. RJafroc: Artificial Intelligence Systems and Observer Performance. https://dpc10ster.github.io/RJafroc/.

Chakraborty, DP, and Tony Svahn. 2011. “Estimating the Parameters of a Model of Visual Search from ROC Data: An Alternate Method for Fitting Proper ROC Curves.” In Medical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment, 7966:189–97. SPIE.

Metz, Charles E, and Xiaochuan Pan. 1999. “‘Proper’ Binormal ROC Curves: Theory and Maximum-Likelihood Estimation.” Journal of Mathematical Psychology 43 (1): 1–33.

Sept. 04, 2014; the version used in this chapter is no longer distributed.↩︎
In accordance with R-package policies white-spaces in the original PROPROC output file names have been removed.↩︎
The VD.lrc file in this directory is the Van Dyke data formatted for input to OR DBM-MRMC 2.5.↩︎