Introduction

  • This vignette is intended for those seeking a quick transition from Windows JAFROC to RJafroc.
  • Described first is the structure of an RJafroc dataset followed by how to read a JAFROC format Excel file to create an RJafroc dataset.

An ROC dataset

Dataset dataset03 corresponding to the Franken ROC data (Franken et al. 1992) is predefined. The following code shows the structure of this dataset.

str(dataset03)
#> List of 3
#>  $ ratings     :List of 3
#>   ..$ NL   : num [1:2, 1:4, 1:100, 1] 3 3 4 3 3 3 4 1 1 3 ...
#>   ..$ LL   : num [1:2, 1:4, 1:67, 1] 5 5 4 4 5 4 4 5 2 2 ...
#>   ..$ LL_IL: logi NA
#>  $ lesions     :List of 3
#>   ..$ perCase: num [1:67] 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ IDs    : num [1:67, 1] 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ weights: num [1:67, 1] 1 1 1 1 1 1 1 1 1 1 ...
#>  $ descriptions:List of 7
#>   ..$ fileName     : chr "dataset03"
#>   ..$ type         : chr "ROC"
#>   ..$ name         : chr "FRANKEN"
#>   ..$ truthTableStr: num [1:2, 1:4, 1:100, 1:2] 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ design       : chr "FCTRL"
#>   ..$ modalityID   : Named chr [1:2] "TREAT1" "TREAT2"
#>   .. ..- attr(*, "names")= chr [1:2] "TREAT1" "TREAT2"
#>   ..$ readerID     : Named chr [1:4] "READER_1" "READER_2" "READER_3" "READER_4"
#>   .. ..- attr(*, "names")= chr [1:4] "READER_1" "READER_2" "READER_3" "READER_4"
  • It is a list with 8 members. The false positive ratings are contained in {NL}, an array with dimensions [1:2,1:4,1:100,1]. The first index corresponds to treatments, and since the dataset has 2 treatments, the corresponding dimension is 2. The second index corresponds to readers, and since the dataset has 4 readers, the corresponding dimension is 4. The third index corresponds to the total number of cases. Since the dataset has 100 cases, the corresponding dimension is 100. But, as you can see from the code below, the entries in this array for cases 34 through 100 are -Inf: i.e., all(dataset03$NL[1,1,34:100,1] == -Inf) = TRUE.

  • This is because in the ROC paradigm false positive are not possible on diseased cases. So the actual FP ratings are contained in the first 33 elements of the array. How did I know that there are 33 non-diseased cases? This can be understood in several ways.

  • LL is an array with dimensions [1:2,1:4,1:67,1]. This implies 67 diseased cases, and by subtraction from 100, there must be 33 non-diseased cases.

  • The list member lesionVector is a vector with length 67, implying 33 non-diseased cases.

  • The list members lesionID and lesionWeight are arrays with dimensions [1:67,1] containing ones. Again, these imply 67 diseased cases.

  • The fields lesionVector, lesionID and lesionWeight, while not needed for ROC data, are needed for the FROC paradigm.

The dataType list member is the character string "ROC", characterizing the ROC dataset.

dataset03$dataType
#> NULL

The modalityID list member is a character string with two entries, "TREAT1" and "TREAT2", corresponding to the two modalities.

dataset03$modalityID
#> NULL

The readerID list member is a character string with four entries, "READER_1", "READER_2", "READER_3" and "READER_4" corresponding to the four readers.

dataset03$readerID
#> NULL

Here are the actual ratings for cases 1:34.

dataset03$NL[1,1,1:33,1]
#> NULL
  • This says that for treatment 1 and reader 1, (non-diseased) case 1 was rated 3, case 2 was rated 1, cases 3-7 were rated 2, case 8 was rated 4, etc.

  • As another example, for treatment 2 and reader 3, the FP ratings are:

dataset03$NL[2,3,1:33,1]
#> NULL

Creating a dataset from a JAFROC format file

There is a file RocData.xlsx that is part of the package installation. Since it is a system file one must get its name as follows.

fileName <- "RocData.xlsx"
sysFileName <- system.file(paste0("extdata/",fileName), package = "RJafroc", mustWork = TRUE)

Next, one uses DfReadDataFile() as follows, assuming it is a JAFROC format file.

ds <- DfReadDataFile(sysFileName, newExcelFileFormat = FALSE)
str(ds)
#> List of 3
#>  $ ratings     :List of 3
#>   ..$ NL   : num [1:2, 1:5, 1:114, 1] 1 3 2 3 2 2 1 2 3 2 ...
#>   ..$ LL   : num [1:2, 1:5, 1:45, 1] 5 5 5 5 5 5 5 5 5 5 ...
#>   ..$ LL_IL: logi NA
#>  $ lesions     :List of 3
#>   ..$ perCase: int [1:45] 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ IDs    : num [1:45, 1] 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ weights: num [1:45, 1] 1 1 1 1 1 1 1 1 1 1 ...
#>  $ descriptions:List of 7
#>   ..$ fileName     : logi NA
#>   ..$ type         : chr "ROC"
#>   ..$ name         : logi NA
#>   ..$ truthTableStr: num [1:2, 1:5, 1:114, 1:2] 1 1 1 1 1 1 1 1 1 1 ...
#>   ..$ design       : chr "FCTRL"
#>   ..$ modalityID   : Named chr [1:2] "0" "1"
#>   .. ..- attr(*, "names")= chr [1:2] "0" "1"
#>   ..$ readerID     : Named chr [1:5] "0" "1" "2" "3" ...
#>   .. ..- attr(*, "names")= chr [1:5] "0" "1" "2" "3" ...

Analysis is illustrated for dataset03, but one could have used the newly created dataset ds.

Analyzing the ROC dataset

This illustrates the StSignificanceTesting() function. The significance testing method is specified as "DBM" and the figure of merit FOM is specified as “Wilcoxon”.

ret <- StSignificanceTesting(dataset03, FOM = "Wilcoxon", method = "DBM")
print(ret)
#> $FOMs
#> $FOMs$foms
#>           rdrREADER_1 rdrREADER_2 rdrREADER_3 rdrREADER_4
#> trtTREAT1  0.85345997  0.86499322  0.85730439  0.81524197
#> trtTREAT2  0.84961556  0.84350972  0.84011759  0.81433740
#> 
#> $FOMs$trtMeans
#>             Estimate
#> trtTREAT1 0.84774989
#> trtTREAT2 0.83689507
#> 
#> $FOMs$trtMeanDiffs
#>                        Estimate
#> trtTREAT1-trtTREAT2 0.010854817
#> 
#> 
#> $ANOVA
#> $ANOVA$TRCanova
#>                  SS  DF           MS
#> T       0.023565410   1 0.0235654097
#> R       0.205217999   3 0.0684059998
#> C      52.528398680  99 0.5305898857
#> TR      0.015060792   3 0.0050202641
#> TC      6.410048814  99 0.0647479678
#> RC     39.242953812 297 0.1321311576
#> TRC    22.660077641 297 0.0762965577
#> Total 121.085323149 799           NA
#> 
#> $ANOVA$VarCom
#>             Estimates
#> VarR    3.7755679e-05
#> VarC    5.1250915e-02
#> VarTR  -7.1276294e-04
#> VarTC  -2.8871475e-03
#> VarRC   2.7917300e-02
#> VarErr  7.6296558e-02
#> 
#> $ANOVA$IndividualTrt
#>       DF   TrtTREAT1   TrtTREAT2
#> msR    3 0.049266349 0.024159915
#> msC   99 0.293967531 0.301370323
#> msRC 297 0.105047872 0.103379843
#> 
#> $ANOVA$IndividualRdr
#>      DF   rdrREADER_1 rdrREADER_2 rdrREADER_3   rdrREADER_4
#> msT   1 0.00073897606 0.023077021 0.014769293 0.00004091217
#> msC  99 0.20387477465 0.223441908 0.214246773 0.28541990211
#> msTC 99 0.09155873437 0.080279256 0.061228980 0.06057067104
#> 
#> 
#> $RRRC
#> $RRRC$FTests
#>           DF           MS     FStat          p
#> Treatment  1 0.0235654097 4.6940577 0.11883786
#> Error      3 0.0050202641        NA         NA
#> 
#> $RRRC$ciDiffTrt
#>                        Estimate       StdErr DF         t      PrGTt
#> trtTREAT1-trtTREAT2 0.010854817 0.0050101218  3 2.1665774 0.11883786
#>                           CILower     CIUpper
#> trtTREAT1-trtTREAT2 -0.0050896269 0.026799261
#> 
#> $RRRC$ciAvgRdrEachTrt
#>             Estimate      StdErr         DF    CILower    CIUpper
#> trtTREAT1 0.84774989 0.024402152  70.121788 0.79908282 0.89641696
#> trtTREAT2 0.83689507 0.023566416 253.644028 0.79048429 0.88330585
#> 
#> 
#> $FRRC
#> $FRRC$FTests
#>           DF          MS      FStat          p
#> Treatment  1 0.023565410 0.36395597 0.54769704
#> Error     99 0.064747968         NA         NA
#> 
#> $FRRC$ciDiffTrt
#>                        Estimate      StdErr DF          t      PrGTt
#> trtTREAT1-trtTREAT2 0.010854817 0.017992772 99 0.60328764 0.54769704
#>                          CILower    CIUpper
#> trtTREAT1-trtTREAT2 -0.024846746 0.04655638
#> 
#> $FRRC$ciAvgRdrEachTrt
#>             Estimate      StdErr DF    CILower    CIUpper
#> trtTREAT1 0.84774989 0.027109386 99 0.79395898 0.90154079
#> trtTREAT2 0.83689507 0.027448603 99 0.78243109 0.89135905
#> 
#> $FRRC$ciDiffTrtEachRdr
#>                                       Estimate      StdErr DF           t
#> rdrREADER_1::trtTREAT1-trtTREAT2 0.00384441429 0.042792227 99 0.089839080
#> rdrREADER_2::trtTREAT1-trtTREAT2 0.02148349163 0.040069753 99 0.536152334
#> rdrREADER_3::trtTREAT1-trtTREAT2 0.01718679331 0.034993994 99 0.491135520
#> rdrREADER_4::trtTREAT1-trtTREAT2 0.00090456807 0.034805365 99 0.025989329
#>                                       PrGTt      CILower     CIUpper
#> rdrREADER_1::trtTREAT1-trtTREAT2 0.92859660 -0.081064648 0.088753476
#> rdrREADER_2::trtTREAT1-trtTREAT2 0.59305592 -0.058023592 0.100990575
#> rdrREADER_3::trtTREAT1-trtTREAT2 0.62441761 -0.052248882 0.086622469
#> rdrREADER_4::trtTREAT1-trtTREAT2 0.97931817 -0.068156827 0.069965963
#> 
#> 
#> $RRFC
#> $RRFC$FTests
#>           DF           MS     FStat          p
#> Treatment  1 0.0235654097 4.6940577 0.11883786
#> Error      3 0.0050202641        NA         NA
#> 
#> $RRFC$ciDiffTrt
#>                        Estimate       StdErr DF         t      PrGTt
#> trtTREAT1-trtTREAT2 0.010854817 0.0050101218  3 2.1665774 0.11883786
#>                           CILower     CIUpper
#> trtTREAT1-trtTREAT2 -0.0050896269 0.026799261
#> 
#> $RRFC$ciAvgRdrEachTrt
#>             Estimate      StdErr DF    CILower    CIUpper
#> trtTREAT1 0.84774989 0.011098012  3 0.81243106 0.88306871
#> trtTREAT2 0.83689507 0.007771730  3 0.81216196 0.86162818

Explanation of the output

The function returns a long unwieldy list. Let us consider them one by one. The function UtilOutputReport(), which can generate an Excel file report, making it much easier to visualize the results, is described in another vignette.

FOMs

  • fomArray contains the [1:2,1:4] FOM values.
ret$fomArray
#> NULL

This shows the 2 x 4 array of FOM values.

Pseudovalue ANOVA table

  • anovaY, where the Y denotes that these are pseudovalue based, is the ANOVA table.
ret$anovaY
#> NULL

Pseudovalue ANOVA table, each treatment

  • anovaYi is the ANOVA table for individual treatments.
ret$anovaYi
#> NULL

The 0 and 1 headers come from the treatment names.

Pseudovalue Variance Components

  • varComp is the variance components (needed for sample size estimation).
ret$varComp
#> NULL

Random-reader random-case (RRRC) analysis

  • ret$FTestStatsRRRC$fRRRC is the F-statistic for testing the NH that the treatments have identical FOMs. RRRC means random-reader random-case generalization.
ret$FTestStatsRRRC$fRRRC
#> NULL

F-statistic and p-value for RRRC analysis

  • ret$FTestStatsRRRC$ddfRRRC is the denominator degrees of freedom of the F-statistic.
ret$FTestStatsRRRC$ddfRRRC
#> NULL
  • ret$FTestStatsRRRC$pRRRC is the p-value of the test.
ret$FTestStatsRRRC$pRRRC
#> NULL

Confidence Intervals for RRRC analysis

  • ciDiffTrtRRRC is the 95% confidence interval of reader-averaged differences between treatments.
ret$ciDiffTrtRRRC
#> NULL
  • ciAvgRdrEachTrtRRRC is the 95% confidence interval of reader-averaged FOMs for each treatments.
ret$ciAvgRdrEachTrtRRRC
#> NULL

Fixed-reader random-case (FRRC) analysis

F-statistic and p-value for RRRC analysis

  • ret$FTestStatsFRRC$fFRRC is the F-statistic for fixed-reader random-case analysis.
ret$FTestStatsFRRC$fFRRC
#> NULL
  • ret$FTestStatsFRRC$ndfFRRC is the numerator degrees of freedom of the F-statistic, always one less than the number of treatments.
ret$FTestStatsFRRC$ndfFRRC
#> NULL
  • ret$FTestStatsFRRC$ddfFRRC is the denominator degreesof freedom of the F-statistic, for fixed-reader random-case analysis.
ret$FTestStatsFRRC$ddfFRRC
#> NULL
  • ret$FTestStatsFRRC$pFRRC is the p-value for fixed-reader random-case analysis.
ret$FTestStatsFRRC$pFRRC
#> NULL

Confidence Intervals for FRRC analysis

  • ciDiffTrtFRRC is the 95% CI of reader-average differences between treatments for fixed-reader random-case analysis
ret$ciDiffTrtFRRC
#> NULL
  • ret$ciAvgRdrEachTrtFRRC is the 95% CI of reader-average FOMs of each treatment for fixed-reader random-case analysis
ret$ciAvgRdrEachTrtFRRC
#> NULL

ANOVA for FRRC analysis

  • ret$msAnovaEachRdrFRRC is the mean-squares ANOVA for each reader
ret$msAnovaEachRdrFRRC
#> NULL

Confidence Intervals for FRRC analysis

  • ciDiffTrtFRRC is the CI for reader-averaged treatment differences, for fixed-reader random-case analysis
ret$ciDiffTrtFRRC
#> NULL

Random-reader fixed-case (RRFC) analysis

F-statistic and p-value for RRFC analysis

  • ret$FTestStatsRRFC$fRRFC is the F-statistic for for random-reader fixed-case analysis
ret$FTestStatsRRFC$fRRFC
#> NULL
  • ret$FTestStatsRRFC$ddfRRFC is the ddf for for random-reader fixed-case analysis
ret$FTestStatsRRFC$ddfRRFC
#> NULL
  • ret$FTestStatsRRFC$pRRFC is the p-value for for random-reader fixed-case analysis
ret$FTestStatsRRFC$pRRFC
#> NULL

Confidence Intervals for RRFC analysis

  • ciDiffTrtRRFC is the CI for reader-averaged inter-treatment FOM differences for random-reader fixed-case analysis
ret$ciDiffTrtRRFC
#> NULL
  • ciAvgRdrEachTrtRRFC is the CI for treatment FOMs for each reader for random-reader fixed-case analysis
ret$ciAvgRdrEachTrtRRFC
#> NULL

OR significance testing

Simply change method = "DBM" to method = "OR".

ret <- StSignificanceTesting(dataset03, FOM = "Wilcoxon", method = "OR")
str(ret)
#> List of 5
#>  $ FOMs :List of 3
#>   ..$ foms        :'data.frame': 2 obs. of  4 variables:
#>   .. ..$ rdrREADER_1: num [1:2] 0.853 0.85
#>   .. ..$ rdrREADER_2: num [1:2] 0.865 0.844
#>   .. ..$ rdrREADER_3: num [1:2] 0.857 0.84
#>   .. ..$ rdrREADER_4: num [1:2] 0.815 0.814
#>   ..$ trtMeans    :'data.frame': 2 obs. of  1 variable:
#>   .. ..$ Estimate: num [1:2] 0.848 0.837
#>   ..$ trtMeanDiffs:'data.frame': 1 obs. of  1 variable:
#>   .. ..$ Estimate: num 0.0109
#>  $ ANOVA:List of 4
#>   ..$ TRanova      :'data.frame':    3 obs. of  3 variables:
#>   .. ..$ SS: num [1:3] 0.000236 0.002052 0.000151
#>   .. ..$ DF: num [1:3] 1 3 3
#>   .. ..$ MS: num [1:3] 2.36e-04 6.84e-04 5.02e-05
#>   ..$ VarCom       :'data.frame':    6 obs. of  2 variables:
#>   .. ..$ Estimates: num [1:6] 2.33e-05 -6.84e-04 7.92e-04 4.84e-04 5.13e-04 ...
#>   .. ..$ Rhos     : num [1:6] NA NA 0.519 0.317 0.336 ...
#>   ..$ IndividualTrt:'data.frame':    2 obs. of  4 variables:
#>   .. ..$ DF         : num [1:2] 3 3
#>   .. ..$ msREachTrt : num [1:2] 0.000493 0.000242
#>   .. ..$ varEachTrt : num [1:2] 0.00152 0.00153
#>   .. ..$ cov2EachTrt: num [1:2] 0.000472 0.000495
#>   ..$ IndividualRdr:'data.frame':    4 obs. of  4 variables:
#>   .. ..$ DF         : num [1:4] 1 1 1 1
#>   .. ..$ msTEachRdr : num [1:4] 7.39e-06 2.31e-04 1.48e-04 4.09e-07
#>   .. ..$ varEachRdr : num [1:4] 0.00148 0.00152 0.00138 0.00173
#>   .. ..$ cov1EachRdr: num [1:4] 0.000562 0.000716 0.000765 0.001124
#>  $ RRRC :List of 3
#>   ..$ FTests         :'data.frame':  2 obs. of  4 variables:
#>   .. ..$ DF   : num [1:2] 1 3
#>   .. ..$ MS   : num [1:2] 2.36e-04 5.02e-05
#>   .. ..$ FStat: num [1:2] 4.69 NA
#>   .. ..$ p    : num [1:2] 0.119 NA
#>   ..$ ciDiffTrt      :'data.frame':  1 obs. of  7 variables:
#>   .. ..$ Estimate: num 0.0109
#>   .. ..$ StdErr  : num 0.00501
#>   .. ..$ DF      : num 3
#>   .. ..$ t       : num 2.17
#>   .. ..$ PrGTt   : num 0.119
#>   .. ..$ CILower : num -0.00509
#>   .. ..$ CIUpper : num 0.0268
#>   ..$ ciAvgRdrEachTrt:'data.frame':  2 obs. of  6 variables:
#>   .. ..$ Estimate: num [1:2] 0.848 0.837
#>   .. ..$ StdErr  : num [1:2] 0.0244 0.0236
#>   .. ..$ DF      : num [1:2] 70.1 253.6
#>   .. ..$ CILower : num [1:2] 0.799 0.79
#>   .. ..$ CIUpper : num [1:2] 0.896 0.883
#>   .. ..$ Cov2    : num [1:2] 0.000472 0.000495
#>  $ FRRC :List of 5
#>   ..$ FTests              :'data.frame': 2 obs. of  4 variables:
#>   .. ..$ MS   : num [1:2] 0.000236 0.000734
#>   .. ..$ Chisq: num [1:2] 0.321 NA
#>   .. ..$ DF   : num [1:2] 1 NA
#>   .. ..$ p    : num [1:2] 0.571 NA
#>   ..$ ciDiffTrt           :'data.frame': 1 obs. of  6 variables:
#>   .. ..$ Estimate: num 0.0109
#>   .. ..$ StdErr  : num 0.0192
#>   .. ..$ z       : num 0.567
#>   .. ..$ PrGTz   : num 0.571
#>   .. ..$ CILower : num -0.0267
#>   .. ..$ CIUpper : num 0.0484
#>   ..$ ciAvgRdrEachTrt     :'data.frame': 2 obs. of  5 variables:
#>   .. ..$ Estimate: num [1:2] 0.848 0.837
#>   .. ..$ StdErr  : num [1:2] 0.0271 0.0274
#>   .. ..$ DF      : num [1:2] 99 99
#>   .. ..$ CILower : num [1:2] 0.795 0.783
#>   .. ..$ CIUpper : num [1:2] 0.901 0.891
#>   ..$ ciDiffTrtEachRdr    :'data.frame': 4 obs. of  6 variables:
#>   .. ..$ Estimate: num [1:4] 0.003844 0.021483 0.017187 0.000905
#>   .. ..$ StdErr  : num [1:4] 0.0428 0.0401 0.035 0.0348
#>   .. ..$ z       : num [1:4] 0.0898 0.5362 0.4911 0.026
#>   .. ..$ PrGTz   : num [1:4] 0.928 0.592 0.623 0.979
#>   .. ..$ CILower : num [1:4] -0.08 -0.0571 -0.0514 -0.0673
#>   .. ..$ CIUpper : num [1:4] 0.0877 0.1 0.0858 0.0691
#>   ..$ IndividualRdrVarCov1:'data.frame': 4 obs. of  2 variables:
#>   .. ..$ varEachRdr : num [1:4] 0.00148 0.00152 0.00138 0.00173
#>   .. ..$ cov1EachRdr: num [1:4] 0.000562 0.000716 0.000765 0.001124
#>  $ RRFC :List of 3
#>   ..$ FTests         :'data.frame':  2 obs. of  4 variables:
#>   .. ..$ DF: num [1:2] 1 3
#>   .. ..$ MS: num [1:2] 2.36e-04 5.02e-05
#>   .. ..$ F : num [1:2] 4.69 NA
#>   .. ..$ p : num [1:2] 0.119 NA
#>   ..$ ciDiffTrt      :'data.frame':  1 obs. of  7 variables:
#>   .. ..$ Estimate: num 0.0109
#>   .. ..$ StdErr  : num 0.00501
#>   .. ..$ DF      : num 3
#>   .. ..$ t       : num 2.17
#>   .. ..$ PrGTt   : num 0.119
#>   .. ..$ CILower : num -0.00509
#>   .. ..$ CIUpper : num 0.0268
#>   ..$ ciAvgRdrEachTrt:'data.frame':  2 obs. of  5 variables:
#>   .. ..$ Estimate: num [1:2] 0.848 0.837
#>   .. ..$ StdErr  : num [1:2] 0.0111 0.00777
#>   .. ..$ DF      : num [1:2] 3 3
#>   .. ..$ CILower : num [1:2] 0.812 0.812
#>   .. ..$ CIUpper : num [1:2] 0.883 0.862

References

Franken, Jr., Edmund A., Kevin S. Berbaum, Susan M. Marley, Wilbur L. Smith, Yutaka Sato, Simon C. S. Kao, and Steven G. Milam. 1992. “Evaluation of a Digital Workstation for Interpreting Neonatal Examinations: A Receiver Operating Characteristic Study.” Journal Article. Investigative Radiology 27 (9): 732–37. http://journals.lww.com/investigativeradiology/Fulltext/1992/09000/Evaluation_of_a_Digital_Workstation_for.16.aspx.