Chapter 41 Split Plot Study Design

41.1 TBA How much finished

10%

41.2 Mean Square R(T)

R(T) is read as “reader nested within treatment” (Hillis 2014).

\[\begin{equation} \text{MS[R(T)]}=\frac{1}{I(J-1)}\sum_{i=1}^{I}\sum_{j=1}^{J}\left ( \theta_{ij} - \theta_{i\bullet} \right )^{2} \tag{41.1} \end{equation}\]

\[\begin{equation} \text{MS[R(T)]}=\frac{1}{I}\sum_{i=1}^{I}\frac{1}{J_i-1}\sum_{j=1}^{J}\left ( \theta_{ij} - \theta_{i\bullet} \right )^{2} \tag{41.2} \end{equation}\]

41.3 References

Alberdi, Eugenio, Andrey A Povyakalo, Lorenzo Strigini, Peter Ayton, and Rosalind Given-Wilson. 2008. “CAD in Mammography: Lesion-Level Versus Case-Level Analysis of the Effects of Prompts on Human Decisions.” International Journal of Computer Assisted Radiology and Surgery 3 (1-2): 115–22.

Bamber, Donald. 1975. “The Area Above the Ordinal Dominance Graph and the Area Below the Receiver Operating Characteristic Graph.” Journal Article. Journal of Mathematical Psychology 12 (4): 387–415. https://doi.org/10.1016/0022-2496(75)90001-2.

Barlow, William E., Chen Chi, Patricia A. Carney, Stephen H. Taplin, Carl D’Orsi, Gary Cutter, R. Edward Hendrick, and Joann G. Elmore. 2004. “Accuracy of Screening Mammography Interpretation by Characteristics of Radiologists.” Journal Article. Journal of the National Cancer Institute 96 (24): 1840–50. https://doi.org/10.1093/jnci/djh333.

Barnes, GT, EA Sabbagh, DP Chakraborty, PH Nath, RF Luna, C Sanders, and RG Fraser. 1989. “A Comparison of Dual-Energy Digital Radiography and Screen-Film Imaging in the Detection of Subtle Interstitial Pulmonary Disease.” Investigative Radiology 24 (8): 585–91. https://doi.org/10.1097/00004424-198908000-00003.

Beam, C. A., Peter M. Layde, and Daniel C. Sullivan. 1996. “Variability in the Interpretation of Screening Mammograms by Us Radiologists. Findings from a National Sample.” Journal Article. Archives of Internal Medicine 156 (2): 209–13.

Berbaum, Kevin S., Donald D. Dorfman, E. A. Franken, and Robert T. Caldwell. 2002. “An Empirical Comparison of Discrete Ratings and Subjective Probability Ratings.” Journal Article. Academic Radiology 9 (7): 756–63. https://doi.org/10.1016/s1076-6332(03)80344-6.

Black, William C. 2000. “Anatomic Extent of Disease: A Critical Variable in Reports of Diagnostic Accuracy.” Journal Article. Radiology 217 (2): 319–20. http://radiology.rsnajnls.org.

Black, William C., and Andrew J. Dwyer. 1990. “Local Versus Global Measures of Accuracy: An Important Distinction for Diagnostic Imaging.” Journal Article. Med Decis Making 10 (4): 266–73. https://doi.org/10.1177/0272989x9001000404.

Bochud, F. O., C. K. Abbey, and M. P. Eckstein. 1999. “Visual Signal Detection in Structured Backgrounds Iv, Calculation of Figures of Merit for Model Observers in Non-Stationary Backgrounds.” Journal Article. Journal of the Optical Society of America, A, Optics, Image Science, and Vision 17 (2): 206–17.

Bolker, Ben, and R Development Core Team. 2020. Bbmle: Tools for General Maximum Likelihood Estimation. https://CRAN.R-project.org/package=bbmle.

Broyden, Charles George. 1970. “The Convergence of a Class of Double-Rank Minimization Algorithms 1. General Considerations.” Journal Article. IMA Journal of Applied Mathematics 6 (1): 76–90.

Bunch, Philip C, John F Hamilton, Gary K Sanderson, and Arthur H Simmons. 1977. “A Free Response Approach to the Measurement and Characterization of Radiographic Observer Performance.” In Application of Optical Instrumentation in Medicine Vi, 127:124–35. International Society for Optics; Photonics.

Bunch, Phillip C., J. F. Hamilton, G. K. Sanderson, and A. H. Simmons. 1977. “A Free-Response Approach to the Measurement and Characterization of Radiographic-Observer Performance.” Journal Article. Proc. SPIE 127: 124–35.

Burgess, Arthur E. 2011. “Visual Perception Studies and Observer Models in Medical Imaging.” In Seminars in Nuclear Medicine, 41:419–36. 6. Elsevier.

Chakraborty, Dev P. 2017. Observer Performance Methods for Diagnostic Imaging: Foundations, Modeling, and Applications with R-Based Examples. Boca Raton, FL: CRC Press.

Chakraborty, Dev P. 2002. “Statistical Power in Observer Performance Studies: A Comparison of the ROC and Free-Response Methods in Tasks Involving Localization.” Journal Article. Acad. Radiol. 9 (2): 147–56. https://doi.org/10.1016/s1076-6332(03)80164-2.

———. 1989. “Maximum Likelihood Analysis of Free-Response Receiver Operating Characteristic (Froc) Data.” Medical Physics 16 (4): 561–68.

———. 2006a. “An Alternate Method for Using a Visual Discrimination Model (Vdm) to Optimize Softcopy Display Image Quality.” Journal Article. Journal of the Society for Information Display 14 (10): 921–26.

———. 2006b. “A Search Model and Figure of Merit for Observer Data Acquired According to the Free-Response Paradigm.” Journal Article. Phys. Med. Biol. 51: 3449–62.

———. 2008. “Validation and Statistical Power Comparison of Methods for Analyzing Free-Response Observer Performance Studies.” Journal Article. Acad Radiol 15 (12): 1554–66. http://www.sciencedirect.com/science/article/B75BK-4TW6D0R-9/2/8f59ae9ff4ba7d2aa596076694b7de09.

Chakraborty, Dev P., and K. S. Berbaum. 2004. “Observer Studies Involving Detection and Localization: Modeling, Analysis and Validation.” Journal Article. Med Phys 31 (8): 2313–30.

Chakraborty, Dev, Peter Philips, and Xuetong Zhai. 2020. RJafroc: Analyzing Diagnostic Observer Performance Studies. https://dpc10ster.github.io/RJafroc/.

Chakraborty, Dev, Peter Phillips, and Xuetong Zhai. 2020. RJafroc: Artificial Intelligence Systems and Observer Performance. https://dpc10ster.github.io/RJafroc/.

Chakraborty, Dev Prasad. 2010. “Prediction Accuracy of a Sample-Size Estimation Method for ROC Studies.” Journal Article. Academic Radiology 17: 628–38. https://doi.org/10.1016/j.acra.2010.01.007.

Chakraborty, Dev P., M. Sivarudrappa, and H. Roehrig. 1999. “Computerized Measurement of Mammographic Display Image Quality.” Conference Proceedings. In Proc Spie Medical Imaging 1999: Physics of Medical Imaging, edited by John M. Boone John M. Boone; James T. Dobbins III, 3659:131–41. SPIE.

Chakraborty, Dev P., and T. Svahn. 2011. “Estimating the Parameters of a Model of Visual Search from ROC Data: An Alternate Method for Fitting Proper ROC Curves.” Journal Article. Proc. SPIE 7966 7966. https://doi.org/10.1117/12.878231.

Chakraborty, Dev P., and H. J. Yoon. 2008. “Operating Characteristics Predicted by Models for Diagnostic Tasks Involving Lesion Localization.” Journal Article. Medical Physics 35 (2): 435–45.

———. 2009. “JAFROC Analysis Revisited: Figure-of-Merit Considerations for Human Observer Studies.” Journal Article. Proc. SPIE Medical Imaging: Image Perception, Observer Performance, and Technology Assessment 7263: 72630T.

Chakraborty, Dev P, and Xuetong Zhai. 2016. “On the Meaning of the Weighted Alternative Free-Response Operating Characteristic Figure of Merit.” Journal Article. Medical Physics 43 (5): 2548–57.

Chakraborty, D. P. 1997a. “Comparison of Computer Analysis of Mammography Phantom Images (Campi) with Perceived Image Quality of Phantom Targets in the Acr Phantom.” Conference Proceedings. In Proc. SPIE Medical Imaging 1997: Image Perception, edited by Harold L. Kundel, 3036:160–67. SPIE.

———. 1997b. “Computer Analysis of Mammography Phantom Images (Campi): An Application to the Measurement of Microcalcification Image Quality of Directly Acquired Digital Images.” Journal Article. Medical Physics 24 (8): 1269–77.

Chakraborty, D. P., E. S. Breatnach, M. V. Yester, B. Soto, G. T. Barnes, and R. G. Fraser. 1986. “Digital and Conventional Chest Imaging: A Modified ROC Study of Observer Performance Using Simulated Nodules.” Journal Article. Radiology 158: 35–39. https://doi.org/10.1148/radiology.158.1.3940394.

Chakraborty, D. P., and Panos P. Fatouros. 1998. “Application of Computer Analyis of Mammography Phantom Images (Campi) Methodology to the Comparison of Two Digital Biopsy Machines.” Conference Proceedings. In Proc Spie Medical Imaging 1998: Physics of Medical Imaging, edited by John M. Boone James T. Dobbins III, 3336:618–28. SPIE.

Clarkson, Eric, Matthew A. Kupinski, and Harrison H. Barrett. 2006. “A Probabilistic Model for the MRMC Method, Part 1: Theoretical Development.” Journal Article. Academic Radiology 13 (11): 1410–21. https://doi.org/10.1016/j.acra.2006.07.016.

Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum Associates.

Daly, S. 1993. “The Visible Differences Predictor: An Algorithm for the Assessment of Image Fidelity.” Book Section. In Digital Images and Human Vision, edited by A. B. Watson, 179–206. Cambridge, Mass: MIT Press.

De Boo, Diederick W, Martin Uffmann, Michael Weber, Shandra Bipat, Eelco F Boorsma, Maeke J Scheerder, Nicole J Freling, and Cornelia M Schaefer-Prokop. 2011. “Computer-Aided Detection of Small Pulmonary Nodules in Chest Radiographs: An Observer Study.” Journal Article. Academic Radiology 18 (12): 1507–14.

DeLong, E. R., D. M. DeLong, and D. L. Clarke-Pearson. 1988. “Comparing the Areas Under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach.” Journal Article. Biometrics 44: 837–45. https://doi.org/10.2307/2531595.

DeSantis, Carol, Rebecca Siegel, Priti Bandi, and Ahmedin Jemal. 2011. “Breast Cancer Statistics, 2011.” CA: A Cancer Journal for Clinicians 61 (6): 408–18.

Dobbins III, James T, H Page McAdams, John M Sabol, Dev P Chakraborty, Ella A Kazerooni, Gautham P Reddy, Jenny Vikgren, and Magnus Båth. 2016. “Multi-Institutional Evaluation of Digital Tomosynthesis, Dual-Energy Radiography, and Conventional Chest Radiography for the Detection and Management of Pulmonary Nodules.” Journal Article. Radiology 282 (1): 236–50.

Dorfman, D. D., and E. Alf. 1969. “Maximum-Likelihood Estimation of Parameters of Signal-Detection Theory and Determination of Confidence Intervals - Rating-Method Data.” Journal Article. Journal of Mathematical Psychology 6: 487–96.

Dorfman, D. D., and K. S. Berbaum. 2000. “A Contaminated Binormal Model for ROC Data: Part Ii. A Formal Model.” Journal Article. Acad Radiol. 7 (6): 427–37. https://doi.org/10.1016/S1076-6332(00)80383-9.

Dorfman, D. D., K. S. Berbaum, and C. E. Metz. 1992. “ROC Characteristic Rating Analysis: Generalization to the Population of Readers and Patients with the Jackknife Method.” Journal Article. Invest. Radiol. 27 (9): 723–31. https://pubmed.ncbi.nlm.nih.gov/1399456.

Dorfman, D. D., K. S. Berbaum, C. E. Metz, R. V. Lenth, J. A. Hanley, and H. Abu Dagga. 1997. “Proper Receiving Operating Characteristic Analysis: The Bigamma Model.” Journal Article. Acad. Radiol. 4 (2): 138–49. https://doi.org/10.1016/S1076-6332(97)80013-X.

Dorfman, Donald D., Kevin S. Berbaum, and Russell V. Lenth. 1995. “Multireader, Multicase Receiver Operating Characteristic Methodology: A Bootstrap Analysis.” Journal Article. Academic Radiology 2 (7): 626–33. https://doi.org/10.1016/S1076-6332(05)80129-1.

Dorfman, Donald D, Kevin S Berbaum, and Charles E Metz. 1992. “Receiver Operating Characteristic Rating Analysis: Generalization to the Population of Readers and Patients with the Jackknife Method.” Investigative Radiology 27 (9): 723–31.

Duchowski, A. T. 2002. Eye Tracking Methodology: Theory and Practice. Book. Clemson, SC: Clemson University.

Edwards, Darrin C, Matthew A Kupinski, Charles E Metz, and Robert M Nishikawa. 2002. “Maximum Likelihood Fitting of Froc Curves Under an Initial-Detection-and-Candidate-Analysis Model.” Medical Physics 29 (12): 2861–70.

Efron, Bradley, and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. Book. Vol. 57. Monographs on Statistics and Applied Probability. Boca Raton: Chapman; Hall/CRC.

Efron, Bradley, and Robert J Tibshirani. 1994. An Introduction to the Bootstrap. CRC press.

Egan, James P. 1975. Signal Detection Theory and ROC Analysis. Book. First. Academic Press Series in Cognition and Perception. New York: Academic Press, Inc.

Egan, J. P., G. Z. Greenburg, and A. I. Schulman. 1961. “Operating Characteristics, Signal Detectability and the Method of Free Response.” Journal Article. J Acoust Soc. Am. 33: 993–1007.

Ernster, Virginia L. 1981. “The Epidemiology of Benign Breast Disease.” Journal Article. Epidemiologic Reviews 3 (1): 184–202.

FDA, U. 2018. “Guidance for Industry and Fda Staff Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data—Premarket Approval (Pma) and Premarket Notification [510 (K)] Submission.”

Fenton, J. J. 2015. “Is It Time to Stop Paying for Computer-Aided Mammography?” Journal Article. JAMA Intern Med. https://doi.org/10.1001/jamainternmed.2015.5319.

Fenton, Joshua J., Stephen H. Taplin, Patricia A. Carney, Linn Abraham, Edward A. Sickles, Carl D’Orsi, Eric A. Berns, et al. 2007. “Influence of Computer-Aided Detection on Performance of Screening Mammography.” Journal Article. N Engl J Med 356 (14): 1399–1409. https://doi.org/10.1056/NEJMoa066099.

Fisher, R. A., and L. H. C. Tippett. 1928. “Limiting Forms of the Frequency Distribution of the Largest and Smallest Member of a Sample.” Journal Article. Proc. Cambridge Phil. Society 24: 180–90.

Fletcher, Roger. 1970. “A New Approach to Variable Metric Algorithms.” Journal Article. The Computer Journal 13 (3): 317–22.

———. 2013. Practical Methods of Optimization. Book. John Wiley; Sons.

Franken, Jr., Edmund A., Kevin S. Berbaum, Susan M. Marley, Wilbur L. Smith, Yutaka Sato, Simon C. S. Kao, and Steven G. Milam. 1992. “Evaluation of a Digital Workstation for Interpreting Neonatal Examinations: A Receiver Operating Characteristic Study.” Journal Article. Investigative Radiology 27 (9): 732–37. http://journals.lww.com/investigativeradiology/Fulltext/1992/09000/Evaluation_of_a_Digital_Workstation_for.16.aspx.

Gallas, Brandon D. 2006. “One-Shot Estimate of MRMC Variance: AUC.” Journal Article. Academic Radiology 13 (3): 353–62. https://doi.org/10.1016/j.acra.2005.11.030.

Gallas, Brandon D., Gene a Pennello, and Kyle J. Myers. 2007. “Multireader Multicase Variance Analysis for Binary Data.” Journal Article. Journal of the Optical Society of America. A, Optics, Image Science, and Vision 24 (12): 70–80. https://doi.org/10.1364/josaa.24.000b70.

Goldfarb, Donald. 1970. “A Family of Variable-Metric Methods Derived by Variational Means.” Journal Article. Mathematics of Computation 24 (109): 23–26.

Green, D. M., and J. A. Swets. 1966. Signal Detection Theory and Psychophysics. Book. New York: John Wiley; Sons.

Gur, David, Andriy I. Bandos, Cathy S. Cohen, Christiane M. Hakim, Lara A. Hardesty, Marie A. Ganott, Ronald L. Perrin, et al. 2008. “The "Laboratory" Effect: Comparing Radiologists’ Performance and Variability During Prospective Clinical and Laboratory Mammography Interpretations.” Journal Article. Radiology 249 (1): 47–53. https://doi.org/10.1148/radiol.2491072025.

Hajian-Tilaki, K. O., James A. Hanley, L. Joseph, and J. P. Collet. 1997. “Extension of Receiver Operating Characteristic Analysis to Data Concerning Multiple Signal Detection Tasks.” Journal Article. Acad Radiol 4: 222–29. https://doi.org/10.1016/S1076-6332(05)80295-8.

Halpern, Scott D, Jason HT Karlawish, and Jesse A Berlin. 2002. “The Continuing Unethical Conduct of Underpowered Clinical Trials.” Journal Article. Jama 288 (3): 358–62.

Hanley, J. A., and B. J. McNeil. 1982. “The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve.” Journal Article. Radiology 143 (1): 29–36. http://radiology.rsnajnls.org/cgi/content/abstract/143/1/29.

Hanley, James A. 1988. “The Robustness of the "Binormal" Assumptions Used in Fitting ROC Curves.” Journal Article. Med. Decis. Making 8 (3): 197–203. https://doi.org/10.1177/0272989X8800800308.

Hanley, James A., and Karim O. Hajian-Tilaki. 1997. “Sampling Variability of Nonparametric Estimates of the Areas Under Receiver Operating Characteristic Curves: An Update.” Journal Article. Academic Radiology 4 (1): 49–58. https://doi.org/10.1016/s1076-6332(97)80161-4.

Hartmann, Lynn C, Thomas A Sellers, Marlene H Frost, Wilma L Lingle, Amy C Degnim, Karthik Ghosh, Robert A Vierkant, et al. 2005. “Benign Breast Disease and the Risk of Breast Cancer.” New England Journal of Medicine 353 (3): 229–37.

Hein, Patrick A, Lasse D Krug, Valentina C Romano, Sonja Kandel, Bernd Hamm, and Patrik Rogalla. 2010. “Computer-Aided Detection in Computed Tomography Colonography with Full Fecal Tagging: Comparison of Standalone Performance of 3 Automated Polyp Detection Systems.” Journal Article. Canadian Association of Radiologists Journal 61 (2): 102–8.

Hillis, S. L., N. A. Obuchowski, K. M. Schartz, and K. S. Berbaum. 2005. “A Comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette Methods for Receiver Operating Characteristic (ROC) Data.” Journal Article. Statistics in Medicine 24 (10): 1579–1607. https://doi.org/10.1002/sim.2024.

Hillis, Stephen L. 2007. “A Comparison of Denominator Degrees of Freedom Methods for Multiple Observer (ROC) Studies.” Journal Article. Statistics in Medicine 26: 596–619. https://doi.org/10.1002/sim.2532.

Hillis, Stephen L. 2007. “A Comparison of Denominator Degrees of Freedom Methods for Multiple Observer (ROC) Analysis.” Statistics in Medicine 26 (3): 596–619.

———. 2014. “A Marginal‐mean ANOVA Approach for Analyzing Multireader Multicase Radiological Imaging Data.” Journal Article. Statistics in Medicine 33 (2): 330–60. https://doi.org/10.1002/sim.5926.

Hillis, Stephen L, Kevin S Berbaum, and Charles E Metz. 2008. “Recent Developments in the Dorfman-Berbaum-Metz Procedure for Multireader Roc Study Analysis.” Academic Radiology 15 (5): 647–61.

Hillis, Stephen L., and K. S. Berbaum. 2004. “Power Estimation for the Dorfman-Berbaum-Metz Method.” Journal Article. Acad. Radiol. 11 (11): 1260–73. https://doi.org/10.1016/j.acra.2004.08.009.

Hillis, Stephen L., K. S. Berbaum, and C. E. Metz. 2008. “Recent Developments in the Dorfman-Berbaum-Metz Procedure for Multireader (ROC) Study Analysis.” Journal Article. Acad Radiol 15 (5): 647–61. https://doi.org/10.1016/j.acra.2007.12.015.

Hillis, Stephen L., Nancy A. Obuchowski, and Kevin S. Berbaum. 2011. “Power Estimation for Multireader ROC Methods: An Updated and Unified Approach.” Journal Article. Academic Radiology 18 (2): 129–42. https://doi.org/10.1016/j.acra.2010.09.007.

Hillis, Stephen L, Nancy A Obuchowski, Kevin M Schartz, and Kevin S Berbaum. 2005. “A Comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette Methods for Receiver Operating Characteristic (ROC) Data.” Statistics in Medicine 24 (10): 1579–1607.

Hupse, Rianne, Maurice Samulski, Marc Lobbes, Ard Heeten, MechliW Imhof-Tas, David Beijerinck, Ruud Pijnappel, Carla Boetes, and Nico Karssemeijer. 2013. “Standalone Computer-Aided Detection Compared to Radiologists’ Performance for the Detection of Mammographic Masses.” Journal Article. European Radiology 23 (1): 93–100. https://doi.org/10.1007/s00330-012-2562-7.

ICRU. 1996. “Medical Imaging: The Assessment of Image Quality.” Journal Article. JOURNAL OF THE ICRU 54 (1): 37–40.

Ishwaran, Hemant, and Constantine A. Gatsonis. 2000. “A General Class of Hierarchical Ordinal Regression Models with Applications to Correlated ROC Analysis.” Journal Article. The Canadian Journal of Statistics 28 (4): 731–50. https://doi.org/10.2307/3315913.

Jiang, Yulei, and Charles E. Metz. 2010. “BI-RADS Data Should Not Be Used to Estimate ROC Curves.” Journal Article. Radiology 256 (1): 29–31. https://doi.org/10.1148/radiol.10091394.

Kooi, Thijs, Albert Gubern-Merida, Jan-Jurre Mordang, Ritse Mann, Ruud Pijnappel, Klaas Schuur, Ard den Heeten, and Nico Karssemeijer. 2016. “A Comparison Between a Deep Convolutional Neural Network and Radiologists for Classifying Regions of Interest in Mammography.” In International Workshop on Breast Imaging, 51–56. Springer.

Kundel, Harold L., Calvin F. Nodine, Emily. F. Conant, and Susan P. Weinstein. 2007. “Holistic Component of Image Perception in Mammogram Interpretation: Gaze-Tracking Study.” Journal Article. Radiology 242 (2): 396–402.

Kundel, H. L., K. S. Berbaum, D. D. Dorfman, D. Gur, C. E. Metz, and R. G. Swensson. 2008. “Receiver Operating Characteristic Analysis in Medical Imaging (Icru Report 79).” Report. International Commission on Radiation Units; Measurments.

Kupinski, Matthew A., Eric Clarkson, and Harrison H. Barrett. 2006. “A Probabilistic Model for the MRMC Method, Part 2: Validation and Applications.” Journal Article. Academic Radiology 13 (11): 1422–30. https://doi.org/10.1016/j.acra.2006.07.015.

Larsen, Richard J., and Morris L. Marx. 2001. An Introduction to Mathematical Statistics and Its Applications. Book. 3rd ed. Upper Saddle River, NJ: Prentice-Hall Inc.

Lubin, J. 1995. A Visual Discrimination Model for Imaging System Design and Evaluation. Book. Visual Models for Target Detection and Recognition. Singapore: World Scientific Publishers.

Lusted, L. B. 1971. “Signal Detectability and Medical Decision Making.” Journal Article. Science 171: 1217–1219. https://doi.org/10.1126/science.171.3977.1217.

Macmillan, N. A., and C. D. Creelman. 1991. Detection Theory: A User’s Guide. Book. New York: Cambridge University Press.

Mann, H. B., and D. R. Whitney. 1947. “On a Test of Whether One of Two Random Variables Is Stochastically Larger Than the Other.” Journal Article. Annals of Mathematical Statistics 18: 50−60.

Metz, C. E. 1978. “Basic Principles of ROC Analysis.” Journal Article. Seminars in Nuclear Medicine 8 (4): 283–98. https://doi.org/10.1016/s0001-2998(78)80014-2.

———. 1989. “Some Practical Issues of Experimental Design and Data Analysis in Radiological ROC Studies.” Journal Article. Investigative Radiology 24: 234–45.

Metz, C. E., and X. Pan. 1999. “Proper Binormal ROC Curves: Theory and Maximum-Likelihood Estimation.” Journal Article. J Math Psychol 43 (1): 1–33.

Metz, Charles E. 1986. “ROC Methodology in Radiologic Imaging.” Journal Article. Investigative Radiology 21 (9): 720–33. http://journals.lww.com/investigativeradiology/Fulltext/1986/09000/ROC_Methodology_in_Radiologic_Imaging.9.aspx.

Metz, Charles E, Stuart J Starr, and Lee B Lusted. 1976. “Observer Performance in Detecting Multiple Radiographic Signals: Prediction and Analysis Using a Generalized Roc Approach.” Radiology 121 (2): 337–47.

Miller, George A. 1956. “The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information.” Journal Article. The Psychological Review 63 (2): 81–97.

Miller, Harold. 1969. “The FROC Curve: A Representation of the Observer’s Performance for the Method of Free Response.” Journal Article. The Journal of the Acoustical Society of America 46 (6(2)): 1473–6.

Niklason, L. T., N. M. Hickey, Dev P. Chakraborty, E. A. Sabbagh, M. V. Yester, R. G. Fraser, and G. T. Barnes. 1986. “Simulated Pulmonary Nodules: Detection with Dual-Energy Digital Versus Conventional Radiography.” Journal Article. Radiology 160: 589–93. https://doi.org/10.1148/radiology.160.3.3526398.

Nishikawa, Robert. 2012. “Estimating Sensitivity and Specificity in an ROC Experiment.” Journal Article. Breast Imaging, 690–96.

Nishikawa, Robert M, and Lorenzo L Pesce. 2011. “Fundamental Limitations in Developing Computer-Aided Detection for Mammography.” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 648: S251–S254.

Noether, Gottfried E. 1967. “Elements of Nonparametric Statistics.” Report. Wiley; Sons.

Obuchowski, Nancy A. 1998. “Sample Size Calculations in Studies of Test Accuracy.” Journal Article. Statistical Methods in Medical Research 7 (4): 371–92. https://doi.org/10.1177/096228029800700405.

———. 2000. “Sample Size Tables for Receiver Operating Characteristic Studies.” Journal Article. Am. J. Roentgenol. 175 (3): 603–8. http://www.ajronline.org/cgi/content/abstract/175/3/603.

Obuchowski, Nancy A., Michael L. Lieber, and Kimerly A. Powell. 2000. “Data Analysis for Detection and Localization of Multiple Abnormalities with Application to Mammography.” Journal Article. Acad. Radiol. 7 (7): 516–25.

Obuchowski, Nancy A., and Howard E. Rockette. 1995. “Hypothesis Testing of Diagnostic Accuracy for Multiple Readers and Multiple Tests an Anova Approach with Dependent Observations.” Communications in Statistics-Simulation and Computation 24 (2): 285–308.

Obuchowski, N. A., and H. E. Rockette. 1995. “Hypothesis Testing of the Diagnostic Accuracy for Multiple Diagnostic Tests: An ANOVA Approach with Dependent Observations.” Journal Article. Communications in Statistics: Simulation and Computation 24: 285–308. https://doi.org/10.1080/03610919508813243.

Pan, Xiaochuan, and Charles E Metz. 1997. “The ‘Proper’ Binormal Model: Parametric Receiver Operating Characteristic Curve Estimation with Degenerate Data.” Journal Article. Academic Radiology 4 (5): 380–89.

Pearson, Karl. 1900. “X. On the Criterion That a Given System of Deviations from the Probable in the Case of a Correlated System of Variables Is Such That It Can Be Reasonably Supposed to Have Arisen from Random Sampling.” Journal Article. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50 (302): 157–75. https://doi.org/10.1080/14786440009463897.

Penedo, Monica, Miguel Souto, Pablo G. Tahoces, Jose M. Carreira, Justo Villalon, Gerardo Porto, Carmen Seoane, et al. 2005. “Free-Response Receiver Operating Characteristic Evaluation of Lossy Jpeg2000 and Object-Based Set Partitioning in Hierarchical Trees Compression of Digitized Mammograms.” Journal Article. Radiology 237 (2): 450–57.

Philpotts, Liane E. 2009. “Can Computer-Aided Detection Be Detrimental to Mammographic Interpretation?” Journal Article. Radiology 253 (1): 17–22. https://doi.org/10.1148/radiol.2531090689.

Pisano, E. D., C. Gatsonis, E. Hendrick, M. Yaffe, J. K. Baum, S. Acharyya, E. F. Conant, et al. 2005. “Diagnostic Performance of Digital Versus Film Mammography for Breast-Cancer Screening.” Journal Article. N Engl J Med 353 (17): 1773–83. https://doi.org/10.1056/NEJMoa052911.

Pollack, Irwin. 1952. “The Information of Elementary Auditory Displays.” Journal Article. The Journal of the Acoustical Society of America 24 (6): 745–49.

———. 1953. “The Information of Elementary Auditory Displays. II.” Journal Article. The Journal of the Acoustical Society of America 25 (4): 765–69.

Popescu, Lucretiu M. 2011. “Nonparametric Signal Detectability Evaluation Using an Exponential Transformation of the FROC Curve.” Journal Article. Medical Physics 38 (10): 5690–5702.

Press, W. H., S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. 2007. Numerical Recipes: The Art of Scientific Computing. Book. 3rd ed. Cambridge: Cambridge University Press.

Rao, Vijay M, David C Levin, Laurence Parker, Barbara Cavanaugh, Andrea J Frangos, and Jonathan H Sunshine. 2010. “How Widely Is Computer-Aided Detection Used in Screening and Diagnostic Mammography?” Journal of the American College of Radiology 7 (10): 802–5.

Rockette, H. E., D. Gur, and C. E. Metz. 1992. “The Use of Continous and Discrete Confidence Judgments in Receiver Operating Characteristic Studies of Diagnostic Imaging Techniques.” Journal Article. Investigative Radiology 27: 169–72.

Roe, C. A., and C. E. Metz. 1997a. “Dorfman-Berbaum-Metz Method for Statistical Analysis of Multireader, Multimodality Receiver Operating Characteristic Data: Validation with Computer Simulation.” Journal Article. Acad Radiol 4: 298–303. https://doi.org/10.1016/S1076-6332(97)80032-3.

———. 1997b. “Variance-Component Modeling in the Analysis of Receiver Operating Characteristic Index Estimates.” Journal Article. Acad. Radiol. 4 (8): 587–600. https://doi.org/10.1016/S1076-6332(97)80210-3.

Ruschin, Mark., Pontus. Timberg, Magnus. Bath, Bengt. Hemdal, Tony. Svahn, Rob. Saunders, Ehsan. Samei, et al. 2007. “Dose Dependence of Mass and Microcalcification Detection in Digital Mammography: Free Response Human Observer Studies.” Journal Article. Medical Physics 34: 400–407.

Satterthwaite, F. E. 1941. “Synthesis of Variance.” Journal Article. Psychometrika 6 (5): 309–16.

———. 1946. “An Approximate Distribution of Estimates of Variance Components.” Journal Article. Biometrics Bulletin 2 (6): 110–14.

Shanno, David F. 1970. “Conditioning of Quasi-Newton Methods for Function Minimization.” Journal Article. Mathematics of Computation 24 (111): 647–56.

Shanno, David F, and Paul C Kettler. 1970. “Optimal Conditioning of Quasi-Newton Methods.” Journal Article. Mathematics of Computation 24 (111): 657–64.

Siddiqui, Khan M, Jeffrey P Johnson, Bruce I Reiner, and Eliot L Siegel. 2005. “Discrete Cosine Transform Jpeg Compression Vs. 2D Jpeg2000 Compression: JNDmetrix Visual Discrimination Model Image Quality Analysis.” In Medical Imaging 2005: PACS and Imaging Informatics, 5748:202–7. International Society for Optics; Photonics.

Skaane, Per, Andriy I Bandos, Randi Gullien, Ellen B Eben, Ulrika Ekseth, Unni Haakenaasen, Mina Izadi, Ingvild N Jebsen, Gunnar Jahr, and Mona Krager. 2013. “Comparison of Digital Mammography Alone and Digital Mammography Plus Tomosynthesis in a Population-Based Screening Program.” Journal Article. Radiology 267 (1): 47–56.

Soh, BaoLin P, Warwick Lee, Mark F McEntee, Peter L Kench, Warren M Reed, Rob Heard, Dev P Chakraborty, and Patrick C Brennan. 2013. “Screening Mammography: Test Set Data Can Reasonably Describe Actual Clinical Reporting.” Journal Article. Radiology 268 (1): 46–53.

Starr, S. J., C. E. Metz, and L. B. Lusted. 1977. “Comments on Generalization of Receiver Operating Characteristic Analysis to Detection and Localization Tasks.” Journal Article. Phys. Med. Biol. 22: 376–79.

Starr, Stuart J, Charles E Metz, Lee B Lusted, and David J Goodenough. 1975. “Visual Detection and Localization of Radiographic Images.” Radiology 116 (3): 533–38.

Stein, Sherman K., and Anthony Barcellos. 1992. Calculus and Analytic Geometry. Book. 5th ed. McGraw-Hill Companies.

Summers, Ronald M, Laurie R Handwerker, Perry J Pickhardt, Robert L Van Uitert, Keshav K Deshpande, Srinath Yeshwant, Jianhua Yao, and Marek Franaszek. 2008. “Performance of a Previously Validated Ct Colonography Computer-Aided Detection System in a New Patient Population.” Journal Article. American Journal of Roentgenology 191 (1): 168–74.

Swensson, Richard G. 1996. “Unified Measurement of Observer Performance in Detecting and Localizing Target Objects on Images.” Journal Article. Medical Physics 23 (10): 1709–25.

Swensson, Richard G. 1996. “Unified Measurement of Observer Performance in Detecting and Localizing Target Objects on Images.” Medical Physics 23 (10): 1709–25.

Swets, John A., and Ronald M. Pickett. 1982. Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Book. First. Series in Cognition and Perception. New York: Academic Press.

Tan, Tao, Bram Platel, Henkjan Huisman, Clara Sánchez, Roel Mus, and Nico Karssemeijer. 2012. “Computer-Aided Lesion Diagnosis in Automated 3-d Breast Ultrasound Using Coronal Spiculation.” Journal Article. Medical Imaging, IEEE Transactions on 31 (5): 1034–42.

Taylor, Stuart A, Steve Halligan, David Burling, Mary E Roddie, Lesley Honeyfield, Justine McQuillan, Hamdam Amin, and Jamshid Dehmeshki. 2006. “Computer-Assisted Reader Software Versus Expert Reviewers for Polyp Detection on Ct Colonography.” Journal Article. American Journal of Roentgenology 186 (3): 696–702.

Thompson, John D, Peter Hogg, David J Manning, Katy Szczepura, and Dev P Chakraborty. 2014. “A Free-Response Evaluation Determining Value in the Computed Tomography Attenuation Correction Image for Revealing Pulmonary Incidental Findings: A Phantom Study.” Journal Article. Academic Radiology 21 (4): 538–45.

Thompson, Mary Lou, and Walter Zucchini. 1989. “On the Statistical Analysis of Roc Curves.” Statistics in Medicine 8 (10): 1277–90.

Toledano, A. Y. 2003. “Three Methods for Analyzing Correlated ROC Curves: A Comparison in Real Data Sets.” Journal Article. Statistics in Medicine 22 (18): 2919–33. https://doi.org/10.1002/sim.1518.

Toledano, A. Y., and C. Gatsonis. 1996. “Ordinal Regression Methodology for ROC Curves Derived from Correlated Data.” Journal Article. Stat Med 15 (16): 1807–26. https://doi.org/10.1002/(SICI)1097-0258(19960830)15:16<1807::AID-SIM333>3.0.CO;2-U.

USAirForce, Report. 1947. “A Statistical Theory of Target Detection by Pulsed Radar.” Santa Monica, CA, US Air Force report.

Van den Branden Lambrecht, Christian J, and Olivier Verscheure. 1996. “Perceptual Quality Measure Using a Spatiotemporal Model of the Human Visual System.” In Digital Video Compression: Algorithms and Technologies 1996, 2668:450–61. International Society for Optics; Photonics.

Van Dyke, C. W., R. D. White, N. A. Obuchowski, M. A. Geisinger, R. J. Lorig, and M. A. Meziane. 1993. “Cine MRI in the Diagnosis of Thoracic Aortic Dissection.” Journal Article. 79th RSNA Meetings.

Vikgren, Jenny, Sara Zachrisson, Angelica Svalkvist, Ase A. Johnsson, Marianne Boijsen, Agneta Flinck, Susanne Kheddache, and Magnus Bath. 2008. “Comparison of Chest Tomosynthesis and Chest Radiography for Detection of Pulmonary Nodules: Human Observer Study of Clinical Cases.” Journal Article. Radiology 249 (3): 1034–41. https://doi.org/10.1148/radiol.2492080304.

Wagner, Robert F., Sergey V. Beiden, and Charles E. Metz. 2001. “Continuous Versus Categorical Data for ROC Analysis: Some Quantitative Considerations.” Journal Article. Academic Radiology 8 (4): 328–34. https://doi.org/10.1016/s1076-6332(03)80502-0.

Warren, Lucy M, Rosalind M Given-Wilson, Matthew G Wallis, Julie Cooke, Mark D Halling-Brown, Alistair Mackenzie, Dev P Chakraborty, Hilde Bosmans, David R Dance, and Kenneth C Young. 2014. “The Effect of Image Processing on the Detection of Cancers in Digital Mammography.” Journal Article. American Journal of Roentgenology 203 (2): 387–93.

Wilcoxon, F. 1945. “Individual Comparison by Ranking Methods.” Journal Article. Biometrics 1: 80–83.

Yoon, H. J., Bin Zheng, B. Sahiner, and Dev P. Chakraborty. 2007. “Evaluating Computer-Aided Detection Algorithms.” Journal Article. Medical Physics 34 (6): 2024–38.

Youden, William J. 1950. “Index for Rating Diagnostic Tests.” Cancer 3 (1): 32–35.

Zanca, Federica, Jurgen Jacobs, Chantal Van Ongeval, Filip Claus, Valerie Celis, Catherine Geniets, Veerle Provost, Herman Pauwels, Guy Marchal, and Hilde Bosmans. 2009. “Evaluation of Clinical Image Processing Algorithms Used in Digital Mammography.” Journal Article. Medical Physics 36 (3): 765–75. https://doi.org/10.1118/1.3077121.

Zanca, F., S. L. Hillis, F. Claus, C. Van Ongeval, V. Celis, V. Provoost, H.-J Yoon, and H. Bosmans. 2012. “Correlation of Free-Response and Receiver-Operating-Characteristic Area-Under-the-Curve Estimates: Results from Independently Conducted FROC/ROC Studies in Mammography.” Journal Article. Med Phys 39 (10): 5917–29.

Zhou, Xiao-Hua, Donna K McClish, and Nancy A Obuchowski. 2009. Statistical Methods in Diagnostic Medicine. Vol. 569. John Wiley & Sons. https://doi.org/110.1002/9780470906514.

Zhou, Xiao-Hua, Nancy A. Obuchowski, and Donna K. McClish. 2002. Statistical Methods in Diagnostic Medicine. Book. New York: John Wiley; Sons.

References

Hillis, Stephen L. 2014. “A Marginal‐mean ANOVA Approach for Analyzing Multireader Multicase Radiological Imaging Data.” Journal Article. Statistics in Medicine 33 (2): 330–60. https://doi.org/10.1002/sim.5926.