Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

On the effect of improperness of binormal ROC curves for estimating full area under the curve

Guo, Ben (2015) On the effect of improperness of binormal ROC curves for estimating full area under the curve. Master's Thesis, University of Pittsburgh. (Unpublished)

[img]
Preview
PDF
Submitted Version

Download (737kB)

Abstract

The “binormal” model is commonly used for evaluating diagnostic performance with smooth Receiver Operating Characteristic (ROC) curves. However, one of the artifacts of the binormal model is the non-concave (improper) shape of the ROC curves, which is sometimes evident as a visible and practically unreasonable “hook”. The artificial hook can often be triggered, when the true ROC curve is concave but has high initial slope. In these scenarios it is natural to be concerned with the bias in the estimates of global summary measures, e.g., in the commonly used area under the ROC curve (AUC). The objective of this study is to evaluate the magnitude of said bias as a function of improperness of the fitted binormal ROC curves. The public health relevance of this work stems from the importance of the ROC methodology for various stages of development and regulatory approval of medical diagnostic systems. This work investigates whether the AUC for a visually improper binormal ROC curve provides an acceptable estimate of the full area under an actually concave ROC curve. For this purpose a simulation study was conducted based on a wide range of scenarios described by the concave bigamma ROC curves. The binormal ROC curves were fitted using the least squares approach. Based on the “mean-to-sigma ratio” criteria proposed in the literature, the fitted binormal curves were divided into the three groups based on the magnitude of their visual improperness. In order to assess bias in these groups of curves the binormal estimates of AUCs were compared with the empirical AUCs (which are unbiased for continuous data). Our results indicate that for continuous data the bias of the binormal estimate of AUC was small regardless of the magnitude of improperness of the fitted curve. Thus, if one is interested only in estimating AUC using continuous diagnostic data, the improper shape of the binormal curve can often be unimportant. We used data from a multireader study with 36 ROC curves, to illustrate the differences between the bigamma and binormal AUC estimates for different shapes of binormal ROC curves fitted to pseudo-continuous data from actual diagnostic accuracy studies.


Share

Citation/Export:
Social Networking:
Share |

Details

Item Type: University of Pittsburgh ETD
Status: Unpublished
Creators/Authors:
CreatorsEmailPitt UsernameORCID
Guo, Benbeg37@pitt.eduBEG37
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Thesis AdvisorBandos, Andriyanb61@pitt.eduANB61
Committee MemberGur, Davidgur@pitt.eduGUR
Committee MemberJeong, Jong Hyeonjeong@nsabp.pitt.eduJJEONG
Date: 28 January 2015
Date Type: Publication
Defense Date: 11 December 2014
Approval Date: 28 January 2015
Submission Date: 24 November 2014
Access Restriction: 2 year -- Restrict access to University of Pittsburgh for a period of 2 years.
Number of Pages: 64
Institution: University of Pittsburgh
Schools and Programs: Graduate School of Public Health > Biostatistics
Degree: MS - Master of Science
Thesis Type: Master's Thesis
Refereed: Yes
Uncontrolled Keywords: ROC curve, Bigamma model, Binormal model
Date Deposited: 28 Jan 2015 15:44
Last Modified: 01 Jan 2017 06:15
URI: http://d-scholarship.pitt.edu/id/eprint/23590

Metrics

Monthly Views for the past 3 years

Plum Analytics


Actions (login required)

View Item View Item