MR plot: A big data tool for distinguishing distributions

Author ORCID Identifier

Keith Millis:https://orcid.org/0000-0001-7639-6484

Publication Title

Statistical Analysis and Data Mining

ISSN

19321864

E-ISSN

19321872

Document Type

Article

Abstract

Big data enables reliable estimation of continuous probability density, cumulative distribution, survival, hazard rate, and mean residual functions (MRFs). We illustrate that plot of the MRF provides the best resolution for distinguishing between distributions. At each point, the MRF gives the mean excess of the data beyond the threshold. Graph of the empirical MRF, called here the MR plot, provides an effective visualization tool. A variety of theoretical and data driven examples illustrate that MR plots of big data preserve the shape of the MRF and complex models require bigger data. The MRF is an optimal predictor of the excess of the random variable. With a suitable prior, the expected MRF gives the Bayes risk in the form of the entropy functional of the survival function, called here the survival entropy. We show that the survival entropy is dominated by the standard deviation (SD) and the equality between the two measures characterizes the exponential distribution. The empirical survival entropy provides a data concentration statistic which is strongly consistent, easy to compute, and less sensitive than the SD to heavy tailed data. An application uses the New York City Taxi database with millions of trip times to illustrate the MR plot as a powerful tool for distinguishing distributions.

First Page

405

Last Page

418

Publication Date

8-1-2020

DOI

10.1002/sam.11464

Keywords

Bayes risk, concentration measures, distributional plots, mean residual plot, survival entropy, taxi trip time

Department

Department of Statistics and Actuarial Science

Link to Full Text

Share

COinS