Wednesday, July 31, 7-8:30
Reader, School of Psychology
Presentation Abstract: Most of scientific work happens behind the scenes: every published scientific result is an intermediate step in a process that may have taken years to realize. Opportunistic analyses, publication bias, and outright fraud may be important to consider when trying to assess the trustworthiness of a result, but they are, of course, not mentioned in the report. However, they might leave statistical traces: for instance, Fisher (1936) pointed out that Mendel’s (1866) results with pea plants appeared to be too close to their theoretical values to be accounted for by chance variation, possibly intentionally falsified either for didactic reasons or by an assistant trying to please Mendel. Similar methods have been at the center of assessing the credibility of more recent research, but the essential character of modern methods is a straighforward extension of Fisher’s logic (which itself is significance testing). We can call these “statistical forensics” methods whose goal is to shed light on whether a body of research is trustworthy and perhaps to try to correct for issues that might cause doubt. I will outline some of these methods, describe where they have been used in practice, and discuss potential objections.
The Legal Sequelae of the 2016 American Statistical Association P-Value Statement
Nathan A. Schachtman
Monday August 5 7-8:30
In 2016, the American Statistical Association issued an unusual guidance document in which it attempted to redress its perception that p-values and statistical significance were widely misunderstood and misinterpreted. In addition to providing guidance on the meaning and use of attained significance probabilities, the ASA also encouraged the use of “other methods that emphasize estimation over testing,” including Bayesian methods. Although the ASA guidance document warned against misuses of p-values, it did not warn of the potential for misapplication of these “other methods.” The reaction of some segments of the legal community was prompt, both in interpreting the 2016 guidance as a rejection of p-values and significance testing, as well as an encouragement to use “other methods,” for which the judiciary would have far less experience and acumen to detect invalid inferences. In this presentation, I will discuss how the ASA Statement was used rhetorically to justify causal claims that had been rejected by the FDA and scientific organizations, and to advance a “Bayesian hypothesis” test to support a claim that a meta-analysis showed that there was an 85 percent probability that testosterone replacement therapy caused either heart attack or stroke. (Fuller Discussion (pdf))
Stephen Senn, Consultant Statistician, Edinburgh
100 years ago, in 1919, Fisher arrived at Rothamsted Agricultural Research Station and began his programme of revolutionising statistics. He realised that it was not enough for the subject of statistics to concern itself with the analysis of data but that it also had to deal with the process of collecting and planning to collect data. Thus, statistics became, under his leadership, a subject not just about analysis of experiments but also about their design.
One of the innovations in design he introduced was randomisation. However, although this has proved to be a practical success in many fields it has become a critical failure amongst many methodologists, in particular, philosophers of science. In my opinion much of the mistrust can be traced to a misunderstanding as to how statistical analysis of randomised experiments proceeds. In this talk I attempt to clear up the misunderstanding and show that many of the criticisms of randomisation turn out to be irrelevant.
Related blogs and articles