InfoSec Seminar: Let's talk about publication bias

Speaker: Jonathan Spring

Date/Time: 10-May-2018, 16:00 UTC

Venue: Roberts 421



We'll mainly focus this discussion around an article in the journal Science that attempted to reproduce 100 significant experimental findings (DOI: 10.1126/science.aac4716). Only 39 reproduced successfully. We'll discuss the causes of this phenomenon -- which include selective reporting, selective analysis, and poor descriptions of experimental conditions -- and the extent to which information security also suffers from these. We might add working under intense and constant deadline pressure to this list for infosec, for example. I'll focus on practical solutions -- we all know we can't all stop what we're doing to replicate existing studies. But we can strike a better balance between novelty and reproduction. For example, we can differentiate between negative results and non-results. A negative result is a well-done study that does not find something -- which is still useful scientific evidence. A non-result is a study with fatal flaws which prevent us from reliably using the results as evidence for anything.




Jonathan Spring is a PhD student at UCL in PPLV, Infosec, and STS. He has about 5 years experience with the CERT program at Carnegie Mellon University's Software Engineering Institute, where he has studied network and DNS analysis and threat intelligence. He also has experience as a research fellow with ICANN's SSAC and an adjunct professor at the University of Pittsburgh.

Add to Calendar

This page was last modified on 27 Mar 2014.