Reimers, S., & Stewart, N. (in press). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript web experiments. Behavior Research Methods.

Web-based research is becoming ubiquitous in the behavioral sciences, facilitated by convenient, readily available participant pools, and relatively straightforward ways of running experiments: most recently, the development of the HTML5 standard. Although in most studies participants give untimed responses, there is a growing interest in being able to record response times online. Existing data on the accuracy and cross-machine variability of online timing measures are limited, and generally compare behavioral data gathered on the web with similar data gathered in the lab. This paper takes a more direct approach, examining two ways of running experiments online—Adobe Flash and HTML5 with CSS3 and JavaScript, across 19 different computer systems. We use specialist hardware to measure stimulus display durations and generate precise response times to visual stimuli to assess measurement accuracy, examining effects of duration, browser, and system-to-system variability such as Windows version, processing power and graphics capability. Findings include: (a) Flash and JavaScript's presentation and response time measurement accuracy are similar (b) Within-system variability is generally small, even in low-powered machines under high load; (c) Variability of measured response times across systems is somewhat larger; (c) Browser type and system hardware appear to have relatively small effects on measured response times. Modeling of the effects of this technical variability suggests that for most within- and between- subjects experiments, Flash and JavaScript can both be used to accurately detect differences in response time across conditions. Concerns are, however, noted about using some correlational or longitudinal designs online.