Software settings, bugs and device drivers in psychology experiments
Modern software used to run psychology experiments has more functionality than ever enabling you to construct a wide range of paradigms that were previously beyond reach. However this complexity comes at a price. Often there are a multitude of settings that you can alter. Each of which can have a subtly different effect on millisecond timing accuracy. It's questionable how many researchers actually use the same settings for each study or fully understand the ramifications of their choices. Some experiment generators may use a multitude of techniques and settings to try and achieve better presentation timings for example.
Modern software whether it be the software you use to write papers with, your experiment generator or even the operating system, will have bugs in it. Simply if certain conditions are met it won't function as intended when running your computer-based psychology experiment. This can lead to unintentional millisecond timing errors which you may never know about unless you check empirically with a Black Box Toolkit v2. There is the well known case of fMRI software miscalculating gradients and bringing years of research across the globe into question. There is also the wider question of whether many fMRI studies can be replicated at all due to the variety of analysis software and techniques used within the field.
Even the "best" software has bugs in it which can have literally devastating consequences. For example, remember the Northeast Power Blackout in the USA (programming bug), NASA Mars Climate Orbiter (metric to imperial measurement "miscommunication"), or something as mundane as the Denver Airport baggage-handling system! These three sets of software bugs alone cost well over 10 billion US Dollars.
Hardware device drivers provided by manufacturers can also be a source of timing error. Whilst drivers should be simple often manufacturers add as many bells and whistles as they can fit in. Settings and bugs in hardware drivers can have a dramatic impact. For example, DSP or effects settings, on soundcards can add 10's of milliseconds to latency.
Sound output delay with two different DSP settings on the same soundcard.
Most people assume that even a "generic" effect will by fine so will leave the
settings at their default unaware of the consequences (from Plant & Turner 2009).
Richard R. Plant and Garry Turner, "Millisecond precision psychological research in a world of commodity computers: New hardware, new problems?", Behavior Research Methods, 2009, Volume 41, Number 3, Pages 598-614