Assuring replication in computer-based psychology experiments

1. WHAT
If you are a psychologist, neuroscientist or vision researcher who uses a computer to run experiments, and report timing accuracy in units of a millisecond, then it's likely your timings are wrong! This can lead to replication failure, spurious results and questionable conclusions. Timing error will affect your work even when you use an experiment generator like E-Prime, SuperLab, Inquisit, MATLAB, Presentation, Paradigm or PsychoPy…

Ask yourself:

Are you always carrying out the experiments you assume you are?
Are you aware of millisecond timing error in your own experiments?
Are you confident you can replicate experiments using different hardware and software in another lab?

The Black Box ToolKit lets you quickly and easily check your own millisecond timing accuracy in terms of stimulus presentation accuracy; stimulus synchronization accuracy; and response time accuracy.

Timing error means that your study is not working as you intended and that your

results might be spurious. Are you putting your reputation at risk?

The Black Box ToolKit guarantees your ability to replicate.

2. WHY
Modern computer hardware may be faster but millisecond timing accuracy gets harder to achieve year-on-year. A common misconception is that "millisecond precision" equals "millisecond accuracy". Precision simply means that timings are reported in units of a millisecond, not that they are accurate!

For example, all TFT monitors and projectors suffer from input lag. If you present an image it will take longer to appear on screen than you might think. On some models this can be upwards of 100 milliseconds. Whatever experiment generator you use, it only knows when it requested a stimulus image be shown and not the time when it physically appeared.

Because of soundcard startup latency audio timing is even more variable. But don't just take our word for it read PST's own findings when using E-Prime.


TFT/projector input lag	Soundcard startup latency	Response devices	Voice keys	fMRI, EEG/ERP, Eye Tracking	OS issues	Software settings	Conditional biases	Laws of physics

3. HOW
To improve replication and enhance credibility researchers should self-validate, or self-certify, their own studies in terms of millisecond presentation, synchronization and response timing accuracy.

Currently self-validation of millisecond timing accuracy can only be done quickly and easily with a Black Box ToolKit. This acts as a programmable virtual human that can detect and respond to stimulus events with sub-millisecond accuracy. It enables you, the researcher, to check the accuracy of your own paradigm whilst running in-situ on your own equipment.

Set-up – Hook up external stimulus sensors (opto-detectors, mics, TTL) and a response device (response pad, robotic key actuator, sounder, TTL)
Test – Use a wizard to select a stimulus pattern to automatically respond to/event mark. Then choose the exact response time, response device and duration
Analyze – Check your stimulus, response and synchronization timings across up to 36 channels. Compare what your experiment recorded with what the BBTK saw