Assuring replication in computer-based psychology experiments

1. WHAT
If you are a psychologist, neuroscientist or vision researcher who uses a computer to run experiments, and reports timing accuracy in units of a millisecond, then it's likely your timings are wrong! This can lead to replication failure, spurious results and questionable conclusions.

Ask yourself:

Are researchers always carrying out the experiments they assume they are in terms of stimulus presentation, synchronization and response timing accuracy?
Are researchers aware of any mistimings that affect stimulus presentation, synchronization and response timing accuracy and can they quantify such together with the affect they may have on the validity of results?
Are researchers confident that they can replicate all experimental parameters so as to ensure that comparable results are obtained when different hardware and software in another laboratory are used?

Timing error means that your study is not working as you intended and that your

results might be spurious. Are you putting your reputation at risk?

The Black Box ToolKit guarantees your ability to replicate.

2. WHY
Modern computer hardware may be faster but millisecond timing accuracy gets harder to achieve year-on-year. A common misconception is that "millisecond precision" equals "millisecond accuracy". Precision simply means that timings are reported in units of a millisecond, not that they are accurate!

For example, all TFT monitors suffer from input lag. If you present an image it will take longer to appear on screen than you might think. On some models this can be upwards of 60 milliseconds. Remember quoted panel response time is not the same as input lag. More >>

Because of startup latency soundcards are even more variable. But don't just take our word for it... read PST's own findings.


TFT/projector input lag	Soundcard startup latency	Response devices	Voice keys	3rd party equipment	OS issues	Software settings	Conditional biases	Laws of physics

3. HOW
To improve replication researchers should self-validate, or self-certify, their own studies in terms of millisecond presentation, synchronization and response timing accuracy.

Currently self-validation can only be done quickly and easily with a Black Box ToolKit. This acts as a programmable virtual human that can detect and respond to stimulus events with sub-millisecond accuracy. It enables you, the researcher, to check the accuracy of your own paradigm whilst running in-situ on your own equipment with no modification needed. More >>

Set-up – Hook up external stimulus sensors (opto-detectors, mics, TTL) and a response device (response pad, robotic key actuator, sounder, TTL)
Test – Use a wizard to select a stimulus pattern to automatically respond to/event mark. Then choose the exact response time, response device and duration
Analyze – Check your stimulus, response and synchronization timings across up to 36 channels. Compare what your experiment recorded with what the BBTK saw