Conditional biases and unaccounted for interactions
Find out more
Unbeknown to you systematic conditional biases can account for your whole experimental effect. The classic file drawer problem in psychological research can be an unintended consequence of unknown biases.

Replication lies at the heart of good science and the scientific method. Repeatability is key. If you think you are holding all other variables constant but later find out many more variables were at play where does that leave your findings and your reputation?

One classic example from our own testing is where in a cross modal priming study it was discovered that on trials where a visual stimulus was presented alongside audio there was a systematic discrepancy in measured reaction time. This error was due to the fact that the software in question could not reliably process the onset of a reaction time while a sound was playing. Without independently checking with a Black Box ToolKit the researcher would have assumed that the experimental effect was real as might you in your own study.

In the case of the cross model priming study the equipment was setup as shown below (Apple Mac). Three Black Box ToolKit opto-detectors monitored for any one of three stars appearing on screen, two digital microphones checked for a sound presentation and an Active Switch Closure was wired into the studies regular response box. When a visual stimulus appeared a response was generated after 300 milliseconds. As can be seen from the summary results there was a clear systematic conditional bias!

Schematic of how the studies timing was validated using a Black Box ToolKit.

 

Measure Expected RT (mS) Observed Mean RT (mS)
Star duration
200
196.46
Tone duration
200
210.66
Tone synchrony to star:
0
6.38
 
50
56.88
 
100
106.62
Reaction Time (no tone)
300
304.58
Reaction Time (tone present)
300
321.78

Summary results from Plant, Hammond & Turner (2004),
"Self-validating presentation and response timing in cognitive paradigms: how and why?"


References
Self-validating presentation and response timing in cognitive paradigms: how and why?, Richard R Plant, Nick Hammond, Garry Turner. Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc. 06/2004; 36(2):291-303.

PsychFileDrawer.org: Archive of Replication Attempts in Experimental Psychology