Platforms /


Capacity and Capabilities

1. To what number of subjects might you reasonably deploy a study?

Amazon MTurk From less than 10 to more than 1000
Cognition.Run From less than 10 to 100
Django From less than 10 to more than 1000
Gorilla From less than 10 to more than 100
hearX From 10 to more than 100
ispring From less than 10 to 100
Jacoti More than 1000
jspsych From less than 10 to more than 100
MATLAB From less than 10 to 100
PART/BGC Science From less than 10 to more than 100
Prolific From less than 10 to more than 1000
Psychstudio From less than 10 to more than 1000
PsyToolkit From less than 10 to more than 1000
Qualtrics From less than 10 to more than 1000
SHOEBOX More than 1000
SpeakPipe From less than 10 to 100
TabSINT From less than 10 to more than 100
TeamHearing More than 1000

2. What factors limit deployability to more or fewer subjects?

Amazon MTurk None. But Turk HITS are designed to not take more than a handful of minutes. Not sure if you could or should get one Turker for an hour or more. They might be nervous to make that big a commitment.
Cognition.Run currently does not have a funding mechanism to support hosting costs, so I would be wary of excessively loading their website until they do. They plan to register as a nonprofit to collect donations soon, but if that does not cover costs they will switch to a license fee structure.
Django Depends on the capacity of the server running Django.
Gorilla Gorilla is a paid service. The biggest limiting factor is the fee associated with each respondent’s data.
hearX cost, number of devices owned
ispring There is not really a set limit. Depending on how a test is deployed, for example using the experimenter’s server or the companion ispringlearn server, costs would be different. But if it is hosted by the experimenter and data processing on the experimenter’s server is set up, there is no extra cost or limits on the number of people who can take it.
Jacoti Accessibility to iPhone/iPad and AppleEarPods
jspsych This is really a function of the server you pair jspsych with.
MATLAB Server hardware or the MATLAB web app server software itself will limit the number of simultaneous users able to access the task. Depending on the amount of time the researcher wants to allow data collection it is possible to collect data from larger numbers. User experience will be impacted by mass recruitment if participants receive messages from the server that it is busy or full.
PART/BGC Science PART can be downloaded onto a user’s device, or a pre-loaded device can be sent to a user. BGC Science was developed in order to allow users to download onto their own device and have the correct tests pushed to the device based on a registered email address. Please contact the developers for any specific use questions, as the NIH funding supports working with researchers and clinicians to develop unique testing approaches.
Prolific While there is no upper limit for deployment imposed by Prolific, it seems to be a good idea to open up the studies to a handful participants at a time (say 20 per batch). This is because participants often message the researcher via Prolific with questions and technical issues, which becomes unmanageable when simultaneously deployed to too many participants. However, participants are found quickly and hence a large N can be obtained in a short time but as a sequence of small batches.
Psychstudio None.
PsyToolkit None.
Qualtrics The number of subjects an experimenter is allowed to collect will depend on the Qualtrics package they have purchased (or that the university has purchased).
SHOEBOX Cost, number of iPad systems available to you
SpeakPipe Manual download of audio files.
TabSINT None
TeamHearing Server management.

3. BRIEFLY describe the capabilities of this platform beyond monaural audio-only air-conduction testing (e.g., binaural, audiovisual, free field, ancillary measures/physiology):

Amazon MTurk Can do anything that requires audio and video presentation through a browser. Via JavaScript you can monitor the entire system, and do your own timing or even capture mouse movements.
Cognition.Run All experiments are implemented in jsPsych, so anything that jsPsych can do.
Django Provides backend logic. This can be used to control flow of the study and serve pages created using front-end platforms like jspsych.
Gorilla Gorilla is capable of monaural & binaural audio playback (embedded in the audio files uploaded to server) and audiovisual studies. There seems to be some variation in the AV synchronization depending on the web browser in use ( It very limited in free field implementation beyond 2-channel audio output. Eye tracking/pupillometry through webcams has limited success and heavily relies on postprocessing. I am not aware of additional physiological measures that are compatible with Gorilla at the moment.
hearX video otoscopy (Hearscope), visual acuity (peek acuity), seal of earplugs (sealCheck probe),
ispring Binaural should work as well with two channel files, but not much more.
Jacoti Uses monaural air conduction results to turn device into an amplifier (JacotiListenApp) or FM system (JacotiLola)
jspsych Binaural stimuli possible (Young folks in the lab are able to hear 20 microsecond ITD clearly), audiovisual stimuli possible. One very useful capability is to record reaction times sufficiently reliably to do classic cognitive tasks.
MATLAB My knowledge about these capabilities is limited but MATLAB web app server apps should allow for any possible combinations of stimulus delivery that are possible through HTML or javascript as the MATLAB web apps support loading and executing HTML/javascript code. I have had success in using both monaural and binaural stimuli over both laptop speakers and headphones.
PART/BGC Science binaural, cognitive testing (progressive matrices, inhibitory control, reaction time)
Prolific Not applicable
Psychstudio Video, audio, text, images and any spatial, temporal and visual blending of such.
Qualtrics Qualtrics has limited capabilities beyond survey questions. Audio, visual, and audiovisual stimuli can be presented.
SHOEBOX High-frequencies, masking, bone conduction, integrated electronic questionnaires, speech reception and speech discrimination testing
TabSINT Supports Dolby 7.1 audio, sound level meter type recordings, bluetooth communication with a computer for additional hardware integrations
TeamHearing Binaural yes, audiovisual yes, tactile yes.

4. Miscellaneous comments on Capabilities:

Amazon MTurk  
Gorilla One of the recruitment mechanisms on Gorilla will allow participants to access an experiment using a unique URL, which will allow them to resume the session. Other recruitment mechnisms will forfeit the ongoing session and participants will have to restart.
jspsych jspsysch doesn’t place many constraints. However, jspsych can be used to detect basic specs of a user’s screen/browser, compatibility with WebAudio API etc. and the experiment programmer can restrict access based on these variables.
PART/BGC Science PART is available for use on iOS, Android, Mac, and Windows, and BGC Science is available for client-side testing. Contact developers if you have a use case that you are not sure how to implement. This project is fully-funded by NIH and so there is substantial support available for helping researchers and clinicians develop the tools they need.
Prolific Prolific allows researchers to let the subjects know that only participants using certain devices should participate. If the researcher is able to verify independently that this instruction was violated, they can reject a submission (and not pay the participant).

5. How are user-designed experiments developed on this platform?

Amazon MTurk Builder interface/GUI and Code-based
Cognition.Run Code-based
Django Code-based
Gorilla Builder interface/GUI and Code-based
hearX Builder interface/GUI
ispring Builder interface/GUI
Jacoti Builder interface/GUI
jspsych Code-based
MATLAB Code-based
PART/BGC Science Builder interface/GUI
Prolific Not available
Psychstudio Builder interface/GUI
Psytoolkit Code-based
Qualtrics Builder interface/GUI and Code-based
SHOEBOX Builder interface/GUI
SpeakPipe Not Available
TabSINT Code-based
TeamHearing Code-based

6. What programming languages are used to interface with the platform and generate experiments (check all)?

Amazon MTurk JavaScript, Python, Other options up to user
Cognition.Run JavaScript, the jsPsych library
Django Python; Needs to be paired with front-end HTML/CSS/Javascript (e.g., using Jspsych)
Gorilla JavaScript
hearX Not sure
ispring The software works as a Power point add on. Tests can be developed with no coding experience. This is both the platform’s strength and weakness. Coding may be needed for data processing depending on the type of experiments, size and scope.
Jacoti Apple App
jspsych JavaScript, Because jspsych needs to be paired with a separate server, some basic backend programming knowledge is also required. The jspsych documentation provides examples for how to integrate with a server running PHP code. Other pairings are possible (e.g., with Django or Cognition.Run).
MATLAB Matlab, JavaScript
PART/BGC Science Matlab, JavaScript,Python can be used, but the main method is using the program itself, which contains a complete graphical interface as well as a custom programming syntax for algorithm specification. Please contact the developers with any questions, as the NIH funding supports one-on-one guidance of researchers and clinicians interested in using PART and BGC Science.
Prolific Not applicable
Psychstudio This platform is entirely user interface driven (there is no programming).
PsyToolkit It’s a language specific to Psytoolkit but is not very complicated and there is extensive documentation.
Qualtrics JavaScript
SpeakPipe Not applicable
TabSINT JavaScript, JSON
TeamHearing JavaScript

7. Are ‘ready-made’ experiments available on this platform?

Amazon MTurk No
Cognition.Run No
Django No
Gorilla Yes, there are some cognitive tasks available for community share. In general, sharing Gorilla tasks is very easy between accounts.
hearX Yes, Pure tone audiometry, high-frequency audiometry, digits in noise speech test (HearDigits)
ispring No
Jacoti Yes, Only a pure tone audiogram
jspsych Yes, Some basic examples are provided in the documentation. There are many more user-generated examples available on the web based on jspsych (e.g., see
MATLAB No officially supported ‘ready-made’ experiments are available. Mathworks provide a rudimentary tutorial on building an app through their GUI based app builder and instructions on how to compile those apps for web use. I am working to complete development work on a port of Stephan Ewert’s "AFC" experiment framework allowing nearly any existing code run in AFC to be run as a web app.
PART/BGC Science Yes, There are a number of batteries that are pre-built and included on the platform at first download, more are available on Github, and the PART team will help support others when researchers are not comfortable setting them up on their own. For descriptions of some of the built-in assessments see the following reference: Lelo de Larrea-Mancera, E.S., Stavropoulos, T., Hoover, E.C., Eddins, D.A., Gallun, F.J., & Seitz, A.R. (2020). Portable Automated Rapid Testing (PART) for auditory research: Validation in a normal hearing population. bioRxiv.
Prolific No
Psychstudio Yes, demonstration and example experiments and templates available.
Psytoolkit Yes, some things like the Stroop task or Posner cueing task are readily available.
Qualtrics No
SHOEBOX Yes, Meant for portable audiometric testing.
SpeakPipe No
TabSINT Yes, A limited number of example experiments are available.
TeamHearing Yes, Wide range of psychophysical and speech recognition experiments are available.

8. Please describe any service that the platform provides or uses to allow investigators to share user-designed experiments with other investigators:

Amazon MTurk Personal communication
Cognition.Run None
Django Not inherent to platform, but any code-sharing platforms (like GitHub) can be used. Django itself is open source with code available on GitHub.
Gorilla Available from within platform
hearX Personal communication, Available from within platform, not explicitly stated – but assuming that if a clinician can access the patient data that they can give other users permissions to view data.
ispring Personal communication. See, e.g.,
Jacoti Can give a health care professional access to results, but meant for direct to consumer.
jspsych All the ways in which javascript code can be shared should work (e.g., GitHub). Jspsych itself is open source with code available on GitHub.
MATLAB The shared online repository for other matlab code could be used as a place for investigators to share experiments and code:
PART/BGC Science Personal communication and GitHub (development ongoing)
Prolific Not applicable
Psychstudio Available within platform. Experiments can be shared collaboratively with other users and can have ownership transferred to other users. Experiments can also be made available to copy and modify by other users within the same lab.
Psytoolkit Shared online repository and personal communication.
Qualtrics Available from within platform
TeamHearing Personal communication, Available from within platform