CapacityAndCapabilities
Capacity and Capabilities
1. To what number of subjects might you reasonably deploy a study?
| Platform | |
| Amazon MTurk | From less than 10 to more than 1000 |
| Cognition.Run | From less than 10 to 100 |
| Django | From less than 10 to more than 1000 |
| Gorilla | From less than 10 to more than 100 |
| hearX | From 10 to more than 100 |
| ispring | From less than 10 to 100 |
| Jacoti | More than 1000 |
| jspsych | From less than 10 to more than 100 |
| MATLAB | From less than 10 to 100 |
| PART/BGC Science | From less than 10 to more than 100 |
| Prolific | From less than 10 to more than 1000 |
| Psychstudio | From less than 10 to more than 1000 |
| PsyToolkit | From less than 10 to more than 1000 |
| Qualtrics | From less than 10 to more than 1000 |
| SHOEBOX | More than 1000 |
| SpeakPipe | From less than 10 to 100 |
| TabSINT | From less than 10 to more than 100 |
| TeamHearing | More than 1000 |
2. What factors limit deployability to more or fewer subjects?
| Platform | |
| Amazon MTurk | None. But Turk HITS are designed to not take more than a handful of minutes. Not sure if you could or should get one Turker for an hour or more. They might be nervous to make that big a commitment. |
| Cognition.Run | Cognition.run currently does not have a funding mechanism to support hosting costs, so I would be wary of excessively loading their website until they do. They plan to register as a nonprofit to collect donations soon, but if that does not cover costs they will switch to a license fee structure. |
| Django | Depends on the capacity of the server running Django. |
| Gorilla | Gorilla is a paid service. The biggest limiting factor is the fee associated with each respondent’s data. |
| hearX | cost, number of devices owned |
| ispring | There is not really a set limit. Depending on how a test is deployed, for example using the experimenter’s server or the companion ispringlearn server, costs would be different. But if it is hosted by the experimenter and data processing on the experimenter’s server is set up, there is no extra cost or limits on the number of people who can take it. |
| Jacoti | Accessibility to iPhone/iPad and AppleEarPods |
| jspsych | This is really a function of the server you pair jspsych with. |
| MATLAB | Server hardware or the MATLAB web app server software itself will limit the number of simultaneous users able to access the task. Depending on the amount of time the researcher wants to allow data collection it is possible to collect data from larger numbers. User experience will be impacted by mass recruitment if participants receive messages from the server that it is busy or full. |
| PART/BGC Science | PART can be downloaded onto a user’s device, or a pre-loaded device can be sent to a user. BGC Science was developed in order to allow users to download onto their own device and have the correct tests pushed to the device based on a registered email address. Please contact the developers for any specific use questions, as the NIH funding supports working with researchers and clinicians to develop unique testing approaches. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | While there is no upper limit for deployment imposed by Prolific, it seems to be a good idea to open up the studies to a handful participants at a time (say 20 per batch). This is because participants often message the researcher via Prolific with questions and technical issues, which becomes unmanageable when simultaneously deployed to too many participants. However, participants are found quickly and hence a large N can be obtained in a short time but as a sequence of small batches. |
| Psychstudio | None. |
| PsyToolkit | None. |
| Qualtrics | The number of subjects an experimenter is allowed to collect will depend on the Qualtrics package they have purchased (or that the university has purchased). |
| SHOEBOX | Cost, number of iPad systems available to you |
| SpeakPipe | Manual download of audio files. |
| TabSINT | None |
| TeamHearing | Server management. |
3. BRIEFLY describe the capabilities of this platform beyond monaural audio-only air-conduction testing (e.g., binaural, audiovisual, free field, ancillary measures/physiology):
| Platform | |
| Amazon MTurk | Can do anything that requires audio and video presentation through a browser. Via JavaScript you can monitor the entire system, and do your own timing or even capture mouse movements. |
| Cognition.Run | All experiments are implemented in jsPsych, so anything that jsPsych can do. |
| Django | Provides backend logic. This can be used to control flow of the study and serve pages created using front-end platforms like jspsych. |
| Gorilla | Gorilla is capable of monaural & binaural audio playback (embedded in the audio files uploaded to server) and audiovisual studies. There seems to be some variation in the AV synchronization depending on the web browser in use (https://psyarxiv.com/d6nu5/). It very limited in free field implementation beyond 2-channel audio output. Eye tracking/pupillometry through webcams has limited success and heavily relies on postprocessing. I am not aware of additional physiological measures that are compatible with Gorilla at the moment. |
| hearX | video otoscopy (Hearscope), visual acuity (peek acuity), seal of earplugs (sealCheck probe), |
| ispring | Binaural should work as well with two channel files, but not much more. |
| Jacoti | Uses monaural air conduction results to turn device into an amplifier (JacotiListenApp) or FM system (JacotiLola) |
| jspsych | Binaural stimuli possible (Young folks in the lab are able to hear 20 microsecond ITD clearly), audiovisual stimuli possible. One very useful capability is to record reaction times sufficiently reliably to do classic cognitive tasks. |
| MATLAB | My knowledge about these capabilities is limited but MATLAB web app server apps should allow for any possible combinations of stimulus delivery that are possible through HTML or javascript as the MATLAB web apps support loading and executing HTML/javascript code. I have had success in using both monaural and binaural stimuli over both laptop speakers and headphones. |
| PART/BGC Science | binaural, cognitive testing (progressive matrices, inhibitory control, reaction time) |
| Prolific | Not applicable |
| Psychstudio | Video, audio, text, images and any spatial, temporal and visual blending of such. |
| PsyToolkit | |
| Qualtrics | Qualtrics has limited capabilities beyond survey questions. Audio, visual, and audiovisual stimuli can be presented. |
| SHOEBOX | High-frequencies, masking, bone conduction, integrated electronic questionnaires, speech reception and speech discrimination testing |
| SpeakPipe | |
| TabSINT | Supports Dolby 7.1 audio, sound level meter type recordings, bluetooth communication with a computer for additional hardware integrations |
| TeamHearing | Binaural yes, audiovisual yes, tactile yes. |
4. Miscellaneous comments on Capabilities:
| Platform | |
| Amazon MTurk | |
| Cognition.Run | |
| Django | |
| Gorilla | One of the recruitment mechanisms on Gorilla will allow participants to access an experiment using a unique URL, which will allow them to resume the session. Other recruitment mechnisms will forfeit the ongoing session and participants will have to restart. |
| hearX | |
| ispring | |
| Jacoti | |
| jspsych | jspsysch doesn’t place many constraints. However, jspsych can be used to detect basic specs of a user’s screen/browser, compatibility with WebAudio API etc. and the experiment programmer can restrict access based on these variables. |
| MATLAB | |
| PART/BGC Science | PART is available for use on iOS, Android, Mac, and Windows, and BGC Science is available for client-side testing. Contact developers if you have a use case that you are not sure how to implement. This project is fully-funded by NIH and so there is substantial support available for helping researchers and clinicians develop the tools they need. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | Prolific allows researchers to let the subjects know that only participants using certain devices should participate. If the researcher is able to verify independently that this instruction was violated, they can reject a submission (and not pay the participant). |
| Psychstudio | |
| PsyToolkit | |
| Qualtrics | |
| SHOEBOX | |
| SpeakPipe | |
| TabSINT | |
| TeamHearing |
5. How are user-designed experiments developed on this platform?
| Platform | |
| Amazon MTurk | Builder interface/GUI and Code-based |
| Cognition.Run | Code-based |
| Django | Code-based |
| Gorilla | Builder interface/GUI and Code-based |
| hearX | Builder interface/GUI |
| ispring | Builder interface/GUI |
| Jacoti | Builder interface/GUI |
| jspsych | Code-based |
| MATLAB | Code-based |
| PART/BGC Science | Builder interface/GUI |
| Prolific | Not available |
| Psychstudio | Builder interface/GUI |
| Psytoolkit | Code-based |
| Qualtrics | Builder interface/GUI and Code-based |
| SHOEBOX | Builder interface/GUI |
| SpeakPipe | Not Available |
| TabSINT | Code-based |
| TeamHearing | Code-based |
6. What programming languages are used to interface with the platform and generate experiments (check all)?
| Platform | |
| Amazon MTurk | JavaScript, Python, Other options up to user |
| Cognition.Run | JavaScript, the jsPsych library |
| Django | Python; Needs to be paired with front-end HTML/CSS/Javascript (e.g., using Jspsych) |
| Gorilla | JavaScript |
| hearX | Not sure |
| ispring | The software works as a Power point add on. Tests can be developed with no coding experience. This is both the platform’s strength and weakness. Coding may be needed for data processing depending on the type of experiments, size and scope. |
| Jacoti | Apple App |
| jspsych | JavaScript, Because jspsych needs to be paired with a separate server, some basic backend programming knowledge is also required. The jspsych documentation provides examples for how to integrate with a server running PHP code. Other pairings are possible (e.g., with Django or Cognition.Run). |
| MATLAB | Matlab, JavaScript |
| PART/BGC Science | Matlab, JavaScript,Python can be used, but the main method is using the program itself, which contains a complete graphical interface as well as a custom programming syntax for algorithm specification. Please contact the developers with any questions, as the NIH funding supports one-on-one guidance of researchers and clinicians interested in using PART and BGC Science. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | Not applicable |
| Psychstudio | This platform is entirely user interface driven (there is no programming). |
| PsyToolkit | It’s a language specific to Psytoolkit but is not very complicated and there is extensive documentation. |
| Qualtrics | JavaScript |
| SHOEBOX | |
| SpeakPipe | Not applicable |
| TabSINT | JavaScript, JSON |
| TeamHearing | JavaScript |
7. Are ‘ready-made’ experiments available on this platform?
| Platform | |
| Amazon MTurk | No |
| Cognition.Run | No |
| Django | No |
| Gorilla | Yes, there are some cognitive tasks available for community share. In general, sharing Gorilla tasks is very easy between accounts. |
| hearX | Yes, Pure tone audiometry, high-frequency audiometry, digits in noise speech test (HearDigits) |
| ispring | No |
| Jacoti | Yes, Only a pure tone audiogram |
| jspsych | Yes, Some basic examples are provided in the documentation. There are many more user-generated examples available on the web based on jspsych (e.g., see expfactory.org). |
| MATLAB | No officially supported ‘ready-made’ experiments are available. Mathworks provide a rudimentary tutorial on building an app through their GUI based app builder and instructions on how to compile those apps for web use. I am working to complete development work on a port of Stephan Ewert’s "AFC" experiment framework allowing nearly any existing code run in AFC to be run as a web app. |
| PART/BGC Science | Yes, There are a number of batteries that are pre-built and included on the platform at first download, more are available on Github, and the PART team will help support others when researchers are not comfortable setting them up on their own. For descriptions of some of the built-in assessments see the following reference: Lelo de Larrea-Mancera, E.S., Stavropoulos, T., Hoover, E.C., Eddins, D.A., Gallun, F.J., & Seitz, A.R. (2020). Portable Automated Rapid Testing (PART) for auditory research: Validation in a normal hearing population. bioRxiv. https://doi.org/10.1101/2020.01.08.899088 |
| Prolific | No |
| Psychstudio | Yes, demonstration and example experiments and templates available. |
| Psytoolkit | Yes, some things like the Stroop task or Posner cueing task are readily available. |
| Qualtrics | No |
| SHOEBOX | Yes, Meant for portable audiometric testing. |
| SpeakPipe | No |
| TabSINT | Yes, A limited number of example experiments are available. |
| TeamHearing | Yes, Wide range of psychophysical and speech recognition experiments are available. |
8. Please describe any service that the platform provides or uses to allow investigators to share user-designed experiments with other investigators:
| Platform | |
|---|---|
| Amazon MTurk | Personal communication |
| Cognition.Run | None |
| Django | Not inherent to platform, but any code-sharing platforms (like GitHub) can be used. Django itself is open source with code available on GitHub. |
| Gorilla | Available from within platform |
| hearX | Personal communication, Available from within platform, not explicitly stated – but assuming that if a clinician can access the patient data that they can give other users permissions to view data. |
| ispring | Personal communication. See, e.g., http://basebattery.org |
| Jacoti | Can give a health care professional access to results, but meant for direct to consumer. |
| jspsych | All the ways in which javascript code can be shared should work (e.g., GitHub). Jspsych itself is open source with code available on GitHub. |
| MATLAB | The shared online repository for other matlab code could be used as a place for investigators to share experiments and code: https://www.mathworks.com/matlabcentral/fileexchange/ |
| PART/BGC Science | Personal communication and GitHub (development ongoing) |
| Prolific | Not applicable |
| Psychstudio | Available within platform. Experiments can be shared collaboratively with other users and can have ownership transferred to other users. Experiments can also be made available to copy and modify by other users within the same lab. |
| Psytoolkit | Shared online repository and personal communication. |
| Qualtrics | Available from within platform |
| SHOEBOX | |
| SpeakPipe | |
| TabSINT | |
| TeamHearing | Personal communication, Available from within platform |