Platforms /


Developers, Support, and Documentation

1. Which operating systems are compatible with this platform (check all that apply)?

Platform Windows 7 Windows 10 MacOS Linux iOS Android
Amazon MTurk Windows 7 Windows 10 MacOS Linux iOS Android
Cognition.Run Windows 7 Windows 10 MacOS Linux iOS Android
Django Windows 7 Windows 10 MacOS Linux iOS Android
Gorilla Windows 7 Windows 10 MacOS iOS Android
hearX Android
ispring Windows 7 Windows 10 MacOS Linux iOS Android
Jacoti iOS
jspsych Windows 7 Windows 10 MacOS Linux iOS Android
MATLAB Windows 10 Linux iOS
PART/BGC Science Windows 7 Windows 10 MacOS Linux iOS Android
Prolific Windows 7 Windows 10 MacOS Linux iOS Android
Psychstudio Windows 7 Windows 10 MacOS Linux iOS Android
PsyToolkit Windows 7 Windows 10 MacOS Linux iOS Android
Qualtrics Windows 7 Windows 10 MacOS Linux iOS Android
SpeakPipe Windows 7 Windows 10 MacOS Linux iOS Android
TabSINT iOS Android
TeamHearing Windows 7 Windows 10 MacOS Linux iOS Android

2. List any known system-specific performance issues (e.g., driver issues on a given OS, lack of robustness to OS updates, browser compatibility issues):

Amazon MTurk Closed ecosystem for subjects.
Django Easier to run on linux server than windows/Mac
Gorilla Even though Gorilla identifies Internet Explorer as compatible, I was not able to load an experiment on IE. I was told that it may be an issue with Web Audio API compatibility with IE.
Jacoti Must have iOS 12.x or 13.x
jspsych When experiments involve a large number of audio files, some browsers might time the request when the stimuli are being preloaded (i.e., downloaded to the subject’s device by the browser). This is reported by some users online, but we have not experienced it so far (we are new anyway).
MATLAB Browser compatibility is an important concern. Playing sound on client computers requires the use of HTML/Javascript code that is executed by the MATLAB web apps to deliver sound files to participant computers. Once developed this HTML/JS could be used by many experiment apps.
PART/BGC Science Frequency response and minimum output level is hardware and perhaps OS dependent. There are issues with presenting signals below 25 dB SPL using the iPad, which limits audiometric procedures. Current work involves integrating support for already existing custom hardware for presenting wideband high dynamic range stimuli appropriate for ultra high-frequency audiometry up to 16 kHz. Please contact the developers for more information, as the project is NIH-funded to support researchers and clinicians.
Qualtrics There are browser compatibility issues with presenting audio files if the participant is using Microsoft Edge, even if the stimuli are hosted on a separate platform, such as YouTube or SoundCloud. Known issues included intermittently delayed or clipped audio. Information about the participant’s browser can be collected automatically, and then any participants using Microsoft Edge can be excluded. Presenting audio is also unreliable for most mobile devices, so participants should use a computer.
TeamHearing None.

3. How is the platform itself implemented (check all)?

Platform Matlab JavaScript Python Unity Other (please specify)
Amazon MTurk No clue… Amazon proprietary.
Cognition.Run Experiments are written in jsPsych, with a wrapper website built in Laravel, Boostrap, and React.
Django Python  
Gorilla JavaScript  
hearX Stand alone platform on a Samsung Galaxy device purchased through the company.
ispring Unknown.
Jacoti Apple Apps
jspsych JavaScript  
MATLAB Matlab  
PART/BGC Science Unity  
Prolific Not sure.
Psychstudio JavaScript Server: Clojure and Java. Web: Transpiled Clojurescript to Javascript.
SHOEBOX Only available on an iPad. Unclear if you can download an app or if you have to purchase the entire system from SHOEBOX
TabSINT JavaScript HTML5
TeamHearing JavaScript Web application combining HTML and PHP.

4. Is the platform open source?

Platform Yes or No Dependencies
Amazon MTurk No  
Cognition.Run Yes  
Django Yes It is python based.
Gorilla No  
hearX Yes  
ispring No  
Jacoti Yes Available for free in the apple app store
jspsych Yes Uses WebAudio API for delivering audio although native HTML5 audio is also supported (with less timing control).
PART/BGC Science No  
Prolific No  
Psychstudio No  
Qualtrics No  
TabSINT Yes TabSINT has a long list of standard dependencies
TeamHearing Yes  

5. Who are the developers of this platform?

Amazon MTurk Amazon
Cognition.Run Elisabet Alzueta (SRI International) Sebastian Michelmann (Princeton) Mar Yebra (LA Cedars Sinai) Javier Vidal Peña (software developer at Facebook)
Django Developed by folks at an independent 501(c)(3) non-profit called the Django Software Foundation
hearX HearXgroup
Jacoti Jacoti, partnered with Med El
jspsych Josh de Leeuw created it. Others have made contributions:
MATLAB Mathworks
PART/BGC Science UC Riverside Brain Game Center
Prolific Managed by a privately owned company.
Psychstudio Ben Howell:
PsyToolkit Professor Gijsbert Stoet
SHOEBOX Scientific advisory board: Nicholas Reed, Susan Scollie, Ian Windmill, Annie Main
TabSINT Creare
TeamHearing Ray Goldsworthy, USC

6. Describe any relationships, permissions, or licensing required to use this platform:

Amazon MTurk None. Everything is fair game.
Cognition.Run None required
Django BSD license (very permissive)
Gorilla Registration and experiment building is free, but it charges a fee at deployment based on the target recruitment size. The fee is also scalable depending on the account type with academic pricing available.
hearX Must purchase platform and subscriptions for use of each test through the company on a month by month or yearly rate.
ispring Purchase from developer.
Jacoti N/A.
jspsych MIT license (which is very open/permissive).
MATLAB A license for the MATLAB web app server must be purchased in order to use this platform.
PART/BGC Science None – these applications were built with NIH funding and are intended to be freely available for researchers and clinicians.
Prolific They have a permissive ToA that is likely to accommodate most researcher’s needs.
Psychstudio Paid account.
PsyToolkit Just need to cite 2 papers.
Qualtrics There is a fee to use the platform if the university does not already provide it. Free accounts are available, but there are limitations as to what types of questions the experimenter can present and the number of participants that are permitted to complete the experiment.
SHOEBOX Must pay for a subscription
TabSINT MIT, BSD, Apache, etc
TeamHearing Ray Goldsworthy, USC

7. Miscellaneous comments on Development from the user perspective:

Amazon MTurk Can be challenging, but anything is possible.
Cognition.Run generates unique URLs to each task, so it is easy for participants to navigate to the task website.
Django Django nicely separates the backend logic from the front-end appearance in the form of a model-view-controller pattern. The documentation makes it easy for non-technical users to setup simple flow controls across different pages that each maybe built using front-end libraries like jspsych. Django has support for securely capturing and saving data to the database. Also bundled with it are secure authentication systems and an easy to use admin interface for examining and manipulating database content. Database logic can be coded using python instead of SQL.
Gorilla The Gorilla platform is built using JavaScript. It allows front-end modular builder and also coding as an alternative. It’s friendly for investigators without any programming background to build complex experiments. But the coding option is more robust in creating versatile experiment GUI, procedures, etc. One critical functionality I found out is that — the staircase procedure needs to be implemented using the coding option.
hearX Seems like a wonderful product, but use will be highly dependent on the HearX group for support.
jspsych Some HTML, CSS knowledge needed to make things appear as desired. Intermediate level familiarity is needed to setup backend/server components. Otherwise, jspsych has been a fantastic platform to work with. Students in my lab and myself are able to quickly create experiments.
MATLAB Users of the platform will likely need support from institutional IT resources to initially set up and configure the server in compliance with IRB/HIPPA guidelines. After initial setup, adding new applications is a fairly straightforward process that should not require skills beyond normal MATLAB development skills.
PART/BGC Science Experiments can be coded within PART or downloaded from Github or shared via email using the JSON file format. MATLAB and Python tools are available for analysis of the data files, which are also stored in JSON format, but not for generating experiments. Thresholds can be read directly from the PART interface or exported as text files. This aspect of the platform is currently undergoing upgrading and development, so researchers and clinicians are encouraged to contact the developers to get the latest tools and to get them customized for their projects.
Prolific Understanding the culture and payment/screening options within Prolific is important for more complex study designs to be successful on the platform (e.g., longitudinal design, online quality check and early termination, etc.).
SHOEBOX Very professional looking and easy to use.
TeamHearing Designed for forced choice experiments, other experiments can be designed but require sophistication from the user.

8. Miscellaneous comments on Development from the developer perspective (platform maintenance, expansion, etc.):

Amazon MTurk  
Cognition.Run The HTML/CSS portions of the website as well as data logging are taken care of, so the only thing the experimenter needs to provide is the jsPsych script to set up and run the experiment.
Django Django seems to be a widely used framework for creating web applications with a large user base. So my best guess is that it would be available and maintained for a long time.
Gorilla Server/platform maintenance is part of Gorilla’s paid service. Investigators are in separate roles with the developers for this platform.
hearX New products and updates will be available for purchase. For example, within the last two weeks the video otoscope became available.
jspsych Josh de Leeuw seems actively interested in maintaining jspsych and is generous with his time in responding in the discussion group. Because the platform is open source, many people can expand the capabilities (and seem to have done so over the years).
MATLAB The development process for web apps is largely similar to developing other software within matlab. Some additional work is necessary to render sounds on client computers, but only needs to be done once and can be reused across many experiments. A final compilation step that is not normally part of matlab software development is required to package the code into an app for deployment onto the web server.
PART/BGC Science Both of these platforms are currently in active development, with new capacities and collaborative projects coming on line every month. The developers encourage users and potential users to reach out with their needs and questions. NIH funds are available for individual support for research and clinical projects.
SHOEBOX Assuming that updates will be made as needed with minimal interruption.
TeamHearing Developer will provide complete copy of code and template files for creating a new module.

9. Cost for software, licenses, or user accounts:

Amazon MTurk Think it’s a percentage of the user payments. It was small compared to user payments.
Cognition.Run Currently free. This is the response that Javier Vidal Peña provided when I asked about it: "We would like to fund the platform by donations. A Nonprofit Public Benefit Corporation is being created in California to articulate that. We will add a ‘Donate’ button in the next days. We are gonna work hard to keep the platform free with this donation system. However, if donations don’t cover the expenses of the platform, we cannot ensure that the organization/structure will remain the same in the long future, and a licensing system would be implemented."
Django Free
Gorilla There is a per-respondent fee. See
hearX There is an annual fee. See
Jacoti Free
jspsych Free
MATLAB There are license fees that vary depending on type of institution, type of license, number of toolboxes installed ,etc.
PART/BGC Science Free
Prolific Fees are charged as a fraction of each dollar paid to participants.
Psychstudio Annual pricing plans at
PsyToolkit Free
Qualtrics Variable annual cost depending on features requested
SHOEBOX Annual fees for license, hardware, annual calibration, set-up, and ongoing support. Cost will vary depending on use case, number of units, and length of license period. Various discount may be available to researchers.
SpeakPipe Free
TabSINT Free
TeamHearing Free

10. Describe the mechanisms for support

Platform Online documentation / Wiki User forums Direct access to developers Other (please specify and rate)
Amazon MTurk Excellent Acceptable Poor  
Cognition.Run Not Available Not Available Excellent  
Django Excellent Excellent Acceptable  
Gorilla Excellent Not Available Excellent Investigators can request a Gorilla programmer to build an experiment with a fee.
hearX Excellent Not Available Acceptable  
ispring Acceptable Not Available Acceptable Customer service can be quite helpful, but you don’t talk to actual developers.
Jacoti Excellent Not Available Acceptable  
jspsych Excellent Excellent Excellent The GitHub repository for jspsych has a "discussion" feature where questions can be posted. Developers or other users are quick to respond and are also often good at deciphering what the user is saying/asking.
MATLAB Acceptable Acceptable UNK I don’t have any experience trying to gain access to mathworks developers and cannot comment on what its like to get direct support in this regard. General customer support is available and has been fairly responsive.
PART/BGC Science Acceptable Acceptable Excellent The best way to learn about using this platform is to reach out to the developers and let them use their NIH-funded time to support you directly.
Prolific Excellent Acceptable Excellent They have recommendations for how to use their platform for various study designs. There are multiple grassroots fora (e.g., on reddit) that also provide useful community-sourced information. Requests for support can be submitted to Prolific directly too, but this is organized customer support (what you usually want) rather than access to developers per se.
Psychstudio Excellent Not Available Excellent  
PsyToolkit Excellent Not Available Excellent  
Qualtrics Excellent Excellent Poor  
SHOEBOX Excellent Not Available Acceptable  
SpeakPipe Not Available Not Available Excellent  
TabSINT Acceptable Acceptable Poor  
TeamHearing Not Available Not Available Excellent  

11. Rate the quality of the documentation:

Platform Complete Understandable Up-to-date
Amazon MTurk Excellent Excellent Excellent
Cognition.Run Excellent Excellent Excellent
Django Excellent Excellent Excellent
Gorilla Excellent Excellent Acceptable
hearX Acceptable Acceptable Acceptable
ispring Acceptable Acceptable Acceptable
Jacoti Acceptable Excellent Excellent
jspsych Acceptable Excellent Excellent
MATLAB Acceptable Acceptable Acceptable
PART/BGC Science Acceptable Acceptable Poor – the platform is changing so rapidly, that the documentation is necessarily lagging. Direct contact with the PART team is encouraged. Once the platform stops advancing so rapidly, the documentation will catch up, with videos and examples as well as user-friendly scripts based on completed projects.
Prolific Acceptable Acceptable Acceptable
Psychstudio Excellent Acceptable Excellent
PsyToolkit Excellent Excellent Excellent
Qualtrics Acceptable Excellent Acceptable
SHOEBOX Excellent Excellent Excellent
SpeakPipe Acceptable Acceptable Acceptable
TabSINT Acceptable Acceptable Excellent