Platforms /

DevelopersSupportAndDocumentation

Developers, Support, and Documentation

1. Which operating systems are compatible with this platform (check all that apply)?

Platform	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Amazon MTurk	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Cognition.Run	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Django	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Gorilla	Windows 7	Windows 10	MacOS	—	iOS	Android
hearX	—	—	—	—	—	Android
ispring	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Jacoti	—	—	—	—	iOS	—
jspsych	Windows 7	Windows 10	MacOS	Linux	iOS	Android
MATLAB	—	Windows 10	—	Linux	iOS	—
PART/BGC Science	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Prolific	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Psychstudio	Windows 7	Windows 10	MacOS	Linux	iOS	Android
PsyToolkit	Windows 7	Windows 10	MacOS	Linux	iOS	Android
Qualtrics	Windows 7	Windows 10	MacOS	Linux	iOS	Android
SHOEBOX	—	—	—	—	iOS	—
SpeakPipe	Windows 7	Windows 10	MacOS	Linux	iOS	Android
TabSINT	—	—	—	—	iOS	Android
TeamHearing	Windows 7	Windows 10	MacOS	Linux	iOS	Android

2. List any known system-specific performance issues (e.g., driver issues on a given OS, lack of robustness to OS updates, browser compatibility issues):

Platform
Amazon MTurk	Closed ecosystem for subjects.
Cognition.Run
Django	Easier to run on linux server than windows/Mac
Gorilla	Even though Gorilla identifies Internet Explorer as compatible, I was not able to load an experiment on IE. I was told that it may be an issue with Web Audio API compatibility with IE.
hearX
ispring
Jacoti	Must have iOS 12.x or 13.x
jspsych	When experiments involve a large number of audio files, some browsers might time the request when the stimuli are being preloaded (i.e., downloaded to the subject’s device by the browser). This is reported by some users online, but we have not experienced it so far (we are new anyway).
MATLAB	Browser compatibility is an important concern. Playing sound on client computers requires the use of HTML/Javascript code that is executed by the MATLAB web apps to deliver sound files to participant computers. Once developed this HTML/JS could be used by many experiment apps.
PART/BGC Science	Frequency response and minimum output level is hardware and perhaps OS dependent. There are issues with presenting signals below 25 dB SPL using the iPad, which limits audiometric procedures. Current work involves integrating support for already existing custom hardware for presenting wideband high dynamic range stimuli appropriate for ultra high-frequency audiometry up to 16 kHz. Please contact the developers for more information, as the project is NIH-funded to support researchers and clinicians. https://braingamecenter.ucr.edu/games/p-a-r-t/
Prolific
Psychstudio
PsyToolkit
Qualtrics	There are browser compatibility issues with presenting audio files if the participant is using Microsoft Edge, even if the stimuli are hosted on a separate platform, such as YouTube or SoundCloud. Known issues included intermittently delayed or clipped audio. Information about the participant’s browser can be collected automatically, and then any participants using Microsoft Edge can be excluded. Presenting audio is also unreliable for most mobile devices, so participants should use a computer.
SHOEBOX
SpeakPipe
TabSINT
TeamHearing	None.

3. How is the platform itself implemented (check all)?

Platform	Matlab	JavaScript	Python	Unity	Other (please specify)
Amazon MTurk	—	—	—	—	No clue… Amazon proprietary.
Cognition.Run	—	—	—	—	Experiments are written in jsPsych, with a wrapper website built in Laravel, Boostrap, and React.
Django	—	—	Python	—
Gorilla	—	JavaScript	—	—
hearX	—	—	—	—	Stand alone platform on a Samsung Galaxy device purchased through the company.
ispring	—	—	—	—	Unknown.
Jacoti	—	—	—	—	Apple Apps
jspsych	—	JavaScript	—	—
MATLAB	Matlab	—	—	—
PART/BGC Science	—	—	—	Unity
Prolific	—	—	—	—	Not sure.
Psychstudio	JavaScript	—	—	—	Server: Clojure and Java. Web: Transpiled Clojurescript to Javascript.
Psytoolkit	—	—	—	—
Qualtrics	—	—	—	—
SHOEBOX	—	—	—	—	Only available on an iPad. Unclear if you can download an app or if you have to purchase the entire system from SHOEBOX
SpeakPipe	—	—	—	—
TabSINT	—	JavaScript	—	—	HTML5
TeamHearing	—	JavaScript	—	—	Web application combining HTML and PHP.

4. Is the platform open source?

Platform	Yes or No	Dependencies
Amazon MTurk	No
Cognition.Run	Yes
Django	Yes	It is python based.
Gorilla	No
hearX	Yes
ispring	No
Jacoti	Yes	Available for free in the apple app store
jspsych	Yes	Uses WebAudio API for delivering audio although native HTML5 audio is also supported (with less timing control).
MATLAB	No
PART/BGC Science	No
Prolific	No
Psychstudio	No
PsyToolkit
Qualtrics	No
SHOEBOX	Yes
SpeakPipe
TabSINT	Yes	TabSINT has a long list of standard dependencies
TeamHearing	Yes

5. Who are the developers of this platform?

Platform
Amazon MTurk	Amazon
Cognition.Run	Elisabet Alzueta (SRI International) Sebastian Michelmann (Princeton) Mar Yebra (LA Cedars Sinai) Javier Vidal Peña (software developer at Facebook)
Django	Developed by folks at an independent 501(c)(3) non-profit called the Django Software Foundation
Gorilla	Gorilla.sc
hearX	HearXgroup https://www.hearxgroup.com/about
ispring	ispringsolutions.com
Jacoti	Jacoti, partnered with Med El
jspsych	Josh de Leeuw created it. Others have made contributions: https://github.com/jspsych/jsPsych/blob/master/contributors.md
MATLAB	Mathworks
PART/BGC Science	UC Riverside Brain Game Center https://braingamecenter.ucr.edu/games/p-a-r-t/
Prolific	Managed by a privately owned company.
Psychstudio	Ben Howell: https://www.psychstudio.com/about#ben-howell
PsyToolkit	Professor Gijsbert Stoet
Qualtrics
SHOEBOX	https://www.shoebox.md/we-are-shoebox-audiometry/ Scientific advisory board: Nicholas Reed, Susan Scollie, Ian Windmill, Annie Main
SpeakPipe
TabSINT	Creare https://www.creare.com/https://gitlab.com/creare-com/tabsint
TeamHearing	Ray Goldsworthy, USC

6. Describe any relationships, permissions, or licensing required to use this platform:

Platform
Amazon MTurk	None. Everything is fair game.
Cognition.Run	None required
Django	BSD license (very permissive)
Gorilla	Registration and experiment building is free, but it charges a fee at deployment based on the target recruitment size. The fee is also scalable depending on the account type with academic pricing available.
hearX	Must purchase platform and subscriptions for use of each test through the company on a month by month or yearly rate.
ispring	Purchase from developer.
Jacoti	N/A.
jspsych	MIT license (which is very open/permissive).
MATLAB	A license for the MATLAB web app server must be purchased in order to use this platform.
PART/BGC Science	None – these applications were built with NIH funding and are intended to be freely available for researchers and clinicians. https://braingamecenter.ucr.edu/games/p-a-r-t/
Prolific	They have a permissive ToA that is likely to accommodate most researcher’s needs. https://www.prolific.co/assets/docs/Researcher_Terms.pdf
Psychstudio	Paid account.
PsyToolkit	Just need to cite 2 papers.
Qualtrics	There is a fee to use the platform if the university does not already provide it. Free accounts are available, but there are limitations as to what types of questions the experimenter can present and the number of participants that are permitted to complete the experiment.
SHOEBOX	Must pay for a subscription
SpeakPipe
TabSINT	MIT, BSD, Apache, etc
TeamHearing	Ray Goldsworthy, USC

7. Miscellaneous comments on Development from the user perspective:

Platform
Amazon MTurk	Can be challenging, but anything is possible.
Cognition.Run	Cognition.run generates unique URLs to each task, so it is easy for participants to navigate to the task website.
Django	Django nicely separates the backend logic from the front-end appearance in the form of a model-view-controller pattern. The documentation makes it easy for non-technical users to setup simple flow controls across different pages that each maybe built using front-end libraries like jspsych. Django has support for securely capturing and saving data to the database. Also bundled with it are secure authentication systems and an easy to use admin interface for examining and manipulating database content. Database logic can be coded using python instead of SQL.
Gorilla	The Gorilla platform is built using JavaScript. It allows front-end modular builder and also coding as an alternative. It’s friendly for investigators without any programming background to build complex experiments. But the coding option is more robust in creating versatile experiment GUI, procedures, etc. One critical functionality I found out is that — the staircase procedure needs to be implemented using the coding option.
hearX	Seems like a wonderful product, but use will be highly dependent on the HearX group for support.
ispring
Jacoti
jspsych	Some HTML, CSS knowledge needed to make things appear as desired. Intermediate level familiarity is needed to setup backend/server components. Otherwise, jspsych has been a fantastic platform to work with. Students in my lab and myself are able to quickly create experiments.
MATLAB	Users of the platform will likely need support from institutional IT resources to initially set up and configure the server in compliance with IRB/HIPPA guidelines. After initial setup, adding new applications is a fairly straightforward process that should not require skills beyond normal MATLAB development skills.
PART/BGC Science	Experiments can be coded within PART or downloaded from Github or shared via email using the JSON file format. MATLAB and Python tools are available for analysis of the data files, which are also stored in JSON format, but not for generating experiments. Thresholds can be read directly from the PART interface or exported as text files. This aspect of the platform is currently undergoing upgrading and development, so researchers and clinicians are encouraged to contact the developers to get the latest tools and to get them customized for their projects. https://braingamecenter.ucr.edu/games/p-a-r-t/
Prolific	Understanding the culture and payment/screening options within Prolific is important for more complex study designs to be successful on the platform (e.g., longitudinal design, online quality check and early termination, etc.).
Psychstudio
PsyToolkit
Qualtrics
SHOEBOX	Very professional looking and easy to use.
SpeakPipe
TabSINT
TeamHearing	Designed for forced choice experiments, other experiments can be designed but require sophistication from the user.

8. Miscellaneous comments on Development from the developer perspective (platform maintenance, expansion, etc.):

Platform
Amazon MTurk
Cognition.Run	The HTML/CSS portions of the website as well as data logging are taken care of, so the only thing the experimenter needs to provide is the jsPsych script to set up and run the experiment.
Django	Django seems to be a widely used framework for creating web applications with a large user base. So my best guess is that it would be available and maintained for a long time.
Gorilla	Server/platform maintenance is part of Gorilla’s paid service. Investigators are in separate roles with the developers for this platform.
hearX	New products and updates will be available for purchase. For example, within the last two weeks the video otoscope became available.
ispring
Jacoti
jspsych	Josh de Leeuw seems actively interested in maintaining jspsych and is generous with his time in responding in the discussion group. Because the platform is open source, many people can expand the capabilities (and seem to have done so over the years).
MATLAB	The development process for web apps is largely similar to developing other software within matlab. Some additional work is necessary to render sounds on client computers, but only needs to be done once and can be reused across many experiments. A final compilation step that is not normally part of matlab software development is required to package the code into an app for deployment onto the web server.
PART/BGC Science	Both of these platforms are currently in active development, with new capacities and collaborative projects coming on line every month. The developers encourage users and potential users to reach out with their needs and questions. NIH funds are available for individual support for research and clinical projects. https://braingamecenter.ucr.edu/games/p-a-r-t/
Prolific
Psychstudio
PsyToolkit
Qualtrics
SHOEBOX	Assuming that updates will be made as needed with minimal interruption.
SpeakPipe
TabSINT
TeamHearing	Developer will provide complete copy of code and template files for creating a new module.

9. Cost for software, licenses, or user accounts:

Platform
Amazon MTurk	Think it’s a percentage of the user payments. It was small compared to user payments.
Cognition.Run	Currently free. This is the response that Javier Vidal Peña provided when I asked about it: "We would like to fund the platform by donations. A Nonprofit Public Benefit Corporation is being created in California to articulate that. We will add a ‘Donate’ button in the next days. We are gonna work hard to keep the platform free with this donation system. However, if donations don’t cover the expenses of the platform, we cannot ensure that the organization/structure will remain the same in the long future, and a licensing system would be implemented."
Django	Free
Gorilla	There is a per-respondent fee. See https://gorilla.sc/pricing
hearX	There is an annual fee. See https://www.hearxgroup.com/shop/hearscreenhttps://www.hearxgroup.com/shop/heartesthttps://www.hearxgroup.com/shop/heardigits
ispring	https://www.ispringsolutions.com/pricing
Jacoti	Free
jspsych	Free
MATLAB	There are license fees that vary depending on type of institution, type of license, number of toolboxes installed ,etc. https://www.mathworks.com/store/
PART/BGC Science	Free
Prolific	Fees are charged as a fraction of each dollar paid to participants.
Psychstudio	Annual pricing plans at https://www.psychstudio.com/#pricing
PsyToolkit	Free
Qualtrics	Variable annual cost depending on features requested
SHOEBOX	Annual fees for license, hardware, annual calibration, set-up, and ongoing support. Cost will vary depending on use case, number of units, and length of license period. Various discount may be available to researchers. https://www.shoebox.md/quote/
SpeakPipe	Free
TabSINT	Free
TeamHearing	Free

10. Describe the mechanisms for support

Platform	Online documentation / Wiki	User forums	Direct access to developers	Other (please specify and rate)
Amazon MTurk	Excellent	Acceptable	Poor
Cognition.Run	Not Available	Not Available	Excellent
Django	Excellent	Excellent	Acceptable
Gorilla	Excellent	Not Available	Excellent	Investigators can request a Gorilla programmer to build an experiment with a fee.
hearX	Excellent	Not Available	Acceptable
ispring	Acceptable	Not Available	Acceptable	Customer service can be quite helpful, but you don’t talk to actual developers.
Jacoti	Excellent	Not Available	Acceptable
jspsych	Excellent	Excellent	Excellent	The GitHub repository for jspsych has a "discussion" feature where questions can be posted. Developers or other users are quick to respond and are also often good at deciphering what the user is saying/asking.
MATLAB	Acceptable	Acceptable	UNK	I don’t have any experience trying to gain access to mathworks developers and cannot comment on what its like to get direct support in this regard. General customer support is available and has been fairly responsive.
PART/BGC Science	Acceptable	Acceptable	Excellent	The best way to learn about using this platform is to reach out to the developers and let them use their NIH-funded time to support you directly. https://braingamecenter.ucr.edu/games/p-a-r-t/
Prolific	Excellent	Acceptable	Excellent	They have recommendations for how to use their platform for various study designs. There are multiple grassroots fora (e.g., on reddit) that also provide useful community-sourced information. Requests for support can be submitted to Prolific directly too, but this is organized customer support (what you usually want) rather than access to developers per se.
Psychstudio	Excellent	Not Available	Excellent
PsyToolkit	Excellent	Not Available	Excellent
Qualtrics	Excellent	Excellent	Poor
SHOEBOX	Excellent	Not Available	Acceptable
SpeakPipe	Not Available	Not Available	Excellent
TabSINT	Acceptable	Acceptable	Poor
TeamHearing	Not Available	Not Available	Excellent

11. Rate the quality of the documentation:

Platform	Complete	Understandable	Up-to-date
Amazon MTurk	Excellent	Excellent	Excellent
Cognition.Run	Excellent	Excellent	Excellent
Django	Excellent	Excellent	Excellent
Gorilla	Excellent	Excellent	Acceptable
hearX	Acceptable	Acceptable	Acceptable
ispring	Acceptable	Acceptable	Acceptable
Jacoti	Acceptable	Excellent	Excellent
jspsych	Acceptable	Excellent	Excellent
MATLAB	Acceptable	Acceptable	Acceptable
PART/BGC Science	Acceptable	Acceptable	Poor – the platform is changing so rapidly, that the documentation is necessarily lagging. Direct contact with the PART team is encouraged. Once the platform stops advancing so rapidly, the documentation will catch up, with videos and examples as well as user-friendly scripts based on completed projects.
Prolific	Acceptable	Acceptable	Acceptable
Psychstudio	Excellent	Acceptable	Excellent
PsyToolkit	Excellent	Excellent	Excellent
Qualtrics	Acceptable	Excellent	Acceptable
SHOEBOX	Excellent	Excellent	Excellent
SpeakPipe	Acceptable	Acceptable	Acceptable
TabSINT	Acceptable	Acceptable	Excellent
TeamHearing