|
Main
Platforms
|
DevelopersSupportAndDocumentation
Developers, Support, and Documentation
1. Which operating systems are compatible with this platform (check all that apply)?
| Platform | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
|---|
| Amazon MTurk | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| Cognition.Run | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| Django | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| Gorilla | Windows 7 | Windows 10 | MacOS | -- | iOS | Android |
| hearX | -- | -- | -- | -- | -- | Android |
| ispring | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| Jacoti | -- | -- | -- | -- | iOS | -- |
| jspsych | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| MATLAB | -- | Windows 10 | -- | Linux | iOS | -- |
| PART/BGC Science | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| Prolific | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| Psychstudio | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| PsyToolkit | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| Qualtrics | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| SHOEBOX | -- | -- | -- | -- | iOS | -- |
| SpeakPipe | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
| TabSINT | -- | -- | -- | -- | iOS | Android |
| TeamHearing | Windows 7 | Windows 10 | MacOS | Linux | iOS | Android |
2. List any known system-specific performance issues (e.g., driver issues on a given OS, lack of robustness to OS updates, browser compatibility issues):
| Platform | |
| Amazon MTurk | Closed ecosystem for subjects. |
| Cognition.Run | |
| Django | Easier to run on linux server than windows/Mac |
| Gorilla | Even though Gorilla identifies Internet Explorer as compatible, I was not able to load an experiment on IE. I was told that it may be an issue with Web Audio API compatibility with IE. |
| hearX | |
| ispring | |
| Jacoti | Must have iOS 12.x or 13.x |
| jspsych | When experiments involve a large number of audio files, some browsers might time the request when the stimuli are being preloaded (i.e., downloaded to the subject's device by the browser). This is reported by some users online, but we have not experienced it so far (we are new anyway). |
| MATLAB | Browser compatibility is an important concern. Playing sound on client computers requires the use of HTML/Javascript code that is executed by the MATLAB web apps to deliver sound files to participant computers. Once developed this HTML/JS could be used by many experiment apps. |
| PART/BGC Science | Frequency response and minimum output level is hardware and perhaps OS dependent. There are issues with presenting signals below 25 dB SPL using the iPad, which limits audiometric procedures. Current work involves integrating support for already existing custom hardware for presenting wideband high dynamic range stimuli appropriate for ultra high-frequency audiometry up to 16 kHz. Please contact the developers for more information, as the project is NIH-funded to support researchers and clinicians. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | |
| Psychstudio | |
| PsyToolkit | |
| Qualtrics | There are browser compatibility issues with presenting audio files if the participant is using Microsoft Edge, even if the stimuli are hosted on a separate platform, such as YouTube or SoundCloud. Known issues included intermittently delayed or clipped audio. Information about the participant's browser can be collected automatically, and then any participants using Microsoft Edge can be excluded. Presenting audio is also unreliable for most mobile devices, so participants should use a computer. |
| SHOEBOX | |
| SpeakPipe | |
| TabSINT | |
| TeamHearing | None. |
3. How is the platform itself implemented (check all)?
| Platform | Matlab | JavaScript | Python | Unity | Other (please specify) |
|---|
| Amazon MTurk | -- | -- | -- | -- | No clue... Amazon proprietary. |
| Cognition.Run | -- | -- | -- | -- | Experiments are written in jsPsych, with a wrapper website built in Laravel, Boostrap, and React. |
| Django | -- | -- | Python | -- | |
| Gorilla | -- | JavaScript | -- | -- | |
| hearX | -- | -- | -- | -- | Stand alone platform on a Samsung Galaxy device purchased through the company. |
| ispring | -- | -- | -- | -- | Unknown. |
| Jacoti | -- | -- | -- | -- | Apple Apps |
| jspsych | -- | JavaScript | -- | -- | |
| MATLAB | Matlab | -- | -- | -- | |
| PART/BGC Science | -- | -- | -- | Unity | |
| Prolific | -- | -- | -- | -- | Not sure. |
| Psychstudio | JavaScript | -- | -- | -- | Server: Clojure and Java. Web: Transpiled Clojurescript to Javascript. |
| Psytoolkit | -- | -- | -- | -- | |
| Qualtrics | -- | -- | -- | -- | |
| SHOEBOX | -- | -- | -- | -- | Only available on an iPad. Unclear if you can download an app or if you have to purchase the entire system from SHOEBOX |
| SpeakPipe | -- | -- | -- | -- | |
| TabSINT | -- | JavaScript | -- | -- | HTML5 |
| TeamHearing | -- | JavaScript | -- | -- | Web application combining HTML and PHP. |
4. Is the platform open source?
| Platform | Yes or No | Dependencies |
| Amazon MTurk | No | |
| Cognition.Run | Yes | |
| Django | Yes | It is python based. |
| Gorilla | No | |
| hearX | Yes | |
| ispring | No | |
| Jacoti | Yes | Available for free in the apple app store |
| jspsych | Yes | Uses WebAudio API for delivering audio although native HTML5 audio is also supported (with less timing control). |
| MATLAB | No | |
| PART/BGC Science | No | |
| Prolific | No | |
| Psychstudio | No | |
| PsyToolkit | | |
| Qualtrics | No | |
| SHOEBOX | Yes | |
| SpeakPipe | | |
| TabSINT | Yes | TabSINT has a long list of standard dependencies |
| TeamHearing | Yes | |
5. Who are the developers of this platform?
6. Describe any relationships, permissions, or licensing required to use this platform:
| Platform | |
| Amazon MTurk | None. Everything is fair game. |
| Cognition.Run | None required |
| Django | BSD license (very permissive) |
| Gorilla | Registration and experiment building is free, but it charges a fee at deployment based on the target recruitment size. The fee is also scalable depending on the account type with academic pricing available. |
| hearX | Must purchase platform and subscriptions for use of each test through the company on a month by month or yearly rate. |
| ispring | Purchase from developer. |
| Jacoti | N/A. |
| jspsych | MIT license (which is very open/permissive). |
| MATLAB | A license for the MATLAB web app server must be purchased in order to use this platform. |
| PART/BGC Science | None - these applications were built with NIH funding and are intended to be freely available for researchers and clinicians. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | They have a permissive ToA that is likely to accommodate most researcher's needs. https://www.prolific.co/assets/docs/Researcher_Terms.pdf |
| Psychstudio | Paid account. |
| PsyToolkit | Just need to cite 2 papers. |
| Qualtrics | There is a fee to use the platform if the university does not already provide it. Free accounts are available, but there are limitations as to what types of questions the experimenter can present and the number of participants that are permitted to complete the experiment. |
| SHOEBOX | Must pay for a subscription |
| SpeakPipe | |
| TabSINT | MIT, BSD, Apache, etc |
| TeamHearing | Ray Goldsworthy, USC |
7. Miscellaneous comments on Development from the user perspective:
| Platform | |
| Amazon MTurk | Can be challenging, but anything is possible. |
| Cognition.Run | Cognition.run generates unique URLs to each task, so it is easy for participants to navigate to the task website. |
| Django | Django nicely separates the backend logic from the front-end appearance in the form of a model-view-controller pattern. The documentation makes it easy for non-technical users to setup simple flow controls across different pages that each maybe built using front-end libraries like jspsych. Django has support for securely capturing and saving data to the database. Also bundled with it are secure authentication systems and an easy to use admin interface for examining and manipulating database content. Database logic can be coded using python instead of SQL. |
| Gorilla | The Gorilla platform is built using JavaScript. It allows front-end modular builder and also coding as an alternative. It's friendly for investigators without any programming background to build complex experiments. But the coding option is more robust in creating versatile experiment GUI, procedures, etc. One critical functionality I found out is that -- the staircase procedure needs to be implemented using the coding option. |
| hearX | Seems like a wonderful product, but use will be highly dependent on the HearX group for support. |
| ispring | |
| Jacoti | |
| jspsych | Some HTML, CSS knowledge needed to make things appear as desired. Intermediate level familiarity is needed to setup backend/server components. Otherwise, jspsych has been a fantastic platform to work with. Students in my lab and myself are able to quickly create experiments. |
| MATLAB | Users of the platform will likely need support from institutional IT resources to initially set up and configure the server in compliance with IRB/HIPPA guidelines. After initial setup, adding new applications is a fairly straightforward process that should not require skills beyond normal MATLAB development skills. |
| PART/BGC Science | Experiments can be coded within PART or downloaded from Github or shared via email using the JSON file format. MATLAB and Python tools are available for analysis of the data files, which are also stored in JSON format, but not for generating experiments. Thresholds can be read directly from the PART interface or exported as text files. This aspect of the platform is currently undergoing upgrading and development, so researchers and clinicians are encouraged to contact the developers to get the latest tools and to get them customized for their projects. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | Understanding the culture and payment/screening options within Prolific is important for more complex study designs to be successful on the platform (e.g., longitudinal design, online quality check and early termination, etc.). |
| Psychstudio | |
| PsyToolkit | |
| Qualtrics | |
| SHOEBOX | Very professional looking and easy to use. |
| SpeakPipe | |
| TabSINT | |
| TeamHearing | Designed for forced choice experiments, other experiments can be designed but require sophistication from the user. |
8. Miscellaneous comments on Development from the developer perspective (platform maintenance, expansion, etc.):
| Platform | |
| Amazon MTurk | |
| Cognition.Run | The HTML/CSS portions of the website as well as data logging are taken care of, so the only thing the experimenter needs to provide is the jsPsych script to set up and run the experiment. |
| Django | Django seems to be a widely used framework for creating web applications with a large user base. So my best guess is that it would be available and maintained for a long time. |
| Gorilla | Server/platform maintenance is part of Gorilla's paid service. Investigators are in separate roles with the developers for this platform. |
| hearX | New products and updates will be available for purchase. For example, within the last two weeks the video otoscope became available. |
| ispring | |
| Jacoti | |
| jspsych | Josh de Leeuw seems actively interested in maintaining jspsych and is generous with his time in responding in the discussion group. Because the platform is open source, many people can expand the capabilities (and seem to have done so over the years). |
| MATLAB | The development process for web apps is largely similar to developing other software within matlab. Some additional work is necessary to render sounds on client computers, but only needs to be done once and can be reused across many experiments. A final compilation step that is not normally part of matlab software development is required to package the code into an app for deployment onto the web server. |
| PART/BGC Science | Both of these platforms are currently in active development, with new capacities and collaborative projects coming on line every month. The developers encourage users and potential users to reach out with their needs and questions. NIH funds are available for individual support for research and clinical projects. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | |
| Psychstudio | |
| PsyToolkit | |
| Qualtrics | |
| SHOEBOX | Assuming that updates will be made as needed with minimal interruption. |
| SpeakPipe | |
| TabSINT | |
| TeamHearing | Developer will provide complete copy of code and template files for creating a new module. |
9. Cost for software, licenses, or user accounts:
| Platform | |
| Amazon MTurk | Think it's a percentage of the user payments. It was small compared to user payments. |
| Cognition.Run | Currently free. This is the response that Javier Vidal Peña provided when I asked about it: "We would like to fund the platform by donations. A Nonprofit Public Benefit Corporation is being created in California to articulate that. We will add a 'Donate' button in the next days. We are gonna work hard to keep the platform free with this donation system. However, if donations don't cover the expenses of the platform, we cannot ensure that the organization/structure will remain the same in the long future, and a licensing system would be implemented." |
| Django | Free |
| Gorilla | There is a per-respondent fee. See https://gorilla.sc/pricing |
| hearX | There is an annual fee. See https://www.hearxgroup.com/shop/hearscreen https://www.hearxgroup.com/shop/heartest https://www.hearxgroup.com/shop/heardigits |
| ispring | https://www.ispringsolutions.com/pricing |
| Jacoti | Free |
| jspsych | Free |
| MATLAB | There are license fees that vary depending on type of institution, type of license, number of toolboxes installed ,etc. https://www.mathworks.com/store/ |
| PART/BGC Science | Free |
| Prolific | Fees are charged as a fraction of each dollar paid to participants. |
| Psychstudio | Annual pricing plans at https://www.psychstudio.com/#pricing |
| PsyToolkit | Free |
| Qualtrics | Variable annual cost depending on features requested |
| SHOEBOX | Annual fees for license, hardware, annual calibration, set-up, and ongoing support. Cost will vary depending on use case, number of units, and length of license period. Various discount may be available to researchers. https://www.shoebox.md/quote/ |
| SpeakPipe | Free |
| TabSINT | Free |
| TeamHearing | Free |
10. Describe the mechanisms for support
| Platform | Online documentation / Wiki | User forums | Direct access to developers | Other (please specify and rate) |
|---|
| Amazon MTurk | Excellent | Acceptable | Poor | |
| Cognition.Run | Not Available | Not Available | Excellent | |
| Django | Excellent | Excellent | Acceptable | |
| Gorilla | Excellent | Not Available | Excellent | Investigators can request a Gorilla programmer to build an experiment with a fee. |
| hearX | Excellent | Not Available | Acceptable | |
| ispring | Acceptable | Not Available | Acceptable | Customer service can be quite helpful, but you don't talk to actual developers. |
| Jacoti | Excellent | Not Available | Acceptable | |
| jspsych | Excellent | Excellent | Excellent | The GitHub repository for jspsych has a "discussion" feature where questions can be posted. Developers or other users are quick to respond and are also often good at deciphering what the user is saying/asking. |
| MATLAB | Acceptable | Acceptable | UNK | I don't have any experience trying to gain access to mathworks developers and cannot comment on what its like to get direct support in this regard. General customer support is available and has been fairly responsive. |
| PART/BGC Science | Acceptable | Acceptable | Excellent | The best way to learn about using this platform is to reach out to the developers and let them use their NIH-funded time to support you directly. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | Excellent | Acceptable | Excellent | They have recommendations for how to use their platform for various study designs. There are multiple grassroots fora (e.g., on reddit) that also provide useful community-sourced information. Requests for support can be submitted to Prolific directly too, but this is organized customer support (what you usually want) rather than access to developers per se. |
| Psychstudio | Excellent | Not Available | Excellent | |
| PsyToolkit | Excellent | Not Available | Excellent | |
| Qualtrics | Excellent | Excellent | Poor | |
| SHOEBOX | Excellent | Not Available | Acceptable | |
| SpeakPipe | Not Available | Not Available | Excellent | |
| TabSINT | Acceptable | Acceptable | Poor | |
| TeamHearing | Not Available | Not Available | Excellent | |
11. Rate the quality of the documentation:
| Platform | Complete | Understandable | Up-to-date |
|---|
| Amazon MTurk | Excellent | Excellent | Excellent |
| Cognition.Run | Excellent | Excellent | Excellent |
| Django | Excellent | Excellent | Excellent |
| Gorilla | Excellent | Excellent | Acceptable |
| hearX | Acceptable | Acceptable | Acceptable |
| ispring | Acceptable | Acceptable | Acceptable |
| Jacoti | Acceptable | Excellent | Excellent |
| jspsych | Acceptable | Excellent | Excellent |
| MATLAB | Acceptable | Acceptable | Acceptable |
| PART/BGC Science | Acceptable | Acceptable | Poor - the platform is changing so rapidly, that the documentation is necessarily lagging. Direct contact with the PART team is encouraged. Once the platform stops advancing so rapidly, the documentation will catch up, with videos and examples as well as user-friendly scripts based on completed projects. |
| Prolific | Acceptable | Acceptable | Acceptable |
| Psychstudio | Excellent | Acceptable | Excellent |
| PsyToolkit | Excellent | Excellent | Excellent |
| Qualtrics | Acceptable | Excellent | Acceptable |
| SHOEBOX | Excellent | Excellent | Excellent |
| SpeakPipe | Acceptable | Acceptable | Acceptable |
| TabSINT | Acceptable | Acceptable | Excellent |
| TeamHearing | | | |
|
