Platforms /

DataHandlingAndStorage

Data Handling and Storage

1. For online-storage options, when/how are data transferred to the server?

Platform Automatically in real-time Automatically at end of run Automatically on a schedule Automatically when connected to internet Manually (upload procedure)
Amazon MTurk
Cognition.Run Required
Django Optional Optional Optional Optional Optional
Gorilla Required Optional Optional
hearX       Required  
ispring Optional
Jacoti   Required     Optional
jspsych
MATLAB Optional Optional Optional Optional Optional
PART/BGC Science Optional Optional Optional
Prolific
Psychstudio Required Required
PsyToolkit Required
Qualtrics Optional Optional Optional
SHOEBOX          
SpeakPipe Optional
TabSINT   Optional     Optional
TeamHearing Optional Required Optional Optional

2. Please describe any user-side input required to maintain the database – e.g., periodic deletion to avoid storage limits:

Platform  
Amazon MTurk None.
Cognition.Run Experimenters are responsible for deleting data when a study is finished.
Django As long as there is space on the server, no extra database maintenance steps are needed. However, backups have to be implemented by the user and is not automatic.
Gorilla There seems to be no limit on the amount (both number and the actual size) of experiments a free account can build and/or share with others. Data storage remains on server until it is manually deleted.
hearX None stated. Differences in cost of some products, but unclear if the differences are due to database limitations.
ispring  
Jacoti None stated. Meant for multiple repetitions of pure tone audiogram. Storage for that data is likely very small.
jspsych No special requirements. jspsych does not constrain what databases you use, nor offers much support for database management. That falls to the user who is setting up the server.
MATLAB File/data management falls entirely on experimenters/users/research IT staff managing the MATLAB web app servers.
PART/BGC Science Data are stored on the local device and optionally synced with a HIPAA compliant AWS server with end-to-end encryption.
Prolific Not applicable.
Psychstudio  
PsyToolkit  
Qualtrics  
SHOEBOX If you cancel or do not renew your cloud subscription, you will have 60 days to download your data after which it will be permanently deleted from the system.
SpeakPipe  
TabSINT  
TeamHearing None.

3. How are user-designed experiments hosted?

Platform  
Amazon MTurk Online – native to platform
Cognition.Run Online – native to platform
Django Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Gorilla Online – native to platform
hearX Entirely on device
ispring Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Jacoti On device after download
jspsych Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
MATLAB Online – native to platform
PART/BGC Science Entirely on device; On device after download; Online – native to platform; or Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Prolific Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Psychstudio Online – native to platform
PsyToolkit On device after download or Online – native to platform
Qualtrics Online – native to platform
SHOEBOX Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
SpeakPipe On device after download or Online – native to platform
TabSINT On device after download
TeamHearing Online – native to platform

4. How are stimulus files hosted?

Platform  
Amazon MTurk Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Cognition.Run Online – native to platform
Django Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Gorilla Online – native to platform
hearX Entirely on device
ispring Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Jacoti On device after download
jspsych Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
MATLAB Entirely on device Online – native to platform
PART/BGC Science On device after download
Prolific Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
Psychstudio Online – native to platform
Psytoolkit On device after download or Online – native to platform
Qualtrics Online – native to platform or Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
SHOEBOX  
SpeakPipe Online – linked to outside server (e.g., Pavlovia, web server, lab PC)
TabSINT On device after download
TeamHearing Online – native to platform

5. How are stimulus files formatted?

Platform
Amazon MTurk Not available
Cognition.Run Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files, or Compressed audio/video (e.g. MP3, AAC, JPEG MP4)
Django Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed archive for upload/download (e.g. ZIP archive), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4)
Gorilla Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4)
hearX  
ispring Compressed audio/video (e.g. MP3, AAC, JPEG MP4)
Jacoti  
jspsych Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4)
MATLAB Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4)
PART/BGC Science Uncompressed, generated on device, with compressed archive used for upload/download of stored stimuli: speech, HRTF files, etc.
Prolific  
Psychstudio Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files)
PsyToolkit Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files)
Qualtrics Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files)
SHOEBOX  
SpeakPipe Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files)
TabSINT Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4
TeamHearing Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed archive for upload/download (e.g. ZIP archive), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4)

6. Internet Connectivity

Platform When is internet connectivity required during testing? Specify any details necessary to clarify interactions between responses
Amazon MTurk At login, During testing Simple survey questions are hosted on the Amazon Turk site. But things like audio must be served from your own host.
Cognition.Run At login, During testing  
Django See details. Connectivity is required for any interaction between browser and server (e.g., loading experiment to browser, saving data, progressing through study flow).
Gorilla At login, During testing Internet is required for participants at all times during testing. For investigators, internet is required for all times during experiment building, recruitment monitoring, and data download.
hearX At login, During testing, After testing Needs to be connected to the internet if clinician/researcher wishes to access data.
ispring At login, during testing, after testing.  
Jacoti See details. Testing is accomplished via apps. Internet is required to download the apps to users phone or iPad and for data to be stored to the cloud. It is unclear if the data is saved in real time or after the test is complete. The only test available is a pure tone audiogram. Audiogram can be transferred manually to apple Health app.
jspsych See details. Internet connectivity is needed for the subject’s browser to download the javascript and stimulus files from wherever the experiment is served. Once the JavaScript and stimuli are downloaded, no connectivity is needed for running the experiment itself. However, connectivity will be needed whenever the experiment code (programmer dependent) makes server requests to save data — the typical use case is that this would happen at the end of the experiment.
MATLAB At login,During testing
PART/BGC Science Needed to download application, can be used to download alternative test batteries, and then can be used to store data after testing, but is not needed at all after app download if desired. It depends on how one wants to get the data, but for initial download and setup, for updates, and then after testing for data to be uploaded is most typical.
Prolific See details. This is a fully online recruitment platform.
Psychstudio At login, During testing Participant data is uploaded after each interaction, thus internet connectivity is required from login to completion of study.
PsyToolkit   If you want to run the experiment online, it must be connected to the internet at all times. But, you can download and run experiments from the local computer, so for the purposes of in person testing you wouldn’t need to be online at all.
Qualtrics At login, During testing  
SHOEBOX Not specified. Documentation states "Automatic back-up from the iPads." Internet would be needed to back-up data from iPads to a central data management system, but this is not specifically stated in the documentation.
SpeakPipe At login, During testing, AFter testing Participants can save an audio recording locally and submit via Qualtrics file upload or other data collection tool. Participants can also provide a link to the recording on the service’s server, e.g., enter link in Qualtrics open-response item, which researcher can access later.
TabSINT Not at all
TeamHearing At login, During testing, After testing Connection needed to access resources and to save data.

7. Miscellaneous comments on Data Handling that would help clarify or expand on responses above:

Platform
Amazon MTurk  
Cognition.Run  
Django Data generated by front-end platforms like jspsych or psychopy can be captured, transferred securely to the server, and saved in encrypted form using Django code. The security and data handling capabilities are in my opinion the highlight of the platform.
Gorilla Participant responses are registered as event-tag style in a spreadsheet. Demographics data that is queried in a survey will be stand-alone spreadsheet. So identifiable data (e.g., birth date, age, sex) does not mix with participant’s task-specific response data. For any new participant completing the experiment, the investigator can request the server to generate the dataset for download. It should be noted that dataset generation is not instantaneous — It can take some time up to a couple of hours to generate a small dataset (only a few subjects). It is a little unclear to me why data generation will take such a long time.
hearX RE: audio compression – unknown. I suspect .wav files stored on the device or a compressed audio file stored on device.
ispring Since this is an elearning software built in data handling options are mostly similar to those for any online education platform. This is sufficient for some simple auditory experience but not for others. There are options to get raw data as CSV or XMLS, but those require some custom programming.
Jacoti Pure tone audiogram from Jacoti HearingCenter app is meant for use in the JacotiListenApp and JacotiLola app. RE: audio compression – company does not specify. I suspect either .wav files or compressed audio. Do not need internet to complete testing, only to download apps.
jspsych jspsych needs to be paired with some server to provide backend components. Combining jspsych with MTurk and PSIturk seems to be a popular pairing. We wrote rudimentary server code using Django and that seems to work well with jspsych. All data handling issues will be dependent on the server, rather than on jspsych.
MATLAB The MATLAB web app platform is not a service so users/experimenters have control over nearly all aspects of data handling, but must provide all the infrastructure themselves. Current experience only pertains to serving full stimulus .wav files to participants via the matlab web app server. According to this experience, participants with poor or unreliable internet connections will likely take longer to complete experimental tasks and extra care would need to be taken if inter trial intervals are important or need to remain a fixed duration.
PART/BGC Science PART is designed to run stand-alone on a tablet or computer, with data saved via hard connection to a secure server. BGC Science was created so that users can download the app, register their email, and the researcher or clinician can define the tests to be pushed to the app, after which data are saved on a secure HIPA-compliant server. For more information on these approaches, please contact the developers, who are NIH-funded to provided hands-on support to researchers and clinicians. https://braingamecenter.ucr.edu/games/p-a-r-t/
Prolific  
Psychstudio  
PsyToolkit One caveat is that the creator has access to all data because of how it is stored on the server. But, he does not own the data and cannot use it for any purpose.
Qualtrics  
SHOEBOX Meant to be an audiology tele-health pure tone hearing test option. It is unclear if data is also stored on the iPad. It is uploaded to SHOEBOX’s HIPAA compliant cloud and accessible at a central database. Unclear if the wavfiles are local or if they need the cloud/internet to access.
SpeakPipe  
TabSINT  
TeamHearing Data can be downloaded to Excel, Matlab, or other program for further analysis. HTML5 audio engine.