|
Main
Platforms
|
Data Handling and Storage
1. For online-storage options, when/how are data transferred to the server?
| Platform | Automatically in real-time | Automatically at end of run | Automatically on a schedule | Automatically when connected to internet | Manually (upload procedure) |
| Amazon MTurk | -- | -- | -- | -- | -- |
| Cognition.Run | Required | -- | -- | -- | -- |
| Django | Optional | Optional | Optional | Optional | Optional |
| Gorilla | Required | Optional | Optional | -- | -- |
| hearX | | | | Required | |
| ispring | -- | Optional | -- | -- | -- |
| Jacoti | | Required | | | Optional |
| jspsych | -- | -- | -- | -- | -- |
| MATLAB | Optional | Optional | Optional | Optional | Optional |
| PART/BGC Science | -- | Optional | -- | Optional | Optional |
| Prolific | -- | -- | -- | -- | -- |
| Psychstudio | Required | -- | -- | Required | -- |
| PsyToolkit | -- | Required | -- | -- | -- |
| Qualtrics | Optional | -- | -- | Optional | Optional |
| SHOEBOX | | | | | |
| SpeakPipe | -- | -- | -- | -- | Optional |
| TabSINT | | Optional | | | Optional |
| TeamHearing | Optional | Required | -- | Optional | Optional |
2. Please describe any user-side input required to maintain the database - e.g., periodic deletion to avoid storage limits:
| Platform | |
| Amazon MTurk | None. |
| Cognition.Run | Experimenters are responsible for deleting data when a study is finished. |
| Django | As long as there is space on the server, no extra database maintenance steps are needed. However, backups have to be implemented by the user and is not automatic. |
| Gorilla | There seems to be no limit on the amount (both number and the actual size) of experiments a free account can build and/or share with others. Data storage remains on server until it is manually deleted. |
| hearX | None stated. Differences in cost of some products, but unclear if the differences are due to database limitations. |
| ispring | |
| Jacoti | None stated. Meant for multiple repetitions of pure tone audiogram. Storage for that data is likely very small. |
| jspsych | No special requirements. jspsych does not constrain what databases you use, nor offers much support for database management. That falls to the user who is setting up the server. |
| MATLAB | File/data management falls entirely on experimenters/users/research IT staff managing the MATLAB web app servers. |
| PART/BGC Science | Data are stored on the local device and optionally synced with a HIPAA compliant AWS server with end-to-end encryption. |
| Prolific | Not applicable. |
| Psychstudio | |
| PsyToolkit | |
| Qualtrics | |
| SHOEBOX | If you cancel or do not renew your cloud subscription, you will have 60 days to download your data after which it will be permanently deleted from the system. |
| SpeakPipe | |
| TabSINT | |
| TeamHearing | None. |
3. How are user-designed experiments hosted?
| Platform | |
| Amazon MTurk | Online - native to platform |
| Cognition.Run | Online - native to platform |
| Django | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Gorilla | Online - native to platform |
| hearX | Entirely on device |
| ispring | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Jacoti | On device after download |
| jspsych | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| MATLAB | Online - native to platform |
| PART/BGC Science | Entirely on device; On device after download; Online - native to platform; or Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Prolific | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Psychstudio | Online - native to platform |
| PsyToolkit | On device after download or Online - native to platform |
| Qualtrics | Online - native to platform |
| SHOEBOX | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| SpeakPipe | On device after download or Online - native to platform |
| TabSINT | On device after download |
| TeamHearing | Online - native to platform |
4. How are stimulus files hosted?
| Platform | |
| Amazon MTurk | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Cognition.Run | Online - native to platform |
| Django | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Gorilla | Online - native to platform |
| hearX | Entirely on device |
| ispring | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Jacoti | On device after download |
| jspsych | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| MATLAB | Entirely on device Online - native to platform |
| PART/BGC Science | On device after download |
| Prolific | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| Psychstudio | Online - native to platform |
| Psytoolkit | On device after download or Online - native to platform |
| Qualtrics | Online - native to platform or Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| SHOEBOX | |
| SpeakPipe | Online - linked to outside server (e.g., Pavlovia, web server, lab PC) |
| TabSINT | On device after download |
| TeamHearing | Online - native to platform |
5. How are stimulus files formatted?
| Platform |
| Amazon MTurk | Not available |
| Cognition.Run | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files, or Compressed audio/video (e.g. MP3, AAC, JPEG MP4) |
| Django | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed archive for upload/download (e.g. ZIP archive), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4) |
| Gorilla | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4) |
| hearX | |
| ispring | Compressed audio/video (e.g. MP3, AAC, JPEG MP4) |
| Jacoti | |
| jspsych | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4) |
| MATLAB | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4) |
| PART/BGC Science | Uncompressed, generated on device, with compressed archive used for upload/download of stored stimuli: speech, HRTF files, etc. |
| Prolific | |
| Psychstudio | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files) |
| PsyToolkit | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files) |
| Qualtrics | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files) |
| SHOEBOX | |
| SpeakPipe | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files) |
| TabSINT | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4 |
| TeamHearing | Uncompressed audio/image/video (e.g. WAV, TIFF, AVI files), or Compressed archive for upload/download (e.g. ZIP archive), or Compressed audio/video (e.g. MP3, AAC, JPEG MP4) |
6. Internet Connectivity
| Platform | When is internet connectivity required during testing? | Specify any details necessary to clarify interactions between responses |
|---|
| Amazon MTurk | At login, During testing | Simple survey questions are hosted on the Amazon Turk site. But things like audio must be served from your own host. |
| Cognition.Run | At login, During testing | |
| Django | See details. | Connectivity is required for any interaction between browser and server (e.g., loading experiment to browser, saving data, progressing through study flow). |
| Gorilla | At login, During testing | Internet is required for participants at all times during testing. For investigators, internet is required for all times during experiment building, recruitment monitoring, and data download. |
| hearX | At login, During testing, After testing | Needs to be connected to the internet if clinician/researcher wishes to access data. |
| ispring | At login, during testing, after testing. | |
| Jacoti | See details. | Testing is accomplished via apps. Internet is required to download the apps to users phone or iPad and for data to be stored to the cloud. It is unclear if the data is saved in real time or after the test is complete. The only test available is a pure tone audiogram. Audiogram can be transferred manually to apple Health app. |
| jspsych | See details. | Internet connectivity is needed for the subject's browser to download the javascript and stimulus files from wherever the experiment is served. Once the JavaScript and stimuli are downloaded, no connectivity is needed for running the experiment itself. However, connectivity will be needed whenever the experiment code (programmer dependent) makes server requests to save data -- the typical use case is that this would happen at the end of the experiment. |
| MATLAB | At login,During testing |
| PART/BGC Science | Needed to download application, can be used to download alternative test batteries, and then can be used to store data after testing, but is not needed at all after app download if desired. | It depends on how one wants to get the data, but for initial download and setup, for updates, and then after testing for data to be uploaded is most typical. |
| Prolific | See details. | This is a fully online recruitment platform. |
| Psychstudio | At login, During testing | Participant data is uploaded after each interaction, thus internet connectivity is required from login to completion of study. |
| PsyToolkit | | If you want to run the experiment online, it must be connected to the internet at all times. But, you can download and run experiments from the local computer, so for the purposes of in person testing you wouldn't need to be online at all. |
| Qualtrics | At login, During testing | |
| SHOEBOX | Not specified. | Documentation states "Automatic back-up from the iPads." Internet would be needed to back-up data from iPads to a central data management system, but this is not specifically stated in the documentation. |
| SpeakPipe | At login, During testing, AFter testing | Participants can save an audio recording locally and submit via Qualtrics file upload or other data collection tool. Participants can also provide a link to the recording on the service's server, e.g., enter link in Qualtrics open-response item, which researcher can access later. |
| TabSINT | Not at all |
| TeamHearing | At login, During testing, After testing | Connection needed to access resources and to save data. |
7. Miscellaneous comments on Data Handling that would help clarify or expand on responses above:
| Platform |
| Amazon MTurk | |
| Cognition.Run | |
| Django | Data generated by front-end platforms like jspsych or psychopy can be captured, transferred securely to the server, and saved in encrypted form using Django code. The security and data handling capabilities are in my opinion the highlight of the platform. |
| Gorilla | Participant responses are registered as event-tag style in a spreadsheet. Demographics data that is queried in a survey will be stand-alone spreadsheet. So identifiable data (e.g., birth date, age, sex) does not mix with participant's task-specific response data. For any new participant completing the experiment, the investigator can request the server to generate the dataset for download. It should be noted that dataset generation is not instantaneous -- It can take some time up to a couple of hours to generate a small dataset (only a few subjects). It is a little unclear to me why data generation will take such a long time. |
| hearX | RE: audio compression - unknown. I suspect .wav files stored on the device or a compressed audio file stored on device. |
| ispring | Since this is an elearning software built in data handling options are mostly similar to those for any online education platform. This is sufficient for some simple auditory experience but not for others. There are options to get raw data as CSV or XMLS, but those require some custom programming. |
| Jacoti | Pure tone audiogram from Jacoti HearingCenter app is meant for use in the JacotiListenApp and JacotiLola app. RE: audio compression - company does not specify. I suspect either .wav files or compressed audio. Do not need internet to complete testing, only to download apps. |
| jspsych | jspsych needs to be paired with some server to provide backend components. Combining jspsych with MTurk and PSIturk seems to be a popular pairing. We wrote rudimentary server code using Django and that seems to work well with jspsych. All data handling issues will be dependent on the server, rather than on jspsych. |
| MATLAB | The MATLAB web app platform is not a service so users/experimenters have control over nearly all aspects of data handling, but must provide all the infrastructure themselves. Current experience only pertains to serving full stimulus .wav files to participants via the matlab web app server. According to this experience, participants with poor or unreliable internet connections will likely take longer to complete experimental tasks and extra care would need to be taken if inter trial intervals are important or need to remain a fixed duration. |
| PART/BGC Science | PART is designed to run stand-alone on a tablet or computer, with data saved via hard connection to a secure server. BGC Science was created so that users can download the app, register their email, and the researcher or clinician can define the tests to be pushed to the app, after which data are saved on a secure HIPA-compliant server. For more information on these approaches, please contact the developers, who are NIH-funded to provided hands-on support to researchers and clinicians. https://braingamecenter.ucr.edu/games/p-a-r-t/ |
| Prolific | |
| Psychstudio | |
| PsyToolkit | One caveat is that the creator has access to all data because of how it is stored on the server. But, he does not own the data and cannot use it for any purpose. |
| Qualtrics | |
| SHOEBOX | Meant to be an audiology tele-health pure tone hearing test option. It is unclear if data is also stored on the iPad. It is uploaded to SHOEBOX's HIPAA compliant cloud and accessible at a central database. Unclear if the wavfiles are local or if they need the cloud/internet to access. |
| SpeakPipe | |
| TabSINT | |
| TeamHearing | Data can be downloaded to Excel, Matlab, or other program for further analysis. HTML5 audio engine. |
|
