TaskFlow
Task Flow
1. What does the platform support in terms of study flow logic, instructions, debriefing, and early termination?
| Platform | Supports control of study flow logic | Provides instructions | Provides debriefing | Supports early termination |
| Amazon MTurk | YES | NO | NO | YES |
| Cognition.Run | YES | YES | YES | YES |
| Django | YES | YES | YES | YES |
| Gorilla | YES | YES | YES | YES |
| hearX | YES | |||
| ispring | YES | YES | YES | YES |
| Jacoti | YES | YES | YES | YES |
| jspsych | NO | YES | YES | YES |
| MATLAB | YES | YES | YES | YES |
| PART/BGC Science | YES | YES | YES | YES |
| Prolific | YES | YES | NO | YES |
| Psychstudio | YES | YES | YES | YES |
| PsyToolkit | YES | YES | YES | YES |
| Qualtrics | YES | YES | YES | YES |
| SHOEBOX | YES | YES | ||
| SpeakPipe | NO | NO | NO | YES |
| TabSINT | YES | YES | YES | YES |
| TeamHearing | YES | YES | YES | YES |
2. What does the platform support in terms of contingencies, pausing, stopping, restarting?
| Platform | Supports Task 2 contingent on completion of Task 1 ("do T2 after T1") | Supports Task 2 contingent on performance on Task 1 ("do T2 if performance is at criterion on T1") | Possible to pause mid-test | Possible to save and quit mid-test | Possible to restore from a save point |
| Amazon MTurk | YES | YES | NO | NO | NO |
| Cognition.Run | YES | YES | YES | NO | NO |
| Django | YES | YES | YES | YES | YES |
| Gorilla | YES | YES | YES | NO | YES |
| hearX | |||||
| ispring | NO | NO | YES | NO | YES |
| Jacoti | |||||
| jspsych | YES | YES | YES | NO | NO |
| MATLAB | YES | YES | YES | YES | YES |
| PART/BGC Science | YES | YES | YES | YES | NO |
| Prolific | NO | NO | NO | NO | NO |
| Psychstudio | YES | YES | NO | NO | |
| PsyToolkit | |||||
| Qualtrics | YES | YES | YES | YES | YES |
| SHOEBOX | |||||
| SpeakPipe | NO | NO | YES | NO | NO |
| TabSINT | YES | YES | YES | YES | YES |
| TeamHearing | YES | YES | YES | YES | YES |
Multiple tasks may be strung together to form a larger hybrid task.
3. Provide a brief description of the task flow (screening/checks, demographics, instructions, task, debrief, etc.):
| Platform | |
| Amazon MTurk | All under experimenter’s control. Anything is possible. |
| Cognition.Run | Same as jsPsych |
| Django | The platform is general enough to program arbitrary task flow structure. For instance, subjects could start on a page that asks them for a subject ID (provided by recruitments platforms like MTurk/Prolific or the experimenter), then a consent page, then a demographic survey, then psychoacoustic tasks implemented using front-end platforms like jsPsych whose outcome is used by Django to screen or route participants appropriately, and then more psychoacoustic tasks. Finally, a debrief page may be shown where they provide feedback and are redirected back to a recruitment platform (MTurk/Prolific). |
| Gorilla | Participants provide online consent (optional, can be done elsewhere such as Qualtrics), then proceed to demographics information. In either the same or separate session, participant starts task by perceptual screening/checks of headphone quality and finding comfortable suprathreshold level (finding quiet threshold may also be possible). Once passing screening, participant is given task instructions either as in text or an embedded video. Participant will continue with practice trials before continuing onto the main task. Participant will be allowed short breaks between conditions and continue at their own pace. At the end of the task, participant will be debriefed and exit the platform. If recruitment is through third-party service such as Prolific, Gorilla may route the participant back to Prolific (not yet cofirmed). Participant can exit the task at anytime by closing the browser. |
| hearX | Not sure. Would need to request a formal demo of the equipment. Website shows clear, easy to follow instructions. unsure if testing can be stopped/started easily while saving data in progress. |
| ispring | Can be custom designed based on the experimenter’s needs. |
| Jacoti | Step by step instructions on how to set up a Jacoti profile and how to take the pure tone test. |
| jspsych | jspsych can help design individual components/tasks within a study (e.g., surveys, audio experiments, instructions, consent). For simple study designs, a hybrid task may be designed with jsPsych that encompasses the invidividual tasks. However, for more complex designs, the flow through the study will need to be controlled by server-side logic (e.g., using Django). |
| MATLAB | Task flow is entirely managed by the experimenter. Matlab web apps can check for existing data from other tasks to ensure participants have completed preliminaries or are eligible to complete each task. It is up to the experiment developer to decide when and how to collect demographics, present instructions, and provide debrief. |
| PART/BGC Science | Platform supports introduction and instruction screens, tests using 4-interval 2-alternative forced choice paradigms, single-interval responding, adaptive staircases, fixed level tests, cognitive assessments, survey responses, and data transfer to server after each threshold estimate or other test. If data upload is turned on, it will try to upload whenever internet is available but will continue to function even if upload is not possible. |
| Prolific | Task/study flow is usually controlled by the external server that participants are redirected to. Prolific only provides subject recruitment, but includes multiple payment options (e.g., partial payment, bonus payment, rejecting submission, etc.) that make it compatible with somewhat complex study flows. |
| Psychstudio | All customizable. You build the experiment design you desire. |
| PsyToolkit | |
| Qualtrics | The consent form can be included as the first question. Participants who agree to consent can then complete the survey questions. Qualtrics is designed for surveys, but simple, pre-recorded audio stimuli can be presented relatively easily (e.g., for melody discrimination or speech perception tasks). If the participant stops the experiment mid-way through the study, they can pick up where they left off as long as they use the same device and browser and have not cleared their cookies. |
| SHOEBOX | Other than an online video, I have not been able to demo this product and do not know how it handles the task flow. Appears to be very user friendly and support is provided in the way of a one hour long tutorial for patients as well as support to the clinicians/researchers (provided with annual subscription). |
| SpeakPipe | |
| TabSINT | The TabSINT platform is equally adept at presenting questionnaires as it is at doing auditory testing. The task flow is therefore very flexible. |
| TeamHearing | Loudness levels, detection thresholds, primary experiment. Each with simple instructions. Recruitment and demographics are handled outside of software. |