ResourcesOLD

Papers and informational resources

Papers with broad/introductory reviews to remote testing

Grootswagers, T. (2020). A primer on running human behavioural experiments online. Behavior research methods.https://link.springer.com/article/10.3758/s13428-020-01395-3Reips, U. D. (2002). Internet-based psychological experimenting: Five dos and five don’ts. Social science computer review, 20(3), 241-249.https://doi.org/10.1177%2F089443930202000302Sauter, M., Draschkow, D., & Mack, W. (2020). Building, Hosting and Recruiting: A Brief Introduction to Running Behavioral Experiments Online. Brain Sciences, 10(4), 251. [OPEN ACCESS]https://doi.org/10.3390/brainsci10040251

Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: a tutorial review. PeerJ, 3, e1058. [OPEN ACCESS] https://doi.org/10.7717/peerj.1058

Papers, theses, proceedings, and preprints that describe and/or validate remote testing approaches

Allen, B. D., Battu, T., Ganev, S. A., Gray, L. C., Harwell, B. N., Kesser, B. W., Kessler, M. A., Lancaster, B. C., Nagel, R. L., & Smith, J. I. (2013, April). Design of a distributable stereo hearing test package. In 2013 IEEE Systems and Information Engineering Design Symposium (pp. 207-212). IEEE.https://doi.org/10.1109/SIEDS.2013.6549520Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior research methods, 52(1), 388-407.https://doi.org/10.3758/s13428-019-01237-xBarbour, D. L., Howard, R. T., Song, X. D., Metzger, N., Sukesan, K. A., DiLorenzo, J. C., Snyder, B. R. D., Chen, J. Y., Degen, E. A., Buchbinder, J. M., & Heisey, K. L. (2019). Online machine learning audiometry. Ear and hearing, 40(4), 918-926.https://doi.org/10.1097/AUD.0000000000000669Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414. [OPEN ACCESS]https://peerj.com/articles/9414/Campos, J. A. D. B., Zucoloto, M. L., Bonafé, F. S. S., Jordani, P. C., & Maroco, J. (2011). Reliability and validity of self-reported burnout in college students: A cross randomized comparison of paper-and-pencil vs. online administration. Computers in Human Behavior, 27(5), 1875-1883.https://doi.org/10.1016/j.chb.2011.04.011Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social science computer review, 19(2), 146-162.https://doi.org/10.1177%2F089443930101900202Cullington, H., Kitterick, P., DeBold, L., Weal, M., Clarke, N., Newberry, E., & Aubert, L. (2016). Personalised long-term follow-up of cochlear implant patients using remote care, compared with those on the standard care pathway: study protocol for a feasibility randomised controlled trial. BMJ open, 6(5). [OPEN ACCESS]https://bmjopen.bmj.com/content/6/5/e011342.shortDandurand, F., Shultz, T. R., & Onishi, K. H. (2008). Comparing online and lab methods in a problem-solving experiment. Behavior research methods, 40(2), 428-434.https://link.springer.com/article/10.3758%2FBRM.40.2.428de Graaff, F., Huysmans, E., Merkus, P., Theo Goverts, S., & Smits, C. (2018). Assessment of speech recognition abilities in quiet and in noise: A comparison between self-administered home testing and testing in the clinic for adult cochlear implant users. International journal of audiology, 57(11), 872-880.https://doi.org/10.1080/14992027.2018.1506168Faust, L. G. (2018). The Test-Retest Reliability of Binaural-Processing Tasks at Home Versus a Clinical Environment. AuD dissertation, James Madison University.https://commons.lib.jmu.edu/diss201019/211/Finger, H., Goeke, C., Diekamp, D., Standvoß, K., & König, P. (2017). LabVanced: a unified JavaScript framework for online studies. In International Conference on Computational Social Science (Cologne). [OPEN ACCESS]https://www.labvanced.com/publication.htmlGallun, F. J., Seitz, A., Eddins, D. A., Molis, M. R., Stavropoulos, T., Jakien, K. M., … & Souza, P. E. (2018, May). Development and validation of Portable Automated Rapid Testing (PART) measures for auditory research. In Proceedings of Meetings on Acoustics 175ASA (Vol. 33, No. 1, p. 050002). Acoustical Society of America. [OPEN ACCESS]https://doi.org/10.1121/2.0000878Ganev, S. A. (2017). Development and Deployment of a Small Stereo-Hearing Testing System. AuD dissertation, James Madison University.https://commons.lib.jmu.edu/diss201019/154/Harrison, P. M., & Müllensiefen, D. (2018). Development and validation of the computerised adaptive beat alignment test (CA-BAT). Scientific Reports, 8(1), 1-19. [OPEN ACCESS]https://doi.org/10.1038/s41598-018-30318-8Harrison, P. M., (2020). psychTestR: An R package for designing and conducting behavioural psychological experiments. Journal of Open Source Software, 5(49), 2088. [OPEN ACCESS]https://doi.org/10.21105/joss.02088Hartshorne, J. K., de Leeuw, J. R., Goodman, N. D., Jennings, M., & O’Donnell, T. J. (2019). A thousand studies for the price of one: Accelerating psychological science with Pushkin. Behavior research methods, 51(4), 1782-1803.https://doi.org/10.3758/s13428-018-1155-zHarwell, B., Battu, T., Ganev, S., Nagel, R. L., Gray, L. C., & Kesser, B. W. (2014, April). Design of a distributable stereo hearing test package. In 2014 Systems and Information Engineering Design Symposium (SIEDS) (pp. 18-22). IEEE.https://doi.org/10.1109/SIEDS.2014.6829922Hazan, A., Rivilla, J., Méndez, N., Wack, N., Paytuvi, O., Zarowski, A., Offeciers, E., & Kinsbergen, J. (2020, June 4). Test-retest analysis of aggregated audiometry testing data using Jacoti Hearing Center self-testing application. Proceedings of the VCCA2020 conference.https://computationalaudiology.com/test-retest-analysis-of-aggregated-audiometry-testing-data-using-jacoti-hearing-center-self-testing-application/Hoerger, M. (2010). Participant dropout as a function of survey length in Internet-mediated university studies: Implications for study design and voluntary participation in psychological research. Cyberpsychology, Behavior, and Social Networking, 13(6), 697-700.https://doi.org/10.1089/cyber.2009.0445Lelo de Larrea-Mancera, E. S., Stavropoulos, T., Hoover, E. C., Eddins, D. A., Gallun, F. J., & Seitz, A. R. (2020). Portable Automated Rapid Testing (PART) for auditory research: Validation in a normal hearing population. bioRxiv. [OPEN ACCESS]https://doi.org/10.1101/2020.01.08.899088Milne, A. E., Bianco, R., Poole, K. C., Zhao, S., Billig, A. J., & Chait, M. (2020). An online headphone screening test based on dichotic pitch. BioRxiv. [OPEN ACCESS]https://doi.org/10.1101/2020.07.21.214395O’Brien, E. (2020, May 14). The critical role of computing infrastructure in computational audiology. Proceedings of the VCCA2020 conference.https://computationalaudiology.com/the-critical-role-of-computing-infrastructure-in-computational-audiology/Pfordresher, P. Q., & Demorest, S. M. (2020). Construction and validation of the Seattle Singing Accuracy Protocol (SSAP): An automated online measure of singing accuracy. In F. Russo, B. Ilari, & A. Cohen (Eds). Routledge Companion to Interdisciplinary Studies in Singing: Vol. 1 Development (pp. 322-333). London: Routledge.Rebchuk, A. D., Deptuck, H. M., O’Neill, Z. R., Fawcett, D. S., Silverberg, N. D., & Field, T. S. (2019). Validation of a Novel Telehealth Administration Protocol for the NIH Toolbox-Cognition Battery. Telemedicine and e-Health, 25(3), 237-242.https://doi.org/10.1089/tmj.2018.0023Rezlescu, C., Danaila, I., Miron, A., & Amariei, C. (2020). More time for science: Using Testable to create and share behavioral experiments faster, recruit better participants, and engage students in hands-on research. Progress in Brain Research, 253, 243-262.https://doi.org/10.1016/bs.pbr.2020.06.005Sevier, J. D., Choi, S., & Hughes, M. L. (2019). Use of Direct-Connect for Remote Speech-Perception Testing in Cochlear Implants. Ear and hearing, 40(5), 1162-1173.https://doi.org/10.1097/AUD.0000000000000693Shafiro, V., Hebb, M., Walker, C., Hsiao, Y., Brown, K., Sheft, S., Li, Y., Vasil, K., Moberly, A. (in press) Development of the 705 Basic Auditory Skills Evaluation (BASE) battery for 706 online testing of cochlear implant listeners. American Journal of Audiology.Sheikh Rashid, M., Dreschler, W. A., & de Laat, J. A. (2017). Evaluation of an internet-based speech-in-noise screening test for school-age children. International journal of audiology, 56(12), 967-975.https://doi.org/10.1080/14992027.2017.1378932Slote, J., & Strand, J. F. (2016). Conducting spoken word recognition research online: Validation and a new timing method. Behavior Research Methods, 48(2), 553-566.https://link.springer.com/article/10.3758/s13428-015-0599-7Stoet, G. (2017). PsyToolkit: A novel web-based method for running online questionnaires and reaction-time experiments. Teaching of Psychology, 44(1), 24-31.https://journals.sagepub.com/doi/10.1177/0098628316677643Swanepoel, D. W., & Clark, J. L. (2019). Hearing healthcare in remote or resource-constrained environments. The Journal of Laryngology & Otology, 133(1), 11–17.https://doi.org/10.1017/S0022215118001159Swanepoel, D. W., De Sousa, K. C., Smits, C., & Moore, D. R. (2019). Mobile applications to detect hearing impairment: Opportunities and challenges. Bulletin of the World Health Organization, 97(10), 717–718.https://doi.org/10.2471/BLT.18.227728Vicente, P., & Reis, E. (2010). Using questionnaire design to fight nonresponse bias in web surveys. Social science computer review, 28(2), 251-267.https://doi.org/10.1177%2F0894439309340751Watson, C. S., Kidd, G. R., Miller, J. D., Smits, C. and Humes, L. E. (2012) Telephone screening tests for functionally impaired hearing: Current use in seven countries and development of a US version. J. Am. Acad. Audiol. 23, 757-767.https://pubmed.ncbi.nlm.nih.gov/23169193/Whitton, J. P., Hancock, K. E., Shannon, J. M., & Polley, D. B. (2016). Validation of a Self-Administered Audiometry Application: An Equivalence Study. Laryngoscope, 126(10), 2382–2388.https://doi.org/10.1002/lary.25988Williams-Sanchez, V., McArdle, R.A., Wilson, R.H., Kidd, G.R., Watson, C.S., Bourne, A.L. (2014) Validation of a screening test of auditory function using the telephone. J. Am. Acad. Aud. ;25:937–951.https://pubmed.ncbi.nlm.nih.gov/25514447/Woods, K. J., Siegel, M. H., Traer, J., & McDermott, J. H. (2017). Headphone screening to facilitate web-based auditory experiments. Attention, Perception, & Psychophysics, 79(7), 2064-2072.https://link.springer.com/article/10.3758/s13414-017-1361-2Wycisk, Y., Kopiez, R., & Wolf, A. (2019, February). Control of Headphone and Loudspeaker Characteristics in Online Experiments. 15th International Conference on Music Perception and Cognition.http://dx.doi.org/10.13140/RG.2.2.23279.48803

Zacharakis, A., Hayes, B., Saitis, C., & Pastiadis, K. (2020, September). Evidence for timbre space robustness to an uncontrolled online stimulus presentation. 2nd International Conference on Timbre.

Other online informational resources

Tutorial on conducting speech perception experiments online using Prolific and Gorilla, contributed by Rachel M. Theodore (University of Connecticut)
Review of articles studying Amazon Mechanical Turk viability as a participant pool, written by Gilan Feldman (University of Hong Kong)
Online information about earphones
BeOnline2020 online conference on browser-based remote testing, with interesting talks on auditory research and research with children
Moving Research Online Workshop intended to help laboratories get their research online, covering the key technologies necessary as well as best practices when designing for research on the Web.

Platform links (for full descriptions see Platform Descriptions

All-in-one web-based solutions (experiment development and hosting)

Also see ready-made and commercial solutions below for prebuilt online and phone-based options.

Cognitionhttps://www.cognition.run

Platform for hosting JavaScript experiments (built with jsPsych) which can be created using Cognition’s online code editor or uploaded by the researcher

Gorillahttps://gorilla.sc

Platform for hosting JavaScript experiments which can be created using Gorilla’s online code editor

Pavloviahttps://pavlovia.org

Platform for hosting JavaScript experiments built with PsychoPy/PsychoJS that also supports jsPsych and lab.js

FindingFivehttps://www.findingfive.com

Build and host experiments using an online code editor

Labvancedhttps://www.labvanced.com

Build and host experiments using an online graphical editor

PsyToolkithttps://www.psytoolkit.org

Build and host experiments using an online code editor

Testablehttps://www.testable.org

Build and host experiments using an online graphical editor

Concertohttps://concertoplatform.com

Build and host experiments using an online graphical editor or upload custom R code

Qualtricshttps://www.qualtrics.com

Used primarily to administer surveys, but an experiment may be implemented by adding custom JavaScript code or using premade question templates

SoSci Surveyhttps://www.soscisurvey.de

Platform for administering surveys which may include sound files and custom JavaScript or PHP code

iSpringhttps://www.ispringsolutions.com

Educational package intended to create online assessments that may be used for audio testing (with some effort)

Pushkinhttps://languagelearninglab.gitbook.io/pushkin/

Pushkin provides a customizable, scalable ecosystem for massive online psychological experiments (some familiarity with JavaScript and React needed)

Inquisithttps://www.millisecond.com

Inquisit provides web or mobile-device software for cognitive, social, neurophysiological, and online psychological experiments.

Take-home/hardware-based solutions

Also see ready-made and commercial solutions below for prebuilt take-home options.

PART/BGC Sciencehttps://ucrbraingamecenter.github.io/PART_Utilities

PART is intended to be used as a stand-alone psychoacoustical test platform, capable of providing identical assessments to large numbers of participants across multiple sites.

TabSINThttps://tabsint.org

Develop custom hearing-related exams or general-purpose questionnaires, then deploy remotely to tablets and mobile devices at multiple sites

mBrainTrainhttps://mbraintrain.com

A mobile EEG device that may be paired with a smartphone or desktop computer

Code libraries/packages/toolkits (manual hosting required)

jsPsychhttps://www.jspsych.org

JavaScript library for front end psychology experiments in a browser

PsychoJShttps://github.com/psychopy/psychojs

JavaScript counterpart to PsychoPy

PsychoPyhttps://www.psychopy.org/

Generally used for laboratory-based experiments, but starting with version 3.0, PsychoPy Builder can automatically generate the equivalent PsychoJS code and upload it to Pavlovia

lab.jshttps://lab.js.org/

Graphical online experiment builder

PsychTestRhttps://pmcharrison.github.io/psychTestR

R package for building experiments that extends R Shiny

R Shinyhttps://shiny.rstudio.com/

R package for building interactive web apps

MATLAB Web App Serverhttps://www.mathworks.com/help/webappserver

Host experiments coded in MATLAB to be run remotely in the web browser of participants

Djangohttps://www.djangoproject.com

Controls web server back end logic when paired with front end libraries like jsPsych

Node.jshttps://nodejs.org

Allows back end web server logic to be controlled with JavaScript

Ready-made and commercial solutions (primarily for hearing screening)

hearXhttps://www.hearxgroup.com

Provides prebuilt solutions for hearing screening, including mobile and desktop applications and hardware

Jacotihttps://www.jacoti.com

Provides prebuilt solutions for hearing screening and assistive listening

SHOEBOXhttps://www.shoebox.md

Provides prebuilt solutions for hearing screening, including mobile and desktop applications

TeamHearinghttps://www.teamhearing.org

Web application for hearing healthcare that includes speech comprehension and binaural hearing exercises, among others

National Hearing Testhttps://www.nationalhearingtest.org

Telephone-based test used for preliminary hearing screening

Recruitment

Prolifichttps://www.prolific.co

Anonymously recruit paid participants for web-based research studies.

Amazon Mechanical Turkhttps://www.mturk.com

Crowdsourcing marketplace whereby researchers can hire workers to complete experimental tasks

Testable Mindshttps://minds.testable.org/

Provides recruitment tools for studies hosted using https://www.testable.org

Children Helping Sciencehttps://childrenhelpingscience.com/

Platform to post studies directed at children and recruit families.

Monitoring

TeamViewerhttps://www.teamviewer.com

Software application for desktop sharing, remote access, and video conferencing

Zoomhttps://zoom.us

Software application primarily for video conferencing, but also supports desktop sharing and remote access

PP Remote Testing Wiki | Resources / ResourcesOLD