PP Remote Testing Wiki | Resources / ResourcesOLD

Uncategorized

Resources /

ResourcesOLD

Papers and informational resources

Papers with broad/introductory reviews to remote testing

Grootswagers, T. (2020). A primer on running human behavioural experiments online. Behavior research methods.https://link.springer.com/article/10.3758/s13428-020-01395-3Reips, U. D. (2002). Internet-based psychological experimenting: Five dos and five don’ts. Social science computer review, 20(3), 241-249.https://doi.org/10.1177%2F089443930202000302Sauter, M., Draschkow, D., & Mack, W. (2020). Building, Hosting and Recruiting: A Brief Introduction to Running Behavioral Experiments Online. Brain Sciences, 10(4), 251. [OPEN ACCESS]https://doi.org/10.3390/brainsci10040251

Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: a tutorial review. PeerJ, 3, e1058. [OPEN ACCESS] https://doi.org/10.7717/peerj.1058

Papers, theses, proceedings, and preprints that describe and/or validate remote testing approaches

Allen, B. D., Battu, T., Ganev, S. A., Gray, L. C., Harwell, B. N., Kesser, B. W., Kessler, M. A., Lancaster, B. C., Nagel, R. L., & Smith, J. I. (2013, April). Design of a distributable stereo hearing test package. In 2013 IEEE Systems and Information Engineering Design Symposium (pp. 207-212). IEEE.https://doi.org/10.1109/SIEDS.2013.6549520Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior research methods, 52(1), 388-407.https://doi.org/10.3758/s13428-019-01237-xBarbour, D. L., Howard, R. T., Song, X. D., Metzger, N., Sukesan, K. A., DiLorenzo, J. C., Snyder, B. R. D., Chen, J. Y., Degen, E. A., Buchbinder, J. M., & Heisey, K. L. (2019). Online machine learning audiometry. Ear and hearing, 40(4), 918-926.https://doi.org/10.1097/AUD.0000000000000669Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414. [OPEN ACCESS]https://peerj.com/articles/9414/Campos, J. A. D. B., Zucoloto, M. L., Bonafé, F. S. S., Jordani, P. C., & Maroco, J. (2011). Reliability and validity of self-reported burnout in college students: A cross randomized comparison of paper-and-pencil vs. online administration. Computers in Human Behavior, 27(5), 1875-1883.https://doi.org/10.1016/j.chb.2011.04.011Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social science computer review, 19(2), 146-162.https://doi.org/10.1177%2F089443930101900202Cullington, H., Kitterick, P., DeBold, L., Weal, M., Clarke, N., Newberry, E., & Aubert, L. (2016). Personalised long-term follow-up of cochlear implant patients using remote care, compared with those on the standard care pathway: study protocol for a feasibility randomised controlled trial. BMJ open, 6(5). [OPEN ACCESS]https://bmjopen.bmj.com/content/6/5/e011342.shortDandurand, F., Shultz, T. R., & Onishi, K. H. (2008). Comparing online and lab methods in a problem-solving experiment. Behavior research methods, 40(2), 428-434.https://link.springer.com/article/10.3758%2FBRM.40.2.428de Graaff, F., Huysmans, E., Merkus, P., Theo Goverts, S., & Smits, C. (2018). Assessment of speech recognition abilities in quiet and in noise: A comparison between self-administered home testing and testing in the clinic for adult cochlear implant users. International journal of audiology, 57(11), 872-880.https://doi.org/10.1080/14992027.2018.1506168Faust, L. G. (2018). The Test-Retest Reliability of Binaural-Processing Tasks at Home Versus a Clinical Environment. AuD dissertation, James Madison University.https://commons.lib.jmu.edu/diss201019/211/Finger, H., Goeke, C., Diekamp, D., Standvoß, K., & König, P. (2017). LabVanced: a unified JavaScript framework for online studies. In International Conference on Computational Social Science (Cologne). [OPEN ACCESS]https://www.labvanced.com/publication.htmlGallun, F. J., Seitz, A., Eddins, D. A., Molis, M. R., Stavropoulos, T., Jakien, K. M., … & Souza, P. E. (2018, May). Development and validation of Portable Automated Rapid Testing (PART) measures for auditory research. In Proceedings of Meetings on Acoustics 175ASA (Vol. 33, No. 1, p. 050002). Acoustical Society of America. [OPEN ACCESS]https://doi.org/10.1121/2.0000878Ganev, S. A. (2017). Development and Deployment of a Small Stereo-Hearing Testing System. AuD dissertation, James Madison University.https://commons.lib.jmu.edu/diss201019/154/Harrison, P. M., & Müllensiefen, D. (2018). Development and validation of the computerised adaptive beat alignment test (CA-BAT). Scientific Reports, 8(1), 1-19. [OPEN ACCESS]https://doi.org/10.1038/s41598-018-30318-8Harrison, P. M., (2020). psychTestR: An R package for designing and conducting behavioural psychological experiments. Journal of Open Source Software, 5(49), 2088. [OPEN ACCESS]https://doi.org/10.21105/joss.02088Hartshorne, J. K., de Leeuw, J. R., Goodman, N. D., Jennings, M., & O’Donnell, T. J. (2019). A thousand studies for the price of one: Accelerating psychological science with Pushkin. Behavior research methods, 51(4), 1782-1803.https://doi.org/10.3758/s13428-018-1155-zHarwell, B., Battu, T., Ganev, S., Nagel, R. L., Gray, L. C., & Kesser, B. W. (2014, April). Design of a distributable stereo hearing test package. In 2014 Systems and Information Engineering Design Symposium (SIEDS) (pp. 18-22). IEEE.https://doi.org/10.1109/SIEDS.2014.6829922Hazan, A., Rivilla, J., Méndez, N., Wack, N., Paytuvi, O., Zarowski, A., Offeciers, E., & Kinsbergen, J. (2020, June 4). Test-retest analysis of aggregated audiometry testing data using Jacoti Hearing Center self-testing application. Proceedings of the VCCA2020 conference.https://computationalaudiology.com/test-retest-analysis-of-aggregated-audiometry-testing-data-using-jacoti-hearing-center-self-testing-application/Hoerger, M. (2010). Participant dropout as a function of survey length in Internet-mediated university studies: Implications for study design and voluntary participation in psychological research. Cyberpsychology, Behavior, and Social Networking, 13(6), 697-700.https://doi.org/10.1089/cyber.2009.0445Lelo de Larrea-Mancera, E. S., Stavropoulos, T., Hoover, E. C., Eddins, D. A., Gallun, F. J., & Seitz, A. R. (2020). Portable Automated Rapid Testing (PART) for auditory research: Validation in a normal hearing population. bioRxiv. [OPEN ACCESS]https://doi.org/10.1101/2020.01.08.899088Milne, A. E., Bianco, R., Poole, K. C., Zhao, S., Billig, A. J., & Chait, M. (2020). An online headphone screening test based on dichotic pitch. BioRxiv. [OPEN ACCESS]https://doi.org/10.1101/2020.07.21.214395O’Brien, E. (2020, May 14). The critical role of computing infrastructure in computational audiology. Proceedings of the VCCA2020 conference.https://computationalaudiology.com/the-critical-role-of-computing-infrastructure-in-computational-audiology/Pfordresher, P. Q., & Demorest, S. M. (2020). Construction and validation of the Seattle Singing Accuracy Protocol (SSAP): An automated online measure of singing accuracy. In F. Russo, B. Ilari, & A. Cohen (Eds). Routledge Companion to Interdisciplinary Studies in Singing: Vol. 1 Development (pp. 322-333). London: Routledge.Rebchuk, A. D., Deptuck, H. M., O’Neill, Z. R., Fawcett, D. S., Silverberg, N. D., & Field, T. S. (2019). Validation of a Novel Telehealth Administration Protocol for the NIH Toolbox-Cognition Battery. Telemedicine and e-Health, 25(3), 237-242.https://doi.org/10.1089/tmj.2018.0023Rezlescu, C., Danaila, I., Miron, A., & Amariei, C. (2020). More time for science: Using Testable to create and share behavioral experiments faster, recruit better participants, and engage students in hands-on research. Progress in Brain Research, 253, 243-262.https://doi.org/10.1016/bs.pbr.2020.06.005Sevier, J. D., Choi, S., & Hughes, M. L. (2019). Use of Direct-Connect for Remote Speech-Perception Testing in Cochlear Implants. Ear and hearing, 40(5), 1162-1173.https://doi.org/10.1097/AUD.0000000000000693Shafiro, V., Hebb, M., Walker, C., Hsiao, Y., Brown, K., Sheft, S., Li, Y., Vasil, K., Moberly, A. (in press) Development of the 705 Basic Auditory Skills Evaluation (BASE) battery for 706 online testing of cochlear implant listeners. American Journal of Audiology.Sheikh Rashid, M., Dreschler, W. A., & de Laat, J. A. (2017). Evaluation of an internet-based speech-in-noise screening test for school-age children. International journal of audiology, 56(12), 967-975.https://doi.org/10.1080/14992027.2017.1378932Slote, J., & Strand, J. F. (2016). Conducting spoken word recognition research online: Validation and a new timing method. Behavior Research Methods, 48(2), 553-566.https://link.springer.com/article/10.3758/s13428-015-0599-7Stoet, G. (2017). PsyToolkit: A novel web-based method for running online questionnaires and reaction-time experiments. Teaching of Psychology, 44(1), 24-31.https://journals.sagepub.com/doi/10.1177/0098628316677643Swanepoel, D. W., & Clark, J. L. (2019). Hearing healthcare in remote or resource-constrained environments. The Journal of Laryngology & Otology, 133(1), 11–17.https://doi.org/10.1017/S0022215118001159Swanepoel, D. W., De Sousa, K. C., Smits, C., & Moore, D. R. (2019). Mobile applications to detect hearing impairment: Opportunities and challenges. Bulletin of the World Health Organization, 97(10), 717–718.https://doi.org/10.2471/BLT.18.227728Vicente, P., & Reis, E. (2010). Using questionnaire design to fight nonresponse bias in web surveys. Social science computer review, 28(2), 251-267.https://doi.org/10.1177%2F0894439309340751Watson, C. S., Kidd, G. R., Miller, J. D., Smits, C. and Humes, L. E. (2012) Telephone screening tests for functionally impaired hearing: Current use in seven countries and development of a US version. J. Am. Acad. Audiol. 23, 757-767.https://pubmed.ncbi.nlm.nih.gov/23169193/Whitton, J. P., Hancock, K. E., Shannon, J. M., & Polley, D. B. (2016). Validation of a Self-Administered Audiometry Application: An Equivalence Study. Laryngoscope, 126(10), 2382–2388.https://doi.org/10.1002/lary.25988Williams-Sanchez, V., McArdle, R.A., Wilson, R.H., Kidd, G.R., Watson, C.S., Bourne, A.L. (2014) Validation of a screening test of auditory function using the telephone. J. Am. Acad. Aud. ;25:937–951.https://pubmed.ncbi.nlm.nih.gov/25514447/Woods, K. J., Siegel, M. H., Traer, J., & McDermott, J. H. (2017). Headphone screening to facilitate web-based auditory experiments. Attention, Perception, & Psychophysics, 79(7), 2064-2072.https://link.springer.com/article/10.3758/s13414-017-1361-2Wycisk, Y., Kopiez, R., & Wolf, A. (2019, February). Control of Headphone and Loudspeaker Characteristics in Online Experiments. 15th International Conference on Music Perception and Cognition.http://dx.doi.org/10.13140/RG.2.2.23279.48803

Zacharakis, A., Hayes, B., Saitis, C., & Pastiadis, K. (2020, September). Evidence for timbre space robustness to an uncontrolled online stimulus presentation. 2nd International Conference on Timbre.

Other articles/projects that use remote testing

Chen, J. X., Whitton, J. P., Parthasarathy, A., Hancock, K. E., & Polley, D. B. (2020). Fluctuations in Subjective Tinnitus Ratings Over Time: Implications for Clinical Research. Otology & Neurotology.https://journals.lww.com/otology-neurotology/Abstract/9000/Fluctuations_in_Subjective_Tinnitus_Ratings_Over.96051.aspxDüvel, N., Kopiez, R., Wolf, A., & Weihe, P. (2020). Confusingly Similar: Discerning between Hardware Guitar Amplifier Sounds and Simulations with the Kemper Profiling Amp. Music & Science, 3.https://doi.org/10.1177%2F2059204320901952Frühauf, J., Kopiez, R., & Platz, F. (2013). Music on the timing grid: The influence of microtiming on the perceived groove quality of a simple drum pattern performance. Musicae Scientiae, 17(2), 246-260.https://doi.org/10.1177%2F1029864913486793H2020 EVOTION Consortium (2016-2020). Developing a platform for remote data collection with hearing aids for public health.https://h2020evotion.eu. Last accessed 2 September 2020.Kopiez, R., Platz, F., & Wolf, A. (2013). The overrated power of background music in television news magazines: A replication of Brosius’ 1990 study. Musicae Scientiae, 17(3), 309-331.https://doi.org/10.1177%2F1029864913489703Kopiez, R., Wolf, A., Platz, F., & Mons, J. (2016). Replacing the orchestra? – The discernibility of sample library and live orchestra sounds. PloS one, 11(7), e0158324. [OPEN ACCESS]https://doi.org/10.1371/journal.pone.0158324Pfordresher, P. Q., & Demorest, S. M. (in press). The prevalence and correlates of accurate singing. Journal of Research in Music Education.Platz, F., & Kopiez, R. (2013). When the first impression counts: Music performers, audience and the evaluation of stage entrance behaviour. Musicae Scientiae, 17(2), 167-197.http://dx.doi.org/10.1177/1029864913486369Whiteford, K. L. Baltzell, L. S., Irsik, V. C., Irvine, A., Mesik, J., Nolan, T., Oakes, B., Schrlau, A. E., Van Hedger, S. C., Bharadwaj, H. M., Johnsrude, I. S., Kidd, G. R., Lim, S., Luebke, A. E., Maddox, R. K., Marvin, E. W., Perrachione, T. K., Shinn-Cunningham, B. G., Oxenham, A. J. (pre-registration). Association of musical training with auditory and speech neural coding and perception.https://osf.io/pxwbu/Whitton, J. P., Hancock, K. E., Shannon, J. M., & Polley, D. B. (2017). Audiomotor Perceptual Training Enhances Speech Intelligibility in Background Noise. Current Biology, 27(21), 3237-3247.e6.https://doi.org/10.1016/j.cub.2017.09.014Wolf, A., Kopiez, R., Platz, F., Lin, H. R., & Mütze, H. (2018). Tendency Towards the Average? The Aesthetic Evaluation of a Quantitatively Average Music Performance: A Successful Replication of Repp’s (1997) Study. Music Perception: An Interdisciplinary Journal, 36(1), 98-108.https://doi.org/10.1525/mp.2018.36.1.98Wolf, A., & Kopiez, R. (2018). Development and validation of the musical ear training assessment (META). Journal of Research in Music Education, 66(1), 53-70.https://doi.org/10.1177%2F0022429418754845

Zhao, S., Yum, N. W., Benjamin, L., Benhamou, E., Yoneya, M., Furukawa, S., Dick, F., Slaney, M., & Chait, M. (2019). Rapid Ocular Responses Are Modulated by Bottom-up-Driven Auditory Salience. Journal of Neuroscience, 39(39), 7703-7714. https://doi.org/10.1523/JNEUROSCI.0776-19.2019

Other articles that describe auditory environments that remote testing may occur

Benítez-Barrera, Carlos R.; Grantham, D. Wesley; Hornsby, Benjamin W.Y. The Challenge of Listening at Home, Ear and Hearing: August 3, 2020 – Volume Publish Ahead of Print – Issue – doi: 10.1097/AUD.0000000000000896

Other online informational resources

  • Tutorial on conducting speech perception experiments online using Prolific and Gorilla, contributed by Rachel M. Theodore (University of Connecticut)
  • Review of articles studying Amazon Mechanical Turk viability as a participant pool, written by Gilan Feldman (University of Hong Kong)
  • Online information about earphones
  • BeOnline2020 online conference on browser-based remote testing, with interesting talks on auditory research and research with children
  • Moving Research Online Workshop intended to help laboratories get their research online, covering the key technologies necessary as well as best practices when designing for research on the Web.

Platform links (for full descriptions see Platform Descriptions

All-in-one web-based solutions (experiment development and hosting)

Also see ready-made and commercial solutions below for prebuilt online and phone-based options.

Cognitionhttps://www.cognition.run

  • Platform for hosting JavaScript experiments (built with jsPsych) which can be created using Cognition’s online code editor or uploaded by the researcher

Gorillahttps://gorilla.sc

  • Platform for hosting JavaScript experiments which can be created using Gorilla’s online code editor

Pavloviahttps://pavlovia.org

  • Platform for hosting JavaScript experiments built with PsychoPy/PsychoJS that also supports jsPsych and lab.js

FindingFivehttps://www.findingfive.com

  • Build and host experiments using an online code editor

Labvancedhttps://www.labvanced.com

  • Build and host experiments using an online graphical editor

PsyToolkithttps://www.psytoolkit.org

  • Build and host experiments using an online code editor

Testablehttps://www.testable.org

  • Build and host experiments using an online graphical editor

Concertohttps://concertoplatform.com

  • Build and host experiments using an online graphical editor or upload custom R code

Qualtricshttps://www.qualtrics.com

  • Used primarily to administer surveys, but an experiment may be implemented by adding custom JavaScript code or using premade question templates

SoSci Surveyhttps://www.soscisurvey.de

  • Platform for administering surveys which may include sound files and custom JavaScript or PHP code

iSpringhttps://www.ispringsolutions.com

  • Educational package intended to create online assessments that may be used for audio testing (with some effort)

Pushkinhttps://languagelearninglab.gitbook.io/pushkin/

  • Pushkin provides a customizable, scalable ecosystem for massive online psychological experiments (some familiarity with JavaScript and React needed)

Inquisithttps://www.millisecond.com

  • Inquisit provides web or mobile-device software for cognitive, social, neurophysiological, and online psychological experiments.

Take-home/hardware-based solutions

Also see ready-made and commercial solutions below for prebuilt take-home options.

PART/BGC Sciencehttps://ucrbraingamecenter.github.io/PART_Utilities

  • PART is intended to be used as a stand-alone psychoacoustical test platform, capable of providing identical assessments to large numbers of participants across multiple sites.

TabSINThttps://tabsint.org

  • Develop custom hearing-related exams or general-purpose questionnaires, then deploy remotely to tablets and mobile devices at multiple sites

mBrainTrainhttps://mbraintrain.com

  • A mobile EEG device that may be paired with a smartphone or desktop computer

Code libraries/packages/toolkits (manual hosting required)

jsPsychhttps://www.jspsych.org

  • JavaScript library for front end psychology experiments in a browser

PsychoJShttps://github.com/psychopy/psychojs

  • JavaScript counterpart to PsychoPy

PsychoPyhttps://www.psychopy.org/

  • Generally used for laboratory-based experiments, but starting with version 3.0, PsychoPy Builder can automatically generate the equivalent PsychoJS code and upload it to Pavlovia

lab.jshttps://lab.js.org/

  • Graphical online experiment builder

PsychTestRhttps://pmcharrison.github.io/psychTestR

  • R package for building experiments that extends R Shiny

R Shinyhttps://shiny.rstudio.com/

  • R package for building interactive web apps

MATLAB Web App Serverhttps://www.mathworks.com/help/webappserver

  • Host experiments coded in MATLAB to be run remotely in the web browser of participants

Djangohttps://www.djangoproject.com

  • Controls web server back end logic when paired with front end libraries like jsPsych

Node.jshttps://nodejs.org

  • Allows back end web server logic to be controlled with JavaScript

Ready-made and commercial solutions (primarily for hearing screening)

hearXhttps://www.hearxgroup.com

  • Provides prebuilt solutions for hearing screening, including mobile and desktop applications and hardware

Jacotihttps://www.jacoti.com

  • Provides prebuilt solutions for hearing screening and assistive listening

SHOEBOXhttps://www.shoebox.md

  • Provides prebuilt solutions for hearing screening, including mobile and desktop applications

TeamHearinghttps://www.teamhearing.org

  • Web application for hearing healthcare that includes speech comprehension and binaural hearing exercises, among others

National Hearing Testhttps://www.nationalhearingtest.org

  • Telephone-based test used for preliminary hearing screening

Recruitment

Prolifichttps://www.prolific.co

  • Anonymously recruit paid participants for web-based research studies.

Amazon Mechanical Turkhttps://www.mturk.com

  • Crowdsourcing marketplace whereby researchers can hire workers to complete experimental tasks

Testable Mindshttps://minds.testable.org/

Children Helping Sciencehttps://childrenhelpingscience.com/

  • Platform to post studies directed at children and recruit families.

Monitoring

TeamViewerhttps://www.teamviewer.com

  • Software application for desktop sharing, remote access, and video conferencing

Zoomhttps://zoom.us

  • Software application primarily for video conferencing, but also supports desktop sharing and remote access