{"id":75,"date":"2022-09-30T08:02:12","date_gmt":"2022-09-30T08:02:12","guid":{"rendered":"https:\/\/metric.qcri.org\/blog\/?p=75"},"modified":"2022-10-21T08:34:51","modified_gmt":"2022-10-21T08:34:51","slug":"who-is-in-the-sample-an-analysis-of-real-and-surrogate-users-as-participants-in-user-study-research-in-the-information-technology-fields","status":"publish","type":"post","link":"https:\/\/metric.qcri.org\/blog\/2022\/09\/30\/who-is-in-the-sample-an-analysis-of-real-and-surrogate-users-as-participants-in-user-study-research-in-the-information-technology-fields\/","title":{"rendered":"Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields"},"content":{"rendered":"\n

\"Who<\/p>\n\n\n\n

This research aims to determine how often user studies reported in peer-reviewed information technology literature sample real users<\/em> or surrogate users<\/em> as participants.<\/p>\n\n\n\n

Constructing a sample of real users<\/em> as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users<\/em> or surrogate users<\/em> is unknown. Therefore, it is uncertain whether or not the use of surrogate users<\/em> in place of real users <\/em>is a widespread problem within user study practice.<\/p>\n\n\n\n

We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users<\/em> (i.e., from the study population) or surrogate users<\/em> (i.e., other than real users<\/em>).<\/p>\n\n\n\n

Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users<\/em>. However, of the targeted studies, only around half (58.4%) sampled real users<\/em>. Of the targeted studies sampling surrogate users<\/em>, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were.<\/p>\n\n\n\n

Key findings are as follows: (a) the state of sampling real users<\/em> in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers.<\/p>\n\n\n\n

The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.<\/p>\n\n\n\n

Salminen, J., Jung, S.G., Kamel, A., Froneman, W., and\u00a0Jansen, B. J.\u00a0(2022)\u00a0Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields<\/a><\/em>.\u00a0PeerJ Computer Science.\u00a08:e1136\u00a0https:\/\/doi.org\/10.7717\/peerj-cs.1136<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

This research aims to determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants. Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-75","post","type-post","status-publish","format-standard","hentry","category-user-study"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":129,"url":"https:\/\/metric.qcri.org\/blog\/2023\/08\/21\/do-you-need-real-users-in-user-studies-user-experiments-and-user-surveys\/","url_meta":{"origin":75,"position":0},"title":"Do you need real users in user studies, user experiments, and user surveys?","date":"August 21, 2023","format":false,"excerpt":"Do you need real users in user studies, user experiments, and user surveys? When you design user studies, user experiments, and user surveys (a.k.a. user research), there is a need to sample from the user population \u2013 that is selecting real users. Selecting real users is often considered vital for\u2026","rel":"","context":"In "analytics"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2023\/08\/real_users.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":186,"url":"https:\/\/metric.qcri.org\/blog\/2024\/06\/09\/choosing-your-study-participants-the-importance-of-segmentation\/","url_meta":{"origin":75,"position":1},"title":"Choosing Your Study Participants: The Importance of Segmentation","date":"June 9, 2024","format":false,"excerpt":"Customer\/audience segmentation is one of the most important parts of marketing user studies, but what is the best way to go about this process? A 2022 research study found that researchers frequently do not define their study participants in adequate detail, which is a disconcerting finding. Best practices will advise\u2026","rel":"","context":"In "user study"","img":{"alt_text":"Market Segmentation","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/06\/Segmentation.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":168,"url":"https:\/\/metric.qcri.org\/blog\/2024\/03\/14\/some-insights-and-recommendations-about-user-studies\/","url_meta":{"origin":75,"position":2},"title":"Some insights and recommendations about user studies","date":"March 14, 2024","format":false,"excerpt":"We have published a series of research articles examining user studies, including implementation suggestions. Some insights and recommendations about user studies Some of our findings and recommendations are: Fair pay: Paying crowdworkers adequately is task dependent Real Users: Students are overused in academic studies; there are workable alternates to get\u2026","rel":"","context":"Similar post","img":{"alt_text":"Some insights and recommendations about user studies","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/03\/insights_into_user_studies_jansen.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":18,"url":"https:\/\/metric.qcri.org\/blog\/2022\/07\/28\/the-ability-of-personas-an-empirical-evaluation-of-altering-incorrect-preconceptions-about-users\/","url_meta":{"origin":75,"position":3},"title":"The Ability of Personas: An Empirical Evaluation of Altering Incorrect Preconceptions About Users","date":"July 28, 2022","format":false,"excerpt":"False preconceptions about users can result in poor design, product development, and marketing decisions, so rectifying these preconceptions is essential for organizations. This research quantitatively evaluates the ability of data-driven personas to alter decision makers\u2019 preconceptions about their online social media users. We conduct a within-participant experiment of 31 professionals\u2026","rel":"","context":"In "user study"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":62,"url":"https:\/\/metric.qcri.org\/blog\/2022\/07\/28\/fixation-and-confusion-investigating-eye-tracking-participants-exposure-to-information-in-personas\/","url_meta":{"origin":75,"position":4},"title":"Fixation and Confusion \u2013 Investigating Eye-tracking Participants\u2019 Exposure to Information in Personas","date":"July 28, 2022","format":false,"excerpt":"To more effectively convey relevant information to end users of persona profiles, we conducted a user study consisting of 29 participants engaging with three persona layout treatments. We were interested in confusion engendered by the treatments on the participants, and conducted a within-subjects study in the actual work environment, using\u2026","rel":"","context":"In "user study"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":34,"url":"https:\/\/metric.qcri.org\/blog\/2022\/07\/28\/suggestions-for-online-user-studies\/","url_meta":{"origin":75,"position":5},"title":"Suggestions for Online User Studies","date":"July 28, 2022","format":false,"excerpt":"During exceptional times when researchers do not have physical access to users of technology, the importance of remote user studies increases. We provide recommendations based on lessons learned from conducting online user studies utilizing four online research platforms (Appen, MTurk, Prolific, and Upwork). Our recommendations aim to help those inexperienced\u2026","rel":"","context":"In "user study"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/75","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/comments?post=75"}],"version-history":[{"count":4,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/75\/revisions"}],"predecessor-version":[{"id":81,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/75\/revisions\/81"}],"wp:attachment":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/media?parent=75"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/categories?post=75"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/tags?post=75"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}