{"id":129,"date":"2023-08-21T13:57:56","date_gmt":"2023-08-21T13:57:56","guid":{"rendered":"https:\/\/metric.qcri.org\/blog\/?p=129"},"modified":"2023-08-21T14:11:46","modified_gmt":"2023-08-21T14:11:46","slug":"do-you-need-real-users-in-user-studies-user-experiments-and-user-surveys","status":"publish","type":"post","link":"https:\/\/metric.qcri.org\/blog\/2023\/08\/21\/do-you-need-real-users-in-user-studies-user-experiments-and-user-surveys\/","title":{"rendered":"Do you need real users in user studies, user experiments, and user surveys?"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full is-style-rounded\"><img loading=\"lazy\" decoding=\"async\" width=\"576\" height=\"398\" src=\"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2023\/08\/real_users.jpg?resize=576%2C398&#038;ssl=1\" alt=\"Do you need real users in user studies, user experiments, and user surveys?\" class=\"wp-image-130\" title=\"Do you need real users in user studies, user experiments, and user surveys?\" srcset=\"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2023\/08\/real_users.jpg?w=576&amp;ssl=1 576w, https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2023\/08\/real_users.jpg?resize=300%2C207&amp;ssl=1 300w\" sizes=\"auto, (max-width: 576px) 100vw, 576px\" data-recalc-dims=\"1\" \/><figcaption class=\"wp-element-caption\">Do you need real users in user studies, user experiments, and user surveys?<\/figcaption><\/figure>\n\n\n\n<p>When you design user studies, user experiments, and user surveys (a.k.a. user research), there is a need to sample from the user population \u2013 that is selecting real users.<\/p>\n\n\n\n<p>Selecting real users is often considered vital for ensuring the validity, usefulness, and applicability of user research.<\/p>\n\n\n\n<p>But what are real users?<\/p>\n\n\n\n<p>Real users are participants drawn from the specific target population.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Target Population<\/strong>: The target population is the entire group of people a researcher is interested in studying and analyzing, often related to a system or phenomenon.<\/li>\n\n\n\n<li><strong>Sample<\/strong>: Real users are selected from the target population using one or more sampling techniques and become the sample used for the study. The sample is the set of people representing the target population participating in the investigation.<\/li>\n\n\n\n<li><strong>Sampling<\/strong>: Sampling is the process of determining the participants taken from the target population for the study and can use techniques such as a sample frame (e.g., a list of individuals in the target population). The methodology used to sample a larger population depends on the analysis and may include, for example, random, stratified, or systematic sampling techniques.<\/li>\n\n\n\n<li><strong>Users<\/strong>: The participants in the user study are referred to as participants. So, the terms \u2018user\u2019 and \u2018participant\u2019 refer to actual people concerning the survey. People use the technology in the \u2018user\u2019 role, and in the \u2018participant\u2019 role, people engage in the study.<\/li>\n<\/ul>\n\n\n\n<p>However, do you really need real users for user research?<\/p>\n\n\n\n<p>While sampling real users is considered essential by many researchers for determining a study\u2019s validity and practical impact, this position has been challenged in the past by the high use of, for example, students and, increasingly is challenged with the discussions of the use of AI as &#8216;simulated&#8217; (a.k.a., fake) users, including the <a href=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S2949719123000171\" data-type=\"link\" data-id=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S2949719123000171\">use of AI for survey respondents<\/a>.<\/p>\n\n\n\n<p>Even given these issues, prior work discusses three foundational problems (FP) of not sampling real users in user research.&nbsp;<\/p>\n\n\n\n<p><strong>FP1 &#8211; Validity<\/strong>: Regarding how well findings can be generalized to other situations, user studies that do not employ real users face external validity issues. If the participants do not belong to the target population, they may lack the motivation, ability, or expertise to give valid responses. Although the employment of real users does not necessarily address the potential lack of motivation, it does ensure a sufficient degree of domain expertise for external and ecological validity (i.e., how well the results predict behavior in real life).<\/p>\n\n\n\n<p><strong>FP2 &#8211; Usefulness<\/strong>: While scientific validity is aimed at the accuracy and precision of results, there is a quintessential question underlying the employment of the research findings: Are the results useful for researchers and practitioners? Usefulness\u2014or accuracy\u2014is unlikely to be achieved when using surrogate users, as the needs and problems discovered may not match those faced by the intended users of the technology. Therefore, regardless of sampling validity, surrogate users are unlikely to provide helpful feedback compared to input drawn from real users.<\/p>\n\n\n\n<p><strong>FP3 &#8211; Applicability<\/strong>: When real users are not included in the sample, researchers may miss valuable insights for improving the technology. This is related to the issue that the sample may not represent the underlying population that will employ the technology. Since surrogate users need to gain intimate knowledge of a domain, subject matter, or problem space, they cannot provide feedback that would foreseeably lead to new features and functionalities addressing an impactful problem for the targeted population.<\/p>\n\n\n\n<p>Are real users necessary for user research? Probably.<\/p>\n\n\n\n<p>Will AI \u2018simulated\u2019 users play an increasingly important role? Probably.<\/p>\n\n\n\n<p>For more on this topic, see:<\/p>\n\n\n\n<p>Salminen, J., Jung, S.G., Kamel, A., Froneman, W., and Jansen, B. J. (2022). PeerJ Computer Science. 8:e1136 <a href=\"https:\/\/doi.org\/10.7717\/peerj-cs.1136\">https:\/\/doi.org\/10.7717\/peerj-cs.1136<\/a><\/p>\n\n\n\n<p>Jansen, B. J., Jung, S.G., and Salminen, J. (2023) <a rel=\"noreferrer noopener\" href=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S2949719123000171\" target=\"_blank\">Employing Large Language Models in Survey Research<\/a>. Natural Language Processing Journal. 4, 100020.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>When you design user studies, user experiments, and user surveys (a.k.a. user research), there is a need to sample from the user population \u2013 that is selecting real users. Selecting real users is often considered vital for ensuring the validity, usefulness, and applicability of user research. But what are real users? Real users are participants &#8230; <a title=\"Do you need real users in user studies, user experiments, and user surveys?\" class=\"read-more\" href=\"https:\/\/metric.qcri.org\/blog\/2023\/08\/21\/do-you-need-real-users-in-user-studies-user-experiments-and-user-surveys\/\" aria-label=\"More on Do you need real users in user studies, user experiments, and user surveys?\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7,5],"tags":[],"class_list":["post-129","post","type-post","status-publish","format-standard","hentry","category-analytics","category-user-study"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":168,"url":"https:\/\/metric.qcri.org\/blog\/2024\/03\/14\/some-insights-and-recommendations-about-user-studies\/","url_meta":{"origin":129,"position":0},"title":"Some insights and recommendations about user studies","date":"March 14, 2024","format":false,"excerpt":"We have published a series of research articles examining user studies, including implementation suggestions. Some insights and recommendations about user studies Some of our findings and recommendations are: Fair pay: Paying crowdworkers adequately is task dependent Real Users: Students are overused in academic studies; there are workable alternates to get\u2026","rel":"","context":"Similar post","img":{"alt_text":"Some insights and recommendations about user studies","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/03\/insights_into_user_studies_jansen.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":75,"url":"https:\/\/metric.qcri.org\/blog\/2022\/09\/30\/who-is-in-the-sample-an-analysis-of-real-and-surrogate-users-as-participants-in-user-study-research-in-the-information-technology-fields\/","url_meta":{"origin":129,"position":1},"title":"Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields","date":"September 30, 2022","format":false,"excerpt":"This research aims to determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants. Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research\u2026","rel":"","context":"In &quot;user study&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":112,"url":"https:\/\/metric.qcri.org\/blog\/2023\/06\/21\/why-are-user-feedback-and-user-testing-important\/","url_meta":{"origin":129,"position":2},"title":"Why are User feedback and User testing important?","date":"June 21, 2023","format":false,"excerpt":"User feedback and user testing play a significant role in developing and improving digital products such as websites and mobile applications as it helps businesses understand their users. Understanding people, their characteristics, capabilities, commonalities, and differences allows designers to create more effective, safer, efficient, and enjoyable systems [2]. Here are\u2026","rel":"","context":"In \"engagement\"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2023\/06\/image-3.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":52,"url":"https:\/\/metric.qcri.org\/blog\/2022\/07\/28\/confusion-prediction-from-eye-tracking-data-experiments-with-machine-learning\/","url_meta":{"origin":129,"position":3},"title":"Confusion Prediction from Eye-Tracking Data: Experiments with Machine Learning","date":"July 28, 2022","format":false,"excerpt":"Predicting user confusion can help improve information presentation on websites, mobile apps, and virtual reality interfaces. One promising information source for such prediction is eye-tracking data about gaze movements on the screen. Coupled with think-aloud records, we explore if user\u2019s confusion is correlated with primarily fixation-level features. We find that\u2026","rel":"","context":"In &quot;user study&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":54,"url":"https:\/\/metric.qcri.org\/blog\/2022\/07\/28\/is-more-better-impact-of-multiple-photos-on-perception-of-persona-profiles\/","url_meta":{"origin":129,"position":4},"title":"Is More Better?: Impact of Multiple Photos on Perception of Persona Profiles","date":"July 28, 2022","format":false,"excerpt":"In this research, we investigate if and how more photos than a single headshot can heighten the level of information provided by persona profiles. We conduct eye-tracking experiments and qualitative interviews with variations in the photos: a single headshot, a headshot and images of the persona in different contexts, and\u2026","rel":"","context":"In &quot;user study&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":282,"url":"https:\/\/metric.qcri.org\/blog\/2024\/07\/04\/actionable-data-creating-personas-from-metric-data-analytics\/","url_meta":{"origin":129,"position":5},"title":"Actionable Data: Creating Personas from METRIC Data Analytics","date":"July 4, 2024","format":false,"excerpt":"Personas represent important segments\u00a0of a target audience or user base, making them indispensable tools in marketing, user experience (UX) design, product development, and education. Personas in marketing help create content and campaigns specifically tailored to various client bases' demands, preferences, and behaviors. They aid in making assessments about the user\u2026","rel":"","context":"In &quot;analytics&quot;","img":{"alt_text":"Personas are essential tools in fields such as marketing, user experience (UX) design, product development, and education, serving to represent key segments of a target audience or user base","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/07\/Personas.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/129","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/comments?post=129"}],"version-history":[{"count":2,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/129\/revisions"}],"predecessor-version":[{"id":132,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/129\/revisions\/132"}],"wp:attachment":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/media?parent=129"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/categories?post=129"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/tags?post=129"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}