{"id":168,"date":"2024-03-14T08:04:39","date_gmt":"2024-03-14T08:04:39","guid":{"rendered":"https:\/\/metric.qcri.org\/blog\/?p=168"},"modified":"2024-03-14T08:04:41","modified_gmt":"2024-03-14T08:04:41","slug":"some-insights-and-recommendations-about-user-studies","status":"publish","type":"post","link":"https:\/\/metric.qcri.org\/blog\/2024\/03\/14\/some-insights-and-recommendations-about-user-studies\/","title":{"rendered":"Some insights and recommendations about user studies"},"content":{"rendered":"\n<p>We have published a series of research articles examining user studies, including implementation suggestions.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"750\" height=\"577\" src=\"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/03\/insights_into_user_studies_jansen.png?resize=750%2C577&#038;ssl=1\" alt=\"Some insights and recommendations about user studies\" class=\"wp-image-169\" srcset=\"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/03\/insights_into_user_studies_jansen.png?w=750&amp;ssl=1 750w, https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/03\/insights_into_user_studies_jansen.png?resize=300%2C231&amp;ssl=1 300w\" sizes=\"auto, (max-width: 750px) 100vw, 750px\" data-recalc-dims=\"1\" \/><figcaption class=\"wp-element-caption\">                               <strong>Some insights and recommendations about user studies<\/strong><\/figcaption><\/figure>\n\n\n\n<p>Some of our findings and recommendations are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Fair pay<\/strong>: Paying crowdworkers adequately is task dependent<\/li>\n\n\n\n<li><strong>Real Users<\/strong>: Students are overused in academic studies; there are workable alternates to get real users<\/li>\n\n\n\n<li><strong>Data Validity<\/strong>: There is an assumption of data validity about people that is often wrong<\/li>\n\n\n\n<li><strong>Enriching Suirveys<\/strong>: Combining surveys with think-aloud provides rich qualitative insights<\/li>\n\n\n\n<li><strong>Participant Selection<\/strong>: Crowdworker platforms vary in quality, and all need careful piloting of recruitment and quality control<\/li>\n<\/ul>\n\n\n\n<p>Read more in these articles!<\/p>\n\n\n\n<p>Salminen, J., Sayed Kamel, A.M., Jung, S.G., Mustak, M. and Jansen, B. J. (2022) <a href=\"http:\/\/www.bernardjjansen.com\/uploads\/2\/4\/1\/8\/24188166\/fair_compensation_of_crowdsourcing_work_the_problem_of_flat_rates.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">Fair compensation of crowdsourcing work: the problem of flat rates<\/a>. Behaviour &amp; Information Technology, 27 (1), 3-35.DOI: 10.1080\/0144929X.2022.2150564<\/p>\n\n\n\n<p>Salminen, J., Jung, S.G., Kamel, A., Froneman, W., and Jansen, B. J. (2022) <em><a href=\"https:\/\/peerj.com\/articles\/cs-1136\/\" target=\"_blank\" rel=\"noreferrer noopener\">Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields<\/a><\/em>. PeerJ Computer Science. 8:e1136 <a href=\"https:\/\/doi.org\/10.7717\/peerj-cs.1136\">https:\/\/doi.org\/10.7717\/peerj-cs.1136<\/a><\/p>\n\n\n\n<p>Jansen, B. J., Salminen, J., Jung, S.G., and Almerekhi, H. (2022) <em><a href=\"https:\/\/www.sciencedirect.com\/science\/article\/pii\/S2543925122001188?via%3Dihub\" target=\"_blank\" rel=\"noreferrer noopener\">The Illusion of Data Validity: Why Numbers About People Are Likely Wron<\/a>g<\/em>. Data and Information Management. 6(4), 100020.<\/p>\n\n\n\n<p>Nielsen, L., Salminen, J., Jung, S.G., and Jansen, B. J. (2021) <a href=\"http:\/\/www.bernardjjansen.com\/uploads\/2\/4\/1\/8\/24188166\/nielsen2021_chapter_think-aloudsurveys.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">Think-Aloud Surveys &#8211; A Method for Eliciting Enhanced Insights During User Studies<\/a>, INTERACT 2021 18th IFIP TC13 International Conference on Human\u2013Computer Interaction. 30 Aug. &#8211; 3 Sept. Bari, Italy.<\/p>\n\n\n\n<p>Salminen J., Jung S., and Jansen B.J. (2021) <a href=\"http:\/\/www.bernardjjansen.com\/uploads\/2\/4\/1\/8\/24188166\/salminen2021_chapter_suggestionsforonlineuserstudie.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">Suggestions for Online User Studies<\/a>. In: Stephanidis C. et al. (eds) HCI International 2021 &#8211; Late Breaking Papers: Design and User Experience. HCII 2021. Lecture Notes in Computer Science, vol. 13094. pp 127-146. <a href=\"https:\/\/doi.org\/10.1007\/978-3-030-90238-4_11\">https:\/\/doi.org\/10.1007\/978-3-030-90238-4_11<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We have published a series of research articles examining user studies, including implementation suggestions. Some of our findings and recommendations are: Read more in these articles! Salminen, J., Sayed Kamel, A.M., Jung, S.G., Mustak, M. and Jansen, B. J. (2022) Fair compensation of crowdsourcing work: the problem of flat rates. Behaviour &amp; Information Technology, 27 &#8230; <a title=\"Some insights and recommendations about user studies\" class=\"read-more\" href=\"https:\/\/metric.qcri.org\/blog\/2024\/03\/14\/some-insights-and-recommendations-about-user-studies\/\" aria-label=\"More on Some insights and recommendations about user studies\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-168","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":34,"url":"https:\/\/metric.qcri.org\/blog\/2022\/07\/28\/suggestions-for-online-user-studies\/","url_meta":{"origin":168,"position":0},"title":"Suggestions for Online User Studies","date":"July 28, 2022","format":false,"excerpt":"During exceptional times when researchers do not have physical access to users of technology, the importance of remote user studies increases. We provide recommendations based on lessons learned from conducting online user studies utilizing four online research platforms (Appen, MTurk, Prolific, and Upwork). Our recommendations aim to help those inexperienced\u2026","rel":"","context":"In &quot;user study&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":123,"url":"https:\/\/metric.qcri.org\/blog\/2023\/07\/23\/using-eye-tracking-in-user-studies-unveiling-visual-attention-patterns\/","url_meta":{"origin":168,"position":1},"title":"Using Eye-Tracking in User Studies: Unveiling Visual Attention Patterns","date":"July 23, 2023","format":false,"excerpt":"Eye tracking is critical in user studies because it provides valuable insights into users\u2019 visual attention and behavior. It allows researchers to see where the users focus their attention while interacting with their digital interfaces, products, or content. The following are reasons why eye tracking is important in user studies:\u2026","rel":"","context":"Similar post","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2023\/07\/image.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":231,"url":"https:\/\/metric.qcri.org\/blog\/2024\/06\/25\/benefits-of-metric-in-teaching-user-study-analytics\/","url_meta":{"origin":168,"position":2},"title":"Benefits of METRIC in Teaching User Study Analytics","date":"June 25, 2024","format":false,"excerpt":"Analytics Illustration [Source: ACUA] Various industries benefit from user analytic studies, both with and without knowledge. E-Commerce, logistics and transportation, hospitality, web development, real estate, healthcare, public services, and even education. Many industries need user feedback to improve their services, so implementing user analytic studies in academia benefits students. Here\u2026","rel":"","context":"In &quot;analytics&quot;","img":{"alt_text":"Benefits of METRIC in Teaching User Study Analytics","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2024\/06\/Screen-Shot-2024-06-13-at-5.59.38-PM.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":129,"url":"https:\/\/metric.qcri.org\/blog\/2023\/08\/21\/do-you-need-real-users-in-user-studies-user-experiments-and-user-surveys\/","url_meta":{"origin":168,"position":3},"title":"Do you need real users in user studies, user experiments, and user surveys?","date":"August 21, 2023","format":false,"excerpt":"Do you need real users in user studies, user experiments, and user surveys? When you design user studies, user experiments, and user surveys (a.k.a. user research), there is a need to sample from the user population \u2013 that is selecting real users. Selecting real users is often considered vital for\u2026","rel":"","context":"In &quot;analytics&quot;","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/metric.qcri.org\/blog\/wp-content\/uploads\/2023\/08\/real_users.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":32,"url":"https:\/\/metric.qcri.org\/blog\/2022\/07\/28\/think-aloud-surveys-a-method-for-eliciting-enhanced-insights-during-user-studies\/","url_meta":{"origin":168,"position":4},"title":"Think-Aloud Surveys &#8211; A Method for Eliciting Enhanced Insights During User Studies","date":"July 28, 2022","format":false,"excerpt":"In a user experiment, we tried out a novel data collection approach consisting of combining surveys with the think aloud method. We coin the phrase \u201cthink-aloud survey method\u201d, where participants think-aloud while completing a questionnaire. We analyzed the transcripts and found that the think aloud survey provides deeper insights into\u2026","rel":"","context":"In &quot;user study&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":75,"url":"https:\/\/metric.qcri.org\/blog\/2022\/09\/30\/who-is-in-the-sample-an-analysis-of-real-and-surrogate-users-as-participants-in-user-study-research-in-the-information-technology-fields\/","url_meta":{"origin":168,"position":5},"title":"Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields","date":"September 30, 2022","format":false,"excerpt":"This research aims to determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants. Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research\u2026","rel":"","context":"In &quot;user study&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/168","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/comments?post=168"}],"version-history":[{"count":2,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/168\/revisions"}],"predecessor-version":[{"id":172,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/168\/revisions\/172"}],"wp:attachment":[{"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/media?parent=168"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/categories?post=168"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/metric.qcri.org\/blog\/wp-json\/wp\/v2\/tags?post=168"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}