• Education
    • Higher Education
    • Scholarships & Grants
    • Online Learning
    • School Reforms
    • Research & Innovation
  • Lifestyle
    • Travel
    • Food & Drink
    • Fashion & Beauty
    • Home & Living
    • Relationships & Family
  • Technology & Startups
    • Software & Apps
    • Startup Success Stories
    • Startups & Innovations
    • Tech Regulations
    • Venture Capital
    • Artificial Intelligence
    • Cybersecurity
    • Emerging Technologies
    • Gadgets & Devices
    • Industry Analysis
  • About us
  • Contact
  • Advertise with Us
  • Privacy & Policy
Today Headline
  • Home
  • World News
    • Us & Canada
    • Europe
    • Asia
    • Africa
    • Middle East
  • Politics
    • Elections
    • Political Parties
    • Government Policies
    • International Relations
    • Legislative News
  • Business & Finance
    • Market Trends
    • Stock Market
    • Entrepreneurship
    • Corporate News
    • Economic Policies
  • Science & Environment
    • Space Exploration
    • Climate Change
    • Wildlife & Conservation
    • Environmental Policies
    • Medical Research
  • Health
    • Public Health
    • Mental Health
    • Medical Breakthroughs
    • Fitness & Nutrition
    • Pandemic Updates
  • Sports
    • Football
    • Basketball
    • Tennis
    • Olympics
    • Motorsport
  • Entertainment
    • Movies
    • Music
    • TV & Streaming
    • Celebrity News
    • Awards & Festivals
  • Crime & Justice
    • Court Cases
    • Cybercrime
    • Policing
    • Criminal Investigations
    • Legal Reforms
No Result
View All Result
  • Home
  • World News
    • Us & Canada
    • Europe
    • Asia
    • Africa
    • Middle East
  • Politics
    • Elections
    • Political Parties
    • Government Policies
    • International Relations
    • Legislative News
  • Business & Finance
    • Market Trends
    • Stock Market
    • Entrepreneurship
    • Corporate News
    • Economic Policies
  • Science & Environment
    • Space Exploration
    • Climate Change
    • Wildlife & Conservation
    • Environmental Policies
    • Medical Research
  • Health
    • Public Health
    • Mental Health
    • Medical Breakthroughs
    • Fitness & Nutrition
    • Pandemic Updates
  • Sports
    • Football
    • Basketball
    • Tennis
    • Olympics
    • Motorsport
  • Entertainment
    • Movies
    • Music
    • TV & Streaming
    • Celebrity News
    • Awards & Festivals
  • Crime & Justice
    • Court Cases
    • Cybercrime
    • Policing
    • Criminal Investigations
    • Legal Reforms
No Result
View All Result
Today Headline
No Result
View All Result
Home Science & Environment Medical Research

Research highlights unreliable responses from most Amazon MTurk users, except for ‘master’ workers

July 16, 2025
in Medical Research
Reading Time: 3 mins read
A A
0
online
5
SHARES
11
VIEWS
Share on FacebookShare on Twitter


online
Credit: CC0 Public Domain

A new study led by Dr. Vadim Axelrod, of the Gonda (Goldschmied) Multidisciplinary Brain Research Center at Bar-Ilan University, has revealed serious concerns about the quality of data collected on Amazon Mechanical Turk’s (MTurk)—a platform widely used for behavioral and psychological research.

MTurk, an online crowdsourcing marketplace where individuals complete small tasks for payment, has served as a key resource for researchers for over 15 years. Despite previous concerns about participant quality, the platform remains popular within the academic community. Dr. Axelrod’s team set out to rigorously assess the current quality of data produced by MTurk participants.

The study, involving over 1,300 participants across main and replication experiments, employed a straightforward but powerful method: repeating identical questionnaire items to measure response consistency. “If a participant is reliable, their answers to repeated questions should be consistent,” added Dr. Axelrod. In addition, the study included different types of “attentional catch” questions that should be easily answered by any attentive respondent.

The findings, just published in Royal Society Open Science, were stark: the majority of participants from MTurk’s general worker pool failed the attention checks and demonstrated highly inconsistent responses, even when the sample was limited to users with a 95% or higher approval rating.

“It’s hard to trust the data of someone who claims a runner isn’t tired after completing a marathon in extremely hot weather or that a cancer diagnosis would make someone glad,” Dr. Axelrod noted.

“The participants did not lack the knowledge to answer such attentional catch questions—they just weren’t paying sufficient attention. The implication is that their responses to the main questionnaire may be equally random.”

By contrast, Amazon’s elite “Master” workers—selected by Amazon based on high performance across previous tasks—consistently produced high-quality data. The authors recommend using Master workers for future research, taking into consideration that these participants are much more experienced and far fewer in number.

“Reliable data is the foundation of any empirical science,” said Dr. Axelrod. “Researchers need to be fully informed about the reliability of their participant pool. Our findings suggest that caution is warranted when using MTurk’s general pool for behavioral research.”

More information:
Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024, Royal Society Open Science (2025). DOI: 10.1098/rsos.250361. royalsocietypublishing.org/doi/10.1098/rsos.250361

Provided by
Bar-Ilan University


Citation:
Research highlights unreliable responses from most Amazon MTurk users, except for ‘master’ workers (2025, July 15)
retrieved 15 July 2025
from https://medicalxpress.com/news/2025-07-highlights-unreliable-responses-amazon-mturk.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.




online
Credit: CC0 Public Domain

A new study led by Dr. Vadim Axelrod, of the Gonda (Goldschmied) Multidisciplinary Brain Research Center at Bar-Ilan University, has revealed serious concerns about the quality of data collected on Amazon Mechanical Turk’s (MTurk)—a platform widely used for behavioral and psychological research.

MTurk, an online crowdsourcing marketplace where individuals complete small tasks for payment, has served as a key resource for researchers for over 15 years. Despite previous concerns about participant quality, the platform remains popular within the academic community. Dr. Axelrod’s team set out to rigorously assess the current quality of data produced by MTurk participants.

The study, involving over 1,300 participants across main and replication experiments, employed a straightforward but powerful method: repeating identical questionnaire items to measure response consistency. “If a participant is reliable, their answers to repeated questions should be consistent,” added Dr. Axelrod. In addition, the study included different types of “attentional catch” questions that should be easily answered by any attentive respondent.

The findings, just published in Royal Society Open Science, were stark: the majority of participants from MTurk’s general worker pool failed the attention checks and demonstrated highly inconsistent responses, even when the sample was limited to users with a 95% or higher approval rating.

“It’s hard to trust the data of someone who claims a runner isn’t tired after completing a marathon in extremely hot weather or that a cancer diagnosis would make someone glad,” Dr. Axelrod noted.

“The participants did not lack the knowledge to answer such attentional catch questions—they just weren’t paying sufficient attention. The implication is that their responses to the main questionnaire may be equally random.”

By contrast, Amazon’s elite “Master” workers—selected by Amazon based on high performance across previous tasks—consistently produced high-quality data. The authors recommend using Master workers for future research, taking into consideration that these participants are much more experienced and far fewer in number.

“Reliable data is the foundation of any empirical science,” said Dr. Axelrod. “Researchers need to be fully informed about the reliability of their participant pool. Our findings suggest that caution is warranted when using MTurk’s general pool for behavioral research.”

More information:
Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024, Royal Society Open Science (2025). DOI: 10.1098/rsos.250361. royalsocietypublishing.org/doi/10.1098/rsos.250361

Provided by
Bar-Ilan University


Citation:
Research highlights unreliable responses from most Amazon MTurk users, except for ‘master’ workers (2025, July 15)
retrieved 15 July 2025
from https://medicalxpress.com/news/2025-07-highlights-unreliable-responses-amazon-mturk.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.



Tags: Health ResearchHealth Research NewsHealth ScienceMedicine ResearchMedicine Research NewsMedicine Science
Previous Post

L.A. County pays $14 million to man wrongfully convicted of murder

Next Post

UK Conservative MP refers himself to standards commissioner over GHGSat contract

Related Posts

Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

July 16, 2025
1
TV remote

Eight everyday bacterial hotspots, and how to clean them

July 16, 2025
10
Next Post

UK Conservative MP refers himself to standards commissioner over GHGSat contract

  • Trending
  • Comments
  • Latest
Family calls for change after B.C. nurse dies by suicide after attacks on the job

Family calls for change after B.C. nurse dies by suicide after attacks on the job

April 2, 2025
Pioneering 3D printing project shares successes

Product reduces TPH levels to non-hazardous status

November 27, 2024

Police ID man who died after Corso Italia fight

December 23, 2024

Hospital Mergers Fail to Deliver Better Care or Lower Costs, Study Finds todayheadline

December 31, 2024
Harris tells supporters 'never give up' and urges peaceful transfer of power

Harris tells supporters ‘never give up’ and urges peaceful transfer of power

0
Des Moines Man Accused Of Shooting Ex-Girlfriend's Mother

Des Moines Man Accused Of Shooting Ex-Girlfriend’s Mother

0

Trump ‘looks forward’ to White House meeting with Biden

0
Catholic voters were critical to Donald Trump’s blowout victory: ‘Harris snubbed us’

Catholic voters were critical to Donald Trump’s blowout victory: ‘Harris snubbed us’

0

NASA to Preview Advanced US-India Radar Mission Ahead of Launch

July 16, 2025
Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

July 16, 2025
Rev. John MacArthur, influential evangelical pastor of Grace Community Church, dies at 86

Rev. John MacArthur, influential evangelical pastor of Grace Community Church, dies at 86

July 16, 2025
NYC Mayor Eric Adams hit with 'criminal conspiracy' suit

NYC Mayor Eric Adams hit with ‘criminal conspiracy’ suit

July 16, 2025

Recent News

NASA to Preview Advanced US-India Radar Mission Ahead of Launch

July 16, 2025
0
Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

July 16, 2025
1
Rev. John MacArthur, influential evangelical pastor of Grace Community Church, dies at 86

Rev. John MacArthur, influential evangelical pastor of Grace Community Church, dies at 86

July 16, 2025
2
NYC Mayor Eric Adams hit with 'criminal conspiracy' suit

NYC Mayor Eric Adams hit with ‘criminal conspiracy’ suit

July 16, 2025
2

TodayHeadline is a dynamic news website dedicated to delivering up-to-date and comprehensive news coverage from around the globe.

Follow Us

Browse by Category

  • Africa
  • Asia
  • Basketball
  • Business & Finance
  • Climate Change
  • Crime & Justice
  • Cybersecurity
  • Economic Policies
  • Elections
  • Entertainment
  • Entrepreneurship
  • Environmental Policies
  • Europe
  • Football
  • Gadgets & Devices
  • Health
  • Medical Research
  • Mental Health
  • Middle East
  • Motorsport
  • Olympics
  • Politics
  • Public Health
  • Relationships & Family
  • Science & Environment
  • Software & Apps
  • Space Exploration
  • Sports
  • Stock Market
  • Technology & Startups
  • Tennis
  • Travel
  • Uncategorized
  • Us & Canada
  • Wildlife & Conservation
  • World News

Recent News

NASA to Preview Advanced US-India Radar Mission Ahead of Launch

July 16, 2025
Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

Understanding Male Intimate Skin Health: How Much Does Pearly Penile Papules Removal Cost?

July 16, 2025
  • Education
  • Lifestyle
  • Technology & Startups
  • About us
  • Contact
  • Advertise with Us
  • Privacy & Policy

© 2024 Todayheadline.co

Welcome Back!

OR

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Business & Finance
  • Corporate News
  • Economic Policies
  • Entrepreneurship
  • Market Trends
  • Crime & Justice
  • Court Cases
  • Criminal Investigations
  • Cybercrime
  • Legal Reforms
  • Policing
  • Education
  • Higher Education
  • Online Learning
  • Entertainment
  • Awards & Festivals
  • Celebrity News
  • Movies
  • Music
  • Health
  • Fitness & Nutrition
  • Medical Breakthroughs
  • Mental Health
  • Pandemic Updates
  • Lifestyle
  • Fashion & Beauty
  • Food & Drink
  • Home & Living
  • Politics
  • Elections
  • Government Policies
  • International Relations
  • Legislative News
  • Political Parties
  • Africa
  • Asia
  • Europe
  • Middle East
  • Artificial Intelligence
  • Cybersecurity
  • Emerging Technologies
  • Gadgets & Devices
  • Industry Analysis
  • Basketball
  • Football
  • Motorsport
  • Olympics
  • Climate Change
  • Environmental Policies
  • Medical Research
  • Science & Environment
  • Space Exploration
  • Wildlife & Conservation
  • Sports
  • Tennis
  • Technology & Startups
  • Software & Apps
  • Startup Success Stories
  • Startups & Innovations
  • Tech Regulations
  • Venture Capital
  • Uncategorized
  • World News
  • Us & Canada
  • Public Health
  • Relationships & Family
  • Travel
  • Research & Innovation
  • Scholarships & Grants
  • School Reforms
  • Stock Market
  • TV & Streaming
  • Advertise with Us
  • Privacy & Policy
  • About us
  • Contact

© 2024 Todayheadline.co