Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

Copycat "News" on "Mturk Bot Panic"

Discussion in 'AMT News' started by Achilles2357, Aug 19, 2018.

  1. Achilles2357

    Achilles2357
    Expand Collapse
    Active Member

    Joined:
    May 24, 2017
    Messages:
    966
    Likes:
    1,457
    In the past week there have been multiple "news" reports that psych researchers are in a panic over the possibility that "bots" are doing psych surveys on mturk. They all basically repeat the same stuff, and you can find them on Google News by searching for mturk. So far as I can tell, there is basically no original reporting involved and they all seem to derive from some observations by the guy (or girl, I guess from the name) who posted this (https://www.reddit.com/r/mturk/comments/95yy96/data_contamination/) on reddit, and apparently much the same stuff on Facebook.
    tl;dr: Researchers need to PUT IN THE WORK to block bots.
    tl;dr: Researchers need to PUT IN THE WORK to eliminate bad data.
    Note to researchers: Jeff Bezos doesn't care about you any more than he cares about us. Accept it and life will get better.
     
    Collapse Signature Expand Signature
  2. TSolo315

    TSolo315
    Expand Collapse
    SnapNCrackle

    • Moderator
    Joined:
    Jan 12, 2016
    Messages:
    2,914
    Likes:
    9,892
    IMO Mturk goes out of their way to exploit researchers who don't know better with how they have set up their fee structure.

    I'm honestly surprised survey bots are a problem with how many people seem to have difficulty even setting up an mturk account.
     
    Collapse Signature Expand Signature
  3. Achilles2357

    Achilles2357
    Expand Collapse
    Active Member

    Joined:
    May 24, 2017
    Messages:
    966
    Likes:
    1,457
    I suppose some originally legitimate accounts get "sold". I don't see how the effort to pull this off will yield any major profits for bad actors, assuming researchers are using reasonable controls. There is absolutely no way that anyone has a bot capable of maintaining a 99% doing only surveys. I suppose I could be wrong, but I simply do not think it so. If it is true, it is a sign of utter cluelessness on the part of researchers, since it would not be difficult at all to deal with this, and significantly increase bot identification.
    Maybe people trying surveys with bots are just testing their bots for other uses.
     
    Collapse Signature Expand Signature
  4. TSolo315

    TSolo315
    Expand Collapse
    SnapNCrackle

    • Moderator
    Joined:
    Jan 12, 2016
    Messages:
    2,914
    Likes:
    9,892
    Amazon will also ban those accounts at some point. Buying accounts only to scam surveys before they get banned doesn't seem like it would be profitable. But maybe I'm overestimating how competent Amazon is in this regard.

    And bots like this are single-purpose for the most part. A bot created to scam surveys wouldn't be good for much more than scamming surveys. I'm skeptical about the testing angle.
     
    Collapse Signature Expand Signature
  5. Achilles2357

    Achilles2357
    Expand Collapse
    Active Member

    Joined:
    May 24, 2017
    Messages:
    966
    Likes:
    1,457
    I suppose that if there are lot of surveys that don't set up their quals reasonably, and don't have adequate controls that will ensure that a good portion of bad work will be both detected and rejected, and if the requesters fail to report suspicious accounts to amazon, then maybe a scammer could temporarily "earn" money at a better rate than the typical mturker. That might be all it takes to get someone trying this.
    As for "testing", I suppose I was thinking of someone simply testing out their ability to make money via illicit means.

    I really think this whole issue can be avoided by ignoring the issue of "bots" per se and focusing on the need to exclude the very bottom tier of mturkers who give nonsense answers. (Bots will be excluded as well.) I really suspect that the research community using online surveys has not given this the actual intellectual attention it deserves. (But maybe that is changing since I did a survey the other day that seemed to be nothing but a series of rather different attention check type stuff, so maybe they are they are actually trying to take this seriously.)

    I do think bots matter, but not for surveys, so long as you exclude "bad workers". Mturk itself likely needs active strategies for bot detection, but I think academic researchers simply need to focus on eliminating completely unengaged workers. I think part of the problem is that the IRB culture actively eschews the idea that researchers are hiring "workers", whereas mturk is unequivocally a work platform.
     
    Collapse Signature Expand Signature
    TSolo315 likes this.