Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

Rep Present New requester: seeking feedback on a series of HITs

Discussion in 'Requesters' started by pilazza, Apr 28, 2018.

  1. pilazza

    Expand Collapse
    New Member

    • Requester
    Apr 27, 2018
    Hi everyone,

    I'm a newbie requester looking to get some feedback on a series of HITs for identifying and labeling forum/blog/Q&A sites. Early prototypes of the HITs (on the worker sandbox) are up at:

    HIT #1: Identify forum sites:
    This is a simple single question with instructions. I've got keyboard shortcuts for the options for easy completion. Would it be helpful to have the site itself in an iframe so it displays inline? Or is it better as a link that opens in a new tab/window?

    HIT #2: Find forum root:
    This is a text box entry, again after looking at and manipulating a website. Same question about iframe vs. link as before. Also, is it clear enough what I mean by forum "root"? Would some examples help?

    HIT #3-5: Label index/thread/post pages:
    These involve a custom Javascript overlay that walks you through labeling several (~20) fields on a webpage. I'm most concerned about the granularity of these HITs - I know that it's best to break HITs up into individual tasks that can be completed quickly, but in these cases, the later stages are dependent upon what's selected in early stages, and I can use software to reduce the amount of work the Turker has to do or improve the error-checking on their input. (For example, if no categories are selected, it should hide the request to label category fields, and the set of legal click targets for the field is modified by the items selected in earlier stages.) Are these too daunting & long? Should I break them up so that each "stage" is a separate HIT, moving some of the software logic that's currently in the Javascript to the server? So that each individual labeling task is a separate HIT? Or is it more convenient to be able to click through a number of fields at once all on the same document?

    I'm also interested in feedback on wording, instruction clarity, UI, and pricing, if you have any opinions on those. I'll likely change the color scheme to match the default templates and add some example screenshots in the future.

    Thanks for any input!
    Bonitagirl33 and TSolo315 like this.
  2. Achilles2357

    Expand Collapse
    Active Member

    May 24, 2017
    I think you need a separate signup to get into the sandbox. I've thought about it but I haven't done it yet, so I'll just respond on the fly....
    1) If the hit is only a few cents, I'd say iframe, otherwise new tab. For workers, every keystroke costs money. Even if you provide shortcuts for easy completion, it is quite plausible that some worker will come up with better shortcuts and release this via a script. But some workers will pick this up and some won't. The track record for requesters grasping worker workflow is not that great. Hits done in the iframe carry the risk of accidental early submission, especially if you are trying to do them efficiently. But they are MUCH better if you have some workflow established.
    2) As for instruction clarity, I'd ask yourself how you would respond if a worker complains about an "unjust" rejection. Anyone liable to respond to your posting here will be a fairly good mturker (some much better than me...) But depending on your pay, some, or many, of your hits will be done by less skilled turkers. I'd ask yourself whether you can clearly justify rejections by citing the instructions you have given.
    3-5) As I said, I don't have the sandbox login right now, but my own sense, and my sense of what others think, is that if there are repeated actions, it is best to have these in multiple hits, so that workers can optimize their own workflow. Multiple hits carry overhead, but if a worker commits to doing many of your hits and trusts you, and has established a good workflow for your hits, then having to do many hits is a good thing, and increases a worker's overall rating (both in hits done and acceptances).

    All of that aside, I'd say the most important issues from the worker side are: 1) pay, and 2) trustworthiness (is there a danger of mass rejection?)
    Collapse Signature Expand Signature