Does anyone here actually use Turk, though? The forum seems nearly abandoned (http://developer.amazonwebservices.com/connect/forum.jspa?forumID=11), and no Amazon employees have responded to any questions. The API has not had any updates in years. Is the Mechanical Turk now a walking corpse?
Read on for a description of the problem I've run into. Maybe a fellow enterprising HN reader dealt with something similar and found a solution. :)
I need to enforce that the same worker does not look at closely-related images. I can enforce that in my app, and basically show a task that says "please return this task and try some of our other tasks in a couple of hours". This is annoying, and drives up the "returned tasks" statistic for the workers. I would very much like to make Amazon filter out tasks for users who have run out of valid work in my system.
At first, it seemed that Turk qualifications fit the bill. Unfortunately, the Turk system does not support "negative" qualification, e.g., it cannot only assign tasks to workers who do not have a particular qualification. Amazon employees have not addressed this question from a year ago: http://developer.amazonwebservices.com/connect/thread.jspa?threadID=32744&tstart=0
Then, I tried blocking workers once they run of tasks, and unblocking them once work shows up. This does not affect task filtering, and instead tells the user about a block. Additionally, blocking may or may not have really bad side effects for workers, another question Amazon employees have ignored: http://developer.amazonwebservices.com/connect/thread.jspa?threadID=48851&tstart=0
Advice and suggestions welcome.