Recent changes to feature-requests

#15 Rate-limiting functionality

Daniel Barowy — Sun, 20 Mar 2016 02:41:52 -0000

Never mind. I just discovered that I was passing "Server.ServiceUnavailable" to your retry mechanism. I have now disable this, which in principle means that I can implement my own backoff mechanism since the SDK ought to just throw an exception. Feel free to close this ticket.

Rate-limiting functionality

Daniel Barowy — Sun, 20 Mar 2016 01:35:08 -0000

The MTurk SDK sorely needs some kind of rate-limiting logic. I have been battling a problem with MTurk where I post a large batch of HITs and then am unable to retrieve the completed assignments because of "Service Unavailable" responses from MTurk. I can only assume that this is because my connection requests are too frequent. This is despite the fact that I have a rate-limiter on my end which ensures that I make no more than 1 connection every n seconds (currently set at 4.5 seconds).

It took me forever to discover that this was actually the problem because the SDK completely concealed the fact that MTurk was returning this error. What I actually saw, in a profiler, was the SDK spinning endlessly in its retry logic. Indeed, what I am seeing on an app that is running RIGHT NOW is that each call to getAllAssignmentsForHIT takes anywhere from 30-45 seconds and then eventually returns nothing. Yet if I sign into my MTurk requester account, I can see that all of the assignments for all of my HITs have indeed been completed.

I can only assume that when this happens, MTurk and the SDK enter into some kind of mutual rate limiting death spiral and nothing makes progress. This is super super frustrating and is costing me (a student, at a university, working on my dissertation) hundreds of dollars per failure. I would gladly implement some kind of exponential backoff on my end, but I get no indication from the API that anything is at all wrong (except that calls take a very long time), which would make doing this a challenge.

I don't know if this is a feature request or a bug. Any help that you can offer would be appreciated. I will gladly fork over logs, source code, etc., if you want to try to reproduce this.

Support for DoesNotExist Comparator

Eran — Mon, 21 Jul 2014 17:28:39 -0000

a DoesNotExist Comparator has been added to the API, and would be very useful if implemented in the Java SDK

#10 always override .success, .results files

Nicole — Thu, 06 Mar 2014 09:27:38 -0000

How do I find my Turk user id

Add a link to manageHIT page in results file

Jang Han Goo — Thu, 21 Feb 2008 22:52:07 -0000

Each HIT ID in the .results file generated by getResults will be linked to the manage HIT page for that HIT in the Requester UI.

Change the default delimiter to tab

jclocke — Mon, 26 Nov 2007 19:10:49 -0000

In the last release, the default delimiter was changed from tab to comma. Change the default back to tab so that the generated files work better with Excel.

Add exponential back off to handle throttling

jclocke — Mon, 26 Nov 2007 18:59:22 -0000

In the developer sandbox environment, Requesters are allowed up to 60 requests, then they're throttled at 2 requests per second. The SDK needs to handle this throttling so that applications using the SDK can be abstracted from the throttling errors.

The fix is to have SDK use retries with the right amount of waits so that it minimizes the amount of throttling errors we get.

always override .success, .results files

jclocke — Wed, 21 Nov 2007 22:52:25 -0000

Right now whenever a user does the run command or getResults command
multiple times, we append the output to the existing .success or .results
file. Users don't expect the data to be appended and sometimes it yields
undesirable behaviour.

Instead we will override existing files. This provides much cleaner and
predictable behaviour.

GetResults should output rows with no assignments

jclocke — Wed, 21 Nov 2007 22:34:27 -0000

related to feature 1826272 in the CLT.

The GetResults operations should output rows even if no assignments have
been submitted yet.

When the CLT gets results, it takes in (a file of) HIT IDs. When a hit has
no submitted assignments, no information is written to the output file for
that hit. This can be disorienting to requesters, particularly when a large
number of HITs have no submitted assignments; it can appear to the
requester that the CLT is not functioning.

Instead of skipping hits that have no assignments, when the call to 'get
assignments for hit' returns no results, modify the CLT to output nothing
for the assignment related fields.

Would prefer nOT to insert default text like "NO RESULTS SUBMITTED" in all
the fields. No data should be represented with no data.

Add some additional line feeds when outputting the URL

jclocke — Wed, 21 Nov 2007 22:32:40 -0000

related to Feature 1834853 in the CLT.

sample output:

"Created HIT 1: HITId=1XKPTG15MWJZ61YGWRK0
<newline>
You may see your HIT(s) with HITTypeId 'F0VZV9Z79YCRGQC24ZMZ' here:
<newline>
http://workersandbox.mturk.com/mturk/preview?groupId=F0VZV9Z79YCRGQC24ZMZ
<newline>
End time: Thu Nov 15 12:11:13 PST 2007"