Lately, I have been playing around a bit with Amazon’s Mechanical Turk service. For those who have no idea what Mechanical Turk is, the following description provided by Amazon may be useful:

What is Amazon Mechanical Turk?

In 1769, Hungarian nobleman Wolfgang von Kempelen astonished Europe by building a mechanical chess-playing automaton that defeated nearly every opponent it faced. A life-sized wooden mannequin, adorned with a fur-trimmed robe and a turban, Kempelen’s “Turk” was seated behind a cabinet and toured Europe confounding such brilliant challengers as Benjamin Franklin and Napoleon Bonaparte. To persuade skeptical audiences, Kempelen would slide open the cabinet’s doors to reveal the intricate set of gears, cogs and springs that powered his invention. He convinced them that he had built a machine that made decisions using artificial intelligence. What they did not know was the secret behind the Mechanical Turk: a human chess master cleverly concealed inside.

Today, we build complex software applications based on the things computers do well, such as storing and retrieving large amounts of information or rapidly performing calculations. However, humans still significantly outperform the most powerful computers at completing such simple tasks as identifying objects in photographs—something children can do even before they learn to speak.

When we think of interfaces between human beings and computers, we usually assume that the human being is the one requesting that a task be completed, and the computer is completing the task and providing the results. What if this process were reversed and a computer program could ask a human being to perform a task and return the results? What if it could coordinate many human beings to perform a task?

Amazon Mechanical Turk provides a web services API for computers to integrate “artificial artificial intelligence” directly into their processing by making requests of humans. Developers use the Amazon Mechanical Turk web service to submit tasks to the Amazon Mechanical Turk web site, approve completed tasks, and incorporate the answers into their software applications. To the application, the transaction looks very much like any remote procedure call: the application sends the request, and the service returns the results. Behind the scenes, a network of humans fuels this artificial artificial intelligence by coming to the web site, searching for and completing tasks, and receiving payment for their work.

I have found the Mechanical Turk service useful for some tasks I needed done but for which I did not have the skills, time or patience.

The first thing I tried was creating a task to translate a short newspaper article from French into English. However, this did not work out very well as I got two submissions that were obviously the product of machine translation from something like AltaVista Babel Fish or Google Translate. The third submission was just barely acceptable but ultimately not usable for my purposes. To make matters worse, since I don’t know much French I couldn’t easily verify the accuracy of the translation. However, Mechanical Turk could certainly be used for translation services (indeed there are a couple companies that have used it for that purpose - however, when I contacted one of them for a quote I never got any reply) if you set up a system to evaluate the quality and accuracy of the translation work.

I had much better success with another type of task: transcribing videos in a foreign language (in this case, French television news reports). I think it helped that I offered a bit more money than most requesters were offering for similar tasks. However, I still paid much less than a professional transcription service would have charged.
More on this later…