We are a software development firm in Seattle that has done a lot of crowdsourcing work. Because of that we had to solve many problems related to getting quality results from workers that we might only know for a few minutes. What we found was that crowdsourcing is very good for distributing the load. For example if you have 1000 documents to read and enter in the database, why not summon 100 people and each of them do 10 document.
The problem you immediately hit is that you don’t know if each of these people actually did the work. Some of them simply type in garbage and now you are stuck looking through everything that was done which doesn’t save you much time.
We faced this problem and created what we call the Progressive Quality Review or PQR process which uses a predictive model to figure out when a worker is about to make a mistake and then have other workers jump in to check it over. The result is the super efficient and scalable system zCrowd.