Google’s Human Evaluation Team

We’ve heard rumours for some time now that in order to keep its results in check, Google uses a team of human evaluators to check the results of its automated search algorithm. The numbers of people involved are unknown but speculation suggests that there are somewhere in the region of 10,000 people worldwide who belong to this team. Scepticism is always rife when it comes to Google and what they do to get their results but recently whilst doing some log file analysis we found the following referral:

https://www.google.com/evaluation/search/rating/task-edit?task=xxxxxx

I wouldn’t recommend trying to open the url as access is obviously limited to those with the correct “evaluation” accounts but it confirmed in our minds that this is a real team of people checking results.

Is Human Evaluation a Good Thing?

It should be. There’s no reason to suspect otherwise but humans are capable of making errors. Without knowing the criteria used for human evaluation it’s impossible to say what problems could arise. It can’t be the highest paid job in the world and as we understand it is usually carried out by students on a part time basis being paid by the hour. The fact that we hear very little about it and the criteria used for checking, we suspect that Google has tied down contractual arrangements pretty well and would therefore hope that those arrangements also extend to the quality of the human evaluation.