"One of our tenets in search quality is to be very data-driven in our decision-making," he
says. "We try hard not to rely on anecdotal examples, which are often misleading in search (where decisions can affect hundreds of millions of queries a day)." Still, humans need to crunch those
numbers, and in many cases, the giant uses human evaluators to see how the algorithmic changes it makes will affect user behavior.
"These evaluators are carefully trained and are asked to evaluate the quality of search results in several different ways," Huffman says. "We sometimes show evaluators whole result sets by themselves or 'side by side' with alternatives; in other cases, we show evaluators a single result at a time for a query and ask them to rate its quality along various dimensions." The company also uses live traffic tests to see how SERP changes affect a broader swath of users as well.