• by Laurie Sullivan , Staff Writer @lauriesullivan, October 4, 2016

Search experts try to answer questions  such as which page do you think ranks better in search engine query results, or what change in the copy or extra link can cause an actual reordering of a specific ranking.

These are interesting questions indeed. In a survey of about 300 people in the U.S. and UK, Will Critchlow, founder of marketing agency Distilled, found that "while accuracy generally increases with experience in both the UK and the U.S., the vast majority of UK respondents performed worse than a coin flip." 

Critchlow's survey asked a few hundred people to pick which of two pages would rank better for a range of keywords, the average rate by search engine experts in the United Kingdom answering those questions came back at 46% accurate. 

"My approach is generally to zoom out and build business cases on assumptions about portfolios of rankings, but it’s been on my mind recently as I think about the ways machine learning should make Google rankings ever more of a black box, and cause the ranking factors to vary more and more between niches," Critchlow writes.

When humans answered those questions for terms like "mortgage calculator" and "mortgage comparable" the accuracy rate came back at came back with an accuracy rate of 60% and 65%, respectfully. The term "hotels near Manchester, had a human accuracy rate of about 59%. 

Even the top search experts have difficulty answering this most basic question, as ranking factors become increasingly complex and Google and Bing build more sophisticated technology into the equation. 

SEO experts with three or more years of experience in the U.S. has an accuracy rate of about 56% versus 50% in the UK. Those with one to three years in the U.S. had an accuracy rate of about 57% versus 45% in the UK. Those with some digital experience in the U.S. had an accuracy rate of about 53% versus 40% in the UK. And laypeople in the U.S. had an accuracy rate of about 50% versus 42% in the UK.

"The question I asked SEOs was 'which page do you think ranks better?' not 'which page is a better result?,' so in general, most of the results say very little about whether Google is picking the right result in terms of user satisfaction," Critchlow wrote, sharing some of the results in the post.