Stockholm Confidential. Greg had just gotten back from a search conference in Sweden. At said conference, there tends to be a handful of black-hat SEOs that hold court after the show shuts down, showing off their spam de jour. For those of us who primarily live on the white side of the fence (and I say primarily because one is never sure exactly where that fence is) it's always a guilty pleasure cornering one of these dark magicians. They're brash, confident and masterful in their manipulation of algorithmic loopholes left by Google, Yahoo and MSN. Using every tactic in their arsenal, they manipulate sites up the rankings and make fistfuls of money in the process. I once asked a black hat if he had any ethical twinges. He replied, "The odd time, but my kids are going to a great college."

This trip, Greg managed to take a black hat to dinner. And in between the courses, a confession came out that stopped Greg in his tracks. "Black hat stuff is getting too hard. I'm actually thinking about turning legit." What? Is this capitulation? Is Courtney Love taking up a nun's habit? What would cause a confirmed black hatter to turn his back on the incredibly lucrative dark side of SEO and step into the light? As much as the army of engineers at Google and Yahoo would like to say it's their constant refinement of their algorithms, I think there's another force at work here. Online is just growing up.

Frontier Mentality. Up to now, online has been the Wild West. The sheriff hadn't come to town yet. Black hats could get mediocre sites to the top of the rankings because the vast majority of legit sites had no clue about search engine optimization. Reams of content were hidden in content management systems, locked off from the search engines by impenetrable dynamic urls. Ill-conceived site architectures meant redirects off the home page to destinations buried four and five levels deep. The essential title tag wasn't even optimized. This is more common than you think. I've participated in a number of search workshops where some of the best-known brands in the world had their sites examined. It's rare to see a keyword show up in the title tag.

But, slowly, things are changing. Brands are clueing into the importance of algorithmic search. Spider friendliness is usually a requirement in evaluations of new CMS solutions or site redesigns. And when you take a site that has thousand of pages of content, with rich internal linking structures and scads of legitimate, authoritative incoming links, it will jump to the top of the search results. It's inevitable. Those are the sites MSN, Yahoo and Google want at the top of their results. Those are the sites we want to see at the top of the results. It's the online universe working as it should.

The Settling of Main Street. Today, these huge brands are turning to white-hat search practitioners to help unlock the full potential of their sites. At this point, it's still a trickle, but it's improving every day. And every time a big brand grabs a spot in that "Golden Triangle" at the top of the search results, a black-hat-manipulated site is moved a little further down the ladder. It doesn't matter what tricks a black hatter has up its sleeve, you can't beat the sheer bulk of these killer sites, as long as they're properly optimized.

So, as the online geography becomes more civilized through the influx of legitimate business, black hats are forced to move off Main Street into the back alleys. There's less territory for them to operate in. And now, they're competing for position against other black hats who are as ruthless as they are, rather than against naïve site owners who have never heard of a meta tag or Pagerank. It gets harder to make a buck.

I'm not discounting the effort that the search engines have made to clean up spam. Google's Florida Update was probably the single biggest blow to black hat optimization and affiliate spam. But, at the end of the day, spam's being eliminated because better sites are being optimized effectively, allowing them to naturally claim their rightful territory in the search listings. And it's the legitimate SEO industry that's making that happen.

Isn't it ironic? As the Web grows up, it appears that many of us in the SEO industry might actually turn out to be the sheriff.

This commentary is insightful. I recommend it to others.