Yet another great post today, this one from SEOmoz is about the problems and choices presented to SEOs by search engines because of their API and automated scraping policies.
I admit it. SEOmoz is a search engine scraper – we do it for our free public tools, for our internal research and we’ve even considered doing it for clients (though I’m seriously concerned about charging for data that’s obtained outside TOS). Many hundreds of large firms in the search space (including a few that are 10-20X our size) do it, too. Why? Because search engine APIs aren’t accurate.
I’m right there with randfish on this. I’ve developed some tools that scrape Google SERP data and return some awesome stuff, but I’m worried about publishing them for public consumption because of course, scraping is against TOS, and the APIs aren’t accurate. I really wish I could get access to real SERP data without pissing off the big G. I’m pretty sure there is some worry about reverse engineering or something that prevents them from allowing us access to this.
Who knows, maybe I’ll take my chances and release the tools, they are pretty sweet.
