0 Comments | AFP, December, 2009
SAN FRANCISCO (AFP) Google News on Wednesday launched a “news-specific crawler” that lets online media automatically keep stories, photos or video out of its index.
The announcement comes a day after the California-based Internet giant said it is letting publishers limit the number of online pages people can view after being routed to their websites by Google’s search engine.
Publishers have always been able to block Google from including their website content in the search engine index.
Google senior business product manager Josh Cohen said in a blog post that a new “web crawler” extends that option to Google News.
Web crawlers are automated programs that scour the Internet for content and then index it in databases routinely mined for results to online…