Notifying search engines of new content

Search engines index the web by crawling along links on web pages. It's rather inefficient for a search engine to reindex pages which have not changed and which have already been indexed. Hence it improves efficiency of the whole internet for search engines to have a way to know what pages are new, what pages are old, and only reindex the new pages.

A few years ago Google developed the XML Sitemap protocol which has since been adopted by all the search engines. It's an XML format which lists pages on a site, the latest update date, and other data. The Drupal XML Sitemap project enables Drupal to generate and submit XML sitemaps.

The built-in ping module does something to notify about new content. In older times the 'ping' concept was formed by the likes of Technorati to notify the blog search sites of new content on a given blog. This ping was a simple process, and if I recall correctly the idea was to simply retrieve the RSS feed and look at the latest entries. Ping is a core built-in module and notifies the service at http://pingomatic.com which then rebroadcasts the ping to other sites that accept blog pings. You simply enable the module and forget about it, as it takes care of itself.