Remember the times when you had to submit your URL to SEs? It appears that Google, Yahoo! and others are aware of your new domain the moment you create it.
Take this website as an example. Whois record shows the following:

Domain Name: SEOWARRIOR.NET
Updated Date: 17-nov-2008
Creation Date: 14-nov-2008
Expiration Date: 14-nov-2010

Without adding my URL to any SEs, just a few days later web spiders started crawling my site.
Visitors included:

crawl-66-249-73-89.googlebot.com (Google)
llf531386.crawl.yahoo.net (Yahoo)
copilot.thunderstone.com (Thunderstone)

I have not added this domain’s URL to any of the above search engines!!

Search Engines are becoming more powerful by day. Depending on your specific circumstances, this behavior may be wanted. For websites that are still being built this may be something you may want to prohibit. Simply create a robots.txt file to disallow all web spiders from crawling your site. Simply create robots.txt file with the following entries:

# disallow all spiders for entire website
User-agent: *
Disallow: /

Note: keep in mind that some spiders may still crawl your website as they are not honoring the Robots Exclusion Protocol (REP).

Post to Twitter

Tags: ,

Comments are closed.