Nicky

Hi everyone,

Does anyone know how to block the contents of a domain URL from being indexed and making it un-searchable via Google etc? Whilst robots.txt, various meta’s and directory/page locking are familiar to protect pages and directories, I just wondered if it was possible.

Why? I have a potential customer who wants website pages to be indexed but not the content within her chosen domain URL. In other words, she wants to remain vaguely anonymous but with a memorable name as a domain URL but does not want people to be able to search on her name and find it in a URL! …aarghh! Why do I always get the easy jobs!

I said; “It can’t be done, so please choose a non-descript domain name that wasn’t so obvious.” Only because that was the only answer I could think of. But nevertheless, that intrigued me. Is it possible to prevent the contents of an URL from being searched-on whilst not affecting the page content itself? Personally, I don’t think it is.

Just curious…

acutecomputing

I can think of a couple of ways;

1. Use Robots.txt - the well-behaved robots respect them.
2. Put all content in images - Still takes a lot of processing power to read them.
3. Use .htaccess to reject search engines (a pain, because you have to find reliable ways of identifying them)
4. Buy several more domains and duplicate the content on them. They might all get penalised and dropped from the index!
5. Don't link to the site - ever. Don't set up email on the site domain (as otherwise, if anyone on gmail emails her, the site will be indexed by Google eventually)

Hard to understand why she wants it that way, of course...


HTH,

Pete.

__________________
Utah based custom PHP and Drupal Development
www.kpdirection.com
www.kids-faith.com

No Name

I vote for .htaccess. I had something like while testing the site and it worked.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Googlebot.*$ [NC]
RewriteRule .* - [F,L]

Never rely on robots.txt alone

__________________
Signature removed by moderator.

thinkbizom

Use Robots.txt

This file can be used to block crawlers from certain pages and folders on your site. Make sure you don't accidentally block them from pages you do want indexed. I personally don't really count the robots.txt file as part of on-page SEO, since it's not directly contributing to where you rank (you don't actually need a robots.txt file at all).

This example allows all robots
User-agent: *
Allow: /

This example keeps all robots out:
User-agent: *
Disallow: /


.htaccess code

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Googlebot.*$ [NC]
RewriteRule .* - [F,L]

DavidAmerland

Nicky, you're right. This is idiosyncratic enough to border on the unique. You can block off entire parts of a site following the 'No Index' command in your robots.txt file - the Google bot will respect it and so will Bing and Yahoo! The rest really are of no real importance. That protects the contents of specific pages within a domain name (or entire sections of it) use a doorway page to present different content to search engine bots and human visitors and not only do you guarantee it will not be searched but if you do any SEO on it Google will drop it from the Index altogether.

I hope this helps.

David