Let’s say you are working on a new Web page and you want to know how it will look on the Web, but you are not ready to release it yet.
You know that search engines have these little things called robots or spiders that constantly ‘crawl’ across the Web, looking for new stuff. How can you prevent Google and other search engines from finding your page and prematurely making its search results available to he public?
Actually, there is a simple remedy. You include a meta tag in the head section of your page as follows.
<meta name=”robots” content=”noindex”>
The search engine spiders may still find your page, but they will ignore it.
You may also want any links inside the page to be ignored. In that case, include the ‘nofollow’ directive:
<meta name=”robots” content=”noindex, nofollow”>