robots.txt

[sarahbrooks]

Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site i.e. it is not a firewall, or a kind of password protection and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

[alfie123]

Robot.txt file is very important for the website. It is simple text file and in this file those pages are being included which the web masters don't want to cache by search engines.

[Paul Jameson]

Robots.txt page is important in SEO cause it shows search engine spiders that how many pages has to crawl. Spider crawls only those pages which Robots.txt file navigate it.

[jenny123]

Robot.txt help to owner so that he can hide some official files or admin penal from user. Basically this is the main use of Robot.txt.