New docs for Web designers on controlling the Googlebot

More information is now available within the Webmaster site (Google Code section) for controlling how the Googlebot crawls and indexes a website. The section is broken down into the following categories:

• Getting Started
• Robots.txt specification
• Robots meta tag and X-Robots-Tag specification
• Google’s crawlers
• References

This new section can help give web designers information on how to, for example, prevent a  PDF file from being indexed. This new section housed on code.google.com provides web designers and webmasters a comprehensive resource for understanding the robots.txt file usage, X-Robots-Tag HTTP header directives and robots meta tags.  From basic definitions of technologies involved to instructions on how to control robots for a website, the new area brings us closer to everyday robots.