We’ve all heard of it and used it too. However, in spite of its widespread popularity in SEO, it remains one of the least understood of web files.
Yes, we’re talking of the Robots.txt file, that inconspicuous little file that even after seeming to have run its course after all these years, refuses to lose its importance even today.
Let’s face it; when you are dealing with a pesky customer like Google (or any other organic search engine for that matter), you need to play your cards right. And the robots file is a trump card (only if you know how to play it right!)
Here’s an excellent guide to the whole business with robots file that should help you make the most of it. Ignore it at your own risk!