Robots.txt file

[Oz.]

I saw a site the other day have this directive in their robots.txt file:

User-agent: Googlebot
Disallow:

What does a blank "disallow" in this situation mean (ie. with no path/file listed)?

I'm guessing it is basically a nonsensical directive?

 

[chris]

Hi Oz,

It looks like it's valid, but used in conjunction with other directives:

http://www.robotstxt.org/robotstxt.html

To allow a single robot

User-agent: Google
Disallow:

User-agent: *
Disallow: /

Cheers,
Chris

 

[FirstPageResults]

It allows complete access to the specified user-agent

 

[James]

I would also include the Sitemap.xml location in the robots.txt as a best practice.