ok I'm starting to work on my first robots.txt  and all i want to do it block them from going to my other directories. the only one i want them to be able to access is my public_html directory..


here is what i have so far::



robots.txt


User-Agent: *

Disallow: /_private_directory/

User-Agent: *

Disallow: /forum_directory/

User-Agent: *

Disallow: /gallery_directory/

User-Agent: *

Disallow: /spenser_directory/If you don't want to permit access to those directories, consider removing any that don't need to be PART of your web tree.so just delete my forum and gallery folders? but wouldnt that mess up my forum and gallery pages??
Yes, it would, wouldn't it...
Your Robots.txt file looks fine. It seems that Google and other search engines will not index those directories.

You MAY want to allow CERTAIN directories to be indexed, and just block the SUBS inside them.

For example:
I have a forms directory. If I wanted to prevent the admin panel from being indexed only, it would be

Disallow user agent: *
disallow: /forms/use/CHSurveyResopnce/admin/ (not a valid directory)
instead of just blocking the ENTIRE /foms/ directory.

I think your file is fine, unless theres something else you would like to add, Rob. I think we both missunderstood your last post.

LOOK! 4 more posts till I'm at my first 1000!In a previous post, you said that you wanted your gallery indexed.
If you want it indexed, you need to allow it. Quote

so just delete my forum and gallery folders? but wouldnt that mess up my forum and gallery pages??