Specific Q about robots.txt

Is there any way I can make the robot ignore all data files without me meta-ing all of them?

 

if you have them all in one directory ... yes

google bot also will accept wildcards "*"

can you explain more what you are trying to do ?

 

i have lots of dat files spread across my site. I don't think I should have G reading files on my site with a few bytes of strings at the top.

 

What do you mean "dat files"?

 

Just a note: If the dat files are just used internaly by php scripts, and are NOT linked from a visible web page, Google won't find them in the first place!

 

oh yeah. You have to addurl if you don't have any IBLS! I forgot that.

*doh*

and J.D. by dat files i mean simple text files with a .dat extension.

 

I meant what are they used for. Anyway, as was pointed out correctly, if these files are just internal server-side files, you shouldn't worry about them.

Another story from the security perspective. Try requesting any of these files with a browser. If you get any of them, somebody else may as well (it's called foreceful browsing). In this case you might want to configure your server to return "404 Not Found" if client requests anything with a .dat extension.

J.D.

 

all this data is publicly displayed. If I wanted anything private, I'd load it into SQL.

but thanks for the tip.