Question about robots meta tag

My website has about 12 pages now with more coming. I had the Index, All tag on my homepage. What is the difference between this and Index, Follow.

Should I use robots.txt to keep my download pages secured from indexing or should I use meta tags on each page?

 

Index all will allow the spider to index all pages. Index follow tells the spider to follow all links throughout the page/site.

If you really do not want those pages indexed a robots.txt should be suffice. However, personally if you did not want anyone to see them, .htaccess is the way to go IMO. At least that is what I do.

Good luck.

 

Having a statement in your robots.txt reveals the location of your downloads. Though it would be enough to prevent Googlebot from displaying it on search results, your downloads page can easily be viewed (robots.txt is accessible to anyone) and the URL can be pasted on the browser. Unless your filenames in that folder is sort of difficult to figure out.

 

Actually, the tag "All" stands for "Index, Follow", which would be the same as "Index, All". Read the HTML Author's Guide for details.

Cryo.

 

In addition to what everyone else has said, the default is that robots are able to index your page and follow its links, so reaffirming this with "index, follow" is unnecessary. You don't need them.

 

Yup, that "index, follow" is the default command to robots.

 

Thanks for all the help. I have been curious about that because I have seen it written both ways.

 

I think it's visible most of the time because many webmaster want to ensure that their pages are indexed even if search engines still do even if they don't see "index,follow" directive on their html pages.

 

i recommend robot.txt coz' you can specify a particular file, page, folder etc. that you dont want to index by the search engine..