which is the best way to use the meta robots? i dont know what is the difference between index, follow and all which one should i use? i've also created a robots.txt file on the server containing: any other advices?
If you aren't trying to prevent the web page from being crawled and don't care if links are followed than you don't need the robots meta tag. Search crawlers will follow your robots.txt just fine, again robots is only used to "prevent" crawling or following.
the robots meta tag is used like this, index, follow - this is the default and does not need to be included. Your pages will be indexed and links will be followed. You can also set it to 'index, nofollow' OR 'noindex, follow'
Yes, robots.txt is what you want to use to prevent crawlers from indexing your pages, if you have any you don't want indexed.
see if you want to stop crawler from indexing some directory or any portion than u can put it in robot.txt file. but if you want to keep crawler away from any particular page than u can put meta tag too. here is the explanation of each term index : index the page for search result follow: follow the links given in that page, it will also pass the PR all : both follow and index follow and no index: follow the link but do not index the page no follow and index: index the page but do not follow the page
thanks guys for all the info... i still don't get something... if i put this in robots.txt, will this keep the crawlers away from indexing my website? cause i don't want that
Hey. I tried to explain this to you earlier, but then ended up getting my reputation points removed because of my description?! Anyway; here goes again. The robots.txt is only used if you need to restrict access to certain directories. For example - I use it so that a directory with promotion/free give-aways is not indexed by google. Since you want your pages indexed, the robots.txt file is not necessary - you can go ahead and delete it. As the default bot behaviour is to index pages and follow meta tags, you can also delete that specific meta tag. If time comes that you have a directory that you don't want in the search results then use the robots file like this: I hope that sorts out your confusion...
ok, everything is clear now. thanks and sorry for the reputation points. i didn't remove your points. thanks again!
No worries. glad you understand it now. Another tip: if you create a google webmaster tools account; it will tell you if any pages were blocked by a robots.txt file. This may be handy if you decide to use the robots file in the future...
You can block specific "pages" in the robots.txt to, I don't know what these 2 are rambling about directories only. View one of my websites robots.txt below and then double check with the site: operator. http://www.civicseo.com/robots.txt The meta tag usage is for situations where you CAN'T upload new files to your web directory and can only edit or create new pages in a control panel. Example would be a free web hosting service. http://www.google.com/support/webmasters/bin/answer.py?answer=61050&hl=en
I think we're "rambling on" about this for the sake of simplifying the answer to the original question. Obviously a simple, straight forward answer was needed....