Is Robots.txt better than Robots in META?

Is Robots.txt better than Robots in META?

-C

 

Obviously. Do you have any doubt in it? Robots.txt provides to flexibility to handle the crawling process of your whole website from a single source. However, if you use meta tags, you will have to invest lots of time. Also, in case of dynamic websites, robots.txt is the best pick to set the crawling priorities.

 

To be honest... i wasnt sure, thats why i asked.

Where can you set up a robots.txt page?


-C

 

No problem, this is the only reason people come to forums.

To add robots.txt file in your website, you should first create a robots.txt file adding the necessary instructions as per your website instructions and then you can add your robots.txt file in the root folder of your website. It should be visible at yourwebsite.com/robots.txt

 

Any way you can show me a proper example of a robots.txt page?

Thanks!

-C

 

robots.txt all the way! I don't even use the robots meta these days.

Jack

 

If you want to see an example of robots.txt file, you can visit our website at http://www.pixelcrayons.com/robots.txt

 

Thanks but you dont need robots if your not disallowing any pages.

 

I do not agree with you on this points. According to Jonathan of Google

"This way [correct setup] when the Google crawler or other search engine checks for a robots.txt file, they get a 200 response if the file is found and a 404 response if it is not found. If they get a 200 response for both cases then it is ambiguous if your site has blocked search engines or not, reducing the likelihood your site will be fully crawled and indexed."

 

I believe Google uses the site maps to index. I also think robots are only for not indexing pages..This is what i recently found.

Can someone please clarify this?

Thanks,

-C

 

you are mixing two things in one thing. Sitemap is basically a way to tell search engine crawlers that these are the pages you should visit. However, if your website is properly interlinked, you do not need to make a sitemap file, Google will automatically index your all pages.

Robots.txt is basically a way to instruct crawlers what portion of a website they should index and what portion they should avoid. It is recommended to have a robots.txt file in whatever case.

 

If you dont instrust them to NOT index any pages then all will be indexed...i dont see the point.

 

yes you are correct. I put this in this way, you said you use sitemap file to index your pages. If you won't add sitemp file, will your website be not indexed. It will. But it is recommended to have a sitemap file in your website so robots.txt

 

THank you for clarifying...i appreciate the meaningful conversaton. Rep added.

 

Thanks Woodcs82

 

Can you clarify on robots.txt issue.

 

Read the posts above...that will clarify.

 

I don't agree with Manish. Because pages in robots.txt are indexed by search engines but arn't displayed in results. If you type exact keyword phrase or file name with your url, you'll see your URL's are indexed even if you have blocked them on robots.txt.

I recommend to use robots meta....

 

You are bit incorrect Candrison. Google says that they might index the pages blocked through robots.txt, but in general, it doesn't happen.

 

hi,
It doesn't meter, you can use any of them.