What is googlebot vs robots.txt ?

hi guys,

i have read thru some info from google, and i so confuse about these 2 term.
Googlebots & Robots.txt

Actually wht is tht? Is that important? Wht if i dun care about it, will it impact my site get index?

Thanks...

 

googlebot = the spider of google search engine
robots.txt - www.robotstxt.org

 

oic,
then how to generate googlebots? and msnbot ?

 

You don't generate anything.

Gogglebot and MSN bot will come around and spider your pages.

robots.txt is a way to tell them to go away or some back different frequencies etc.

You should probably read up at w3schools on robots.txt

 

oic...that mean we no need to do anything on "validate" or "create" our robots.txt file for our site, unless we dun want search engine to crawl our site.

Am i rite?

Can i say, just ignore about the robots.txt for all the search engine?

By the way, can i say tht once i have submit my sitemap to google/ yahoo/ msn.... my job is done? Just waiting to see my site get index?

Thanks,,,,

 

Many webmaster recommend to ignore robots.txt file if you want your entire website to be crawled by searchengine bots.

But in my opinion one must have robots.txt file. I write the following in my robots.txt file:

User-agent: *
Disallow:

Sitemap: http://www.example.com/sitemap.xml 

Code (markup):

Now this helps any search engine bot to get my sitemap xml easily & crawl it easily. This way I don't have to submit my sitemap xml in all the search engines.

 

hi ann_india, great !

Am i just need to create 1 file call robots.txt and upload into my server at
www.mysite.com/robots.txt ?

And the content for the robots.txt is just a few words as u mention previous?

User-agent: *
Disallow:

Sitemap: http://www.example.com/sitemap.xml

am i correct?

 

Yes you are correct. Regarding the content in the file, let me explain it to you.

There are many other things also which could be included. More details can be found at robotstxt.org and en.wikipedia.org/wiki/Robots.txt