Optimize Robots.txt file

Discussion in 'Search Engine Optimization' started by jaguar34, Jun 28, 2015.

0
  1. #1
    How do allow Google and Bing Bots to crawl my website's sitemap.xml file and index the pages in the sitemap.xml file? what codes to put in robots.txt file to make this happen?
     
    jaguar34, Jun 28, 2015 IP
  2. Mkcoy

    Mkcoy Well-Known Member

    Messages:
    1,602
    Likes Received:
    77
    Best Answers:
    2
    Trophy Points:
    195
    #2
    Nothing. Google and Bing bots will crawl your sitemap file anyway.

    All you need to do is add it in your Google/Bing webmasters tools.

    If you have a virtual robots.txt file then your sitemap url is automatically inserted into it.

    Else just put this in your robots.txt

    Sitemap: http://whateveryoursiteis.com/sitemap.xml.gz
    Code (markup):
     
    Mkcoy, Jun 28, 2015 IP
  3. LinkGenie

    LinkGenie Active Member

    Messages:
    794
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    60
    #3
    You can use this code to allow any bot to crawl your site

    User-agent: *
    Sitemap: URL/sitemap.xml

    if you allow Bing and Google only, you can use following code.

    User-agent: googlebot
    User-agent: google
    User-agent: bingbot
    User-agent: bing

    User-agent: *
    Disallow: /
     
    LinkGenie, Jun 30, 2015 IP
  4. abinjoy

    abinjoy Member

    Messages:
    110
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #4
    To allow all robots complete access write below code in your robots.txt file
    User-agent: *
    Disallow:
    Sitemap: http://www.    yoursite.com/sitemap.xml

    To exclude all robots from the entire server write below code in your robots.txt file
    User-agent: *
    Disallow: /
     
    abinjoy, Jun 30, 2015 IP
  5. Aryans

    Aryans Well-Known Member

    Messages:
    1,854
    Likes Received:
    31
    Best Answers:
    1
    Trophy Points:
    178
    #5
    This code is allow to Google? what is this crap??? in dis code you are writing "disallow"
     
    Aryans, Jul 1, 2015 IP
  6. patco

    patco Well-Known Member

    Messages:
    2,035
    Likes Received:
    47
    Best Answers:
    17
    Trophy Points:
    100
    #6
    And why do you have to put such things in the robots.txt? Isn't robots.txt used ONLY if you want to EXCLUSE something from being INDEXED? :)
     
    patco, Jul 2, 2015 IP
  7. AmitSolanki1188

    AmitSolanki1188 Member

    Messages:
    34
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #7
    You can also submit your sitemap.xml in Webmaster tool.
     
    AmitSolanki1188, Jul 4, 2015 IP
  8. abinjoy

    abinjoy Member

    Messages:
    110
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #8
    code is allow to all search engines. I didnt mentioned google antwhere.
     
    abinjoy, Jul 14, 2015 IP
  9. Aryans

    Aryans Well-Known Member

    Messages:
    1,854
    Likes Received:
    31
    Best Answers:
    1
    Trophy Points:
    178
    #9
    Hope you know the meaning of "disallow" and "/"
     
    Aryans, Jul 14, 2015 IP
  10. reenajoon

    reenajoon Greenhorn

    Messages:
    72
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    23
    #10
    I completely agree with your statement. We need to follow some steps if we want to allow particular search engine to crawl our website.
     
    reenajoon, Jul 15, 2015 IP
  11. Aryans

    Aryans Well-Known Member

    Messages:
    1,854
    Likes Received:
    31
    Best Answers:
    1
    Trophy Points:
    178
    #11
    If you didn't commend anything to Robots.txt then it ll didn't index your website?? Well one thing I must want to clear here we are using robots.txt when we want to restrict any page or folder from search engine, If you didn't do any commend to robots then it ll automatically crawl your complete website no need add any code for this.
     
    Aryans, Jul 15, 2015 IP
  12. abinjoy

    abinjoy Member

    Messages:
    110
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #12
    it block the crwalers
     
    abinjoy, Jul 16, 2015 IP
  13. sheyi

    sheyi Greenhorn

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #13
    How effective is thisthis abinjoy?
     
    sheyi, Jul 16, 2015 IP
  14. sheyi

    sheyi Greenhorn

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #14
    How will it look like pls?
     
    sheyi, Jul 16, 2015 IP
  15. seojesica

    seojesica Greenhorn

    Messages:
    44
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #15

    Thanks, Got It....
     
    seojesica, Jul 16, 2015 IP
  16. reenajoon

    reenajoon Greenhorn

    Messages:
    72
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    23
    #16
    Aryans, I think you didn't read my reply seriously. I was saying that if you want to allow particular search engine to crawl your website, then you need to mention the search engine name in your robots.txt file. For an example, in robots file we use the * star for allowing all search engines, but for a particular search engine you need to mention the search engine boot name at the place of * star. Hope you understand my point of views, and thanks for considering my comment dude.
     
    reenajoon, Jul 16, 2015 IP
  17. rsrikanth05

    rsrikanth05 Well-Known Member

    Messages:
    1,362
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    190
    #17
    My robots.txt only specifies which directories shouldn't be crawled and points to my Sitemap.
    That's all that is needed to be honest. Majority of traffic today comes from Google. Google will index your site, unless you explicitly tell Google to de-index it. Bing will be the second to crawl it, it would take a few weeks, but once that is done, all is well.
     
    rsrikanth05, Jul 17, 2015 IP