Is this the correct way to create robots.txt

Discussion in 'Search Engine Optimization' started by jazznchew, Feb 20, 2008.

0
  1. #1
    Hi guys,

    If i want to create a robots.txt tht will allow all search engine to crawl my site at all pages, is this content correct? Pls advice....thanks man! :)

    User-agent: *
    Disallow:

    Sitemap: http://www.mysite.com/sitemap.xml
     
    jazznchew, Feb 20, 2008 IP
  2. dylanphelan

    dylanphelan Active Member

    Messages:
    86
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    91
    #2
    Yeah...

    you can specify a directory to disallow like this:

    User-agent: *
    Disallow: /directory_name
     
    dylanphelan, Feb 20, 2008 IP
  3. jazznchew

    jazznchew Peon

    Messages:
    94
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    all rite man ! i got it. i will do it now!

    By the way, wht will be the impact if i not create this robots.txt for my site, but i only submit my sitemap to google. Wht is the impact?
     
    jazznchew, Feb 20, 2008 IP
  4. abdulmalick

    abdulmalick Active Member

    Messages:
    264
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    88
    #4
    yeah man.. thats only way, suggest to spider that to crawl in webpage
     
    abdulmalick, Feb 20, 2008 IP
  5. breddg

    breddg Member

    Messages:
    10
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    38
    #5
    yes, it is perfect :)
     
    breddg, Feb 20, 2008 IP
  6. Aryans

    Aryans Well-Known Member

    Messages:
    1,854
    Likes Received:
    31
    Best Answers:
    1
    Trophy Points:
    178
    #6
    if you don't give any kind of command to robot and submit your sitemap in google then crawler will automatically crawled your all pages.
     
    Aryans, Feb 20, 2008 IP
  7. dylanphelan

    dylanphelan Active Member

    Messages:
    86
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    91
    #7
    Well; what do you actually want to do?

    you're talking about submitting a sitemap, which means that you want your pages crawled and indexed, but at the start of this thread you use a robots.txt file with a 'Disallow' command in it.
    That stops the bots from crawling certain pages, directories etc...
     
    dylanphelan, Feb 20, 2008 IP
  8. jazznchew

    jazznchew Peon

    Messages:
    94
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    no...u get me wrong. i am seeking for advice whether my robots.txt content content is correct or not? if i want search engine search all my site pages...

    Is this correct ?

    User-agent: *
    Disallow:

    Sitemap: http://www.mysite.com/sitemap.xml
     
    jazznchew, Feb 20, 2008 IP