1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Should i still make a robots.txt file if i have sitemap file?

Discussion in 'Google Sitemaps' started by mapleisaac, Feb 26, 2011.

0
  1. #1
    Is robots.txt important?After I have uploaded my sitemap file, should I still make a robots.txt file?
     
    mapleisaac, Feb 26, 2011 IP
  2. gobinathdpi

    gobinathdpi Member

    Messages:
    130
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #2
    This two are different concepts and the answer is yes you require a robots.txt file even though you have sitemap file which will help robots to search properly. robots.txt is useful if you want to restrict robot entry to some of your sites pages
     
    gobinathdpi, Mar 1, 2011 IP
  3. idcatalin

    idcatalin Peon

    Messages:
    35
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Sitemap and robots.txt are very different, sitemap is made to let google discovering new pages and robots.txt is made to restrict access to some areas of your site for search engines
     
    idcatalin, Mar 2, 2011 IP
  4. artemporio

    artemporio Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    robot.txt is always require even if you have the sitemap, because sitemap is for indexing your webpages properly on search engine and through robot.txt you can define that which part of your website should be crawl or not by google bots.
     
    artemporio, Mar 2, 2011 IP
  5. jennex05

    jennex05 Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Sitemaps help googlebot find deep pages it wouldn't have found while the robots.txt file says, hey googlebot,dont crawl these files!
     
    jennex05, Mar 8, 2011 IP
  6. diogop

    diogop Peon

    Messages:
    39
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    robots and sitemap are not the same thing so yes definitely create a robots.
     
    diogop, Mar 8, 2011 IP
  7. sandraseeves

    sandraseeves Peon

    Messages:
    37
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    To further complicate the topic, there is also a robot meta tags. In my opinion, it wouldn't hurt if you are still going to use a robots.txt. If Google or other search engine said it is bad then don't use it.
     
    sandraseeves, Mar 8, 2011 IP
  8. larendaniel

    larendaniel Peon

    Messages:
    18
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Not only new pages but also old pages should also keep on getting indexed.
     
    larendaniel, Apr 12, 2011 IP
  9. will83

    will83 Peon

    Messages:
    31
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Yes i always use a robots.txt file.
     
    will83, Apr 13, 2011 IP
  10. duffers2010

    duffers2010 Peon

    Messages:
    7
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    By having the robots.txt file you can still prevent bots from using up resources on your server. Also google doesnt like crawling pages that it doesn't have to - google webmaster guidelines still state using robots.txt to restrict crawler access is advisable.
     
    duffers2010, Apr 13, 2011 IP