1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

What does this robots.txt file do?

Discussion in 'robots.txt' started by PeterA, Jan 15, 2010.

0
  1. #1
    What does this robots.txt file do?

    User-agent: *
    Disallow:


    I was just sitting in my folder, does it do anything at all? do I need it?

    If it had the slash it would stop all spiders, yes? Like this below.

    User-agent: *
    Disallow: /

    Peter
     
    PeterA, Jan 15, 2010 IP
  2. josephwaldman1984

    josephwaldman1984 Peon

    Messages:
    306
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    User-agent: googlebot
    Disallow:/images

    If we mention above code in the robots.txt file, then it will prevent google search engine to crawl images folder.

    Similarly we can prevent spiders of different engines to crawl unwanted folders or web pages.
    Thus with the help of robots.txt file we can increase the indexing of important pages of our website.
     
    josephwaldman1984, Jan 18, 2010 IP
  3. PeterA

    PeterA Guest

    Messages:
    107
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Yes I understand that, but what does this code do?

    User-agent: *
    Disallow:

    Peter
     
    PeterA, Jan 18, 2010 IP
  4. linksale77

    linksale77 Peon

    Messages:
    43
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    User-agent: the robot the following rule applies to
    Disallow: the URL you want to block

    User-agent: *
    Google uses several different bots (user-agents). The bot we use for our web search is Googlebot. Our other bots like Googlebot-Mobile and Googlebot-Image follow rules you set up for Googlebot, but you can set up specific rules for these specific bots as well

    Disallow:
    it will not Block any folder you have on the site
     
    linksale77, Jan 18, 2010 IP
  5. kabir5800

    kabir5800 Active Member

    Messages:
    40
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #5
    It will disallow Google from crawling any of the dynamically generated pages. It will allow the
    altavista scooter bot to access
    every page
     
    kabir5800, Jul 31, 2012 IP
  6. andydxgame

    andydxgame Member

    Messages:
    117
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #6
    It's basically in place so pages you don't want Google to index won't get indexed
     
    andydxgame, Aug 3, 2012 IP