1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Wow is ROBOTS.txt important?

Discussion in 'robots.txt' started by skionxb, Jul 26, 2006.

0
Page 2 of 2
  1. X FILES

    X FILES Peon

    Messages:
    324
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    0
    #21
    Google just index limited pages from each site everyday depending of the strongth of it.Robots.txt will be useful for the newly site to index the important pages firest
    You can also use this If you don`t want spider prevent any thing of your pages

    # Allow all
User-agent:  *
Disallow:
    Code (markup):
     
    X FILES, Dec 6, 2006 IP
  2. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #22
    you are welcome to the webmaster club ;) robots.txt is one of the key factor web site designing;)
     
    trichnosis, Dec 18, 2006 IP
  3. jacobs

    jacobs Banned

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #23
    you r right skionxb

    if u really want to disallow anything then it matters otherwise i dont think its really important..to include robots.txt
     
    jacobs, Dec 21, 2006 IP
  4. Jirsh

    Jirsh Peon

    Messages:
    34
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #24
    Would a phpBB2 forum need a robots.txt file or only html based sites? What about flash sites?
     
    Jirsh, Jan 3, 2007 IP
  5. explorer

    explorer Well-Known Member

    Messages:
    463
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    110
    #25
    Yes, you can use a robots.txt for .php pages if you want to - to keep bots out of places that you don't want to appear in SE results. If your forum is in a folder called forum, you would use a robots.txt to exclude bots from pages like avatar.php and faq.php as follows.

    User-agent: *
    Disallow: /forum/avatar.php
    Disallow: /forum/faq.php
     
    explorer, Jan 3, 2007 IP
  6. melefire

    melefire Peon

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #26
    all the files with no content should be coded out
     
    melefire, Jan 5, 2007 IP
  7. Dan_Grr

    Dan_Grr Peon

    Messages:
    72
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #27
    I don't know if anyone has posted it yet, but there is some nice information here: http://www.thesitewizard.com/archive/robotstxt.shtml
     
    Dan_Grr, Jan 12, 2007 IP
Page 2 of 2