1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt File Format is Right or wrong?

Discussion in 'robots.txt' started by yogeshseo, Mar 6, 2012.

  1. #1
    Hello Frinds:

    This is my website.. http://www.ibazaarindia.com/robots.txt Please check and let me know this robots.txt file format is right or wrong?

    Pls suggest me if have any problem..


    Thanks
     
    yogeshseo, Mar 6, 2012 IP
  2. FoxIX

    FoxIX Well-Known Member

    Messages:
    211
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    140
    #2
    It seems a bit bloated, but I can't see anything major wrong with it - unless being that bloated is a problem. I would suggest going to Google's Webmaster Tools and where you can test your robots file.
     
    FoxIX, Mar 6, 2012 IP
  3. RobertStev

    RobertStev Peon

    Messages:
    92
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    you could also try generating it
     
    RobertStev, Mar 19, 2012 IP
  4. p.caspian

    p.caspian Peon

    Messages:
    964
    Likes Received:
    6
    Best Answers:
    1
    Trophy Points:
    0
    #4
    Are you facing any problem with that robot file? If yes then what problem are you facing? I have checked your file and I think its ok.
     
    p.caspian, Mar 21, 2012 IP
  5. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    have you tried a check on the file to check validation? my webhost does that
     
    tiffanywilliams12i2, Apr 10, 2012 IP
  6. Knocks

    Knocks Well-Known Member

    Messages:
    85
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    100
    #6
    It's too long and ugly. Use wildcards instead of listing all those entries individually.
     
    Knocks, Apr 20, 2012 IP
  7. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #7
    Instead of making a useless and meaningless post, why don't you provide useful help and explain what the problem is?

    @yogeshseo, the robots.txt is long, but I assume you have all in there that you want. You do not need:

    Allow: /
    Code (markup):
    Not all robots are equal, so best remove it to avoid any of them getting confused and crawling areas you don't want them to.

    You might want to make use of the robots meta tag for pages you're wanting to specifically exclude from being crawled.

    However, I noticed some of these pages you're blocking with robots.txt don't exist any more (I didn't check all of them). This is wrong really. Instead of blocking them, you should redirect to an alternative product, category or your home using .htaccess and a 301. I'm guessing you used robots.txt to block these URLs due to crawl errors. The cleaner way is the 301 redirect, plus any backlinks you have to these pages are then not lost and still directing traffic to other relevant pages (or your home page).

    Good luck.
     
    ryan_uk, Aug 5, 2012 IP
  8. Renatus

    Renatus Member

    Messages:
    216
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    35
    #8
    Looks okay but too many "disallow" /products/ and that makes it bloated. Wouldn't it be easier to simply write one line:

    
    Disallow: /products
    
    Code (markup):
     
    Renatus, Dec 3, 2012 IP
    ryan_uk likes this.