Robots.txt

Discussion in 'Google Sitemaps' started by Bombaywala, Dec 11, 2006.

0
  1. #1
    How do I tell GBot not to visit any pages that has:

    domain.com/ezineready.php?......

    What should I put in my robots.txt for this?
     
    Bombaywala, Dec 11, 2006 IP
  2. Darkhodge

    Darkhodge Well-Known Member

    Messages:
    2,111
    Likes Received:
    76
    Best Answers:
    1
    Trophy Points:
    185
    #2
    You can create a robot.txt and put this code into it. It's set to stop all spiders from crawling ezineready.php:

    User-Agent: *
Disallow: /ezineready.php
    Code (markup):
    If that's not the only URL you want to stop robots crawling or you want the robot.txt to specifically exclude Google bots, you could use this online tool : Robot.txt Generator

    P.S. Sitemaps and robot.txt are 2 different things btw... :eek:
     
    Darkhodge, Dec 11, 2006 IP
  3. Bombaywala

    Bombaywala Peon

    Messages:
    1,249
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks maan. I hear ya - Sitemaps and robot.txt - two different things - but after looking under Google Sitemaps stats - I saw it showing domain/ezineready.php which I didn't want to show up.

    Thanks again.
     
    Bombaywala, Dec 11, 2006 IP
  4. Darkhodge

    Darkhodge Well-Known Member

    Messages:
    2,111
    Likes Received:
    76
    Best Answers:
    1
    Trophy Points:
    185
    #4
    No probs - glad I could help :)
     
    Darkhodge, Dec 11, 2006 IP