1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

What is Robot.txt file?

Discussion in 'robots.txt' started by w3seotraining, Jul 3, 2013.

Thread Status:
Not open for further replies.
0
  1. #1
    What is Robot.txt file? if anybody have good knowledge please give answer pelase?
     
    w3seotraining, Jul 3, 2013 IP
  2. topnichewebsites

    topnichewebsites Member

    Messages:
    35
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    36
    #2
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called: The Robots Exclusion Protocol.

    It works likes this: a robot wants to visits a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt.

    Note: not all spiders aka robots will follow the inclusion / exclusions

    For example:

    User-agent: *
    Disallow /

    Would tell the robots / spiders not to index any part of a website

    This example


    User-agent: *
    Disallow /cgi-bin/
    Disallow /temp/
    Disallow /~images/

    Would tell the robots / spiders to index the whole site except cgi-bin, temp and ~images

     
    topnichewebsites, Jul 3, 2013 IP
  3. Vincentas

    Vincentas Active Member

    Messages:
    102
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #3
    You can read more about it http://en.wikipedia.org/wiki/Robots_exclusion_standard
     
    Vincentas, Jul 8, 2013 IP
Thread Status:
Not open for further replies.