1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robot.txt file

Discussion in 'Search Engine Optimization' started by seovinod86, Apr 29, 2014.

0
  1. #1
    how robot text file is create in onpage seo
     
    seovinod86, Apr 29, 2014 IP
  2. iwebsocial

    iwebsocial Well-Known Member

    Messages:
    1,715
    Likes Received:
    69
    Best Answers:
    4
    Trophy Points:
    170
    #2
    just search for it in Google. you will find many tools to create a robots.txt file.
     
    iwebsocial, Apr 30, 2014 IP
  3. sayjohn

    sayjohn Member

    Messages:
    82
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    48
    #3
    If you want to block all the content of the website to the web crawlers
    User-agent: * Disallow: /

    If you want to access all the content of the website you do not need the robots.txt file.
     
    sayjohn, Apr 30, 2014 IP
  4. Seo Joy

    Seo Joy Active Member

    Messages:
    408
    Likes Received:
    1
    Best Answers:
    2
    Trophy Points:
    63
    #4
    Robots.txt is a simple text (not html) file you put on your website root directory. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all... for more information, you can search for it in google.
     
    Seo Joy, Apr 30, 2014 IP
  5. Martin Taylor

    Martin Taylor Active Member

    Messages:
    172
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #5
    Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
    It works likes this: a robot wants to visits a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

    User-agent: *
    Disallow: /
    The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
     
    Martin Taylor, Apr 30, 2014 IP