1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to create robots.txt file

Discussion in 'robots.txt' started by sarahashlin, Nov 28, 2011.

  1. #1
    Please suggest me tips to crate and implement the roboots.tsx file.
     
    sarahashlin, Nov 28, 2011 IP
  2. john_active

    john_active Peon

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Just create a file named robots.txt, put it on your root directory of your website(where index.htm/index.php is), edit with a text editor and add these:

    1st line:
    User-agent: *
    [The * means that you refer to all crawlers. if you want to refer to a specific crawler just replace * with its name(like "googlebot")]

    2nd and every other line:
    Disallow: *something*

    eg.
    Disallow: /
    this means that you block crawlers from all your pages. if you want to block a specific page like "contact.htm" you should put:
    Disallow: /contact.htm
    if you want to block a complete subdirectory like "/admin/" you should put:
    Disallow: /admin/

    if you want to allow everything you should type:
    Disallow:

    (Note: you can have only one rule on each line)
     
    john_active, Dec 17, 2011 IP
  3. hansimorn

    hansimorn Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Simple Rule have checking for robots.txt file go the end of url and write the roabts.txt then enter
     
    hansimorn, Dec 20, 2011 IP
  4. MarcusN

    MarcusN Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    0
    #4
    May be could be interesting to read how google treats and likes robots.txt
    I cannot post links, just learned it :), but the page can be easily found by a little search with this keywords "google robots.txt specifications"
     
    MarcusN, Dec 21, 2011 IP
  5. Icecube_media

    Icecube_media Peon

    Messages:
    656
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #5
    hi
    it depends upon which page you want to create, you can made it on webmaster tool, or on notepad .
     
    Icecube_media, Dec 21, 2011 IP
  6. JashandeepMax

    JashandeepMax Peon

    Messages:
    72
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Webmaster tool is a best way .Try to not make a practice of use in in your real life.
     
    JashandeepMax, Dec 23, 2011 IP
  7. tessori

    tessori Peon

    Messages:
    138
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Hi,

    Most of the time , it depends in what pages you want to get access to crawler & what pages you don't.
     
    tessori, Dec 23, 2011 IP
  8. tripbuilder

    tripbuilder Peon

    Messages:
    72
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    its very simple you can use robots.txt generator.
     
    tripbuilder, Dec 28, 2011 IP
  9. MessiahofMoney

    MessiahofMoney Greenhorn

    Messages:
    25
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #9
    Def best way! Well..from my experiences neways lol
     
    MessiahofMoney, Dec 30, 2011 IP
  10. WPC

    WPC Peon

    Messages:
    116
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #10
    WPC, Dec 31, 2011 IP
  11. fancy1594

    fancy1594 Peon

    Messages:
    37
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    This is my robots.txt, you can follow:
    sitemap: http://freebestarticles.net/sitemap.xml
    sitemap: http://freebestarticles.net/sitemap.xml.gz

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /go/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /author/
    Disallow: /page/
    Disallow: /wp-images/
    Disallow: /images/
    Disallow: /backup/
    Disallow: /banners/
    Disallow: /archives/
    Disallow: /trackback/


    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Mediapartners-Google
    Allow: /

    User-agent: duggmirror
    Disallow: /
     
    fancy1594, Jan 1, 2012 IP
  12. sisatel

    sisatel Active Member

    Messages:
    1,391
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    90
    #12
    sisatel, Jan 2, 2012 IP
  13. rosy

    rosy Peon

    Messages:
    89
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #13
    john you have nicely explained how to create robot.txt file. I will add up to it some thing more. After creating upload it in the root directory and then check whether you can see it.
    for eg. www.site.com/robot.txt and hit enter you can view your robot.txt file.
     
    rosy, Jan 4, 2012 IP
  14. WFgirls

    WFgirls Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    you can install KBrobots plugin on your wordpress site
     
    WFgirls, Jan 14, 2012 IP
  15. Manikanta Reddy

    Manikanta Reddy Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    Hi

    Robot txt file is for our website content or internal page not appear to search engines to crawl.

    that is the reason we are using robot txt file.
    when search engines are crawling a website they are searching for robot txt file after watching the file instructions search engines will crawl the websites.
     
    Manikanta Reddy, Jan 24, 2012 IP
  16. WPC

    WPC Peon

    Messages:
    116
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #16
    There are good scripts on the internet, just google robot generators.
     
    WPC, Jan 27, 2012 IP
  17. hunk.jaat

    hunk.jaat Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #17
    thks coz i want to also know about robot.txt
     
    hunk.jaat, Feb 5, 2012 IP
  18. lananhplus

    lananhplus Member

    Messages:
    35
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #18
    you could use/edit robots.txt in GA tool but it's not apply to your website. You have to edit file robots.txt in host. ex: seta-international.com/robots.txt.
     
    lananhplus, Feb 5, 2012 IP
  19. bobbyloin

    bobbyloin Guest

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #19
    Hi thanks for the link, that was helpful
     
    bobbyloin, Feb 7, 2012 IP
  20. nitishaddict

    nitishaddict Peon

    Messages:
    35
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #20
    Hi thanks for the information. Now i got my answer about robot.txt file.
     
    nitishaddict, Feb 11, 2012 IP