robots.txt

Find help with setting up your robots.txt file on your web site.

Page 1 of 19
Sort by:
Title Start Date
Replies Views ↓
Last Message
  1. seoworld1234

    How to check robots.txt on site?

    seoworld1234, Apr 21, 2008 ... 2
    Replies:
    33
    Views:
    85,437
    imranbashir.seo
    Mar 7, 2023
  2. danieloffice

    how to block all other BOT except Google, MSN, Yahoo

    danieloffice, Jan 9, 2008 ... 2
    Replies:
    23
    Views:
    70,046
    moneyroad
    Jul 10, 2015
  3. hmansfield

    List of Bad Bots to disallow?

    hmansfield, May 20, 2007
    Replies:
    14
    Views:
    25,843
    Sergio Minozzi
    Feb 22, 2017
  4. jack_sparrow

    Help needed - robts.txt and sitemap

    jack_sparrow, Aug 3, 2007 ... 3 4 5
    Replies:
    89
    Views:
    22,888
    kevoh99
    May 25, 2010
  5. Ralarash

    Robots txt file permission

    Ralarash, Apr 30, 2010
    Replies:
    9
    Views:
    22,303
    manish.chauhan
    Jun 11, 2010
  6. latoya

    How to hide robots.txt from visitors

    latoya, Aug 23, 2008
    Replies:
    16
    Views:
    21,513
    mrgroove
    Apr 1, 2010
  7. Mysmasken

    robots.txt to deny all spiders for proxy?

    Mysmasken, Nov 29, 2007
    Replies:
    3
    Views:
    21,000
    Mysmasken
    Nov 30, 2007
  8. Angelabailey

    Robots.txt

    Angelabailey, Nov 15, 2013
    Replies:
    15
    Views:
    20,405
    Panda123
    Mar 27, 2016
  9. blabla11

    Name of yahoo, google and msn bot

    blabla11, Aug 8, 2007
    Replies:
    9
    Views:
    19,926
    seoworld
    Sep 11, 2007
  10. pradip200

    Robots.txt for SMF Forum

    pradip200, Mar 23, 2013
    Replies:
    4
    Views:
    19,705
    Svoboda
    Jun 30, 2017
  11. aleale

    Is my robots.txt working fine?

    aleale, Sep 10, 2014
    Replies:
    2
    Views:
    19,479
    maddenitrous
    Oct 3, 2014
  12. Marker1982

    Very Strange Problem

    Marker1982, Jul 21, 2015
    Replies:
    3
    Views:
    19,440
    Jacob.k
    Jul 25, 2015
  13. MarcL

    Fix to ban google search from Cloaking and/or sneaky redirects

    MarcL, Jul 30, 2015
    Replies:
    4
    Views:
    19,317
    MarcL
    Aug 3, 2015
  14. Googles76

    robots.txt for phpBB forum

    Googles76, Dec 3, 2005
    Replies:
    18
    Views:
    19,308
    linux-lover
    May 14, 2009
  15. enikram

    How to control when a bot(yahoo,google) can crawl a website

    enikram, Mar 18, 2015
    Replies:
    4
    Views:
    18,756
    Nigel Lew
    Mar 18, 2015
  16. MoneyPlanters

    Why cant Google access my robots.txt ?????

    MoneyPlanters, Sep 27, 2015
    Replies:
    1
    Views:
    18,692
    Mit Thakkar
    Sep 27, 2015
  17. HomeRun

    Advantages of robots.txt

    HomeRun, Feb 24, 2008 ... 2
    Replies:
    26
    Views:
    18,131
    agitetech
    Dec 27, 2012
  18. Minesh Parikh

    Blocked Url's by robots.txt in google webmaster tool how to fix it?

    Minesh Parikh, Apr 6, 2015
    Replies:
    0
    Views:
    17,931
    Minesh Parikh
    Apr 6, 2015
  19. netboy

    Remove this stupid robots.txt

    netboy, Feb 22, 2008
    Replies:
    17
    Views:
    17,782
    Rituja
    Feb 15, 2012
  20. Jeffr2014

    Do you block Majestic bots?

    Jeffr2014, Apr 9, 2015
    Replies:
    4
    Views:
    17,271
    neteater
    Feb 20, 2016
  21. Ratman2050

    Is 32,000 Alexa Rank Good?

    Ratman2050, Oct 8, 2006 ... 2
    Replies:
    35
    Views:
    17,148
    icyfiren
    Aug 26, 2011
  22. Kushal Biswas

    Robots.txt Help to Block a Particular File Name

    Kushal Biswas, Feb 3, 2016
    Replies:
    1
    Views:
    17,120
    Anubhav-soin
    Feb 25, 2016
  23. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    16,951
    danieloffice
    Jan 9, 2008
  24. Jaden Alvin

    Webmaster Blocked URL's Understanding Issue

    Jaden Alvin, Jul 9, 2014
    Replies:
    19
    Views:
    16,909
    Jaden Alvin
    Nov 24, 2014
  25. deluxdon

    Twiceler wont stop even when banned from robots.txt

    deluxdon, Jun 24, 2007 ... 2
    Replies:
    24
    Views:
    15,581
    elladrone
    Dec 28, 2009
  26. sck

    spiders.text & robot.text

    sck, Aug 11, 2006
    Replies:
    3
    Views:
    15,536
    explorer
    Sep 15, 2006
  27. Mrinal_Roy

    Robots.txt is blocking my labels

    Mrinal_Roy, Feb 5, 2015
    Replies:
    5
    Views:
    15,417
    Gideon Duah
    Jul 21, 2021
  28. web_master

    i can't stop Baidu, pls help!!!

    web_master, Dec 27, 2011
    Replies:
    7
    Views:
    14,938
    RoundShots
    Sep 11, 2015
  29. EducationLinks

    How to block archive.org bot with both robots.txt AND .htaccess

    EducationLinks, Aug 9, 2007
    Replies:
    6
    Views:
    14,841
    hans
    Oct 21, 2007
  30. Abdullah adil
    Locked

    robot.txt

    Abdullah adil, Feb 11, 2017
    Replies:
    6
    Views:
    14,726
    sarahk
    Feb 16, 2017
  31. myfastrupee

    Robot.txt file not found on my blog

    myfastrupee, Jun 21, 2013
    Replies:
    4
    Views:
    14,628
    meethi jain
    Jun 3, 2015
  32. wartes

    Robot.txt for Google (error)

    wartes, Apr 10, 2017
    Replies:
    6
    Views:
    14,243
    Md.Saddam Hossain
    May 7, 2017
  33. amanamission

    How do I write my robots.txt to exclude SSL pages only?

    amanamission, Sep 18, 2007
    Replies:
    5
    Views:
    14,168
    Webnauts
    Dec 5, 2008
  34. consultsoft

    Disallow subdirectory in robots.txt

    consultsoft, Dec 24, 2008
    Replies:
    4
    Views:
    13,846
    manish.chauhan
    Jan 20, 2009
  35. Jon12345

    What bots ignore robots.txt?

    Jon12345, Aug 16, 2005
    Replies:
    3
    Views:
    13,346
    minstrel
    Oct 15, 2005
  36. iluvm

    Robot.txt file

    iluvm, Nov 30, 2005 ... 2
    Replies:
    20
    Views:
    12,555
    Emily Stephens
    Jan 3, 2012
  37. gree124

    robots.txt please check

    gree124, Nov 27, 2014
    Replies:
    7
    Views:
    12,165
    rajendra1682
    Oct 6, 2015
  38. Prasseo

    How to create a robots file for PHP webpage to crawl

    Prasseo, Jan 12, 2015
    Replies:
    3
    Views:
    12,092
    abinjoy
    Jul 14, 2015
  39. angellica2017

    Dangerous Crawler and User Agent! Must Read!

    angellica2017, Sep 19, 2006 ... 2
    Replies:
    22
    Views:
    11,689
    manish.chauhan
    Apr 6, 2008
  40. prabhjot.singh

    How to Remove 404 Error?

    prabhjot.singh, Aug 26, 2012
    Replies:
    18
    Views:
    11,665
    Rakeshraj
    Jul 15, 2013
  41. scott2342

    How do I unblock my robots.txt

    scott2342, Nov 24, 2014
    Replies:
    3
    Views:
    11,526
    meethi jain
    Jun 3, 2015
  42. aj845

    unknown bots.

    aj845, Jul 6, 2007
    Replies:
    12
    Views:
    11,363
    cquinndesign
    Jul 3, 2011
  43. eniva

    Best Robots.txt For Wordpress Installed Blogs

    eniva, Sep 23, 2008
    Replies:
    12
    Views:
    10,985
    seo.lover
    Aug 29, 2010
  44. rob587

    Would a robot.txt fix this SEO issue?

    rob587, May 10, 2013
    Replies:
    19
    Views:
    10,980
    msjaved
    Nov 8, 2013
  45. offsitenoc

    How to Download google robot.txt file?

    offsitenoc, Jul 26, 2010
    Replies:
    3
    Views:
    10,940
    -[z]-
    Nov 7, 2012
  46. style fashion

    Best Robots.txt For Wordpress Blog ....

    style fashion, Oct 1, 2009
    Replies:
    13
    Views:
    10,698
    mohisha
    May 10, 2011
  47. mariush

    Need advice on how to block certain page using robots.txt

    mariush, Aug 12, 2006
    Replies:
    7
    Views:
    10,493
    learn2success
    Jun 6, 2012
  48. nikmahajan

    Blogger Robots.txt Problem- Google is not crawling

    nikmahajan, Aug 5, 2008
    Replies:
    8
    Views:
    10,417
    g4ngst3r
    Sep 25, 2011
  49. full_time_writer

    Robot text and partial fetch issues - need suggestions

    full_time_writer, Jul 28, 2015
    Replies:
    0
    Views:
    10,058
    full_time_writer
    Jul 28, 2015
  50. linkladdy

    Error - URLs restricted by robots.txt errors How Fix IT

    linkladdy, Jul 2, 2014
    Replies:
    1
    Views:
    9,983
    evildon
    Jul 2, 2014
Showing threads 1 to 50 of 911

Thread Display Options

Loading...
Page 1 of 19