Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 1 of 18
Sort by:
Title Start Date
Replies Views ↓
Last Message
  1. seoworld1234

    How to check robots.txt on site?

    seoworld1234, Apr 21, 2008 ... 2
    Replies:
    29
    Views:
    65,166
    Stephen Thomas
    May 19, 2017
  2. danieloffice

    how to block all other BOT except Google, MSN, Yahoo

    danieloffice, Jan 9, 2008 ... 2
    Replies:
    23
    Views:
    52,835
    moneyroad
    Jul 10, 2015
  3. jack_sparrow

    Help needed - robts.txt and sitemap

    jack_sparrow, Aug 3, 2007 ... 3 4 5
    Replies:
    89
    Views:
    19,761
    kevoh99
    May 25, 2010
  4. Mysmasken

    robots.txt to deny all spiders for proxy?

    Mysmasken, Nov 29, 2007
    Replies:
    3
    Views:
    19,561
    Mysmasken
    Nov 30, 2007
  5. blabla11

    Name of yahoo, google and msn bot

    blabla11, Aug 8, 2007
    Replies:
    9
    Views:
    17,847
    seoworld
    Sep 11, 2007
  6. aleale

    Is my robots.txt working fine?

    aleale, Sep 10, 2014
    Replies:
    2
    Views:
    17,561
    maddenitrous
    Oct 3, 2014
  7. hmansfield

    List of Bad Bots to disallow?

    hmansfield, May 20, 2007
    Replies:
    14
    Views:
    17,345
    Sergio Minozzi
    Feb 22, 2017
  8. netboy

    Remove this stupid robots.txt

    netboy, Feb 22, 2008
    Replies:
    17
    Views:
    16,984
    Rituja
    Feb 15, 2012
  9. Googles76

    robots.txt for phpBB forum

    Googles76, Dec 3, 2005
    Replies:
    18
    Views:
    16,951
    linux-lover
    May 14, 2009
  10. Ralarash

    Robots txt file permission

    Ralarash, Apr 30, 2010
    Replies:
    9
    Views:
    16,233
    manish.chauhan
    Jun 11, 2010
  11. latoya

    How to hide robots.txt from visitors

    latoya, Aug 23, 2008
    Replies:
    16
    Views:
    16,145
    mrgroove
    Apr 1, 2010
  12. enikram

    How to control when a bot(yahoo,google) can crawl a website

    enikram, Mar 18, 2015
    Replies:
    4
    Views:
    15,719
    Nigel Lew
    Mar 18, 2015
  13. HomeRun

    Advantages of robots.txt

    HomeRun, Feb 24, 2008 ... 2
    Replies:
    26
    Views:
    15,676
    agitetech
    Dec 27, 2012
  14. Ratman2050

    Is 32,000 Alexa Rank Good?

    Ratman2050, Oct 8, 2006 ... 2
    Replies:
    35
    Views:
    15,664
    icyfiren
    Aug 26, 2011
  15. Jaden Alvin

    Webmaster Blocked URL's Understanding Issue

    Jaden Alvin, Jul 9, 2014
    Replies:
    19
    Views:
    15,455
    Jaden Alvin
    Nov 24, 2014
  16. sck

    spiders.text & robot.text

    sck, Aug 11, 2006
    Replies:
    3
    Views:
    14,961
    explorer
    Sep 15, 2006
  17. Angelabailey

    Robots.txt

    Angelabailey, Nov 15, 2013
    Replies:
    15
    Views:
    14,860
    Panda123
    Mar 27, 2016
  18. Minesh Parikh

    Blocked Url's by robots.txt in google webmaster tool how to fix it?

    Minesh Parikh, Apr 6, 2015
    Replies:
    0
    Views:
    14,728
    Minesh Parikh
    Apr 6, 2015
  19. MarcL

    Fix to ban google search from Cloaking and/or sneaky redirects

    MarcL, Jul 30, 2015
    Replies:
    4
    Views:
    14,281
    MarcL
    Aug 3, 2015
  20. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    14,106
    danieloffice
    Jan 9, 2008
  21. deluxdon

    Twiceler wont stop even when banned from robots.txt

    deluxdon, Jun 24, 2007 ... 2
    Replies:
    24
    Views:
    13,857
    elladrone
    Dec 28, 2009
  22. EducationLinks

    How to block archive.org bot with both robots.txt AND .htaccess

    EducationLinks, Aug 9, 2007
    Replies:
    6
    Views:
    13,787
    hans
    Oct 21, 2007
  23. amanamission

    How do I write my robots.txt to exclude SSL pages only?

    amanamission, Sep 18, 2007
    Replies:
    5
    Views:
    13,267
    Webnauts
    Dec 5, 2008
  24. consultsoft

    Disallow subdirectory in robots.txt

    consultsoft, Dec 24, 2008
    Replies:
    4
    Views:
    13,154
    manish.chauhan
    Jan 20, 2009
  25. pradip200

    Robots.txt for SMF Forum

    pradip200, Mar 23, 2013
    Replies:
    4
    Views:
    13,132
    Svoboda
    Jun 30, 2017
  26. Marker1982

    Very Strange Problem

    Marker1982, Jul 21, 2015
    Replies:
    3
    Views:
    12,998
    Jacob.k
    Jul 25, 2015
  27. MoneyPlanters

    Why cant Google access my robots.txt ?????

    MoneyPlanters, Sep 27, 2015
    Replies:
    1
    Views:
    12,236
    Mit Thakkar
    Sep 27, 2015
  28. Jon12345

    What bots ignore robots.txt?

    Jon12345, Aug 16, 2005
    Replies:
    3
    Views:
    12,168
    minstrel
    Oct 15, 2005
  29. iluvm

    Robot.txt file

    iluvm, Nov 30, 2005 ... 2
    Replies:
    20
    Views:
    11,511
    Emily Stephens
    Jan 3, 2012
  30. Jeffr2014

    Do you block Majestic bots?

    Jeffr2014, Apr 9, 2015
    Replies:
    4
    Views:
    11,295
    neteater
    Feb 20, 2016
  31. Kushal Biswas

    Robots.txt Help to Block a Particular File Name

    Kushal Biswas, Feb 3, 2016
    Replies:
    1
    Views:
    10,954
    Anubhav-soin
    Feb 25, 2016
  32. prabhjot.singh

    How to Remove 404 Error?

    prabhjot.singh, Aug 26, 2012
    Replies:
    18
    Views:
    10,416
    Rakeshraj
    Jul 15, 2013
  33. eniva

    Best Robots.txt For Wordpress Installed Blogs

    eniva, Sep 23, 2008
    Replies:
    12
    Views:
    10,321
    seo.lover
    Aug 29, 2010
  34. aj845

    unknown bots.

    aj845, Jul 6, 2007
    Replies:
    12
    Views:
    10,312
    cquinndesign
    Jul 3, 2011
  35. offsitenoc

    How to Download google robot.txt file?

    offsitenoc, Jul 26, 2010
    Replies:
    3
    Views:
    10,228
    -[z]-
    Nov 7, 2012
  36. web_master

    i can't stop Baidu, pls help!!!

    web_master, Dec 27, 2011
    Replies:
    7
    Views:
    9,971
    RoundShots
    Sep 11, 2015
  37. rob587

    Would a robot.txt fix this SEO issue?

    rob587, May 10, 2013
    Replies:
    19
    Views:
    9,951
    msjaved
    Nov 8, 2013
  38. mariush

    Need advice on how to block certain page using robots.txt

    mariush, Aug 12, 2006
    Replies:
    7
    Views:
    9,843
    learn2success
    Jun 6, 2012
  39. myfastrupee

    Robot.txt file not found on my blog

    myfastrupee, Jun 21, 2013
    Replies:
    4
    Views:
    9,842
    meethi jain
    Jun 3, 2015
  40. angellica2017

    Dangerous Crawler and User Agent! Must Read!

    angellica2017, Sep 19, 2006 ... 2
    Replies:
    22
    Views:
    9,820
    manish.chauhan
    Apr 6, 2008
  41. style fashion

    Best Robots.txt For Wordpress Blog ....

    style fashion, Oct 1, 2009
    Replies:
    13
    Views:
    9,733
    mohisha
    May 10, 2011
  42. nikmahajan

    Blogger Robots.txt Problem- Google is not crawling

    nikmahajan, Aug 5, 2008
    Replies:
    8
    Views:
    9,638
    g4ngst3r
    Sep 25, 2011
  43. Sunil Keshari

    Robots.txt code

    Sunil Keshari, Jan 9, 2014
    Replies:
    3
    Views:
    9,510
    ashokdhan
    Jan 14, 2014
  44. David Lowe

    Can I add specific urls to robots.txt to stop google indexing them?

    David Lowe, Jun 25, 2013
    Replies:
    2
    Views:
    9,398
    BabyDoll123
    Jun 19, 2014
  45. linkladdy

    Error - URLs restricted by robots.txt errors How Fix IT

    linkladdy, Jul 2, 2014
    Replies:
    1
    Views:
    9,203
    evildon
    Jul 2, 2014
  46. BabyDoll123

    Magento Robots.txt file

    BabyDoll123, Jun 12, 2014
    Replies:
    3
    Views:
    9,156
    Nigel Lew
    Jun 14, 2014
  47. James101

    What is robot.txt file?

    James101, Apr 1, 2014
    Replies:
    4
    Views:
    8,959
    John Dave
    Apr 2, 2014
  48. SebastienD

    Dissalow folder with the same name as allowed folder

    SebastienD, Jan 15, 2014
    Replies:
    2
    Views:
    8,865
    Nette
    Feb 2, 2014
  49. gvinaykumar

    How to block a specific folder in robots.txt file?

    gvinaykumar, Jul 17, 2014
    Replies:
    0
    Views:
    8,834
    gvinaykumar
    Jul 17, 2014
  50. amedhussaini

    should i disallow images?

    amedhussaini, Dec 1, 2006
    Replies:
    14
    Views:
    8,642
    sarpras2005@yahoo.co.in
    Aug 11, 2009
Showing threads 1 to 50 of 900

Thread Display Options

Loading...
Page 1 of 18