Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 1 of 19
Sort by:
Title Start Date
Replies ↓ Views
Last Message
  1. jack_sparrow

    Help needed - robts.txt and sitemap

    jack_sparrow, Aug 3, 2007 ... 3 4 5
    Replies:
    89
    Views:
    19,940
    kevoh99
    May 25, 2010
  2. fryman

    is a blank robots file ok?

    fryman, Jul 13, 2005 ... 2 3
    Replies:
    40
    Views:
    4,547
    minstrel
    Oct 14, 2005
  3. NewComputer

    robots.txt pointing to index page

    NewComputer, Sep 26, 2004 ... 2
    Replies:
    39
    Views:
    5,230
    Blogmaster
    Apr 19, 2005
  4. Ratman2050

    Is 32,000 Alexa Rank Good?

    Ratman2050, Oct 8, 2006 ... 2
    Replies:
    35
    Views:
    15,748
    icyfiren
    Aug 26, 2011
  5. Edz

    The most ideal robots.txt? how should it look like?

    Edz, Nov 24, 2005 ... 2
    Replies:
    34
    Views:
    5,896
    minstrel
    Dec 10, 2005
  6. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    14,240
    danieloffice
    Jan 9, 2008
  7. alka007

    What is Robots.txt ?

    alka007, Feb 23, 2012 ... 2
    Replies:
    32
    Views:
    7,218
    sahostking
    Nov 22, 2012
  8. VijayanKumar

    What Is Robots.txt?

    VijayanKumar, Jun 9, 2011 ... 2
    Replies:
    32
    Views:
    7,966
    webgfx
    Sep 14, 2011
  9. seoworld1234

    How to check robots.txt on site?

    seoworld1234, Apr 21, 2008 ... 2
    Replies:
    29
    Views:
    66,529
    Stephen Thomas
    May 19, 2017
  10. j3r0m3

    Have robot.txt. but always 404

    j3r0m3, Mar 23, 2006 ... 2
    Replies:
    27
    Views:
    3,539
    ryan_uk
    Apr 3, 2006
  11. HomeRun

    Advantages of robots.txt

    HomeRun, Feb 24, 2008 ... 2
    Replies:
    26
    Views:
    15,796
    agitetech
    Dec 27, 2012
  12. skionxb

    Wow is ROBOTS.txt important?

    skionxb, Jul 26, 2006 ... 2
    Replies:
    26
    Views:
    8,016
    Dan_Grr
    Jan 12, 2007
  13. weathor

    Too many googlebot hits

    weathor, Apr 6, 2010 ... 2
    Replies:
    25
    Views:
    4,847
    kevoh99
    May 25, 2010
  14. techfreakd

    How to disallow Google crawl link.???

    techfreakd, May 18, 2012 ... 2
    Replies:
    24
    Views:
    7,194
    Max796
    Oct 10, 2012
  15. Woodcs82

    Is Robots.txt better than Robots in META?

    Woodcs82, Jun 10, 2010 ... 2
    Replies:
    24
    Views:
    5,119
    fsdnetwork
    Feb 9, 2011
  16. deluxdon

    Twiceler wont stop even when banned from robots.txt

    deluxdon, Jun 24, 2007 ... 2
    Replies:
    24
    Views:
    13,959
    elladrone
    Dec 28, 2009
  17. danieloffice

    how to block all other BOT except Google, MSN, Yahoo

    danieloffice, Jan 9, 2008 ... 2
    Replies:
    23
    Views:
    54,606
    moneyroad
    Jul 10, 2015
  18. jnelson563

    Help with blocking affiliate URLs

    jnelson563, Jun 17, 2010 ... 2
    Replies:
    23
    Views:
    2,501
    manish.chauhan
    Jul 12, 2010
  19. poseidon

    How to create robot.txt file ? how to stop indexing files from cgi-bin ?

    poseidon, Jan 23, 2006 ... 2
    Replies:
    23
    Views:
    8,237
    sgtcory
    Jun 14, 2009
  20. amitpatel_3001

    Is this Gud/Bad??? Google Bot eats 330MB/Day Bandwidth

    amitpatel_3001, Dec 9, 2005 ... 2
    Replies:
    23
    Views:
    6,482
    amitpatel_3001
    Feb 23, 2007
  21. sarahashlin

    How to create robots.txt file

    sarahashlin, Nov 28, 2011 ... 2
    Replies:
    22
    Views:
    6,330
    madison37
    Mar 6, 2012
  22. PuneetJvw

    How bad is it to NOT have a robots.txt file

    PuneetJvw, Jan 16, 2010 ... 2
    Replies:
    22
    Views:
    4,542
    smartamit04
    Aug 4, 2010
  23. Algore

    Is My Robot.txt Okay ???

    Algore, Oct 20, 2008 ... 2
    Replies:
    22
    Views:
    5,179
    buldozerceto
    Feb 19, 2009
  24. angellica2017

    Dangerous Crawler and User Agent! Must Read!

    angellica2017, Sep 19, 2006 ... 2
    Replies:
    22
    Views:
    9,936
    manish.chauhan
    Apr 6, 2008
  25. lim (x² - 5x³) = -∞

    robots.txt - How to ban everyone except Google

    lim (x² - 5x³) = -∞, Mar 27, 2008 ... 2
    Replies:
    21
    Views:
    3,380
    dermax
    Jun 2, 2008
  26. Yukio

    Should I deny access to my site forum using googlebot?

    Yukio, Jun 21, 2005 ... 2
    Replies:
    21
    Views:
    4,211
    nevetS
    Jun 29, 2005
  27. iluvm

    Robot.txt file

    iluvm, Nov 30, 2005 ... 2
    Replies:
    20
    Views:
    11,583
    Emily Stephens
    Jan 3, 2012
  28. Jaden Alvin

    Webmaster Blocked URL's Understanding Issue

    Jaden Alvin, Jul 9, 2014
    Replies:
    19
    Views:
    15,652
    Jaden Alvin
    Nov 24, 2014
  29. rob587

    Would a robot.txt fix this SEO issue?

    rob587, May 10, 2013
    Replies:
    19
    Views:
    10,059
    msjaved
    Nov 8, 2013
  30. sofiaisabella01

    What is robot.txt flie?

    sofiaisabella01, Feb 19, 2012
    Replies:
    19
    Views:
    2,787
    mbitsol
    Apr 7, 2012
  31. alokseo

    Anybody tell me about robots.txt? hwo to usefull for seo?

    alokseo, Jun 18, 2010
    Replies:
    19
    Views:
    2,577
    Prateekgupta
    Oct 17, 2010
  32. Quench

    bots overloading my server everyday.

    Quench, Aug 7, 2008
    Replies:
    19
    Views:
    6,639
    Quench
    Oct 9, 2008
  33. john269

    About the search engine robot.txt file and SEO!

    john269, Aug 12, 2006
    Replies:
    19
    Views:
    2,704
    john269
    Aug 21, 2006
  34. girbaud

    robots.txt disallow

    girbaud, Jan 13, 2006
    Replies:
    19
    Views:
    3,779
    minstrel
    Mar 8, 2006
  35. Moneyfolk

    Robots.txt question

    Moneyfolk, Jan 12, 2006
    Replies:
    19
    Views:
    3,801
    ServerUnion
    Jan 13, 2006
  36. prabhjot.singh

    How to Remove 404 Error?

    prabhjot.singh, Aug 26, 2012
    Replies:
    18
    Views:
    10,500
    Rakeshraj
    Jul 15, 2013
  37. Burano

    Is my wordpress site on nofollow?

    Burano, Jan 1, 2010
    Replies:
    18
    Views:
    2,359
    john12
    Mar 16, 2010
  38. Googles76

    robots.txt for phpBB forum

    Googles76, Dec 3, 2005
    Replies:
    18
    Views:
    17,144
    linux-lover
    May 14, 2009
  39. mikelorentz

    robots.txt helps to improve your page rank

    mikelorentz, Jul 16, 2008
    Replies:
    18
    Views:
    2,149
    catanich
    Aug 21, 2008
  40. netboy

    Remove this stupid robots.txt

    netboy, Feb 22, 2008
    Replies:
    17
    Views:
    17,038
    Rituja
    Feb 15, 2012
  41. allanswagger

    Robots.txt

    allanswagger, Sep 2, 2010
    Replies:
    17
    Views:
    3,589
    manish.chauhan
    May 4, 2011
  42. mcsp

    Robots.txt question

    mcsp, Jul 26, 2005
    Replies:
    17
    Views:
    3,778
    gatordun
    Jul 29, 2005
  43. latoya

    How to hide robots.txt from visitors

    latoya, Aug 23, 2008
    Replies:
    16
    Views:
    16,573
    mrgroove
    Apr 1, 2010
  44. shailendra

    Confused about robots.txt

    shailendra, Mar 24, 2009
    Replies:
    16
    Views:
    3,167
    Aaron111
    Sep 2, 2009
  45. jallikatu

    Crawling restricted area

    jallikatu, Feb 1, 2009
    Replies:
    16
    Views:
    2,021
    manish.chauhan
    Feb 23, 2009
  46. bruce-banner

    Site with no robots.txt

    bruce-banner, Nov 14, 2008
    Replies:
    16
    Views:
    3,310
    shailendra
    Feb 19, 2009
  47. Angelabailey

    Robots.txt

    Angelabailey, Nov 15, 2013
    Replies:
    15
    Views:
    15,079
    Panda123
    Mar 27, 2016
  48. DevAdelicia

    What is Robots.txt ?

    DevAdelicia, Nov 17, 2011
    Replies:
    15
    Views:
    5,542
    xiaokaige
    Sep 18, 2012
  49. junrey24

    URL restricted by robots.txt

    junrey24, Mar 28, 2010
    Replies:
    15
    Views:
    1,995
    LookAtHerBeautifulFace
    Mar 29, 2010
  50. slayer__

    ideal robots.txt for all sites.

    slayer__, Jul 15, 2009
    Replies:
    15
    Views:
    2,760
    theapparatus
    Sep 19, 2009
Showing threads 1 to 50 of 904

Thread Display Options

Loading...
Page 1 of 19