Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 1 of 19
Sort by:
Title Start Date
Replies ↓ Views
Last Message
  1. jack_sparrow

    Help needed - robts.txt and sitemap

    jack_sparrow, Aug 3, 2007 ... 3 4 5
    Replies:
    89
    Views:
    19,821
    kevoh99
    May 25, 2010
  2. fryman

    is a blank robots file ok?

    fryman, Jul 13, 2005 ... 2 3
    Replies:
    40
    Views:
    4,480
    minstrel
    Oct 14, 2005
  3. NewComputer

    robots.txt pointing to index page

    NewComputer, Sep 26, 2004 ... 2
    Replies:
    39
    Views:
    5,176
    Blogmaster
    Apr 19, 2005
  4. Ratman2050

    Is 32,000 Alexa Rank Good?

    Ratman2050, Oct 8, 2006 ... 2
    Replies:
    35
    Views:
    15,688
    icyfiren
    Aug 26, 2011
  5. Edz

    The most ideal robots.txt? how should it look like?

    Edz, Nov 24, 2005 ... 2
    Replies:
    34
    Views:
    5,831
    minstrel
    Dec 10, 2005
  6. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    14,137
    danieloffice
    Jan 9, 2008
  7. alka007

    What is Robots.txt ?

    alka007, Feb 23, 2012 ... 2
    Replies:
    32
    Views:
    7,156
    sahostking
    Nov 22, 2012
  8. VijayanKumar

    What Is Robots.txt?

    VijayanKumar, Jun 9, 2011 ... 2
    Replies:
    32
    Views:
    7,928
    webgfx
    Sep 14, 2011
  9. seoworld1234

    How to check robots.txt on site?

    seoworld1234, Apr 21, 2008 ... 2
    Replies:
    29
    Views:
    65,650
    Stephen Thomas
    May 19, 2017
  10. j3r0m3

    Have robot.txt. but always 404

    j3r0m3, Mar 23, 2006 ... 2
    Replies:
    27
    Views:
    3,490
    ryan_uk
    Apr 3, 2006
  11. HomeRun

    Advantages of robots.txt

    HomeRun, Feb 24, 2008 ... 2
    Replies:
    26
    Views:
    15,715
    agitetech
    Dec 27, 2012
  12. skionxb

    Wow is ROBOTS.txt important?

    skionxb, Jul 26, 2006 ... 2
    Replies:
    26
    Views:
    7,966
    Dan_Grr
    Jan 12, 2007
  13. weathor

    Too many googlebot hits

    weathor, Apr 6, 2010 ... 2
    Replies:
    25
    Views:
    4,760
    kevoh99
    May 25, 2010
  14. techfreakd

    How to disallow Google crawl link.???

    techfreakd, May 18, 2012 ... 2
    Replies:
    24
    Views:
    7,129
    Max796
    Oct 10, 2012
  15. Woodcs82

    Is Robots.txt better than Robots in META?

    Woodcs82, Jun 10, 2010 ... 2
    Replies:
    24
    Views:
    5,054
    fsdnetwork
    Feb 9, 2011
  16. deluxdon

    Twiceler wont stop even when banned from robots.txt

    deluxdon, Jun 24, 2007 ... 2
    Replies:
    24
    Views:
    13,895
    elladrone
    Dec 28, 2009
  17. danieloffice

    how to block all other BOT except Google, MSN, Yahoo

    danieloffice, Jan 9, 2008 ... 2
    Replies:
    23
    Views:
    53,431
    moneyroad
    Jul 10, 2015
  18. jnelson563

    Help with blocking affiliate URLs

    jnelson563, Jun 17, 2010 ... 2
    Replies:
    23
    Views:
    2,450
    manish.chauhan
    Jul 12, 2010
  19. poseidon

    How to create robot.txt file ? how to stop indexing files from cgi-bin ?

    poseidon, Jan 23, 2006 ... 2
    Replies:
    23
    Views:
    8,161
    sgtcory
    Jun 14, 2009
  20. amitpatel_3001

    Is this Gud/Bad??? Google Bot eats 330MB/Day Bandwidth

    amitpatel_3001, Dec 9, 2005 ... 2
    Replies:
    23
    Views:
    6,443
    amitpatel_3001
    Feb 23, 2007
  21. sarahashlin

    How to create robots.txt file

    sarahashlin, Nov 28, 2011 ... 2
    Replies:
    22
    Views:
    6,273
    madison37
    Mar 6, 2012
  22. PuneetJvw

    How bad is it to NOT have a robots.txt file

    PuneetJvw, Jan 16, 2010 ... 2
    Replies:
    22
    Views:
    4,496
    smartamit04
    Aug 4, 2010
  23. Algore

    Is My Robot.txt Okay ???

    Algore, Oct 20, 2008 ... 2
    Replies:
    22
    Views:
    5,127
    buldozerceto
    Feb 19, 2009
  24. angellica2017

    Dangerous Crawler and User Agent! Must Read!

    angellica2017, Sep 19, 2006 ... 2
    Replies:
    22
    Views:
    9,862
    manish.chauhan
    Apr 6, 2008
  25. lim (x² - 5x³) = -∞

    robots.txt - How to ban everyone except Google

    lim (x² - 5x³) = -∞, Mar 27, 2008 ... 2
    Replies:
    21
    Views:
    3,341
    dermax
    Jun 2, 2008
  26. Yukio

    Should I deny access to my site forum using googlebot?

    Yukio, Jun 21, 2005 ... 2
    Replies:
    21
    Views:
    4,163
    nevetS
    Jun 29, 2005
  27. iluvm

    Robot.txt file

    iluvm, Nov 30, 2005 ... 2
    Replies:
    20
    Views:
    11,533
    Emily Stephens
    Jan 3, 2012
  28. Jaden Alvin

    Webmaster Blocked URL's Understanding Issue

    Jaden Alvin, Jul 9, 2014
    Replies:
    19
    Views:
    15,495
    Jaden Alvin
    Nov 24, 2014
  29. rob587

    Would a robot.txt fix this SEO issue?

    rob587, May 10, 2013
    Replies:
    19
    Views:
    9,975
    msjaved
    Nov 8, 2013
  30. sofiaisabella01

    What is robot.txt flie?

    sofiaisabella01, Feb 19, 2012
    Replies:
    19
    Views:
    2,760
    mbitsol
    Apr 7, 2012
  31. alokseo

    Anybody tell me about robots.txt? hwo to usefull for seo?

    alokseo, Jun 18, 2010
    Replies:
    19
    Views:
    2,539
    Prateekgupta
    Oct 17, 2010
  32. Quench

    bots overloading my server everyday.

    Quench, Aug 7, 2008
    Replies:
    19
    Views:
    6,603
    Quench
    Oct 9, 2008
  33. john269

    About the search engine robot.txt file and SEO!

    john269, Aug 12, 2006
    Replies:
    19
    Views:
    2,675
    john269
    Aug 21, 2006
  34. girbaud

    robots.txt disallow

    girbaud, Jan 13, 2006
    Replies:
    19
    Views:
    3,749
    minstrel
    Mar 8, 2006
  35. Moneyfolk

    Robots.txt question

    Moneyfolk, Jan 12, 2006
    Replies:
    19
    Views:
    3,763
    ServerUnion
    Jan 13, 2006
  36. prabhjot.singh

    How to Remove 404 Error?

    prabhjot.singh, Aug 26, 2012
    Replies:
    18
    Views:
    10,444
    Rakeshraj
    Jul 15, 2013
  37. Burano

    Is my wordpress site on nofollow?

    Burano, Jan 1, 2010
    Replies:
    18
    Views:
    2,330
    john12
    Mar 16, 2010
  38. Googles76

    robots.txt for phpBB forum

    Googles76, Dec 3, 2005
    Replies:
    18
    Views:
    17,008
    linux-lover
    May 14, 2009
  39. mikelorentz

    robots.txt helps to improve your page rank

    mikelorentz, Jul 16, 2008
    Replies:
    18
    Views:
    2,123
    catanich
    Aug 21, 2008
  40. netboy

    Remove this stupid robots.txt

    netboy, Feb 22, 2008
    Replies:
    17
    Views:
    16,999
    Rituja
    Feb 15, 2012
  41. allanswagger

    Robots.txt

    allanswagger, Sep 2, 2010
    Replies:
    17
    Views:
    3,554
    manish.chauhan
    May 4, 2011
  42. mcsp

    Robots.txt question

    mcsp, Jul 26, 2005
    Replies:
    17
    Views:
    3,740
    gatordun
    Jul 29, 2005
  43. latoya

    How to hide robots.txt from visitors

    latoya, Aug 23, 2008
    Replies:
    16
    Views:
    16,307
    mrgroove
    Apr 1, 2010
  44. shailendra

    Confused about robots.txt

    shailendra, Mar 24, 2009
    Replies:
    16
    Views:
    3,147
    Aaron111
    Sep 2, 2009
  45. jallikatu

    Crawling restricted area

    jallikatu, Feb 1, 2009
    Replies:
    16
    Views:
    1,997
    manish.chauhan
    Feb 23, 2009
  46. bruce-banner

    Site with no robots.txt

    bruce-banner, Nov 14, 2008
    Replies:
    16
    Views:
    3,290
    shailendra
    Feb 19, 2009
  47. Angelabailey

    Robots.txt

    Angelabailey, Nov 15, 2013
    Replies:
    15
    Views:
    14,929
    Panda123
    Mar 27, 2016
  48. DevAdelicia

    What is Robots.txt ?

    DevAdelicia, Nov 17, 2011
    Replies:
    15
    Views:
    5,520
    xiaokaige
    Sep 18, 2012
  49. junrey24

    URL restricted by robots.txt

    junrey24, Mar 28, 2010
    Replies:
    15
    Views:
    1,964
    LookAtHerBeautifulFace
    Mar 29, 2010
  50. slayer__

    ideal robots.txt for all sites.

    slayer__, Jul 15, 2009
    Replies:
    15
    Views:
    2,739
    theapparatus
    Sep 19, 2009
Showing threads 1 to 50 of 901

Thread Display Options

Loading...
Page 1 of 19