1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 18 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. icare

    What is robots.txt?

    icare, Feb 12, 2006
    Replies:
    11
    Views:
    5,168
    lionstarr
    Feb 23, 2006
  2. irka

    Robots.txt file generator with PHP

    irka, Jan 29, 2006
    Replies:
    6
    Views:
    5,564
    irka
    Feb 16, 2006
  3. Tim_Myth

    Referrer logs show odd referrer: somesite.com\robots.txt

    Tim_Myth, Feb 3, 2006
    Replies:
    7
    Views:
    3,611
    Tim_Myth
    Feb 5, 2006
  4. hmilesjr

    Question about Robots.txt file

    hmilesjr, Feb 2, 2006
    Replies:
    1
    Views:
    2,305
    gordonfreeman
    Feb 2, 2006
  5. TheConnollyKid

    Any way to exclude include files from spiders?

    TheConnollyKid, Jan 27, 2006
    Replies:
    5
    Views:
    2,031
    minstrel
    Jan 29, 2006
  6. rockinaway

    Robots.txt?

    rockinaway, Jan 13, 2006
    Replies:
    7
    Views:
    1,581
    rockinaway
    Jan 14, 2006
  7. Moneyfolk

    Robots.txt question

    Moneyfolk, Jan 12, 2006
    Replies:
    19
    Views:
    4,149
    ServerUnion
    Jan 13, 2006
  8. new

    spider specific and general rules

    new, Jan 12, 2006
    Replies:
    4
    Views:
    1,440
    new
    Jan 12, 2006
  9. jhantraz

    htaccess and robots.txt

    jhantraz, Jan 3, 2006
    Replies:
    5
    Views:
    5,948
    falco85
    Jan 7, 2006
  10. mdvaldosta

    Need help controlling bots (robots.txt - htaccess - etc)

    mdvaldosta, Dec 1, 2005
    Replies:
    2
    Views:
    2,121
    Integrov
    Jan 1, 2006
  11. cgo85

    Some help please...

    cgo85, Dec 28, 2005
    Replies:
    2
    Views:
    1,230
    Integrov
    Jan 1, 2006
  12. JEET

    Please see if this is right.

    JEET, Dec 24, 2005
    Replies:
    4
    Views:
    1,708
    SiteExpress
    Dec 27, 2005
  13. dshah

    using robots.txt during development

    dshah, Dec 15, 2005
    Replies:
    2
    Views:
    1,630
    dshah
    Dec 18, 2005
  14. Edz

    The most ideal robots.txt? how should it look like?

    Edz, Nov 24, 2005 ... 2
    Replies:
    34
    Views:
    6,883
    minstrel
    Dec 10, 2005
  15. jda

    Robots.txt Disallow Question

    jda, Nov 28, 2005
    Replies:
    4
    Views:
    2,202
    jda
    Dec 2, 2005
  16. dshah

    Optimal commands in robots.txt

    dshah, Nov 28, 2005
    Replies:
    2
    Views:
    2,740
    minstrel
    Dec 1, 2005
  17. noelves

    robots

    noelves, Nov 28, 2005
    Replies:
    2
    Views:
    1,578
    noelves
    Nov 28, 2005
  18. Design Agent

    Robots.txt, Vbulletin & Google

    Design Agent, Nov 26, 2005
    Replies:
    10
    Views:
    2,755
    Design Agent
    Nov 26, 2005
  19. Dio

    Blocking Bad Bots With HTACCESS rather than robots.txt

    Dio, Nov 23, 2005
    Replies:
    1
    Views:
    5,151
    tflight
    Nov 23, 2005
  20. SERPalert

    Robots.txt

    SERPalert, Oct 26, 2005
    Replies:
    2
    Views:
    1,890
    dj1471
    Nov 19, 2005
  21. tweetylover8402

    Robots.txt?

    tweetylover8402, Oct 25, 2005
    Replies:
    14
    Views:
    2,333
    jazzylee77
    Nov 13, 2005
  22. misohoni

    How to stop Google Spidering my Javascript...

    misohoni, Aug 31, 2005
    Replies:
    8
    Views:
    3,649
    DangerMouse
    Nov 11, 2005
  23. mjonsson

    406: Not Acceptable (encoding)

    mjonsson, Nov 9, 2005
    Replies:
    1
    Views:
    2,683
    minstrel
    Nov 9, 2005
  24. briandunning

    Can I use robots.txt to block *.html?

    briandunning, Nov 2, 2005
    Replies:
    4
    Views:
    2,246
    minstrel
    Nov 5, 2005
  25. melaniejk

    What happened? robots.txt changed it?

    melaniejk, Oct 18, 2005
    Replies:
    5
    Views:
    1,633
    ServerUnion
    Oct 18, 2005
  26. Jon12345

    What bots ignore robots.txt?

    Jon12345, Aug 16, 2005
    Replies:
    3
    Views:
    13,192
    minstrel
    Oct 15, 2005
  27. fryman

    is a blank robots file ok?

    fryman, Jul 13, 2005 ... 2 3
    Replies:
    40
    Views:
    5,522
    minstrel
    Oct 14, 2005
  28. misohoni

    Seen Google's Robot.txt?

    misohoni, Oct 13, 2005
    Replies:
    2
    Views:
    4,540
    minstrel
    Oct 14, 2005
  29. Davilac

    Some "advanced" help

    Davilac, Sep 22, 2005
    Replies:
    1
    Views:
    1,460
    minstrel
    Sep 30, 2005
  30. aardvark

    NOINDEX valid in robots.txt?

    aardvark, Sep 21, 2005
    Replies:
    7
    Views:
    1,968
    webmistress
    Sep 24, 2005
  31. 802networks

    Will robots.txt block duplicate content effectively

    802networks, Aug 18, 2005
    Replies:
    3
    Views:
    3,953
    aeiouy
    Sep 14, 2005
  32. Alfred_ans

    Pages getting indexed despite diallowing them with robots.txt

    Alfred_ans, Sep 13, 2005
    Replies:
    1
    Views:
    1,595
    Kim Steinhaug
    Sep 13, 2005
  33. Solicitors Mortgages

    robots/htaccess

    Solicitors Mortgages, Apr 21, 2005
    Replies:
    3
    Views:
    1,950
    gatordun
    Jul 29, 2005
  34. mcsp

    Robots.txt question

    mcsp, Jul 26, 2005
    Replies:
    17
    Views:
    4,180
    gatordun
    Jul 29, 2005
  35. vprp

    robots.txt help... trying to disallow file only from Google.

    vprp, Jul 16, 2005
    Replies:
    6
    Views:
    2,793
    minstrel
    Jul 29, 2005
  36. ian_ok

    Google no following robots.txt

    ian_ok, Jun 7, 2005
    Replies:
    5
    Views:
    3,509
    ziandra
    Jul 13, 2005
  37. Yukio

    Should I deny access to my site forum using googlebot?

    Yukio, Jun 21, 2005 ... 2
    Replies:
    21
    Views:
    4,951
    nevetS
    Jun 29, 2005
  38. NewComputer

    We were unable to access the URL you provided due to a restriction in robots.txt.

    NewComputer, Jun 20, 2005
    Replies:
    10
    Views:
    3,169
    minstrel
    Jun 23, 2005
  39. Tuning

    Checking robots.txt?

    Tuning, Jun 7, 2005
    Replies:
    9
    Views:
    1,932
    noppid
    Jun 8, 2005
  40. THT

    Robots.txt question

    THT, Jun 5, 2005
    Replies:
    12
    Views:
    2,641
    ZuraX
    Jun 6, 2005
  41. ashiezai

    robot.txt help

    ashiezai, May 31, 2005
    Replies:
    2
    Views:
    3,583
    ashiezai
    May 31, 2005
  42. l234244

    Blocking robots?

    l234244, May 30, 2005
    Replies:
    9
    Views:
    3,908
    Will.Spencer
    May 30, 2005
  43. nevetS

    AWSM Bots?

    nevetS, May 12, 2005
    Replies:
    0
    Views:
    2,344
    nevetS
    May 12, 2005
  44. TommyD

    robotsxx.txt and other strange robot files...

    TommyD, May 10, 2005
    Replies:
    2
    Views:
    3,946
    Dio
    May 11, 2005
  45. cellphone_store

    Meta noindex OR robots.txt OR rel=nofollow

    cellphone_store, May 5, 2005
    Replies:
    8
    Views:
    4,957
    ServerUnion
    May 6, 2005
  46. tripower

    AdSense, subdomains and robots.txt

    tripower, May 4, 2005
    Replies:
    5
    Views:
    6,898
    paymentapprovaltooslow
    May 6, 2005
  47. Owlcroft

    Infant-level robots.txt question.

    Owlcroft, Dec 26, 2004
    Replies:
    9
    Views:
    3,128
    mcfox
    Apr 29, 2005
  48. digitalpoint

    Disallowing All But Certain Spiders

    digitalpoint, Apr 23, 2005
    Replies:
    3
    Views:
    2,412
    fryman
    Apr 24, 2005
  49. Epica

    Robots.txt - hide sitewide HTML pages only

    Epica, Apr 20, 2005
    Replies:
    4
    Views:
    3,406
    Epica
    Apr 20, 2005
  50. NewComputer

    robots.txt pointing to index page

    NewComputer, Sep 26, 2004 ... 2
    Replies:
    39
    Views:
    6,011
    Blogmaster
    Apr 19, 2005
Showing threads 851 to 900 of 911

Thread Display Options

Loading...
Page 18 of 19