robots.txt

Find help with setting up your robots.txt file on your web site.

Page 18 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. icare

    What is robots.txt?

    icare, Feb 12, 2006
    Replies:
    11
    Views:
    5,312
    lionstarr
    Feb 23, 2006
  2. irka

    Robots.txt file generator with PHP

    irka, Jan 29, 2006
    Replies:
    6
    Views:
    5,686
    irka
    Feb 16, 2006
  3. Tim_Myth

    Referrer logs show odd referrer: somesite.com\robots.txt

    Tim_Myth, Feb 3, 2006
    Replies:
    7
    Views:
    3,796
    Tim_Myth
    Feb 5, 2006
  4. hmilesjr

    Question about Robots.txt file

    hmilesjr, Feb 2, 2006
    Replies:
    1
    Views:
    2,393
    gordonfreeman
    Feb 2, 2006
  5. TheConnollyKid

    Any way to exclude include files from spiders?

    TheConnollyKid, Jan 27, 2006
    Replies:
    5
    Views:
    2,119
    minstrel
    Jan 29, 2006
  6. rockinaway

    Robots.txt?

    rockinaway, Jan 13, 2006
    Replies:
    7
    Views:
    1,682
    rockinaway
    Jan 14, 2006
  7. Moneyfolk

    Robots.txt question

    Moneyfolk, Jan 12, 2006
    Replies:
    19
    Views:
    4,318
    ServerUnion
    Jan 13, 2006
  8. new

    spider specific and general rules

    new, Jan 12, 2006
    Replies:
    4
    Views:
    1,594
    new
    Jan 12, 2006
  9. jhantraz

    htaccess and robots.txt

    jhantraz, Jan 3, 2006
    Replies:
    5
    Views:
    6,147
    falco85
    Jan 7, 2006
  10. mdvaldosta

    Need help controlling bots (robots.txt - htaccess - etc)

    mdvaldosta, Dec 1, 2005
    Replies:
    2
    Views:
    2,224
    Integrov
    Jan 1, 2006
  11. cgo85

    Some help please...

    cgo85, Dec 28, 2005
    Replies:
    2
    Views:
    1,299
    Integrov
    Jan 1, 2006
  12. JEET

    Please see if this is right.

    JEET, Dec 24, 2005
    Replies:
    4
    Views:
    1,839
    SiteExpress
    Dec 27, 2005
  13. dshah

    using robots.txt during development

    dshah, Dec 15, 2005
    Replies:
    2
    Views:
    1,816
    dshah
    Dec 18, 2005
  14. Edz

    The most ideal robots.txt? how should it look like?

    Edz, Nov 24, 2005 ... 2
    Replies:
    34
    Views:
    7,234
    minstrel
    Dec 10, 2005
  15. jda

    Robots.txt Disallow Question

    jda, Nov 28, 2005
    Replies:
    4
    Views:
    2,330
    jda
    Dec 2, 2005
  16. dshah

    Optimal commands in robots.txt

    dshah, Nov 28, 2005
    Replies:
    2
    Views:
    2,890
    minstrel
    Dec 1, 2005
  17. noelves

    robots

    noelves, Nov 28, 2005
    Replies:
    2
    Views:
    1,718
    noelves
    Nov 28, 2005
  18. Design Agent

    Robots.txt, Vbulletin & Google

    Design Agent, Nov 26, 2005
    Replies:
    10
    Views:
    2,903
    Design Agent
    Nov 26, 2005
  19. Dio

    Blocking Bad Bots With HTACCESS rather than robots.txt

    Dio, Nov 23, 2005
    Replies:
    1
    Views:
    5,308
    tflight
    Nov 23, 2005
  20. SERPalert

    Robots.txt

    SERPalert, Oct 26, 2005
    Replies:
    2
    Views:
    2,011
    dj1471
    Nov 19, 2005
  21. tweetylover8402

    Robots.txt?

    tweetylover8402, Oct 25, 2005
    Replies:
    14
    Views:
    2,482
    jazzylee77
    Nov 13, 2005
  22. misohoni

    How to stop Google Spidering my Javascript...

    misohoni, Aug 31, 2005
    Replies:
    8
    Views:
    3,778
    DangerMouse
    Nov 11, 2005
  23. mjonsson

    406: Not Acceptable (encoding)

    mjonsson, Nov 9, 2005
    Replies:
    1
    Views:
    2,812
    minstrel
    Nov 9, 2005
  24. briandunning

    Can I use robots.txt to block *.html?

    briandunning, Nov 2, 2005
    Replies:
    4
    Views:
    2,344
    minstrel
    Nov 5, 2005
  25. melaniejk

    What happened? robots.txt changed it?

    melaniejk, Oct 18, 2005
    Replies:
    5
    Views:
    1,776
    ServerUnion
    Oct 18, 2005
  26. Jon12345

    What bots ignore robots.txt?

    Jon12345, Aug 16, 2005
    Replies:
    3
    Views:
    13,371
    minstrel
    Oct 15, 2005
  27. fryman

    is a blank robots file ok?

    fryman, Jul 13, 2005 ... 2 3
    Replies:
    40
    Views:
    6,013
    minstrel
    Oct 14, 2005
  28. misohoni

    Seen Google's Robot.txt?

    misohoni, Oct 13, 2005
    Replies:
    2
    Views:
    4,657
    minstrel
    Oct 14, 2005
  29. Davilac

    Some "advanced" help

    Davilac, Sep 22, 2005
    Replies:
    1
    Views:
    1,540
    minstrel
    Sep 30, 2005
  30. aardvark

    NOINDEX valid in robots.txt?

    aardvark, Sep 21, 2005
    Replies:
    7
    Views:
    2,086
    webmistress
    Sep 24, 2005
  31. 802networks

    Will robots.txt block duplicate content effectively

    802networks, Aug 18, 2005
    Replies:
    3
    Views:
    4,052
    aeiouy
    Sep 14, 2005
  32. Alfred_ans

    Pages getting indexed despite diallowing them with robots.txt

    Alfred_ans, Sep 13, 2005
    Replies:
    1
    Views:
    1,705
    Kim Steinhaug
    Sep 13, 2005
  33. Solicitors Mortgages

    robots/htaccess

    Solicitors Mortgages, Apr 21, 2005
    Replies:
    3
    Views:
    2,053
    gatordun
    Jul 29, 2005
  34. mcsp

    Robots.txt question

    mcsp, Jul 26, 2005
    Replies:
    17
    Views:
    4,329
    gatordun
    Jul 29, 2005
  35. vprp

    robots.txt help... trying to disallow file only from Google.

    vprp, Jul 16, 2005
    Replies:
    6
    Views:
    2,884
    minstrel
    Jul 29, 2005
  36. ian_ok

    Google no following robots.txt

    ian_ok, Jun 7, 2005
    Replies:
    5
    Views:
    3,626
    ziandra
    Jul 13, 2005
  37. Yukio

    Should I deny access to my site forum using googlebot?

    Yukio, Jun 21, 2005 ... 2
    Replies:
    21
    Views:
    5,152
    nevetS
    Jun 29, 2005
  38. NewComputer

    We were unable to access the URL you provided due to a restriction in robots.txt.

    NewComputer, Jun 20, 2005
    Replies:
    10
    Views:
    3,291
    minstrel
    Jun 23, 2005
  39. Tuning

    Checking robots.txt?

    Tuning, Jun 7, 2005
    Replies:
    9
    Views:
    2,026
    noppid
    Jun 8, 2005
  40. THT

    Robots.txt question

    THT, Jun 5, 2005
    Replies:
    12
    Views:
    2,760
    ZuraX
    Jun 6, 2005
  41. ashiezai

    robot.txt help

    ashiezai, May 31, 2005
    Replies:
    2
    Views:
    3,677
    ashiezai
    May 31, 2005
  42. l234244

    Blocking robots?

    l234244, May 30, 2005
    Replies:
    9
    Views:
    4,106
    Will.Spencer
    May 30, 2005
  43. nevetS

    AWSM Bots?

    nevetS, May 12, 2005
    Replies:
    0
    Views:
    2,449
    nevetS
    May 12, 2005
  44. TommyD

    robotsxx.txt and other strange robot files...

    TommyD, May 10, 2005
    Replies:
    2
    Views:
    4,052
    Dio
    May 11, 2005
  45. cellphone_store

    Meta noindex OR robots.txt OR rel=nofollow

    cellphone_store, May 5, 2005
    Replies:
    8
    Views:
    5,076
    ServerUnion
    May 6, 2005
  46. tripower

    AdSense, subdomains and robots.txt

    tripower, May 4, 2005
    Replies:
    5
    Views:
    7,043
    paymentapprovaltooslow
    May 6, 2005
  47. Owlcroft

    Infant-level robots.txt question.

    Owlcroft, Dec 26, 2004
    Replies:
    9
    Views:
    3,342
    mcfox
    Apr 29, 2005
  48. digitalpoint

    Disallowing All But Certain Spiders

    digitalpoint, Apr 23, 2005
    Replies:
    3
    Views:
    2,500
    fryman
    Apr 24, 2005
  49. Epica

    Robots.txt - hide sitewide HTML pages only

    Epica, Apr 20, 2005
    Replies:
    4
    Views:
    3,514
    Epica
    Apr 20, 2005
  50. NewComputer

    robots.txt pointing to index page

    NewComputer, Sep 26, 2004 ... 2
    Replies:
    39
    Views:
    6,503
    Blogmaster
    Apr 19, 2005
Showing threads 851 to 900 of 911

Thread Display Options

Loading...
Page 18 of 19