robots.txt

Find help with setting up your robots.txt file on your web site.

Page 2 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↑
  1. cgo85

    Some help please...

    cgo85, Dec 28, 2005
    Replies:
    2
    Views:
    1,316
    Integrov
    Jan 1, 2006
  2. mdvaldosta

    Need help controlling bots (robots.txt - htaccess - etc)

    mdvaldosta, Dec 1, 2005
    Replies:
    2
    Views:
    2,245
    Integrov
    Jan 1, 2006
  3. jhantraz

    htaccess and robots.txt

    jhantraz, Jan 3, 2006
    Replies:
    5
    Views:
    6,212
    falco85
    Jan 7, 2006
  4. new

    spider specific and general rules

    new, Jan 12, 2006
    Replies:
    4
    Views:
    1,637
    new
    Jan 12, 2006
  5. Moneyfolk

    Robots.txt question

    Moneyfolk, Jan 12, 2006
    Replies:
    19
    Views:
    4,340
    ServerUnion
    Jan 13, 2006
  6. rockinaway

    Robots.txt?

    rockinaway, Jan 13, 2006
    Replies:
    7
    Views:
    1,718
    rockinaway
    Jan 14, 2006
  7. TheConnollyKid

    Any way to exclude include files from spiders?

    TheConnollyKid, Jan 27, 2006
    Replies:
    5
    Views:
    2,140
    minstrel
    Jan 29, 2006
  8. hmilesjr

    Question about Robots.txt file

    hmilesjr, Feb 2, 2006
    Replies:
    1
    Views:
    2,409
    gordonfreeman
    Feb 2, 2006
  9. Tim_Myth

    Referrer logs show odd referrer: somesite.com\robots.txt

    Tim_Myth, Feb 3, 2006
    Replies:
    7
    Views:
    3,821
    Tim_Myth
    Feb 5, 2006
  10. irka

    Robots.txt file generator with PHP

    irka, Jan 29, 2006
    Replies:
    6
    Views:
    5,711
    irka
    Feb 16, 2006
  11. icare

    What is robots.txt?

    icare, Feb 12, 2006
    Replies:
    11
    Views:
    5,348
    lionstarr
    Feb 23, 2006
  12. steve_gts

    Which Robots text?

    steve_gts, Feb 28, 2006
    Replies:
    7
    Views:
    1,458
    hans
    Feb 28, 2006
  13. jennypretty

    How/where to get robot.txt file

    jennypretty, Feb 23, 2006
    Replies:
    3
    Views:
    2,341
    neroux
    Feb 28, 2006
  14. girbaud

    robots.txt disallow

    girbaud, Jan 13, 2006
    Replies:
    19
    Views:
    4,328
    minstrel
    Mar 8, 2006
  15. yapyap2006

    status result for robots?

    yapyap2006, Mar 3, 2006
    Replies:
    2
    Views:
    1,269
    maildeepak
    Mar 13, 2006
  16. nevetS

    robots.txt + adsense

    nevetS, Jan 4, 2005
    Replies:
    3
    Views:
    2,163
    Slapyo
    Mar 16, 2006
  17. ian_ok

    Robots and php include files

    ian_ok, Mar 18, 2006
    Replies:
    1
    Views:
    1,593
    Slapyo
    Mar 20, 2006
  18. webmaster

    Don't want indexing!

    webmaster, Mar 16, 2006
    Replies:
    4
    Views:
    1,469
    webmaster
    Mar 26, 2006
  19. MediaHustler

    Can anyone find any mistakes in my robots.txt

    MediaHustler, Mar 24, 2006
    Replies:
    3
    Views:
    1,533
    alifan
    Mar 30, 2006
  20. mortgage-pro-seo

    Robots.txt ?

    mortgage-pro-seo, Feb 28, 2006
    Replies:
    6
    Views:
    2,072
    alifan
    Mar 30, 2006
  21. quaffapint

    FastFox Robots.txt block

    quaffapint, Feb 21, 2006
    Replies:
    4
    Views:
    2,342
    Jean-Luc
    Mar 31, 2006
  22. digitalpoint

    Google Not Consistent With robots.txt

    digitalpoint, Mar 22, 2006
    Replies:
    4
    Views:
    3,774
    Jean-Luc
    Apr 2, 2006
  23. j3r0m3

    Have robot.txt. but always 404

    j3r0m3, Mar 23, 2006 ... 2
    Replies:
    27
    Views:
    4,683
    ryan_uk
    Apr 3, 2006
  24. Riboflavin

    Robots.txt and indexing question

    Riboflavin, Apr 3, 2006
    Replies:
    5
    Views:
    1,588
    exam
    Apr 5, 2006
  25. R0ck$tAr

    Is This A Bad Bot?

    R0ck$tAr, Apr 12, 2006
    Replies:
    15
    Views:
    2,131
    exam
    Apr 13, 2006
  26. jordan5815

    Robots.txt

    jordan5815, Apr 11, 2006
    Replies:
    15
    Views:
    2,100
    Crazy_Rob
    Apr 14, 2006
  27. potski

    Help writing Robots.txt file

    potski, Apr 10, 2006
    Replies:
    3
    Views:
    1,799
    minstrel
    Apr 19, 2006
  28. hulo

    Robots.txt Exclusion and Sandbox?

    hulo, Apr 20, 2006
    Replies:
    1
    Views:
    1,296
    INV
    Apr 20, 2006
  29. pzn

    robots.txt?

    pzn, Apr 19, 2006
    Replies:
    4
    Views:
    1,863
    Mystique
    Apr 20, 2006
  30. girbaud

    what this code */ means?

    girbaud, Apr 20, 2006
    Replies:
    3
    Views:
    1,434
    girbaud
    Apr 24, 2006
  31. stayyoung

    robots.txt not found in google sitemap program

    stayyoung, Apr 24, 2006
    Replies:
    2
    Views:
    1,768
    shabda
    Apr 29, 2006
  32. abcdefGARY

    Robots.txt?

    abcdefGARY, Apr 29, 2006
    Replies:
    2
    Views:
    1,414
    abcdefGARY
    Apr 29, 2006
  33. icare

    is robots.txt a good thing to have in forum root?

    icare, May 1, 2006
    Replies:
    3
    Views:
    1,766
    elcash
    May 5, 2006
  34. cmeinck

    Robots Meta Tag needed on every page of site?

    cmeinck, May 6, 2006
    Replies:
    1
    Views:
    1,489
    Jean-Luc
    May 6, 2006
  35. Percept

    Is this robots.txt blocking Slurp ?

    Percept, Feb 12, 2006
    Replies:
    3
    Views:
    2,060
    mussolinihitler
    May 7, 2006
  36. bobby9101

    block bots

    bobby9101, May 13, 2006
    Replies:
    4
    Views:
    3,521
    bobby9101
    May 18, 2006
  37. hype

    Need search engine spider!

    hype, May 5, 2006
    Replies:
    2
    Views:
    2,186
    MaxPowers
    May 26, 2006
  38. maurus

    Dublicate Content and robots.txt

    maurus, May 9, 2006
    Replies:
    1
    Views:
    1,343
    thes
    Jun 2, 2006
  39. tim_ver

    Robot.txt on sharred server

    tim_ver, Jun 7, 2006
    Replies:
    6
    Views:
    1,414
    mad4
    Jun 8, 2006
  40. grobar

    robots.txt OR ...

    grobar, Jun 8, 2006
    Replies:
    1
    Views:
    1,388
    Kaabi
    Jun 8, 2006
  41. explorer

    How could I block the google search spider but still allow adsense?

    explorer, Jun 11, 2006
    Replies:
    3
    Views:
    4,784
    Jean-Luc
    Jun 12, 2006
  42. seoindiaweb

    robots.txt

    seoindiaweb, May 29, 2006
    Replies:
    4
    Views:
    1,912
    maldives
    Jun 13, 2006
  43. UnEmployedAfterSchool

    Boogle Bot prevention for Private Membership Forder

    UnEmployedAfterSchool, Jun 14, 2006
    Replies:
    1
    Views:
    1,751
    ServerUnion
    Jun 14, 2006
  44. samib

    help with robots.txt

    samib, Jun 15, 2006
    Replies:
    1
    Views:
    1,648
    Jean-Luc
    Jun 16, 2006
  45. whateverandever2

    robots.txt

    whateverandever2, Jun 17, 2006
    Replies:
    10
    Views:
    2,047
    Jean-Luc
    Jun 17, 2006
  46. samib

    Robots.txt Question

    samib, Jun 15, 2006
    Replies:
    3
    Views:
    1,337
    Jean-Luc
    Jun 19, 2006
  47. MicahCarrick

    Don't want linked page listed in google

    MicahCarrick, Jun 21, 2006
    Replies:
    2
    Views:
    1,412
    Jean-Luc
    Jun 21, 2006
  48. fcmisc

    Correct robots.txt syntax to block ask jeeves?

    fcmisc, May 7, 2006
    Replies:
    5
    Views:
    2,795
    fcmisc
    Jun 23, 2006
  49. fesite

    How can I use robots.txt?

    fesite, Jun 24, 2006
    Replies:
    1
    Views:
    1,783
    minstrel
    Jun 24, 2006
  50. fsmedia

    how to stop alexa??

    fsmedia, May 7, 2006
    Replies:
    8
    Views:
    2,612
    fsmedia
    Jun 25, 2006
Showing threads 51 to 100 of 911

Thread Display Options

Loading...
Page 2 of 19