robots.txt

Find help with setting up your robots.txt file on your web site.

Page 15 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. Brajeshwar

    Disallow search bots beyond the root folder content

    Brajeshwar, Jul 26, 2007
    Replies:
    4
    Views:
    2,089
    Brajeshwar
    Aug 6, 2007
  2. othonas

    I really need your help

    othonas, Jul 19, 2007
    Replies:
    7
    Views:
    1,358
    trichnosis
    Aug 6, 2007
  3. coolcow

    Please review my Robots.txt

    coolcow, Jul 17, 2007
    Replies:
    1
    Views:
    1,155
    trichnosis
    Aug 6, 2007
  4. wijasenna

    robot.txt

    wijasenna, Jul 13, 2007
    Replies:
    3
    Views:
    1,897
    trichnosis
    Aug 6, 2007
  5. SaN-DeeP

    psbot and gigabot wont stop even when banned from robots.txt

    SaN-DeeP, Feb 28, 2006
    Replies:
    7
    Views:
    3,866
    trichnosis
    Aug 6, 2007
  6. markowe

    Picsearch/PSbot - what's the verdict, ban it?

    markowe, May 20, 2007
    Replies:
    2
    Views:
    2,830
    trichnosis
    Aug 6, 2007
  7. autorave

    Can I use robots.txt to protect my blog feed?

    autorave, Mar 28, 2007
    Replies:
    6
    Views:
    2,576
    trichnosis
    Aug 6, 2007
  8. frio

    Robots.txt please help!!!

    frio, Jul 30, 2007
    Replies:
    3
    Views:
    1,473
    trichnosis
    Aug 6, 2007
  9. theultimatepublishing

    www. & non-www. (fixing robots.text)

    theultimatepublishing, Jul 30, 2007
    Replies:
    5
    Views:
    1,315
    trichnosis
    Aug 6, 2007
  10. trichnosis

    blocking url like /?option=com_content&task=emailform&id=1000&itemid =8

    trichnosis, May 13, 2007
    Replies:
    3
    Views:
    2,101
    trichnosis
    Jul 22, 2007
  11. astounding

    This one is for robots.txt expertsL How does one pull his off...

    astounding, Jul 18, 2007
    Replies:
    3
    Views:
    1,156
    trichnosis
    Jul 22, 2007
  12. vishwa

    Autodiscovery of Sitemaps Using robots.txt

    vishwa, Apr 19, 2007
    Replies:
    7
    Views:
    2,107
    danieloffice
    Jul 16, 2007
  13. paulinmargarita

    Is robots.txt really necessary??

    paulinmargarita, Jun 23, 2007
    Replies:
    14
    Views:
    2,263
    antman
    Jul 10, 2007
  14. paladin2

    How to stop Googlebot to spider this page?

    paladin2, Jun 25, 2007
    Replies:
    3
    Views:
    5,239
    trichnosis
    Jul 3, 2007
  15. Phishie

    Disallow /images/ but allow /images/dir/dir. Is it possible?

    Phishie, Jun 21, 2007
    Replies:
    3
    Views:
    1,211
    trichnosis
    Jul 3, 2007
  16. markowe

    IP's in range 124.115.*.*

    markowe, Jun 19, 2007
    Replies:
    5
    Views:
    2,189
    DeViAnThans3
    Jun 27, 2007
  17. papek

    My robots.txt file, is it too silly example?

    papek, Jun 7, 2007
    Replies:
    12
    Views:
    1,964
    DavidK1
    Jun 12, 2007
  18. weknowtheworld

    robots.txt

    weknowtheworld, Mar 29, 2007
    Replies:
    5
    Views:
    1,722
    kirby009
    Jun 12, 2007
  19. steveb

    Robots.txt Question

    steveb, Mar 21, 2007
    Replies:
    7
    Views:
    1,722
    kirby009
    Jun 12, 2007
  20. Phishie

    Can somebody look over this real fast?

    Phishie, Jun 11, 2007
    Replies:
    4
    Views:
    1,126
    kirby009
    Jun 12, 2007
  21. Mysmasken

    Prevent subdomain listings?

    Mysmasken, May 28, 2007
    Replies:
    4
    Views:
    1,373
    trichnosis
    Jun 8, 2007
  22. Phishie

    Blocking /?Itemid=1

    Phishie, May 29, 2007
    Replies:
    2
    Views:
    1,441
    trichnosis
    Jun 8, 2007
  23. Dave Hybrid

    Blocking query strings in URLs

    Dave Hybrid, Jun 1, 2007
    Replies:
    8
    Views:
    2,654
    Dave Hybrid
    Jun 1, 2007
  24. me4you

    I have some Question about robots.txt

    me4you, May 31, 2007
    Replies:
    2
    Views:
    1,251
    pommy
    May 31, 2007
  25. visualscope

    block all except in robots.txt

    visualscope, May 26, 2007
    Replies:
    1
    Views:
    2,387
    tinkerbox
    May 27, 2007
  26. Hostaff

    Excluding feeds in Wordpress

    Hostaff, May 22, 2007
    Replies:
    0
    Views:
    960
    Hostaff
    May 22, 2007
  27. jgjg

    Please give me a robots.txt file to exclude as many search engines as possible!!

    jgjg, May 18, 2007
    Replies:
    12
    Views:
    1,761
    Christine8
    May 20, 2007
  28. sck

    Preventing Bad bots (robots)

    sck, May 9, 2007
    Replies:
    4
    Views:
    1,634
    sarahk
    May 12, 2007
  29. vic_msn

    Adding path to sitemaps in robots.txt

    vic_msn, Apr 26, 2007
    Replies:
    4
    Views:
    1,746
    trichnosis
    May 11, 2007
  30. AHA7

    Is this valid?

    AHA7, May 8, 2007
    Replies:
    1
    Views:
    1,057
    Spartan_Strategy
    May 10, 2007
  31. rldowling03

    Letting robots browse forum posts

    rldowling03, Apr 19, 2007
    Replies:
    5
    Views:
    1,414
    rldowling03
    May 3, 2007
  32. swiftmed
    Locked

    Bad?

    swiftmed, Feb 25, 2007
    Replies:
    4
    Views:
    1,377
    kgrad
    May 2, 2007
  33. AHA7

    A few simple questions about robots.txt

    AHA7, Apr 29, 2007
    Replies:
    2
    Views:
    1,361
    AHA7
    Apr 29, 2007
  34. imparas

    What is the code.

    imparas, Apr 25, 2007
    Replies:
    10
    Views:
    1,482
    imparas
    Apr 27, 2007
  35. rjdpa

    How large can the robots.txt file be?

    rjdpa, Apr 27, 2007
    Replies:
    3
    Views:
    1,580
    m1t0s1s
    Apr 27, 2007
  36. ish1301
    Locked

    How to block spam robots ?

    ish1301, Nov 14, 2006
    Replies:
    11
    Views:
    4,593
    ruby
    Apr 26, 2007
  37. imnotadoctor

    the new Sitemap parameter and robot.txt file help

    imnotadoctor, Apr 12, 2007
    Replies:
    3
    Views:
    3,503
    Claudek
    Apr 23, 2007
  38. srinet

    Robot.txt

    srinet, Jan 19, 2007
    Replies:
    7
    Views:
    2,123
    RECEP
    Apr 11, 2007
  39. abcde

    Help on Robots.txt Please.. even google groups couldt help :-(

    abcde, Apr 4, 2007
    Replies:
    1
    Views:
    1,117
    abcde
    Apr 4, 2007
  40. wisam74us

    Robot.txt for 2 parked domains

    wisam74us, Apr 3, 2007
    Replies:
    2
    Views:
    1,455
    wisam74us
    Apr 3, 2007
  41. Jona

    Blocking spam IP in htaccess doens't work ... What's wrong ?

    Jona, Mar 9, 2007
    Replies:
    1
    Views:
    1,375
    logodesignteam
    Mar 25, 2007
  42. gio

    How To Disable All Others Bots Except Googlebot, Yahoo! Slurp, and MSNbot?

    gio, Feb 20, 2007
    Replies:
    5
    Views:
    3,922
    alexus
    Mar 18, 2007
  43. dougadam

    Which IP's to block?

    dougadam, Mar 17, 2007
    Replies:
    0
    Views:
    1,155
    dougadam
    Mar 17, 2007
  44. Will.Spencer

    Chinese robots no understand robots.txt?

    Will.Spencer, Mar 5, 2007
    Replies:
    8
    Views:
    2,665
    Bondat
    Mar 13, 2007
  45. kmike

    Does Yahoo Slurp really support wildcards in robots.txt?

    kmike, Mar 1, 2007
    Replies:
    4
    Views:
    1,690
    kmike
    Mar 10, 2007
  46. samantha pia

    domain pointing/redirects and url's in address bar

    samantha pia, Mar 10, 2007
    Replies:
    0
    Views:
    1,404
    samantha pia
    Mar 10, 2007
  47. AHA7

    Disabling WP feed pages

    AHA7, Mar 6, 2007
    Replies:
    1
    Views:
    1,315
    agnivo007
    Mar 7, 2007
  48. VishalVasani

    I want to disallow spam bot

    VishalVasani, Feb 21, 2007
    Replies:
    1
    Views:
    1,286
    WebGeek182
    Mar 5, 2007
  49. The Rock

    My Robot Txt Syntax?

    The Rock, Mar 1, 2007
    Replies:
    3
    Views:
    1,258
    ac_roma
    Mar 2, 2007
  50. amitpatel_3001

    Is this Gud/Bad??? Google Bot eats 330MB/Day Bandwidth

    amitpatel_3001, Dec 9, 2005 ... 2
    Replies:
    23
    Views:
    7,566
    amitpatel_3001
    Feb 23, 2007
Showing threads 701 to 750 of 911

Thread Display Options

Loading...
Page 15 of 19