robots.txt

Find help with setting up your robots.txt file on your web site.

Page 4 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↑
  1. pr0xy122

    What is robots.txt and should I modify it?

    pr0xy122, Oct 25, 2006
    Replies:
    8
    Views:
    3,359
    jonbt
    Jan 30, 2007
  2. rusting

    please help me!!!

    rusting, Jan 22, 2007
    Replies:
    4
    Views:
    1,252
    jonbt
    Jan 30, 2007
  3. Jon12345

    Help me design a simple robots file!

    Jon12345, Jan 8, 2007
    Replies:
    6
    Views:
    1,554
    jonbt
    Jan 30, 2007
  4. circuscircus
    Locked

    How important is robot.txt in SEO?

    circuscircus, Jan 21, 2007
    Replies:
    5
    Views:
    1,975
    jonbt
    Jan 30, 2007
  5. mvhs

    How to block spiders?

    mvhs, Jan 20, 2007
    Replies:
    8
    Views:
    4,657
    jonbt
    Jan 30, 2007
  6. m0rtal

    Would this work?

    m0rtal, Jan 29, 2007
    Replies:
    4
    Views:
    1,565
    m0rtal
    Jan 30, 2007
  7. alarik

    google indexed my website :(what did i do wrong?

    alarik, Jan 1, 2007
    Replies:
    6
    Views:
    2,063
    alarik
    Jan 31, 2007
  8. Ogden2k

    robots.txt question

    Ogden2k, Feb 5, 2007
    Replies:
    1
    Views:
    1,493
    nevetS
    Feb 5, 2007
  9. eXe

    Question

    eXe, Feb 15, 2007
    Replies:
    3
    Views:
    1,318
    Jean-Luc
    Feb 20, 2007
  10. relixx

    robots.txt and blocking querystrings

    relixx, Feb 15, 2007
    Replies:
    3
    Views:
    1,579
    relixx
    Feb 20, 2007
  11. skionxb

    BLOCKED ROBOTS.txt and CUSTOM 404 PAGE ISSUE

    skionxb, Jul 27, 2006
    Replies:
    9
    Views:
    3,483
    WebGeek182
    Feb 22, 2007
  12. amitpatel_3001

    Is this Gud/Bad??? Google Bot eats 330MB/Day Bandwidth

    amitpatel_3001, Dec 9, 2005 ... 2
    Replies:
    23
    Views:
    7,751
    amitpatel_3001
    Feb 23, 2007
  13. The Rock

    My Robot Txt Syntax?

    The Rock, Mar 1, 2007
    Replies:
    3
    Views:
    1,280
    ac_roma
    Mar 2, 2007
  14. VishalVasani

    I want to disallow spam bot

    VishalVasani, Feb 21, 2007
    Replies:
    1
    Views:
    1,309
    WebGeek182
    Mar 5, 2007
  15. AHA7

    Disabling WP feed pages

    AHA7, Mar 6, 2007
    Replies:
    1
    Views:
    1,338
    agnivo007
    Mar 7, 2007
  16. samantha pia

    domain pointing/redirects and url's in address bar

    samantha pia, Mar 10, 2007
    Replies:
    0
    Views:
    1,430
    samantha pia
    Mar 10, 2007
  17. kmike

    Does Yahoo Slurp really support wildcards in robots.txt?

    kmike, Mar 1, 2007
    Replies:
    4
    Views:
    1,734
    kmike
    Mar 10, 2007
  18. Will.Spencer

    Chinese robots no understand robots.txt?

    Will.Spencer, Mar 5, 2007
    Replies:
    8
    Views:
    2,721
    Bondat
    Mar 13, 2007
  19. dougadam

    Which IP's to block?

    dougadam, Mar 17, 2007
    Replies:
    0
    Views:
    1,181
    dougadam
    Mar 17, 2007
  20. gio

    How To Disable All Others Bots Except Googlebot, Yahoo! Slurp, and MSNbot?

    gio, Feb 20, 2007
    Replies:
    5
    Views:
    3,947
    alexus
    Mar 18, 2007
  21. Jona

    Blocking spam IP in htaccess doens't work ... What's wrong ?

    Jona, Mar 9, 2007
    Replies:
    1
    Views:
    1,401
    logodesignteam
    Mar 25, 2007
  22. wisam74us

    Robot.txt for 2 parked domains

    wisam74us, Apr 3, 2007
    Replies:
    2
    Views:
    1,472
    wisam74us
    Apr 3, 2007
  23. abcde

    Help on Robots.txt Please.. even google groups couldt help :-(

    abcde, Apr 4, 2007
    Replies:
    1
    Views:
    1,140
    abcde
    Apr 4, 2007
  24. srinet

    Robot.txt

    srinet, Jan 19, 2007
    Replies:
    7
    Views:
    2,149
    RECEP
    Apr 11, 2007
  25. imnotadoctor

    the new Sitemap parameter and robot.txt file help

    imnotadoctor, Apr 12, 2007
    Replies:
    3
    Views:
    3,527
    Claudek
    Apr 23, 2007
  26. ish1301
    Locked

    How to block spam robots ?

    ish1301, Nov 14, 2006
    Replies:
    11
    Views:
    4,669
    ruby
    Apr 26, 2007
  27. rjdpa

    How large can the robots.txt file be?

    rjdpa, Apr 27, 2007
    Replies:
    3
    Views:
    1,620
    m1t0s1s
    Apr 27, 2007
  28. imparas

    What is the code.

    imparas, Apr 25, 2007
    Replies:
    10
    Views:
    1,548
    imparas
    Apr 27, 2007
  29. AHA7

    A few simple questions about robots.txt

    AHA7, Apr 29, 2007
    Replies:
    2
    Views:
    1,422
    AHA7
    Apr 29, 2007
  30. swiftmed
    Locked

    Bad?

    swiftmed, Feb 25, 2007
    Replies:
    4
    Views:
    1,405
    kgrad
    May 2, 2007
  31. rldowling03

    Letting robots browse forum posts

    rldowling03, Apr 19, 2007
    Replies:
    5
    Views:
    1,457
    rldowling03
    May 3, 2007
  32. AHA7

    Is this valid?

    AHA7, May 8, 2007
    Replies:
    1
    Views:
    1,073
    Spartan_Strategy
    May 10, 2007
  33. vic_msn

    Adding path to sitemaps in robots.txt

    vic_msn, Apr 26, 2007
    Replies:
    4
    Views:
    1,813
    trichnosis
    May 11, 2007
  34. sck

    Preventing Bad bots (robots)

    sck, May 9, 2007
    Replies:
    4
    Views:
    1,658
    sarahk
    May 12, 2007
  35. jgjg

    Please give me a robots.txt file to exclude as many search engines as possible!!

    jgjg, May 18, 2007
    Replies:
    12
    Views:
    1,809
    Christine8
    May 20, 2007
  36. Hostaff

    Excluding feeds in Wordpress

    Hostaff, May 22, 2007
    Replies:
    0
    Views:
    984
    Hostaff
    May 22, 2007
  37. visualscope

    block all except in robots.txt

    visualscope, May 26, 2007
    Replies:
    1
    Views:
    2,405
    tinkerbox
    May 27, 2007
  38. me4you

    I have some Question about robots.txt

    me4you, May 31, 2007
    Replies:
    2
    Views:
    1,286
    pommy
    May 31, 2007
  39. Dave Hybrid

    Blocking query strings in URLs

    Dave Hybrid, Jun 1, 2007
    Replies:
    8
    Views:
    2,681
    Dave Hybrid
    Jun 1, 2007
  40. Phishie

    Blocking /?Itemid=1

    Phishie, May 29, 2007
    Replies:
    2
    Views:
    1,467
    trichnosis
    Jun 8, 2007
  41. Mysmasken

    Prevent subdomain listings?

    Mysmasken, May 28, 2007
    Replies:
    4
    Views:
    1,404
    trichnosis
    Jun 8, 2007
  42. Phishie

    Can somebody look over this real fast?

    Phishie, Jun 11, 2007
    Replies:
    4
    Views:
    1,164
    kirby009
    Jun 12, 2007
  43. steveb

    Robots.txt Question

    steveb, Mar 21, 2007
    Replies:
    7
    Views:
    1,771
    kirby009
    Jun 12, 2007
  44. weknowtheworld

    robots.txt

    weknowtheworld, Mar 29, 2007
    Replies:
    5
    Views:
    1,751
    kirby009
    Jun 12, 2007
  45. papek

    My robots.txt file, is it too silly example?

    papek, Jun 7, 2007
    Replies:
    12
    Views:
    2,095
    DavidK1
    Jun 12, 2007
  46. markowe

    IP's in range 124.115.*.*

    markowe, Jun 19, 2007
    Replies:
    5
    Views:
    2,232
    DeViAnThans3
    Jun 27, 2007
  47. Phishie

    Disallow /images/ but allow /images/dir/dir. Is it possible?

    Phishie, Jun 21, 2007
    Replies:
    3
    Views:
    1,245
    trichnosis
    Jul 3, 2007
  48. paladin2

    How to stop Googlebot to spider this page?

    paladin2, Jun 25, 2007
    Replies:
    3
    Views:
    5,289
    trichnosis
    Jul 3, 2007
  49. paulinmargarita

    Is robots.txt really necessary??

    paulinmargarita, Jun 23, 2007
    Replies:
    14
    Views:
    2,331
    antman
    Jul 10, 2007
  50. vishwa

    Autodiscovery of Sitemaps Using robots.txt

    vishwa, Apr 19, 2007
    Replies:
    7
    Views:
    2,168
    danieloffice
    Jul 16, 2007
Showing threads 151 to 200 of 911

Thread Display Options

Loading...
Page 4 of 19