robots.txt

Find help with setting up your robots.txt file on your web site.

Page 3 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↑
  1. C. Szeler

    Googlebot not following the rules

    C. Szeler, Jun 25, 2006
    Replies:
    15
    Views:
    2,320
    mvandemar
    Jun 27, 2006
  2. nvidura

    If i restrict google bot?

    nvidura, Jun 29, 2006
    Replies:
    8
    Views:
    2,336
    Jean-Luc
    Jun 29, 2006
  3. beyondthegrave

    robots.txt with multiple websites on one webserver

    beyondthegrave, Jul 4, 2006
    Replies:
    3
    Views:
    1,850
    Jean-Luc
    Jul 4, 2006
  4. Ollie001

    When you ask a robot to return in 31 days..

    Ollie001, Jul 5, 2006
    Replies:
    7
    Views:
    2,044
    mad4
    Jul 6, 2006
  5. twelfty

    Help please: Flash then HTML then robots.txt ??

    twelfty, Jul 14, 2006
    Replies:
    0
    Views:
    1,493
    twelfty
    Jul 14, 2006
  6. websiteideas

    Bots that you disallow because they eat bandwidth

    websiteideas, Mar 3, 2006
    Replies:
    15
    Views:
    3,616
    websiteideas
    Jul 15, 2006
  7. ian_ok

    Hide content and remove from robots.txt????

    ian_ok, May 30, 2006
    Replies:
    4
    Views:
    2,124
    kirpy
    Jul 17, 2006
  8. CWN

    Robots.txt

    CWN, Jul 19, 2006
    Replies:
    13
    Views:
    2,862
    wrmineo
    Jul 20, 2006
  9. joeychgo

    robots.txt

    joeychgo, Nov 13, 2004
    Replies:
    6
    Views:
    3,486
    GiddyGamer
    Jul 25, 2006
  10. Pammer

    Robots.txt: Do you care that??

    Pammer, Jul 27, 2006
    Replies:
    1
    Views:
    1,517
    mad4
    Jul 27, 2006
  11. scylla

    how to block the bots going to this address?

    scylla, Jul 28, 2006
    Replies:
    1
    Views:
    1,616
    Cryogenius
    Jul 28, 2006
  12. Scarface.

    bots and my site

    Scarface., Jul 26, 2006
    Replies:
    2
    Views:
    1,360
    dagg3r
    Jul 31, 2006
  13. ritchie111

    bots viewing only half my site

    ritchie111, Jul 30, 2006
    Replies:
    2
    Views:
    1,360
    glennhefley
    Aug 4, 2006
  14. john269

    Using the robot.txt file to block the search engines

    john269, Aug 12, 2006
    Replies:
    4
    Views:
    2,990
    john269
    Aug 12, 2006
  15. zachbb

    spiders ignore robots.txt !

    zachbb, Jul 29, 2006
    Replies:
    3
    Views:
    1,842
    zachbb
    Aug 16, 2006
  16. racer22

    Help needed - Robots.txt file

    racer22, Aug 18, 2006
    Replies:
    5
    Views:
    2,550
    hans
    Aug 19, 2006
  17. john269

    About the search engine robot.txt file and SEO!

    john269, Aug 12, 2006
    Replies:
    19
    Views:
    3,388
    john269
    Aug 21, 2006
  18. Notorious

    Yahoo Ignoring robots.txt

    Notorious, Aug 21, 2006
    Replies:
    3
    Views:
    1,941
    Andrey Markin
    Aug 22, 2006
  19. SaleemY

    is this robots.txt ok??

    SaleemY, Sep 9, 2005
    Replies:
    6
    Views:
    6,093
    MLDesigners
    Aug 29, 2006
  20. klown

    google spiders webbing around robot tags

    klown, Jun 26, 2006
    Replies:
    3
    Views:
    1,875
    MLDesigners
    Aug 29, 2006
  21. White Monkey

    Questions about robots.txt...

    White Monkey, Aug 22, 2006
    Replies:
    5
    Views:
    1,687
    makeit easy
    Aug 29, 2006
  22. Laszlo

    Do I have to include the robots.txt

    Laszlo, Aug 12, 2006
    Replies:
    7
    Views:
    1,695
    dhruv37
    Aug 30, 2006
  23. evolving

    google indexing many pages incorrectly

    evolving, Aug 30, 2006
    Replies:
    2
    Views:
    1,389
    Jean-Luc
    Aug 30, 2006
  24. uma

    IS there any tool to find out the particular page got spidered?

    uma, Sep 1, 2006
    Replies:
    0
    Views:
    1,073
    uma
    Sep 1, 2006
  25. trichnosis

    one question about robots.txt

    trichnosis, Sep 3, 2006
    Replies:
    4
    Views:
    2,592
    Jean-Luc
    Sep 5, 2006
  26. northstar

    Robots.txt help

    northstar, Sep 12, 2006
    Replies:
    8
    Views:
    1,815
    Jean-Luc
    Sep 12, 2006
  27. sck

    spiders.text & robot.text

    sck, Aug 11, 2006
    Replies:
    3
    Views:
    15,584
    explorer
    Sep 15, 2006
  28. northstar

    Help with blocking duplicate content

    northstar, Sep 7, 2006
    Replies:
    4
    Views:
    1,375
    Jean-Luc
    Sep 17, 2006
  29. sarathy

    Removing a single Dynamic url

    sarathy, Sep 26, 2006
    Replies:
    3
    Views:
    1,577
    sarathy
    Oct 1, 2006
  30. murph

    Is it possible to have */

    murph, Sep 14, 2006
    Replies:
    1
    Views:
    1,178
    zerohex
    Oct 3, 2006
  31. Kingdom85

    Need a bit of help here please.

    Kingdom85, Oct 5, 2006
    Replies:
    4
    Views:
    1,230
    Kingdom85
    Oct 5, 2006
  32. goodfolks

    How do I block this bot?

    goodfolks, Oct 5, 2006
    Replies:
    3
    Views:
    1,785
    goodfolks
    Oct 5, 2006
  33. ColinR

    Robots.txt

    ColinR, Sep 24, 2004
    Replies:
    6
    Views:
    3,210
    TheSyndicate
    Oct 18, 2006
  34. mnymkr

    What should my robot.txt say?

    mnymkr, Oct 4, 2006
    Replies:
    7
    Views:
    3,794
    blabla11
    Nov 5, 2006
  35. bwb

    quick robots.txt question... please help :)

    bwb, Nov 11, 2006
    Replies:
    7
    Views:
    1,878
    bwb
    Nov 17, 2006
  36. ramki

    Restricting Search Engine Spider to crawl some portion of the webpage? Please Help!

    ramki, Nov 22, 2006
    Replies:
    1
    Views:
    2,496
    KC TAN
    Nov 22, 2006
  37. stephensaid

    Limiting search engines listing to main page

    stephensaid, Nov 24, 2006
    Replies:
    3
    Views:
    1,803
    stephensaid
    Nov 25, 2006
  38. 993ti

    Using an outbound script and robots.txt

    993ti, Nov 28, 2006
    Replies:
    1
    Views:
    1,986
    993ti
    Nov 28, 2006
  39. weknowtheworld

    Help required for understanding..

    weknowtheworld, Dec 9, 2006
    Replies:
    1
    Views:
    1,195
    Monty
    Dec 9, 2006
  40. lordmenace

    External URLS

    lordmenace, Sep 20, 2006
    Replies:
    6
    Views:
    1,670
    trichnosis
    Dec 18, 2006
  41. i30

    Simple bots file to block one section of a forum

    i30, Dec 18, 2006
    Replies:
    2
    Views:
    1,429
    i30
    Dec 18, 2006
  42. surekhaweb

    Is robots.txt required?

    surekhaweb, Dec 24, 2006
    Replies:
    8
    Views:
    1,902
    Monty
    Dec 24, 2006
  43. 1-script.com

    What in the world is bot/1.0 (bot; http://; bot@bot.bot)

    1-script.com, Dec 12, 2006
    Replies:
    1
    Views:
    4,266
    explorer
    Dec 26, 2006
  44. ASCFulfillment

    Robots.txt file, do i even need it?

    ASCFulfillment, Dec 14, 2006
    Replies:
    3
    Views:
    1,778
    explorer
    Dec 26, 2006
  45. minnseoelite

    Robots.txt

    minnseoelite, Nov 11, 2006
    Replies:
    2
    Views:
    1,603
    explorer
    Dec 26, 2006
  46. johan-cr

    How to delay/slow down googlebot?

    johan-cr, Dec 29, 2006
    Replies:
    4
    Views:
    3,205
    bizhobby
    Jan 1, 2007
  47. stocknsurf

    Is robots.txt needed for a totally dynamic website?

    stocknsurf, Jan 9, 2007
    Replies:
    2
    Views:
    1,740
    stocknsurf
    Jan 9, 2007
  48. skionxb

    Wow is ROBOTS.txt important?

    skionxb, Jul 26, 2006 ... 2
    Replies:
    26
    Views:
    9,366
    Dan_Grr
    Jan 12, 2007
  49. sarathy

    Good bad and Ugly

    sarathy, Dec 2, 2006
    Replies:
    3
    Views:
    1,640
    zumo
    Jan 14, 2007
  50. Arcos

    Including a folder within a Disallow folder?

    Arcos, Jan 15, 2007
    Replies:
    1
    Views:
    1,471
    Jean-Luc
    Jan 16, 2007
Showing threads 101 to 150 of 911

Thread Display Options

Loading...
Page 3 of 19