robots.txt

Find help with setting up your robots.txt file on your web site.

Page 16 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. skionxb

    BLOCKED ROBOTS.txt and CUSTOM 404 PAGE ISSUE

    skionxb, Jul 27, 2006
    Replies:
    9
    Views:
    3,522
    WebGeek182
    Feb 22, 2007
  2. relixx

    robots.txt and blocking querystrings

    relixx, Feb 15, 2007
    Replies:
    3
    Views:
    1,592
    relixx
    Feb 20, 2007
  3. eXe

    Question

    eXe, Feb 15, 2007
    Replies:
    3
    Views:
    1,330
    Jean-Luc
    Feb 20, 2007
  4. Ogden2k

    robots.txt question

    Ogden2k, Feb 5, 2007
    Replies:
    1
    Views:
    1,502
    nevetS
    Feb 5, 2007
  5. alarik

    google indexed my website :(what did i do wrong?

    alarik, Jan 1, 2007
    Replies:
    6
    Views:
    2,093
    alarik
    Jan 31, 2007
  6. m0rtal

    Would this work?

    m0rtal, Jan 29, 2007
    Replies:
    4
    Views:
    1,597
    m0rtal
    Jan 30, 2007
  7. mvhs

    How to block spiders?

    mvhs, Jan 20, 2007
    Replies:
    8
    Views:
    4,696
    jonbt
    Jan 30, 2007
  8. circuscircus
    Locked

    How important is robot.txt in SEO?

    circuscircus, Jan 21, 2007
    Replies:
    5
    Views:
    2,008
    jonbt
    Jan 30, 2007
  9. Jon12345

    Help me design a simple robots file!

    Jon12345, Jan 8, 2007
    Replies:
    6
    Views:
    1,592
    jonbt
    Jan 30, 2007
  10. rusting

    please help me!!!

    rusting, Jan 22, 2007
    Replies:
    4
    Views:
    1,266
    jonbt
    Jan 30, 2007
  11. pr0xy122

    What is robots.txt and should I modify it?

    pr0xy122, Oct 25, 2006
    Replies:
    8
    Views:
    3,445
    jonbt
    Jan 30, 2007
  12. Arcos

    Including a folder within a Disallow folder?

    Arcos, Jan 15, 2007
    Replies:
    1
    Views:
    1,477
    Jean-Luc
    Jan 16, 2007
  13. sarathy

    Good bad and Ugly

    sarathy, Dec 2, 2006
    Replies:
    3
    Views:
    1,662
    zumo
    Jan 14, 2007
  14. skionxb

    Wow is ROBOTS.txt important?

    skionxb, Jul 26, 2006 ... 2
    Replies:
    26
    Views:
    9,528
    Dan_Grr
    Jan 12, 2007
  15. stocknsurf

    Is robots.txt needed for a totally dynamic website?

    stocknsurf, Jan 9, 2007
    Replies:
    2
    Views:
    1,751
    stocknsurf
    Jan 9, 2007
  16. johan-cr

    How to delay/slow down googlebot?

    johan-cr, Dec 29, 2006
    Replies:
    4
    Views:
    3,218
    bizhobby
    Jan 1, 2007
  17. minnseoelite

    Robots.txt

    minnseoelite, Nov 11, 2006
    Replies:
    2
    Views:
    1,623
    explorer
    Dec 26, 2006
  18. ASCFulfillment

    Robots.txt file, do i even need it?

    ASCFulfillment, Dec 14, 2006
    Replies:
    3
    Views:
    1,807
    explorer
    Dec 26, 2006
  19. 1-script.com

    What in the world is bot/1.0 (bot; http://; bot@bot.bot)

    1-script.com, Dec 12, 2006
    Replies:
    1
    Views:
    4,280
    explorer
    Dec 26, 2006
  20. surekhaweb

    Is robots.txt required?

    surekhaweb, Dec 24, 2006
    Replies:
    8
    Views:
    1,953
    Monty
    Dec 24, 2006
  21. i30

    Simple bots file to block one section of a forum

    i30, Dec 18, 2006
    Replies:
    2
    Views:
    1,450
    i30
    Dec 18, 2006
  22. lordmenace

    External URLS

    lordmenace, Sep 20, 2006
    Replies:
    6
    Views:
    1,703
    trichnosis
    Dec 18, 2006
  23. weknowtheworld

    Help required for understanding..

    weknowtheworld, Dec 9, 2006
    Replies:
    1
    Views:
    1,208
    Monty
    Dec 9, 2006
  24. 993ti

    Using an outbound script and robots.txt

    993ti, Nov 28, 2006
    Replies:
    1
    Views:
    1,998
    993ti
    Nov 28, 2006
  25. stephensaid

    Limiting search engines listing to main page

    stephensaid, Nov 24, 2006
    Replies:
    3
    Views:
    1,820
    stephensaid
    Nov 25, 2006
  26. ramki

    Restricting Search Engine Spider to crawl some portion of the webpage? Please Help!

    ramki, Nov 22, 2006
    Replies:
    1
    Views:
    2,509
    KC TAN
    Nov 22, 2006
  27. bwb

    quick robots.txt question... please help :)

    bwb, Nov 11, 2006
    Replies:
    7
    Views:
    1,928
    bwb
    Nov 17, 2006
  28. mnymkr

    What should my robot.txt say?

    mnymkr, Oct 4, 2006
    Replies:
    7
    Views:
    3,823
    blabla11
    Nov 5, 2006
  29. ColinR

    Robots.txt

    ColinR, Sep 24, 2004
    Replies:
    6
    Views:
    3,231
    TheSyndicate
    Oct 18, 2006
  30. goodfolks

    How do I block this bot?

    goodfolks, Oct 5, 2006
    Replies:
    3
    Views:
    1,802
    goodfolks
    Oct 5, 2006
  31. Kingdom85

    Need a bit of help here please.

    Kingdom85, Oct 5, 2006
    Replies:
    4
    Views:
    1,253
    Kingdom85
    Oct 5, 2006
  32. murph

    Is it possible to have */

    murph, Sep 14, 2006
    Replies:
    1
    Views:
    1,197
    zerohex
    Oct 3, 2006
  33. sarathy

    Removing a single Dynamic url

    sarathy, Sep 26, 2006
    Replies:
    3
    Views:
    1,593
    sarathy
    Oct 1, 2006
  34. northstar

    Help with blocking duplicate content

    northstar, Sep 7, 2006
    Replies:
    4
    Views:
    1,396
    Jean-Luc
    Sep 17, 2006
  35. sck

    spiders.text & robot.text

    sck, Aug 11, 2006
    Replies:
    3
    Views:
    15,634
    explorer
    Sep 15, 2006
  36. northstar

    Robots.txt help

    northstar, Sep 12, 2006
    Replies:
    8
    Views:
    1,856
    Jean-Luc
    Sep 12, 2006
  37. trichnosis

    one question about robots.txt

    trichnosis, Sep 3, 2006
    Replies:
    4
    Views:
    2,617
    Jean-Luc
    Sep 5, 2006
  38. uma

    IS there any tool to find out the particular page got spidered?

    uma, Sep 1, 2006
    Replies:
    0
    Views:
    1,098
    uma
    Sep 1, 2006
  39. evolving

    google indexing many pages incorrectly

    evolving, Aug 30, 2006
    Replies:
    2
    Views:
    1,408
    Jean-Luc
    Aug 30, 2006
  40. Laszlo

    Do I have to include the robots.txt

    Laszlo, Aug 12, 2006
    Replies:
    7
    Views:
    1,720
    dhruv37
    Aug 30, 2006
  41. White Monkey

    Questions about robots.txt...

    White Monkey, Aug 22, 2006
    Replies:
    5
    Views:
    1,715
    makeit easy
    Aug 29, 2006
  42. klown

    google spiders webbing around robot tags

    klown, Jun 26, 2006
    Replies:
    3
    Views:
    1,894
    MLDesigners
    Aug 29, 2006
  43. SaleemY

    is this robots.txt ok??

    SaleemY, Sep 9, 2005
    Replies:
    6
    Views:
    6,150
    MLDesigners
    Aug 29, 2006
  44. Notorious

    Yahoo Ignoring robots.txt

    Notorious, Aug 21, 2006
    Replies:
    3
    Views:
    1,983
    Andrey Markin
    Aug 22, 2006
  45. john269

    About the search engine robot.txt file and SEO!

    john269, Aug 12, 2006
    Replies:
    19
    Views:
    3,462
    john269
    Aug 21, 2006
  46. racer22

    Help needed - Robots.txt file

    racer22, Aug 18, 2006
    Replies:
    5
    Views:
    2,630
    hans
    Aug 19, 2006
  47. zachbb

    spiders ignore robots.txt !

    zachbb, Jul 29, 2006
    Replies:
    3
    Views:
    1,860
    zachbb
    Aug 16, 2006
  48. john269

    Using the robot.txt file to block the search engines

    john269, Aug 12, 2006
    Replies:
    4
    Views:
    3,006
    john269
    Aug 12, 2006
  49. ritchie111

    bots viewing only half my site

    ritchie111, Jul 30, 2006
    Replies:
    2
    Views:
    1,377
    glennhefley
    Aug 4, 2006
  50. Scarface.

    bots and my site

    Scarface., Jul 26, 2006
    Replies:
    2
    Views:
    1,390
    dagg3r
    Jul 31, 2006
Showing threads 751 to 800 of 911

Thread Display Options

Loading...
Page 16 of 19