1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 3 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↑
  1. C. Szeler

    Googlebot not following the rules

    C. Szeler, Jun 25, 2006
    Replies:
    15
    Views:
    2,140
    mvandemar
    Jun 27, 2006
  2. nvidura

    If i restrict google bot?

    nvidura, Jun 29, 2006
    Replies:
    8
    Views:
    2,191
    Jean-Luc
    Jun 29, 2006
  3. beyondthegrave

    robots.txt with multiple websites on one webserver

    beyondthegrave, Jul 4, 2006
    Replies:
    3
    Views:
    1,736
    Jean-Luc
    Jul 4, 2006
  4. Ollie001

    When you ask a robot to return in 31 days..

    Ollie001, Jul 5, 2006
    Replies:
    7
    Views:
    1,833
    mad4
    Jul 6, 2006
  5. twelfty

    Help please: Flash then HTML then robots.txt ??

    twelfty, Jul 14, 2006
    Replies:
    0
    Views:
    1,410
    twelfty
    Jul 14, 2006
  6. websiteideas

    Bots that you disallow because they eat bandwidth

    websiteideas, Mar 3, 2006
    Replies:
    15
    Views:
    3,415
    websiteideas
    Jul 15, 2006
  7. ian_ok

    Hide content and remove from robots.txt????

    ian_ok, May 30, 2006
    Replies:
    4
    Views:
    1,989
    kirpy
    Jul 17, 2006
  8. CWN

    Robots.txt

    CWN, Jul 19, 2006
    Replies:
    13
    Views:
    2,627
    wrmineo
    Jul 20, 2006
  9. joeychgo

    robots.txt

    joeychgo, Nov 13, 2004
    Replies:
    6
    Views:
    3,238
    GiddyGamer
    Jul 25, 2006
  10. Pammer

    Robots.txt: Do you care that??

    Pammer, Jul 27, 2006
    Replies:
    1
    Views:
    1,431
    mad4
    Jul 27, 2006
  11. scylla

    how to block the bots going to this address?

    scylla, Jul 28, 2006
    Replies:
    1
    Views:
    1,495
    Cryogenius
    Jul 28, 2006
  12. Scarface.

    bots and my site

    Scarface., Jul 26, 2006
    Replies:
    2
    Views:
    1,262
    dagg3r
    Jul 31, 2006
  13. ritchie111

    bots viewing only half my site

    ritchie111, Jul 30, 2006
    Replies:
    2
    Views:
    1,285
    glennhefley
    Aug 4, 2006
  14. john269

    Using the robot.txt file to block the search engines

    john269, Aug 12, 2006
    Replies:
    4
    Views:
    2,915
    john269
    Aug 12, 2006
  15. zachbb

    spiders ignore robots.txt !

    zachbb, Jul 29, 2006
    Replies:
    3
    Views:
    1,767
    zachbb
    Aug 16, 2006
  16. racer22

    Help needed - Robots.txt file

    racer22, Aug 18, 2006
    Replies:
    5
    Views:
    2,307
    hans
    Aug 19, 2006
  17. john269

    About the search engine robot.txt file and SEO!

    john269, Aug 12, 2006
    Replies:
    19
    Views:
    3,076
    john269
    Aug 21, 2006
  18. Notorious

    Yahoo Ignoring robots.txt

    Notorious, Aug 21, 2006
    Replies:
    3
    Views:
    1,801
    Andrey Markin
    Aug 22, 2006
  19. SaleemY

    is this robots.txt ok??

    SaleemY, Sep 9, 2005
    Replies:
    6
    Views:
    5,853
    MLDesigners
    Aug 29, 2006
  20. klown

    google spiders webbing around robot tags

    klown, Jun 26, 2006
    Replies:
    3
    Views:
    1,785
    MLDesigners
    Aug 29, 2006
  21. White Monkey

    Questions about robots.txt...

    White Monkey, Aug 22, 2006
    Replies:
    5
    Views:
    1,610
    makeit easy
    Aug 29, 2006
  22. Laszlo

    Do I have to include the robots.txt

    Laszlo, Aug 12, 2006
    Replies:
    7
    Views:
    1,602
    dhruv37
    Aug 30, 2006
  23. evolving

    google indexing many pages incorrectly

    evolving, Aug 30, 2006
    Replies:
    2
    Views:
    1,292
    Jean-Luc
    Aug 30, 2006
  24. uma

    IS there any tool to find out the particular page got spidered?

    uma, Sep 1, 2006
    Replies:
    0
    Views:
    1,005
    uma
    Sep 1, 2006
  25. trichnosis

    one question about robots.txt

    trichnosis, Sep 3, 2006
    Replies:
    4
    Views:
    2,465
    Jean-Luc
    Sep 5, 2006
  26. northstar

    Robots.txt help

    northstar, Sep 12, 2006
    Replies:
    8
    Views:
    1,705
    Jean-Luc
    Sep 12, 2006
  27. sck

    spiders.text & robot.text

    sck, Aug 11, 2006
    Replies:
    3
    Views:
    15,438
    explorer
    Sep 15, 2006
  28. northstar

    Help with blocking duplicate content

    northstar, Sep 7, 2006
    Replies:
    4
    Views:
    1,315
    Jean-Luc
    Sep 17, 2006
  29. sarathy

    Removing a single Dynamic url

    sarathy, Sep 26, 2006
    Replies:
    3
    Views:
    1,468
    sarathy
    Oct 1, 2006
  30. murph

    Is it possible to have */

    murph, Sep 14, 2006
    Replies:
    1
    Views:
    1,108
    zerohex
    Oct 3, 2006
  31. Kingdom85

    Need a bit of help here please.

    Kingdom85, Oct 5, 2006
    Replies:
    4
    Views:
    1,153
    Kingdom85
    Oct 5, 2006
  32. goodfolks

    How do I block this bot?

    goodfolks, Oct 5, 2006
    Replies:
    3
    Views:
    1,585
    goodfolks
    Oct 5, 2006
  33. ColinR

    Robots.txt

    ColinR, Sep 24, 2004
    Replies:
    6
    Views:
    3,128
    TheSyndicate
    Oct 18, 2006
  34. mnymkr

    What should my robot.txt say?

    mnymkr, Oct 4, 2006
    Replies:
    7
    Views:
    3,707
    blabla11
    Nov 5, 2006
  35. bwb

    quick robots.txt question... please help :)

    bwb, Nov 11, 2006
    Replies:
    7
    Views:
    1,704
    bwb
    Nov 17, 2006
  36. ramki

    Restricting Search Engine Spider to crawl some portion of the webpage? Please Help!

    ramki, Nov 22, 2006
    Replies:
    1
    Views:
    2,449
    KC TAN
    Nov 22, 2006
  37. stephensaid

    Limiting search engines listing to main page

    stephensaid, Nov 24, 2006
    Replies:
    3
    Views:
    1,730
    stephensaid
    Nov 25, 2006
  38. 993ti

    Using an outbound script and robots.txt

    993ti, Nov 28, 2006
    Replies:
    1
    Views:
    1,917
    993ti
    Nov 28, 2006
  39. weknowtheworld

    Help required for understanding..

    weknowtheworld, Dec 9, 2006
    Replies:
    1
    Views:
    1,147
    Monty
    Dec 9, 2006
  40. lordmenace

    External URLS

    lordmenace, Sep 20, 2006
    Replies:
    6
    Views:
    1,536
    trichnosis
    Dec 18, 2006
  41. i30

    Simple bots file to block one section of a forum

    i30, Dec 18, 2006
    Replies:
    2
    Views:
    1,356
    i30
    Dec 18, 2006
  42. surekhaweb

    Is robots.txt required?

    surekhaweb, Dec 24, 2006
    Replies:
    8
    Views:
    1,742
    Monty
    Dec 24, 2006
  43. 1-script.com

    What in the world is bot/1.0 (bot; http://; bot@bot.bot)

    1-script.com, Dec 12, 2006
    Replies:
    1
    Views:
    4,178
    explorer
    Dec 26, 2006
  44. ASCFulfillment

    Robots.txt file, do i even need it?

    ASCFulfillment, Dec 14, 2006
    Replies:
    3
    Views:
    1,673
    explorer
    Dec 26, 2006
  45. minnseoelite

    Robots.txt

    minnseoelite, Nov 11, 2006
    Replies:
    2
    Views:
    1,501
    explorer
    Dec 26, 2006
  46. johan-cr

    How to delay/slow down googlebot?

    johan-cr, Dec 29, 2006
    Replies:
    4
    Views:
    3,063
    bizhobby
    Jan 1, 2007
  47. stocknsurf

    Is robots.txt needed for a totally dynamic website?

    stocknsurf, Jan 9, 2007
    Replies:
    2
    Views:
    1,633
    stocknsurf
    Jan 9, 2007
  48. skionxb

    Wow is ROBOTS.txt important?

    skionxb, Jul 26, 2006 ... 2
    Replies:
    26
    Views:
    8,837
    Dan_Grr
    Jan 12, 2007
  49. sarathy

    Good bad and Ugly

    sarathy, Dec 2, 2006
    Replies:
    3
    Views:
    1,528
    zumo
    Jan 14, 2007
  50. Arcos

    Including a folder within a Disallow folder?

    Arcos, Jan 15, 2007
    Replies:
    1
    Views:
    1,386
    Jean-Luc
    Jan 16, 2007
Showing threads 101 to 150 of 911

Thread Display Options

Loading...
Page 3 of 19