1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 2 of 19
Sort by:
Title Start Date
Replies ↓ Views
Last Message
  1. sarathy

    How to remove this dynamic url using robots.txt from SE's

    sarathy, Oct 13, 2005
    Replies:
    15
    Views:
    8,475
    C. Berntsen
    Jul 17, 2008
  2. Giorgi

    how to completly block engine from crawling my site?

    Giorgi, Jan 16, 2008
    Replies:
    15
    Views:
    2,001
    manish.chauhan
    Apr 7, 2008
  3. websiteideas

    Bots that you disallow because they eat bandwidth

    websiteideas, Mar 3, 2006
    Replies:
    15
    Views:
    3,415
    websiteideas
    Jul 15, 2006
  4. C. Szeler

    Googlebot not following the rules

    C. Szeler, Jun 25, 2006
    Replies:
    15
    Views:
    2,140
    mvandemar
    Jun 27, 2006
  5. jordan5815

    Robots.txt

    jordan5815, Apr 11, 2006
    Replies:
    15
    Views:
    2,016
    Crazy_Rob
    Apr 14, 2006
  6. R0ck$tAr

    Is This A Bad Bot?

    R0ck$tAr, Apr 12, 2006
    Replies:
    15
    Views:
    2,007
    exam
    Apr 13, 2006
  7. hmansfield

    List of Bad Bots to disallow?

    hmansfield, May 20, 2007
    Replies:
    14
    Views:
    24,398
    Sergio Minozzi
    Feb 22, 2017
  8. abrahamgold

    Hello, I am new, and need some help on robots.txt

    abrahamgold, Oct 21, 2005
    Replies:
    14
    Views:
    8,853
    SPA assurance
    Mar 8, 2012
  9. Arcos

    How to STOP indexing web pages

    Arcos, Dec 23, 2005
    Replies:
    14
    Views:
    8,225
    jhon786
    Aug 25, 2011
  10. harryoxford

    Robots.txt

    harryoxford, Feb 10, 2011
    Replies:
    14
    Views:
    3,619
    manish.chauhan
    May 4, 2011
  11. Sumitthewriter

    Can I Hide Duplicate Content By Robots.txt

    Sumitthewriter, Oct 28, 2010
    Replies:
    14
    Views:
    3,625
    ACME Squares
    Mar 15, 2011
  12. Coldfaun

    Problem with robots.txt

    Coldfaun, Jul 12, 2010
    Replies:
    14
    Views:
    1,603
    Groovystar
    Jul 17, 2010
  13. AzureHaze

    Disallow All Pages Except Homepage

    AzureHaze, Jul 22, 2009
    Replies:
    14
    Views:
    4,485
    jabz.biz
    Sep 8, 2009
  14. amedhussaini

    should i disallow images?

    amedhussaini, Dec 1, 2006
    Replies:
    14
    Views:
    9,171
    sarpras2005@yahoo.co.in
    Aug 11, 2009
  15. login

    Avoid indexing of ../lyrics/print,74306.html

    login, Jan 11, 2008
    Replies:
    14
    Views:
    1,342
    SwapsRulez
    Jan 14, 2008
  16. paulinmargarita

    Is robots.txt really necessary??

    paulinmargarita, Jun 23, 2007
    Replies:
    14
    Views:
    2,110
    antman
    Jul 10, 2007
  17. tweetylover8402

    Robots.txt?

    tweetylover8402, Oct 25, 2005
    Replies:
    14
    Views:
    2,332
    jazzylee77
    Nov 13, 2005
  18. style fashion

    Best Robots.txt For Wordpress Blog ....

    style fashion, Oct 1, 2009
    Replies:
    13
    Views:
    10,527
    mohisha
    May 10, 2011
  19. saadhome2

    What Is Use Of Robot Text File.

    saadhome2, Mar 24, 2010
    Replies:
    13
    Views:
    2,271
    seo-india
    Aug 27, 2010
  20. mithu.singh
    Locked

    What is robot.txt files.

    mithu.singh, May 22, 2010
    Replies:
    13
    Views:
    1,162
    goldenminds
    May 25, 2010
  21. girbaud

    Unauthorized bots?

    girbaud, Jan 17, 2006
    Replies:
    13
    Views:
    2,619
    linkdealer
    Jun 8, 2009
  22. jitendraag

    America's new robots.txt

    jitendraag, Jan 20, 2009
    Replies:
    13
    Views:
    1,539
    frih
    Feb 2, 2009
  23. Owlcroft

    Stopping/Discouraging SpamBots.

    Owlcroft, Jan 14, 2005
    Replies:
    13
    Views:
    6,357
    Quench
    Aug 7, 2008
  24. nonflasher

    How to put robots.txt

    nonflasher, Feb 9, 2006
    Replies:
    13
    Views:
    3,366
    Hunnigs
    Jun 6, 2008
  25. jtothek3030

    How do i stop Internet Shinchakubin?

    jtothek3030, May 11, 2007
    Replies:
    13
    Views:
    4,470
    scoopy82
    Mar 10, 2008
  26. compindustries

    Help with robots.txt file

    compindustries, Jun 19, 2006
    Replies:
    13
    Views:
    2,846
    muks
    Nov 1, 2007
  27. CWN

    Robots.txt

    CWN, Jul 19, 2006
    Replies:
    13
    Views:
    2,627
    wrmineo
    Jul 20, 2006
  28. gedarnet

    i'm need best robots.txt for wordpress photographers

    gedarnet, Nov 18, 2012
    Replies:
    12
    Views:
    2,977
    gedarnet
    Dec 7, 2012
  29. aj845

    unknown bots.

    aj845, Jul 6, 2007
    Replies:
    12
    Views:
    11,122
    cquinndesign
    Jul 3, 2011
  30. gravy834

    Robots.txt vs <meta name="robots" content="noindex,nofollow">

    gravy834, Mar 2, 2009
    Replies:
    12
    Views:
    8,527
    fsdnetwork
    Feb 9, 2011
  31. hectoth

    Robots.txt

    hectoth, Oct 7, 2010
    Replies:
    12
    Views:
    2,280
    Pronto! SEO
    Oct 28, 2010
  32. eniva

    Best Robots.txt For Wordpress Installed Blogs

    eniva, Sep 23, 2008
    Replies:
    12
    Views:
    10,799
    seo.lover
    Aug 29, 2010
  33. kevoh99

    Is robots.txt really safe?

    kevoh99, Jul 17, 2010
    Replies:
    12
    Views:
    2,635
    seo.lover
    Aug 29, 2010
  34. YSNY

    help with my robots.txt file

    YSNY, Sep 25, 2009
    Replies:
    12
    Views:
    2,827
    more_sem
    Mar 14, 2010
  35. Rian

    Free Online Robots.txt Generator

    Rian, Sep 26, 2008
    Replies:
    12
    Views:
    5,462
    kabo0m
    Feb 23, 2009
  36. birdsq

    Giv me an intro to robots.txt

    birdsq, Aug 29, 2007
    Replies:
    12
    Views:
    1,763
    manish.chauhan
    Jan 20, 2009
  37. raptrex

    hows my robots.txt

    raptrex, Jul 11, 2008
    Replies:
    12
    Views:
    2,102
    Jean-Luc
    Sep 29, 2008
  38. laxmi

    robots.txt?

    laxmi, Jun 28, 2008
    Replies:
    12
    Views:
    1,270
    laxmi
    Jul 5, 2008
  39. ThomaSeidler

    Help in encouraging Yahoo and MSN robots to obedience

    ThomaSeidler, Jun 14, 2007
    Replies:
    12
    Views:
    2,556
    evera
    Aug 28, 2007
  40. papek

    My robots.txt file, is it too silly example?

    papek, Jun 7, 2007
    Replies:
    12
    Views:
    1,811
    DavidK1
    Jun 12, 2007
  41. jgjg

    Please give me a robots.txt file to exclude as many search engines as possible!!

    jgjg, May 18, 2007
    Replies:
    12
    Views:
    1,702
    Christine8
    May 20, 2007
  42. THT

    Robots.txt question

    THT, Jun 5, 2005
    Replies:
    12
    Views:
    2,640
    ZuraX
    Jun 6, 2005
  43. TheSyndicate

    Is it normal that the /robots.txt is the most downloaded item from Server stats

    TheSyndicate, Mar 29, 2012
    Replies:
    11
    Views:
    2,610
    TheSyndicate
    Apr 9, 2012
  44. OSSEO

    How to restrict url from robots.txt file

    OSSEO, Sep 7, 2011
    Replies:
    11
    Views:
    3,838
    MyWebsiteNow
    Oct 17, 2011
  45. covetousrat

    Can someone check my Robot.txt My site is still not indexed yet.

    covetousrat, Mar 27, 2011
    Replies:
    11
    Views:
    4,227
    us2006
    May 5, 2011
  46. pradeep.balua

    how to install a robots.txt for a wordpress blog ?

    pradeep.balua, Jan 22, 2008
    Replies:
    11
    Views:
    4,657
    ayushi infotech
    Dec 28, 2010
  47. Talker

    Robots.txt for Vbulletin

    Talker, Oct 13, 2006
    Replies:
    11
    Views:
    6,674
    zinghana
    Jul 31, 2010
  48. remember123

    Best robots.txt for vBulletin forums

    remember123, Aug 22, 2009
    Replies:
    11
    Views:
    6,713
    Brandon Sheley
    Jan 8, 2010
  49. jkapadia1983

    Google Crawler

    jkapadia1983, Oct 1, 2009
    Replies:
    11
    Views:
    3,387
    redesignunit
    Dec 4, 2009
  50. rayqsl

    Do I Need A Robots.txt?

    rayqsl, Oct 13, 2007
    Replies:
    11
    Views:
    1,594
    Kuldeep1952
    Nov 2, 2007
Showing threads 51 to 100 of 911

Thread Display Options

Loading...
Page 2 of 19