1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 5 of 19
Sort by:
Title Start Date
Replies ↓ Views
Last Message
  1. z3nyt

    Is robots.txt available for all engines?

    z3nyt, Jan 19, 2009
    Replies:
    7
    Views:
    1,292
    manish.chauhan
    Feb 4, 2009
  2. kie

    Prohibit SE to crawl subdomain

    kie, Jan 23, 2008
    Replies:
    7
    Views:
    1,575
    manish.chauhan
    Apr 7, 2008
  3. ialwaysforget

    robot.txt help

    ialwaysforget, Mar 13, 2008
    Replies:
    7
    Views:
    1,421
    manish.chauhan
    Apr 6, 2008
  4. crazyryan

    Stop ANY search engine indexing my site

    crazyryan, Feb 2, 2008
    Replies:
    7
    Views:
    1,961
    manish.chauhan
    Apr 4, 2008
  5. artboy

    How to get robots.txt re-crawled by google/yahoo, etc?

    artboy, Mar 10, 2008
    Replies:
    7
    Views:
    1,461
    manish.chauhan
    Apr 4, 2008
  6. Jinx3000

    First time setting up robots.txt

    Jinx3000, Jan 14, 2008
    Replies:
    7
    Views:
    1,320
    catanich
    Feb 2, 2008
  7. Jon12345

    What does this robots.txt text mean?

    Jon12345, Jan 8, 2008
    Replies:
    7
    Views:
    1,732
    catanich
    Feb 2, 2008
  8. frio

    Disallow

    frio, Dec 20, 2007
    Replies:
    7
    Views:
    2,111
    catanich
    Feb 2, 2008
  9. lowprofilejoe

    Disallow Flash Files

    lowprofilejoe, Jan 4, 2008
    Replies:
    7
    Views:
    1,877
    SwapsRulez
    Jan 12, 2008
  10. eathosting.com

    What is a Robot???

    eathosting.com, Dec 1, 2007
    Replies:
    7
    Views:
    1,387
    gab1982
    Dec 20, 2007
  11. softarea51

    Limit bot access frequency

    softarea51, Nov 1, 2007
    Replies:
    7
    Views:
    5,228
    ajsa52
    Nov 5, 2007
  12. bad_bob00

    Error pages on my site - robots.txt ?

    bad_bob00, Sep 21, 2007
    Replies:
    7
    Views:
    1,492
    Nucca_The_Warrior
    Oct 22, 2007
  13. othonas

    I really need your help

    othonas, Jul 19, 2007
    Replies:
    7
    Views:
    1,250
    trichnosis
    Aug 6, 2007
  14. SaN-DeeP

    psbot and gigabot wont stop even when banned from robots.txt

    SaN-DeeP, Feb 28, 2006
    Replies:
    7
    Views:
    3,750
    trichnosis
    Aug 6, 2007
  15. vishwa

    Autodiscovery of Sitemaps Using robots.txt

    vishwa, Apr 19, 2007
    Replies:
    7
    Views:
    1,971
    danieloffice
    Jul 16, 2007
  16. steveb

    Robots.txt Question

    steveb, Mar 21, 2007
    Replies:
    7
    Views:
    1,615
    kirby009
    Jun 12, 2007
  17. srinet

    Robot.txt

    srinet, Jan 19, 2007
    Replies:
    7
    Views:
    2,077
    RECEP
    Apr 11, 2007
  18. bwb

    quick robots.txt question... please help :)

    bwb, Nov 11, 2006
    Replies:
    7
    Views:
    1,703
    bwb
    Nov 17, 2006
  19. mnymkr

    What should my robot.txt say?

    mnymkr, Oct 4, 2006
    Replies:
    7
    Views:
    3,707
    blabla11
    Nov 5, 2006
  20. Laszlo

    Do I have to include the robots.txt

    Laszlo, Aug 12, 2006
    Replies:
    7
    Views:
    1,602
    dhruv37
    Aug 30, 2006
  21. Ollie001

    When you ask a robot to return in 31 days..

    Ollie001, Jul 5, 2006
    Replies:
    7
    Views:
    1,833
    mad4
    Jul 6, 2006
  22. steve_gts

    Which Robots text?

    steve_gts, Feb 28, 2006
    Replies:
    7
    Views:
    1,363
    hans
    Feb 28, 2006
  23. Tim_Myth

    Referrer logs show odd referrer: somesite.com\robots.txt

    Tim_Myth, Feb 3, 2006
    Replies:
    7
    Views:
    3,611
    Tim_Myth
    Feb 5, 2006
  24. rockinaway

    Robots.txt?

    rockinaway, Jan 13, 2006
    Replies:
    7
    Views:
    1,581
    rockinaway
    Jan 14, 2006
  25. aardvark

    NOINDEX valid in robots.txt?

    aardvark, Sep 21, 2005
    Replies:
    7
    Views:
    1,967
    webmistress
    Sep 24, 2005
  26. Mr.Dog

    Ban subdomain from robots, stop Google from indexing it

    Mr.Dog, Dec 1, 2018
    Replies:
    6
    Views:
    4,558
    Borislav Arapchev
    May 13, 2020
  27. macau2009

    Google or robots.txt problem

    macau2009, Feb 19, 2018
    Replies:
    6
    Views:
    4,745
    qwikad.com
    Feb 21, 2018
  28. wartes

    Robot.txt for Google (error)

    wartes, Apr 10, 2017
    Replies:
    6
    Views:
    13,283
    Md.Saddam Hossain
    May 7, 2017
  29. Mubashir Ahmad

    Robot.txt or remove the page

    Mubashir Ahmad, Mar 21, 2017
    Replies:
    6
    Views:
    6,194
    Tutorials Feed
    Apr 19, 2017
  30. Abdullah adil
    Locked

    robot.txt

    Abdullah adil, Feb 11, 2017
    Replies:
    6
    Views:
    13,742
    sarahk
    Feb 16, 2017
  31. Vel

    I need help on how to Noindex categories, tags, author, archive

    Vel, Jul 25, 2013
    Replies:
    6
    Views:
    8,713
    tymsh
    Dec 9, 2013
  32. msaeedkk

    disallow:/?

    msaeedkk, Mar 7, 2013
    Replies:
    6
    Views:
    2,188
    ryan_uk
    Mar 14, 2013
  33. Deepak_Kotian

    Robot.txt beginner

    Deepak_Kotian, Nov 1, 2012
    Replies:
    6
    Views:
    2,795
    agitetech
    Dec 27, 2012
  34. Adhe Carere

    Problem With Robot.txt

    Adhe Carere, May 30, 2012
    Replies:
    6
    Views:
    3,833
    kashishjain
    Nov 24, 2012
  35. Sunil Keshari

    Robot.txt file crawling pages very fast

    Sunil Keshari, Apr 26, 2012
    Replies:
    6
    Views:
    4,257
    selectaupairs
    Oct 30, 2012
  36. erebussymk

    Is it go to index wordpress Tag or not ?

    erebussymk, Oct 11, 2011
    Replies:
    6
    Views:
    4,361
    manish.chauhan
    Oct 12, 2012
  37. Alex Decosta

    How Robots.txt works

    Alex Decosta, Apr 14, 2012
    Replies:
    6
    Views:
    4,048
    XpErT
    Aug 5, 2012
  38. thundercow

    How to use Robots.txt to get only front page indexed?

    thundercow, Dec 12, 2008
    Replies:
    6
    Views:
    4,330
    Jason0078
    Jul 30, 2012
  39. TheSyndicate

    How do you get archiver to READ my robots?

    TheSyndicate, Mar 31, 2012
    Replies:
    6
    Views:
    3,149
    TheSyndicate
    Apr 27, 2012
  40. godsgift

    robots.txt error when allowing all

    godsgift, Jan 9, 2012
    Replies:
    6
    Views:
    4,366
    tiffanywilliams12i2
    Apr 10, 2012
  41. i_am_dhaval

    is am i right with robots.txt file

    i_am_dhaval, Jul 11, 2011
    Replies:
    6
    Views:
    3,050
    manish.chauhan
    Aug 1, 2011
  42. premierfabio

    Robots Problem

    premierfabio, May 23, 2011
    Replies:
    6
    Views:
    3,026
    RamCity
    May 24, 2011
  43. Pablo Peillard

    My Website is being indexed a little too much

    Pablo Peillard, Dec 25, 2010
    Replies:
    6
    Views:
    2,485
    fsdnetwork
    Feb 9, 2011
  44. adwaitk007

    I need guidance! For good robot.txt| All Gurus are invited!

    adwaitk007, Sep 12, 2010
    Replies:
    6
    Views:
    1,529
    The_Hatta
    Sep 20, 2010
  45. astrohope

    What to Allow and Disallow on robots.txt

    astrohope, Jul 5, 2010
    Replies:
    6
    Views:
    2,015
    joseph pop
    Aug 5, 2010
  46. Neil UK

    Robots.txt and Google crawl error

    Neil UK, Jul 10, 2010
    Replies:
    6
    Views:
    2,023
    mattstevens40
    Jul 27, 2010
  47. fadetoblack22

    Robots.txt not working for wordpress.

    fadetoblack22, Jul 8, 2010
    Replies:
    6
    Views:
    2,149
    fadetoblack22
    Jul 9, 2010
  48. teohcl

    How to protect our robots.txt?

    teohcl, Mar 28, 2010
    Replies:
    6
    Views:
    3,866
    chauhanmanish
    Apr 21, 2010
  49. FileBux

    Webmaster Tools - Robots.txt issue

    FileBux, Sep 2, 2009
    Replies:
    6
    Views:
    1,665
    jabz.biz
    Sep 11, 2009
  50. fadetoblack22

    Robots.txt goes in which directory?

    fadetoblack22, Jul 6, 2009
    Replies:
    6
    Views:
    1,481
    TheMagpie
    Jul 8, 2009
Showing threads 201 to 250 of 911

Thread Display Options

Loading...
Page 5 of 19