1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 18 of 19
Sort by:
Title Start Date
Replies ↓ Views
Last Message
  1. Dio

    Blocking Bad Bots With HTACCESS rather than robots.txt

    Dio, Nov 23, 2005
    Replies:
    1
    Views:
    5,151
    tflight
    Nov 23, 2005
  2. mjonsson

    406: Not Acceptable (encoding)

    mjonsson, Nov 9, 2005
    Replies:
    1
    Views:
    2,683
    minstrel
    Nov 9, 2005
  3. Davilac

    Some "advanced" help

    Davilac, Sep 22, 2005
    Replies:
    1
    Views:
    1,460
    minstrel
    Sep 30, 2005
  4. Alfred_ans

    Pages getting indexed despite diallowing them with robots.txt

    Alfred_ans, Sep 13, 2005
    Replies:
    1
    Views:
    1,595
    Kim Steinhaug
    Sep 13, 2005
  5. inthedark

    robots.txt

    inthedark, Mar 25, 2005
    Replies:
    1
    Views:
    2,462
    riziko
    Mar 26, 2005
  6. exam

    robots.txt special excusion needed

    exam, Dec 11, 2004
    Replies:
    1
    Views:
    2,245
    exam
    Dec 12, 2004
  7. l234244

    Images - robots.txt

    l234244, Sep 4, 2004
    Replies:
    1
    Views:
    2,655
    stephfoster
    Sep 5, 2004
  8. gree124

    Robots is good

    gree124, May 2, 2019
    Replies:
    0
    Views:
    3,236
    gree124
    May 2, 2019
  9. dinrock

    I have default robots.txt and asking how to customize it

    dinrock, Oct 13, 2016
    Replies:
    0
    Views:
    5,185
    dinrock
    Oct 13, 2016
  10. Official Gates

    What is the correct way to write my “robots.txt” file?

    Official Gates, Aug 16, 2016
    Replies:
    0
    Views:
    4,495
    Official Gates
    Aug 16, 2016
  11. sensor88

    Addon Domains Showing With Main Domain In URL

    sensor88, Aug 9, 2016
    Replies:
    0
    Views:
    4,971
    sensor88
    Aug 9, 2016
  12. full_time_writer

    Robot text and partial fetch issues - need suggestions

    full_time_writer, Jul 28, 2015
    Replies:
    0
    Views:
    9,192
    full_time_writer
    Jul 28, 2015
  13. Minesh Parikh

    Blocked Url's by robots.txt in google webmaster tool how to fix it?

    Minesh Parikh, Apr 6, 2015
    Replies:
    0
    Views:
    17,822
    Minesh Parikh
    Apr 6, 2015
  14. Steviebone

    Robots.txt code to disallow by default but allow domain name

    Steviebone, Mar 4, 2015
    Replies:
    0
    Views:
    8,124
    Steviebone
    Mar 4, 2015
  15. Olvia

    can give an example of a good robots.txt to wordpress

    Olvia, Nov 13, 2014
    Replies:
    0
    Views:
    7,926
    Olvia
    Nov 13, 2014
  16. gvinaykumar

    How to block a specific folder in robots.txt file?

    gvinaykumar, Jul 17, 2014
    Replies:
    0
    Views:
    9,366
    gvinaykumar
    Jul 17, 2014
  17. garry420

    I have at-last found site that creates xml file of https sites

    garry420, Mar 8, 2014
    Replies:
    0
    Views:
    8,035
    garry420
    Mar 8, 2014
  18. manish_khanna

    Things you should be doing to block bad BOTS.

    manish_khanna, Feb 13, 2014
    Replies:
    0
    Views:
    8,494
    manish_khanna
    Feb 13, 2014
  19. sam223

    Robots.txt - What is the proper format for a Crawl Delay for multiple user agents?

    sam223, Jun 29, 2013
    Replies:
    0
    Views:
    4,243
    sam223
    Jun 29, 2013
  20. Dee2007

    Folder Question

    Dee2007, Dec 9, 2012
    Replies:
    0
    Views:
    1,530
    Dee2007
    Dec 9, 2012
  21. pooja111

    Robots.txt was inaccessible

    pooja111, Sep 15, 2012
    Replies:
    0
    Views:
    2,890
    pooja111
    Sep 15, 2012
  22. cuteboy

    Staging IP indexed by Google

    cuteboy, Sep 4, 2012
    Replies:
    0
    Views:
    2,809
    cuteboy
    Sep 4, 2012
  23. cloudseoguy

    search engine specific robot.txt [HELP]

    cloudseoguy, Aug 19, 2012
    Replies:
    0
    Views:
    2,051
    cloudseoguy
    Aug 19, 2012
  24. rjd1265

    Robot.txt Troubles...again

    rjd1265, Feb 2, 2012
    Replies:
    0
    Views:
    1,966
    rjd1265
    Feb 2, 2012
  25. TheSyndicate

    Amazon AWS Services - Cloudfront - FRA What to do with these?

    TheSyndicate, Dec 21, 2011
    Replies:
    0
    Views:
    2,812
    TheSyndicate
    Dec 21, 2011
  26. mjindustry

    Quick Disallow Question for Gallery2 items

    mjindustry, Nov 24, 2011
    Replies:
    0
    Views:
    2,385
    mjindustry
    Nov 24, 2011
  27. TheSyndicate

    What is the best robot.txt for Wordpress a 2011

    TheSyndicate, Nov 15, 2011
    Replies:
    0
    Views:
    3,340
    TheSyndicate
    Nov 15, 2011
  28. mbpalmer87

    What is everyone's favourite robot?

    mbpalmer87, Nov 9, 2011
    Replies:
    0
    Views:
    2,324
    mbpalmer87
    Nov 9, 2011
  29. Kalibaster

    Robots.txt for wordpress autoblog setup.

    Kalibaster, Sep 21, 2011
    Replies:
    0
    Views:
    3,147
    Kalibaster
    Sep 21, 2011
  30. prclicks

    Create robots.txt File, Need Your Suggestion

    prclicks, Jun 3, 2011
    Replies:
    0
    Views:
    2,663
    prclicks
    Jun 3, 2011
  31. penpebso

    Should i continue to block robots from redirect link?

    penpebso, Jan 11, 2011
    Replies:
    0
    Views:
    1,735
    penpebso
    Jan 11, 2011
  32. vincent87

    robots.txt for smf forum

    vincent87, Dec 8, 2010
    Replies:
    0
    Views:
    1,840
    vincent87
    Dec 8, 2010
  33. nuraddiansyah

    PR3 blog search advertisers

    nuraddiansyah, Nov 15, 2010
    Replies:
    0
    Views:
    1,646
    nuraddiansyah
    Nov 15, 2010
  34. mwcole83

    Does a "NoArchive" metatag affect your page rank on seach engines?

    mwcole83, Mar 22, 2010
    Replies:
    0
    Views:
    1,420
    mwcole83
    Mar 22, 2010
  35. makenoiz

    is /*? (astrisk question mark) the same as /? (question mark)

    makenoiz, Mar 3, 2010
    Replies:
    0
    Views:
    869
    makenoiz
    Mar 3, 2010
  36. Beefandy

    Trouble with Google News Crawling site

    Beefandy, Oct 27, 2009
    Replies:
    0
    Views:
    1,069
    Beefandy
    Oct 27, 2009
  37. Sxperm

    Please help with my robots.txt

    Sxperm, Sep 19, 2009
    Replies:
    0
    Views:
    1,294
    Sxperm
    Sep 19, 2009
  38. franklinrony

    multiple pages in a post generate duplicate content

    franklinrony, Sep 17, 2009
    Replies:
    0
    Views:
    1,270
    franklinrony
    Sep 17, 2009
  39. itsallwhite

    Advice about robots.txt and duplicate content

    itsallwhite, Aug 23, 2009
    Replies:
    0
    Views:
    1,410
    itsallwhite
    Aug 23, 2009
  40. bigfishphilip

    Wildcards and directory depth

    bigfishphilip, Jul 25, 2009
    Replies:
    0
    Views:
    1,352
    bigfishphilip
    Jul 25, 2009
  41. gloglo

    Help me plz

    gloglo, Jun 18, 2009
    Replies:
    0
    Views:
    1,080
    gloglo
    Jun 18, 2009
  42. rnvr
    Locked

    How to create a robots.txt, easy steps

    rnvr, May 26, 2009
    Replies:
    0
    Views:
    1,119
    rnvr
    May 26, 2009
  43. hoshheid

    Robots.txt for http and https on Linux server

    hoshheid, Feb 23, 2009
    Replies:
    0
    Views:
    1,165
    hoshheid
    Feb 23, 2009
  44. wibblet
    Locked

    How To Block An Entire Sub-Forum?

    wibblet, Jan 24, 2009
    Replies:
    0
    Views:
    1,030
    wibblet
    Jan 24, 2009
  45. alanbutt

    What are these dam Robots up to?

    alanbutt, Oct 31, 2008
    Replies:
    0
    Views:
    957
    alanbutt
    Oct 31, 2008
  46. filmixpress
    Locked

    Robot

    filmixpress, Oct 28, 2008
    Replies:
    0
    Views:
    773
    filmixpress
    Oct 28, 2008
  47. iloveRP.com

    Stop Google From Indexing WAP2 Version of Your Site

    iloveRP.com, Oct 22, 2008
    Replies:
    0
    Views:
    1,637
    iloveRP.com
    Oct 22, 2008
  48. kiransarv

    robots.txt

    kiransarv, Oct 21, 2008
    Replies:
    0
    Views:
    847
    kiransarv
    Oct 21, 2008
  49. TheMrAnderson

    Excluding a page and all spiders except Google

    TheMrAnderson, Oct 20, 2008
    Replies:
    0
    Views:
    873
    TheMrAnderson
    Oct 20, 2008
  50. jbperkins

    Need robots.txt Help

    jbperkins, Sep 24, 2008
    Replies:
    0
    Views:
    812
    jbperkins
    Sep 24, 2008
Showing threads 851 to 900 of 911

Thread Display Options

Loading...
Page 18 of 19