1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 1 of 19
Sort by:
Title Start Date
Replies ↑ Views
Last Message
  1. gree124

    Robots is good

    gree124, May 2, 2019
    Replies:
    0
    Views:
    3,233
    gree124
    May 2, 2019
  2. dinrock

    I have default robots.txt and asking how to customize it

    dinrock, Oct 13, 2016
    Replies:
    0
    Views:
    5,182
    dinrock
    Oct 13, 2016
  3. Official Gates

    What is the correct way to write my “robots.txt” file?

    Official Gates, Aug 16, 2016
    Replies:
    0
    Views:
    4,494
    Official Gates
    Aug 16, 2016
  4. sensor88

    Addon Domains Showing With Main Domain In URL

    sensor88, Aug 9, 2016
    Replies:
    0
    Views:
    4,968
    sensor88
    Aug 9, 2016
  5. full_time_writer

    Robot text and partial fetch issues - need suggestions

    full_time_writer, Jul 28, 2015
    Replies:
    0
    Views:
    9,190
    full_time_writer
    Jul 28, 2015
  6. Minesh Parikh

    Blocked Url's by robots.txt in google webmaster tool how to fix it?

    Minesh Parikh, Apr 6, 2015
    Replies:
    0
    Views:
    17,821
    Minesh Parikh
    Apr 6, 2015
  7. Steviebone

    Robots.txt code to disallow by default but allow domain name

    Steviebone, Mar 4, 2015
    Replies:
    0
    Views:
    8,123
    Steviebone
    Mar 4, 2015
  8. Olvia

    can give an example of a good robots.txt to wordpress

    Olvia, Nov 13, 2014
    Replies:
    0
    Views:
    7,925
    Olvia
    Nov 13, 2014
  9. gvinaykumar

    How to block a specific folder in robots.txt file?

    gvinaykumar, Jul 17, 2014
    Replies:
    0
    Views:
    9,365
    gvinaykumar
    Jul 17, 2014
  10. garry420

    I have at-last found site that creates xml file of https sites

    garry420, Mar 8, 2014
    Replies:
    0
    Views:
    8,034
    garry420
    Mar 8, 2014
  11. manish_khanna

    Things you should be doing to block bad BOTS.

    manish_khanna, Feb 13, 2014
    Replies:
    0
    Views:
    8,494
    manish_khanna
    Feb 13, 2014
  12. sam223

    Robots.txt - What is the proper format for a Crawl Delay for multiple user agents?

    sam223, Jun 29, 2013
    Replies:
    0
    Views:
    4,243
    sam223
    Jun 29, 2013
  13. Dee2007

    Folder Question

    Dee2007, Dec 9, 2012
    Replies:
    0
    Views:
    1,529
    Dee2007
    Dec 9, 2012
  14. pooja111

    Robots.txt was inaccessible

    pooja111, Sep 15, 2012
    Replies:
    0
    Views:
    2,890
    pooja111
    Sep 15, 2012
  15. cuteboy

    Staging IP indexed by Google

    cuteboy, Sep 4, 2012
    Replies:
    0
    Views:
    2,808
    cuteboy
    Sep 4, 2012
  16. cloudseoguy

    search engine specific robot.txt [HELP]

    cloudseoguy, Aug 19, 2012
    Replies:
    0
    Views:
    2,051
    cloudseoguy
    Aug 19, 2012
  17. rjd1265

    Robot.txt Troubles...again

    rjd1265, Feb 2, 2012
    Replies:
    0
    Views:
    1,964
    rjd1265
    Feb 2, 2012
  18. TheSyndicate

    Amazon AWS Services - Cloudfront - FRA What to do with these?

    TheSyndicate, Dec 21, 2011
    Replies:
    0
    Views:
    2,811
    TheSyndicate
    Dec 21, 2011
  19. mjindustry

    Quick Disallow Question for Gallery2 items

    mjindustry, Nov 24, 2011
    Replies:
    0
    Views:
    2,384
    mjindustry
    Nov 24, 2011
  20. TheSyndicate

    What is the best robot.txt for Wordpress a 2011

    TheSyndicate, Nov 15, 2011
    Replies:
    0
    Views:
    3,340
    TheSyndicate
    Nov 15, 2011
  21. mbpalmer87

    What is everyone's favourite robot?

    mbpalmer87, Nov 9, 2011
    Replies:
    0
    Views:
    2,324
    mbpalmer87
    Nov 9, 2011
  22. Kalibaster

    Robots.txt for wordpress autoblog setup.

    Kalibaster, Sep 21, 2011
    Replies:
    0
    Views:
    3,146
    Kalibaster
    Sep 21, 2011
  23. prclicks

    Create robots.txt File, Need Your Suggestion

    prclicks, Jun 3, 2011
    Replies:
    0
    Views:
    2,661
    prclicks
    Jun 3, 2011
  24. penpebso

    Should i continue to block robots from redirect link?

    penpebso, Jan 11, 2011
    Replies:
    0
    Views:
    1,735
    penpebso
    Jan 11, 2011
  25. vincent87

    robots.txt for smf forum

    vincent87, Dec 8, 2010
    Replies:
    0
    Views:
    1,838
    vincent87
    Dec 8, 2010
  26. nuraddiansyah

    PR3 blog search advertisers

    nuraddiansyah, Nov 15, 2010
    Replies:
    0
    Views:
    1,645
    nuraddiansyah
    Nov 15, 2010
  27. mwcole83

    Does a "NoArchive" metatag affect your page rank on seach engines?

    mwcole83, Mar 22, 2010
    Replies:
    0
    Views:
    1,420
    mwcole83
    Mar 22, 2010
  28. makenoiz

    is /*? (astrisk question mark) the same as /? (question mark)

    makenoiz, Mar 3, 2010
    Replies:
    0
    Views:
    869
    makenoiz
    Mar 3, 2010
  29. Beefandy

    Trouble with Google News Crawling site

    Beefandy, Oct 27, 2009
    Replies:
    0
    Views:
    1,069
    Beefandy
    Oct 27, 2009
  30. Sxperm

    Please help with my robots.txt

    Sxperm, Sep 19, 2009
    Replies:
    0
    Views:
    1,293
    Sxperm
    Sep 19, 2009
  31. franklinrony

    multiple pages in a post generate duplicate content

    franklinrony, Sep 17, 2009
    Replies:
    0
    Views:
    1,270
    franklinrony
    Sep 17, 2009
  32. itsallwhite

    Advice about robots.txt and duplicate content

    itsallwhite, Aug 23, 2009
    Replies:
    0
    Views:
    1,410
    itsallwhite
    Aug 23, 2009
  33. bigfishphilip

    Wildcards and directory depth

    bigfishphilip, Jul 25, 2009
    Replies:
    0
    Views:
    1,352
    bigfishphilip
    Jul 25, 2009
  34. gloglo

    Help me plz

    gloglo, Jun 18, 2009
    Replies:
    0
    Views:
    1,080
    gloglo
    Jun 18, 2009
  35. rnvr
    Locked

    How to create a robots.txt, easy steps

    rnvr, May 26, 2009
    Replies:
    0
    Views:
    1,119
    rnvr
    May 26, 2009
  36. hoshheid

    Robots.txt for http and https on Linux server

    hoshheid, Feb 23, 2009
    Replies:
    0
    Views:
    1,165
    hoshheid
    Feb 23, 2009
  37. wibblet
    Locked

    How To Block An Entire Sub-Forum?

    wibblet, Jan 24, 2009
    Replies:
    0
    Views:
    1,030
    wibblet
    Jan 24, 2009
  38. alanbutt

    What are these dam Robots up to?

    alanbutt, Oct 31, 2008
    Replies:
    0
    Views:
    957
    alanbutt
    Oct 31, 2008
  39. filmixpress
    Locked

    Robot

    filmixpress, Oct 28, 2008
    Replies:
    0
    Views:
    773
    filmixpress
    Oct 28, 2008
  40. iloveRP.com

    Stop Google From Indexing WAP2 Version of Your Site

    iloveRP.com, Oct 22, 2008
    Replies:
    0
    Views:
    1,637
    iloveRP.com
    Oct 22, 2008
  41. kiransarv

    robots.txt

    kiransarv, Oct 21, 2008
    Replies:
    0
    Views:
    847
    kiransarv
    Oct 21, 2008
  42. TheMrAnderson

    Excluding a page and all spiders except Google

    TheMrAnderson, Oct 20, 2008
    Replies:
    0
    Views:
    873
    TheMrAnderson
    Oct 20, 2008
  43. jbperkins

    Need robots.txt Help

    jbperkins, Sep 24, 2008
    Replies:
    0
    Views:
    812
    jbperkins
    Sep 24, 2008
  44. weilies

    Sitemap indexing for forum: Dynamic or Static is better?

    weilies, Sep 21, 2008
    Replies:
    0
    Views:
    933
    weilies
    Sep 21, 2008
  45. poseidon

    How to stop all bots other than Google/yahoo/Live through robots.txt ?

    poseidon, Sep 5, 2008
    Replies:
    0
    Views:
    1,742
    poseidon
    Sep 5, 2008
  46. abercrombie

    ignore addon domain folders in robots.txt?

    abercrombie, Aug 1, 2008
    Replies:
    0
    Views:
    1,060
    abercrombie
    Aug 1, 2008
  47. SuperNovaArcade

    All About Robots.txt

    SuperNovaArcade, Jun 4, 2008
    Replies:
    0
    Views:
    1,504
    SuperNovaArcade
    Jun 4, 2008
  48. frogman49th

    How to use robots.txt in blogs?

    frogman49th, Oct 6, 2007
    Replies:
    0
    Views:
    966
    frogman49th
    Oct 6, 2007
  49. Hostaff

    Excluding feeds in Wordpress

    Hostaff, May 22, 2007
    Replies:
    0
    Views:
    909
    Hostaff
    May 22, 2007
  50. dougadam

    Which IP's to block?

    dougadam, Mar 17, 2007
    Replies:
    0
    Views:
    1,111
    dougadam
    Mar 17, 2007
Showing threads 1 to 50 of 911

Thread Display Options

Loading...
Page 1 of 19