1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 2 of 19
Sort by:
Title Start Date
Replies Views ↓
Last Message
  1. David Lowe

    Can I add specific urls to robots.txt to stop google indexing them?

    David Lowe, Jun 25, 2013
    Replies:
    2
    Views:
    9,734
    BabyDoll123
    Jun 19, 2014
  2. BabyDoll123

    Magento Robots.txt file

    BabyDoll123, Jun 12, 2014
    Replies:
    3
    Views:
    9,667
    Nigel Lew
    Jun 14, 2014
  3. gvinaykumar

    How to block a specific folder in robots.txt file?

    gvinaykumar, Jul 17, 2014
    Replies:
    0
    Views:
    9,364
    gvinaykumar
    Jul 17, 2014
  4. James101

    What is robot.txt file?

    James101, Apr 1, 2014
    Replies:
    4
    Views:
    9,333
    John Dave
    Apr 2, 2014
  5. SebastienD

    Dissalow folder with the same name as allowed folder

    SebastienD, Jan 15, 2014
    Replies:
    2
    Views:
    9,225
    Nette
    Feb 2, 2014
  6. full_time_writer

    Robot text and partial fetch issues - need suggestions

    full_time_writer, Jul 28, 2015
    Replies:
    0
    Views:
    9,176
    full_time_writer
    Jul 28, 2015
  7. amedhussaini

    should i disallow images?

    amedhussaini, Dec 1, 2006
    Replies:
    14
    Views:
    9,171
    sarpras2005@yahoo.co.in
    Aug 11, 2009
  8. VijayanKumar

    What Is Robots.txt?

    VijayanKumar, Jun 9, 2011 ... 2
    Replies:
    32
    Views:
    9,038
    webgfx
    Sep 14, 2011
  9. jiangnancun

    How can i find those website in which robots.txt disallow google spider?

    jiangnancun, May 13, 2014
    Replies:
    2
    Views:
    8,963
    neroux
    May 13, 2014
  10. poseidon

    How to create robot.txt file ? how to stop indexing files from cgi-bin ?

    poseidon, Jan 23, 2006 ... 2
    Replies:
    23
    Views:
    8,955
    sgtcory
    Jun 14, 2009
  11. bright776

    how to disallow a folder from being indexed

    bright776, Feb 1, 2014
    Replies:
    3
    Views:
    8,930
    Ehteshamshaikh
    Mar 7, 2014
  12. cathandrew

    How to clear these errors in robots.txt?

    cathandrew, Feb 25, 2014
    Replies:
    1
    Views:
    8,876
    BabyDoll123
    Jun 19, 2014
  13. abrahamgold

    Hello, I am new, and need some help on robots.txt

    abrahamgold, Oct 21, 2005
    Replies:
    14
    Views:
    8,851
    SPA assurance
    Mar 8, 2012
  14. skionxb

    Wow is ROBOTS.txt important?

    skionxb, Jul 26, 2006 ... 2
    Replies:
    26
    Views:
    8,831
    Dan_Grr
    Jan 12, 2007
  15. anna30

    How to add robots.txt file in Magento

    anna30, Oct 16, 2015
    Replies:
    1
    Views:
    8,793
    paco78
    Oct 16, 2015
  16. ronkoo
    Locked

    asking you for a information

    ronkoo, Mar 13, 2018
    Replies:
    4
    Views:
    8,772
    hostechsupport
    Aug 17, 2018
  17. cathandrew

    Is it necessary to specify sitemap.xml in robots.txt file

    cathandrew, Feb 25, 2014
    Replies:
    1
    Views:
    8,742
    Ehteshamshaikh
    Feb 27, 2014
  18. Vel

    I need help on how to Noindex categories, tags, author, archive

    Vel, Jul 25, 2013
    Replies:
    6
    Views:
    8,711
    tymsh
    Dec 9, 2013
  19. mjfnd

    how to dissalow an image?

    mjfnd, Apr 29, 2014
    Replies:
    1
    Views:
    8,620
    BabyDoll123
    Jun 19, 2014
  20. gravy834

    Robots.txt vs <meta name="robots" content="noindex,nofollow">

    gravy834, Mar 2, 2009
    Replies:
    12
    Views:
    8,525
    fsdnetwork
    Feb 9, 2011
  21. imaginexhacked

    Blocking auto generated pages with robots.txt

    imaginexhacked, Jun 9, 2014
    Replies:
    1
    Views:
    8,497
    O-D-T
    Jun 9, 2014
  22. vatu

    Robots.txt File Help

    vatu, Jan 15, 2014
    Replies:
    8
    Views:
    8,493
    alsgroup
    Jan 23, 2014
  23. manish_khanna

    Things you should be doing to block bad BOTS.

    manish_khanna, Feb 13, 2014
    Replies:
    0
    Views:
    8,493
    manish_khanna
    Feb 13, 2014
  24. sarathy

    How to remove this dynamic url using robots.txt from SE's

    sarathy, Oct 13, 2005
    Replies:
    15
    Views:
    8,475
    C. Berntsen
    Jul 17, 2008
  25. alka007

    What is Robots.txt ?

    alka007, Feb 23, 2012 ... 2
    Replies:
    32
    Views:
    8,289
    sahostking
    Nov 22, 2012
  26. Arcos

    How to STOP indexing web pages

    Arcos, Dec 23, 2005
    Replies:
    14
    Views:
    8,224
    jhon786
    Aug 25, 2011
  27. Prasseo

    Will robots file help to cache the content in the web page

    Prasseo, Jan 12, 2015
    Replies:
    1
    Views:
    8,146
    Sycrid
    Jan 18, 2015
  28. wayne_hoton

    How to create robots.txt for bookmarking website. Is it necessary?

    wayne_hoton, Jan 11, 2015
    Replies:
    1
    Views:
    8,136
    wayne_hoton
    Jan 14, 2015
  29. chadhaajay

    Please review my robots.txt code

    chadhaajay, Oct 10, 2013
    Replies:
    2
    Views:
    8,126
    makeit easy
    Nov 17, 2013
  30. Steviebone

    Robots.txt code to disallow by default but allow domain name

    Steviebone, Mar 4, 2015
    Replies:
    0
    Views:
    8,122
    Steviebone
    Mar 4, 2015
  31. infopages

    Help with Robots.txt

    infopages, Sep 29, 2017
    Replies:
    4
    Views:
    8,039
    Elevate
    Aug 3, 2018
  32. garry420

    I have at-last found site that creates xml file of https sites

    garry420, Mar 8, 2014
    Replies:
    0
    Views:
    8,033
    garry420
    Mar 8, 2014
  33. techfreakd

    How to disallow Google crawl link.???

    techfreakd, May 18, 2012 ... 2
    Replies:
    24
    Views:
    8,021
    Max796
    Oct 10, 2012
  34. gdtechind

    How to Stop Web Robot from Engines other than G and Y and MSN

    gdtechind, Oct 27, 2005
    Replies:
    9
    Views:
    8,002
    Spearheadltd
    Jun 1, 2012
  35. Olvia

    can give an example of a good robots.txt to wordpress

    Olvia, Nov 13, 2014
    Replies:
    0
    Views:
    7,923
    Olvia
    Nov 13, 2014
  36. kusadasi-guy

    Allowing only Mediapartners-Google

    kusadasi-guy, Dec 30, 2004
    Replies:
    9
    Views:
    7,390
    bannitim
    Aug 13, 2009
  37. wan.ariffuddin

    How do i remove my history from INTERNET ARCHIVE WAYBACK MACHINE

    wan.ariffuddin, Jul 8, 2010
    Replies:
    1
    Views:
    7,222
    manish.chauhan
    Jul 8, 2010
  38. Harris96

    Robots.txt error

    Harris96, Jan 24, 2016
    Replies:
    1
    Views:
    7,215
    Nigel Lew
    Jan 24, 2016
  39. amitpatel_3001

    Is this Gud/Bad??? Google Bot eats 330MB/Day Bandwidth

    amitpatel_3001, Dec 9, 2005 ... 2
    Replies:
    23
    Views:
    7,209
    amitpatel_3001
    Feb 23, 2007
  40. Quench

    bots overloading my server everyday.

    Quench, Aug 7, 2008
    Replies:
    19
    Views:
    7,130
    Quench
    Oct 9, 2008
  41. sarahashlin

    How to create robots.txt file

    sarahashlin, Nov 28, 2011 ... 2
    Replies:
    22
    Views:
    7,090
    madison37
    Mar 6, 2012
  42. tripower

    AdSense, subdomains and robots.txt

    tripower, May 4, 2005
    Replies:
    5
    Views:
    6,896
    paymentapprovaltooslow
    May 6, 2005
  43. Edz

    The most ideal robots.txt? how should it look like?

    Edz, Nov 24, 2005 ... 2
    Replies:
    34
    Views:
    6,880
    minstrel
    Dec 10, 2005
  44. disgust

    googlebot, vbulletin, and robots.txt

    disgust, Aug 5, 2004
    Replies:
    5
    Views:
    6,849
    digitalpoint
    Aug 5, 2004
  45. Xfactormaster

    Robots.txt has 'Disallow: /forums/-/' line, what is the meaning

    Xfactormaster, Mar 1, 2016
    Replies:
    2
    Views:
    6,766
    stilliard
    Mar 7, 2016
  46. remember123

    Best robots.txt for vBulletin forums

    remember123, Aug 22, 2009
    Replies:
    11
    Views:
    6,712
    Brandon Sheley
    Jan 8, 2010
  47. Talker

    Robots.txt for Vbulletin

    Talker, Oct 13, 2006
    Replies:
    11
    Views:
    6,673
    zinghana
    Jul 31, 2010
  48. kartarsingh

    I need anyone expert who can create robots.txt file for myBB forum

    kartarsingh, Mar 17, 2012
    Replies:
    3
    Views:
    6,600
    Pepdeal
    Jul 28, 2012
  49. PaRaDiGm13

    Robots.txt Question

    PaRaDiGm13, Mar 16, 2016
    Replies:
    4
    Views:
    6,549
    Cado Magenge
    Feb 28, 2017
  50. yourarnav

    Googlebot can't access my site (Robot Txt)

    yourarnav, Sep 25, 2013
    Replies:
    4
    Views:
    6,538
    shakiba.proba
    Oct 9, 2013
Showing threads 51 to 100 of 911

Thread Display Options

Loading...
Page 2 of 19