robots.txt

Find help with setting up your robots.txt file on your web site.

Page 6 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. lateuk

    Should i block my custom 404 page with robots.txt file?

    lateuk, May 27, 2008
    Replies:
    5
    Views:
    4,466
    christina258
    Apr 14, 2011
  2. georgerogers

    E-mail from google about robots.txt?

    georgerogers, Apr 12, 2011
    Replies:
    3
    Views:
    2,806
    georgerogers
    Apr 13, 2011
  3. umairraja52

    Purpose of robots.txt

    umairraja52, Mar 16, 2011
    Replies:
    9
    Views:
    4,463
    Oliveras11
    Apr 7, 2011
  4. iBullet

    How to make Robots.txt

    iBullet, Mar 21, 2011
    Replies:
    8
    Views:
    3,187
    brad.smith4321
    Apr 4, 2011
  5. gravy834

    Wildcard usage on User-agent

    gravy834, Mar 4, 2011
    Replies:
    4
    Views:
    3,396
    ACME Squares
    Mar 23, 2011
  6. Jooy

    Robots.txt

    Jooy, Mar 16, 2011
    Replies:
    5
    Views:
    2,756
    vijay01
    Mar 22, 2011
  7. Naveenseo

    Site not index in Google?

    Naveenseo, Mar 16, 2011
    Replies:
    5
    Views:
    2,615
    manish.chauhan
    Mar 22, 2011
  8. Sumitthewriter

    Can I Hide Duplicate Content By Robots.txt

    Sumitthewriter, Oct 28, 2010
    Replies:
    14
    Views:
    3,881
    ACME Squares
    Mar 15, 2011
  9. Voro

    Vbulletin Access

    Voro, Jun 4, 2007
    Replies:
    2
    Views:
    2,941
    shakac
    Mar 9, 2011
  10. Teepeg

    Leaching

    Teepeg, Mar 8, 2011
    Replies:
    3
    Views:
    2,304
    ACME Squares
    Mar 9, 2011
  11. jagan123

    Main site redirects to subdomain. How to stop indexing of subdomain?

    jagan123, Jan 11, 2011
    Replies:
    4
    Views:
    3,279
    manish.chauhan
    Feb 17, 2011
  12. Woodcs82

    Is Robots.txt better than Robots in META?

    Woodcs82, Jun 10, 2010 ... 2
    Replies:
    24
    Views:
    7,066
    fsdnetwork
    Feb 9, 2011
  13. Pablo Peillard

    My Website is being indexed a little too much

    Pablo Peillard, Dec 25, 2010
    Replies:
    6
    Views:
    2,649
    fsdnetwork
    Feb 9, 2011
  14. gravy834

    Robots.txt vs <meta name="robots" content="noindex,nofollow">

    gravy834, Mar 2, 2009
    Replies:
    12
    Views:
    8,767
    fsdnetwork
    Feb 9, 2011
  15. illianafrazer

    How can i block Google bot alone

    illianafrazer, Feb 2, 2011
    Replies:
    4
    Views:
    2,636
    fsdnetwork
    Feb 9, 2011
  16. ljk6802

    Layman's guide to robots.txt (and sitemap)

    ljk6802, Jan 24, 2011
    Replies:
    8
    Views:
    2,714
    ljk6802
    Feb 7, 2011
  17. n00bl3t

    Google webcache banned by robots.txt

    n00bl3t, Nov 25, 2010
    Replies:
    3
    Views:
    2,661
    gmendoza_0101
    Feb 7, 2011
  18. philipseocc

    Can robotx.txt block specific IPs/domain names?

    philipseocc, Jan 25, 2011
    Replies:
    7
    Views:
    2,794
    philipseocc
    Feb 1, 2011
  19. lxguy

    How to block some particular urls in my robots.txt?

    lxguy, Jan 16, 2011
    Replies:
    1
    Views:
    6,031
    nit Green
    Jan 19, 2011
  20. penpebso

    Should i continue to block robots from redirect link?

    penpebso, Jan 11, 2011
    Replies:
    0
    Views:
    1,809
    penpebso
    Jan 11, 2011
  21. DoDo Me

    Help! I want a robots.txt allow only root and disallow all others

    DoDo Me, Aug 30, 2010
    Replies:
    4
    Views:
    5,553
    techvivek
    Dec 31, 2010
  22. ayushi infotech

    Pls rectify my problem in Robots.txt

    ayushi infotech, Dec 28, 2010
    Replies:
    1
    Views:
    1,812
    techvivek
    Dec 31, 2010
  23. articleshop4all

    Robots.txt Please Check mine

    articleshop4all, Oct 30, 2010
    Replies:
    9
    Views:
    2,338
    reihaan
    Dec 30, 2010
  24. pradeep.balua

    how to install a robots.txt for a wordpress blog ?

    pradeep.balua, Jan 22, 2008
    Replies:
    11
    Views:
    4,808
    ayushi infotech
    Dec 28, 2010
  25. Woodcs82

    POLL: Do Large Websites your robots.txt ?

    Woodcs82, Nov 17, 2010
    Replies:
    1
    Views:
    1,687
    Pablo Peillard
    Dec 25, 2010
  26. jeel2008

    whats the root?

    jeel2008, Oct 22, 2010
    Replies:
    2
    Views:
    1,759
    computerlaptop
    Dec 23, 2010
  27. scott.allerdice

    robots.txt for https

    scott.allerdice, Dec 13, 2010
    Replies:
    4
    Views:
    1,894
    scott.allerdice
    Dec 19, 2010
  28. vincent87

    robots.txt for smf forum

    vincent87, Dec 8, 2010
    Replies:
    0
    Views:
    1,940
    vincent87
    Dec 8, 2010
  29. fokerss

    what is agent for bing ,blekko,bweezy?

    fokerss, Nov 17, 2010
    Replies:
    1
    Views:
    1,564
    underground-stockholm
    Nov 30, 2010
  30. duffer77

    Block specific part of the site

    duffer77, Nov 22, 2010
    Replies:
    2
    Views:
    1,598
    The_Hatta
    Nov 24, 2010
  31. nuraddiansyah

    PR3 blog search advertisers

    nuraddiansyah, Nov 15, 2010
    Replies:
    0
    Views:
    1,719
    nuraddiansyah
    Nov 15, 2010
  32. daliah

    my robot file doesnot work

    daliah, Oct 15, 2010
    Replies:
    9
    Views:
    2,035
    lyco
    Nov 14, 2010
  33. sorcha

    Can I limt a crawler to the root fodler with a single command?

    sorcha, Nov 1, 2010
    Replies:
    1
    Views:
    1,443
    AtomicPages
    Nov 3, 2010
  34. avatarmaddy

    Is really the contents provided by robots are meaningful ?

    avatarmaddy, Oct 27, 2010
    Replies:
    4
    Views:
    1,772
    avatarmaddy
    Oct 31, 2010
  35. hectoth

    Robots.txt

    hectoth, Oct 7, 2010
    Replies:
    12
    Views:
    2,442
    Pronto! SEO
    Oct 28, 2010
  36. Aragorn

    Separate page for mobile crawlers

    Aragorn, Oct 4, 2007
    Replies:
    2
    Views:
    2,541
    manish.chauhan
    Oct 25, 2010
  37. tshirtdeal

    Objects Filename Restricted By Google Robots?

    tshirtdeal, Apr 14, 2009
    Replies:
    2
    Views:
    2,158
    manish.chauhan
    Oct 25, 2010
  38. relkeel

    Mapped subdomains and robots.txt

    relkeel, Oct 7, 2010
    Replies:
    2
    Views:
    1,755
    manish.chauhan
    Oct 24, 2010
  39. cahyoaja

    robot.txt. on blogspot

    cahyoaja, Oct 22, 2010
    Replies:
    1
    Views:
    1,527
    manish.chauhan
    Oct 24, 2010
  40. ahmadlucky

    somebody plz solve my problem

    ahmadlucky, Sep 28, 2010
    Replies:
    8
    Views:
    2,267
    ahmadlucky
    Oct 17, 2010
  41. alokseo

    Anybody tell me about robots.txt? hwo to usefull for seo?

    alokseo, Jun 18, 2010
    Replies:
    19
    Views:
    3,252
    Prateekgupta
    Oct 17, 2010
  42. thetaxclub

    Robots help - The Tax Club

    thetaxclub, Dec 6, 2006
    Replies:
    2
    Views:
    1,898
    manish.chauhan
    Oct 6, 2010
  43. Alevoor

    I want to handpick Robots, spiders or crawlers, But How?

    Alevoor, Apr 22, 2010
    Replies:
    2
    Views:
    1,507
    seoguy5
    Oct 3, 2010
  44. Sammy04

    Question about blocking a link

    Sammy04, Mar 18, 2010
    Replies:
    2
    Views:
    1,601
    seoguy5
    Oct 3, 2010
  45. cahyoaja

    my robot.txt

    cahyoaja, Mar 23, 2010
    Replies:
    3
    Views:
    1,771
    seoguy5
    Oct 3, 2010
  46. peterzhou

    The robots.txt file in my site does not work?

    peterzhou, Aug 5, 2010
    Replies:
    3
    Views:
    1,636
    Sudhaa
    Sep 28, 2010
  47. thomasonk

    Blocking DNS aliases/CNAMES

    thomasonk, Mar 9, 2007
    Replies:
    1
    Views:
    1,782
    Sketch5646
    Sep 28, 2010
  48. life31

    DotBot!!! Is it of any use!!! Allow or Disallow???

    life31, Jun 21, 2009
    Replies:
    1
    Views:
    2,172
    Sketch5646
    Sep 28, 2010
  49. aka_le_Mulder

    Optimal 'robots.txt' file for Joomla! 1.5?

    aka_le_Mulder, Jan 5, 2010
    Replies:
    1
    Views:
    4,359
    Sketch5646
    Sep 28, 2010
  50. adwaitk007

    I need guidance! For good robot.txt| All Gurus are invited!

    adwaitk007, Sep 12, 2010
    Replies:
    6
    Views:
    1,667
    The_Hatta
    Sep 20, 2010
Showing threads 251 to 300 of 911

Thread Display Options

Loading...
Page 6 of 19