robots.txt

Find help with setting up your robots.txt file on your web site.

Page 4 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. russel_shuvo2

    Robot.txt block my major part

    russel_shuvo2, Jun 11, 2012
    Replies:
    2
    Views:
    3,157
    atxsurf
    Jul 3, 2012
  2. iGeekSoftware

    Idea for robots.txt selective restrictions [done easy]

    iGeekSoftware, Jun 14, 2012
    Replies:
    1
    Views:
    3,109
    atxsurf
    Jul 3, 2012
  3. mariush

    Need advice on how to block certain page using robots.txt

    mariush, Aug 12, 2006
    Replies:
    7
    Views:
    10,654
    learn2success
    Jun 6, 2012
  4. hvwebseo

    How to stop crawling of multi language site

    hvwebseo, Jun 5, 2012
    Replies:
    1
    Views:
    2,736
    RoseHosting
    Jun 5, 2012
  5. matthecool

    Blocking .php from google in robots.txt

    matthecool, Jun 4, 2012
    Replies:
    1
    Views:
    2,773
    digitalpoint
    Jun 4, 2012
  6. AlisaRozzy

    Help regarding robots.txt file for my website

    AlisaRozzy, May 19, 2012
    Replies:
    1
    Views:
    2,914
    war_machine
    Jun 3, 2012
  7. riteshvyas52

    What is robots.txt ?

    riteshvyas52, May 4, 2012
    Replies:
    5
    Views:
    3,260
    JFSG
    Jun 3, 2012
  8. gdtechind

    How to Stop Web Robot from Engines other than G and Y and MSN

    gdtechind, Oct 27, 2005
    Replies:
    9
    Views:
    8,424
    Spearheadltd
    Jun 1, 2012
  9. joerudy1988

    robot.txt to allow everything

    joerudy1988, May 3, 2012
    Replies:
    3
    Views:
    3,554
    joerudy1988
    May 17, 2012
  10. satish nandanwar

    How we use robot.txt file for our site pages?

    satish nandanwar, May 9, 2012
    Replies:
    1
    Views:
    3,002
    bsdstr
    May 17, 2012
  11. TheSyndicate

    How do you get archiver to READ my robots?

    TheSyndicate, Mar 31, 2012
    Replies:
    6
    Views:
    3,440
    TheSyndicate
    Apr 27, 2012
  12. kshama

    I want to block folder /Portals/ and /portals/

    kshama, Apr 16, 2012
    Replies:
    1
    Views:
    3,039
    blacknet
    Apr 17, 2012
  13. sumitsaurabhmca

    robots.txt correct or not?

    sumitsaurabhmca, Apr 3, 2012
    Replies:
    7
    Views:
    3,157
    ryan_uk
    Apr 12, 2012
  14. Rajnish357

    What this robot.txt will do ? Please help !!

    Rajnish357, Jan 23, 2012
    Replies:
    2
    Views:
    3,552
    carlhenson
    Apr 11, 2012
  15. wijasenna

    does my robots.txt block my URLs?

    wijasenna, Apr 9, 2012
    Replies:
    3
    Views:
    3,759
    carlhenson
    Apr 11, 2012
  16. roynetwork7

    Noindex, nofollow on a wordpress blog

    roynetwork7, Mar 26, 2012
    Replies:
    2
    Views:
    3,305
    roynetwork7
    Apr 10, 2012
  17. sameerseo

    Robots.txt Allow/ Disallow Dynamic Pages

    sameerseo, Mar 26, 2012
    Replies:
    1
    Views:
    3,656
    tiffanywilliams12i2
    Apr 10, 2012
  18. matthecool

    blocking directory certain part of url from robots.txt

    matthecool, Oct 20, 2011
    Replies:
    3
    Views:
    4,391
    tiffanywilliams12i2
    Apr 10, 2012
  19. world pharmacare

    how to change the robots.txt file in open cart

    world pharmacare, Dec 31, 2011
    Replies:
    1
    Views:
    4,810
    tiffanywilliams12i2
    Apr 10, 2012
  20. godsgift

    robots.txt error when allowing all

    godsgift, Jan 9, 2012
    Replies:
    6
    Views:
    4,678
    tiffanywilliams12i2
    Apr 10, 2012
  21. TheSyndicate

    Robot directory

    TheSyndicate, Jan 24, 2012
    Replies:
    1
    Views:
    3,262
    tiffanywilliams12i2
    Apr 10, 2012
  22. qwikad.com

    Is there a way to make the bots respect the robots.txt file at all times?

    qwikad.com, Jan 31, 2012
    Replies:
    2
    Views:
    3,266
    tiffanywilliams12i2
    Apr 10, 2012
  23. aniket123

    Query about robots.txt function in wordpress?

    aniket123, Feb 1, 2012
    Replies:
    3
    Views:
    3,571
    tiffanywilliams12i2
    Apr 10, 2012
  24. varsha reddy

    yes it is important to keep robots.txt

    varsha reddy, Feb 1, 2012
    Replies:
    1
    Views:
    3,215
    tiffanywilliams12i2
    Apr 10, 2012
  25. rjd1265

    Robot.txt Kicking My A$$ Please Help!

    rjd1265, Feb 2, 2012
    Replies:
    2
    Views:
    2,985
    tiffanywilliams12i2
    Apr 10, 2012
  26. stilv

    how to get more yahoo and bing like google? [photo]

    stilv, Mar 3, 2012
    Replies:
    2
    Views:
    2,810
    tiffanywilliams12i2
    Apr 10, 2012
  27. AlisaRozzy

    Use of robot.txt file

    AlisaRozzy, Mar 23, 2012
    Replies:
    2
    Views:
    3,140
    tiffanywilliams12i2
    Apr 10, 2012
  28. arivin29

    thumbnail does not appear

    arivin29, Mar 26, 2012
    Replies:
    1
    Views:
    2,333
    tiffanywilliams12i2
    Apr 10, 2012
  29. TheSyndicate

    Is it normal that the /robots.txt is the most downloaded item from Server stats

    TheSyndicate, Mar 29, 2012
    Replies:
    11
    Views:
    2,842
    TheSyndicate
    Apr 9, 2012
  30. sofiaisabella01

    What is robot.txt flie?

    sofiaisabella01, Feb 19, 2012
    Replies:
    19
    Views:
    3,610
    mbitsol
    Apr 7, 2012
  31. NewComputer

    Question Re: Robots.txt

    NewComputer, Jun 12, 2004
    Replies:
    4
    Views:
    4,470
    anna30
    Apr 3, 2012
  32. mpchekuri

    How to block certain pages with url parameters from crawler's access using robots.txt

    mpchekuri, Feb 29, 2012
    Replies:
    4
    Views:
    3,301
    pindians007
    Mar 26, 2012
  33. abrahamgold

    Hello, I am new, and need some help on robots.txt

    abrahamgold, Oct 21, 2005
    Replies:
    14
    Views:
    9,361
    SPA assurance
    Mar 8, 2012
  34. sarahashlin

    How to create robots.txt file

    sarahashlin, Nov 28, 2011 ... 2
    Replies:
    22
    Views:
    7,764
    madison37
    Mar 6, 2012
  35. netboy

    Remove this stupid robots.txt

    netboy, Feb 22, 2008
    Replies:
    17
    Views:
    17,988
    Rituja
    Feb 15, 2012
  36. DeepakDwivedi

    Hi friends i want to disallow only this type of page in robot.txt

    DeepakDwivedi, Feb 9, 2012
    Replies:
    2
    Views:
    2,609
    DeepakDwivedi
    Feb 13, 2012
  37. rjd1265

    Robot.txt Troubles...again

    rjd1265, Feb 2, 2012
    Replies:
    0
    Views:
    2,075
    rjd1265
    Feb 2, 2012
  38. pla gilbert

    Webmaster Tools- Crawl errors Restricted by robots.txt ‎(14)‎

    pla gilbert, Jan 14, 2012
    Replies:
    2
    Views:
    3,064
    Cristina001
    Jan 26, 2012
  39. waleed.khaliqi

    google webmaster tool crawl error "Restricted by robots.txt ‎(473)‎"

    waleed.khaliqi, Jan 11, 2012
    Replies:
    1
    Views:
    2,580
    umerplz
    Jan 25, 2012
  40. ademmeda

    robots.txt and images

    ademmeda, Sep 11, 2011
    Replies:
    9
    Views:
    4,828
    WFgirls
    Jan 14, 2012
  41. iluvm

    Robot.txt file

    iluvm, Nov 30, 2005 ... 2
    Replies:
    20
    Views:
    12,850
    Emily Stephens
    Jan 3, 2012
  42. e-marketing guru

    Analyze These Robots.txt which one is better?

    e-marketing guru, Jun 23, 2008
    Replies:
    1
    Views:
    2,964
    TheSyndicate
    Dec 21, 2011
  43. TheSyndicate

    Amazon AWS Services - Cloudfront - FRA What to do with these?

    TheSyndicate, Dec 21, 2011
    Replies:
    0
    Views:
    2,989
    TheSyndicate
    Dec 21, 2011
  44. lok123

    Problem with timed out and unreacheable crawl error in google

    lok123, Dec 1, 2011
    Replies:
    2
    Views:
    2,853
    onlinegame6
    Dec 17, 2011
  45. wijasenna

    is this robot.txt good enough?

    wijasenna, Oct 25, 2011
    Replies:
    7
    Views:
    3,806
    xDiglett
    Dec 16, 2011
  46. mjindustry

    Quick Disallow Question for Gallery2 items

    mjindustry, Nov 24, 2011
    Replies:
    0
    Views:
    2,506
    mjindustry
    Nov 24, 2011
  47. TheSyndicate

    What is the best robot.txt for Wordpress a 2011

    TheSyndicate, Nov 15, 2011
    Replies:
    0
    Views:
    3,453
    TheSyndicate
    Nov 15, 2011
  48. aniket123

    Robots.txt Disallow: / function

    aniket123, Nov 1, 2011
    Replies:
    3
    Views:
    3,091
    tripbuilder
    Nov 14, 2011
  49. mbpalmer87

    What is everyone's favourite robot?

    mbpalmer87, Nov 9, 2011
    Replies:
    0
    Views:
    2,446
    mbpalmer87
    Nov 9, 2011
  50. MyArtGallery

    Please help me understanding robots.txt file

    MyArtGallery, Sep 6, 2011
    Replies:
    9
    Views:
    4,012
    pr0t0n
    Nov 8, 2011
Showing threads 151 to 200 of 911

Thread Display Options

Loading...
Page 4 of 19