robots.txt

Find help with setting up your robots.txt file on your web site.

Page 2 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. scott2342

    How do I unblock my robots.txt

    scott2342, Nov 24, 2014
    Replies:
    3
    Views:
    11,509
    meethi jain
    Jun 3, 2015
  2. Minesh Parikh

    Blocked Url's by robots.txt in google webmaster tool how to fix it?

    Minesh Parikh, Apr 6, 2015
    Replies:
    0
    Views:
    17,922
    Minesh Parikh
    Apr 6, 2015
  3. enikram

    How to control when a bot(yahoo,google) can crawl a website

    enikram, Mar 18, 2015
    Replies:
    4
    Views:
    18,741
    Nigel Lew
    Mar 18, 2015
  4. Steviebone

    Robots.txt code to disallow by default but allow domain name

    Steviebone, Mar 4, 2015
    Replies:
    0
    Views:
    8,211
    Steviebone
    Mar 4, 2015
  5. Prasseo

    Will robots file help to cache the content in the web page

    Prasseo, Jan 12, 2015
    Replies:
    1
    Views:
    8,238
    Sycrid
    Jan 18, 2015
  6. wayne_hoton

    How to create robots.txt for bookmarking website. Is it necessary?

    wayne_hoton, Jan 11, 2015
    Replies:
    1
    Views:
    8,269
    wayne_hoton
    Jan 14, 2015
  7. Jaden Alvin

    Webmaster Blocked URL's Understanding Issue

    Jaden Alvin, Jul 9, 2014
    Replies:
    19
    Views:
    16,851
    Jaden Alvin
    Nov 24, 2014
  8. Olvia

    can give an example of a good robots.txt to wordpress

    Olvia, Nov 13, 2014
    Replies:
    0
    Views:
    8,010
    Olvia
    Nov 13, 2014
  9. aleale

    Is my robots.txt working fine?

    aleale, Sep 10, 2014
    Replies:
    2
    Views:
    19,466
    maddenitrous
    Oct 3, 2014
  10. gvinaykumar

    How to block a specific folder in robots.txt file?

    gvinaykumar, Jul 17, 2014
    Replies:
    0
    Views:
    9,465
    gvinaykumar
    Jul 17, 2014
  11. linkladdy

    Error - URLs restricted by robots.txt errors How Fix IT

    linkladdy, Jul 2, 2014
    Replies:
    1
    Views:
    9,971
    evildon
    Jul 2, 2014
  12. David Lowe

    Can I add specific urls to robots.txt to stop google indexing them?

    David Lowe, Jun 25, 2013
    Replies:
    2
    Views:
    9,836
    BabyDoll123
    Jun 19, 2014
  13. cathandrew

    How to clear these errors in robots.txt?

    cathandrew, Feb 25, 2014
    Replies:
    1
    Views:
    8,999
    BabyDoll123
    Jun 19, 2014
  14. mjfnd

    how to dissalow an image?

    mjfnd, Apr 29, 2014
    Replies:
    1
    Views:
    8,716
    BabyDoll123
    Jun 19, 2014
  15. BabyDoll123

    Magento Robots.txt file

    BabyDoll123, Jun 12, 2014
    Replies:
    3
    Views:
    9,832
    Nigel Lew
    Jun 14, 2014
  16. imaginexhacked

    Blocking auto generated pages with robots.txt

    imaginexhacked, Jun 9, 2014
    Replies:
    1
    Views:
    8,595
    O-D-T
    Jun 9, 2014
  17. jiangnancun

    How can i find those website in which robots.txt disallow google spider?

    jiangnancun, May 13, 2014
    Replies:
    2
    Views:
    9,214
    neroux
    May 13, 2014
  18. James101

    What is robot.txt file?

    James101, Apr 1, 2014
    Replies:
    4
    Views:
    9,501
    John Dave
    Apr 2, 2014
  19. garry420

    I have at-last found site that creates xml file of https sites

    garry420, Mar 8, 2014
    Replies:
    0
    Views:
    8,165
    garry420
    Mar 8, 2014
  20. bright776

    how to disallow a folder from being indexed

    bright776, Feb 1, 2014
    Replies:
    3
    Views:
    9,073
    Ehteshamshaikh
    Mar 7, 2014
  21. cathandrew

    Is it necessary to specify sitemap.xml in robots.txt file

    cathandrew, Feb 25, 2014
    Replies:
    1
    Views:
    8,811
    Ehteshamshaikh
    Feb 27, 2014
  22. manish_khanna

    Things you should be doing to block bad BOTS.

    manish_khanna, Feb 13, 2014
    Replies:
    0
    Views:
    8,584
    manish_khanna
    Feb 13, 2014
  23. SebastienD

    Dissalow folder with the same name as allowed folder

    SebastienD, Jan 15, 2014
    Replies:
    2
    Views:
    9,328
    Nette
    Feb 2, 2014
  24. vatu

    Robots.txt File Help

    vatu, Jan 15, 2014
    Replies:
    8
    Views:
    8,629
    alsgroup
    Jan 23, 2014
  25. Sunil Keshari

    Robots.txt code

    Sunil Keshari, Jan 9, 2014
    Replies:
    3
    Views:
    9,945
    ashokdhan
    Jan 14, 2014
  26. Vel

    I need help on how to Noindex categories, tags, author, archive

    Vel, Jul 25, 2013
    Replies:
    6
    Views:
    8,797
    tymsh
    Dec 9, 2013
  27. chadhaajay

    Please review my robots.txt code

    chadhaajay, Oct 10, 2013
    Replies:
    2
    Views:
    8,210
    makeit easy
    Nov 17, 2013
  28. rob587

    Would a robot.txt fix this SEO issue?

    rob587, May 10, 2013
    Replies:
    19
    Views:
    10,945
    msjaved
    Nov 8, 2013
  29. Ricky Martin 3124

    How to create Robots.txt Files?

    Ricky Martin 3124, Oct 31, 2013
    Replies:
    1
    Views:
    5,753
    Sayeri
    Oct 31, 2013
  30. yourarnav

    Googlebot can't access my site (Robot Txt)

    yourarnav, Sep 25, 2013
    Replies:
    4
    Views:
    6,656
    shakiba.proba
    Oct 9, 2013
  31. suganux

    Google Webmaster, URLs restricted by robots.txt

    suganux, Aug 13, 2013
    Replies:
    4
    Views:
    6,279
    suganux
    Aug 23, 2013
  32. PalakBhatt83

    Need Help with Robots.txt

    PalakBhatt83, Jul 24, 2013
    Replies:
    4
    Views:
    5,690
    blogdum
    Aug 12, 2013
  33. Goran Nikolov

    Robots.txt file can't be seen but SEOQuake says it exists?

    Goran Nikolov, Jul 11, 2013
    Replies:
    7
    Views:
    5,966
    MokricaIrina
    Jul 17, 2013
  34. prabhjot.singh

    How to Remove 404 Error?

    prabhjot.singh, Aug 26, 2012
    Replies:
    18
    Views:
    11,633
    Rakeshraj
    Jul 15, 2013
  35. HiSoC8Y

    Development site, don't want search engines...how?

    HiSoC8Y, Jul 12, 2013
    Replies:
    4
    Views:
    2,863
    ryan_uk
    Jul 13, 2013
  36. ntrdtr

    Blocked URL checking

    ntrdtr, Jun 11, 2013
    Replies:
    9
    Views:
    3,690
    Kyle Stester
    Jul 11, 2013
  37. w3seotraining
    Locked

    What is Robot.txt file?

    w3seotraining, Jul 3, 2013
    Replies:
    2
    Views:
    2,885
    Vincentas
    Jul 8, 2013
  38. tanmay.kamath1992

    Want to block other domain from crawling

    tanmay.kamath1992, Jun 22, 2013
    Replies:
    5
    Views:
    2,987
    Stephen Moyers
    Jul 4, 2013
  39. sam223

    Robots.txt - What is the proper format for a Crawl Delay for multiple user agents?

    sam223, Jun 29, 2013
    Replies:
    0
    Views:
    4,340
    sam223
    Jun 29, 2013
  40. beafriend

    When I search my blog on SE-I saw this-User-agent:Mediapartners-Google: * Disallow:

    beafriend, May 13, 2013
    Replies:
    3
    Views:
    4,801
    BobbyMcNiel
    Jun 3, 2013
  41. TiffanyJ.SSS

    Robots.txt security protection setup

    TiffanyJ.SSS, May 7, 2013
    Replies:
    2
    Views:
    2,785
    ryan_uk
    May 15, 2013
  42. sam2220

    Help regarding Robot.txt

    sam2220, Mar 19, 2013
    Replies:
    3
    Views:
    1,849
    topcoder
    May 14, 2013
  43. Tony Brar

    Stop Bing from indexing my website

    Tony Brar, Apr 22, 2013
    Replies:
    10
    Views:
    5,517
    Revelations-Decoder
    May 5, 2013
  44. Devtard

    How to prevent robots.txt from being indexed?

    Devtard, Apr 17, 2013
    Replies:
    5
    Views:
    1,576
    ryan_uk
    May 3, 2013
  45. Sanjeev SIngh

    IS robots.txt is safe for site?

    Sanjeev SIngh, Feb 21, 2013
    Replies:
    8
    Views:
    2,216
    blogdum
    Apr 25, 2013
  46. Devtard

    How to allow crawling a URL with a certain parameter

    Devtard, Apr 16, 2013
    Replies:
    1
    Views:
    1,168
    Devtard
    Apr 17, 2013
  47. Mr.Dog

    How Do I Ban Google From Seeing A Folder?

    Mr.Dog, Jan 16, 2013
    Replies:
    8
    Views:
    2,242
    MayurKawale
    Mar 29, 2013
  48. viveknathr

    Block a URL using Robots.txt

    viveknathr, Mar 24, 2013
    Replies:
    3
    Views:
    1,890
    indyonline
    Mar 24, 2013
  49. zeel_shah

    right and default format of robots.txt

    zeel_shah, Mar 13, 2013
    Replies:
    2
    Views:
    1,924
    ryan_uk
    Mar 14, 2013
  50. msaeedkk

    disallow:/?

    msaeedkk, Mar 7, 2013
    Replies:
    6
    Views:
    2,309
    ryan_uk
    Mar 14, 2013
Showing threads 51 to 100 of 911

Thread Display Options

Loading...
Page 2 of 19