robots.txt

Find help with setting up your robots.txt file on your web site.

Page 18 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↑
  1. linkladdy

    Error - URLs restricted by robots.txt errors How Fix IT

    linkladdy, Jul 2, 2014
    Replies:
    1
    Views:
    10,057
    evildon
    Jul 2, 2014
  2. gvinaykumar

    How to block a specific folder in robots.txt file?

    gvinaykumar, Jul 17, 2014
    Replies:
    0
    Views:
    9,533
    gvinaykumar
    Jul 17, 2014
  3. aleale

    Is my robots.txt working fine?

    aleale, Sep 10, 2014
    Replies:
    2
    Views:
    19,564
    maddenitrous
    Oct 3, 2014
  4. Olvia

    can give an example of a good robots.txt to wordpress

    Olvia, Nov 13, 2014
    Replies:
    0
    Views:
    8,064
    Olvia
    Nov 13, 2014
  5. Jaden Alvin

    Webmaster Blocked URL's Understanding Issue

    Jaden Alvin, Jul 9, 2014
    Replies:
    19
    Views:
    17,133
    Jaden Alvin
    Nov 24, 2014
  6. wayne_hoton

    How to create robots.txt for bookmarking website. Is it necessary?

    wayne_hoton, Jan 11, 2015
    Replies:
    1
    Views:
    8,351
    wayne_hoton
    Jan 14, 2015
  7. Prasseo

    Will robots file help to cache the content in the web page

    Prasseo, Jan 12, 2015
    Replies:
    1
    Views:
    8,304
    Sycrid
    Jan 18, 2015
  8. Steviebone

    Robots.txt code to disallow by default but allow domain name

    Steviebone, Mar 4, 2015
    Replies:
    0
    Views:
    8,288
    Steviebone
    Mar 4, 2015
  9. enikram

    How to control when a bot(yahoo,google) can crawl a website

    enikram, Mar 18, 2015
    Replies:
    4
    Views:
    18,859
    Nigel Lew
    Mar 18, 2015
  10. Minesh Parikh

    Blocked Url's by robots.txt in google webmaster tool how to fix it?

    Minesh Parikh, Apr 6, 2015
    Replies:
    0
    Views:
    18,047
    Minesh Parikh
    Apr 6, 2015
  11. scott2342

    How do I unblock my robots.txt

    scott2342, Nov 24, 2014
    Replies:
    3
    Views:
    11,662
    meethi jain
    Jun 3, 2015
  12. myfastrupee

    Robot.txt file not found on my blog

    myfastrupee, Jun 21, 2013
    Replies:
    4
    Views:
    15,205
    meethi jain
    Jun 3, 2015
  13. danieloffice

    how to block all other BOT except Google, MSN, Yahoo

    danieloffice, Jan 9, 2008 ... 2
    Replies:
    23
    Views:
    71,396
    moneyroad
    Jul 10, 2015
  14. Prasseo

    How to create a robots file for PHP webpage to crawl

    Prasseo, Jan 12, 2015
    Replies:
    3
    Views:
    12,585
    abinjoy
    Jul 14, 2015
  15. Marker1982

    Very Strange Problem

    Marker1982, Jul 21, 2015
    Replies:
    3
    Views:
    19,945
    Jacob.k
    Jul 25, 2015
  16. full_time_writer

    Robot text and partial fetch issues - need suggestions

    full_time_writer, Jul 28, 2015
    Replies:
    0
    Views:
    10,522
    full_time_writer
    Jul 28, 2015
  17. MarcL

    Fix to ban google search from Cloaking and/or sneaky redirects

    MarcL, Jul 30, 2015
    Replies:
    4
    Views:
    19,844
    MarcL
    Aug 3, 2015
  18. web_master

    i can't stop Baidu, pls help!!!

    web_master, Dec 27, 2011
    Replies:
    7
    Views:
    15,444
    RoundShots
    Sep 11, 2015
  19. MoneyPlanters

    Why cant Google access my robots.txt ?????

    MoneyPlanters, Sep 27, 2015
    Replies:
    1
    Views:
    19,185
    Mit Thakkar
    Sep 27, 2015
  20. gree124

    robots.txt please check

    gree124, Nov 27, 2014
    Replies:
    7
    Views:
    12,629
    rajendra1682
    Oct 6, 2015
  21. anna30

    How to add robots.txt file in Magento

    anna30, Oct 16, 2015
    Replies:
    1
    Views:
    10,174
    paco78
    Oct 16, 2015
  22. Harris96

    Robots.txt error

    Harris96, Jan 24, 2016
    Replies:
    1
    Views:
    8,716
    Nigel Lew
    Jan 24, 2016
  23. Jeffr2014

    Do you block Majestic bots?

    Jeffr2014, Apr 9, 2015
    Replies:
    4
    Views:
    17,894
    neteater
    Feb 20, 2016
  24. Kushal Biswas

    Robots.txt Help to Block a Particular File Name

    Kushal Biswas, Feb 3, 2016
    Replies:
    1
    Views:
    17,598
    Anubhav-soin
    Feb 25, 2016
  25. Xfactormaster

    Robots.txt has 'Disallow: /forums/-/' line, what is the meaning

    Xfactormaster, Mar 1, 2016
    Replies:
    2
    Views:
    8,286
    stilliard
    Mar 7, 2016
  26. Angelabailey

    Robots.txt

    Angelabailey, Nov 15, 2013
    Replies:
    15
    Views:
    21,089
    Panda123
    Mar 27, 2016
  27. soseooooo

    Block a folder except some URL's

    soseooooo, Mar 9, 2016
    Replies:
    1
    Views:
    7,279
    MayurKawale
    Jun 29, 2016
  28. irfansyah

    Problem mobile friendly test

    irfansyah, Jun 23, 2016
    Replies:
    8
    Views:
    7,230
    Host Little
    Jul 14, 2016
  29. ketanseo23

    Robots.txt Issues

    ketanseo23, Apr 14, 2016
    Replies:
    1
    Views:
    7,264
    MayurKawale
    Aug 4, 2016
  30. sensor88

    Addon Domains Showing With Main Domain In URL

    sensor88, Aug 9, 2016
    Replies:
    0
    Views:
    6,303
    sensor88
    Aug 9, 2016
  31. Official Gates

    What is the correct way to write my “robots.txt” file?

    Official Gates, Aug 16, 2016
    Replies:
    0
    Views:
    5,180
    Official Gates
    Aug 16, 2016
  32. Noobnl

    Disallow google to index specific tekst with Robot.txt on webpage?

    Noobnl, Apr 10, 2016
    Replies:
    4
    Views:
    7,154
    Furquan Ahmed
    Aug 25, 2016
  33. shoprat

    Problem with file

    shoprat, Sep 23, 2016
    Replies:
    2
    Views:
    6,205
    JeffMichaels
    Sep 23, 2016
  34. dinrock

    I have default robots.txt and asking how to customize it

    dinrock, Oct 13, 2016
    Replies:
    0
    Views:
    6,518
    dinrock
    Oct 13, 2016
  35. Boxwell

    Can someone tell me what is wrong with my robots.txt file?

    Boxwell, Nov 24, 2016
    Replies:
    2
    Views:
    6,803
    GameO.Over
    Nov 24, 2016
  36. Abdullah adil
    Locked

    robot.txt

    Abdullah adil, Feb 11, 2017
    Replies:
    6
    Views:
    15,264
    sarahk
    Feb 16, 2017
  37. hmansfield

    List of Bad Bots to disallow?

    hmansfield, May 20, 2007
    Replies:
    14
    Views:
    26,565
    Sergio Minozzi
    Feb 22, 2017
  38. smokey99

    Noindex and robots disallow

    smokey99, Dec 22, 2016
    Replies:
    2
    Views:
    6,524
    PoPSiCLe
    Feb 24, 2017
  39. PaRaDiGm13

    Robots.txt Question

    PaRaDiGm13, Mar 16, 2016
    Replies:
    4
    Views:
    7,993
    Cado Magenge
    Feb 28, 2017
  40. Mubashir Ahmad

    Robot.txt or remove the page

    Mubashir Ahmad, Mar 21, 2017
    Replies:
    6
    Views:
    7,692
    Tutorials Feed
    Apr 19, 2017
  41. wartes

    Robot.txt for Google (error)

    wartes, Apr 10, 2017
    Replies:
    6
    Views:
    14,824
    Md.Saddam Hossain
    May 7, 2017
  42. pradip200

    Robots.txt for SMF Forum

    pradip200, Mar 23, 2013
    Replies:
    4
    Views:
    20,299
    Svoboda
    Jun 30, 2017
  43. rants1990

    Vbulletin 4 robots txt?

    rants1990, Sep 19, 2017
    Replies:
    3
    Views:
    5,662
    sarahk
    Sep 19, 2017
  44. Divvy

    Help with robots.txt file

    Divvy, Jun 14, 2017
    Replies:
    4
    Views:
    6,286
    Arun Kumar Soni
    Sep 24, 2017
  45. macau2009

    Google or robots.txt problem

    macau2009, Feb 19, 2018
    Replies:
    6
    Views:
    6,255
    qwikad.com
    Feb 21, 2018
  46. infopages

    Help with Robots.txt

    infopages, Sep 29, 2017
    Replies:
    4
    Views:
    9,601
    Elevate
    Aug 3, 2018
  47. ronkoo
    Locked

    asking you for a information

    ronkoo, Mar 13, 2018
    Replies:
    4
    Views:
    10,312
    hostechsupport
    Aug 17, 2018
  48. Kamaldeep Singh SEO

    Issue Regarding Robots.txt

    Kamaldeep Singh SEO, Oct 24, 2018
    Replies:
    4
    Views:
    6,732
    sigen
    Oct 25, 2018
  49. rosn007

    Not Indexing Well. Is it robots.txt?

    rosn007, Dec 14, 2018
    Replies:
    3
    Views:
    5,803
    Joshuathompson
    Jan 3, 2019
  50. Melissa2007

    Want to exclude TEST folder from Google indexing

    Melissa2007, Feb 3, 2019
    Replies:
    8
    Views:
    5,380
    Melissa2007
    Feb 4, 2019
Showing threads 851 to 900 of 911

Thread Display Options

Loading...
Page 18 of 19