robots.txt

Find help with setting up your robots.txt file on your web site.

Page 1 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. Johnsmi2323

    Robots

    Johnsmi2323, Aug 23, 2023
    Replies:
    1
    Views:
    2,599
    seomanualsubmission
    Aug 24, 2023
  2. seoworld1234

    How to check robots.txt on site?

    seoworld1234, Apr 21, 2008 ... 2
    Replies:
    33
    Views:
    86,059
    imranbashir.seo
    Mar 7, 2023
  3. Mrinal_Roy

    Robots.txt is blocking my labels

    Mrinal_Roy, Feb 5, 2015
    Replies:
    5
    Views:
    15,787
    Gideon Duah
    Jul 21, 2021
  4. MayurKawale

    Is it good to block BaiduSpider from accessing/crawling the site?

    MayurKawale, Jan 7, 2021
    Replies:
    1
    Views:
    3,914
    qwikad.com
    Jan 10, 2021
  5. kabali

    importance of robots.txt

    kabali, Sep 27, 2016
    Replies:
    5
    Views:
    7,621
    MayurKawale
    Nov 24, 2020
  6. dabidovic123

    Disallow all Bots

    dabidovic123, Oct 13, 2019
    Replies:
    4
    Views:
    5,667
    boshiant
    Jun 6, 2020
  7. Mr.Dog

    Ban subdomain from robots, stop Google from indexing it

    Mr.Dog, Dec 1, 2018
    Replies:
    6
    Views:
    5,732
    Borislav Arapchev
    May 13, 2020
  8. vema123

    How to Block All Robots to Crawl/Index All html Pages within My Website

    vema123, Jul 9, 2019
    Replies:
    2
    Views:
    4,809
    mmerlinn
    Jul 9, 2019
  9. gree124

    Robots is good

    gree124, May 2, 2019
    Replies:
    0
    Views:
    4,288
    gree124
    May 2, 2019
  10. Melissa2007

    Is there a robots.txt interpreter for those of us who are newbies?

    Melissa2007, Apr 29, 2019
    Replies:
    2
    Views:
    4,330
    Melissa2007
    Apr 30, 2019
  11. Melissa2007

    Why do I need a robots.txt , and what to put in it?

    Melissa2007, Feb 23, 2019
    Replies:
    8
    Views:
    5,837
    mmerlinn
    Feb 26, 2019
  12. Melissa2007

    Want to exclude TEST folder from Google indexing

    Melissa2007, Feb 3, 2019
    Replies:
    8
    Views:
    5,067
    Melissa2007
    Feb 4, 2019
  13. rosn007

    Not Indexing Well. Is it robots.txt?

    rosn007, Dec 14, 2018
    Replies:
    3
    Views:
    5,526
    Joshuathompson
    Jan 3, 2019
  14. Kamaldeep Singh SEO

    Issue Regarding Robots.txt

    Kamaldeep Singh SEO, Oct 24, 2018
    Replies:
    4
    Views:
    6,441
    sigen
    Oct 25, 2018
  15. ronkoo
    Locked

    asking you for a information

    ronkoo, Mar 13, 2018
    Replies:
    4
    Views:
    10,026
    hostechsupport
    Aug 17, 2018
  16. infopages

    Help with Robots.txt

    infopages, Sep 29, 2017
    Replies:
    4
    Views:
    9,290
    Elevate
    Aug 3, 2018
  17. macau2009

    Google or robots.txt problem

    macau2009, Feb 19, 2018
    Replies:
    6
    Views:
    5,966
    qwikad.com
    Feb 21, 2018
  18. Divvy

    Help with robots.txt file

    Divvy, Jun 14, 2017
    Replies:
    4
    Views:
    6,020
    Arun Kumar Soni
    Sep 24, 2017
  19. rants1990

    Vbulletin 4 robots txt?

    rants1990, Sep 19, 2017
    Replies:
    3
    Views:
    5,389
    sarahk
    Sep 19, 2017
  20. pradip200

    Robots.txt for SMF Forum

    pradip200, Mar 23, 2013
    Replies:
    4
    Views:
    19,972
    Svoboda
    Jun 30, 2017
  21. wartes

    Robot.txt for Google (error)

    wartes, Apr 10, 2017
    Replies:
    6
    Views:
    14,531
    Md.Saddam Hossain
    May 7, 2017
  22. Mubashir Ahmad

    Robot.txt or remove the page

    Mubashir Ahmad, Mar 21, 2017
    Replies:
    6
    Views:
    7,414
    Tutorials Feed
    Apr 19, 2017
  23. PaRaDiGm13

    Robots.txt Question

    PaRaDiGm13, Mar 16, 2016
    Replies:
    4
    Views:
    7,715
    Cado Magenge
    Feb 28, 2017
  24. smokey99

    Noindex and robots disallow

    smokey99, Dec 22, 2016
    Replies:
    2
    Views:
    6,265
    PoPSiCLe
    Feb 24, 2017
  25. hmansfield

    List of Bad Bots to disallow?

    hmansfield, May 20, 2007
    Replies:
    14
    Views:
    26,158
    Sergio Minozzi
    Feb 22, 2017
  26. Abdullah adil
    Locked

    robot.txt

    Abdullah adil, Feb 11, 2017
    Replies:
    6
    Views:
    14,974
    sarahk
    Feb 16, 2017
  27. Boxwell

    Can someone tell me what is wrong with my robots.txt file?

    Boxwell, Nov 24, 2016
    Replies:
    2
    Views:
    6,539
    GameO.Over
    Nov 24, 2016
  28. dinrock

    I have default robots.txt and asking how to customize it

    dinrock, Oct 13, 2016
    Replies:
    0
    Views:
    6,265
    dinrock
    Oct 13, 2016
  29. shoprat

    Problem with file

    shoprat, Sep 23, 2016
    Replies:
    2
    Views:
    5,939
    JeffMichaels
    Sep 23, 2016
  30. Noobnl

    Disallow google to index specific tekst with Robot.txt on webpage?

    Noobnl, Apr 10, 2016
    Replies:
    4
    Views:
    6,873
    Furquan Ahmed
    Aug 25, 2016
  31. Official Gates

    What is the correct way to write my “robots.txt” file?

    Official Gates, Aug 16, 2016
    Replies:
    0
    Views:
    5,086
    Official Gates
    Aug 16, 2016
  32. sensor88

    Addon Domains Showing With Main Domain In URL

    sensor88, Aug 9, 2016
    Replies:
    0
    Views:
    6,060
    sensor88
    Aug 9, 2016
  33. ketanseo23

    Robots.txt Issues

    ketanseo23, Apr 14, 2016
    Replies:
    1
    Views:
    7,015
    MayurKawale
    Aug 4, 2016
  34. irfansyah

    Problem mobile friendly test

    irfansyah, Jun 23, 2016
    Replies:
    8
    Views:
    6,936
    Host Little
    Jul 14, 2016
  35. soseooooo

    Block a folder except some URL's

    soseooooo, Mar 9, 2016
    Replies:
    1
    Views:
    7,030
    MayurKawale
    Jun 29, 2016
  36. Angelabailey

    Robots.txt

    Angelabailey, Nov 15, 2013
    Replies:
    15
    Views:
    20,721
    Panda123
    Mar 27, 2016
  37. Xfactormaster

    Robots.txt has 'Disallow: /forums/-/' line, what is the meaning

    Xfactormaster, Mar 1, 2016
    Replies:
    2
    Views:
    8,010
    stilliard
    Mar 7, 2016
  38. Kushal Biswas

    Robots.txt Help to Block a Particular File Name

    Kushal Biswas, Feb 3, 2016
    Replies:
    1
    Views:
    17,347
    Anubhav-soin
    Feb 25, 2016
  39. Jeffr2014

    Do you block Majestic bots?

    Jeffr2014, Apr 9, 2015
    Replies:
    4
    Views:
    17,557
    neteater
    Feb 20, 2016
  40. Harris96

    Robots.txt error

    Harris96, Jan 24, 2016
    Replies:
    1
    Views:
    8,445
    Nigel Lew
    Jan 24, 2016
  41. anna30

    How to add robots.txt file in Magento

    anna30, Oct 16, 2015
    Replies:
    1
    Views:
    9,910
    paco78
    Oct 16, 2015
  42. gree124

    robots.txt please check

    gree124, Nov 27, 2014
    Replies:
    7
    Views:
    12,386
    rajendra1682
    Oct 6, 2015
  43. MoneyPlanters

    Why cant Google access my robots.txt ?????

    MoneyPlanters, Sep 27, 2015
    Replies:
    1
    Views:
    18,931
    Mit Thakkar
    Sep 27, 2015
  44. web_master

    i can't stop Baidu, pls help!!!

    web_master, Dec 27, 2011
    Replies:
    7
    Views:
    15,190
    RoundShots
    Sep 11, 2015
  45. MarcL

    Fix to ban google search from Cloaking and/or sneaky redirects

    MarcL, Jul 30, 2015
    Replies:
    4
    Views:
    19,582
    MarcL
    Aug 3, 2015
  46. full_time_writer

    Robot text and partial fetch issues - need suggestions

    full_time_writer, Jul 28, 2015
    Replies:
    0
    Views:
    10,282
    full_time_writer
    Jul 28, 2015
  47. Marker1982

    Very Strange Problem

    Marker1982, Jul 21, 2015
    Replies:
    3
    Views:
    19,689
    Jacob.k
    Jul 25, 2015
  48. Prasseo

    How to create a robots file for PHP webpage to crawl

    Prasseo, Jan 12, 2015
    Replies:
    3
    Views:
    12,335
    abinjoy
    Jul 14, 2015
  49. danieloffice

    how to block all other BOT except Google, MSN, Yahoo

    danieloffice, Jan 9, 2008 ... 2
    Replies:
    23
    Views:
    70,649
    moneyroad
    Jul 10, 2015
  50. myfastrupee

    Robot.txt file not found on my blog

    myfastrupee, Jun 21, 2013
    Replies:
    4
    Views:
    14,883
    meethi jain
    Jun 3, 2015
Showing threads 1 to 50 of 911

Thread Display Options

Loading...
Page 1 of 19