1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 1 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. Johnsmi2323

    Robots

    Johnsmi2323, Aug 23, 2023
    Replies:
    1
    Views:
    1,337
    seomanualsubmission
    Aug 24, 2023
  2. seoworld1234

    How to check robots.txt on site?

    seoworld1234, Apr 21, 2008 ... 2
    Replies:
    33
    Views:
    82,971
    imranbashir.seo
    Mar 7, 2023
  3. Mrinal_Roy

    Robots.txt is blocking my labels

    Mrinal_Roy, Feb 5, 2015
    Replies:
    5
    Views:
    14,308
    Gideon Duah
    Jul 21, 2021
  4. MayurKawale

    Is it good to block BaiduSpider from accessing/crawling the site?

    MayurKawale, Jan 7, 2021
    Replies:
    1
    Views:
    2,784
    qwikad.com
    Jan 10, 2021
  5. kabali

    importance of robots.txt

    kabali, Sep 27, 2016
    Replies:
    5
    Views:
    6,189
    MayurKawale
    Nov 24, 2020
  6. dabidovic123

    Disallow all Bots

    dabidovic123, Oct 13, 2019
    Replies:
    4
    Views:
    4,510
    boshiant
    Jun 6, 2020
  7. Mr.Dog

    Ban subdomain from robots, stop Google from indexing it

    Mr.Dog, Dec 1, 2018
    Replies:
    6
    Views:
    4,567
    Borislav Arapchev
    May 13, 2020
  8. vema123

    How to Block All Robots to Crawl/Index All html Pages within My Website

    vema123, Jul 9, 2019
    Replies:
    2
    Views:
    3,553
    mmerlinn
    Jul 9, 2019
  9. gree124

    Robots is good

    gree124, May 2, 2019
    Replies:
    0
    Views:
    3,237
    gree124
    May 2, 2019
  10. Melissa2007

    Is there a robots.txt interpreter for those of us who are newbies?

    Melissa2007, Apr 29, 2019
    Replies:
    2
    Views:
    3,217
    Melissa2007
    Apr 30, 2019
  11. Melissa2007

    Why do I need a robots.txt , and what to put in it?

    Melissa2007, Feb 23, 2019
    Replies:
    8
    Views:
    4,198
    mmerlinn
    Feb 26, 2019
  12. Melissa2007

    Want to exclude TEST folder from Google indexing

    Melissa2007, Feb 3, 2019
    Replies:
    8
    Views:
    3,779
    Melissa2007
    Feb 4, 2019
  13. rosn007

    Not Indexing Well. Is it robots.txt?

    rosn007, Dec 14, 2018
    Replies:
    3
    Views:
    4,380
    Joshuathompson
    Jan 3, 2019
  14. Kamaldeep Singh SEO

    Issue Regarding Robots.txt

    Kamaldeep Singh SEO, Oct 24, 2018
    Replies:
    4
    Views:
    5,180
    sigen
    Oct 25, 2018
  15. ronkoo
    Locked

    asking you for a information

    ronkoo, Mar 13, 2018
    Replies:
    4
    Views:
    8,790
    hostechsupport
    Aug 17, 2018
  16. infopages

    Help with Robots.txt

    infopages, Sep 29, 2017
    Replies:
    4
    Views:
    8,065
    Elevate
    Aug 3, 2018
  17. macau2009

    Google or robots.txt problem

    macau2009, Feb 19, 2018
    Replies:
    6
    Views:
    4,757
    qwikad.com
    Feb 21, 2018
  18. Divvy

    Help with robots.txt file

    Divvy, Jun 14, 2017
    Replies:
    4
    Views:
    4,892
    Arun Kumar Soni
    Sep 24, 2017
  19. rants1990

    Vbulletin 4 robots txt?

    rants1990, Sep 19, 2017
    Replies:
    3
    Views:
    4,200
    sarahk
    Sep 19, 2017
  20. pradip200

    Robots.txt for SMF Forum

    pradip200, Mar 23, 2013
    Replies:
    4
    Views:
    18,345
    Svoboda
    Jun 30, 2017
  21. wartes

    Robot.txt for Google (error)

    wartes, Apr 10, 2017
    Replies:
    6
    Views:
    13,292
    Md.Saddam Hossain
    May 7, 2017
  22. Mubashir Ahmad

    Robot.txt or remove the page

    Mubashir Ahmad, Mar 21, 2017
    Replies:
    6
    Views:
    6,202
    Tutorials Feed
    Apr 19, 2017
  23. PaRaDiGm13

    Robots.txt Question

    PaRaDiGm13, Mar 16, 2016
    Replies:
    4
    Views:
    6,574
    Cado Magenge
    Feb 28, 2017
  24. smokey99

    Noindex and robots disallow

    smokey99, Dec 22, 2016
    Replies:
    2
    Views:
    5,191
    PoPSiCLe
    Feb 24, 2017
  25. hmansfield

    List of Bad Bots to disallow?

    hmansfield, May 20, 2007
    Replies:
    14
    Views:
    24,417
    Sergio Minozzi
    Feb 22, 2017
  26. Abdullah adil
    Locked

    robot.txt

    Abdullah adil, Feb 11, 2017
    Replies:
    6
    Views:
    13,750
    sarahk
    Feb 16, 2017
  27. Boxwell

    Can someone tell me what is wrong with my robots.txt file?

    Boxwell, Nov 24, 2016
    Replies:
    2
    Views:
    5,425
    GameO.Over
    Nov 24, 2016
  28. dinrock

    I have default robots.txt and asking how to customize it

    dinrock, Oct 13, 2016
    Replies:
    0
    Views:
    5,186
    dinrock
    Oct 13, 2016
  29. shoprat

    Problem with file

    shoprat, Sep 23, 2016
    Replies:
    2
    Views:
    4,780
    JeffMichaels
    Sep 23, 2016
  30. Noobnl

    Disallow google to index specific tekst with Robot.txt on webpage?

    Noobnl, Apr 10, 2016
    Replies:
    4
    Views:
    5,786
    Furquan Ahmed
    Aug 25, 2016
  31. Official Gates

    What is the correct way to write my “robots.txt” file?

    Official Gates, Aug 16, 2016
    Replies:
    0
    Views:
    4,496
    Official Gates
    Aug 16, 2016
  32. sensor88

    Addon Domains Showing With Main Domain In URL

    sensor88, Aug 9, 2016
    Replies:
    0
    Views:
    4,973
    sensor88
    Aug 9, 2016
  33. ketanseo23

    Robots.txt Issues

    ketanseo23, Apr 14, 2016
    Replies:
    1
    Views:
    5,932
    MayurKawale
    Aug 4, 2016
  34. irfansyah

    Problem mobile friendly test

    irfansyah, Jun 23, 2016
    Replies:
    8
    Views:
    5,685
    Host Little
    Jul 14, 2016
  35. soseooooo

    Block a folder except some URL's

    soseooooo, Mar 9, 2016
    Replies:
    1
    Views:
    5,936
    MayurKawale
    Jun 29, 2016
  36. Angelabailey

    Robots.txt

    Angelabailey, Nov 15, 2013
    Replies:
    15
    Views:
    19,255
    Panda123
    Mar 27, 2016
  37. Xfactormaster

    Robots.txt has 'Disallow: /forums/-/' line, what is the meaning

    Xfactormaster, Mar 1, 2016
    Replies:
    2
    Views:
    6,790
    stilliard
    Mar 7, 2016
  38. Kushal Biswas

    Robots.txt Help to Block a Particular File Name

    Kushal Biswas, Feb 3, 2016
    Replies:
    1
    Views:
    16,268
    Anubhav-soin
    Feb 25, 2016
  39. Jeffr2014

    Do you block Majestic bots?

    Jeffr2014, Apr 9, 2015
    Replies:
    4
    Views:
    16,179
    neteater
    Feb 20, 2016
  40. Harris96

    Robots.txt error

    Harris96, Jan 24, 2016
    Replies:
    1
    Views:
    7,241
    Nigel Lew
    Jan 24, 2016
  41. anna30

    How to add robots.txt file in Magento

    anna30, Oct 16, 2015
    Replies:
    1
    Views:
    8,812
    paco78
    Oct 16, 2015
  42. gree124

    robots.txt please check

    gree124, Nov 27, 2014
    Replies:
    7
    Views:
    11,278
    rajendra1682
    Oct 6, 2015
  43. MoneyPlanters

    Why cant Google access my robots.txt ?????

    MoneyPlanters, Sep 27, 2015
    Replies:
    1
    Views:
    17,794
    Mit Thakkar
    Sep 27, 2015
  44. web_master

    i can't stop Baidu, pls help!!!

    web_master, Dec 27, 2011
    Replies:
    7
    Views:
    13,936
    RoundShots
    Sep 11, 2015
  45. MarcL

    Fix to ban google search from Cloaking and/or sneaky redirects

    MarcL, Jul 30, 2015
    Replies:
    4
    Views:
    18,462
    MarcL
    Aug 3, 2015
  46. full_time_writer

    Robot text and partial fetch issues - need suggestions

    full_time_writer, Jul 28, 2015
    Replies:
    0
    Views:
    9,194
    full_time_writer
    Jul 28, 2015
  47. Marker1982

    Very Strange Problem

    Marker1982, Jul 21, 2015
    Replies:
    3
    Views:
    18,567
    Jacob.k
    Jul 25, 2015
  48. Prasseo

    How to create a robots file for PHP webpage to crawl

    Prasseo, Jan 12, 2015
    Replies:
    3
    Views:
    11,212
    abinjoy
    Jul 14, 2015
  49. danieloffice

    how to block all other BOT except Google, MSN, Yahoo

    danieloffice, Jan 9, 2008 ... 2
    Replies:
    23
    Views:
    67,601
    moneyroad
    Jul 10, 2015
  50. myfastrupee

    Robot.txt file not found on my blog

    myfastrupee, Jun 21, 2013
    Replies:
    4
    Views:
    13,596
    meethi jain
    Jun 3, 2015
Showing threads 1 to 50 of 911

Thread Display Options

Loading...
Page 1 of 19