robots.txt

Find help with setting up your robots.txt file on your web site.

Page 6 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↑
  1. gree124

    Exert rebots.txt

    gree124, Dec 3, 2007
    Replies:
    2
    Views:
    1,487
    just-4-teens
    Dec 23, 2007
  2. Browning

    I need a robots.txt for my proxy site. Don't know what goes in/how to do this

    Browning, Dec 25, 2007
    Replies:
    2
    Views:
    1,450
    lpstong
    Dec 26, 2007
  3. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    17,414
    danieloffice
    Jan 9, 2008
  4. Mr bloggy

    Will robots search even if I use web forwarding?

    Mr bloggy, Jan 9, 2008
    Replies:
    3
    Views:
    1,039
    SwapsRulez
    Jan 12, 2008
  5. lowprofilejoe

    Disallow Flash Files

    lowprofilejoe, Jan 4, 2008
    Replies:
    7
    Views:
    1,996
    SwapsRulez
    Jan 12, 2008
  6. ecctao

    Is it necessary to write a list of my all my pages in a tobots.txt

    ecctao, Jan 1, 2008
    Replies:
    4
    Views:
    1,219
    SwapsRulez
    Jan 12, 2008
  7. crystak

    Difference between /category/ and /category disallowing ?

    crystak, Dec 29, 2007
    Replies:
    3
    Views:
    1,199
    SwapsRulez
    Jan 12, 2008
  8. login

    Avoid indexing of ../lyrics/print,74306.html

    login, Jan 11, 2008
    Replies:
    14
    Views:
    1,592
    SwapsRulez
    Jan 14, 2008
  9. dalailama

    avoid robots to index mainsite.com/undersite.com

    dalailama, Jan 11, 2008
    Replies:
    3
    Views:
    1,813
    rich_uk
    Jan 23, 2008
  10. kong707

    is robot.txt very important for a forum?

    kong707, Dec 18, 2007
    Replies:
    8
    Views:
    2,094
    Chris-777
    Feb 2, 2008
  11. nino2007

    What is the problem with Google and my site?

    nino2007, Jan 22, 2008
    Replies:
    3
    Views:
    1,649
    catanich
    Feb 2, 2008
  12. nshadab

    Help required for ROBOTS.txt?

    nshadab, Jan 22, 2008
    Replies:
    4
    Views:
    1,059
    catanich
    Feb 2, 2008
  13. frio

    Disallow

    frio, Dec 20, 2007
    Replies:
    7
    Views:
    2,260
    catanich
    Feb 2, 2008
  14. Jon12345

    What does this robots.txt text mean?

    Jon12345, Jan 8, 2008
    Replies:
    7
    Views:
    1,882
    catanich
    Feb 2, 2008
  15. Jinx3000

    First time setting up robots.txt

    Jinx3000, Jan 14, 2008
    Replies:
    7
    Views:
    1,439
    catanich
    Feb 2, 2008
  16. coletrickle

    Robots.txt not visible via HTTP for wordpress blog...

    coletrickle, Feb 11, 2008
    Replies:
    4
    Views:
    1,301
    coletrickle
    Feb 18, 2008
  17. mytechguru

    URLs restricted by robots.txt

    mytechguru, Mar 6, 2008
    Replies:
    4
    Views:
    1,729
    LittleJonSupportSite
    Mar 7, 2008
  18. jtothek3030

    How do i stop Internet Shinchakubin?

    jtothek3030, May 11, 2007
    Replies:
    13
    Views:
    4,607
    scoopy82
    Mar 10, 2008
  19. warley

    Help Required

    warley, Mar 11, 2008
    Replies:
    2
    Views:
    865
    warley
    Mar 11, 2008
  20. HomeRun

    Is this a good robots.txt for vbulletin?

    HomeRun, Mar 25, 2008
    Replies:
    2
    Views:
    1,324
    FightRice
    Mar 31, 2008
  21. artboy

    How to get robots.txt re-crawled by google/yahoo, etc?

    artboy, Mar 10, 2008
    Replies:
    7
    Views:
    1,599
    manish.chauhan
    Apr 4, 2008
  22. coolfriend

    Method to edit "robots.txt" in blogspot

    coolfriend, Mar 25, 2008
    Replies:
    3
    Views:
    2,302
    manish.chauhan
    Apr 4, 2008
  23. TheSites

    How to block out sessions from being indexed

    TheSites, Mar 29, 2008
    Replies:
    1
    Views:
    1,043
    manish.chauhan
    Apr 4, 2008
  24. templates

    ? abouta bot

    templates, Feb 23, 2008
    Replies:
    2
    Views:
    1,382
    manish.chauhan
    Apr 4, 2008
  25. HomeRun

    Is this a valid robots.txt file?

    HomeRun, Mar 17, 2008
    Replies:
    6
    Views:
    1,159
    manish.chauhan
    Apr 4, 2008
  26. BigJuicy

    I'm a NOOB google wont spider me =(

    BigJuicy, Dec 11, 2007
    Replies:
    5
    Views:
    1,507
    Dreads
    Apr 4, 2008
  27. crazyryan

    Stop ANY search engine indexing my site

    crazyryan, Feb 2, 2008
    Replies:
    7
    Views:
    2,191
    manish.chauhan
    Apr 4, 2008
  28. HomeRun

    How can I block / disallow this spider?

    HomeRun, Mar 20, 2008
    Replies:
    2
    Views:
    1,169
    manish.chauhan
    Apr 5, 2008
  29. ialwaysforget

    robot.txt help

    ialwaysforget, Mar 13, 2008
    Replies:
    7
    Views:
    1,558
    manish.chauhan
    Apr 6, 2008
  30. Death4you

    [.htaccess and bad bots list] robots.txt of course not works

    Death4you, Mar 22, 2008
    Replies:
    4
    Views:
    2,089
    manish.chauhan
    Apr 6, 2008
  31. angellica2017

    Dangerous Crawler and User Agent! Must Read!

    angellica2017, Sep 19, 2006 ... 2
    Replies:
    22
    Views:
    11,870
    manish.chauhan
    Apr 6, 2008
  32. Malmsteen

    Quick Question on a robots.txt file - help!?

    Malmsteen, Feb 29, 2008
    Replies:
    3
    Views:
    1,262
    manish.chauhan
    Apr 6, 2008
  33. Giorgi

    how to completly block engine from crawling my site?

    Giorgi, Jan 16, 2008
    Replies:
    15
    Views:
    2,197
    manish.chauhan
    Apr 7, 2008
  34. kie

    Prohibit SE to crawl subdomain

    kie, Jan 23, 2008
    Replies:
    7
    Views:
    1,654
    manish.chauhan
    Apr 7, 2008
  35. gogo2005

    Error on my site Robots could block them?

    gogo2005, Dec 19, 2007
    Replies:
    4
    Views:
    2,078
    manish.chauhan
    Apr 7, 2008
  36. Maciej Ziemczonek

    robots.txt and IP addresses

    Maciej Ziemczonek, Apr 8, 2008
    Replies:
    2
    Views:
    1,280
    Maciej Ziemczonek
    Apr 8, 2008
  37. movvadinesh

    What is robot.txt and how to add this to blog

    movvadinesh, Feb 2, 2008
    Replies:
    4
    Views:
    2,224
    manish.chauhan
    Apr 9, 2008
  38. sudip_dg77

    Robot.txt?

    sudip_dg77, Apr 4, 2008
    Replies:
    6
    Views:
    1,443
    SubmitShop
    Apr 10, 2008
  39. IndianMonk

    Need help In my robots.txt

    IndianMonk, Apr 10, 2008
    Replies:
    4
    Views:
    1,183
    manish.chauhan
    Apr 13, 2008
  40. TheDutchMan

    Automatic no-follow???

    TheDutchMan, Apr 13, 2008
    Replies:
    6
    Views:
    1,370
    henery008
    Apr 17, 2008
  41. abcdefGARY

    quick question about robots.txt

    abcdefGARY, Feb 9, 2008
    Replies:
    9
    Views:
    1,771
    manish.chauhan
    Apr 20, 2008
  42. smoked1

    Help with robots.txt and dynamic URLs

    smoked1, Apr 25, 2008
    Replies:
    3
    Views:
    1,399
    smoked1
    Apr 28, 2008
  43. lateuk

    Blocking a forum with robots.txt

    lateuk, May 1, 2008
    Replies:
    1
    Views:
    1,148
    manish.chauhan
    May 1, 2008
  44. Molon Lave

    Bandwidth problem due to bots

    Molon Lave, Apr 10, 2008
    Replies:
    4
    Views:
    2,099
    manish.chauhan
    May 7, 2008
  45. bryanw

    Help needed - ban robots from crawling redirect script

    bryanw, May 7, 2008
    Replies:
    9
    Views:
    1,442
    Trusted Writer
    May 7, 2008
  46. akhilesh243

    Ideal Robots.txt file for wordpress website

    akhilesh243, Nov 19, 2007
    Replies:
    8
    Views:
    6,207
    arunsubru
    May 12, 2008
  47. mnymkr

    robot.txt wildcards are they allowed?

    mnymkr, May 4, 2008
    Replies:
    2
    Views:
    3,181
    Trusted Writer
    May 13, 2008
  48. nitu

    Use of robots.txt

    nitu, May 3, 2008
    Replies:
    3
    Views:
    1,219
    fifty-one
    May 17, 2008
  49. SeoVeteran33

    Wicked Dump Robots Question

    SeoVeteran33, May 17, 2008
    Replies:
    2
    Views:
    1,178
    manish.chauhan
    May 20, 2008
  50. risoknop

    Review my robots.txt

    risoknop, May 20, 2008
    Replies:
    6
    Views:
    1,315
    manish.chauhan
    May 20, 2008
Showing threads 251 to 300 of 911

Thread Display Options

Loading...
Page 6 of 19