1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Find help with setting up your robots.txt file on your web site.

Page 6 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↑
  1. gree124

    Exert rebots.txt

    gree124, Dec 3, 2007
    Replies:
    2
    Views:
    1,347
    just-4-teens
    Dec 23, 2007
  2. Browning

    I need a robots.txt for my proxy site. Don't know what goes in/how to do this

    Browning, Dec 25, 2007
    Replies:
    2
    Views:
    1,339
    lpstong
    Dec 26, 2007
  3. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    16,090
    danieloffice
    Jan 9, 2008
  4. Mr bloggy

    Will robots search even if I use web forwarding?

    Mr bloggy, Jan 9, 2008
    Replies:
    3
    Views:
    922
    SwapsRulez
    Jan 12, 2008
  5. lowprofilejoe

    Disallow Flash Files

    lowprofilejoe, Jan 4, 2008
    Replies:
    7
    Views:
    1,877
    SwapsRulez
    Jan 12, 2008
  6. ecctao

    Is it necessary to write a list of my all my pages in a tobots.txt

    ecctao, Jan 1, 2008
    Replies:
    4
    Views:
    1,107
    SwapsRulez
    Jan 12, 2008
  7. crystak

    Difference between /category/ and /category disallowing ?

    crystak, Dec 29, 2007
    Replies:
    3
    Views:
    1,051
    SwapsRulez
    Jan 12, 2008
  8. login

    Avoid indexing of ../lyrics/print,74306.html

    login, Jan 11, 2008
    Replies:
    14
    Views:
    1,342
    SwapsRulez
    Jan 14, 2008
  9. dalailama

    avoid robots to index mainsite.com/undersite.com

    dalailama, Jan 11, 2008
    Replies:
    3
    Views:
    1,685
    rich_uk
    Jan 23, 2008
  10. kong707

    is robot.txt very important for a forum?

    kong707, Dec 18, 2007
    Replies:
    8
    Views:
    1,898
    Chris-777
    Feb 2, 2008
  11. nino2007

    What is the problem with Google and my site?

    nino2007, Jan 22, 2008
    Replies:
    3
    Views:
    1,528
    catanich
    Feb 2, 2008
  12. nshadab

    Help required for ROBOTS.txt?

    nshadab, Jan 22, 2008
    Replies:
    4
    Views:
    981
    catanich
    Feb 2, 2008
  13. frio

    Disallow

    frio, Dec 20, 2007
    Replies:
    7
    Views:
    2,111
    catanich
    Feb 2, 2008
  14. Jon12345

    What does this robots.txt text mean?

    Jon12345, Jan 8, 2008
    Replies:
    7
    Views:
    1,732
    catanich
    Feb 2, 2008
  15. Jinx3000

    First time setting up robots.txt

    Jinx3000, Jan 14, 2008
    Replies:
    7
    Views:
    1,320
    catanich
    Feb 2, 2008
  16. coletrickle

    Robots.txt not visible via HTTP for wordpress blog...

    coletrickle, Feb 11, 2008
    Replies:
    4
    Views:
    1,210
    coletrickle
    Feb 18, 2008
  17. mytechguru

    URLs restricted by robots.txt

    mytechguru, Mar 6, 2008
    Replies:
    4
    Views:
    1,625
    LittleJonSupportSite
    Mar 7, 2008
  18. jtothek3030

    How do i stop Internet Shinchakubin?

    jtothek3030, May 11, 2007
    Replies:
    13
    Views:
    4,470
    scoopy82
    Mar 10, 2008
  19. warley

    Help Required

    warley, Mar 11, 2008
    Replies:
    2
    Views:
    778
    warley
    Mar 11, 2008
  20. HomeRun

    Is this a good robots.txt for vbulletin?

    HomeRun, Mar 25, 2008
    Replies:
    2
    Views:
    1,253
    FightRice
    Mar 31, 2008
  21. artboy

    How to get robots.txt re-crawled by google/yahoo, etc?

    artboy, Mar 10, 2008
    Replies:
    7
    Views:
    1,461
    manish.chauhan
    Apr 4, 2008
  22. coolfriend

    Method to edit "robots.txt" in blogspot

    coolfriend, Mar 25, 2008
    Replies:
    3
    Views:
    2,207
    manish.chauhan
    Apr 4, 2008
  23. TheSites

    How to block out sessions from being indexed

    TheSites, Mar 29, 2008
    Replies:
    1
    Views:
    952
    manish.chauhan
    Apr 4, 2008
  24. templates

    ? abouta bot

    templates, Feb 23, 2008
    Replies:
    2
    Views:
    1,286
    manish.chauhan
    Apr 4, 2008
  25. HomeRun

    Is this a valid robots.txt file?

    HomeRun, Mar 17, 2008
    Replies:
    6
    Views:
    1,030
    manish.chauhan
    Apr 4, 2008
  26. BigJuicy

    I'm a NOOB google wont spider me =(

    BigJuicy, Dec 11, 2007
    Replies:
    5
    Views:
    1,401
    Dreads
    Apr 4, 2008
  27. crazyryan

    Stop ANY search engine indexing my site

    crazyryan, Feb 2, 2008
    Replies:
    7
    Views:
    1,961
    manish.chauhan
    Apr 4, 2008
  28. HomeRun

    How can I block / disallow this spider?

    HomeRun, Mar 20, 2008
    Replies:
    2
    Views:
    1,074
    manish.chauhan
    Apr 5, 2008
  29. ialwaysforget

    robot.txt help

    ialwaysforget, Mar 13, 2008
    Replies:
    7
    Views:
    1,421
    manish.chauhan
    Apr 6, 2008
  30. Death4you

    [.htaccess and bad bots list] robots.txt of course not works

    Death4you, Mar 22, 2008
    Replies:
    4
    Views:
    1,993
    manish.chauhan
    Apr 6, 2008
  31. angellica2017

    Dangerous Crawler and User Agent! Must Read!

    angellica2017, Sep 19, 2006 ... 2
    Replies:
    22
    Views:
    11,414
    manish.chauhan
    Apr 6, 2008
  32. Malmsteen

    Quick Question on a robots.txt file - help!?

    Malmsteen, Feb 29, 2008
    Replies:
    3
    Views:
    1,184
    manish.chauhan
    Apr 6, 2008
  33. Giorgi

    how to completly block engine from crawling my site?

    Giorgi, Jan 16, 2008
    Replies:
    15
    Views:
    2,001
    manish.chauhan
    Apr 7, 2008
  34. kie

    Prohibit SE to crawl subdomain

    kie, Jan 23, 2008
    Replies:
    7
    Views:
    1,575
    manish.chauhan
    Apr 7, 2008
  35. gogo2005

    Error on my site Robots could block them?

    gogo2005, Dec 19, 2007
    Replies:
    4
    Views:
    1,940
    manish.chauhan
    Apr 7, 2008
  36. Maciej Ziemczonek

    robots.txt and IP addresses

    Maciej Ziemczonek, Apr 8, 2008
    Replies:
    2
    Views:
    1,158
    Maciej Ziemczonek
    Apr 8, 2008
  37. movvadinesh

    What is robot.txt and how to add this to blog

    movvadinesh, Feb 2, 2008
    Replies:
    4
    Views:
    2,124
    manish.chauhan
    Apr 9, 2008
  38. sudip_dg77

    Robot.txt?

    sudip_dg77, Apr 4, 2008
    Replies:
    6
    Views:
    1,336
    SubmitShop
    Apr 10, 2008
  39. IndianMonk

    Need help In my robots.txt

    IndianMonk, Apr 10, 2008
    Replies:
    4
    Views:
    1,114
    manish.chauhan
    Apr 13, 2008
  40. TheDutchMan

    Automatic no-follow???

    TheDutchMan, Apr 13, 2008
    Replies:
    6
    Views:
    1,240
    henery008
    Apr 17, 2008
  41. abcdefGARY

    quick question about robots.txt

    abcdefGARY, Feb 9, 2008
    Replies:
    9
    Views:
    1,567
    manish.chauhan
    Apr 20, 2008
  42. smoked1

    Help with robots.txt and dynamic URLs

    smoked1, Apr 25, 2008
    Replies:
    3
    Views:
    1,314
    smoked1
    Apr 28, 2008
  43. lateuk

    Blocking a forum with robots.txt

    lateuk, May 1, 2008
    Replies:
    1
    Views:
    1,068
    manish.chauhan
    May 1, 2008
  44. Molon Lave

    Bandwidth problem due to bots

    Molon Lave, Apr 10, 2008
    Replies:
    4
    Views:
    1,926
    manish.chauhan
    May 7, 2008
  45. bryanw

    Help needed - ban robots from crawling redirect script

    bryanw, May 7, 2008
    Replies:
    9
    Views:
    1,270
    Trusted Writer
    May 7, 2008
  46. akhilesh243

    Ideal Robots.txt file for wordpress website

    akhilesh243, Nov 19, 2007
    Replies:
    8
    Views:
    5,798
    arunsubru
    May 12, 2008
  47. mnymkr

    robot.txt wildcards are they allowed?

    mnymkr, May 4, 2008
    Replies:
    2
    Views:
    3,047
    Trusted Writer
    May 13, 2008
  48. nitu

    Use of robots.txt

    nitu, May 3, 2008
    Replies:
    3
    Views:
    1,105
    fifty-one
    May 17, 2008
  49. SeoVeteran33

    Wicked Dump Robots Question

    SeoVeteran33, May 17, 2008
    Replies:
    2
    Views:
    1,061
    manish.chauhan
    May 20, 2008
  50. risoknop

    Review my robots.txt

    risoknop, May 20, 2008
    Replies:
    6
    Views:
    1,187
    manish.chauhan
    May 20, 2008
Showing threads 251 to 300 of 911

Thread Display Options

Loading...
Page 6 of 19