robots.txt

Find help with setting up your robots.txt file on your web site.

Page 14 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. nino2007

    What is the problem with Google and my site?

    nino2007, Jan 22, 2008
    Replies:
    3
    Views:
    1,631
    catanich
    Feb 2, 2008
  2. kong707

    is robot.txt very important for a forum?

    kong707, Dec 18, 2007
    Replies:
    8
    Views:
    2,058
    Chris-777
    Feb 2, 2008
  3. dalailama

    avoid robots to index mainsite.com/undersite.com

    dalailama, Jan 11, 2008
    Replies:
    3
    Views:
    1,784
    rich_uk
    Jan 23, 2008
  4. login

    Avoid indexing of ../lyrics/print,74306.html

    login, Jan 11, 2008
    Replies:
    14
    Views:
    1,541
    SwapsRulez
    Jan 14, 2008
  5. crystak

    Difference between /category/ and /category disallowing ?

    crystak, Dec 29, 2007
    Replies:
    3
    Views:
    1,169
    SwapsRulez
    Jan 12, 2008
  6. ecctao

    Is it necessary to write a list of my all my pages in a tobots.txt

    ecctao, Jan 1, 2008
    Replies:
    4
    Views:
    1,200
    SwapsRulez
    Jan 12, 2008
  7. lowprofilejoe

    Disallow Flash Files

    lowprofilejoe, Jan 4, 2008
    Replies:
    7
    Views:
    1,976
    SwapsRulez
    Jan 12, 2008
  8. Mr bloggy

    Will robots search even if I use web forwarding?

    Mr bloggy, Jan 9, 2008
    Replies:
    3
    Views:
    1,021
    SwapsRulez
    Jan 12, 2008
  9. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    17,136
    danieloffice
    Jan 9, 2008
  10. Browning

    I need a robots.txt for my proxy site. Don't know what goes in/how to do this

    Browning, Dec 25, 2007
    Replies:
    2
    Views:
    1,438
    lpstong
    Dec 26, 2007
  11. gree124

    Exert rebots.txt

    gree124, Dec 3, 2007
    Replies:
    2
    Views:
    1,473
    just-4-teens
    Dec 23, 2007
  12. eathosting.com

    What is a Robot???

    eathosting.com, Dec 1, 2007
    Replies:
    7
    Views:
    1,562
    gab1982
    Dec 20, 2007
  13. Mysmasken

    robots.txt to deny all spiders for proxy?

    Mysmasken, Nov 29, 2007
    Replies:
    3
    Views:
    21,151
    Mysmasken
    Nov 30, 2007
  14. MrThePlague

    Follow signup.php but NOT the proceeding query string

    MrThePlague, Nov 27, 2007
    Replies:
    1
    Views:
    1,552
    Kuldeep1952
    Nov 28, 2007
  15. acrimon

    Sort of a catch-22 here.

    acrimon, Nov 15, 2007
    Replies:
    4
    Views:
    1,352
    Ladadadada
    Nov 22, 2007
  16. Avalanche

    Robots.txt for Virtual Subdomains?

    Avalanche, Nov 14, 2007
    Replies:
    1
    Views:
    1,276
    energy.fs
    Nov 21, 2007
  17. Pammer

    How to block keywords via robots.txt ?

    Pammer, Nov 19, 2007
    Replies:
    2
    Views:
    3,297
    Pammer
    Nov 19, 2007
  18. Seobiznezz

    What should a robots.txt say?

    Seobiznezz, Nov 7, 2007
    Replies:
    4
    Views:
    1,410
    silence
    Nov 17, 2007
  19. schwab

    robots.txt

    schwab, Nov 12, 2007
    Replies:
    5
    Views:
    1,211
    schwab
    Nov 13, 2007
  20. snooks

    Robots .txt restricted - assistance needed

    snooks, Nov 9, 2007
    Replies:
    2
    Views:
    1,722
    snooks
    Nov 9, 2007
  21. ||NeX||

    Let index just the HP

    ||NeX||, Nov 3, 2007
    Replies:
    4
    Views:
    1,078
    ||NeX||
    Nov 9, 2007
  22. sirisaacnewton2007

    Can I tell spiders to only come at night?

    sirisaacnewton2007, Oct 17, 2007
    Replies:
    8
    Views:
    1,313
    harrysmith
    Nov 7, 2007
  23. Armaan143

    How to make robots.txt file for my site

    Armaan143, Oct 24, 2007
    Replies:
    8
    Views:
    5,230
    Armaan143
    Nov 6, 2007
  24. softarea51

    Limit bot access frequency

    softarea51, Nov 1, 2007
    Replies:
    7
    Views:
    5,330
    ajsa52
    Nov 5, 2007
  25. rayqsl

    Do I Need A Robots.txt?

    rayqsl, Oct 13, 2007
    Replies:
    11
    Views:
    1,698
    Kuldeep1952
    Nov 2, 2007
  26. login

    block pages in robots.txt

    login, Nov 2, 2007
    Replies:
    2
    Views:
    2,360
    Monty
    Nov 2, 2007
  27. Boardwalk

    Robots.txt Guide

    Boardwalk, Oct 16, 2007
    Replies:
    6
    Views:
    1,984
    hans
    Nov 1, 2007
  28. compindustries

    Help with robots.txt file

    compindustries, Jun 19, 2006
    Replies:
    13
    Views:
    3,000
    muks
    Nov 1, 2007
  29. frio

    Excluding subdmains

    frio, Oct 30, 2007
    Replies:
    3
    Views:
    1,024
    Kuldeep1952
    Oct 30, 2007
  30. clickalot

    http code 200 mystery

    clickalot, Oct 13, 2007
    Replies:
    4
    Views:
    1,860
    clickalot
    Oct 23, 2007
  31. surekhaweb

    robots.txt and delet the pages

    surekhaweb, Oct 2, 2007
    Replies:
    1
    Views:
    1,005
    Nucca_The_Warrior
    Oct 22, 2007
  32. bad_bob00

    Error pages on my site - robots.txt ?

    bad_bob00, Sep 21, 2007
    Replies:
    7
    Views:
    1,595
    Nucca_The_Warrior
    Oct 22, 2007
  33. EducationLinks

    How to block archive.org bot with both robots.txt AND .htaccess

    EducationLinks, Aug 9, 2007
    Replies:
    6
    Views:
    14,929
    hans
    Oct 21, 2007
  34. greatlogix

    Adding 301 redirect for more than 1000 pages

    greatlogix, Oct 3, 2007
    Replies:
    1
    Views:
    1,638
    Ladadadada
    Oct 14, 2007
  35. n0chance

    301 Redirect the robots.txt outside the root domain?

    n0chance, Oct 9, 2007
    Replies:
    3
    Views:
    3,569
    mcfox
    Oct 9, 2007
  36. frogman49th

    How to use robots.txt in blogs?

    frogman49th, Oct 6, 2007
    Replies:
    0
    Views:
    1,018
    frogman49th
    Oct 6, 2007
  37. Markas

    Does robots.txt work for all search engines?

    Markas, Sep 29, 2007
    Replies:
    5
    Views:
    3,000
    harrysmith
    Oct 2, 2007
  38. birdsq

    Only googlebot,slurp and msnbot should crawl my site.

    birdsq, Sep 10, 2007
    Replies:
    5
    Views:
    2,821
    harrysmith
    Sep 28, 2007
  39. blabla11

    Name of yahoo, google and msn bot

    blabla11, Aug 8, 2007
    Replies:
    9
    Views:
    19,976
    seoworld
    Sep 11, 2007
  40. scuba5794

    should i change this and my robots.txt and how?

    scuba5794, Sep 6, 2007
    Replies:
    4
    Views:
    1,046
    seoworld
    Sep 11, 2007
  41. Abhik

    How can I block a particular section of my forum?

    Abhik, Aug 29, 2007
    Replies:
    1
    Views:
    1,143
    evera
    Sep 10, 2007
  42. ThomaSeidler

    Help in encouraging Yahoo and MSN robots to obedience

    ThomaSeidler, Jun 14, 2007
    Replies:
    12
    Views:
    2,740
    evera
    Aug 28, 2007
  43. Diablos

    Standard Disallowed Pages with Robots.txt

    Diablos, Aug 23, 2007
    Replies:
    2
    Views:
    1,187
    Directory_SEO
    Aug 27, 2007
  44. sufi

    How to block images directory?

    sufi, May 26, 2007
    Replies:
    3
    Views:
    2,878
    trichnosis
    Aug 27, 2007
  45. cocaine

    I dont know the very basic thing about robots.txt

    cocaine, Feb 26, 2007
    Replies:
    10
    Views:
    2,251
    twistedspikes
    Aug 8, 2007
  46. st_jimi

    Can someone check my robots.txt file

    st_jimi, Jul 2, 2007
    Replies:
    3
    Views:
    1,670
    trichnosis
    Aug 7, 2007
  47. turboadds

    Traffic bot

    turboadds, May 7, 2007
    Replies:
    2
    Views:
    1,694
    trichnosis
    Aug 7, 2007
  48. danieloffice

    A default robots.txt for all my site.......... Please review

    danieloffice, Jul 13, 2007
    Replies:
    1
    Views:
    1,454
    trichnosis
    Aug 7, 2007
  49. ArtfulWebSites

    Bad Bot Advice

    ArtfulWebSites, Mar 23, 2007
    Replies:
    10
    Views:
    2,777
    trichnosis
    Aug 6, 2007
  50. Audiomad

    how many sitemaps do I put in the robots.txt file

    Audiomad, Jul 10, 2007
    Replies:
    2
    Views:
    1,638
    trichnosis
    Aug 6, 2007
Showing threads 651 to 700 of 911

Thread Display Options

Loading...
Page 14 of 19