robots.txt

Find help with setting up your robots.txt file on your web site.

Page 14 of 19
Sort by:
Title Start Date
Replies Views
Last Message ↓
  1. nino2007

    What is the problem with Google and my site?

    nino2007, Jan 22, 2008
    Replies:
    3
    Views:
    1,644
    catanich
    Feb 2, 2008
  2. kong707

    is robot.txt very important for a forum?

    kong707, Dec 18, 2007
    Replies:
    8
    Views:
    2,083
    Chris-777
    Feb 2, 2008
  3. dalailama

    avoid robots to index mainsite.com/undersite.com

    dalailama, Jan 11, 2008
    Replies:
    3
    Views:
    1,801
    rich_uk
    Jan 23, 2008
  4. login

    Avoid indexing of ../lyrics/print,74306.html

    login, Jan 11, 2008
    Replies:
    14
    Views:
    1,568
    SwapsRulez
    Jan 14, 2008
  5. crystak

    Difference between /category/ and /category disallowing ?

    crystak, Dec 29, 2007
    Replies:
    3
    Views:
    1,189
    SwapsRulez
    Jan 12, 2008
  6. ecctao

    Is it necessary to write a list of my all my pages in a tobots.txt

    ecctao, Jan 1, 2008
    Replies:
    4
    Views:
    1,213
    SwapsRulez
    Jan 12, 2008
  7. lowprofilejoe

    Disallow Flash Files

    lowprofilejoe, Jan 4, 2008
    Replies:
    7
    Views:
    1,990
    SwapsRulez
    Jan 12, 2008
  8. Mr bloggy

    Will robots search even if I use web forwarding?

    Mr bloggy, Jan 9, 2008
    Replies:
    3
    Views:
    1,036
    SwapsRulez
    Jan 12, 2008
  9. johan-cr

    Is Dreamhost support stupid or am i missing something?

    johan-cr, Dec 27, 2006 ... 2
    Replies:
    33
    Views:
    17,264
    danieloffice
    Jan 9, 2008
  10. Browning

    I need a robots.txt for my proxy site. Don't know what goes in/how to do this

    Browning, Dec 25, 2007
    Replies:
    2
    Views:
    1,447
    lpstong
    Dec 26, 2007
  11. gree124

    Exert rebots.txt

    gree124, Dec 3, 2007
    Replies:
    2
    Views:
    1,481
    just-4-teens
    Dec 23, 2007
  12. eathosting.com

    What is a Robot???

    eathosting.com, Dec 1, 2007
    Replies:
    7
    Views:
    1,611
    gab1982
    Dec 20, 2007
  13. Mysmasken

    robots.txt to deny all spiders for proxy?

    Mysmasken, Nov 29, 2007
    Replies:
    3
    Views:
    21,199
    Mysmasken
    Nov 30, 2007
  14. MrThePlague

    Follow signup.php but NOT the proceeding query string

    MrThePlague, Nov 27, 2007
    Replies:
    1
    Views:
    1,568
    Kuldeep1952
    Nov 28, 2007
  15. acrimon

    Sort of a catch-22 here.

    acrimon, Nov 15, 2007
    Replies:
    4
    Views:
    1,364
    Ladadadada
    Nov 22, 2007
  16. Avalanche

    Robots.txt for Virtual Subdomains?

    Avalanche, Nov 14, 2007
    Replies:
    1
    Views:
    1,289
    energy.fs
    Nov 21, 2007
  17. Pammer

    How to block keywords via robots.txt ?

    Pammer, Nov 19, 2007
    Replies:
    2
    Views:
    3,312
    Pammer
    Nov 19, 2007
  18. Seobiznezz

    What should a robots.txt say?

    Seobiznezz, Nov 7, 2007
    Replies:
    4
    Views:
    1,430
    silence
    Nov 17, 2007
  19. schwab

    robots.txt

    schwab, Nov 12, 2007
    Replies:
    5
    Views:
    1,228
    schwab
    Nov 13, 2007
  20. snooks

    Robots .txt restricted - assistance needed

    snooks, Nov 9, 2007
    Replies:
    2
    Views:
    1,733
    snooks
    Nov 9, 2007
  21. ||NeX||

    Let index just the HP

    ||NeX||, Nov 3, 2007
    Replies:
    4
    Views:
    1,094
    ||NeX||
    Nov 9, 2007
  22. sirisaacnewton2007

    Can I tell spiders to only come at night?

    sirisaacnewton2007, Oct 17, 2007
    Replies:
    8
    Views:
    1,341
    harrysmith
    Nov 7, 2007
  23. Armaan143

    How to make robots.txt file for my site

    Armaan143, Oct 24, 2007
    Replies:
    8
    Views:
    5,268
    Armaan143
    Nov 6, 2007
  24. softarea51

    Limit bot access frequency

    softarea51, Nov 1, 2007
    Replies:
    7
    Views:
    5,344
    ajsa52
    Nov 5, 2007
  25. rayqsl

    Do I Need A Robots.txt?

    rayqsl, Oct 13, 2007
    Replies:
    11
    Views:
    1,743
    Kuldeep1952
    Nov 2, 2007
  26. login

    block pages in robots.txt

    login, Nov 2, 2007
    Replies:
    2
    Views:
    2,390
    Monty
    Nov 2, 2007
  27. Boardwalk

    Robots.txt Guide

    Boardwalk, Oct 16, 2007
    Replies:
    6
    Views:
    2,004
    hans
    Nov 1, 2007
  28. compindustries

    Help with robots.txt file

    compindustries, Jun 19, 2006
    Replies:
    13
    Views:
    3,045
    muks
    Nov 1, 2007
  29. frio

    Excluding subdmains

    frio, Oct 30, 2007
    Replies:
    3
    Views:
    1,047
    Kuldeep1952
    Oct 30, 2007
  30. clickalot

    http code 200 mystery

    clickalot, Oct 13, 2007
    Replies:
    4
    Views:
    1,887
    clickalot
    Oct 23, 2007
  31. surekhaweb

    robots.txt and delet the pages

    surekhaweb, Oct 2, 2007
    Replies:
    1
    Views:
    1,017
    Nucca_The_Warrior
    Oct 22, 2007
  32. bad_bob00

    Error pages on my site - robots.txt ?

    bad_bob00, Sep 21, 2007
    Replies:
    7
    Views:
    1,635
    Nucca_The_Warrior
    Oct 22, 2007
  33. EducationLinks

    How to block archive.org bot with both robots.txt AND .htaccess

    EducationLinks, Aug 9, 2007
    Replies:
    6
    Views:
    14,965
    hans
    Oct 21, 2007
  34. greatlogix

    Adding 301 redirect for more than 1000 pages

    greatlogix, Oct 3, 2007
    Replies:
    1
    Views:
    1,657
    Ladadadada
    Oct 14, 2007
  35. n0chance

    301 Redirect the robots.txt outside the root domain?

    n0chance, Oct 9, 2007
    Replies:
    3
    Views:
    3,584
    mcfox
    Oct 9, 2007
  36. frogman49th

    How to use robots.txt in blogs?

    frogman49th, Oct 6, 2007
    Replies:
    0
    Views:
    1,034
    frogman49th
    Oct 6, 2007
  37. Markas

    Does robots.txt work for all search engines?

    Markas, Sep 29, 2007
    Replies:
    5
    Views:
    3,029
    harrysmith
    Oct 2, 2007
  38. birdsq

    Only googlebot,slurp and msnbot should crawl my site.

    birdsq, Sep 10, 2007
    Replies:
    5
    Views:
    2,849
    harrysmith
    Sep 28, 2007
  39. blabla11

    Name of yahoo, google and msn bot

    blabla11, Aug 8, 2007
    Replies:
    9
    Views:
    20,070
    seoworld
    Sep 11, 2007
  40. scuba5794

    should i change this and my robots.txt and how?

    scuba5794, Sep 6, 2007
    Replies:
    4
    Views:
    1,061
    seoworld
    Sep 11, 2007
  41. Abhik

    How can I block a particular section of my forum?

    Abhik, Aug 29, 2007
    Replies:
    1
    Views:
    1,164
    evera
    Sep 10, 2007
  42. ThomaSeidler

    Help in encouraging Yahoo and MSN robots to obedience

    ThomaSeidler, Jun 14, 2007
    Replies:
    12
    Views:
    2,794
    evera
    Aug 28, 2007
  43. Diablos

    Standard Disallowed Pages with Robots.txt

    Diablos, Aug 23, 2007
    Replies:
    2
    Views:
    1,198
    Directory_SEO
    Aug 27, 2007
  44. sufi

    How to block images directory?

    sufi, May 26, 2007
    Replies:
    3
    Views:
    2,902
    trichnosis
    Aug 27, 2007
  45. cocaine

    I dont know the very basic thing about robots.txt

    cocaine, Feb 26, 2007
    Replies:
    10
    Views:
    2,277
    twistedspikes
    Aug 8, 2007
  46. st_jimi

    Can someone check my robots.txt file

    st_jimi, Jul 2, 2007
    Replies:
    3
    Views:
    1,681
    trichnosis
    Aug 7, 2007
  47. turboadds

    Traffic bot

    turboadds, May 7, 2007
    Replies:
    2
    Views:
    1,707
    trichnosis
    Aug 7, 2007
  48. danieloffice

    A default robots.txt for all my site.......... Please review

    danieloffice, Jul 13, 2007
    Replies:
    1
    Views:
    1,465
    trichnosis
    Aug 7, 2007
  49. ArtfulWebSites

    Bad Bot Advice

    ArtfulWebSites, Mar 23, 2007
    Replies:
    10
    Views:
    2,810
    trichnosis
    Aug 6, 2007
  50. Audiomad

    how many sitemaps do I put in the robots.txt file

    Audiomad, Jul 10, 2007
    Replies:
    2
    Views:
    1,654
    trichnosis
    Aug 6, 2007
Showing threads 651 to 700 of 911

Thread Display Options

Loading...
Page 14 of 19