1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Which Robots text?

Discussion in 'robots.txt' started by steve_gts, Feb 28, 2006.

0
  1. #1
    Hi,

    I have a site which I want the bots to crawl. I have this text in there at the moment and I am being indexed although am ranking very low:

    <meta name="ROBOTS" content="index, follow">
    <meta name="ROBOTS" content="ALL">

    I ran a test on the site yesterday and it said that there is no robots text in there. Looking at another site it suggests just using this:

    User-agent: *
    Disallow: /

    If I change it to this do I just add these terms in the "" "" or take out the first one completely and replace with this one (i.e. not meta name etc)

    What's best?

    Thanks
     
    steve_gts, Feb 28, 2006 IP
  2. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Hi,

    These lines don't hurt, but they don't help either. Their contents indicate that the web robots are welcome. If these lines are not present, robots consider they are welcome anyway. Other forms of these lines could be used to disallow access to robots.

    This is the contents of a robots.txt file for a site that disallows access to robots. Probably not what you are looking for!

    To improve your ranking in search engines, you need external quality links pointing to your web site. Search engines will take this as an evidence that the "popularity" of your site is high. If there are not enough external quality links, your site will not be considered as "popular" and its ranking will be poor.

    Jean-Luc
     
    Jean-Luc, Feb 28, 2006 IP
  3. vlead

    vlead Peon

    Messages:
    215
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Basically a robots.txt file is useful when you do not want search engine robots to access certain sections of your website.

    User-agent: *
    Disallow: /
    The above means that no search engines should access the site.
     
    vlead, Feb 28, 2006 IP
  4. Colleen

    Colleen Illustrious Member

    Messages:
    6,777
    Likes Received:
    725
    Best Answers:
    1
    Trophy Points:
    430
    #4
    Yeah, if you read up on meta tags, the only ones that matter anymore are description and keywords, and of course the "content-type" meta tag.

    Block robots as the others suggested.
     
    Colleen, Feb 28, 2006 IP
  5. hans

    hans Well-Known Member

    Messages:
    2,923
    Likes Received:
    126
    Best Answers:
    1
    Trophy Points:
    173
    #5
    in the robots.txt
    normally you are happy for SE to come
    however one of the most important folder to exclude normally would be the
    /cgi-bin
    as well as any other admin folder to assure hacker have no way to use G or other SE to find vulnerable login or hacker files

    to index all and to follow all links is default nowadays by all major bots
    therefor i statrted to remove the meta tag robots line used once in earlier years such as the 2 meta name="ROBOTS" lines mentioned earlier

    there also is a robots.txt validator available to prevent problems with robots.txt and more info about robots.txt out there in the web
     
    hans, Feb 28, 2006 IP
  6. steve_gts

    steve_gts Active Member

    Messages:
    1,170
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    80
    #6
    Thanks folks,

    So basically leave it as it is, seems to be the general oppinion?

    I realise I need the links, however the site I am potimising has numerous relevant links and probably around 3oo directory submissions, I was expecting it to go from a 3 to a 5 at the update but instead it's remained the same. I thought I better start looking at some onpage optimisation factors instead. Google is seeing 250 links to my site but some of my competitors only have around 20 (and not from authority sites) and are much higher than me in the searches.

    Colleen - Perhaps it's my content type thats wrong ? It was there in the template I stared off with and I've not changed it. Does this look right to you?

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
    "http://www.w3.org/TR/html4/loose.dtd"><html><head><meta http-equiv="Content-Type" content="text/html; charset=windows-1252"><meta http-equiv="Content-Language" content="en-gb">

    Thanks again
     
    steve_gts, Feb 28, 2006 IP
  7. steve_gts

    steve_gts Active Member

    Messages:
    1,170
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    80
    #7
    Hi Hans,

    You were typing that at the same time as me. The problem I have with these validators is they just seem to say that everything is wrong and not knowing html it's an impossible task to change it. I used FP03 to create the site so you would assume that MS would create the right code!!! Although I'm starting to think they dont.

    Cheers
     
    steve_gts, Feb 28, 2006 IP
  8. hans

    hans Well-Known Member

    Messages:
    2,923
    Likes Received:
    126
    Best Answers:
    1
    Trophy Points:
    173
    #8
    the robots.txt validator is a different validator on its own
    much easier than html validator

    i needed that robots.txt validator weeks ago urgently
    i had a robots.txt exactly as publihsed on G pages that time
    and that was wrong and the sitemap was NOT found thanks to that wrong robots.txt

    as to html

    well a fish who wants to swim in ocean has to learn to swim :)
    a fish who wants to9 learn to swim in www has to learn html and MUCH more
    far beyond HTML

    as soon as you are told by your host that yoiur site has been hacked and you are supposed to act and repair you will know what you should have learned before starting a web site ...

    as to content type etc
    i may have missed to see your URL/domain you are talking about
    if yoiu ahve any HTML problem in a template I may help' you instantly when back from dinner - but please by direct email if you wish so - its faster, but you would need instant direct ftp access and a firefox 1.5 browser with a web developer toolbar installed for fast success in any problemsolving
     
    hans, Feb 28, 2006 IP