Stoping SE Spiders Scanning/Indexing my Page? - what is the HTML code?

Discussion in 'HTML & Website Design' started by entry, Jul 8, 2011.

  1. #1
    What is the HTML Code that I should insert in my html page, to stop Search engine spiders from scanning that page, so that it IS NOT in any of the search engines?

    do i need to insert code to the individual page?

    or change some settings to the folder, on the control panel?
     
    entry, Jul 8, 2011 IP
  2. unigogo

    unigogo Peon

    Messages:
    286
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #2
    <meta name="robots" content="noindex,nofollow" />


    Insert the code in the head of individual page
     
    unigogo, Jul 8, 2011 IP
  3. MarketerScientist

    MarketerScientist Peon

    Messages:
    22
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    OP you can use a file called robots.txt. With it you can specify in more detail which pages are included and excluded from spidering. Please have a look at this guide for more info and examples:
    http://www.robotstxt.org/orig.html
    Code (markup):
    N.B. Not all spiders take into account the robots.txt file AND/OR unigogo's solution
     
    MarketerScientist, Jul 9, 2011 IP
  4. brinkmedia

    brinkmedia Peon

    Messages:
    52
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    yeah, robots are the best solution, for most of SEs... you can also use meta, nofollow and noindex tags, but with tags - SEs have different policies. For example russian yandex.ru accepts using noindex tag for any piece of content (noindex a paragraph of text), but google doesn't understand that and will index.
     
    brinkmedia, Jul 9, 2011 IP