what is crawling of spiders..how can e stop them if required fo a particular sentence

Discussion in 'Link Development' started by Jonathansmith, Dec 30, 2009.

0
  1. #1
    what is crawling of spiders..how can e stop them if required fo a particular sentence
     
    Jonathansmith, Dec 30, 2009 IP
  2. vstar

    vstar Well-Known Member

    Messages:
    906
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    150
    #2
    You should include <meta name="robots" content="noindex,nofollow">

    that will prevent them from crawling your page
     
    vstar, Dec 31, 2009 IP
  3. mjnetwork

    mjnetwork Peon

    Messages:
    70
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    This one will obviously works :)
     
    mjnetwork, Dec 31, 2009 IP
  4. intellg

    intellg Well-Known Member

    Messages:
    397
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    130
    #4
    You can use "nofollow" attribute for particular page or use robot.txt.
     
    intellg, Jan 1, 2010 IP