I have a site of mine that got indexed on MSN, just the homepage, but the crawler did not go through the entire website that is featuring several pages... how can I help this? I also have some pr6 - pr7 pointing on it. Thank you very much for your contributions
True, but MSN *does* have its own popularity score which is similar. It is a good assumption that a site that has a high PR with google will likely be seen as similar in popularity in other engines.
* MSN loves robot.txt. To be able to gain a better ranking or fast indexing robt.txt is very very important. * As usual get links from websites which are related to the content of your site. The more links the better!
Thank you maldives for your answer Actually I am not using a robots.txt in my sites, so I am going to get some info about it!
so you say get links from sites which are related to the content of your website. If I am advertising a college tshirt website, does this mean I can put my website up in a college related forum. I cant figure out if a page like "myspace.com" this thing where kids put profiles of themselves, is considered indexed and is doing me any good to put links on it to my site. Or is the only way to get linked by indexed websites with relative content is to contact them and beg? appreciate this amazing forum
Funnily, I got indexed by MSN way quicker than Google. It usually pays to have a sitemap link on your index page and let it crawl through that!
Interesting, I had never thought of using robots.txt as part of seo before. Exactly what does MSN love about it?
Another thing MSN seems to like is fresh content. Throw an RSS feed on the page (CARP is good) or a daily thought or some thing. Seems to get the spiders legs twitching ! Regards Brian www.bestrealincome.com
robot.txt tells the robot (spider) which files it may spider (download). Each time MSNBot crawls your website, it looks in your web server's root directory for a robots.txt file. If the file exists, MSNBot checks to see if MSNBot is an allowed user agent, and if any crawling or indexing restrictions have been set. You just need to create robots.txt file in the top level directory of your website and specify which web crawlers to allow or block. MSNBot complies with the standards for robots.txt, not all web crawlers comply
When starting a site I usually allow every user agent to crawl my site. After some time viewing the stats and the traffic generated I decide whether to exclude some crawler from my site i.e. it's sucking tons of bandwidth from my server and it's not leading traffic to me