Recently i was looking at one of my sites and noticed this MSNBot last vist 12 Jan 2006 - 21:50 I am wondering how can i get msn bot to come back almost daily?? I have many backlinks and even some from PR6 high traffic sites the bot should be there dailiy to update my sites serps??
what is a good example of a robot.txt and what it should basicly state to let it crawl everything besides cgi ? EDIT: checked my site and it has both robot.txt & robots.txt dunno if the s on it matters but on msn they referr to it as robots.txt
1. Allowing all the robots: User-agent: * Disallow: 2. Allow only MSNbot: User-agent: msnbot Disallow:User-agent: * Disallow: /
User-agent: * Disallow: /cgi-bin/ Disallow: /images/ that is how my robots.txt is looking now so it looks to be fine, any other ways to get msn to visit more often.
Get more links, thats all you get really do. Trying to tweak your robot file and metatags other then making sure the bots arent blocked is pointless.
my site has 2,509 and some are from site that would get crawled dailiy just weird usually i see msnbot every day or every 2nd day on my other sites weird.
I add content every week at minimum. As frequently you add, they index more. It is even woth to program some dynamic contant that changes every day or week.
i added a blog to my site and if i update that should that cause the bots to crawl my domain.com and update there listings? blog is linked to from main page and from blog to main page so the bots should find it all.