I have a video portal when i allow robot then its index very fast so my server crash so my hosting says remove the site but also suggest me to disallow from search engine so i have put these code but by these way my site is not index in search engine i have checked in webmaster there its show more then 300000 links are block(Googlebot is blocked from http://www.bwithyou.co.in/) I have also set google webmaster crawling (custom) minimum suggest me some one that what robot i use that my site get index in search engine and not consume more cpu usage. waiting for some positive reply Thanks
You have to allow some in but it may just be that you have selected cheap hosting on an overloaded server or a poor script. Hostgator, for instance, get ratty if you run openx on their servers - good hosting but they don't like resource heavy scripts. You may have to consider changing hosting or changing scripts. Unless you are being targetted by a rogue spider for some reasons (your logs will tell you which one) then you shouldn't be getting any more spider visits than anyone else..
Thank you for the nice tip but regarding hosting i am already using Hostgator services. Website is hosted on Hostgator server.
You use that hosting service that not use more cpu memory and if you do the disallow index definetly any search engine do not crawl your site . So you don't find your site in Search engine if you want to crawl by serach engine you must be allowed that.