Hello i have problem with my host i open topic before here http://forums.digitalpoint.com/showthread.php?t=1399085 my host send new message for me ==================== Hello, I've attached an image of your usage. As you can see usage is still to high. Checking your webalizer logs in cpanel's web/ftp stats, you appear to have a lot of bots crawling through your site, which may be the cause of the high usage. Top 30 of 39099 Total Sites # Hits Files KBytes Visits Hostname 1 51615 1.55% 51313 2.25% 365248 3.87% 114 0.69% crawl-66-249-71-20.googlebot.com 2 27600 0.83% 27150 1.19% 29856 0.32% 15 0.09% idargi.lunarmania.com 3 18202 0.55% 14477 0.64% 52714 0.56% 101 0.61% dynamic.dsl.nesma.net.sa 4 17233 0.52% 17125 0.75% 123552 1.31% 118 0.71% crawl-66-249-71-17.googlebot.com You have a robots.txt but you aren't blocking anything with it. You may consider blocking or limiting how often they can access your forum. More information about robots.txt can be found at: http://www.robotstxt.org/ If you have any further questions or concerns, feel free to contact us. ======================= what i can do for robots.txt my file now here http://www.extra-pc.com/robots.txt my forum www.extra-pc.com/forum my subdomain www.games.extra-pc.com www.up.extra-pc.com what's the best setting for robots.txt for my site? and for phpbb3 forum?
Your host is suggesting you add robot crawl restrictions to the forum section of your website. Visit the robottxt.org site they sent you and it will tell you how to do that. Slowing the Google crawl rate with the webmaster tools may help, but it looks like at least two other sites are crawling your forums constantly. Your robot.txt at the moment looks empty, so any spider from search engines will crawl as often as they like.
i make new one now User-agent: * Disallow: /cgi-bin/ Disallow: forum/admin/ Disallow: forum/cache/ Disallow: forum/images/ Disallow: forum/store/ Disallow: forum/mods/ Disallow: forum/richedit/ Disallow: forum/files/ Disallow: forum/download/ Disallow: forum/docs/ Disallow: forum/includes/ Disallow: forum/language/ Disallow: forum/styles/ Disallow: forum/faq.php Disallow: forum/login.php Disallow: forum/memberlist.php Disallow: forum/mcp.php Disallow: forum/rss.php Disallow: forum/posting.php Disallow: forum/ucp.php Disallow: forum/search.php Disallow: forum/viewonline.php Disallow: /*.swf$ User-agent: Googlebot # disallow files ending with these extensions Disallow: /*.inc$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ # allow google image bot to search all images User-agent: Googlebot-Image Disallow: Allow: /* # allow adsense bot on entire site User-agent: Mediapartners-Google* Disallow: Allow: /* good? what did u mean?