does anyone know where I can get a list of the IP addresses for all the good bots like googlebot, googleadwords bot and yahoo has like 50 diff bots etc. Also hoping to find a list of all the bad bots aswel, at the moment I am banning bots manually, by checking their IP address to see who they are and banning IP addresses. I would just like to create a table that my system can check to see if a user is a bot and if they are a bot are they good or bad. thanks
I would just start logging bots, adding the friendly ones to a list, and googling the ones you don't know about. Work your own DB form there....
Not a good idea to ban IP addresses on your server unless force majeure, a small mistake will make you might ban or prevent a friendly one to visit your site...
it would be hard to know all and they might change but here is some i think* IP range (66.249.64.0 - 66.249.95.255) are probally going to be from google .. anyway id suggest you also dont ban ips thats not a good way.. if you never read this page before id suggest you read this.. http://www.robotstxt.org/wc/faq.html its great info on robots how to ban them ect.. use robots.txt to alllow or disallow robots.. here its a big list http://www.robotstxt.org and details on how to properly make a robots txt.. is also on there.. Database of Web Robots, Overview http://www.robotstxt.org/wc/active/html/index.html p.s. the ips for each robots on there.. if you still wanted that..
If you dont want Google visiting your site go to google.com/webmaster/ and log in and they have stuff