I have a website that blocks access from almost all countries (including US). This is causing conflict with the googlebot the site is not being indexed. I use an htaccess like this: <Limit GET HEAD POST> Order deny, allow Allow from 8.6.48.0/21 Allow from 64.68.80.0/21 Allow from 64.68.80.0/22 Allow from 64.68.80.0/24 Deny from all </ Limit> How can I do to allow google bots access?
Get googlebot's IP addresses and allow them. Hmmm. If you are denying access to users, why would you need google to index them at all?
Is it reliable that google does not change IP's? I do not need users from other countries but I still need to be well indexed in google for users from my country.
@mmerlinn you wonder why OP is denying access to certain users. It should be mentioned that some websites are not relevant to people in other countries. If you have a website for a carwash in a small town in the UK, do you want for example German people to index or waste bandwidth on your server? The answer is very obvious. As you don't advertise to attract German car owners, you don't have to allow IP addresses from there to access your website. However if you own for example a hotel, you will not benefit from blocking foreign IP addresses. Foreigners travelling to the area will do their research and booking in advance, and by blocking them you might lose customers. The conclusion is that the question of blocking IP addresses is very much dependent on what business you run. @F.Royale I don't think that you should bother about Googlebot changing IP. In the case this becomes an issue, Google will use Search Console to notify you that Googleboot is being blocked.
Block from the actual website by looking at remote ip AND the user agent In @MrKing01's example the carwash isn't going to get so many hits that bandwidth is a problem - If the site is set up right and your business is set up in Google Places then Google will recognise that the site won't appeal to people looking for a carwash in Berlin. They're quite clever. I look after a huge site that gets appropriate traffic for it's niche and I've never heard a murmur from the hosting company about bandwidth. Once upon a time I used to get blocked from American real estate sites where the owner had blocked all APNIC countries. Honestly, I don't know why they bothered. The only sites that really have to worry about blocking countries are those peddling "adult content" that may get them in hot water in some countries. If the problem is spam use better captcha.
@sarahk As you look after a huge site, I think it's appropriate to ask you following questions. Have you ever been in a situation where you have found it justified to block IP addresses from certain foreign countries? Also let's say that you find out that you get loads of spam comments from country "X" and hacking attempts from country "Y". What would you do to tackle these issues?
I'd start with a service like cloudflare I'd use hidden captcha I'd track their hacking attempts to see if they've identified any vulnerabilities but if you're getting targetted malicious hacking attempts they'll bounce around IP addresses and you won't be able to block them all
Allow IPs from whole USA, google use only USA IPs, Google all the time change their IP and use new ones.