One of the biggest problems a website can get is when it is attacked by bots. These bot attacks can slow down the website for other website users.
Bots can be designed for various purposes. Each bot can be assigned a task to be carried upon. These tasks can include scraping, indexing, tracking, etc.
Learn more about bots and crawlers in our SEO terminology post here: Common SEO Terms For Beginners In Digital Marketing
These bots can be designed to also harm websites. To find security loopholes on your website and try to gain important information from it.
This is the reason many WordPress security plugins and malware scanners have IP lists inside them to blacklist or whitelist a set of IP addresses. So that the right bots are allowed on the website. And bots that have negative intentions for your website are blocked.
But how would you know which Googlebot is the right bot or faking it?
When Google itself gives you the list of IP addresses its bot's use it will be easier for programmers and companies to secure themselves from other bots that try to hack WordPress websites.
Googlebot IP Addresses List From Google
Google has recently disclosed its Googlebot IP addresses list here: https://developers.google.com/search/apis/ipranges/googlebot.json
This is to be sure that your website is scanned by a real Googlebot and not a fake one.
The list provided by Google includes IP addresses of all kinds of robots of Google from different surfaces of Google like Google Shopping, Google Maps, Google Adwords, etc.
To differentiate a real Googlebot from a fake Googlebot, one of the ways is by doing a reverse DNS method on the IP address of the bot.
When you do reverse DNS on an IP address and it matches with any of the IP addresses from the list given by Google itself in JSON format, then you are safe. But if it does not match, you can go ahead and ban that bot IP address.
Most of the WordPress plugins already use this list coded inside their programs.
Bingbot IP Addresses List From Microsoft
Following the footsteps of Google, Microsoft also recently released a similar list of Bingbot's IP addresses. So that you just not unblock the Googlebots but also Bingbots.
The Bingbots IP addresses have also been made online in JSON format here: https://www.bing.com/toolbox/bingbot.json
All these Bingbot IP addresses are in IPV4 format. I guess Bingbots are still to be upgraded to get IPV6 addresses for themselves.
This list of Googlebot spiders and Bingbot robots IP addresses should always stay unblocked for your website. So that these crawlers can crawl your website and index them in their search engines.
Server administrators use a lot of Firewall rules to protect websites hosted on their servers.
One of the firewall rules is to blacklist or whitelist IP addresses in the server itself. This way even if a script or plugin in the WordPress website misses tracking the IP address of the search engine bots, the server OS will not allow it to get any files hosted on it following the firewall rules created by the server administrators.
When you have a list like this given by the official sources of Google and Microsoft itself. The task to protect your websites from malicious spiders, bots, and crawlers becomes easy.
A good website is easily accessible to everyone. That includes search engine bots like Googlebot and Bingbot too.
When your website is easily accessible, it will be indexed on the search engines and then ranked based on the SEO of the website.
But if you block the wrong IP addresses to your website, it might lead to de-indexing of the website and Google ranking drop too. So to prevent you not to doing that the Google and Microsoft have released their robots IP addresses that you can safely approve to crawl your website.
Keep your website safe and secure by using the right WordPress plugins and whitelisting the right Googlebot and Bingbot IP addresses.
Thanks for the Google bots IP addresses. I had always had issues with the right IP addresses. With the official Google information, things get much easier.
Easiest resource to find Googlebot and Bingebot IP addresses. Hope they keep this list updated and in the same place for a while. Else I need to change its address in my script every time they change its URL.