There are plugin’s out there that can handle this, but I’m keen on the less is more approach, so I add my bad bots list directly to my robots.txt file.
Caching plugin’s can cause issues with bad bot plugin’s too, or at the very least require additional steps to get them to play nicely together. Another reason to consider handling this manually.
This is my go to resource on Github, the Apache Ultimate Bad Bot Blocker.
Mitchell updates this regularly.
Bad bots are essentially any bot that hits your website with no benefit to you. Blocking them can reduce stress on your server and potentially save you bandwidth, hosting costs, and speed up your site.
I do also have a few rules at Cloudflare that help in this area – but I’ll save that for another post.
A number of hosts are starting to offer this as an add-on service, no doubt due to WordPress’s popularity.