Hey, we have been dealing with a lot of bot traffic on our site, although our site is mostly static and not affected, it skews our traffic data and as we use cloudinary for image hosting it is starting to exceed our bandwidth. Because of how netlify authenticates our domain, we are unsure how we could pass this through some kind of WAF or filter. Even if we could filter a percentage of bot traffic would be useful
Do you have any solutions to this problem that could become quite serious. A way to filter through netlify, or to pass through cloudfront or something.
(We have implemented a robots.txt file already. This is asking about malicious or bots not respecting robots.txt)
Sorry to say we don’t have any native facilities for handling that at present. It’s an interesting conundrum: how would you have us identify bad vs good traffic, since they’re already not behaving well and presumably do bad tricks like user agent spoofing to not look like robots?
I, too, hope there can be some progress on this front, not only for general purposes but also to make Netlify Analytics data more useful. I have recently seen a bizarre amount of home page traffic from Germany that doesn’t show up in a competing firm’s analytics product that does screen out bot traffic. No idea why my site is being targeted, but the bots are greatly skewing my Netlify Analytics data and the reported bandwidth far in excess of real-world, real-user traffic.
Thanks for the feedback! We are still contemplating a bot filter for analytics, and I have added your voice to that open feature request so we’ll remember to follow up here in case we release it.