Netlify.app - blacklist issue

Some services use blacklists like this
https://phishing.army/download/phishing_army_blocklist_extended.txt
to filter content.
In my case, I use DNS resolver https://nextdns.io/ (similar to Cloudflare’s 1.1.1.1), and all *.netlify.app sites are blocked since you’re on that list.

Also reported on twitter:

hey there, thanks for bringing this up. I know it is alarming to see that. We’re actually aware of the issue, and working to fix it as soon as we can. Hopefully we’ll have an update for you at some point soon - and if you notice that you are able to post links again, please comment here and let us know! :slight_smile:

Can you provide an update?

hi there, we are still working to resolve this issue - we are making some headway, but unfortunately it is a complex issue not something we can control directly. Please do know we are aware that this a concern that is on our minds as much as yours - we hope to see some noticeable progress here soon. :muscle:

Hello,

I’m experiencing a similar issue. My domain is laurabrown.xyz and I’m seeing it blacklisted on a couple of lists here: Network Tools: DNS,IP,Email (sometimes I have to re-run the check more than once to get blacklisted results, see here Image 2021-02-24 at 7.53.43 PM). What can be done about this?

Thanks,
Laura

I’m not sure if my previous message went through. My domain’s IP address appears to be blacklisted according to Network Tools: DNS,IP,Email

Shared with CloudApp

What can be done to solve this?

Just a followup, I’m seeing additional issues here: MultiRBL.valli.org - Results of the query laurabrown.xyz

Since we don’t send any email from those IP’s, I’m not sure why they are listed in email blacklists, but shouldn’t impact you at all, since you don’t send mail from those IP’s :slight_smile:

The fact you are not personally sending emails is not the problem. I have also been having this problem and have emailed in but had no response.

The issue is Digital Ocean servers are getting picked up by spam lists when they have incorrect DNS settings, namely not PTR record being set for the server. AWS does not have this issue as they are configured correctly.

This means you can test your domain multiple times and sometimes it’ll be blacklisted and sometimes not.

This does cause a huge issue though when your sending emails which contain a link to your website as it shows them up as bad links giving you a spam score.

Who ever is in charge of your Digital Ocean servers needs to get it configured correctly.

Hey @markwilde,

I see DO IPs appearing on two blacklists:

Firstly –

If you are on the UCEPROTECTL2 / L3, you have an IP Address from your ISP that falls into a poor reputation range; i.e. the entire range of IP Addresses is blocked as a result of the provider hosting spammers.

Secondly, SpamRATS! - SpamRATS Lookup Tool!

You ONLY need to remove the IP Address from our list if you are running an outgoing mail server. If you are NOT running an email server, then this should not affect you from sending email, however you should ask your provider to provide reverse DNS as it can affect other services as well.

We’re not running a mail server so neither occurrence is likely to be the culprit of any mail send issues you’re seeing.

The reason they are being listed on these blacklists is due to the PTR record not being set and it classing it as misconfigured DNS.

When sending email campaigns, if they are link checked by the receiving server and find your email to contain links listed on a blacklist, it does have an effect.

Unfortunately only the owner of the servers can fix this (I assume this is yourselves).

AWS servers don’t have this problem.

I’m not really sure why this is being classed as “we don’t send email so its not our problem”. As you can see, AWS servers do seem to be configured correctly.

Hey @markwilde ,

With how things are configured at DO (no FQDN etc.) the addition of a PTR record isn’t actually possible. I think the best solution here is to make use of a URL shortener, masker or intermediary like most large subscription email services provide by default.

The only alternative, given constraints, is to reach out to each spam list vendor on a case-by-case basis and discuss how our infrastructure works. Then, we’d need to reach out each time DO rotates an IP or we introduce new/different nodes with this vendor.

Sorry I don’t have better news for you!

@Scott, thanks for your reply.

If you have control of your own servers on DO then adding PTR records within the DNS settings is fairly simple.

Unfortunately URL shortners only redirect to the domain name so when blacklists check them, they they go numerous levels deep. Email servers are fairly sophisticated at checking links, other wise all spammers and people phishing would just use bit.ly and spam away :rofl:

For me to reach out to each blacklist every time my IP changes (this is every refresh) is going to be impossible.

How come DO is rotating your IP’s? surely if you add a server to your stack, you get a fixed IP unless these are automatic and added and removed as needed with capacity? If this is the case there there is a problem in your deployment,

Can you not just restrict my deployment to use AWS only… would save me moving my deployment there which is a shame as Netlify is actually quite nice.

Hey @markwilde,

What I can do is file an internal feature request for you so that our site reliability team can be aware of this concern. No promises on a fix or ETA but at least the applicable team are aware. We’ll be sure to feed back in this topic if there is any progress!

Hello! Just pitching in on this matter, and please understand that my knowledge of this matter is probably not as large as yours.

My portfolio website (hosted here on Netlify) is currently being blocked by Facebook, therefore all my links to my website (from my profile, Page and even in private Messages) are considered malicious.

The only obvious reason would be because my website was on a blacklist due to the DigitalOcean IP having a bad reputation. I was on the UCEPROTECTL2/3. As of writing this message, I am not anymore. Apparently random if my research are to explain something here.

I am currently investigating with Facebook Business support and will request explanation from them as to what triggered their blocklist in the first place. I will write those reasons here if it helps and if I can actually know why I was blocked.

Cheers!

Hi, @NicolasGiuristante. UCEPROTECT and Spamhaus are two organizations I would love to see rehabilitated but at this time both flag innocent domains quite frequently.

These related forum topics might be helpful:

https://answers.netlify.com/search?q=UCEPROTECT
https://answers.netlify.com/search?q=Spamhaus

UCEPROTECT flags entire network blocks as all being “bad” if many of the surround IP address have been reported. The IP address which was reported is part of the cloud providers we use and someone using an nearby IP address has done something bad.

That is what a L2/L3 listing means. It isn’t the IP address for your site but the entire network block that is flagged.

At this time, there is no solution for this issue. UCEPROTECT blames harmless IP addresses with their L2 reporting and they offer no recourse to fix it. We have a feature request to try to workaround their mistakes on our side but at this time there has been no solution. We will post an update here though if something is found.

If there are other questions about this, please let us know.

2 Likes

Thank you @luke for this explanation. My instinct told me it was not a Netlify issue but wanted to make sure.

In the meantime, for anyone stumbling on this thread for similar issues: I had a chat with someone at Facebook Business support. I was not able to confirm whether the UCEPROTECT was the culprit of my website being blocked, but they did review it and acknowledged that their system had erroneously blocked my website, then lifted the block. I have no proof whatsoever that appearing on UCEPROTECT’s blocklist could trigger a block from potentially high heuristic URL scan of major tech companies.

Cheers!