ISP Routing Issue to Netlify Domains?

I still haven’t heard from support (more than 48 hours later).

I’m going to circle back with our ISP and see what the deal is.

Just wanted to keep the communication updated here.

I’ve seen this situation before (not with Netlify sites), but it was long enough ago that I don’t remember the resolution.

Two things you can check:

First, try visiting the afflicted sites in a new private window in your browser.

Second, try clearing your locate DNS cache. There’s probably a snazzy terminal command, but I use CleanMyMac X so I don’t know what that command might be.

My two cents.

Thanks for those suggestions, Greg - also two of our support team’s favorites, but in this case, it seems pretty clear that this ISP has decided to block some of our CDN nodes and we will almost certainly need help from them. So glad that @Riley is able to talk to them as a customer, since most ISP’s don’t have a good escalation path for non-customers…

1 Like

Anybody have any recommendations on how exactly to talk about this issue to TDS support? I’ve called twice and both times the person I talked to was convinced it wasn’t their issue and I couldn’t even “file a ticket” or anything with them. What was your technique @Riley?

I would confirm that others are sharing your issue and that it is not an isolated incident. If necessary you can bring up this thread as proof that the issue is widespread.

Super annoying, I’m now required to use a VPN to reference websites like vuejs.org (and anything hosted with Netlify). However, what is annoying for me is complete failure for other TDS customers who don’t leverage a VPN. Without VPN and testing I’d assume a portion of these websites were consistently down, which is a major issue.

My initial call was handled the way you’ve experienced, with support trying to encourage me to check my own network, etc; however, I ran a lot of tests before following up with another call. Your thread was a great indication that others were having this issue (even if it was just you and I at the time :upside_down_face:). Having @fool’s IP list here, I was able to ping and traceroute each to identify what wasn’t working. I’d recommend doing the same, but I can say that I think the two blocked IPs are 104.248.78.23 & 104.248.78.24.

I think the key in dealing with support is confidence. You need to be clear that you have tried whatever basic protocol they are running through, let them know this is an issue that other customers are experiencing, and be sufficiently technical in your hypothesis. Let them know you’ve run tests and isolated two IPs (if your tests are mirroring my results) and tell them you’d like confirmation that they have no problem accessing those IPs themselves. This is what moved me through that boilerplate feedback loop that you sometimes get with tech support. Ask them to try to access the site, then ask them if they have access to an outside ISP or service (even their phone) and have them test a website again (netlify.com was what I recommended them looking at). The proof should be apparent at that point.

Persistence is the key here, I will continue to follow up in a respectful by consistent manner to see where this goes. It’s such an odd case that I feel like the average user experiencing this issue assumes that the website in question is at fault and not their ISP, which convolutes the responsibility here.

The larger problem is that TDS is a nationwide provider for suburban and rural internet, so this issue is much more widespread than we would hope. TDS Provides Internet, TV and Phone to Communities Across America

Quick question @fool / @luke , and probably unlikely, but could Netlify potentially temporarily suspend using these IPs until the issue is resolved? We’re two weeks into this and I really don’t want to go through the process of migrating clients to another service. However, having to run QA on the state of this daily is becoming less and less effective as the hope of a resolution drags out from TDS.

Otherwise, does anyone have a recommendation on avoiding these IPs for a particular site?

1 Like

Quick update, I called again this morning (~10 minutes ago) and asked them about my ticket and they said that I could use Google DNS in the meantime and that they are still looking into this issue—6 days later.

This recommendation does not solve the problem, and I’m unimpressed that the offered solution is to mask the symptoms versus a resolution. I had explained that other traffic is not going to try anything like this, they’ll just assume these websites are down.

UPDATE (11/4/19 at ~8pm PST) : @jimniels @bheadwhite The issue with the IPs in question seems to be resolved. I got word from @luke that TDS had reached out this evening and that they have been unblocked. Many thanks to everyone involved here!

I also have new information on the support ticket I opened with their Internet support team. This is my understanding based on my last call:

  • they (TDS) were routing traffic to those IP two address to the “blackhole route” which makes the IP addresses unreachable from their network
  • this was done because of a report of malicious content found at those IP address (we are a hosting company and this does happen)
  • they state they have removed the blackhole routing and these IP addresses should be reachable with TDS internet service now

Personally, I haven’t ever seen an ISP blackhole route IP addresses based on a report of content hosted at that IP address before. This is not an standard industry practice as far as I know and I’ve never heard of an ISP blocking one of our IP addresses before. Countries like China and Russia have been known to do this but not any ISPs and especially not ISPs in the USA or Canada.

Normally, if someone hosts phishing or other malicious content at Netlify an abuse complaint would be filed with the “owner” of the IP address, which will be one of the various cloud provider we use to build our CDN and other services. The cloud provider then forwards the report to Netlify. We then disable the site in question and remove the user that created it from our service. An ISP blocking the IP address, though, that was a new one for me.

Blackhole routing the IP address and not having a timely correction process in place for errors like this one seems to be a bad experience for both our customers and theirs. I’m not sure why they would handle a malicious content report in this way.

Again, I’ve only seen this done by nation states and they do so for other reasons - for example - to block tools that enable free speech and not for phishing content.

I’m trying to get someone at TDS to confirm that they won’t do this again and instead report the content so we can correct the issue. However, I’ve not been able to speak to anyone at TDS so far that can help me with that.

If I make a connection at TDS and someone does help me get this practice changes, I’ll update this community topic to share the news.

Wow. What a fascinating end to this story. When I first posted this issue, it seemed so odd I wasn’t even sure how to describe it. I thought for sure I’d be stuck in a weird place where it would never get fixed because TDS wouldn’t believe what I was telling them—which is why I posted here, in case somebody else out there in the world had the same issue. Turns out, it wasn’t just me.

Similar to @Riley, all along the way TDS support was providing no support at all, suggesting things like switching my DNS (which didn’t help anyway, I’d already tried). Big thanks to @Luke and anyone else at Netlify who was involved in the resolution, as I was unable to make any headway the entire time with TDS support.

1 Like

fantastic work luke!!

This is still happening with Frontier. I’m in Southern California on FIOS, and sporadically my Netlify sites won’t load. Very frustrating. I keep saying I’m giving up on this issue because it’s so sporadic that getting details is like nailing Jello to a wall, but then after awhile I think of something else to try. Something else I don’t understand about this is why, when I have 1.1.1.1 set as my DNS, it takes seven hops to reach the Cloudflare DNS. The hang-up always seems to be on some frontier.net server.

Inspired by the OP, I contacted Frontier tech support. He checked my cable modem settings and made a change. I then restarted the modem and things seem better for now. It was, however, a Frontier DNS issue apparently caused by an old configuration setting in their cable modem.

Greg, I’m so impressed by your diligence. This is super helpful for anyone who might find themselves in the same spot. Thanks a bunch for sharing & going down this rabbit hole.

is anyone getting this issue again. Im getting this is a strange way again. ISP is TDS.

last year it was netlify.com not loading at all. This year preview sites that have the ending netlify.app works but certain domains that is pointing to netlify servers from godaddy is not loading. turn off the wifi on the phone it also works. done all that restart router etc still didn’t fix the issue.

Same scenario. without vpn it doesn’t load, just a white screen. but when VPN is active it loads.

@stefantrinh1 That would mean this is still a local issue, not an issue with Netlify.