Query string cached when redirecting

Since this morning we noticed that query params in redirects are suddenly beeing cached. Without any code changes nor environment changes or Netlify settings. This looks like a major issue since we pass personal tokens in the query params.

When I visit the following URL:

1. /path1/path2?tracecode=1234&zipcode=1234 [301]
response headers: location: /path3/?tracecode=1234&zipcode=1234

2. /path3/?tracecode=1234&zipcode=1234 [301]
response headers: location: domain.com/path3/?tracecode=5678&zipcode=5678

3. domain.com/path3/?tracecode=5678&zipcode=567 [304]

?tracecode=5678&zipcode=5678 are query params we used before.

netlify.toml:

[[redirects]]
from = “/path1/path2”
to = “/path3/”
status = 302
force = true # COMMENT: ensure that we always redirect
headers = {X-From = “Netlify”}

2 Likes

Hi @annejan!

Is there a redirect rule causing /path3/ to redirect? Or were you expecting a status 200 for it?

Hi,

We noticed the same problem since today where url with query string are globally cached. On our side that means that users where able to access the content of other users because of the redirection.

Attached you can find an example of such a redirect that happened (the redirect change the parameter, especially in this case individualId and documentId

Our netlify.toml looks like:

[[headers]]
  # Define which paths this specific [[headers]] block will cover.
  for = "/static/registry.json"
    [headers.values]
    Access-Control-Allow-Origin = "*"
[[headers]]
  for = "/?q*"
    [headers.values]
    cache-control = '''
      max-age=0,
      no-cache,
      no-store,
      must-revalidate'''

We added the cache invalidation on /?q to try to fix the issue, unsuccessfully. We have no redirect rules

Redirects from path to full url (domain + path) without this having setup in redirects.

We are experiencing same issue, with a large user base receiving personalized data via individual user tokens. Because of forced redirects, user-data was breached this AM, with all system users being directed to other individuals’ dashboards, which are served by API to an SSG using a token. Help me understand the rationale in such an aggressive caching/redirecting behaviour being released into stable environments.

Netlify, what have you done? ! The most basic promise of the internet is that I type in a link and you send me to that link.By failing to keep that promise, you have broken systems which rely on individual user tokens. You have leaked many peoples’ data. Today, I type in a link with my token, and I end up seeing someone else’s data. It feels like a betrayal, to be honest. You’ve broken a system for me which many people worked very hard on for a very long time. With no warning, for no good reason. @marcus

hey @f3rg! thanks for sharing that. We appreciate it - we are aware of some of the issues you are describing and currently working on it. Not the experience we want you to have, for sure. Stay tuned, we’ll update here as soon as we have news.

1 Like

@perry Perry, Imagine a scenario where a hypothetical technology company rolled out a hypothetical COVID19 screening tool to a hypothetical large factory where hypothetical employees had to be screened daily or were refused access to the factory. Your reply is helpful and conveys good intentions, but I want to ensure you understand the magnitude of the problem.

I hear you, that is definitely very impactful.

Here’s what we are doing:
We are actively looking in to this internally - i see some of our most knowledgeable folks on the case - and we will present options to you as soon as we have them.

In the meantime, I have some information from one of our devs that could be a short term mitigation. is there any way you shift to not using the apex domain, but the www subdomain for sensitive query params so they will not be redirected?

thanks for your patience - again, this isn’t the experience we are hoping for and we are working to mitigate.

1 Like

That is a helpful mitigation for the immediate concerns. We will look at doing that while we prepare our migration out.

a quick update that we are making some progress, but are not in the all clear yet. I or someone from my team will circle back here when we have something specific to share.

2 Likes

All of our activation emails have something like this “https://app.ourdomain.com/?access_token=abcdefg
Our users are reporting when they click this link that they are logging in as other users. I assume this is the same issue reported above. We likely need to bring our entire company site down until this is fixed. Is there any update?

Yup, that does sound like the same thing, @Enalmada. Check out my next update in this thread for current status and next steps in case you are still affected.

1 Like

Hi folks,

I understand this was a highly impactful issue to the folks in this thread and a few others we had heard from. I am personally deeply sorry for this impact, and I do speak for Netlify on this as well.

Our engineers have found the cause and repaired it in our codebase and deployed that fix, and have started a process to repair our CDN’s cache for all impacted sites. We are working on sites from paid customers first, and then working our way to the free ones. We expect all paid sites to be fixed by tomorrow, and the free ones will likely not complete until sometime tomorrow or even next week.

However, you can self-serve on the fix already: redeploying your affected site(s) will clear its cache on the CDN, so as soon as you do so, the problem should no longer occur.

We’d love to hear from you as to whether the fix has worked for you (or perhaps more actionably, if it hasn’t, and at what example URL or x-nf-request-id in case your URL’s are sensitive or unshareable publicly (we can find the URL from that, but an attacker could not).

@f3rg I do hear you loud and clear that this was a breach of trust. We will be doing an internal retrospective call about the causes and how to prevent them from happening in the future, and will report back here. I don’t believe anything we say can rebuild trust, but I do believe that things we do - improving testing, discovering the full nature of the root cause of both the problem and missing it during our (extensive!) pre-testing of the feature - are one step in that process, as is the transparency we’ll provide here. You of course have to make the call about whether any of this is enough for you, but as your advocate to the business, my promise is that I am doing all that I can to see those actions implemented and reporting back here as to the progress.

3 Likes

Redeploying our affected site seems to have solved the problem. Thank you for the fix.

2 Likes

Just wanted to report back to this thread that our cache repair for all free customers without a custom domain who have not deployed in the past 6 days (the last group to complete, millions of sites) was just completed.

So, nobody should see any leakage anymore, but I think everyone in this thread had probably redeployed already and was no longer affected.