My Netlify function is now returning doubled headers. AFAICT, nothing has changed on my end. Did some Netlify CDN setting change? This is breaking my redirects because the location field is repeated and browsers don’t understand it.
I’m seeing doubled headers even on old Deploy Previews, which makes me think it can’t be on my end.
It’s happening on almanack.data.spotlightpa.org but not www.spotlightpa.org, so it’s not all Netlify sites. Old deploy previews that worked before are now broken by the double Location header, so it’s not a code change on my end.
$ curl -IXGET https://almanack.data.spotlightpa.org/api/bookmarklet/pennsylvania-house-speaker-mike-turzai-retirement HTTP/2 307 almanack-app-version: 88035c2 almanack-app-version: 88035c2 cache-control: no-cache content-type: text/html; charset=utf-8 content-type: text/html; charset=utf-8 location: /articles/TQNXUFYXD5E6TBYC7GVI7Y6PKQ/schedule location: /articles/TQNXUFYXD5E6TBYC7GVI7Y6PKQ/schedule date: Wed, 10 Jun 2020 13:00:48 GMT content-length: 81 age: 0 server: Netlify x-nf-request-id: bd28a56d-24a6-41ed-9961-10974459e5c0-6562887
Still not working:
$ curl -IXGET https://almanack.data.spotlightpa.org/api/healthcheck HTTP/2 200 almanack-app-version: c3859cc almanack-app-version: c3859cc cache-control: public, max-age=60 cache-control: public, max-age=60 content-type: text/plain content-type: text/plain date: Thu, 11 Jun 2020 13:13:26 GMT content-length: 659 age: 0 server: Netlify x-nf-request-id: d72edf8e-32bb-4a2d-8086-550b04065480-3535263
Can’t reproduce this locally.
We’re looking into this but have not had other reports of duplicate headers on functions that are proxied to. We are able to replicate what you’re seeing for a bunch of the routes here:
curl -svo /dev/null https://almanack.data.spotlightpa.org/api/scheduled-articles/5FLSBYA36JANHM5AOKCXGNC33U
curl -svo /dev/null https://almanack.data.spotlightpa.org/api/proxy-image/aHR0cHM6Ly93d3cuaW5xdWlyZXIuY29tL3Jlc2l6ZXIvVUU3dkhHQVhodWR1QU9mVThSVFJDTDNmZHpVPS9hcmMtYW5nbGVyZmlzaC1hcmMyLXByb2QtcG1uL3B1YmxpYy81VjVWSjJTRUJOR1pWQU9FNFVPTllWWEdCNC5qcGc=
Is it possible that your middleware is adding duplicate headers somewhere?
We’re continuing to investigate but wanted to share the update and ask about that middleware.
I don’t believe it’s my middleware because in local testing, there are no duplicate headers. It’s possible that there’s a difference between local and production because of the AWS Lambda wrapper, but it hasn’t been a problem before. I will see if I can find another way to test it.
I forked my project, but I’m still seeing double headers in the fork:
$ curl -IXGET https://boring-elion-2589ea.netlify.app/api/healthcheck HTTP/2 200 almanack-app-version: 2427cd2 almanack-app-version: 2427cd2 cache-control: public, max-age=60 cache-control: public, max-age=60 content-type: text/plain content-type: text/plain date: Thu, 11 Jun 2020 17:28:30 GMT content-length: 865 age: 15 server: Netlify x-nf-request-id: d15e1760-f3ad-42eb-9aa2-fa5d2cca3bb4-4020284
I changed to a different AWS Lambda adaptor, and it seems to have fixed the issue! I will file a ticket with the old adaptor. It’s strange because I have not changed the adaptor in some time, so I don’t know why it would only have the issue now instead of before.
Thanks for your help.
$ curl -IXGET 'https://boring-elion-2589ea.netlify.app/api/healthcheck' HTTP/2 200 almanack-app-version: 70101ea cache-control: public, max-age=60 content-type: text/plain date: Thu, 11 Jun 2020 17:37:01 GMT content-length: 726 age: 6 server: Netlify x-nf-request-id: d15e1760-f3ad-42eb-9aa2-fa5d2cca3bb4-4373964
This appears to be the problem: https://github.com/piotrkubisa/apigo/commit/e12c4810a2cfba1c4e9e033f286a42b435418880
The AWS Lambda adaptor is setting MultiValueHeaders as well as regular Headers. I believe the Lambda should realize that two sets of Headers are the same, but it apparently causes a duplication.
Wow, nice find and thanks a lot for reporting back on the root cause and your fix.
TIL about AWS Lambda adaptors! For others who come across this issue in the future, I’m adding a nice article here on how they fit into serverless workflows:
I wrote up how I use Netlify + Go back in March: https://blog.carlmjohnson.net/post/2020/how-to-host-golang-on-netlify-for-free/ Basically, I’m using an adaptor so I can use normal Go HTTP handlers instead of special AWS Event handlers. It makes testing in local development much simpler than having to create a Lambda simulator or whatever.