Netlify Functions returning duplicate headers

Support
1

My Netlify function is now returning doubled headers. AFAICT, nothing has changed on my end. Did some Netlify CDN setting change? This is breaking my redirects because the location field is repeated and browsers don’t understand it.

2

I’m seeing doubled headers even on old Deploy Previews, which makes me think it can’t be on my end.

3

It’s happening on almanack.data.spotlightpa.org but not www.spotlightpa.org, so it’s not all Netlify sites. Old deploy previews that worked before are now broken by the double Location header, so it’s not a code change on my end.

4 
$ curl -IXGET https://almanack.data.spotlightpa.org/api/bookmarklet/pennsylvania-house-speaker-mike-turzai-retirement
HTTP/2 307
almanack-app-version: 88035c2
almanack-app-version: 88035c2
cache-control: no-cache
content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
location: /articles/TQNXUFYXD5E6TBYC7GVI7Y6PKQ/schedule
location: /articles/TQNXUFYXD5E6TBYC7GVI7Y6PKQ/schedule
date: Wed, 10 Jun 2020 13:00:48 GMT
content-length: 81
age: 0
server: Netlify
x-nf-request-id: bd28a56d-24a6-41ed-9961-10974459e5c0-6562887
8

Still not working:

$ curl -IXGET https://almanack.data.spotlightpa.org/api/healthcheck
HTTP/2 200
almanack-app-version: c3859cc
almanack-app-version: c3859cc
cache-control: public, max-age=60
cache-control: public, max-age=60
content-type: text/plain
content-type: text/plain
date: Thu, 11 Jun 2020 13:13:26 GMT
content-length: 659
age: 0
server: Netlify
x-nf-request-id: d72edf8e-32bb-4a2d-8086-550b04065480-3535263

Can’t reproduce this locally.

9

Hey @carlmjohnson,
We’re looking into this but have not had other reports of duplicate headers on functions that are proxied to. We are able to replicate what you’re seeing for a bunch of the routes here:
https://github.com/spotlightpa/almanack/blob/master/pkg/api/routes.go#L26

like:

  • curl -svo /dev/null https://almanack.data.spotlightpa.org/api/scheduled-articles/5FLSBYA36JANHM5AOKCXGNC33U
  • curl -svo /dev/null https://almanack.data.spotlightpa.org/api/proxy-image/aHR0cHM6Ly93d3cuaW5xdWlyZXIuY29tL3Jlc2l6ZXIvVUU3dkhHQVhodWR1QU9mVThSVFJDTDNmZHpVPS9hcmMtYW5nbGVyZmlzaC1hcmMyLXByb2QtcG1uL3B1YmxpYy81VjVWSjJTRUJOR1pWQU9FNFVPTllWWEdCNC5qcGc=

Is it possible that your middleware is adding duplicate headers somewhere?

We’re continuing to investigate but wanted to share the update and ask about that middleware.

10

I don’t believe it’s my middleware because in local testing, there are no duplicate headers. It’s possible that there’s a difference between local and production because of the AWS Lambda wrapper, but it hasn’t been a problem before. I will see if I can find another way to test it.

12

I forked my project, but I’m still seeing double headers in the fork:

https://boring-elion-2589ea.netlify.app/api/healthcheck

$ curl -IXGET https://boring-elion-2589ea.netlify.app/api/healthcheck
HTTP/2 200
almanack-app-version: 2427cd2
almanack-app-version: 2427cd2
cache-control: public, max-age=60
cache-control: public, max-age=60
content-type: text/plain
content-type: text/plain
date: Thu, 11 Jun 2020 17:28:30 GMT
content-length: 865
age: 15
server: Netlify
x-nf-request-id: d15e1760-f3ad-42eb-9aa2-fa5d2cca3bb4-4020284
13

I changed to a different AWS Lambda adaptor, and it seems to have fixed the issue! I will file a ticket with the old adaptor. It’s strange because I have not changed the adaptor in some time, so I don’t know why it would only have the issue now instead of before.

Thanks for your help.

$ curl -IXGET 'https://boring-elion-2589ea.netlify.app/api/healthcheck'
HTTP/2 200
almanack-app-version: 70101ea
cache-control: public, max-age=60
content-type: text/plain
date: Thu, 11 Jun 2020 17:37:01 GMT
content-length: 726
age: 6
server: Netlify
x-nf-request-id: d15e1760-f3ad-42eb-9aa2-fa5d2cca3bb4-4373964
14

This appears to be the problem: Add multi-value headers, sync with fork, init go-mod · piotrkubisa/apigo@e12c481 · GitHub

The AWS Lambda adaptor is setting MultiValueHeaders as well as regular Headers. I believe the Lambda should realize that two sets of Headers are the same, but it apparently causes a duplication.

15

Wow, nice find and thanks a lot for reporting back on the root cause and your fix.

TIL about AWS Lambda adaptors! For others who come across this issue in the future, I’m adding a nice article here on how they fit into serverless workflows:

16

I wrote up how I use Netlify + Go back in March: How to Use Netlify to Deploy a Free Go Web Application · The Ethically-Trained Programmer Basically, I’m using an adaptor so I can use normal Go HTTP handlers instead of special AWS Event handlers. It makes testing in local development much simpler than having to create a Lambda simulator or whatever.

1 Like