Lambda function using binary to generate pdf has stopped working because tar has disappeared

I have previously developed a node lambda function for generating pdfs using wkhtmltopdf. In order to achieve this feat, I put the latest Centos build of wkhtmltopdf binary into a tar archive then base64 encoded that, then put that string into a json file which is included with the function (so it got packaged up when built).

During execution, the function would decode the string from the json file, write the resulting tar file to the tmp directory, then untar it there. Then using a node wkhtmltopdf library I was able to generate a pdf using a handlebar.js html template and posted data.

It worked and I got pretty pdfs.

netlify site: divshares.netlify.com
function: prospect-report.js -> divshares.netlify.com/.netlify/functions/prospect-report
NB: The function can only be called in POST. It requires a form encoded string consisting of a single field data whose value is a json string setup for reasons I can’t remember. Here is a sample submission:

data=%7B%22name%22%3A%22Person+Name%22%2C%22email%22%3A%22person%40example.com%22%2C%22prospects%22%3A%5B%5D%2C%22dataDate%22%3A%229th+March+2019%22%2C%22dummyData%22%3Atrue%2C%22values%22%3A%7B%22sort%22%3A%22company%22%2C%22PE_RATIO%22%3A%5B1%2C15%5D%2C%22PAYOUT_RATIO%22%3A%5B50%2C75%5D%2C%22NET_YIELD%22%3A%5B6%2C22%5D%2C%22FRANKING%22%3A%5Btrue%2Ctrue%2Ctrue%5D%2C%22EXCLUDE_SECTOR%22%3A%5B%5D%2C%22EXCLUDE_INDUSTRY%22%3A%5B%5D%2C%22EXCLUDE_COMPANY%22%3A%5B%22EBG%22%2C%22LYL%22%2C%22MTO%22%2C%22REX%22%2C%22RXP%22%2C%22SDG%22%2C%22SGF%22%2C%22SRG%22%2C%22TRS%22%2C%22VLW%22%2C%22VTG%22%2C%22CVF%22%2C%22CAA%22%2C%22ASH%22%2C%22AFG%22%5D%7D%7D

Recently, the function has stopped working. Initially it had 500 errors, when I investigated the logs it reported an ENOENT error no command found for tar. I wrote up a custom endpoint that reads the path variable and runs ls -1 on every directory it finds:

http://divshares.netlify.com/.netlify/functions/sys-probe

the current result is:

{"results":["node","npm","npx","","","[","alias","arch","awk","base64","basename","bash","bashbug","bashbug-64","bg","ca-legacy","captoinfo","cat","catchsegv","cd","chcon","chgrp","chmod","chown","cksum","clear","comm","command","cp","csplit","cut","date","dd","df","dgawk","dir","dircolors","dirname","du","echo","egrep","env","expand","expr","factor","false","fc","fg","fgrep","fmt","fold","gawk","gencat","getconf","getent","getopts","grep","groups","head","hostid","iconv","id","igawk","info","infocmp","infokey","infotocap","install","jobs","join","ldd","link","ln","locale","localedef","logname","ls","makedb","md5sum","mkdir","mkfifo","mknod","mktemp","mv","nice","nl","nohup","nproc","numfmt","od","p11-kit","paste","pathchk","pgawk","pinky","pldd","pr","printenv","printf","ptx","pwd","read","readlink","realpath","reset","rm","rmdir","rpcgen","runcon","sed","seq","sh","sha1sum","sha224sum","sha256sum","sha384sum","sha512sum","shred","shuf","sleep","sort","sotruss","split","sprof","stat","stdbuf","stty","sum","sync","tabs","tac","tail","tee","test","tic","timeout","toe","touch","tput","tr","true","truncate","trust","tset","tsort","tty","tzselect","umask","unalias","uname","unexpand","uniq","unlink","update-ca-trust","users","vdir","wait","wc","who","whoami","yes","","[","alias","arch","awk","base64","basename","bash","bashbug","bashbug-64","bg","ca-legacy","captoinfo","cat","catchsegv","cd","chcon","chgrp","chmod","chown","cksum","clear","comm","command","cp","csplit","cut","date","dd","df","dgawk","dir","dircolors","dirname","du","echo","egrep","env","expand","expr","factor","false","fc","fg","fgrep","fmt","fold","gawk","gencat","getconf","getent","getopts","grep","groups","head","hostid","iconv","id","igawk","info","infocmp","infokey","infotocap","install","jobs","join","ldd","link","ln","locale","localedef","logname","ls","makedb","md5sum","mkdir","mkfifo","mknod","mktemp","mv","nice","nl","nohup","nproc","numfmt","od","p11-kit","paste","pathchk","pgawk","pinky","pldd","pr","printenv","printf","ptx","pwd","read","readlink","realpath","reset","rm","rmdir","rpcgen","runcon","sed","seq","sh","sha1sum","sha224sum","sha256sum","sha384sum","sha512sum","shred","shuf","sleep","sort","sotruss","split","sprof","stat","stdbuf","stty","sum","sync","tabs","tac","tail","tee","test","tic","timeout","toe","touch","tput","tr","true","truncate","trust","tset","tsort","tty","tzselect","umask","unalias","uname","unexpand","uniq","unlink","update-ca-trust","users","vdir","wait","wc","who","whoami","yes","",""]}

Notably missing is tar, and also zip. I can imagine a reason for why it may have been removed, but I couldn’t find any announcements from either Netlify or Amazon that this had occurred. When I downloaded Amazon Linux 2, it had tar included.

I have tried deploying that tar binary in a similar way to the wkhtmltopdf binary, although it is small enough that I don’t need to compress it, I simply base64 encode it, then dump to decoded version also file in /tmp/. I can’t get any useful output from that either (I get an exit code of 2). Here is my latest log dump from the function:

3:03:49 PM: 2020-03-18T04:03:49.105Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	1584504229105 -> Invoked
3:03:49 PM: 2020-03-18T04:03:49.105Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	{
  name: 'Person Name',
  email: 'person@example.com',
  prospects: [],
  dataDate: '9th March 2019',
  dummyData: true,
  values: {
    sort: 'company',
    PE_RATIO: [ 1, 15 ],
    PAYOUT_RATIO: [ 50, 75 ],
    NET_YIELD: [ 6, 22 ],
    FRANKING: [ true, true, true ],
    EXCLUDE_SECTOR: [],
    EXCLUDE_INDUSTRY: [],
    EXCLUDE_COMPANY: []
  }
}
3:03:49 PM: 2020-03-18T04:03:49.172Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	/var/task
3:03:49 PM: 2020-03-18T04:03:49.184Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	Running command: chmod +x /tmp/tar
3:03:49 PM: 2020-03-18T04:03:49.204Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	
3:03:49 PM: 2020-03-18T04:03:49.204Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	Running command: tar --version
3:03:49 PM: 2020-03-18T04:03:49.208Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	TAR version tar (GNU tar) 1.26
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

3:03:49 PM: 2020-03-18T04:03:49.209Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	1584504229208 -> Writing out executable
3:03:49 PM: 2020-03-18T04:03:49.370Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	1584504229370 -> Expanding TAR
3:03:49 PM: 2020-03-18T04:03:49.371Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	Running command: tar --warning="all" xzvf /tmp/wkhtmltopdf.tar.gz -C /tmp/
3:03:49 PM: 2020-03-18T04:03:49.375Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	TAR output 
3:03:49 PM: 2020-03-18T04:03:49.375Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	Running command: ls -1la /tmp/
3:03:49 PM: 2020-03-18T04:03:49.382Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	total 15328
drwx------  2 sbx_user1051  495     4096 Mar 18 04:03 .
dr-xr-xr-x 17 root         root     4096 Feb 26 15:40 ..
-rw-rw-r--  1 sbx_user1051  495     2052 Mar 18 04:03 logo.svg
-rw-rw-r--  1 sbx_user1051  495     7544 Mar 18 04:03 randomfile.html
-rwxrwxr-x  1 sbx_user1051  495   341984 Mar 18 04:03 tar
-rw-rw-r--  1 sbx_user1051  495 15329991 Mar 18 04:03 wkhtmltopdf.tar.gz

3:03:49 PM: 2020-03-18T04:03:49.383Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	1584504229383 -> Executable created and untarred
3:03:49 PM: 2020-03-18T04:03:49.383Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	[node-wkhtmltopdf] [debug] [command] /tmp/wkhtmltopdf --page-size "A4" --debug-javascript --orientation "Landscape" --footer-right "Page [page] of [toPage]" --footer-center "P/E: 1-15%, PR: 50-75, NY: 6-22%, F: f,p,n" --footer-left "Demo-data for evaluation only" - -
3:03:49 PM: 2020-03-18T04:03:49.386Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	[node-wkhtmltopdf] [debug] /bin/bash: /tmp/wkhtmltopdf: No such file or directory

3:03:49 PM: 2020-03-18T04:03:49.387Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	[]
3:03:49 PM: 2020-03-18T04:03:49.387Z	2a3ced8c-d9bb-4f40-9d06-602c4c2eb531	INFO	1584504229387 -> PDF Generated
3:03:49 PM: Duration: 283.56 ms	Memory Usage: 269 MB

It’s a bit of a mess, the ls command which lists the /tmp/ directory is performed after the untarring, so it should have the wkhtmltopdf binary, but doesn’t. The tar command also doesn’t output any text, despite using the warning flag. Using the same binary on Amazon Linux 2 with the same decoded wkhtmltopdf.tar.gz ejects the binary fine and the filename appears in the console. Rather than trying to figure out why uploading tar doesn’t work (I can confirm that the encoded version works when decoded again to a file) I would rather understand why tar has gone missing, but any help getting any tar working would be great.

Any help would be much appreciated.

I have been able to resolve this.

I fixed the tar issue by using a node library (tar). However, wkhtmltopdf then complained of missing a bunch of dynamic libraries, so I added a new tar with those added. Then the text didn’t render, so I had to add a ttf into the mix also then modify the style sheet to use the font.

Now it works again.

Well, that is weird! We didn’t change anything on our side and it would be unusual for Amazon to change without notice as well, but I at least didn’t receive any or see any note that anyone else received it.

Very glad that you managed to find a fix and very thankful to you for following up to let us know how you’d done so.