Large html static files stopped working recently

Greetings ,

Your CDN service is great because you allow showing large html files generated by HoloViz technologies. However, for some reason, the large plots no longer show.

Here are links to plots that worked well int he past and stopped working now - they still work locally, yet no longer on Netlify.
https://jacob-barhak.netlify.app/thereferencemodel/results_2020_01_04_visual_2020_02_08/combinedplot

https://jacob-barhak.netlify.app/thereferencemodel/results_2020_03_21_visual_2020_03_23/combinedplot

Notice that newer smaller plots like for the COVID-19 model work nicely:
https://jacob-barhak.netlify.app/thereferencemodel/results_covid19_2020_06_27/combinedplot

However, the older larger plots stopped working. They did work in the past - I checked them a few months ago.

I uploaded the site only by dragging a file folder structure onto Netlify. The same files work nicely from my local machine, so I assume some changes in Netlify made things stop working.

Hopefully you can help explain what is going on and hopefully help fix this.

        Jacob

@jbarhak Welcome to the Netlify community.

Could the three non-loading assets have anything to do with this? I note, though, that on the working page there are four assets that fail to load, so maybe not. I just a firm believer in starting off by fixing what you know to be wrong before moving into speculatory areas.

Hi @jbarhak!

It seems we recently introduced a time limit on how long our file origin is allowed to take to serve HTML files. This limit is now 30 seconds. This is enough for the common use case of hosting websites at Netlify since you won’t expect most site visitors to wait more than 30 seconds for a page to load. Most visitors leave way earlier.

We constantly iterate on our file serving architecture to make it reliable and scalable for all users. This limit better helps us deal with the scale we are operating at.

This is how you can still make it work:
A best practice for Jamstack sites is to take advantage of javascript to load large assets asynchronously, so you can give the visitor an indication of “this site is not broken” and ask them to be patient while you load the data.
My advice is to put the data that seems to be currently inlined in the HTML page into its own file and have a smaller html file that loads it. Remember we are not imposing the 30 second limit on non-HTML files.

Have fun building!

Thanks Marcus,

This workaround may be functional, yet it really is not helpful in my situation. I did try your solution to the best of my ability, yet you are aiming for web developers while your platform is simple enough for much simpler users. It seems you are missing this audience with your recent modifications.

You see, I do not really control the html files - those are generated for me by Holoviz tools. using the save command with embed:
https://panel.holoviz.org/_modules/panel/io/save.html

One of the plots that stopped working are key in interactive posters I showed at the NIH and other locations, you can see one here:
https://jacob-barhak.github.io/Poster_MSM_IMAG_2020.html

If you click on the Results button in the interactive poster , you should see the plots - and there is a timeout there, yet it still does not load the file - due to the problem you highlighted.

You can see the link to the other non working plot in this publication:
https://www.modsimworld.org/papers/2020/MODSIM_2020_paper_42_.pdf

Both of these were working nicely in the past - so I know the problem is not on my side.

Look, I am not a web developer - I just use third party tools to generate the files I am showing. Can you help me by one of the following:

  1. Being more specific with a code example of what I should do - I really do not want to mess with the code inside the html - it is auto generated. I tried renaming the file locally to .plt and using the following redirection code in an html file:
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Refresh" content="7; url=CombinedPlot.plt" />
</head>
<body>
<p>Please wait for <a href="CombinedPlot.plt">this link</a> to load.</p>
</body>
</html>

However, it just shows the file text in the browser - so this does not work for me, even if I used type="text/html". This is my level of web development - you must know a better solution.

  1. Instead of trying to trick the system, is there a way that netlify can render the large files if their html is generated by certain systems such as panel / holoviews that generate those huge files? The reason I chose Netlify was because it allowed those huge html files - this was once a service no other system provided. And those huge files do take time to render - it is normal so 30 seconds may not be enough. Hopefully Netlify will still provide this service in the future.

Look Marcus, this is pandemic time and I develop disease models - not web pages. I worked hard to make those plots available to the public and those specific plots that now stopped working show a technique that allows including human interpretations with the disease model. If someone now tries to look at those plots and get nothing, in the worst case they may think bad things about my work , with no fault of mine or in the best case may not understand the technique because they cannot play with the sliders and see the results change. So things not working at this time are bad for me.

Can you / Netlify help me make those work again like it worked so nice in the past? Netlify has a great service that no one else offers, please help keep it great.

Jacob

And @marcus,

Is there a way to register large files so that Netlify can inspect those and approve them for long loading files?

You see, the files you see will only grow in size if I add more populations / models / interpretations. As far as I know you are the only service that was able to handle those huge static files. And those files are important for reproducibility - this is a large issue in scientific modeling these days and If I can package all the results in one file - it is an important capability - despite the disadvantage of slow loading. If I was aiming for speed I would have created a server to serve those results and not use Netlify.

Yet by using Netlify, I get the advantage of documentation being interactive - think about a powerpoint presentation that is interactive - this is an important capability. Until now Netlify supported it.

Hopefully I make sense and you can help me resolve this issue.

Jacob

Thanks @gregraven

Your post was initially missed, yet since you took a look, I wanted to reply.

The files work nicely locally and used to work well in the past - I think the reason things are not working are what @marcus mentioned. Netlify changed something in the service which made things break.

You mention errors - those do not seem to change anything on either edge or chrome locally and things worked nicely in the past. If you can explain how to reach those errors, I can ask Holoviz why those errors are generated - it may be even bokeh or some other library that needs fixing or something my plot generation code caused - you see those html files are auto generated from python code - I really do not know what is inside those just like I do not know what is inside a doc file or a powerpoint file, yet it is great that those are generated statically while having interactive capabilities.

Anyway, I hope Netlify can fix this so things can work again like they did a few months ago.

Jacob

@jbarhak I just took another look. They missing assets are only .map files, so you are correct that they are probably not causing a failure of any kind. Sorry about the false alarm.

Thanks @gregraven ,

Your second look is appreciated. This is not a false alarm, this means you found something that needs investigation that I did not know about - I will ask the Holoviz community what those mean. It may help improve their already very good product.

Jacob

Hi @marcus

Are you sure that there are no limits on files other than html?

A friend sent me a javascript load function that should load the huge file while displaying the message and I deployed it with the huge file with a .plt extension instead of html. You can find it here:
:
https://jacob-barhak-test.netlify.app/loadfile

The code is below.

For some reason this does not work. Are you sure that Netlify blocks only html files that take more than 30 seconds to load?

Hopefully there is a simple example solution that can be followed.

Jacob

<html>
  <head>
    <title>Put original title here</title>
  </head>
  <script>
    function loadFile(file) {
      // read text from URL location
      var request = new XMLHttpRequest();
      request.open("GET", file, true);
      request.send(null);
      request.onreadystatechange = function () {
        if (request.readyState === 4 && request.status === 200) {
          var type = request.getResponseHeader("Content-Type");
          if (type.indexOf("text") !== 1) {
            document.write(request.responseText);
          }
        }
      };
    }
    loadFile(
      "https://jacob-barhak-test.netlify.app/CombinedPlot.plt"
    );
  </script>
  <body>
    Loading large file... this may take a while.
  </body>
</html>

Hi, @jbarhak. I’ve tested the download of the asset. First, I wanted to be sure the content type is correct (and not text/html for some reason). The content type is correct as application/octet-stream:

$ curl -I https://jacob-barhak-test.netlify.app/CombinedPlot.plt
HTTP/2 200
accept-ranges: bytes
cache-control: public, max-age=0, must-revalidate
content-length: 96627347
content-type: application/octet-stream
date: Thu, 01 Oct 2020 00:03:17 GMT
etag: "b604284abd015386d21dc878041931cf-ssl"
strict-transport-security: max-age=31536000; includeSubDomains; preload
age: 2
server: Netlify
x-nf-request-id: b70f413d-736e-41c1-a9bb-ac01cf61983a-11572916

Next, I checked to see if the download itself was stopped at the 30 second mark. However, it was not:

$ time wget https://jacob-barhak-test.netlify.app/CombinedPlot.plt
--2020-09-30 17:05:37--  https://jacob-barhak-test.netlify.app/CombinedPlot.plt
Resolving jacob-barhak-test.netlify.app (jacob-barhak-test.netlify.app)... 206.189.73.52, 167.172.215.127
Connecting to jacob-barhak-test.netlify.app (jacob-barhak-test.netlify.app)|206.189.73.52|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96627347 (92M) [application/octet-stream]
Saving to: ‘CombinedPlot.plt’

CombinedPlot.plt                         100%[========================================================================================>]  92.15M  1.18MB/s    in 79s

2020-09-30 17:06:57 (1.17 MB/s) - ‘CombinedPlot.plt.1’ saved [96627347/96627347]


real	1m19.394s
user	0m0.397s
sys	0m0.967s

I also tested the code itself (because it is part of that same test site):

The full file was sent in that test and it took much longer than 30 seconds.

To summarize, I do think you should be able to load this via javascript and it isn’t being cut off by the CDN at 30 seconds.

If you are seeing it cut off, a HAR file recording or the x-nf-request-id header for the truncated response would be helpful.

Thanks @luke,

Your tests look promising. First I see that the entire file seems to be downloaded - it shows locally as:
92.1 MB (96,627,347 bytes)

And I see that this was detected correctly on your tests. I am not sure what the content type should be, yet this is an html file generated by HoloViz panel and it works nicely in many other plots - it just breaks in the large ones now.

This is the same html file that was once accessible via :

https://jacob-barhak.netlify.app/thereferencemodel/results_2020_01_04_visual_2020_02_08/combinedplot

I just changed the file name extension from html to plt when I uploaded the test example.

If you report that the file is fully loaded by the javascript script, this means that it is not rendered due to another reason. It did show nicely in the past and it does render when I open the html locally. If the 30 second cut off that @marcus reported is not the reason for things breaking, I might need to check details more closely and use your advice.

I am not a web developer, so I appreciate your guidance and I will check the har file recording option in hope I can figure out what is the problem with the original file. I will report here once I figure out details.

Your help is very much appreciated.

Jacob

Hey Jacob,

Just spoke about this one with our traffic team. Yeah, changing the file type and serving through a smaller HTML file, via JavaScript, would be our advised solution here as we currently timeout HTML requests after 30 seconds. It’s a combination of both sets of advice thus far but definitely something which can be worked around!

Thanks @Scott

You find me working on the solution, yet since I see no recipe for the small loader file it makes it hard on someone like me who uses higher level tools to create web pages.

Both the html redirect and the javascript loader I sent before as possible solutions do not seem to work - I am not sure why and it seems this confuses several web developers I know. I really am trying to follow the instructions the community gives and using other resources.

In fact I made some changes to the test site and used two html files with plots. One is larger and one is smaller and used both plt and html extension for both. Then I tried the javascript loader in various forms to see if the problem is the file extension or the file size. In all cases the javascript above failed.

For example, this small html file works:
https://jacob-barhak-test.netlify.app/combinedplotsmall

However, this big html file does not work:
https://jacob-barhak-test.netlify.app/combinedplotbig

Those were html versions - I also have .plt versions:

I tried the javascript loader file on all 4 variations and it failed in all of them:

https://jacob-barhak-test.netlify.app/loadfilebig

https://jacob-barhak-test.netlify.app/loadfilebightml

https://jacob-barhak-test.netlify.app/loadfilesmall

https://jacob-barhak-test.netlify.app/loadfilesmallhtml

So I concluded that the javascript loader I added is not working - especially since it cannot even load the small html.

If you have an example of such a small loader file or know why it will not show the small file html example after loading the larger file, please let me know.

Hopefully I can get this running back quickly.

Jacob

I am asking some friends who do web development to help.

Thanks @Scott, Thanks Greg, Thanks @luke, Thanks Marcus,

You all looked at the problem and helped me reach the solution.

The javascript loader that loads the large files works now.

I had two friends who helped: Guy who wrote the skeleton code and Kevin that helped me debug it. Thanks to them both.

I am sending the working solution to the community in case it will be useful:

<html>
  <head>
    <title>Put original title here</title>
  </head>
  <script>
    function loadFile(file) {
      // read text from URL location
      var request = new XMLHttpRequest();
      request.open("GET", file, true);
      request.send(null);
      request.onreadystatechange = function () {
        if (request.readyState === 4 && request.status === 200) {
          var type = request.getResponseHeader("Content-Type");
          if (type.indexOf("text") !== 1) {
            document.write(request.responseText);
            document.close()
          }
        }
      };
    }
    loadFile(
      "CombinedPlot.plt"
    );
  </script>
  <body>
    Loading large file... this may take a while.
  </body>
</html>

I hope Netlify will approve of this solution.

Jacob

thank you for sticking with this and sharing the solution, @jbarhak! glad we found a way forward together :smiley:

Well @perry,

You have to understand that this was critical for me to fix. You see, the plots that were not working show a new modeling technique that merges human interpretation with disease models. It took me years to develop this technology.

I am showing this for Diabetes, yet I plan to apply it to COVID-19 once I get enough expert opinions to merge with the ensemble model.

Just to explain, those interpretations allow adding assumptions such as: the number of incidences is over reported for a reported outcome. For example for COVID19 it can mean the number of infections or number of deaths.

Those plots were published earlier this year and therefore important - especially since those techniques are applicable to the COVID-19 pandemic. So it was a high priority to solve this problem. So all those who contributed to solve this issue, should know this was an important contribution.

I really do not know how long those files were not accessible - I do know they worked around March / April when I published the paper and poster. However, if the community here helps spread the word about this technology, perhaps this incident will have a positive outcome.

And I must add that Netlify is currently the only service I know that will host such large files for free with a very easy interface that fits people like me.

Jacob

1 Like