Pretty URLs, URL Routing, and Trailing-Slashes

Greetings :wave:t2:

I feel like this post may be most pertinent for @fool since this topic’s been handled a number of times here on The Community (by him and others) but I’m looking to clarify implementation details and strongly urge some extended documentation and/or UI changes.

The “Pretty URLs” check-box has global routing side effects that are un-documented (as far as I can tell) and un-beholden to the “Disable Asset Optimization” flag, meaning that the “Pretty URLs” checkbox has side effects that are beyond the scope of the parent “disable” flag, even though the “Pretty URLs” checkbox itself becomes disabled when the parent flag is checked.

To walk this out, I made four sites cataloging the routing (and markup re-writing) behavior of when the Pretty URLs flag is checked or unchecked and when the “Disable Asset Optimization” flag is checked or unchecked:

The good news is that the advertised benefit of the “Pretty URLs” feature seems to work fine when the box is checked and asset optimization is turned on - the part where anchor tags in the HTML markup have their href= changed to remove the .html tail (at least when the file exists). This makes sense. That’s the control pattern I would expect - if AO is disabled overall, anchor-href-swapping shouldn’t occur. If AO is enabled overall but Pretty URLs is disabled, anchor-href-swapping shouldn’t occur. If AO is enabled overall and Pretty URLs is enabled, anchor-href-swapping should occur. This is the working behavior and that’s great.

What’s not so great and quite confusing is that the value of the Pretty URLs check-box, even when it’s disabled (e.g. “disable all asset optimization” is enabled) has meaningful routing changes for the site. Those four example sites document that routing fairly well, but there are some serious impacts there. Namely, SEO gets super knocked if you’re serving the same content on two paths (notably like ./blog/post no-slash and ./blog/post/ slashed) - which is controlled by the Pretty URLs routing side-effect.

In regards to the Pretty URLs feature, the docs currently state:

In addition to forwarding paths like /about to /about/ (a common practice in static sites and single-page apps), it will also rewrite paths like /about.html to /about/

But that’s it. I think the first part, “In addition to forwarding paths like /about to /about/may be referencing routing changes? And the latter mentions the markup changes / anchor-href-swapping? But overall there could be a lot more documentation on what’s going on with that flag. Especially thoroughly documenting that, even when that flag is not editable, its checked-or-not-checked status will have significant routing impacts on the site.

Outside of adding more documentation, could this functionality be refactored and split? The concept of parsing my HTML and anchor-href-swapping sounds good, and I understand is related to the server’s routing to some degree, but I think they could be de-coupled and it would make things easier to understand. Ideally I’d love to see a separate panel inside the Build & Deploy screen that allows a site to switch between “Resolve regardless of slash trailing” and “Follow document / directory based routing for trailing slash” (and perhaps a warning that the former can hurt SEO? :stuck_out_tongue_winking_eye: ) – this toggle should be switchable and controllable regardless of what the Asset Optimization settings are, and that would allow the removal of the global routing side-effects tied to the Pretty URLs feature. Pretty URLs would truly just be anchor-href-swapping at that point (which is what we’d expect given the available controls).

I realize that would be a breaking change and would probably require some cross-over time for things to get hashed out, but the current functionality is difficult to work with and mostly undocumented.


For what it’s worth, I’m also not remiss to the fact that a lot of people are running PWAs on Netlify and are having client-side routing issues too, making this whole conversation more convoluted and difficult, but I think it would actually be more helpful to sort out PWA routing problems if the Netlify controls were more clear on exactly what the Edge is going to do regarding its own routing.

Thanks for the long read :slight_smile: Technically @Scott called for it :wink:

Jon :netliheart:

Edit: Gatsby community reference

4 Likes

@jonsully Wow, this must have been a lot of work to figure out.

I just use canonical meta tags:

1 Like

Haha. Yeah Canonical tags can be super useful and are a great tool, but even they always work best when routing is clear and straightforward. They also can’t help with client-side routing, which is a whole bag of worms on its own :rofl:

Plus analytics platforms recognize the slashed and non-slashed URLs as different and that can break up your analytics data in bad ways too :grimacing: that’s actually how I first realized my site was having this issue and what lead me to dig ALL THE WAY DOWN the rabbit hole :laughing: - (note the /blog/ and /blog as separate records)

1 Like

Hey @jonsully,

Love the repros. I’ll get this raised internally and we’ll take it from there. As always, thanks for your help and input!

1 Like

Hey Jon,

I’ve briefly spoken with our traffic team ahead of our meeting later this week. We’ve got a plan moving forward – we’re going to have a sit down, work out what happens (using your repros) for each use case and I’ll get a support guide published with our advised configs, the role that the SSG plays in trailing slashes and how we make use of canonical link headers. Stay tuned!

1 Like

Awesome. Thanks @Scott. Hopefully getting a good guide in place (especially covering the awkward controls in that UI panel) will clear things up for a lot of folks. :netliheart:

1 Like

Greetings, friends :wave:

Just wanted to check in on this topic, as it’s been a little over a year and I think the “pretty URLs” checkbox/control is still contained inside of a “Disable All” checkbox/control even though they appear to behave independently :slight_smile:

We’re having some really great discussions and developments around how to handle trailing slashes in the Gatsby Community so I thought it would be a good time to hopefully nudge this topic too. :netliheart:


Jon

Thanks for the reminder, Jon! I think the long-open UX improvement request probably fell off our team’s radar due to some cleanups and manager handoffs for that squad. I just pinged the current manager about our proposal to improve it and while I haven’t heard back yet, I do know he can’t just “squeeze it in” since it needs a lot of work. However I can promise it is now back on our radar.

Since you have written (thank you!) our best docs on asset optimization after you researched, if you did have a “quick win” suggestion - he will review this thread. Could you let us know what you think if there is some low hanging fruit that isn’t totally overhaul the thing?

2 Likes

Aw thanks! Happy to help :grin: and appreciate you soliciting my input, that means a lot!

Genuinely my honest take is that the whole problem can be solved by some very low hanging fruit. Seriously :laughing:. Just some markup tweaks that shouldn’t even involve any javascript functionality changes muchless backend API changes. Given the context that this is the “Not-editing” portion of the view that’s in question:

and that this is the “While-editing” portion of the view:

My suggestion for making things more clear is to change the “Not-editing” view to look like this:

Which should simply be changing the text inside the header to better indicate that there are two separate control groups inside this panel … and maybe move the “URLs” line to the bottom of the table since “X & Y” implies that Y comes after X

Then inside the “While-editing” panel just move a few rows around, change a couple of header sizes, and change a little bit of the copy to more clearly indicate what-does-what:

I’m not a front-end person and mocked this up with DOM manipulation, so it needs a little :nail_care: but bear with me. The parts I’d consider important:

  • URL Handling is named as such (“Pretty URLs” is an opaque concept) and explained with just a touch of detail in the description, but left as default ON and called “standard handling” since the default is indeed the standard web-server behavior since the dawn of the internet
  • URL Handling is given its own equal-weight header to Asset Optimization and horizontal lines help separate it from the group of actual asset optimizations (controlled by the “Disable Asset Opt.” checkbox) — could even indent the actual asset optimization options a little too
  • “Disable Asset Optimizations” checkbox description clarifies that it’s only for the CSS/JS/Img controls below, just doubly clarifying that that checkbox is separate and unrelated to the URL Handling control

All of this while retaining that all inputs are still inside the same semantic <form> and the JS handler code for that form wouldn’t need to change at all, but the UI way better reflects that the URL handling is fully separate from the “Disable Asset Opt” control.

Hope that helps or inspires.


Jon

HTML Code from DOM in Details Block
<form class="floating-labels">
  <div>
    <div class="table-body">
      <dl>
        <dt>CSS</dt>
        <dd>
          <div class=""><label><input data-testid="checkbox"
                class="tw-w-[20px] tw-h-[20px] tw-p-0 tw-border tw-mr-1 tw-mt-[2px] tw-mb-0 tw-ml-[2px] tw-box-border tw-absolute tw-top-auto before:tw-content-empty before:tw-absolute before:tw-origin-top-left focus:tw-shadow-checkbox tw-cursor-pointer hover:tw-border-teal tw-border-gray focus:tw-border-gray focus:hover:tw-border-teal focus:hover:checked:tw-border-teal-darkest checked:tw-bg-teal-darker checked:tw-border-teal-darker focus:checked:tw-border-teal-darker hover:checked:tw-bg-teal-darkest hover:checked:tw-border-teal-darkest dark:hover:checked:tw-bg-teal dark:hover:checked:tw-border-teal tw-bg-transparent before:tw-h-[11px] before:tw-inline-block before:tw-w-[3px] before:tw-rounded-sm before:tw-left-[7px] before:tw-top-[13px] before:tw-transform before:tw-rotate-[-135deg] after:tw-w-[3px] after:tw-h-[7px] after:tw-rounded-sm after:tw-content-empty after:tw-absolute after:tw-top-[7px] after:tw-transform after:tw--rotate-45 after:tw-left-[3px] dark:after:tw-bg-transparent checked:before:tw-bg-gray-lightest checked:after:tw-bg-gray-lightest dark:checked:after:tw-bg-gray-darkest dark:checked:before:tw-bg-gray-darkest dark:checked:tw-bg-teal-lighter dark:checked:tw-border-teal-lighter"
                type="checkbox" value=""><span
                class="tw-pl-[32px] tw-block tw-cursor-pointer tw-text-base tw-text-gray-darkest tw-font-semibold dark:tw-text-gray-lightest">Bundle
                CSS</span>
              <div
                class="tw-text-muted tw-text-sm tw-ml-[32px] tw-mt-0 tw-text-gray-darker tw-font-regular dark:tw-text-gray-lighter">
                Concatenate consecutive CSS files together to reduce HTTP requests.</div>
            </label></div>
          <div class="tw-mt-2 md:tw-mt-2"><label><input data-testid="checkbox"
                class="tw-w-[20px] tw-h-[20px] tw-p-0 tw-border tw-mr-1 tw-mt-[2px] tw-mb-0 tw-ml-[2px] tw-box-border tw-absolute tw-top-auto before:tw-content-empty before:tw-absolute before:tw-origin-top-left focus:tw-shadow-checkbox tw-cursor-pointer hover:tw-border-teal tw-border-gray focus:tw-border-gray focus:hover:tw-border-teal focus:hover:checked:tw-border-teal-darkest checked:tw-bg-teal-darker checked:tw-border-teal-darker focus:checked:tw-border-teal-darker hover:checked:tw-bg-teal-darkest hover:checked:tw-border-teal-darkest dark:hover:checked:tw-bg-teal dark:hover:checked:tw-border-teal tw-bg-transparent before:tw-h-[11px] before:tw-inline-block before:tw-w-[3px] before:tw-rounded-sm before:tw-left-[7px] before:tw-top-[13px] before:tw-transform before:tw-rotate-[-135deg] after:tw-w-[3px] after:tw-h-[7px] after:tw-rounded-sm after:tw-content-empty after:tw-absolute after:tw-top-[7px] after:tw-transform after:tw--rotate-45 after:tw-left-[3px] dark:after:tw-bg-transparent checked:before:tw-bg-gray-lightest checked:after:tw-bg-gray-lightest dark:checked:after:tw-bg-gray-darkest dark:checked:before:tw-bg-gray-darkest dark:checked:tw-bg-teal-lighter dark:checked:tw-border-teal-lighter"
                type="checkbox" value=""><span
                class="tw-pl-[32px] tw-block tw-cursor-pointer tw-text-base tw-text-gray-darkest tw-font-semibold dark:tw-text-gray-lightest">Minify
                CSS</span>
              <div
                class="tw-text-muted tw-text-sm tw-ml-[32px] tw-mt-0 tw-text-gray-darker tw-font-regular dark:tw-text-gray-lighter">
                Run CSS through a minifier to reduce file size.</div>
            </label></div>
        </dd>
      </dl>
      <dl>
        <dt>JS</dt>
        <dd>
          <div class=""><label><input data-testid="checkbox"
                class="tw-w-[20px] tw-h-[20px] tw-p-0 tw-border tw-mr-1 tw-mt-[2px] tw-mb-0 tw-ml-[2px] tw-box-border tw-absolute tw-top-auto before:tw-content-empty before:tw-absolute before:tw-origin-top-left focus:tw-shadow-checkbox tw-cursor-pointer hover:tw-border-teal tw-border-gray focus:tw-border-gray focus:hover:tw-border-teal focus:hover:checked:tw-border-teal-darkest checked:tw-bg-teal-darker checked:tw-border-teal-darker focus:checked:tw-border-teal-darker hover:checked:tw-bg-teal-darkest hover:checked:tw-border-teal-darkest dark:hover:checked:tw-bg-teal dark:hover:checked:tw-border-teal tw-bg-transparent before:tw-h-[11px] before:tw-inline-block before:tw-w-[3px] before:tw-rounded-sm before:tw-left-[7px] before:tw-top-[13px] before:tw-transform before:tw-rotate-[-135deg] after:tw-w-[3px] after:tw-h-[7px] after:tw-rounded-sm after:tw-content-empty after:tw-absolute after:tw-top-[7px] after:tw-transform after:tw--rotate-45 after:tw-left-[3px] dark:after:tw-bg-transparent checked:before:tw-bg-gray-lightest checked:after:tw-bg-gray-lightest dark:checked:after:tw-bg-gray-darkest dark:checked:before:tw-bg-gray-darkest dark:checked:tw-bg-teal-lighter dark:checked:tw-border-teal-lighter"
                type="checkbox" value=""><span
                class="tw-pl-[32px] tw-block tw-cursor-pointer tw-text-base tw-text-gray-darkest tw-font-semibold dark:tw-text-gray-lightest">Bundle
                JS</span>
              <div
                class="tw-text-muted tw-text-sm tw-ml-[32px] tw-mt-0 tw-text-gray-darker tw-font-regular dark:tw-text-gray-lighter">
                Concatenate consecutive JS files together to reduce HTTP requests.</div>
            </label></div>
          <div class="tw-mt-2 md:tw-mt-2"><label><input data-testid="checkbox"
                class="tw-w-[20px] tw-h-[20px] tw-p-0 tw-border tw-mr-1 tw-mt-[2px] tw-mb-0 tw-ml-[2px] tw-box-border tw-absolute tw-top-auto before:tw-content-empty before:tw-absolute before:tw-origin-top-left focus:tw-shadow-checkbox tw-cursor-pointer hover:tw-border-teal tw-border-gray focus:tw-border-gray focus:hover:tw-border-teal focus:hover:checked:tw-border-teal-darkest checked:tw-bg-teal-darker checked:tw-border-teal-darker focus:checked:tw-border-teal-darker hover:checked:tw-bg-teal-darkest hover:checked:tw-border-teal-darkest dark:hover:checked:tw-bg-teal dark:hover:checked:tw-border-teal tw-bg-transparent before:tw-h-[11px] before:tw-inline-block before:tw-w-[3px] before:tw-rounded-sm before:tw-left-[7px] before:tw-top-[13px] before:tw-transform before:tw-rotate-[-135deg] after:tw-w-[3px] after:tw-h-[7px] after:tw-rounded-sm after:tw-content-empty after:tw-absolute after:tw-top-[7px] after:tw-transform after:tw--rotate-45 after:tw-left-[3px] dark:after:tw-bg-transparent checked:before:tw-bg-gray-lightest checked:after:tw-bg-gray-lightest dark:checked:after:tw-bg-gray-darkest dark:checked:before:tw-bg-gray-darkest dark:checked:tw-bg-teal-lighter dark:checked:tw-border-teal-lighter"
                type="checkbox" value=""><span
                class="tw-pl-[32px] tw-block tw-cursor-pointer tw-text-base tw-text-gray-darkest tw-font-semibold dark:tw-text-gray-lightest">Minify
                JS</span>
              <div
                class="tw-text-muted tw-text-sm tw-ml-[32px] tw-mt-0 tw-text-gray-darker tw-font-regular dark:tw-text-gray-lighter">
                Run JS through a minifier to reduce file size.</div>
            </label></div>
        </dd>
      </dl>
      <dl style="border-bottom: gray;">
        <dt>Images</dt>
        <dd>
          <div class=""><label><input data-testid="checkbox"
                class="tw-w-[20px] tw-h-[20px] tw-p-0 tw-border tw-mr-1 tw-mt-[2px] tw-mb-0 tw-ml-[2px] tw-box-border tw-absolute tw-top-auto before:tw-content-empty before:tw-absolute before:tw-origin-top-left focus:tw-shadow-checkbox tw-cursor-pointer hover:tw-border-teal tw-border-gray focus:tw-border-gray focus:hover:tw-border-teal focus:hover:checked:tw-border-teal-darkest checked:tw-bg-teal-darker checked:tw-border-teal-darker focus:checked:tw-border-teal-darker hover:checked:tw-bg-teal-darkest hover:checked:tw-border-teal-darkest dark:hover:checked:tw-bg-teal dark:hover:checked:tw-border-teal tw-bg-transparent before:tw-h-[11px] before:tw-inline-block before:tw-w-[3px] before:tw-rounded-sm before:tw-left-[7px] before:tw-top-[13px] before:tw-transform before:tw-rotate-[-135deg] after:tw-w-[3px] after:tw-h-[7px] after:tw-rounded-sm after:tw-content-empty after:tw-absolute after:tw-top-[7px] after:tw-transform after:tw--rotate-45 after:tw-left-[3px] dark:after:tw-bg-transparent checked:before:tw-bg-gray-lightest checked:after:tw-bg-gray-lightest dark:checked:after:tw-bg-gray-darkest dark:checked:before:tw-bg-gray-darkest dark:checked:tw-bg-teal-lighter dark:checked:tw-border-teal-lighter"
                type="checkbox" value=""><span
                class="tw-pl-[32px] tw-block tw-cursor-pointer tw-text-base tw-text-gray-darkest tw-font-semibold dark:tw-text-gray-lightest">Compress
                Images</span>
              <div
                class="tw-text-muted tw-text-sm tw-ml-[32px] tw-mt-0 tw-text-gray-darker tw-font-regular dark:tw-text-gray-lighter">
                Run all images through lossless image compression.</div>
            </label></div>
        </dd>
      </dl>
    </div>
    <div class="table-header"></div>
    <div class="table-header">
      <div class="table-body">
        <dl>
          <dt
            class="tw-flex tw-font-semibold tw-gap-1 tw-justify-start tw-leading-tight tw-text-xl after:tw-content-none md:tw-justify-between">
            URL Handling</dt>
          <dd>
            <div class=""><label><input data-testid="checkbox"
                  class="tw-w-[20px] tw-h-[20px] tw-p-0 tw-border tw-mr-1 tw-mt-[2px] tw-mb-0 tw-ml-[2px] tw-box-border tw-absolute tw-top-auto before:tw-content-empty before:tw-absolute before:tw-origin-top-left focus:tw-shadow-checkbox tw-cursor-pointer hover:tw-border-teal tw-border-gray focus:tw-border-gray focus:hover:tw-border-teal focus:hover:checked:tw-border-teal-darkest checked:tw-bg-teal-darker checked:tw-border-teal-darker focus:checked:tw-border-teal-darker hover:checked:tw-bg-teal-darkest hover:checked:tw-border-teal-darkest dark:hover:checked:tw-bg-teal dark:hover:checked:tw-border-teal tw-bg-transparent before:tw-h-[11px] before:tw-inline-block before:tw-w-[3px] before:tw-rounded-sm before:tw-left-[7px] before:tw-top-[13px] before:tw-transform before:tw-rotate-[-135deg] after:tw-w-[3px] after:tw-h-[7px] after:tw-rounded-sm after:tw-content-empty after:tw-absolute after:tw-top-[7px] after:tw-transform after:tw--rotate-45 after:tw-left-[3px] dark:after:tw-bg-transparent checked:before:tw-bg-gray-lightest checked:after:tw-bg-gray-lightest dark:checked:after:tw-bg-gray-darkest dark:checked:before:tw-bg-gray-darkest dark:checked:tw-bg-teal-lighter dark:checked:tw-border-teal-lighter"
                  type="checkbox" value=""><span
                  class="tw-pl-[32px] tw-block tw-cursor-pointer tw-text-base tw-text-gray-darkest tw-font-semibold dark:tw-text-gray-lightest">Standard
                  File/URL Behavior</span>
                <div
                  class="tw-text-muted tw-text-sm tw-ml-[32px] tw-mt-0 tw-text-gray-darker tw-font-regular dark:tw-text-gray-lighter">
                  Serves named html files without trailing slashes and named directories (with index.html files) with
                  trailing slashes</div>
              </label></div>
          </dd>
        </dl>
      </div>
    </div>
  </div>
  <div
    class="tw-flex tw-flex-wrap tw-actions-flex-gap-2 tw-items-center tw-justify-end md:tw-justify-start tw-w-full md:tw-w-auto tw-mt-4 md:tw-mt-4">
    <button class="btn btn-default btn-primary btn-primary--standard" type="submit">Save</button><button
      class="btn btn-default btn-secondary btn-secondary--standard" type="button">Cancel</button></div>
</form>
2 Likes

Thank you so, so much for the comprehensive and thoughtful write up! We appreciate the time you have taken to put these thoughts together. As @fool has said, we have shared this directly with the team that would work on this.

2 Likes

Absolutely my pleasure. Perhaps I could help more directly :laughing: jk, but I sincerely hope the above info helps!

2 Likes