A/B Split Testing Issue

Hi there,

Recently we tried launching an A/B test using Netifly Split Testing but have run into some issues, we’re having difficulty solving. The test appears to be working in that we’re seeing sessions show up in our custom dimension in GA, the issue is that transactions don’t appear to be working. Currently for the last six days, it shows 149 transactions for the main branch and 1 for the test branch – even if the test branch is not as good, it seems highly unlikely that this would happen.

For additional context, our tech stack is Gatsby, Craft and Shopify (for the backend). We’re using Google Tag Manager extensively as well. My guess is that the GTM trigger for transactions is not registering the test branch but I’m really not sure. Can anyone help?

Site name: diggs-production
URL: diggs.pet

Hi @bartfish. We cannot troubleshoot what your Google Tag Manager behavior is as that is outside the scope of our support. We can and will troubleshoot what Netlify is doing though.

To clarify, I cannot debug the javascript responsible for sending the data. I can troubleshoot if our CDN is sending the correct files to the user’s browser. (There are custom support contracts which do include troubleshooting custom code and we can get you in contact with our sales team if you are interested in that.)

For example, regarding what Netlify is doing, I checked the time window since these two deploys occurred:

Since that time I do see a 50/50 traffic split for those two deploys. I can see the traffic split is working.

If in debugging this, you believe our service is sending an incorrect file, please let us know.

If you see us sending incorrect files, we need to be able to track the HTTP response with this issue. The simplest way to do this is to send us the x-nf-request-id header which we send with every HTTP response.

There more information about this header here:

If that header isn’t available for any reason, please send the information it replaces (or as many of these details as possible). Those details are:

  • the complete URL requested
  • the IP address for the system making the request
  • the IP address for the CDN node that responded
  • the day of the request
  • the time of the request
  • the timezone the time is in

Again, we can troubleshoot what Netlify is doing but we cannot troubleshoot the site javascript. Someone else here in community might have the bandwidth to troubleshoot it though (but our support team does not).

If there are other questions of if there a response which is incorrect, please let us know.