Update 6/11/2020: WIX support reached out to me last night and told me they were investigating. As of 9:30 AM Eastern this remains unresolved. Check for yourself here.
On May 30th the Sectigo AddTrust External CA Root expired. While generally not an issue since modern browsers try to fill in the intermediate CAs etc. If you're behind a firewall this becomes an issue. Like it has for some of you.
A little explanation on how SSL/TLS certs work is key to understanding this.
When a browser requests a secure website (where the URL starts with https://) an authentication process takes place, where the server has to prove that it is who it says it is, it does this by presenting the SSL/TLS certificate assigned for that website. Think of this like proving you are who you say you are by presenting your ID.
The website proves that it's not using a face SSL/TLS cert (Fake ID) by also providing a certificate chain of the issuing authority, known as a Root Certification Authority (Root CA). Think of this as certain "tell tale" clues to look for on an ID to make sure it's not fake (Hologram, the second picture, etc.) Browsers usually store the most common Root CA of these to make your life easier as a user, so that you don't have to track these down to make your browser not display "this page is not secure". Something IT professionals are well acquainted with due to the self signed cert.
If the Root CA is valid or trusted by the browser, the web page displays without issue. If it is not, an error like this is shown:
Sometimes, certificates are issued by what are known as an "Intermediate Certificate Authority", essentially a 3rd party that has been vetted by a Root CA to issue certificates on behalf of said Root CA. This is where things get weird because now a web server with a cert issued by an Intermediate CA needs to display the full chain and it usually looks like this:
As you can see in this example USERTrust RSA Certification Authority is our root CA, and the Sectigo RSA Domain Validation Secure Server CA is the intermediate. Finally you have ultraviolet.network which is the subject name.
A quick look at this, and you'd think everything is AOK. Except it might not be. What do you mean?! Remember I mentioned that browsers have a number of Root CA's cached, same goes for Intermediate. Here's where the fun begins.
A browser's main job is to provide a user experience, that user experience is to browse the web with ease and convenience. Browsers will try to find the right CA/Intermediate despite the fact that the web server may not present the correct chain. One thing that adds a wrinkle to the scenario surrounding this post is the fact that the Sectigo AddTrust signed certs are also "Cross Signed" by other CAs, as explained by Sectigo:
CAs often control multiple root certificates, and generally the older the root the more widely distributed it is on older platforms. In order to take advantage of this fact, CAs generate cross certificates to ensure that their certificates are as widely supported as possible. A cross certificate is where one root certificate is used to sign another.
The cross certificate uses the same public key and Subject as the root being signed.
This is why you're seeing the "proper" chain for this website in the image above.
However, many users have reported issues with browsing certain websites starting May 30th. And they all had something in common - they were all signed by the Sectigo AddTrust CA Root. This is what the error looked like if you were behind a FortiGate:
Depending on the vendor of the firewall, if you experienced an error it may look different.
But wait a minute?! You just said that browsers will try to do this for me.
I did, and they will. But a Firewall is not a browser, a Firewall is different. A Firewall trusts nothing unless you tell it to, the default policy on a firewall is an implicit deny. Well in the same vain, if you present a broken chain, or a chain with an expired cert, it will not look at whether or not the cert is cross signed, it will only validate the presented chain and will not look to correct any errors in presentation.
As such the reason you may have gotten this error is due to the fact that what's actually being presented to your Fortigate (and browser) looks like this:
Which brings me to the title of this post. WIX may or may not decide that I'm too much of a noise maker and they're better off without me as a tenant. Frankly I'm considering it anyway given what's transpired.
After posting another post yesterday, a reader hit me up and said they got hit with a similar error to the "This Connection is Invalid" presented above. When I started digging I looked at the cert presented in my browser and saw "USERTrust RSA etc.." and that looked fine. But then I remembered when I was writing Let's Encrypt with Fortigate I ran into a similar issue with using curl and the requests python library; despite the browser showing a valid chain, cURL and requests would say it was not a trusted cert. After some Googling I found the above seecert () shell function and with it, the output.
As you can see, this is what is being presented when your browser, requests www.ultraviolet.network. As I mentioned above, your browser goes "This doesn't look right, but I see it's cross signed, so let's check those out. Oh cool the cross signed certs are still good, we'll put those in the chain". A Flow-Based firewall policy would have no impact on this either.
However with a Proxy-Based firewall policy, a high-level explanation is your browser sends the request to the website, and since you're behind a firewall, it responds (i.e. impersonates) as the websit and takes your request, and forwards it to the website, the website responds, and the firewall receives the response, inspects it according to the policy requirements, and then forwards on to your browser.
If one of those requirements is to reject invalid certs, any certificate presenting with an expired, incomplete, or otherwise invalid certificate, intermediate CA, or root CA will be blocked. Which is why many websites started getting blocked on May 30th.
So I reach out to support, and I spent about an hour and a half with a gentleman we'll call Fred. Fred starts looking into the issue and tells me that everything looks fine on his end, and I explain the issue, provide screenshots, as well as information regarding the Sectigo AddTrust CA root, and he tells me he needs to escalate to the proper team, it will be a brief hold while he escalates. 15 minutes later he comes back saying that he did some DNS lookups and noted that:
ultraviolet.network DNS zone was not managed by WIX, in order for the HTTPS cert to present correctly, it needs to be managed by WIX.
DNSSEC needs to be set to unsigned.
I explain to Fred that to my understanding these would have no impact on presentation of the website over HTTPS, I did acknowledge that they could be an issue if I was having resolution failures, but I am not. And that the issue is an invalid certificate chain issue being presented. Again, he reiterated that if I fixed the above the cert issue would go away. I asked for an explanation for how an SSL/TLS cert could be impacted by the DNS zone being hosted somewhere other than WIX, and it was all working fine until May 30th. And the above was reiterated a 3rd time. I requested the escalation again, and he opened a ticket with the proper group and escalated.
Last night I receive this email from WIX Support:
It's at this point that I start to feel frustrated, because I am being dismissed by the "it works in my browser" approach. And I re-iterated that the browser's job is to make sure it provides the convenient yet secure browsing experience, but a firewall is very different. I provided the screenshots above. Requested that at a minimum that I be allowed to present whatever cert I'd like to present instead of having WIX do it on behalf of me.
I have yet to hear back, however I am contemplating migrating off this platform due to this fiasco. The point here is this, I don't monetize this platform, I don't have a ton of readers, I'd be surprised if that number was over 50. But I do pay for a service, and I don't feel like this service is being provided properly, all I am asking for is for it to be fixed.
If you made it this far, thanks for reading a crazy man's ramblings.
Comments