SSL Issues, Jan 2018

Some users have been reporting issues getting SSL certificates issued.  We’ve looked into it, and it turns out that yes, there’s A Problem.

Now I should say right now, this probably won’t affect you.  We’ve got some short-term measures in place that should last just fine until we deliver a long-term fix.

But we like to be transparent, and we’d rather you understood what was happening rather than panic if it does raise its ugly little head at some point.

So here’s the details.

We use a service called Let’s Encrypt that lets us issue SSL certificates for any domain pointed to us, because they’re fully automatable and generally awesome.  They also provide tool packages called Managers to do the heavy lifting involved in issuing these certificates, and we use one of their off-the-shelf Managers to do this work.

The flow looks a bit like this:

Just over a week ago, they found a security hole in their old way of checking that we actually controlled the domain we wanted a certificate for.

So they switched it off.  Got to admit, would have been nice if they’d given us a bit of warning, but security’s important so we can kinda see their point of view here.

Anyhow, we swapped to their new way of doing it, and everything worked fine again.

Until yesterday, when we started getting more reports of SSL failures.

And it turned out there was a problem.

Sometimes, this new method fails.  We don’t know yet if this is due to how it works or due to something we’ve done in our setup, but sometimes we ask for an SSL certificate and Let’s Encrypt goes:

Most of the time, the request is retried and everything works fine, but the failure stays in the queue.

And our existing Manager doesn’t support clearing those failures.

So when the queue fills up, we can’t issue any more SSL certificates.

Now, this queue’s pretty big, which is why it took over a week to fill up and for us to realise there was a problem here.

We’ve set up a new queue so currently, SSL certificates will be issued as normal.  That’s the short-term fix.

The long-term fix is developing our own Manager package, which we’re working on now.

Should this matter to you?  Probably not.  But we’d like you to know what’s going on, so if you do have repeated failures in getting SSL certificates for your Convertri sites, you can let us know and we can give it a kick again.  It’s possible we’ll need to set up more queues before this is fully sorted.

We’ll update this post once the long-term fix is in place.  Until then, we hope you won’t have any issues, but let us know if things do start to flare up.