Affecting Server - ST1

• 07/29/2022 02:18 - 07/29/2022 03:18
• Last Updated 07/29/2022 02:33

We are aware of issues with sending email via Webmail on the ST1 server and we are investigating and working to resolve them.

At this time it looks as though regular SMTP with a stand-alone mail client [ Outlook, Mac Mail, Thunderbird, etc ] are still working.

Update - 2:23 AM ET

This issue appears to be affecting inbound email as well.  This appears to have started with the nightly cPanel update.  We've engaged cPanel emergency support and are continuing to investigate on our end.

Affecting Other - Lets Encrypt

• 09/29/2021 00:00 - 10/01/2021 09:01
• Last Updated 10/01/2021 09:01

One of Let's Encrypt's intermediate certificates, used by browsers and other software to verify the issued certificates, expired September 29th [yesterday].  This intermediate certificate was used to issue and renew certificates whose expiration dates were after September 29th.

Some browsers and applications, such as mail clients, may report these certificates as expired or invalid.

The solution to this issue is to re-issue any certificates having issues via cPanel -> Let's Encrypt.  It may take up to 15 minutes for the new certificate to take effect.

For more details you can see this thread on Let's Encrypt's community forums: https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired-it-issued-certs-past-its-expiration-date/

Affecting System - SpamExperts Spam Filtering

• 09/15/2020 14:20 - 09/16/2020 00:19
• Last Updated 09/15/2020 17:28

SpamExperts is experiencing issues and they're aware of them and working to resolve them.  We wish we could provide an ETA but SpamExperts has not provided an ETA.

This is the first issue we've seen with them in nearly a decade of providing spam filtering through them so this is a very unusual incident.  When we have more information we will provide it here.

• 09/21/2018 18:10 - 09/26/2018 01:46
• Last Updated 08/26/2020 10:47

Update 7

All of our attempts to speed this process up haven't been successful.  As it stands we're restoring several dozen accounts per server all at the same time.  If you're seeing a cPanel IP Error page and you're using our DNS/Nameservers - your site isn't restored yet.  If you can't log into your cPanel with your cPanel username and password - your site isn't restored yet.

Nobody wants you back online more than us - I wish we could snap our fingers and everybody was online.  Better yet if we could go back and avoid this issue entirely that would be great.  As it stands - if you're not online yet - we're sorry and we're working on it.  If you are back online now - if you have any issues please let us know.

====
Update 6

We are working on two issues right now:
1. The cPanel Restoration process is not restoring MySQL data although the data is in the backup and verified good.  We're working with cPanel Emergency Support on this.
2. We are working on also getting a secondary copy of our backup data local to the backup server that we're going to be spreading across 3 servers with 10 GBPS links so that we can ideally cut the restoration time down as low as, in theory. 2.5 hours.  In reality I doubt we'll be able to saturate the connections due to cPanel restoration overhead - but we're doing our best to both get a second copy of the data just in case as well as giving us more throughput for restorations.

Ideally we'll have everything back online within 10 to 20 hours at the most - but we're still very much working-in-progress on this.

====
Update 5

It looks like our luck is against us.  The damage to the servers was more extensive than we believed originally.  We have the servers in a state of being ready to restore client data and when we attempted to mount the current data - the data can't be mounted / isn't workable.

We still have an administrator working on recovering this local data, however, it is looking like we're going to have to restore all client data from our latest backups.  We're not any more happy about this than any of our clients.  At this point it looks like it could take up to 30 or 40 hours to restore this data.

Once everything is back online we are going to be doing a complete overhaul on our backup system so that a full restore such as this will take only 4 to 6 hours.  There are some bottlenecks in our current backup system that are going to keep things from going quickly.  We could make some changes to the system now in an attempt to speed things up but as the backups we have now are the last remaining copy of the data we aren't going to be taking any chances.  As it stands while we hate to be offline for an extended period - the risk of migrating our backup drives to another server is too great.

====
Update 4

We are performing final updates  to the servers and software in preparation for bringing accounts back online.  There is the possibility that we may have missed some settings / software and there is some software that isn't a priority, like Softaculous, that we'll focus on once services are restored.

We will be getting started on bringing accounts back online shortly.  We're also working to try and make sure that account IP addresses do not change.  While we can't make any promises - we're doing our best.

====
Update 3

All servers are online - we're working on configuring all software - cPanel, LiteSpeed, MySQL, etc - to get them ready to bring accounts back online.

====
Update 2

We are working to bring online the servers that host all clients.  This is primarily a function of provisioning the guests on the hosts and configuring them to be ready to accept accounts.  Once this is done we will be conducing cPanel restorations to the servers of just the cPanel and MySQL data and re-connecting the accounts to the home directory data from before the outage.

We do have current and up-to-date MySQL data from before the outage, however, we're going to be restoring MySQL data from our backups taken last night and then will be working to restore any MySQL data lost between then and the outage with clients on a one-on-one basis as needed.

We hope to have everything back online tonight, however, we will keep you updated as we progress with the disaster recovery.

Once we are fully back online we will be providing a complete Reason For Outage [RFO] to all affected clients.  This RFO will outline what happened, why it happened, what we're changing to prevent it from happening again, and what we're changing so that should anything like this happen again we can recover from it substantially faster.

Our goal is always to be as open and transparent as we can be - and we will continue this.  Right now we're focused on restoring services and will provide more details once services are back online.

====
Update 1

We are having to bring new servers online to restore services.  We are working hard to get this done as quickly as possible and are going to do our best to do this with as little disruption as possible.

No customer data has been damaged or lost - only system-level data.  We will provide full details as to what caused this outage, what we did to resolve it, and what we're going to do to prevent it from happening again as soon as we have the chance.

====
Initial Message

We are experiencing a major outage across all services at this time. We are aware of the issue and are working to restore services as quickly as possible.

We will provide more detail when we can, however, we are focused on restoring services and diverting all energy to those tasks presently.

Server Status

Below is a real-time overview of our servers where you can check if there's any known issues.