Urgent - DATABASE IS EMPTY OF RECORDS

Appreciate there was going to be 5 mins of downtime on the 20th but its 3am on the 21st in the USA and 4pm here in Australia.

Two hours+ ago all the records from our database disappeared. No warning.
With no telephone numbers I am relying on support to read emails.
Not good enough when this catastrophic event has occured and I ahve no way of knowing why.

Anyone else dealing with such circumtances ?

1 Like

I cannot also access both the front and backend.

2 Likes

Same here, US East Coast, no front or back, or api response.

1 Like

Don’t mean to belittle your misery but its reassuring to know it’s not just our database that has issues.

Be damned NICE FOR PUBLIC ANNOUNCEMENTS of what’s going on and whats hapenning !!

I FEEL YOUR PAIN

Same here (UK). Urgently need an update from Knack on what’s going on!

Same here (UK as well) can login to the dashboard but unable to access any of our apps… All embedded apps on our live corp website are not loading as well. Angry customers already at our end! Urgently need an update from Knack. Really not good enough…

@RayWindlow47079 @Godfred @Delvin - just had this response from Knack

“Hey there,
You are having issues accessing your live apps and builders because of the planned maintenance.
The maintenance process is taking a bit longer than initially estimated but we are expecting it to be over soon.
Once the maintenance is finished, we will let you know.
Let us know if you have any additional questions.”

Our users reported at around 01:50 EST that they got white screen when accessing the Apps.

Email sent to Support, and so far no response (2+ hours passed).

BTW, where can we see the server location of our Apps ? Because some of the Apps are accessible, we suspected that the Apps in our Knack Account are distributed in various servers.

Maintenance was scheduled for 2 - 3 AM on the TWENTIETH not 2-3 am US TIME ON THE 21ST ?

Our data is back online albeit slow.

I am suggesting to my users as it is nighttime here that we stay away overnight until everyone is loaded back and functioning properly !

Hey Folks!

So sorry that the scheduled downtime went on longer than expected. I am not too sure about the details just yet, but I am sure there will be more info from the team that we can share with you soon.

Your apps should all be back up and running, but if you are still having any troubles please do reach out to support@knack.com or tag me here with a ticket number and we can jump on and have a look for you.

Thanks so much for your patience while we grow from good to great!

Thanks for jumping in on this one @Ally :+1:

Awesome that you stepped up for the Knack Management!

This highlights the serious nature of managing a SaaS. Thousands of businesses rely on some sort of system reliability and Knack is failing here.

I’ve been a user since inception and have seen upgrades that really side-grades/downgrades over their maturity. To the point that I now look at different options, and that’s sad.

It’s one thing building a complicated web app with lots of JS/CSS/API to make it truly something special, but if you can’t trust the reliability and the CIO to stand behind the efforts then we need to look for other options.

That said, good work on being the front guy.

Many thanks
Phil

1 Like

I TEND TO AGREE, THIS LAST EFFORT FAILED DISMALLY WITH (AS IT TURNED OUT) WITH GLOBAL INTERRUPTIONS. (Sorry too lazy go back retype it again. Stuck Capslock.)
The worst part is NO TELEPHONE NO TO CALL TO FIND OUT WHAT THE HELL WAS GOING ON.
I had researchers asking me what was happening and all I could say is have patience lets sit it out.
Problem was I couldn’t understand why it was down !
The message re upgrades called for it to be done between 2 and 3 am I believe on the 20th yet here it was the 21st at 3am and a classic stuff up. Surely there is a way of doing a universal message to all users on login that something has gone astray rather than setting off panic mode when apps come back up but with no data.
We are far from the biggest users but there is 3 years of damned hard research gone into our records and my people were absolutely panic stricken despite my assurances there were backups etc.

So what I am getting at is your interrface with your valued users needs to improve 1000% over what is happening now. Unacceptable what just took place. Cheers

1 Like

Knack Software Service Disruption Post-Mortem and Improvements

We want to extend our gratitude for your patience during last week’s software outage. We understand the inconvenience and frustration it caused, and we acknowledge that our communication during the incident fell short of expectations. In this post-mortem, we aim to provide more clarity on the cause of the outage and the steps we are taking to prevent similar issues in the future.

What happened during the disruption?

Last week, we performed a major version upgrade on our database systems. Our cloud provider assured us that downtime would be minimal, and our testing environments validated this. However, for our European and US production regions, the downtime was longer than anticipated. The majority of customers experienced 30 minutes or less of downtime, but some faced up to 4 hours.

Our lack of proactive communication on the status page compounded the problem, leaving many of you in the dark.

How are we addressing this?

In light of these events, our leadership team conducted a comprehensive post-mortem to analyze both the technical and communication aspects of the outage. As a historically engineering-centric company, we recognize the need to improve our communication processes as we continue to grow.

On the technical front, our engineering team is revamping our testing environments to more accurately mirror production systems. This will allow us to simulate peak production traffic and better understand the impact of changes on our infrastructure.

To improve communication, we are re-evaluating and redefining our internal processes for maintenance and outages, with a focus on keeping customers informed in real-time. We are working towards providing fine-tuned status updates for all our services. In the meantime, we have ensured that more Knack staff members are trained in managing our status page (https://status.knack.com/) and are committed to increased transparency regarding the status of the application.

We sincerely apologize for any inconvenience the prolonged disruption caused and appreciate your continued trust in Knack. We are taking the necessary steps to prevent similar incidents in the future and enhance our communication efforts going forward.

Thank you for reading - please reach out to me if you want to discuss, or email support@knack.com for information or updates on existing tickets.

Kara

5 Likes