Why is my production application that was healthy in production returning 404s when a rollback deploy fails?

Posted on April 10, 2021

I originally had a healthy application deployed to Digital Ocean Apps. I created a new deploy, which failed. This caused a rollback deploy to occur. This rollback also failed.

about 9 hours ago
Error
Automated rollback after failed deployment of "073dc20c-3910-42ca-aaeb-4c47c545ef3f"
Details
about 9 hours ago
Error
Commit 6c543f6 pushed to gitlab.com/...

At this time the live production app starts throwing 404s and is effectively taken offline. I am now forced to do a manual deploy to get my app running again.

Why is this the default functionality for deploys? I have health checks for a reason. My production containers should never be touched unless the deploy is successful. Is this not the case? If so it needs to be made very clear in big bold letters everywhere.

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

alexdo

October 31, 2023

Hey, I’ve had my fair share of deployment rollbacks, and I know how frustrating they can be. Especially in production.

Deployment systems often implement automated rollbacks as a safety measure. If a new deployment fails, the system will automatically revert to a previous stable state. This minimizes the impact on production.

Health checks are important, but they can’t catch everything. Sometimes, an application might appear healthy in terms of health checks, but there might still be issues that prevent it from functioning correctly under certain conditions.

You can check our Rollbacks docs here:

https://docs.digitalocean.com/products/app-platform/how-to/manage-deployments/#roll-back-to-a-previous-deployment

Regards