August 22nd, 2023 Deno Deploy Post-mortem
On August 22nd, 2023, beginning at 07:00 UTC and ending at 08:30 UTC, the
deno.land experienced a
distributed denial of service (DDoS) attack.
This attack not only impacted
deno.com and Deno Deploy dashboard users, but
also Deno runtime users importing packages through HTTPS URLs on
We sincerely apologize for any disruption casued by this attack on our infrastructure. We have taken steps to protect against this style of attack in the future, and remain firm in our commitment to providing reliable service for Deno Deploy and any software that depends on our infrastructure for dependency management.
During a 90-minute period, the Deno Deploy admin console and the
deno.land websites were unreachable. This also impacted our third-party module
deno.land/x and the standard library at
deno.land/std. Lack of
access to the registry may have caused build time errors for some Deno runtime
Deno applications where dependencies were already cached or vendored were not
impacted. Other applications aside from
deno.com already deployed and running
on Deno Deploy were not impacted.
All times in UTC, on August 22nd, 2023.
- 07:00 - DDoS attack begins
- 07:02 - Team members alerted to elevated errors on deno.com
- 07:10 - Team members discover a DDoS attack underway and begin investigation
- 07:41 - Additional team members paged to assist mitigation efforts
- 08:21 - Attack mitigated as offending IP addresses are blocked
- 08:29 - Team confirms system recovery
- 08:59 - Incident is formally resolved
We estimate a downtime of approximately 90 minutes from when our systems first started failing until full recovery was achieved.
A DDoS attack was mounted against
deno.com, which is hosted on Deno Deploy.
The very large volume of requests exceeded the ability of Deno Deploy to scale
up and allocate more resources to this application, causing server errors that
made web pages and modules hosted on
Thankfully, resource isolation between “deployments” (versions of apps hosted on Deno Deploy across projects and customers) was effective in shielding other applications on Deno Deploy from significant impact.
We realize how disruptive this service interruption was to many Deno users, both on Deploy and using the Deno runtime. We are taking swift action to ensure that this type of incident is not possible in the future.
- We are improving our ability to allocate additional resources to Deno Deploy applications that are experiencing an abnormally large spike in traffic.
- We are improving our on-call process, so that incidents are escalated more quickly and appropriately.
- We have also identified our need for more effective communication during incidents, and will ensure that outage reports are more promptly added to denostatus.com.
- More broadly, we are looking at changing how we host Deno module code to
prevent incidents on
deno.comand Deno Deploy impacting dependency management for other Deno programs.
Once again, we deeply regret the impact this incident may have had on your work. Thanks to your support and collaboration, we will continue to evolve and harden Deno and Deno Deploy in the months and years to come.
If you have additional questions or concerns, please start a conversation with us here.