Operations

Code Deployments

Service Code

We release small-increment (atomic) updates to our multi-tenant services on a constant basis, following the continuous delivery practice called Scaled Trunk-based Development. On average, there are roughly 80-100 releases per month. After ensuring 100% test coverage and a (human) code review, service updates are automatically rolled out to all customers simultaneously. The majority of them are dependency updates which typically go completely unnoticed, or fixes for issues some or all customers were having. Features or enhancements can come feature-flagged for single customers or projects, meaning they will only be activated on an opt-in basis via configuration.

Customer Code

Unlike traditional AEM as a Cloud Service, there is no CI/CD pipeline for Edge Delivery Service. Code changes are picked up directly from the branches in your GitHub or bring your own git repository. Within seconds, every branch is automatically published under its own distinct URL for testing and staging or changes. Production code is typically served from the main branch.

We strongly recommend project developers follow the same Scaled Trunk-based Development model as we do for our services. This ensures you merge small pull requests into production often, but the quality assurance & review efforts are limited to small change sets. Nobody wants to review and test large pull requests, and long-lived branches with lots of changes tend to be difficult (and dangerous) to merge. For more details, please read our developer best practices.

Observability

Logging

All our technical services feed into a centralized, and redundant SIEM system, powered by Coralogix and Splunk. Based on this, alerts for critical levels of errors are set based on thresholds that are constantly updated based on the evolution of the services. Our observability infrastructure is operated independently from the operational infrastructure.

Our logging is focussed on perimeter logging, so that the operational properties of the individual services are prioritized over internal state. Depending on the nature of the collected logs, different retention policies are applied, ranging from two weeks to 25 months.

Monitoring

Our observability setup consists of a fine-grained set of highly sensitive synthetic monitors and log-based alerts for all our technical services. The slightest anomaly, be it related to a change we made ourselves or an issue with one of our 3rd party vendors, immediately alerts our on-call rotation.

We also have extra synthetic monitoring in place for our top 10 customer sites by traffic, which gives us the confidence that our delivery service is performing and scaling as intended.

Shared Responsibility Model

Edge Delivery Services is a service offering that relies on a shared responsibility security and operational model. These responsibilities are shared between Adobe, the customer, and third-parties like cloud service providers and content delivery network (CDN) providers.

Adobe Responsibilities

Adobe is responsible for maintenance, security, operations, and availability of the following service endpoints:

The following resources also fall under Adobe's responsibility in terms of maintenance, security and availability:

Furthermore, Adobe is responsible for monitoring the health of cloud services used to render its own services, and working with its cloud service vendors to resolve any issues that may occur.

Customer Responsibilities

As a customer, you are responsible for the following:

Third-Party Responsibilities

Cloud service providers are responsible for the maintenance, availability, network and physical security of the data centers and cloud services that host Edge Delivery Services endpoints as well as the customer's project code and content.

Incident Management

Our operations team is assigned to a 24/7 on-call rotation split into two 12 hour shifts on two continents. Adobe On-Call ensures prompt notification of on-call engineers on several channels including phone calls, text messages and push notifications. We vow to acknowledge every incident within 15 minutes, although in reality we are typically a lot quicker.

We maintain detailed runbooks for each type of incident to ensure we can restore the affected service as fast as possible. Our process includes root cause analysis (RCA) and we publish postmortems for every single incident, no matter how small the customer impact was.

Disaster Recovery

Adobe maintains detailed disaster recovery plans for all business services and regularly conducts disaster recovery tests to validate that both delivery and API services can be restored well within their respective intended recovery time objectives (RTOs).

Publishing

The Admin API is a single-cloud deployment. In case of a disastrous outage in this service, the intended RTO is 12 hours.

The recovery point objective (RPO) depends on the type of content source:

Delivery

The Content Hub is where all published content, media, and code is stored. For this tier, we rely on active/active replication in a multi-cloud setup. Unlike traditional approaches to disaster recovery like active/standby or multi-region deployments, an active/active multi-cloud setup ensures that any published content, media, or code is stored redundantly in at least two different cloud providers with different, but functionally identical software stacks.

In case of an outage, even a global outage of the first cloud provider's control plane that would affect a multi-region setup, all content is still available in the second cloud provider and operations can resume without data loss.

The active/active deployment means that during normal operations, the workloads are split roughly equally between our cloud providers and only in case of an outage of one, the remaining providers will pick up the load.

The delivery service itself is also deployed redundantly, so in case of an outage at one cloud provider, Adobe can plan to switch to the other and resume delivery near-instantaneously. The intended RTO for this service is 15 minutes. Thanks to our redundant dual-stack architecture, in the event of an outage data can be restored up until the very moment a disaster occurred, making the RPO 0.

Backup and Restore

How content is restored and who is reposinsible depends on what caused the loss:

Restoring after an outage

Recovering from a platform outage or disaster is Adobe's responsibility and requires no customer action. Published content, media, and code in the Content Hub are protected by two independent mechanisms:

In the event of an outage, Adobe restores the affected service from redundant storage or from backup within the documented RTO and RPO.

Restoring your own content and code

Recovering from your own changes (for example an accidental deletion, an unwanted edit, or a rollback to an earlier state) is done at the source. Edge Delivery Services follows a source-of-truth model in which your content and code live in systems you control, and the delivery tier serves published copies of them. This differs fundamentally from traditional AEM as a Cloud Service, which restores content from Adobe-managed backups through Cloud Manager. In Edge Delivery Services you restore at the source using its native version history, and then re-publish:

After restoring the source, preview and publish the affected content, or redeploy the code, so the change propagates to *.aem.page and *.aem.live.

Data Handling

Data Residency

By default, all customer data is stored in US EAST data centers of different cloud service providers. If you have special requirements regarding data residency or cloud service providers used, please contact Adobe.

Data Retention

All customer data (including audit logs) is kept by Adobe for the entirety of the license term.

Customer data (including backups) will be deleted by Adobe within 90 days after customer notification or termination of the license term.