Multi-region high availability for Azure Content Understanding

Azure Content Understanding and Azure OpenAI Service are regional services. A production document-processing workflow often cannot treat a regional endpoint, its model deployments, or its analyzer definitions as a single point of failure. In this post, we discuss a highly available solution built across multiple regions to handle regional failures.

Architecture

Figure 1. Implemented regional stacks and cross-region flows.

The diagram shows two distinct planes: solid lines represent synchronous request traffic, while dashed lines represent asynchronous analyzer reconciliation. This separation addresses two related resilience challenges: routing requests to a healthy Content Understanding backend and keeping analyzer configurations consistent across regions. API Management selects a local-first backend synchronously within each available gateway, while two queue-driven Azure Functions synchronize analyzers asynchronously.

Reliable entry point

Users access the application through a DNS record which would be mapped to Azure Traffic Manager which acts as a DNS based load balancer which will route the traffic to primary region as long as the health check for the primary is good when the health check fails it will automatically route the traffic to the secondary region.

In both primary and secondary region following resources are deployed:

• Microsoft Foundry AI Services resource which includes Azure Content Understanding service (ACU) and Azure OpenAI Service (AOAI).

• AOAI contains a chat model (e.g. gpt-5.2) and an embedding model (e.g. text-embedding-3-large) deployed as required by ACU

Traffic Manager priority routing

The primary APIM external endpoint has priority 1, and the secondary has priority 2. Traffic Manager returns the primary while its probe is healthy and switches DNS answers when that endpoint is degraded.

APIM backend pools and circuit breakers

Each region has an APIM gateway. Its backend pool gives the same-region Content Understanding endpoint priority 1 and the peer endpoint priority 2.

pool: {
  services: [
    { id: primaryAcuBackend.id, priority: 1, weight: 100 },
    { id: secondaryAcuBackend.id, priority: 2, weight: 100 },
  ]
}

The circuit breaker covers HTTP status codes 429–599, including 429, all 5xx responses, and codes 430–499. In the sample configuration, it trips when failures reach 1% within five minutes and remains open for at least one minute. A Retry-After value can extend this duration.

Client authentication

The APIM API requires a subscription key. A scoped APIM subscription is created with tracing disabled. This client-facing key is separate from backend authentication: APIM uses its managed identity toward Foundry, so a caller never receives a Cognitive Services key. In a production API, subscription keys are usually combined with stronger caller identity and authorization policies appropriate to the application.

Analyzer resiliency

Analyzer definitions are application configuration. They must exist in both regions before backend failover is useful. Content Understanding does not emit a native analyzer-created Event Grid event, so the implementation uses a periodic scan and treats direct caller notifications as an optional latency optimization.

Broadly there are two common scenarios:

1. Your use case only needs a set of analyzers, and they will be updated infrequently

2. Your use case requires you to create/update/delete analyzers frequently

If your use case is the first one, then you shall disable the time-based trigger and only use the fast path programmatically. Whereas if your use case is the second one, then it’s recommended to consider the time-based automatic synchronization.

Timer, queues, and worker

1. A timer fires every minute and enqueues a compact scan of work items.

2. The reconciliation queue trigger builds the regional clients and processes either the full scan or one analyzer ID.

3. The worker takes the union of local, peer, and checkpointed analyzer IDs, so one-sided absence is visible.

4. The authority compares service-provided lastModifiedAt values and selects the strictly later analyzer as the source.

5. The destination is reread and verified before the local/peer timestamp pair is checkpointed.

Queue processing is serialized within a controller. host.json sets batchSize to 1 and newBatchThreshold to 0, while the Flex plan is capped at one instance. This creates a single writer for each regional checkpoint partition. More importantly, REPLICATION_AUTHORITY is true only on primary deployment. The standby can scan and decide, but decide returns NO_ACTION before any authorization, copy, or checkpoint write. Don’t enable both authorities at the same time to avoid creating a circular loop.

Optional fast path

A caller that directly creates or updates an analyzer can send the following message to the local analyzer-events queue after the API operation succeeds. The event trigger validates analyzerId and forwards the work to the same reconciliation queue as the timer.

{ 
  "analyzerId": "my-analyzer"
}

If the event is lost or never sent, the next periodic scan remains authoritative. Caller identities should receive Storage Queue Data Message Sender only on the appropriate regional event queue.

Regional state

Each Function has its own StorageV2 account with locally redundant storage. The storage account contains a deployment container, the reconciliation queue, the optional event queue, and the AnalyzerReplication table. A regional storage outage therefore pauses that controller but does not directly interrupt the end user request to APIM endpoint.

Timestamp based reconciliation

Last writer wins, under one authority

The service response is reduced to analyzerId, status, and the time zone aware lastModifiedAt value. If one ready analyzer is strictly newer, the authority copies it over the older analyzer. If one side is absent, the surviving ready analyzer is copied to the missing side. If either existing side is not ready, the authority waits rather than overwrite a creating, deleting, or failed resource.

The timestamp-pair checkpoint

A native copy can give the destination a later timestamp than the source even though the definitions are now equivalent. Without a loop guard, the next scan would treat the destination as a new edit and copy it back. The active authority’s checkpoint stores the exact local and peer timestamps observed after a successful copy. When that authority sees the pair again, the analyzer is synchronized. The analyzer’s ID is retained as a property, and its SHA-256 digest is used only as a Table Storage-safe row key.

Decision table

Why there is one authority

The authority is allowed to copy in either direction because ‘local’ describes where the Function runs, not which region owns the data. If the peer analyzer is later, the primary Function grants authorization at the peer source and starts the copy on its local destination. This asymmetry prevents two schedules from issuing opposite copies. Promotion is operational: set the former authority false before setting the standby true.

Promotion, and delete semantics

Promote the standby explicitly

The secondary Function already has the required Foundry and Storage role assignments, so promotion does not require redeploying RBAC. First set REPLICATION_AUTHORITY=false on the former authority and confirm that setting has taken effect. Then set REPLICATION_AUTHORITY=true on the standby. Enabling both at once violates the single-writer invariant and possibility of creating competing copies. Traffic Manager failover does not automatically promote the analyzer controller; request-plane and control-plane failover are separate runbooks.

Primary deletes use checkpoint evidence

The authority scans the union of local, peer, and checkpointed analyzer IDs. If primary is missing and secondary still has the exact peer modified time recorded in the last synchronized checkpoint, the authority rereads both regions and reruns the decision. If the evidence still holds, it deletes secondary idempotently, verifies both are absent, and replaces the checkpoint with a deletion tombstone. The tombstone suppresses stale secondary recreation.

An untracked secondary-only analyzer still seeds primary because absence alone is not deletion evidence. If secondary changes after synchronization and that change is visible at the pre-delete reread, the authority waits rather than delete it. Recreating a tombstoned analyzer in primary intentionally replaces secondary and starts a new synchronized lifecycle.

Failure Behavior and recovery expectations

Deploying the implementation

Example code which enables you to bootstrap this design is available in this GitHub repo: https://github.com/p-prakash/multi-region-ha-acu

Prerequisites

• Azure CLI with Bicep support and permission to deploy resources and role assignments.
• Azure Functions Core Tools v4 for remote Python build and publish.
• Two regions that support GPT-5.2 version 2025-12-11, text-embedding-3-large version 1, and Python 3.14 Flex Consumption.
• Sufficient model quota in both selected regions.
• Required resource providers registered in the subscription.

Review the parameters

main.bicepparam sets the sample regions, publisher metadata, model capacities, circuit-breaker threshold, reconciliation schedule, instance limit, and copy timeout. Foundry and APIM names are deterministic and globally unique by default. Confirm every value before deployment, especially the 200-unit sample model capacities and the 1 percent breaker threshold, and DNS/probe timing.

Validate without deploying

.\scripts\deploy-analyzer-replication.ps1  -ResourceGroupName <resource-group> -ValidateOnly

The PowerShell script resolves and verifies the Content Understanding Contributor role ID, compiles the parameter file, runs Azure resource-group validation, and runs what-if. ValidateOnly then exits without creating resources or publishing code. The Bash script provides the equivalent workflow on Linux and macOS.

Deploy infrastructure and code

.\scripts\deploy-analyzer-replication.ps1 -ResourceGroupName <resource-group>

Without ValidateOnly, the script creates or updates the Bicep resources, reads both Function App names from deployment outputs, and publishes the same Python package to each app using a remote build. The deployment is incremental at resource-group scope. Existing unrelated resources are not removed.

Cost and complexity tradeoffs

The design duplicates APIM, Foundry deployments, Function hosting, storage, and monitoring. It also creates operational work for quotas, alerts, and conflict handling. In return, request serving and analyzer configuration no longer depend on one regional stack. Teams should compare that benefit with their business RTO/RPO and the cost of operating two complete regions.

Conclusion

A resilient multi-region Content Understanding design must address both backend selection and configuration convergence. This implementation separates the synchronous request plane from an asynchronous, conflict-aware analyzer control plane. Each reachable APIM gateway provides local-first backend selection, while independent regional controllers reconcile analyzer definitions using verified native copy operations. APIM endpoint steering is managed through Traffic Manager.