Isolating a VoIP datacenter in broad daylight, without dropping a call

An operator runs two datacenters. One day you need to work on one of them: an SBC reboot, a firmware upgrade, a twenty-minute maintenance, in the middle of the day, while customers are on the phone. That operation did not exist before. You waited for the 2 a.m. window, or you took the risk of dropping calls.

I built the missing lever with n8n and DNS. A short intervention, in daylight, without a single customer losing a call.

The problem: two datacenters, one shared resolution

Two datacenters, DC-A and DC-B. Each fronts the Microsoft Teams Direct Routing service with an access SBC, and customers resolve teams.example.net (plus a <pbx_id>.teams.example.net record per customer) to A records pointing at both DCs. Taking DC-A out of service is a clean cycle: stop new sessions landing on it, let the calls in progress drain, work, then put it back.

By hand in the DNS console, record by record, under time pressure, that is how you cut a customer by mistake.

Why DNS, and not the SBC

I am API-first. The lever I trust is the one with a clean, deterministic, reversible API behind it. The DNS provider has exactly that. Manipulating the SBC directly, refusing new calls on DC-A at the equipment, I judged less reliable and more invasive.

There is also a question of responsibility. Sensitive data, and the operation that touches it, belong with a professional whose job this is, on shared, hardened infrastructure, not on a single, non-shared appliance that lacks the right reflexes. The DNS provider is built for that. The SBC is not.

So the drain happens at the DNS layer. You pull DC-A's IP out of resolution. New calls go to DC-B. Calls in progress keep flowing on DC-A until they hang up. That is connection draining, the move a load balancer makes on an instance it removes: cut the new, let the old finish. An anycast network does the same by withdrawing a BGP route. I do it at the layer I drive by API, DNS.

The model is two moves: delete-by-content, then mirror-restore. To isolate, I do not touch records by name. There is the apex, the wildcard, and one record per customer. I delete every A record whose content is DC-A's IP. To restore, I do not replay a snapshot. I recreate by mirroring: for every A record that currently points at DC-B, the survivor, I add the same name pointing at DC-A.

Mirror-restore is idempotent. Rerun it and it converges to the same state, which matters the day a restore gets replayed after a hiccup. The price is deliberate: it trusts the surviving DC as the reference, so it aligns DC-A on DC-B's state, drift included. As long as the two DCs are meant to be symmetric, that is exactly what you want.

# Isolate: list every A record pointing at DC-A's IP
GET /client/v4/zones/{zone_id}/dns_records?type=A&content=<DC_A_IP>&per_page=1000
Authorization: Bearer $CF_TOKEN

# Then, for each record returned, delete it by id
DELETE /client/v4/zones/{zone_id}/dns_records/{record_id}
Authorization: Bearer $CF_TOKEN

The restore never reads a backup. It mirrors the surviving DC:

// n8n Code Node - Restore: recreate DC-A's A records mirroring DC-B
// Input: the A records currently pointing at the survivor IP (DC-B).
const survivors = $input.all();
const dcAip = $('Normalize & Config').first().json.dc.public_ip; // DC-A's IP

return survivors.map((r) => ({
  json: {
    type: "A",
    name: r.json.name,   // same name: apex, wildcard, <pbx_id>.teams...
    content: dcAip,      // repoint this name at DC-A
    ttl: 120,            // low standing TTL, see below
    proxied: false,
  },
}));

The drain, and the decision that makes operators frown

Pulling the IP stops new resolutions. It does not hang up the calls already established on DC-A's SBC. Before I intervene, I need the number of active sessions still on that SBC. The SBC exposes an active-call gauge over RESTCONF. The workflow reads it and shows it to me.

// n8n - Helper "Get SBC Session Count": one SBC's active-call count.
// The HTTP node runs onError=continue: a wrong RESTCONF path
// degrades to "unavailable", it never breaks isolation.
const res = $input.first().json;
if (!res || res.error) {
  return [{ json: { ok: false, active_calls: null, reason: "SBC unreachable" } }];
}
// Parse the gauge the SBC exposes (adapt to the model and version).
const active = Number(res?.callCountStatus?.activeCalls ?? 0);
return [{ json: { ok: true, active_calls: active } }];

Here is the decision that makes an operator frown. The workflow is report-only. It shows me the count and lets me judge: low enough, I go. It does not block until the count hits zero.

That is deliberate, but it is a stage, not an ideal. The target is an automatic hard gate that drains to zero before cutover. Today that gate stays imagined: it waits on a real answer, queried against a live FQDN, the proof the drain signal is trustworthy in production. And it waits on a second, structural thing: control of every path into DC-A. The Teams zone is under my managed DNS. The SIP-trunk zones are not. DC-A keeps carrying trunk traffic after I pull its Teams records, so a "zero Teams sessions" gate would lie about the datacenter's real state.

You do not gate on a signal you do not control end to end. Report-only is the honest posture until every ingress sits under the same DNS and the count is trustworthy everywhere.

Block hard, or only warn

Two guardrails, two postures.

A hard block on double isolation: if DC-A is already isolated, the workflow refuses to isolate DC-B. Isolating both is a self-inflicted outage, an always-wrong state. The automation says no, full stop.

A plain warning, not a block, when I isolate the datacenter that hosts the only AS (Application Server) cluster. Pulling that DC's access SBC does not protect the AS, and the workflow cannot isolate an AS that has no twin. That is a context-dependent limit, to weigh against the maintenance need. The automation warns and lets me decide.

My rule: block hard on states that are always wrong, warn on limits that depend on context.

The real trap: the TTL

The prerequisite that makes or breaks all of it is the standing TTL. A DNS drain is only as fast as the TTL allows. At the default 3600, pulling DC-A's records still leaves resolvers and clients hitting it for up to an hour. The drain is useless for the whole length of your maintenance.

So the records have to sit at a low standing TTL, around 120s, all the time, not lowered the morning of. I set 120s as a permanent infra prerequisite, and the workflow recreates at 120 on restore. Even then, the TTL is a ceiling, not a promise: RFC 8767 lets a resolver serve an already-expired record to ride out an outage, and client caches are not always polite. So the drain is never instant or total. Plan for a couple of minutes of tail, and never confuse "records pulled" with "datacenter silent".

When it is the right tool, when it is a trap

DNS-based drain is the right tool when two conditions hold: ingress is centralized under one DNS, and that DNS is driven by API. Then it is deterministic and reversible, and it costs nothing on the equipment itself.

It is a trap when you need immediate effect. The propagation delay, the TTL plus resolver and client caching, means there is always a lag before the DC actually goes quiet. If your operation cannot tolerate that lag, DNS is the wrong lever.

So there is an envelope to respect. DNS drain is for an intervention the surviving DC absorbs and that tolerates a few minutes of tail: a config change, a firmware upgrade on an N+1 pair. It is not a substitute for a hard cutover on an operation that would drop the sessions still established. Those need the real zero.

What is left

This workflow turns a capability that did not exist, working on a live datacenter in daylight, into a two-click form. Next comes the hard drain-to-zero gate. It is not a matter of more code: it is the day every ingress, Teams and trunks alike, sits under the same DNS I drive, when "zero sessions" becomes a signal I own end to end. Until then, the only actor who sees what the automation cannot is the human. Report-only plus judgment is the honest version, not the lazy one.

If you run more than one datacenter and you still wait for the night window, the missing piece is probably not more infrastructure. It is a centralized DNS you can drive by API. I described the underlying automation building block in my article on automating VoIP operations with n8n.

Notes and sources

• Cloudflare, DNS Records API (list, create, delete per zone), developer documentation.
• n8n, HTTP Request and Code nodes, onError handling.
• RFC 8767 (serve-stale) and RFC 1035, DNS TTL and caching: a resolver may serve an expired record, and a drain is never instant even below the TTL.
• Connection draining and deregistration delay (load-balancer layer): the same move, cut the new and let the old finish, one layer up.

A datacenter maintenance that has to run in daylight, a Teams Direct Routing cutover to automate, an SBC drain to make reliable? Get in touch.