Active Directory · Discovery · Change safety
Retiring a domain controller that held every forest role
One on-premises domain controller had collected every forest-critical role over about a decade. It had to be retired onto two AWS-hosted controllers with no maintenance window, no reliable dependency inventory, and no contact with end users allowed during discovery. Everything had to be proven from the servers themselves.
At a glance
How to read the tagsStarting point
The server held all five FSMO roles, the forest’s Enterprise Root CA, DHCP for the main site, 28 shared print queues, DNS, software deployment packages, and logon scripts. The job was to retire it from a live multi-site environment.
Discovery
The AD subnet table covered only a fraction of the routed networks, and everything still worked, which hid the problem. Clients with no site don’t fail. They authenticate to whichever controller answers first, which is how a decommission target ends up carrying authentication load nobody knows about.
A sweep of security-log logon events and netlogon logs surfaced more than 480,000 site-less authentication events in the sampled window, and it found what no inventory contained:
- The surviving cloud controller used the decommission target as its secondary resolver.
- A cloud directory connector still listed a long-demoted controller as one of its two DNS servers.
- Three cloud member servers had the on-premises controller hardcoded as static DNS, so they authenticated across the WAN while two healthy controllers sat in the same VPC.
- Five logon scripts referenced server-specific paths that would break at power-off.
- A stale DNS record and a dead conditional forwarder were still replicating forest-wide.
Twelve hard dependencies in all, any of which could have caused an outage at power-off.
DNS and site topology
DNS was corrected in the VPC DHCP option set and the AD subnet table rather than by pinning static resolvers on each member server. Hardcoded static DNS is how the hidden dependency was created in the first place, and pinning resolvers again would have set up the same problem for the next migration.
Subnets went in smallest first as a canary, with a soak between each and the busiest network last. The main-site subnet moved on a morning when someone was watching, because moving it is effectively a dry run of the power-off.
Key decision
Transfer the FSMO roles early, or wait for the power-off window?
The decommission was on hold pending an unrelated performance investigation, with all five roles still on the box.
Why
A graceful transfer between healthy controllers is reversible with one command. Powering off a role holder first forces a seize, and a seized controller can never safely come back. Waiting would have given up the rollback path for no benefit.
Outcome
The roles moved online with replication clean on both sides. The performance problem was traced to the network, not directory topology. I also recommended holding the decommission date five weeks past its original target once discovery surfaced the twelve dependencies. Powering off a role holder with open dependencies was the worse trade.
Certificate authority
The root CA moved first, in one evening window, by the same-name, same-key method. Both key generations were preserved, every issued certificate stayed valid, and nothing was reissued. Validation was a full chain check with revocation fetch, certutil -verify -urlfetch, returning clean on every link.
Print migration
Twenty-eight queues and 24 mappings spread across three Group Policy delivery mechanisms moved in three stages: additive, a soak, and a subtractive stage gated on written sign-off. No maintenance window was taken. The phased approach came from a change of plan partway through. It took more work, and it was the safer design.
What broke
Printer GPO left disabled after staging
The first acceptance round delivered no printers after a refresh, a re-login, or a reboot. The policy had been fully disabled during staging and never re-enabled.
FixAdded an explicit all-settings-enabled check to the runbook as a hard gate.
CA restore failed on a non-empty target
The configuration wizard had quietly created a starter database before the restore ran.
FixVerify the target is empty before restoring into it.
Escaping error in a registry write
Batch-file percent escaping was used at an interactive prompt, where it doesn’t collapse.
FixEach command is written for the shell it will actually run in. The handoff shipped with an error-code decoder.
Cleanup script with false negatives
It filtered on the wrong property for local queries and mixed machine-scope and user-scope printer connections, which need separate execution contexts.
FixRewritten as dry-run by default, idempotent, context-aware, and CSV-logged, and reviewed like any other change.
Multi-line commands broken on paste
Paste-order concatenation in the target consoles mangled multi-line commands.
FixEvery command in every later runbook is written as a single line.
Lessons
- Disable before delete at every destructive step. It turns rollback into a sub-minute operation instead of a restore.
- Cloud directory services carry DNS settings independent of the VPC DHCP option set. Fixing one doesn’t fix the other, and the second failure only shows up at cutover.
- Burstable instance classes disqualify a host from time-authority and CA roles. CPU credit exhaustion on the forest time root is a real failure mode.
- Never apply a preference deletion and unlink its policy in the same action. Unlinking kills the deletions before they run and orphans every mapping they were meant to remove.
Stack
- AD DS
- AD Sites and Services
- AD CS
- Group Policy (Deployed Printers, Preferences, WMI and security filtering)
- DFSR
- EC2
- VPC DHCP option sets
- Transit Gateway
- AWS Directory Service (AD Connector)
- CloudWatch Logs Insights
- VPC Flow Logs
- PowerShell
- ntdsutil
- repadmin
- dcdiag
- nltest
- certutil
Organizations, people, dates, and identifying environment details are omitted or generalized. Each figure carries a tag for the kind of evidence behind it.