qasim@wiki:~$ ▍
Wiki Home Portfolio

Incident Postmortem — Apex Vhost Fallthrough

What happened

A client's apex domain briefly served a different, unrelated client's website on the shared MainDomains redirect server.

Root cause

  1. A new vhost was created for the affected domain by copying an existing client's .conf file and editing the domain-specific lines.
  2. The final Redirect permanent / https://www.<domain>/ line was accidentally omitted during the edit.
  3. Because sites-enabled/ loads alphabetically and no explicit catch-all existed, the alphabetically-first vhost was Apache's implicit default for unmatched requests. The new domain's redirect logic being incomplete meant its traffic wasn't confidently routed to itself, and the affected window exposed the fallthrough to that default vhost — a different client's site.

Contributing factors

  • Copying a live client's vhost as a template — brings along that client's cert paths and redirect target, easy to miss one line.
  • No catch-all/default-deny vhost existed — any config gap had a real, unrelated client's site as its fallback instead of a safe error.
  • No automated post-deploy check to verify a new domain actually redirects to itself before considering onboarding "done."

Fix applied

  1. Added 000-catchall.conf — loads first for both *:80 and *:443, returns 403 Require all denied for any unmatched Host/SNI. Confirmed via apache2ctl -S as the default server ahead of all client vhosts.
  2. Verified with curl --resolve across all existing client domains — all redirect correctly to their own www. subdomain.
  3. Verified an unmapped test domain now returns 403 instead of another client's content.
  4. Replaced the "copy an existing client's vhost" workflow with a clean domain-only template + certbot-generated SSL block — see Apache Config Templates.

Verification evidence

client-a.com.pk       -> 301  Location: https://www.client-a.com.pk/
client-b.com.pk       -> 301  Location: https://www.client-b.com.pk/
client-c.co.za        -> 301  Location: https://www.client-c.co.za/
client-d.com.sa       -> 301  Location: https://www.client-d.com.sa/
client-e.com          -> 301  Location: https://www.client-e.com/
client-f.et           -> 301  Location: https://www.client-f.et/
client-promo-site.com -> 301  Location: https://www.client-promo-site.com/

unmapped-test-domain.com -> 403

Follow-up items identified

  • A cert-path copy/paste mismatch was also found on an unrelated existing vhost (using another domain's Let's Encrypt cert path) — same root cause class, not yet triggered. To be corrected using the same template + certbot approach.
  • Long-term architecture options evaluated to reduce reliance on the shared server entirely — see Static-IP Options Compared.

Prevention going forward

See the Onboarding Checklist — every new apex domain must go through this before being considered live.

← Back to the DNS series index