Knowledge Base · Security & Hardening
Hardening live production servers without taking them down
A repeatable method for bringing internet-facing Linux estates to audit standard — built around the constraint that the applications must keep serving traffic the entire time.
Representative output. The finding list is the runbook's table of contents.
Why the usual advice fails here
Hardening guides assume a server you can reboot. Production doesn't offer that.
Most hardening material is written for a machine being built. The work I do is almost always the opposite: a system that is already live, already carrying revenue traffic, often deployed by someone else under time pressure, and now facing an external security audit.
That inverts the problem. The question stops being what is the correct configuration — that part is well documented — and becomes in what order can I reach the correct configuration without an outage, and how do I prove I got there. Three constraints drive every decision:
- No unplanned downtime. Most controls can be applied with a config reload or no service interruption at all. The few that genuinely require a process restart get batched into one announced window instead of being scattered across the work.
- No lockout. Access hardening can strand you on your own server. Every change to SSH, firewall, or sudo is sequenced add-verify-remove, with a second session held open and an out-of-band console confirmed working first.
- Evidence, not assertions. An auditor accepts a before/after artifact. Every control ends with a verification command whose output goes into the evidence pack.
Control domains
The checklist I work through, independent of distro or stack.
Remote access
Source-restricted SSH, key-only auth, no root login, forwarding disabled, session timeouts, brute-force banning.
Network exposure
Host firewall plus cloud security groups. Every listening port justified or bound to loopback.
Secrets at rest
Credentials out of compose files and scripts into mode-600 env files, rotated after exposure, mirrored to a password manager.
Privilege model
Sudo scoped and logged, service accounts separated from human ones, dormant accounts locked, root-equivalent group membership reviewed.
Transport security
Modern TLS only, key permissions, certificate lifecycle and renewal hooks, security response headers.
Web server posture
Version disclosure off, catch-all deny for unmatched hosts, stale virtual hosts archived, compression and caching verified.
Data services
Databases and caches bound to loopback or private interfaces, authentication mandatory, no trust-based access.
Container runtime
Ports published to loopback, no unnecessary privileged containers, daemon configured to survive its own restart.
Kernel & PAM
Core dump restrictions, address space randomisation, redirect handling, empty-password authentication removed.
Audit & logging
File integrity rules on every sensitive path, sudo session logging, log retention, time synchronisation.
Patch posture
Automatic security updates, pending-update visibility, documented patch cadence for the auditor.
Recovery
Verified backups with encryption at rest, restore path tested, process definitions captured as code.
Execution model
Ordered by blast radius. Downtime is concentrated, never distributed.
Establish a way back
Machine image snapshot, application-consistent database dumps, and a confirmed out-of-band console before touching access controls. If the recovery path is untested, the hardening hasn't started yet.
No impactPerimeter first
Cloud security groups and host firewall, narrowed to known administrative sources. This closes the findings an external scanner sees, and it closes them before the scanner runs.
No impactHost controls
SSH policy, brute-force protection, privilege model, kernel parameters, authentication modules, audit rules, time sync, patch automation. Additive or reload-only — running services are untouched.
No impactWeb tier
Centralised TLS policy, security headers, catch-all deny, virtual host review, compression and caching. Every change validated with a config test before reload; workers keep serving the old config until the new one passes.
No impactThe one window
Everything that genuinely requires a process restart — credential migration and rotation, service rebinding, runtime flags — batched into a single announced off-peak window with a per-service verification loop and a rollback for each step.
Seconds per serviceMake it durable
Backups with encrypted offsite storage, a scheduled self-check that re-verifies every control monthly and alerts on regression, and process definitions committed as code so the next deployment doesn't quietly undo the work.
No impactSelf-scan before they scan
Re-run discovery and diff it against the baseline. Probe the estate the way the auditor's tooling will. Assemble the evidence pack: before/after outputs, control table, and written risk acceptance for anything deliberately left open.
No impactField notes
Failure modes that cost me time, documented so they cost less next time.
A config test run as the wrong user lies to you
Certificate directories are readable only by root. Running a web server config test as an unprivileged user reports a certificate loading failure that does not exist, which looks exactly like a broken production config.
Always validate as root, and read the error class before reacting — Permission denied on a path is a different problem from a syntax error in a file.
Removing passwordless sudo can strand you
Cloud images ship a provisioning account with unrestricted passwordless sudo. Removing that grant is correct — but the account frequently has no password set at all, and root login over SSH is being disabled in the same pass. Both doors close together.
Set and store a password before touching the sudoers entry. Then check whether CI pipelines depend on the grant: if deployment jobs run privileged commands, an unscoped removal breaks deploys instead of protecting anything. Scope the grant to the exact commands, and treat a dedicated non-privileged runner account as the real fix.
The environment variable that looks applied but isn't
An application framework may honour a bind-address environment variable in one launch mode and silently ignore it in another. I have found servers carrying a loopback binding in their process configuration while still listening on every interface — the fix had been applied months earlier and never verified.
Never trust configuration as evidence of state. Confirm with a socket listing every time; the kernel is the only reliable witness.
Process managers don't reload arguments on restart
Editing a process manager's config file and issuing a restart reapplies the environment but keeps the original launch arguments. The change appears successful and nothing actually changed.
Delete and re-create the process from its definition file. Confirm with the socket listing, then persist the state so it survives a reboot.
Blocking HTTP methods breaks REST APIs
Restricting a virtual host to GET, POST and HEAD is standard hardening advice and standard breakage for any REST application. It fails progressively: the first restriction breaks record updates, the fix reveals deletes are broken too, and each discovery arrives as a production incident.
Method authorisation belongs in the application, not the proxy. Where an auditor expects to see method control, document that placement as a deliberate decision rather than shipping a fragile deny list.
Applications running from a CI working directory
Deployment pipelines that build in place leave live services running out of the runner's workspace. A workspace clean, a runner upgrade, or a concurrent job takes production down, and any server-side configuration fix is silently reverted on the next deploy.
Treat it as an architecture finding, not a config finding. Fixes belong in the repository; the deployment target belongs outside the build workspace.
Environment separation is discovered, not documented
Servers named for staging routinely carry production virtual hosts. The name is not evidence. Enumerate what is actually served and reconcile it with what the business believes is running there — that reconciliation has been a finding on its own more than once.
Map every listening port to its process and its public hostname before planning changes. Orphaned listeners and dead upstreams both surface in the same pass.
Tooling
Three scripts, one per stage. All read-only against production.
Auto-detecting inventory pass. Captures listening sockets, firewall state, access policy, privilege model, TLS material, container and process-manager topology, database bindings, and a secrets sweep — then auto-evaluates roughly thirty controls into a pass/warn/fail summary. Secret values are redacted at source so the output is safe to share. Nothing is written outside the report.
Probes every public hostname the way an audit scanner does: legacy CMS and admin paths, backup and version-control artifacts, traversal patterns, security headers, redirect behaviour, version disclosure, and unmatched-host handling. Benign requests only.
Scheduled re-verification of every control, emailing a report and exiting non-zero on regression so it can gate monitoring. This is what stops a hardened server from quietly drifting back over two quarters of deployments.
What the deliverable looks like
The runbook is the product, not the commands.
Each engagement produces a single document that a colleague could execute without me: a findings table with current state and target state per control, phased steps with the exact commands, a verification step and a rollback path for every change, an explicit note of which step carries the only downtime, and a decisions section listing the items that need business sign-off rather than a technical fix.
That last section matters more than it looks. Co-located environments, CI systems holding privileged access, and third-party workloads on shared infrastructure are not problems an engineer resolves alone — they are risk decisions. Surfacing them in writing, with options and trade-offs, is part of the work; a documented accepted risk passes an audit that an undocumented one fails.