Multi-Client Deploy Architecture

Deploying to a dozen isolated service accounts on one shared host — per-user nvm, per-user PM2, least-privilege sudo, no root anywhere.

The hosting model

One staging host runs API servers for many clients. Each client is isolated by Linux user account:

Each service user owns its code, its own ~/.nvm, and its own PM2 daemon. Nothing is shared. A client on Node 20 and a client on Node 24 coexist without conflict, and a runaway process affects one account rather than the host.

Why not deploy as root

The deploy chain

Setup — per client

echo "ubuntu ALL=(kfc) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/ubuntu-to-kfc
sudo chmod 440 /etc/sudoers.d/ubuntu-to-kfc

# ALWAYS validate before closing the session
sudo visudo -c

# -n = non-interactive: fail immediately rather than prompting
sudo -n -u kfc whoami
# kfc

sudo -u kfc bash -c '
  source /home/kfc/.nvm/nvm.sh
  node -v
  npm -v
  pm2 list
'

# On the Jenkins controller, as the jenkins user
sudo -u jenkins ssh-keygen -t ed25519 -C "jenkins-deploy-stg" \
  -f /var/lib/jenkins/.ssh/stg_deploy_key -N ""

sudo cat /var/lib/jenkins/.ssh/stg_deploy_key.pub

# On the staging host, as ubuntu
mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "<paste public key>" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
chmod 755 ~            # home must NOT be group-writable
chown -R ubuntu:ubuntu ~/.ssh

sudo -u jenkins ssh -i /var/lib/jenkins/.ssh/stg_deploy_key \
  -o StrictHostKeyChecking=no ubuntu@10.10.1.223 \
  'sudo -u kfc bash -c "source /home/kfc/.nvm/nvm.sh && node -v && pm2 list"'

The deploy stage

stage('Deploy to Staging') {
  when { expression { currentBuild.currentResult == 'SUCCESS' } }
  steps {
    sshagent(['stg-server-ssh-key']) {
      sh """
        ssh -o StrictHostKeyChecking=no ${DEPLOY_USER}@${DEPLOY_HOST} \
          'sudo -u ${SERVICE_USER} bash -c "
            source ${NVM_DIR}/nvm.sh
            set -euo pipefail

            cd ${DEPLOY_PATH}
            git fetch --all
            git reset --hard origin/${DEPLOY_BRANCH}

            npm ci
            npm run build
            pm2 restart ${PM2_APP_NAME}

            for i in 1 2 3 4 5; do
              if curl -s -o /dev/null -w '%{http_code}' http://localhost:${HEALTH_PORT}/ | grep -q 200; then
                echo Service responding on port ${HEALTH_PORT}.
                exit 0
              fi
              sleep 3
            done
            echo Health check failed after 5 attempts.
            exit 1
          "'
      """
    }
  }
}

Quoting — three nested layers

set -euo pipefail

• -e — exit on any command failure. Without it, a failed npm ci is followed cheerfully by pm2 restart, restarting a half-built application.
• -u — error on undefined variables, catching typos rather than expanding them to empty strings.
• -o pipefail — a pipeline fails if any element fails, not just the last one.

git reset --hard rather than git pull

A deploy target should be a exact mirror of the branch, not a merge participant. git pull can produce merge conflicts or merge commits on a server nobody is watching. git fetch --all && git reset --hard origin/<branch> guarantees the working tree matches origin exactly, every time.

Health check

The retry loop with sleep 3 exists because PM2 returns from restart as soon as the process is spawned, not when the application is ready to serve. Without the loop, the check runs against a still-booting app and fails spuriously.

# Preferred — a real health endpoint
curl -sf http://localhost:4016/health

# Fallback when the app has no /health route:
# accept any HTTP 200 from the root path
curl -s -o /dev/null -w '%{http_code}' http://localhost:4016/ | grep -q 200

Onboarding a new client

With the pattern established, each additional client is roughly thirty minutes.

Discovering the values

# Node version, PM2 processes, and top-level dirs for each service user
for user in kfc dom br phpak phqatar; do
  echo "=== $user ==="
  sudo -u $user bash -c 'source ~/.nvm/nvm.sh 2>/dev/null && node -v; pm2 list 2>/dev/null | grep -E "online|stopped"'
  echo
done

# Which port a process listens on — read it from the app's own startup log
sudo -u kfc bash -c 'source ~/.nvm/nvm.sh && pm2 logs kfc-server --lines 50 --nostream' | grep -i port

# Or from the socket table
ss -lntp | grep 4016

Sudoers policy

Worth writing down, because it is the thing an auditor will ask about:

A one-file-per-user layout means ls /etc/sudoers.d/ is a complete access inventory, and revoking one client's access is a single rm with no risk of breaking the others. If grants are ever made ahead of the pipelines that justify them, record the date and business reason in the change log — an unexplained gap between "users with access" and "clients with pipelines" is exactly what audit findings are made of.

Verification checklist