Troubleshooting Playbook

Every failure encountered building this pipeline — symptom, root cause, fix.

Jenkins

Build queues forever — "Waiting for next available executor"

Started by an SCM change
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor

Cause. The built-in node's executor count was set to 0 (correct hardening) but no build agent exists to take the work.

Fix. Either provision an agent, or temporarily restore executors on the controller:

# Manage Jenkins → Nodes → Built-In Node → Configure
#   Number of executors: 2

Tool name not found

Cause. The string in tools { nodejs '...' } does not exactly match a configured NodeJS installation name.

Fix. Compare character by character against Manage Jenkins → Tools. Node.js 24.18.x and Node.js 24.18 are different tools.

UI elements missing / plugin install hangs behind a proxy

Cause. Nginx is not forwarding WebSocket upgrade headers, so live-updating UI components never connect. The page renders, so it presents as a UI bug rather than a proxy problem.

# http{} block
map $http_upgrade $connection_upgrade { default upgrade; '' close; }

# location block
proxy_http_version 1.1;
proxy_set_header Upgrade    $http_upgrade;
proxy_set_header Connection $connection_upgrade;

SCM checkout

Couldn't find remote ref

hudson.plugins.git.GitException: Command "git fetch --tags --force --progress --prune --
  origin +refs/heads/release/staging:refs/remotes/origin/release/staging" returned status code 128:
stderr: fatal: couldn't find remote ref refs/heads/release/staging

Cause. The branch configured in the job does not exist in the remote. Authentication succeeded — this is purely a missing branch.

Fix. Confirm the branch name in the SCM UI, then either point the job at an existing branch or create the missing one. Verify with:

git ls-remote --heads https://bitbucket.org/<workspace>/<repo>.git

Authentication fails on an SSH-form URL

Cause. A git@host:org/repo.git URL paired with a username/password credential.

Fix. Match the pair — either switch the URL to https:// and keep the app-password credential, or add an SSH private key credential. The HTTPS route is usually simpler to operate.

Node and npm

Node version mismatch every build

+ REQUIRED=v20.20.2
+ ACTUAL=20.20.2
+ [ 20.20.2 != v20.20.2 ]
Node version mismatch: required v20.20.2, got 20.20.2

Cause. .nvmrc contains a leading v. The check strips v from node -v output but compares against the file verbatim.

# In the repo, not on the server
echo "20.20.2" > .nvmrc     # no leading v

.nvmrc edited on the server has no effect

Cause. Jenkins checks out fresh from source control on every build. Files created in the deploy directory on the target host are invisible to the build.

Fix. Commit .nvmrc to the repository on the deploy branch. Anything the build must see lives in version control, without exception.

npm ci fails with ERESOLVE

npm error code ERESOLVE
npm error While resolving: eslint-plugin-import@2.21.2
npm error Found: eslint@7.1.0
npm error peer eslint@"^2 || ^3 || ^4 || ^5 || ^6 || ^7.2.0" from eslint-plugin-import@2.21.2
npm error Conflicting peer dependency: eslint@7.32.0

Cause. A genuine peer-dependency conflict in the project: a plugin requires eslint >= 7.2.0 while the project pins exactly 7.1.0.

Two legitimate fixes — choose with the dev team, not unilaterally:

1. If the team already installs locally with --legacy-peer-deps, match that in CI so both environments behave identically.
2. Better: raise the pinned version to satisfy the peer range and regenerate the lockfile.

SonarQube

sonar-scanner: command not found

Cause. The scanner's PATH entry was added to ~/.bashrc only. Jenkins sh steps run non-interactive shells, which never read it.

# Reproduce the failure exactly as Jenkins sees it
sudo -u jenkins bash -c 'sonar-scanner --version'

Fix. Export in both .bashrc and .profile, or — preferably — call the absolute path in the Jenkinsfile and remove the environment dependency entirely.

403 when submitting analysis

Cause. The token's owning account lacks Execute Analysis on the project, or the token has expired.

Fix. Project Settings → Permissions → tick Execute Analysis on the token owner's row. To identify the owner and check expiry: avatar → My Account → Security.

Quality gate passes on obviously poor code

Cause. First scan — no New Code baseline exists yet, so New Code conditions have nothing to evaluate. Additionally, if tests failed, no coverage report was produced and the coverage condition is inert.

What to do. Nothing technical — but say so out loud when presenting results. A green gate on scan #1 is not evidence of code health, and letting a stakeholder believe otherwise creates a problem later. The gate becomes meaningful from the second scan onward.

Coverage shows 0%

Cause. coverage/lcov.info was never written, usually because the test run failed. Or sonar.javascript.lcov.reportPaths points at the wrong path.

# Confirm the file exists after the test stage
ls -la coverage/lcov.info

# The scanner log states explicitly whether it imported coverage
# Look for a line mentioning the LCOV report path

Analysis is extremely slow

A ~500-file TypeScript project taking ~11 minutes and producing a 10 MB report is normal. If it is much worse, check:

• sonar.exclusions covers node_modules, dist, and coverage
• Compute Engine heap (sonar.ce.javaOpts) is not starved
• Files reported as analysed "without type information" — extend the tsconfig includes

SSH and deployment

Permission denied (publickey)

Work the causes in this order — cheapest first:

sshd[…]: User ubuntu from 10.10.1.215 not allowed because not listed in AllowUsers

Cause. An AllowUsers directive restricts SSH to an explicit list that omits the CI account. Keys and permissions were never the problem.

# Add the account to the existing list, then
sudo sshd -t                 # validate config BEFORE reloading
sudo systemctl reload ssh

sudo asks for a password in the pipeline

Cause. No sudoers.d rule for that service user, or the file has wrong permissions.

echo "ubuntu ALL=(<user>) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/ubuntu-to-<user>
sudo chmod 440 /etc/sudoers.d/ubuntu-to-<user>
sudo visudo -c

# Verify without hanging on a prompt
sudo -n -u <user> whoami

node: command not found on the deploy target

Cause. nvm is a shell function loaded from .bashrc; the non-interactive remote shell never sources it.

# Every remote block must source nvm explicitly, first
sudo -u kfc bash -c '
  source /home/kfc/.nvm/nvm.sh
  node -v
'

Cannot list a service user's home as the CI account

ls: cannot open directory '/home/dom/': Permission denied

Not a problem. The CI account does not need to read those files as itself — it needs to become the service user and act there. Confirm what actually matters:

sudo -u dom bash -c 'source ~/.nvm/nvm.sh && node -v && pm2 list'

Adding the CI account to each service group would make manual inspection easier at the cost of broader standing read access. The sudo path is sufficient and tighter.

Health check fails immediately after a successful restart

Cause. PM2 returns as soon as the process spawns, not when the app is ready to serve.

Fix. Retry loop with a delay, as in Part 5. If the endpoint 404s, confirm the route exists at all:

curl -i http://localhost:4016/health   # 404 → no such route
curl -i http://localhost:4016/         # 200 → app is up

Where no health route exists, fall back to accepting a 200 on / — and record the missing endpoint as a gap for the dev team rather than treating the weaker check as equivalent.

Email

553 relay error / sender address rejected

Cause. The envelope sender does not match the authenticated SMTP account.

Fix. Manage Jenkins → System → Jenkins Location → System Admin e-mail address must be identical to the SMTP username.

Microsoft 365 SMTP authentication fails

Cause. Basic SMTP AUTH is disabled tenant-wide by default on modern M365 tenants.

Fix. A tenant admin must enable SMTP AUTH for that mailbox, or use a provider that supports app passwords. No Jenkins-side configuration can work around it.

Console links in emails are broken

Cause. Jenkins URL is unset or wrong, so ${env.BUILD_URL} renders against localhost.

Fix. Manage Jenkins → System → Jenkins Location → Jenkins URL — set to the externally reachable URL.

Diagnostic order

When a build fails and the cause is not obvious, work outward from the smallest unit:

Useful one-liners

# Reproduce a step in Jenkins' actual shell environment
sudo -u jenkins bash -c 'cd /var/lib/jenkins/workspace/<job> && npm ci'

# Full deploy chain in one command
sudo -u jenkins ssh -i /var/lib/jenkins/.ssh/stg_deploy_key ubuntu@<host> \
  'sudo -u <svc> bash -c "source /home/<svc>/.nvm/nvm.sh && node -v && pm2 list"'

# Watch SSH auth failures live
sudo tail -f /var/log/auth.log

# Confirm every sudoers file parses
sudo visudo -c

# List branches on a remote without cloning
git ls-remote --heads <url>

# SonarQube health
curl -s -u <token>: http://<sonar>:9000/api/system/status

# What is actually listening on a port
ss -lntp | grep <port>

# App startup log — often the only place the port is stated
sudo -u <svc> bash -c 'source ~/.nvm/nvm.sh && pm2 logs <app> --lines 50 --nostream'

Patterns worth internalising