Production CI/CD Deploy Failure: SSH Credential Boundary
Diagnosed a production deployment failure where Docker image pulls were denied on the remote VM -- traced the root cause to GitHub Actions credentials not being forwarded across an SSH boundary, fixed with a single line.
Production deploy failed mid-pipeline. GitHub Actions had built the Docker image, pushed it to GHCR successfully — the push step was green. But the next step, which SSHed into the production VM to pull and deploy, returned error response from daemon: denied: denied. The image was in the registry. The VM couldn’t get it.
The agent’s first diagnostic move was to read the full workflow file and trace exactly what was happening at the boundary between the runner and the remote machine. This is where the bug became clear.
The docker/login-action step authenticates with GHCR by writing credentials to ~/.docker/config.json on the GitHub Actions runner. That authentication is scoped to the runner’s filesystem. The appleboy/ssh-action step that follows it opens an SSH connection to the production VM and executes commands there. Those commands run in a shell on the VM — a completely different machine with a completely different filesystem. The ~/.docker/config.json that docker/login-action wrote doesn’t exist on the VM. So when docker compose pull runs on the VM, Docker has no credentials for GHCR and the pull is denied.
The first attempted fix made a common mistake. It set GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }} as an environment variable in the SSH action, then used echo $GHCR_TOKEN | docker login ghcr.io inside the SSH command block. This doesn’t work. Environment variables defined in a GitHub Actions step aren’t forwarded through SSH. They exist in the runner’s process environment; the SSH session is a separate process on a separate host with its own environment. $GHCR_TOKEN inside the SSH block refers to a variable that doesn’t exist on the remote machine — it expands to an empty string and docker login authenticates with nothing.
The correct fix is a single line that understands GitHub Actions’ template expansion timing. Inside the SSH command block:
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdinThe ${{ secrets.GITHUB_TOKEN }} and ${{ github.actor }} expressions are GitHub Actions template syntax, not shell variable syntax. They’re expanded by the GitHub Actions runner before the SSH command is assembled — the token value is literally interpolated into the command string before it’s sent over the wire. By the time the SSH session receives the command, it contains the actual token value as a string literal, not a variable reference. No forwarding needed. The VM logs into GHCR with a valid credential, docker compose pull succeeds, and the deploy completes.
One line changed, three lines added to the workflow. The conceptual gap — understanding that ${{ }} expansion happens before SSH while $VAR evaluation happens after — is the kind of thing that turns a two-hour outage into a thirty-minute fix.