SSH Connection refused after multiple previous successful connections

I am running a Wordpress One-Click install and I am using GitLab CI/CD to deploy to my droplet. My script is successfully connecting via SSH creating some folders, then using SCP to copy some files to the server successfully, then after a few SSH connections the droplet refuses the SSH connection and the job fails. Here is a chunk of the log from GitLab:

$ ssh-add <(echo "$SSH_PRIVATE_KEY")
Identity added: /dev/fd/63 (rsa w/o comment)
$ ssh "$PROD_SERVER_USER"@"$PROD_SERVER_ADDRESS" "mkdir -p \"$PROD_DIRECTORY\"/_tmp && mkdir -p \"$PROD_DIRECTORY\"/_old && mkdir -p \"$PROD_DIRECTORY\"/build"
Warning: Permanently added '#######' (ECDSA) to the list of known hosts.
$ scp "$BUILD_DIRECTORY"/build/* "$PROD_SERVER_USER"@"$PROD_SERVER_ADDRESS":"$PROD_DIRECTORY"/_tmp
$ ssh "$PROD_SERVER_USER"@"$PROD_SERVER_ADDRESS" "mv \"$PROD_DIRECTORY\"/build \"$PROD_DIRECTORY\"/_old && mv \"$PROD_DIRECTORY\"/_tmp \"$PROD_DIRECTORY\"/build"
ssh: connect to host ####### port 22: Connection refused
ERROR: Job failed: exit code 1

Does anyone know why it is suddenly refusing to connect?

Thanks