SSH stops working when I run docker compose up with DOCKER_HOST

I am trying to use docker and docker-compose with DOCKER_HOST set to ssh://user@my_droplet.

Ssh works perfectly, and most docker compose commands seem to work fine except for docker compose up and down. They run for a second or two then fail with error “connect to host XXX.XXX.XXX.XXX port 22: Connection refused”

error during connect: Get "http://docker/v1.41/images/redis:alpine/json": command [ssh -l user -- XXX.XXX.XXX.XXX docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host XXX.XXX.XXX.XXX port 22: Connection refused

After that ssh stops working entirely. For a couple minutes I cannot connect to ssh at all, and docker commands that worked previously start failing with the same error (connection refused). Then ssh starts working again.

I tried to google this issue and it seems that for other people docker either works over ssh or not. In my case it works without any problem until I run docker compose up. And even docker compose up sometimes manages to do something before failing.