(I’m using BitBucket so I have to containerize everything, rather an using git repos and Procfiles.)

I’ve made a Docker image that runs a simple HTTP server (“api-server”) and another that has a long-running script (“test-long-script”).

The api container looks like it’s running fine, but how do I get the API server to call the long-script container? To run it from the commandline, I’d use this command:

docker run --rm test-long-script:latest

I want people to be able to make HTTP requests to my api-server web service, like POST /encode-video or GET /do-something-slow?foo=bar, and have the request handler send the POSTdata / query params to the slow script:

docker run --rm test-long-script:latest --foo=bar

I can’t find any documentation on how to use workers, or how to communicate between components, besides “you can use the component name” and something about internal ports. What am I missing?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
2 answers

There’s no way for any part of the app to directly send information to the worker, so the worker process should continuously poll some kind of job queue from a database, and process jobs one by one. Does that sound correct?

Yup, that sounds correct.

It sounds like I can’t accomplish this with just a web service component and a worker component; at minimum, I need to add a database as well? And then I need to wrap my long-running-script in a service that runs continuously, and wrap that with some kind of job queue package, like “Bull” for Node.js or “Celery” for Python. Am I on the right track?

I think you might be able to do this without a database, it might make it easier to use one though. It really depends on the work you’re doing.

The worker process has to be a long-running script. In the past I’ve done this by having my code call an endpoint on my service every 30 seconds to see if any new jobs have been requested. If a job exists, it will process it and send the results back to the service.

If you want to to introduce a database you could have a table that has “tasks”, your worker would check for new “tasks” and if a new one exists, set it to “in-progress” and start doing the work, once complete saving the results back in the database and marking it as complete.

As a possible future feature, would you prefer to be able to call out to the worker privately? If so, what kind of behavior would you expect?

  • Thanks again for clarifying. I saw the description of worker components that said it was suitable for things like “video encoding” (both Heroku and DOAP say this) and I assumed it was something you could invoke on demand, I guess like a serverless function. I think I was way off with that assumption. But, as long as we’re talking feature requests, it would be awesome to abstract away the job queue part and have DOAP handle that!

    If I could do something like add_to_DO_AP_job_queue(<some details>) in my web service component, and then have the app platform pass those details to some arbitrary script on demand, that would be amazing. But I suppose that’s more of a dream for when DO tackles lambda functions :)

    In the mean time, I’ll settle for some more explicit documentation for the worker components. It’s probably fine for anyone switching from Heroku, but if you’ve never used a worker dyno before, you can only guess about them.

    • Thanks for the great feedback. I really appreciate it, I’ll take some notes and get some tickets created internally to bring to the team.

      I can understand the confusion, we don’t support serverless stuff… yet. That is a concept that the team is excited about but also requires a lot of planning and work to be executed correctly. Hopefully we have more good news coming on that topic in the future.

      Cheers!

👋 @fsduncan

Great question! Workers are not routable, meaning they cannot be accessed remotely, however, they can access services.

This means that your worker (eg. test-long-script) can reach out to a service within the network. For example, if your api component is called api-server you can make HTTP request to it at http://api-server. You can also create an environment variable that automatically injects the private URL to a local service like: API-SERVICE=${api-service.PRIVATE_URL}. Here’s more bind variable information.

Services also support internal ports within the app spec (this isn’t included in the UI yet). If your api-server component starts two servers, one can be mapped for public usage, and another can be used for internal traffic with your workers.

Another common pattern is to share a database between your service and workers, where jobs to be done are put in the database by the service and the worker picks them up when it checks for new work.

In conclusion, workers best work with a pattern of checking for work by calling an exposed (public or private) api endpoint from a service or checking a database for new work.

Here’s how internal ports are set in the app spec (though this app does not use them):

name: internal-ports
region: nyc
services:
- github:
    branch: master
    repo: digitalocean/sample-golang
  instance_count: 1
  instance_size_slug: basic-xxs
  internal_ports:
  - 8082
  - 4001
  name: web

Here’s more information on app specs

Hope this helps!

  • Thanks for the reply, I have since found out that worker processes need to be running continuously like a server. The docs made it sound like I could invoke a script (like for video encoding, image resizing, etc) that I assumed would start and end. I’ll have to re-think my whole setup if I want this to work in App Platform.

    But if I understand correctly:

    Workers are not routable, meaning they cannot be accessed remotely, however, they can access services.

    There’s no way for any part of the app to directly send information to the worker, so the worker process should continuously poll some kind of job queue from a database, and process jobs one by one. Does that sound correct?

    It sounds like I can’t accomplish this with just a web service component and a worker component; at minimum, I need to add a database as well? And then I need to wrap my long-running-script in a service that runs continuously, and wrap that with some kind of job queue package, like “Bull” for Node.js or “Celery” for Python. Am I on the right track?