cURL with cron job not working

April 24, 2018 1.5k views
Nginx Ubuntu 16.04

Hello

I've tried to create a cron job with a curl to a specific page on my website. On that page, when loaded, it's sending out an e-mail to some specific people. I've tested the script, and it's working just as intended.

What I've done to enable my cron job is:

  • Log in to my server
  • Run: sudo crontab -e
  • Added: * 17 * * * curl http://url.com/cron/email to the file and saved

What I tried to here was to make a cron job calling the url once a day at 5pm. For some reason this is never happening - not even at another time of the day, it's like it's never run.

What am I doing wrong?

3 comments
  • Here are a couple suggestions:

    1. Use absolute paths in cron: cron jobs are executed by crond in a minimal environment without a shell (unless you force one to be created), the regular shell setup (including things like your PATH variable that helps map curl to /usr/bin/curl) never happens. More reference

    2. Do you intend to log the entire response HTML? I think right now, you are saving everything that comes back from http://url.com/cron/email in /var/log/cron. If your response is just a short plaintext thing that should be fine, but if it is returning html there's going to be a lot of junk in your logs. Add the --silent option to avoid that.

    3. Pipe output to dev/null The Cron daemon that runs your command wants to send the output to the email address associated with your account, but I am guessing you don't have email configured and you dont need that so add >/dev/null 2>&1 to the end of the command to pipe all output to /dev/null (same as ignoring it)

    So with all that, try:

    * 17 * * * /usr/bin/curl --silent http://url.com/cron/email >/dev/null 2>&1
    
  • Hey @aha

    Thank you for answering my question.

    Unfortunately it's not working. I've tried to add the line you've recommended, and made it run every minute * * * * * just for testing purpose, but nothing seems to work.

    I have no intention of logging anything. I just want to run the URL once a day at 5pm - nothing else. Since on that webpage it's doing some logic. Maybe this is not even possible? To run a webpage like if you visited it in the browser?

  • One more tip for debugging: to see the logged results of the cron task you set up to run every minute, use grep CRON /var/log/syslog to print out all the lines related to cron. You should see something like:

    CRON[2179]: (root) CMD (/usr/bin/curl http://url.com/cron/email)
    

    However - you are right, there is a big difference between running curl http://url.com/cron/email and visiting http://url.com/cron/email in the browser:

    When you visit http://url.com/cron/email in the browser, if it returns an HTML doctype, the browser will: A) Download the HTML, B) Download any Javascript, CSS, Images referenced in the HTML, C)Render the page, D)Execute all Javascript.

    When you curl http://url.com/cron/email, it is just A) Downloading the HTML

    So if the thing that causes the email send is executed via javascript on the page, curling it will not help.

2 Answers

This question was answered by @aha:

One more tip for debugging: to see the logged results of the cron task you set up to run every minute, use grep CRON /var/log/syslog to print out all the lines related to cron. You should see something like:

CRON[2179]: (root) CMD (/usr/bin/curl http://url.com/cron/email)

However - you are right, there is a big difference between running curl http://url.com/cron/email and visiting http://url.com/cron/email in the browser:

When you visit http://url.com/cron/email in the browser, if it returns an HTML doctype, the browser will: A) Download the HTML, B) Download any Javascript, CSS, Images referenced in the HTML, C)Render the page, D)Execute all Javascript.

When you curl http://url.com/cron/email, it is just A) Downloading the HTML

So if the thing that causes the email send is executed via javascript on the page, curling it will not help.

View the original comment

If you're looking for a simpler cron job solution, you might consider webcron services like easycron.com which will load a specific URL at a given time.

Have another answer? Share your knowledge.