Can't get clock to stay in sync using NTP
I'm having some serious issues getting the clock on my droplet to sync decently (I wanted to use my droplet to provide NTP service as part of the pool, but it's annoying that I can't provide decently accurate time). ntpd is synced to good time servers, and while initially the clock will be very good (getting down to 1ms or less offset), over time it gets out of sync to the tune of 20-50ms. I know NTP will keep the clock in sync better than this, and I've heard of people running NTP servers on DO droplets with much better results than I'm seeing. So it has to be something in my config, but damned if I know what.
Any ideas on what to try? I'm running one of the 512 MB droplets, currently using Ubuntu 16.04 (have also tried CentOS 7.3, with similar issues). Using a pretty standard ntpd config (I am installing/configuring it through Puppet using the puppetlabs module, all I am tweaking is to specify the upstream servers to sync to).