The issue with custom name severs is that you need to, ideally, setup multiple IP's -- one for each of the name servers you create. Since DigitalOcean only technically allows a single IP per Droplet, this isn't possible. The purpose of using multiple IP's is for redundancy -- should one of the name servers go down, the other can pick up the requests.
That being said, custom name servers require registration with your registrar.
You can set your domains DNS to any name servers(s) you wish, valid or not, and the registrar will do as you tell them without actually confirming if they are valid (some do, most don't).
The best option would be to set your domains DNS to use:
That will allow you to add your domain through the DigitalOcean Dashboard and manage your DNS entries for each domain there.
Once your domain has been added, you can add your A, CNAME, MX, TXT, AAAA, and other DNS records.
You'd then point the main A entry for your domain to your Droplet IP. Optionally, you'd add a CNAME for
www and point that to your domain, like so:
A @ DROPLET_IP
CNAME www domain.com
That'll make the setup much easier and get things working for you far quicker without having to worry about setting up private name servers (or vanity name servers, depending on where you look).