This is a good question. This is a subject very dear to me, I love running email servers. First, let me tell you why you shouldn’t run a mail server. We put this together a few years ago and it still rings true:
Aside from that, I run mail servers for a lot of people, and I can tell you first hand that nothing in life has ever caused me more pain and suffering. Whether or not it would bring you the same depends on your needs. A lot of people just want to receive email, and very rarely send any out. Those users will be more easily pleased. I deal a lot in sending, and that is where the frustration lies. I’ll explain.
Unless you are yourself a large corporation with incredible power and influence, with your own IP address space, someone is going to block the emails you send. With Google and Outlook, for example, they have that power and influence, while specifically managing an email infrastructure on their IP space, and it protects you. With your own server, sharing IP space with other customers, neither us nor you can intervene in some of these blocks that occur. Your email will reach Gmail unless you send spam toward them (don’t forward email to their service, this causes blocks). Your email might reach Outlook (Hotmail/Live) and, if it doesn’t, they might unblock you if you fill out a request. Your email will probably never reach Verizon or AT&T, and they will not respond to requests to lift a block (from anyone, for any or no reason).
To add to that, there are things you have to specifically do to play by the right rules with email. If you don’t play by the rules, anyone you send mail to considers you a spammer. Your server’s hostname must be set (ex. mail.yourdomain.tld), the mail server must be configured to send that hostname as it’s HELO statement (ex. “Hi, I’m mail.yourdomain.tld”), and the A record for the hostname must point to the IP of that mail server (ex. mail.yourdomain.tld > 126.96.36.199). You should have an SPF record in your DNS like this, saying email from yourdomain.tld is allowed to be sent from the droplet IP (let’s use 188.8.131.52 as the example IP). It would look like this:
Record name: @ or blank
Record type: TXT
Record contents: “v=spf1 ip4:184.108.40.206 -all”
Many will recommend DKIM as well, and it has it’s benefits, but I’d recommend you get comfortable with everything else before you tackle DKIM. I’ve seen it trip up too many people, you don’t need that as an initial roadblock.
Now that I’ve told you the scary parts, let me talk about how to set up a mail server. This is the easy part. You may as well use a pre-built stack, just because it’s there and makes life easier. I’m fond of this one right now:
I’ve also had a lot of success running this stack in the past:
If you want guidance on setting up your own from scratch, this tutorial is pretty in depth:
Though I prefer rspamd to SpamAssassin myself, for filtering inbound and outbound spam.
Hope this helps :)