@iMetalliX
When it comes to running a mail server, there’s three primary things you need to work with, other than the mail server itself – SPF, DKIM, and DMARC.
Since SPF and DKIM are more widely accepted than they used to be, many providers, GMail included, now verify whether or not the sender has these items setup by validating DNS entries against the mail server. DMARC isn’t as widely enforced, though I’d say it won’t be long before providers begin asking for such as well.
The link below is a pretty in-depth guide on how to set each of these things up. At the very least, you really need to setup both SPF and DKIM, though I’d advise all three.
https://www.skelleton.net/2015/03/21/how-to-eliminate-spam-and-protect-your-name-with-dmarc/
As an aside, this is one reason why many prefer not to run their own mail server and instead offload much of the work to third-party solutions such as Mailgun, SendGrid, and similar services. Unless you plan on spending a good deal of time managing your mail server (as in checking logs, preventing any potential abuse, locking down the server to prevent break-in and break-in attempts, etc), it’s often best to simply offload this portion of your workload to a service that just works with very little configuration.
Beyond just configuration, you also have to worry about Spam Blacklists which can cause your mail to be rejected if a provider is using a specific RBL with your IP in it, even with all three of the above properly configured.
By no means am I telling you not to run your own, rather, I’m saying it’s a lot of work at times and if you would rather focus on application development and/or your site without having to worry too much about anything more than a plugin or two, I’d offload. WordPress has plugins for both of those services.
