Website without www not working however with www its works.

My website www.mydomain.com is working fine but mydomain.com giving 403 Forbidden error.

I have Apache Ubuntu server. Here is my configuration.

My host file /etc/host

...
127.0.0.1 localhost
XX.XX.XX.32 mydomain.com www.mydomain.com

My mydomain.com.conf file

ServerName mydomain.com
ServerAlias www.mydomain.com
ServerAdmin admin@mydomain.com
DocumentRoot /var/www/mydomain.com/public_html


RewriteEngine on
RewriteCond %{SERVER_NAME} = mydomain.com [OR]
RewriteCond %{SERVER_NAME} = www.mydomain.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

I already have configure A record to my IP address XX.XX.XX.32

I also set my CNAME to www.mydomain.com with TTL 43200

I have installed SSL for HTTPS as well. Having same above mydomain.com-le-ssl.conf

I already check root folder by removing with .htaccess its file. so no problem at htaccess file at all.

I am not sure what else I need to do. I am not able to access my website on mydomain.com or https://mydomain.com

Any suggestion please.