Apache will always display the first virtualhost in it’s configuration if a VirtualHost does not match with what is requested (as with the IP address). You can specify a third VirtualHost block in your configuration that does not have a ServerName or ServerAlias directive to direct requests for the IP address to a different folder on your droplet:

<VirtualHost *>
DocumentRoot /var/www/default
</VirtualHost>

Place this in a file in /etc/apache2/sites-enabled that would come alphabetically first in a list of the files in that directory (sorting is 0-9 then A-Z). Then create the directory /var/www/default (or make sure you are specifying a directory that already exists) with an index.html file. If you want nothing to display you can create a blank index.html file.

Once you’ve done this, reload your apache configuration and you should be all set:

service apache2 reload