I believe that it is easiest to install Virtualmin when starting from scratch, on a fresh system.
That way you would only use the install.sh script, which would install and configure all of the necessary the system automatically, and everything would work out of the box.
My guess would be that it is quite likely to cause some problems on a live system.
What I think you could do is to do a manual installation. It’s more complex, but with some tweaking it should certainly. Here’s a link to the manual installation guide from the official Virtualmin website:
Note: skip the “Partitioning” part.
Of course if you decide to go with that, make sure that you have a backup!
But I think it would be easier, to create a new server, install Virtualmin and Webmin and then simply transfer your sites over to the new server. I think that this would require less manual work and it can be done with 0 downtime.
Hope that this helps!