In most cases, removing unused drivers or modules is OK, although they may be looped in to various other packages. In these cases, you will be notified via yum of their dependencies.
If you're in doubt of a package (such as lvm2), it is likely best to keep the package in-tact. Be careful when removing other kernel modules or specific packages, as one wrong move could cause push your droplet into an unrecoverable state! Take a snapshot before proceeding with any removals, and ensure your `/etc/yum.conf` file has `tsflags=repackage`.
Hope that helps! If you need anything further, feel free to submit a ticket.