ServerAuth provides you with the ability to restrict who can SSH into your servers.

Invite members to your team, where they can manage their own SSH Keys. Then assign team members to individual server accounts, restricting when they can log in (e.g only during work hours), and to which server accounts.