No, that I looked for specifically and there's no way to restrict voting to a subset of user groups. I might try taking a look at the voting plugin and see if there's an easy way to hack that in, since plugins are less likely to need re-patching after every upgrade.