You can't really "convert" .gym to MP3/Vorbis as such since a .gym file is not actually a sound recording; it would be necessary to synthesize it and then encode the synthesized file. The main thing that comes to mind is to get a .gym player plugin for Winamp and use the disk writer output plugin, then use the MP3 or Vorbis encoder of your choice on the resulting files.
edit: okay, I see you've tried that. I don't know what else to suggest. Ss for the quality issue, you will never find a .gym player that is anywhere close to 100% accurate because the format itself limits the frequency of updates to the sound configuration which is no problem for normal synthesis but breaks a bunch of special effects.