I've been using Gentoo for a few years now, and although the package management is simply awesome, I can say that the speed claims are exaggerated. You're not going to want to use -O3 all the time because compiles will fail and it doesn't speed things up that much anyway. In my experience, 80% of tweaking Linux has to do with making sure the kernel is set up in the most efficient way for your use. Second in importance would be doing things like HD tuning and so forth.
As for the 'five days of compile time' thing, it shouldn't take nearly that long, even on a slower system. GNOME/KDE tend to take the longest to compile anyway, and as you won't be using those, there's no need to worry about that.
About the mplayer crash thing, maybe a soundcard issue (probably not, but it's possible)? Which card are you using, and are you using OSS or ALSA? Try switching from one to the other and see if it makes a difference. But before you do that, try changing the ao option in your mplayer config and change the output driver to something else. If you're using ALSA, you can use the alsa, oss, or sdl drivers, for example, whereas with OSS you could use oss or sdl. I think oss is the default, so try sdl for the hell of it.
As for the 'five days of compile time' thing, it shouldn't take nearly that long, even on a slower system. GNOME/KDE tend to take the longest to compile anyway, and as you won't be using those, there's no need to worry about that.
About the mplayer crash thing, maybe a soundcard issue (probably not, but it's possible)? Which card are you using, and are you using OSS or ALSA? Try switching from one to the other and see if it makes a difference. But before you do that, try changing the ao option in your mplayer config and change the output driver to something else. If you're using ALSA, you can use the alsa, oss, or sdl drivers, for example, whereas with OSS you could use oss or sdl. I think oss is the default, so try sdl for the hell of it.