An emulation can look better than original hardware. There are really a limited number of things that are reasonable to do before compatibility starts suffering though. For example, an emulator could filter individual textures/sprites or the entire screen (e.g. with interpolation) and perform antialiasing on polygon edges. However, things like adding polygons to models or extending visible distance are too game-specific to really be feasible at the emulator level.
However, if the emulator author is willing to do work on a per-game basis, radical changes in visual quality can be made by creating new sets of graphics for existing games. At least two emulators, RetroFX
, are based on this concept. Nebula also allows the use of external semitransparency masks.
In any case, the generic stuff is nothing that couldn't be done with new hardware; the division between hardware and software is largely arbitrary anyway...