The first thing that comes to mind is to check that your RGB pixels have the MSB set. The manual specifies that all RGB format pixels must have the MSB set, but IIRC that's so VDP2 can identify the pixel format and if you're not using any VDP2 effects RGB pixels will display fine with the MSB clear. However the documentation also says that some of the VDP1 color effects including transparency will only be performed if the MSB of the framebuffer pixel is set, otherwise the source pixel will be drawn using the normal "replace" mode.