So to get the border area to display black instead of the back screen color, I had to set the TVMD register at vblank, just like you did when you were setting the screen to hi res. This is what I added in function vblank_requirements of render.c, the value is the same as Jo engine for NTSC but with the BDCLMD bit set to 0 :
The values that Jo engine put in this register at init set the border to the back screen color (JO_VDP2_TVMD in core.c). But changing those values has no effect on the border color, and I couldn't find something else that modified the TVMD register.