Translating Grandia

Ardiloso

Member
Hi. Finally got a Pseudo Saturn Kai cart for my launch model US Saturn. I have the original Japanese game, and that plays flawlessly. I burned disc 1 of the English patched game on a Taiyo Yuden lacquer CDR. Other burned games play fine, but English patched Grandia frequently freezes on a black screen when accessing the save menu or items menu (music continues playing). Especially the A button menu, seems to freeze 9 times out of 10. Those are the only times the game ever freezes.

I'm still using my original laser lens, and it's kind of flakey, but works for other games. I just found it odd that it's always the same particular actions that seem to lock the game.
Grandia constantly streams from disc and is very picky with burning speeds, your lens being tired don't help either. To have a proper working Grandia burned disc you need to burn it at 1-2x max (you need an old burner), or at least new lens to alleviate these problems.
 

DJ Clae

New Member
Thank you for your efforts, TrekkiesUnite118. I've actually been following this project since 2018. You made dreams come true for a lot of diehard Saturn fans like myself with this project.

Grandia constantly streams from disc and is very picky with burning speeds, your lens being tired don't help either. To have a proper working Grandia burned disc you need to burn it at 1-2x max (you need an old burner), or at least new lens to alleviate these problems.
Thanks for your advice. I went and found the oldest burner in the house, and was able to burn it at 4x (as opposed to 10x as before). I know it doesn't fit your 1-2x max you're looking for. If anything, this new 4X copy seems to be a little worse. I kind of bought into the idea that it's not the speed that matters so much as preventing the drive from changing speeds mid-burn.

I may have to break down and buy a new lens, or just an ODE if I want to play this. It's just a shame to put in a cheapo replacement lens when the original lens still works for everything else. Maybe it's time to get a second Saturn.
 

Ardiloso

Member
Thank you for your efforts, TrekkiesUnite118. I've actually been following this project since 2018. You made dreams come true for a lot of diehard Saturn fans like myself with this project.


Thanks for your advice. I went and found the oldest burner in the house, and was able to burn it at 4x (as opposed to 10x as before). I know it doesn't fit your 1-2x max you're looking for. If anything, this new 4X copy seems to be a little worse. I kind of bought into the idea that it's not the speed that matters so much as preventing the drive from changing speeds mid-burn.

I may have to break down and buy a new lens, or just an ODE if I want to play this. It's just a shame to put in a cheapo replacement lens when the original lens still works for everything else. Maybe it's time to get a second Saturn.
This is really weird. Must be something between the lens or pseudo saturn. I have a modchipped console with new lens from aliexpress and had many problems with a 10x burned disc (my laptop's slower option) then used an old laptop to burn the discs at 8x and had great improvements. I finished the game with only 2 or 3 hang ups in a 65h playthrough.
One last tip, try your burned discs with ssf or mednafen. It should play fine.
 

DJ Clae

New Member
What kind of Psuedo Saturn cartridge? The cheap red ones on ebay tend to be pretty cheap and faulty and cause problems
It's not red, but it is from ebay. This one to be exact:

I haven't had any issues with it, other than the burn of patched Grandia sometimes locking up opening the item screen, A button menu, and save menu.

I just loaded up my original Japanese Grandia disc using the Pseudo Saturn cart and didn't have any of the same issues.
 

DJ Clae

New Member
Just burned a CDR copy of the Japanese disc at the same 10x, and I'm having the same problems with it. So I've established there's nothing wrong with the patched ISO I'm using.

I'll probably bite the bullet and order a new lens.
 

privateye

New Member
It's not red, but it is from ebay. This one to be exact:

I haven't had any issues with it, other than the burn of patched Grandia sometimes locking up opening the item screen, A button menu, and save menu.

I just loaded up my original Japanese Grandia disc using the Pseudo Saturn cart and didn't have any of the same issues.
These carts have compatibility issues with some games (including Grandia) on certain Saturn hardware revisions, as documented here: PPCenter :: Pseudo Saturn Kai Compatibility List

These issues can usually be fixed by shielding the PCB in the cartridge: PPCenter :: Pseudo Saturn Kai Compatibility List
 

nanash1

New Member
At least mathematically I understand the iDCT being used now. As I said, at its core a DCT or iDCT is just a matrix multiplication. A 2D DCT is two matrix multiplications, one for the rows and one for the columns. So if X is our image data, the DCT is Y=C X C^T and the iDCT X = C^T Y C. This works because the scaled DCT matrix C is orthogonal.

The codec in Grandia is altered though. All negative entries of the transposed DCT matrix are offset by +1. The non-transposed matrix however is unchanged. You can easily generate these matrices yourself with this Python code.

Python:
import numpy as np

def gen_dct_mat1():

    res = np.zeros((8,8))

    for j in range(0,8):
        for k in range(0,8):
            if j == 0:
                en = np.sqrt(2)/2
            else:
                en = 1
            
            res[j,k] = en*np.cos((j*(2*k+1)*np.pi)/16)
            if res[j,k] < 0:
                res[j,k] += 2
        
    return (0.5*res).T

def gen_dct_mat2():
  
    res = np.zeros((8,8))
  
    for j in range(0,8):
        for k in range(0,8):
            if j == 0:
                en = np.sqrt(2)/2
            else:
                en = 1
              
            res[j,k] = en*np.cos((j*(2*k+1)*np.pi)/16)
          
    return 0.5*res

I think this means that technically this isn't a DCT transform in the normal sense. At least for the columns of the iDCT input matrix. So in Grandia the equation for the iDCT is X = C_1^T Y C_2, where C_1^T != C_1^-1 != C_2 and C_2^T == C_2^^-1. But C_1 is still invertible, so the DCT should be Y = (C_1^T)^-1 X C_2^T.

When I implement the iDCT with floating point operations and compare it with the results from the game there are some differences. Like here for example:

mat_comp.PNG

We can see that in the integer result the values get smaller to the upper right, but in the floating point version they actually get slightly bigger. Since the floating point version works with 64 bit floats and I don't see potential for numerical instability here, I must assume that floating point version is more accurate and the differences are caused by the fixed point math used in the integer version. So think what I have to do is implement the DCT with high precision floating point operations and round the result after the quantization step to integers. The decoded results will probably be different from the pre-encoded data not only because of the quantization, but also because of errors in the fixed point math the game uses, but as far as I know that is unavoidable. If someone knows better though, please speak up.

I think with this I can begin to write an encoder. It will probably be pretty bare-bones and rather slow. At the moment my plan is to write most of it in Python with some C extension for stuff that would be too slow in pure Python. The most complicated part is figuring out how to determine what quantization to use per frame, but I think I can actually get around that. Since we'll be encoding pretty much the same video data as the original, I think I can just pull the quantization levels from the existing files and use those. This should hopefully yield very similar results in quality and compression to the original data. It may be necessary to bump up the quality of frames with subtitles though. DCTs are not good with the hard edges you see in text. And some of the video frames look pretty blocky. Like here for example:

city_image.png

I doubt that subtitles would be very readable with this strong compression level.
 
Last edited:

DJ Clae

New Member
These carts have compatibility issues with some games (including Grandia) on certain Saturn hardware revisions, as documented here: PPCenter :: Pseudo Saturn Kai Compatibility List

These issues can usually be fixed by shielding the PCB in the cartridge: PPCenter :: Pseudo Saturn Kai Compatibility List
Wow, you are the GOAT! I had no idea this was a known issue. Completely fixed my problem! Thank you! Thank you!

The instructions weren't great, and I have no idea why it says it will take 2 hours to stick some tape to a piece of aluminum foil. I managed to do it in about 15 minutes, and it worked!

By the way, the page says the problem effects the English-patched version, but I confirmed it happens with the original game on a CDR too.
 
At least mathematically I understand the iDCT being used now. As I said, at its core a DCT or iDCT is just a matrix multiplication. A 2D DCT is two matrix multiplications, one for the rows and one for the columns. So if X is our image data, the DCT is Y=C X C^T and the iDCT X = C^T Y C. This works because the scaled DCT matrix C is orthogonal.

The codec in Grandia is altered though. All negative entries of the transposed DCT matrix are offset by +1. The non-transposed matrix however is unchanged. You can easily generate these matrices yourself with this Python code.

Python:
import numpy as np

def gen_dct_mat1():

    res = np.zeros((8,8))

    for j in range(0,8):
        for k in range(0,8):
            if j == 0:
                en = np.sqrt(2)/2
            else:
                en = 1
           
            res[j,k] = en*np.cos((j*(2*k+1)*np.pi)/16)
            if res[j,k] < 0:
                res[j,k] += 2
       
    return (0.5*res).T

def gen_dct_mat2():
 
    res = np.zeros((8,8))
 
    for j in range(0,8):
        for k in range(0,8):
            if j == 0:
                en = np.sqrt(2)/2
            else:
                en = 1
             
            res[j,k] = en*np.cos((j*(2*k+1)*np.pi)/16)
         
    return 0.5*res

I think this means that technically this isn't a DCT transform in the normal sense. At least for the columns of the iDCT input matrix. So in Grandia the equation for the iDCT is X = C_1^T Y C_2, where C_1^T != C_1^-1 != C_2 and C_2^T == C_2^^-1. But C_1 is still invertible, so the DCT should be Y = (C_1^T)^-1 X C_2^T.

When I implement the iDCT with floating point operations and compare it with the results from the game there are some differences. Like here for example:

View attachment 5502

We can see that in the integer result the values get smaller to the upper right, but in the floating point version they actually get slightly bigger. Since the floating point version works with 64 bit floats and I don't see potential for numerical instability here, I must assume that floating point version is more accurate and the differences are caused by the fixed point math used in the integer version. So think what I have to do is implement the DCT with high precision floating point operations and round the result after the quantization step to integers. The decoded results will probably be different from the pre-encoded data not only because of the quantization, but also because of errors in the fixed point math the game uses, but as far as I know that is unavoidable. If someone knows better though, please speak up.

I think with this I can begin to write an encoder. It will probably be pretty bare-bones and rather slow. At the moment my plan is to write most of it in Python with some C extension for stuff that would be too slow in pure Python. The most complicated part is figuring out how to determine what quantization to use per frame, but I think I can actually get around that. Since we'll be encoding pretty much the same video data as the original, I think I can just pull the quantization levels from the existing files and use those. This should hopefully yield very similar results in quality and compression to the original data. It may be necessary to bump up the quality of frames with subtitles though. DCTs are not good with the hard edges you see in text. And some of the video frames look pretty blocky. Like here for example:

View attachment 5503

I doubt that subtitles would be very readable with this strong compression level.
This is really nice to see. For the issue of Subtitles being readable, I'd say we wont really know until we try. If you can get an encoder going we can try different fonts out and see. For what it's worth, we can take a look at the intros as those have credits in them. My thought was we could encode the PS1 versions for English credits.
 

klarth

New Member
If you do need to go the route of increasing the video size to accommodate the subtitles, I messaged the Mode developer the other week and he seems to think that the saturn can take discs up to 99 minutes since its using BCD for access of data sectors. Still need to test with actual media
 

Ardiloso

Member
This is really nice to see. For the issue of Subtitles being readable, I'd say we wont really know until we try. If you can get an encoder going we can try different fonts out and see. For what it's worth, we can take a look at the intros as those have credits in them. My thought was we could encode the PS1 versions for English credits.
The Steam HD release have those videos with english credits too and are at much higher quality and framerate. The first video (the 4:3 one) is a problem since they framed it in 16:9 with those shitty border effects and slapped the credits all over them, so the PS1 would still be the better option in this one.
 

Attachments

  • op1.png
    op1.png
    874.4 KB · Views: 12
  • op2.png
    op2.png
    1.1 MB · Views: 12
The Steam HD release have those videos with english credits too and are at much higher quality and framerate. The first video (the 4:3 one) is a problem since they framed it in 16:9 with those shitty border effects and slapped the credits all over them, so the PS1 would still be the better option in this one.
The Saturn FMVs appear to be either 10fps or 12fps. It seems some of the PS1 FMVS are capped at similar frame rates so I don't think there's much to be gained from using the HD Remaster ones unless there's some awful JPEG compression on the PS1 FMVs.
 

nanash1

New Member
That's really a shame with the HD remaster videos. Starting from a cleaner source would be much nicer than using th PS1 versions. Slapping a pretty strong lossy compression on video that is already very compressed isn't exactly a recipe nice picture quality. Does the PS1 use at least a higher resolution than the Saturn?

PS: I currently trying to understand how exactly the game handles the color space conversion and it seems to be handled by the SCU DSP. Is anyone here familiar with the DSP can tell what exactly is done here? You just have to set a code breakpoint at address 0x0 during fmv playback to see the routine.

PPS: Project is on Github, but there not much to see for now.

 
Last edited:
That's really a shame with the HD remaster videos. Starting from a cleaner source would be much nicer than using th PS1 versions. Slapping a pretty strong lossy compression on video that is already very compressed isn't exactly a recipe nice picture quality. Does the PS1 use at least a higher resolution than the Saturn?

PS: I currently trying to understand how exactly the game handles the color space conversion and it seems to be handled by the SCU DSP. Is anyone here familiar with the DSP can tell what exactly is done here? You just have to set a code breakpoint at address 0x0 during fmv playback to see the routine.

PS1's FMVs are 320x240 as far as I can tell. So lower horizontal, but higher vertical. For some the HD Remaster videos might be usable, but for ones like the opening before the title screen I think the PS1 version will have to do, or just leave the JP Saturn one. The other issue with the HD version is they seemed to do the "AI Learning" meme where they just upscale and blur it. So there's also that issue.
 

Ardiloso

Member
That's really a shame with the HD remaster videos. Starting from a cleaner source would be much nicer than using th PS1 versions. Slapping a pretty strong lossy compression on video that is already very compressed isn't exactly a recipe nice picture quality. Does the PS1 use at least a higher resolution than the Saturn?

PS: I currently trying to understand how exactly the game handles the color space conversion and it seems to be handled by the SCU DSP. Is anyone here familiar with the DSP can tell what exactly is done here? You just have to set a code breakpoint at address 0x0 during fmv playback to see the routine.

PPS: Project is on Github, but there not much to see for now.

Besides the opening, I think the fmv in need of subtitles are all the original widescreen ones so they don't suffer the border effect hell.

@TrekkiesUnite118 Yes they seem A.I upscaled but I think they look good. Also, I subtitled the jpn Megaman X4 fmv on the saturn and used as source some A.I upscaled youtube videos to enconde back to the cpk and they look virtually the same at that low resolution (maybe a little better but must be me).
 

Ponut

New Member
That's really a shame with the HD remaster videos. Starting from a cleaner source would be much nicer than using th PS1 versions. Slapping a pretty strong lossy compression on video that is already very compressed isn't exactly a recipe nice picture quality. Does the PS1 use at least a higher resolution than the Saturn?

PS: I currently trying to understand how exactly the game handles the color space conversion and it seems to be handled by the SCU DSP. Is anyone here familiar with the DSP can tell what exactly is done here? You just have to set a code breakpoint at address 0x0 during fmv playback to see the routine.

PPS: Project is on Github, but there not much to see for now.

I don't have a Grandia ROM handy. A plaintext past of the decompilation _from anything other than Yabause_ would help.
Of course, a program RAM dump at that time would do just as much.
 

nanash1

New Member
I don't have a Grandia ROM handy. A plaintext past of the decompilation _from anything other than Yabause_ would help.
Of course, a program RAM dump at that time would do just as much.

Thanks for the help. This should be a dump of the DSP program. Please tell me if you need anything else. I know that the first few commands take the data from the Y channel and cast the 16 bit values to 32 bit and store them in the DATA RAM. Then 0x7C seems to be added.
 

Attachments

  • colorspace_conv_dsp_dump.zip
    524 bytes · Views: 3

nanash1

New Member
I think I was able to figure out what the DSP program does. It sort of looks like the YCbCr to RGB conversion on Wikipedia, but no exactly.

Python:
def YCbCr2RGB(y, cr, cb):
    
    def mask(x):
        if x > 0:
            return x & 0xff
        else:
            return x & 0x00
        
    """
    y stored in M1
    cr stored in M2
    cb stored in M3
    """
    y0 = int().from_bytes(y[0:2], 'big', signed=True)
    y1 = int().from_bytes(y[2:4], 'big', signed=True)
    cr = int().from_bytes(cr, 'big', signed=True)
    cb = int().from_bytes(cb, 'big', signed=True)
    
    y0 += 124
    y1 += 124
    
    green0 = mask(y0 + cb + cr)   
    green1 = mask(y1 + cb + cr)
    red = mask(-2*cr + y0)
    blue = mask(-2*cb + y0)
        
    rgb0 = blue << 16 | green0 << 8 | red
    rgb1 = blue << 16 | green1 << 8 | red
    
    return rgb0, rgb1

# example
YCbCr2RGB(b'\xff\xae\xff\xad', b'\x00\x00\x00\x08', b'\xff\xff\xff\xf6')
 
Top