You might not need to keep the filesize the same. If its like lunar, there is a region of memory where the current script always gets loaded into. If the script is <= that size, youre fine. You do need to rebuild the cd though if you expand files. Instructions on that are on rockin -b’s site.
- Thread starter TrekkiesUnite118
- Start date
TrekkiesUnite118
Established Member
Do you have a link to where on his site those instructions are?
http://www.rockin-b.de/saturn/saturngametutorial/SaturnGameTutorial.htm#createISO
mkisofs: https://opensourcepack.blogspot.com/p/cdrtools.html
SegaCueMaker: http://www.racketboy.com/downloads/sega-cue-maker.htm
Fixing SegaCueMaker for XP+ (VB Issue): http://www.racketboy.com/forum/viewtopic.php?t=10106
mkisofs: https://opensourcepack.blogspot.com/p/cdrtools.html
SegaCueMaker: http://www.racketboy.com/downloads/sega-cue-maker.htm
Fixing SegaCueMaker for XP+ (VB Issue): http://www.racketboy.com/forum/viewtopic.php?t=10106
Last edited:
TrekkiesUnite118
Established Member
So I wrote a program that allows me to parse the MDT and MDP files from the Saturn and Playstation versions. This basically strips the first 512 bytes and reads them as a pointer table. This information is then used to parse the rest of the file. Knowing that 0x60 marks the table entry for the script, I wrote some code to copy the bytes defined in the table and write them to a file. This has essentially allowed me to dump the raw script for the game out of all the MDT/MDP files.
I've made it so it also writes the pointer table to a file, as well as the pre-script and post-script sections to files. This way it should be easy to make some code to do the reverse and put all the pieces back together and recalculate the pointer table.
On to the more interesting stuff though. After dumping the script for the Japanese Saturn, Japanese PS1, and English PS1 versions, I discovered that the Saturn and Japanese PS1 script data is identical. The initial cutscene I was looking at appears to be a special snowflake where they've added a bunch of additional control codes in the PS1 version. After that, things start lining up between the two. The only difference is the non-script parts of the data are byte swapped due to endian differences.
This is making it really easy to start identifying pointer values and control codes by comparing the different versions. I'm still figuring things out in these files, but I'm starting to think it might be possible to write a program that could automatically convert the English PS1 script files into a format the Saturn is expecting.
I've attached some samples for people to look at if they wish. The extension says .SCRIPT but that is just something I came up with to keep things organized. They're just raw data that you can look at in a hex editor.
I've made it so it also writes the pointer table to a file, as well as the pre-script and post-script sections to files. This way it should be easy to make some code to do the reverse and put all the pieces back together and recalculate the pointer table.
On to the more interesting stuff though. After dumping the script for the Japanese Saturn, Japanese PS1, and English PS1 versions, I discovered that the Saturn and Japanese PS1 script data is identical. The initial cutscene I was looking at appears to be a special snowflake where they've added a bunch of additional control codes in the PS1 version. After that, things start lining up between the two. The only difference is the non-script parts of the data are byte swapped due to endian differences.
This is making it really easy to start identifying pointer values and control codes by comparing the different versions. I'm still figuring things out in these files, but I'm starting to think it might be possible to write a program that could automatically convert the English PS1 script files into a format the Saturn is expecting.
I've attached some samples for people to look at if they wish. The extension says .SCRIPT but that is just something I came up with to keep things organized. They're just raw data that you can look at in a hex editor.
TrekkiesUnite118
Established Member
So I think I might have some logic for how to do an automated conversion. Since the script format is the same, but with portions byte swapped, converting is straight forward from a manual process, but it's very time consuming. So automating it would probably save time, but it's a bit tricky. From what I've found of the script files, there's no real common pattern I can use confidently to know exactly where the the text portion starts, and where the pre-text header/control code portion is. Basically some control codes are optional and aren't always used which makes knowing what to scan for difficult.
But I do know the following 2 pieces of information seem to be consistent enough:
1) Each text sequence ends in the byte 0x07
2) There will always be at least a 2 byte sequence that will tell us the amount of bytes to the end of the text sequence, starting immediately after those bytes. From this point on is the portion that doesn't need to be byte swapped. Now this isn't always in the same pattern sadly. I've seen at least the following variations
Now since we know the file format is identical between versions for the script data, and the control codes are the same, the only differences is the portions that precede and include the size bytes are byte swapped, and obviously the English text isn't the same size. Control codes appear to be the same. However for the Saturn version the control code 03 needs to be added at the start of each line to tell it to use the 8x16 font.
So with all of this in mind this was my thought process for a how it could be converted automatically:
1) Split Script file for both English PS1 and Japanese Saturn or PS1 files on byte value 07 to break it down into it's individual script sequences.
For English PS1 Script Pieces
2) Scan the script piece two bytes at a time, checking to see if the value of the low half byte of the first byte combined with the value of the second byte is equal to the remaining size of the piece when byte flipped.
*When doing the check, check the size starting at the offset immediately after these two bytes..
3) When we have a match, check the previous 2 bytes to see if they are also equal in distance from the offset immediately before them. If yes, make note of the value and it's offset.
4) Byte flip the piece from offset 0 to the last offset of the size byte.
5) From here on compare this piece to it's Japanese equivalent. Store size difference as a variable.
6) Scan byte by byte until we don't match, the data prior to this mismatch should be control codes. At this point insert 03. Keep a count of each 03.
7) If Japanese file is larger, jump English offset ahead by Size difference. If English is larger, jump Japanese offset ahead by size.
8) Scan until we do get a match. We're Hoping that we've found a control code, so check that the matched value starts with 0.
9) If not the end of the piece, Return to 6)
10) If end of piece, return to size values and increase them by the number of 03's added.
11) Save and move to next piece.
When all pieces are converted
12) Append all pieces back together
13) Reconstruct MDT file.
Obviously this probably still needs some more thought to it, but I think it's enough that I can code something together and then test it out and see how it does. Errors will probably still happen, but if this can help convert the bulk of the text with only a few errors that need hand checked and fixed I'd count that as a win here. If this can't be automated, the next option would be to do it all by hand which while doable, is very time consuming and opens itself up for it's own set of human errors.
One thing that could make this less complex and easier to implement though is if the default font can be set to the 8x16 font. This would eliminate the need for using the 0x03 control code to use that font and changing the size bytes. As a result this would then just become find the size byte, then byte swap the the necessary portion of the script piece.
But I do know the following 2 pieces of information seem to be consistent enough:
1) Each text sequence ends in the byte 0x07
2) There will always be at least a 2 byte sequence that will tell us the amount of bytes to the end of the text sequence, starting immediately after those bytes. From this point on is the portion that doesn't need to be byte swapped. Now this isn't always in the same pattern sadly. I've seen at least the following variations
- 0x2X XX
- 0x3X XX
- 0x9X XX
Now since we know the file format is identical between versions for the script data, and the control codes are the same, the only differences is the portions that precede and include the size bytes are byte swapped, and obviously the English text isn't the same size. Control codes appear to be the same. However for the Saturn version the control code 03 needs to be added at the start of each line to tell it to use the 8x16 font.
So with all of this in mind this was my thought process for a how it could be converted automatically:
1) Split Script file for both English PS1 and Japanese Saturn or PS1 files on byte value 07 to break it down into it's individual script sequences.
For English PS1 Script Pieces
2) Scan the script piece two bytes at a time, checking to see if the value of the low half byte of the first byte combined with the value of the second byte is equal to the remaining size of the piece when byte flipped.
*When doing the check, check the size starting at the offset immediately after these two bytes..
3) When we have a match, check the previous 2 bytes to see if they are also equal in distance from the offset immediately before them. If yes, make note of the value and it's offset.
4) Byte flip the piece from offset 0 to the last offset of the size byte.
5) From here on compare this piece to it's Japanese equivalent. Store size difference as a variable.
6) Scan byte by byte until we don't match, the data prior to this mismatch should be control codes. At this point insert 03. Keep a count of each 03.
7) If Japanese file is larger, jump English offset ahead by Size difference. If English is larger, jump Japanese offset ahead by size.
8) Scan until we do get a match. We're Hoping that we've found a control code, so check that the matched value starts with 0.
9) If not the end of the piece, Return to 6)
10) If end of piece, return to size values and increase them by the number of 03's added.
11) Save and move to next piece.
When all pieces are converted
12) Append all pieces back together
13) Reconstruct MDT file.
Obviously this probably still needs some more thought to it, but I think it's enough that I can code something together and then test it out and see how it does. Errors will probably still happen, but if this can help convert the bulk of the text with only a few errors that need hand checked and fixed I'd count that as a win here. If this can't be automated, the next option would be to do it all by hand which while doable, is very time consuming and opens itself up for it's own set of human errors.
One thing that could make this less complex and easier to implement though is if the default font can be set to the 8x16 font. This would eliminate the need for using the 0x03 control code to use that font and changing the size bytes. As a result this would then just become find the size byte, then byte swap the the necessary portion of the script piece.
Last edited:
You might want to try expanding the text manually first to see if it works before you go through all of that coding. It may break completely. Lunar has a bunch of relative jumps in control codes that come before/after text. If the file contents shift, those jumps break and the game freezes. Grandia might be similar... you did mention that the preceding control codes differ. Its also suspicious that the Korean version did not attempt to expand anywhere.
Last edited:
TrekkiesUnite118
Established Member
Well so far from what I'm seeing I'm not sure if that would be an issue if you go the route of just converting the PS1 files. As I stated the script format between consoles appears to be the same. Any jumping with pointers and size seems to be relative to within the scope of the script file. So if that script portion is converted accurately, the pointers should still line up fine I'd imagine.
Where things could get messy fast though is with that 03 control code to activate the 8x16 font. I'm not seeing that used anywhere in the English PS1 version so I'm guessing they reassigned it to the default font. Having to account for each one of those codes makes it very easy to get errors in the pointers, since I think each line of text needs the code if I remember correctly from testing it earlier.
Don't get me wrong, more research still needs to be done before it gets to that step to do it accurately. That was simply my thoughts on the info I had. However I just noticed my parsing code had a bug. I wasn't parsing out the chunk of data defined in the Pointer Table right before the script. I realized this and fixed the parser to parse it out and print it out to a file. This looks to be another piece of the puzzle. This is another pointer table by the looks of things. Each entry looks to be 4 bytes long. The first 2 Bytes seem to represent a specific ID text box event, the next 2 bytes appear to be the offset in the script file for where that events text starts.
With this information I can probably break the individual script file into these event pieces. This might make it easier to then examine control codes to get a better idea of what means what.
Oh, i misread that the other day. So there is similarity between the jap and eng ps1 versions, thats reassuring. Btw it looks like YabaSanshiro 1.6.1 sort of supports the game, might be able to figure something out at the asm level. Its really buggy, but I was able to get it to some dialog boxes.
Last edited:
TrekkiesUnite118
Established Member
So time for some news.
First thanks to some help from Klarth I've found out how to enable the 8x16 English font. The address 0x0601CEA4 has a byte that seems to control this. When it's set to x00 we get the 16x16 Japanese font. When it's set to x01 we get the ASCII 8x16 font.
Digging further in the debugger for YabaSanshiro I was able to find the assembly code that controls this. At 0x0601E2A4 there's these lines:
The last 2 lines are the important code here. This code sets r0 to x00 and then writes it to the address at r3. That address happens to be 0x0601CEA4. Doing some testing, I was able to successfully force the game to use the 8x16 font by changing the code to this:
This seems to do the trick. When viewing the memory address for this code I found these values:
This appears to align with the Opcodes for these instructions according to the SH-2 manual. If I'm understanding things correctly, I believe I need to change it to this:
Which would align with the Opcode for the replacement mov instruction. The problem is however, I can't seem to find this sequence of hex values anywhere in the game's executable, or anywhere in the games BIN/ISO for that matter. I found 2 values that were close, but they didn't seem to make any difference when I changed them. So I'm not sure what to do here.
Moving on however, I have some more news concerning this. I have continued work on my program for converting the files and I believe I may have successfully converted the PS1 script data over to the Saturn format. When combined with the memory edits above, I was able to get this:
Now this still isn't perfect. As while this cutscene plays fine, the game seems to hang afterwards when it tries to load Parm. The music and sound effects play, but the screen is black and the game doesn't seem to be progressing any further. Also not all the sound effects and voice clips seem to be playing during that first cutscene. So probably some debugging is needed to find out what's gone wrong. However the fact this seems to work in some cases looks promising that we can convert the English PS1 files over to the Saturn game rather easily.
I intend to release my code as I get it cleaned up as it's a bit sloppy at the moment. But if I get it working nicely, I'd imagine it could make it really easy for any future retranslation projects to apply their translation to both versions of the game.
First thanks to some help from Klarth I've found out how to enable the 8x16 English font. The address 0x0601CEA4 has a byte that seems to control this. When it's set to x00 we get the 16x16 Japanese font. When it's set to x01 we get the ASCII 8x16 font.
Digging further in the debugger for YabaSanshiro I was able to find the assembly code that controls this. At 0x0601E2A4 there's these lines:
Code:
extu.b r0, r0
shlr2 r0
and #0x01, r0
mov.b r0, @r3The last 2 lines are the important code here. This code sets r0 to x00 and then writes it to the address at r3. That address happens to be 0x0601CEA4. Doing some testing, I was able to successfully force the game to use the 8x16 font by changing the code to this:
Code:
extu.b r0, r0
shlr2 r0
mov #0x01, r0
mov.b r0, @r3This seems to do the trick. When viewing the memory address for this code I found these values:
Code:
600C 4009 C901 2300This appears to align with the Opcodes for these instructions according to the SH-2 manual. If I'm understanding things correctly, I believe I need to change it to this:
Code:
600C 4009 E001 2300Which would align with the Opcode for the replacement mov instruction. The problem is however, I can't seem to find this sequence of hex values anywhere in the game's executable, or anywhere in the games BIN/ISO for that matter. I found 2 values that were close, but they didn't seem to make any difference when I changed them. So I'm not sure what to do here.
Moving on however, I have some more news concerning this. I have continued work on my program for converting the files and I believe I may have successfully converted the PS1 script data over to the Saturn format. When combined with the memory edits above, I was able to get this:
Now this still isn't perfect. As while this cutscene plays fine, the game seems to hang afterwards when it tries to load Parm. The music and sound effects play, but the screen is black and the game doesn't seem to be progressing any further. Also not all the sound effects and voice clips seem to be playing during that first cutscene. So probably some debugging is needed to find out what's gone wrong. However the fact this seems to work in some cases looks promising that we can convert the English PS1 files over to the Saturn game rather easily.
I intend to release my code as I get it cleaned up as it's a bit sloppy at the moment. But if I get it working nicely, I'd imagine it could make it really easy for any future retranslation projects to apply their translation to both versions of the game.
Last edited:
TrekkiesUnite118
Established Member
So I've been able to figure out how to get the game to use the 8x16 font by default. Turns out the reason I couldn't find the code that I needed to change is because it doesn't exist in the games compiled code. Instead those instructions are generated at run time based on values the game reads from other files. This could be a form of compression, or it could be some kind of automated code generation. I'm not really sure which.
However, I was able to find what value it's using to generate the code that determines what font to use. Turns out it's in the SCNRL.BIN file. At offset 0x22A4 is the value 0x9D. This value get's XOR'd with the value 0x54 to create 0xC9. This corresponds to the And instruction for SH-2. Changing this value to 0xB4 will instead result in the value 0xE0 being created, which is the Mov instruction we need instead to use the 8x16 Japanese font.
Now the game is still crashing after the intro cutscene when it tries to load Parm. However I think this is due to the MDT file having an issue with it, as when I swap back in the original Japanese one it no longer crashes. So I'm going to need to dig back into these files and figure out whats wrong.
However, I was able to find what value it's using to generate the code that determines what font to use. Turns out it's in the SCNRL.BIN file. At offset 0x22A4 is the value 0x9D. This value get's XOR'd with the value 0x54 to create 0xC9. This corresponds to the And instruction for SH-2. Changing this value to 0xB4 will instead result in the value 0xE0 being created, which is the Mov instruction we need instead to use the 8x16 Japanese font.
Now the game is still crashing after the intro cutscene when it tries to load Parm. However I think this is due to the MDT file having an issue with it, as when I swap back in the original Japanese one it no longer crashes. So I'm going to need to dig back into these files and figure out whats wrong.
TrekkiesUnite118
Established Member
I removed the MDT file that was causing the game to crash and replaced it with it's Japanese counterpart for the time being while I figure out whats wrong. On the bright side though some of the other areas look to be working ok with some minor graphical glitches:
From the looks of it, in this area the palette data is wrong. So I probably need to figure out where it's trying to read from the wrong spot and fix it. The English PS1 version added/removed chunks of null data from the files, probably to increase/reduce padding for this very issue.
From the looks of it, in this area the palette data is wrong. So I probably need to figure out where it's trying to read from the wrong spot and fix it. The English PS1 version added/removed chunks of null data from the files, probably to increase/reduce padding for this very issue.
TrekkiesUnite118
Established Member
Found out what was wrong with Parm. There's a value that controls the fade in and fade out. In the first chunk in the script file there's a value at offset 0x4D that controls it. In the PS1 this is set to 0xD4, however on the Saturn it needs to be set to 0x7A. After making that change Parm loads fine and seems to be working ok.
TrekkiesUnite118
Established Member
So I'm still working on trying to figure out the Palette bug in some of the areas. It seems to be related to file size expansion, but it's kind of odd. Basically the offset in the MDT file header at 0x0D8 seems to be the entry that points to this data. And while the offset was updated in the file, it seems the game is only partially obeying it. Basically the game is still starting the read of the data from the correct spot in when it writes it into RAM. However it's stopping at what would be the originally stopping point for the original offset prior to the file size changing. The size of this data hasn't changed and it's still correct in the header. As a result of this, old data from the previous MDT file is still in RAM and is being copied over as Palette data, which isn't correct and is causing the partial graphical corruptions I'm seeing.
So I'm not entirely sure why this problem is happening. The only explanation is that it must be getting the value for where to stop reading from somewhere else. But where it's getting that info I haven't been able to track down yet.
In other news, I've put the code for the tools I've been writing up on GitHub:
https://github.com/TrekkiesUnite118/GrandiaTranslation
The tools are written in Java and hopefully should be easy to figure out. I'll be putting more info on the file formats as well as how to use the tools in the next few days when I have time. If anyone wants access as a contributor so they can add code/make changes let me know and I can add you.
So I'm not entirely sure why this problem is happening. The only explanation is that it must be getting the value for where to stop reading from somewhere else. But where it's getting that info I haven't been able to track down yet.
In other news, I've put the code for the tools I've been writing up on GitHub:
https://github.com/TrekkiesUnite118/GrandiaTranslation
The tools are written in Java and hopefully should be easy to figure out. I'll be putting more info on the file formats as well as how to use the tools in the next few days when I have time. If anyone wants access as a contributor so they can add code/make changes let me know and I can add you.
Last edited:
TrekkiesUnite118
Established Member
So it's been a while since I've posted an update, but I've finally solved the palette bug I'd been having with the automated conversion. Turns out the offsets in the MDT header 0x180 & 0x1F4 specify how many sectors need to be copied over from the CD for the file's data. This wasn't being updated so the game wasn't copying over all of the data. I've updated the tools to account for this and properly update the values when reconstructing the MDT files.
So with that out of the way I can move on to fixing some of the more minor issues that seem to be related to control codes. Basically audio isn't syncing as expected, and some of the text boxes aren't being positioned correctly or cleaning up correctly. I'd imagine this is probably some subtle difference between control codes across the Saturn and PS1.
Now, for something a bit more fun. I thought you guys might like to see some footage of the translated text in action. So here's a small bit I recorded from Mednafen:
Now to reiterate, this isn't perfect yet and there are bugs. So you'll get to see all of those as well.
So with that out of the way I can move on to fixing some of the more minor issues that seem to be related to control codes. Basically audio isn't syncing as expected, and some of the text boxes aren't being positioned correctly or cleaning up correctly. I'd imagine this is probably some subtle difference between control codes across the Saturn and PS1.
Now, for something a bit more fun. I thought you guys might like to see some footage of the translated text in action. So here's a small bit I recorded from Mednafen:
Now to reiterate, this isn't perfect yet and there are bugs. So you'll get to see all of those as well.
TrekkiesUnite118
Established Member
So I was able to fix the Text Box bugs that were visible in the previous video. Basically for the smaller text boxes the game does a calculation based on the number of characters in it to determine it's size. The game was still acting like the font was 16x16 instead of 8x16 which was causing the boxes to be bigger than they should. This was resulting in some having an overflow and bugging out.
I did some digging and found that when the game read the 0x03 control code, it calls another subroutine for these text boxes that sets a value at the memory location of 0x06001ECC. For 16x16 font this value is set to 0x00, for 8x16 it should be set to 0x01. I was able to find where this value was coming from in the SCNRL.BIN file and make it so it permanently is set to 0x01 which has fixed the text boxes as you can see in the below picture:
With that out of the way I think I have the bulk of the issues ironed out with the main script. What remains to be done for the main script is the following:
1) Fix any other bugs that come about through play-testing (I'm expecting more to crop up that were similar to the fade-in issue with Parm).
2) Fix the audio synchronization issue.
The first will need to be done on a case by case basis as issues are discovered through play testing. The latter though is going to be tricky to figure out. One thing that really makes it hard is that emulators like Mednafen seem to draw faster than a real Saturn. So the audio synchronization is different between emulators and real hardware. On Real hardware the audio for the most part always plays, and the text severely lags behind in speed. So I'm thinking it may be an issue of text speed. I've reached out to the creator of the UnDub PS1 patch for help and they've been giving some suggestions to look at.
For now though I'm thinking of shifting focus onto the menus and what not as what remains with the main script isn't as pressing of an issue I'd say.
I did some digging and found that when the game read the 0x03 control code, it calls another subroutine for these text boxes that sets a value at the memory location of 0x06001ECC. For 16x16 font this value is set to 0x00, for 8x16 it should be set to 0x01. I was able to find where this value was coming from in the SCNRL.BIN file and make it so it permanently is set to 0x01 which has fixed the text boxes as you can see in the below picture:
With that out of the way I think I have the bulk of the issues ironed out with the main script. What remains to be done for the main script is the following:
1) Fix any other bugs that come about through play-testing (I'm expecting more to crop up that were similar to the fade-in issue with Parm).
2) Fix the audio synchronization issue.
The first will need to be done on a case by case basis as issues are discovered through play testing. The latter though is going to be tricky to figure out. One thing that really makes it hard is that emulators like Mednafen seem to draw faster than a real Saturn. So the audio synchronization is different between emulators and real hardware. On Real hardware the audio for the most part always plays, and the text severely lags behind in speed. So I'm thinking it may be an issue of text speed. I've reached out to the creator of the UnDub PS1 patch for help and they've been giving some suggestions to look at.
For now though I'm thinking of shifting focus onto the menus and what not as what remains with the main script isn't as pressing of an issue I'd say.
I've been watching this thread since it started, seeing Trekkies figure this out gradually with a bit of assistance from you has been pretty cool. I just made this account to also say to him great job, his efforts are not going unnoticed and they are much appreciated.
XL2
Established Member
Same thing here, I've been watching your work, but I must say : Good job and keep up the good work!
