• Rise from your gwave!

Translating Lunar Silver Star Story

Discussion in 'Saturn Dev' started by TrekkiesUnite118, Aug 5, 2013.

  1. TrekkiesUnite118

    TrekkiesUnite118 New Member

    It's been a while since I posted here but I figured I'd post this project I've been fiddling with lately.

    Over at SegaSaturn.co.uk I've been discussing the idea of taking the US PS1 script and audio and somehow putting it into the Saturn game. I'm posting this here as I could use some more experienced Saturn programmers/hackers as I'm in a bit over my head.

    From my thread over at SegaSaturn.co.uk, here is what I've figured out so far:

    My first attempt with a straight port over failed miserably. The game booted, but crashed when I tried to start a new game. I pretty much expected this but I decided to give it a try anyways just to be sure.

    I've been able to extract the files from iso's of the Japanese and English versions of the game as well as my copy of the MPEG version and a BIN I got of the standard Saturn version. Comparing the Saturn files to the Japanese PS1 files they look to be the same for the most part. Comparing with the English versions there's obvious data differences, but the format still appears to be the same.

    The problem I've run into appears to be with the Endian difference. The PS1 as far as I know is Little Endian where as the Saturn is Big Endian. The conversion between the two is looking to be a bit of an issue. Comparing the Japanese files is posing more questions than answers at this point. In some cases bytes are just flipped which you would expect with the difference in endian-ness. In other cases data is byte for byte identical. And in some cases it looks as though someone cut the file down the middle and flipped the two sides, so for Example:

    ABCDEFGH

    Becomes this:

    EFGHABCD

    The main issue is there's no consistency with in the files themselves. The 3 scenarios I listed above can all happen within the same file. So I'm not sure how to go about converting that with any sense of accuracy or if there even is a way to convert that. I'm not very experienced with this kind of thing so if someone more experienced wants to step in feel free to.

    Moving on, this is what I have found out about the files.

    The Saturn version is laid out in a very clear and logical manner. Files are stored in clearly marked folders and are organized quite neatly. The Structure for the Saturn version is as follows:

    Folders:

    BTL_DATA - I assume this is Battle Data. It contains 52 Files called B_DATAXX.CMP, 1 B_PARTY.CMP, 99 EFF_XX.DAT, 61 MXX.PCM, OPEN_BG.CMP, OPEN_OBJ.CMP, and 3 SOUNDXX.DAT files. My guess is this is mostly stats and Sound effects but I could be wrong.
    CPK - Holds all the FMVs
    MAP - Holds about 153 MAP.DAT files
    SOUND - Holds about 3 small files, LUNARMAP.REL, SCSPDRV.BIN, SO_PACK.DAT. My guess is drivers. By the file names maybe instruments or tables pointing to them, but they are awfully small for that.
    SYSTEM - This holds a lot of DAT Files, TBL Files, BIN Files, and PAL Files with a few others. Most of these by the names appear to be where all the different Face pictures are stored. It also holds the Map Table, Symbol data, as well as the Shop List among other things.
    TBL - Holds 155 Files called PEO_XXX.DAT. I'm not sure what these are. They could be stat Tables, or they might be Kana/Character tables. I'm leaning towards the latter since in the English version these are changed and are dramatically smaller in size. But I'm not 100% sure on what they are.
    TEXT - Contains 61 TEX_XXX.DAT files, 2 FONT.DAT files, and AR_BOOK.TXT which appears to be the entire script for the FMVs in plain text. This I'm guessing is where the text is stored. These are also changed heavily in the English version.

    Files:
    0 - Not sure what this is, its 504K. My best guess is maybe the main program. What I can decipher from it are a few references to files and folder and calls to what appear to be a Cinepak player.
    CDDA1 - Flute Song + Luna Singing
    CDDA2 - Luna telling Alex that this is a Saturn game and to take it out of his CD Player.
    ~99 PCM files - Mostly voice clips and sound effects
    MAP_TONE.BIN - No idea what this is.
    SE_MAP.SEQ - Might be a MIDI Sequence file, but again not really sure
    85 SE_MAP.DAT files
    SE_SYS.SEQ - Might be a MIDI Sequence file, but again not really sure
    68 SOUND.DAT files
    SYS_TONE.BIN - 32KB Binary file. Not sure what this is, probably part of the main program.
    SYSTEM.DAT - Data file of some kind. Might be for menus and stuff.


    The MPEG version differs slightly. The Main difference is the Folders have now been turned in to Archive files. Basically all the files in them are now added together into one big file with a header on top.

    From what I've looked at, the Header is pretty straight forward. The First 12 Bytes are the fire name. The Next 2 sets of 4 bytes I'm not sure on. All I know is that when you add them together you get the value of the first 4 bytes in the line after the next (if you have your Hex Editor set to 12 Bytes per line). The final 4 bytes are the file size. After each file there is a padding of 0's to the next 0x800 bytes.

    The other big difference is that the FMVs are now MPEGs and stored in the main directory. The SND archive now has the SE_MAP.DAT files along with the two SEQ files and both TONE.BIN files in it along with what was in the SOUND folder. The SYSTEM.DAT file is now in the SYSTEM Archive file. Probably to avoid having two files with the same name.

    There is also a file that doesn't correspond to a previous folder in the previous version called NW.DAT. This is where all the PCM data has been moved to. However it has no header like the others. It's header is instead stored in SYSTEM the System archive as a file called NW.DIR.

    Finally there is one file that only shows up in the Original Saturn version. It's called TEXT.DIC in the TEXT folder. I don't know what this is and I'm not even sure if it's used. But it's there none the less.

    The PS1 version has most of the same files, though the Sound is stored different it seems. There is one PEO Table missing from what I can tell, PEO_255.DAT. I'm not sure if this is actually used though as I said it's missing in both the Japanese and English PS1 versions.

    The main difference in the PS1 version is that instead of having folders or archive files, all the files are stored in one big archive file. You can extract them by using a tool called BioFAT. Though it seems only older versions support it.
    [hr]
    And here is more info pulled from my thread over at SegaSaturn.co.uk:

    Ok, so I've taken a short break to play around with the audio to see what complications are involved in converting the English Audio. So far I've found the following:

    M01.PCM - M61.PCM = Battle voices and game sound effects
    M62.PCM - M94.PCM = In game cutscene voices
    M95.PCM - M99.PCM = Final few sound effects of the game

    Audio files are in 8-bit PCM format at 16000Hz in Mono.

    The PS1 audio files in comparison are 8-bit PCM Format at 37800Hz in Mono. So right off the bat there's going to be a hit in audio quality. My first attempt at converting the PS1 Files to the above Saturn format didn't quite work. The game crashed when it tried to load the audio. The files were too large for RAM.

    My Next test was to try the conversion again only this time cutting off the trailing silence at the end of the files, and slightly reducing some dramatic pauses. This worked and got the files to fit in RAM only there was another problem:

    http://www.youtube.com/watch?v=m472VMaPloM

    The audio clips were too long and got cut off. The game also got hung up and stopped. My guess is that the files were still too big and the Saturn didn't know what to do with itself.

    So I tried reducing the Sample Rate to see if smaller files might fix the issue. This made the samples play a bit longer but the same thing happned. So some hex editing of pointers will probably need to happen at some point is my guess. However this test did have some hilarious results:

    http://www.youtube.com/watch?v=m6yjkA4Hrhg
     
  2. FacundoARG

    FacundoARG New Member

    Nice project.
    Good luck and keep pushing, because it's not a easy way :biggrin:
     
  3. SaturnAR

    SaturnAR New Member

    Hi,

    seems that you're with SS hacking too. We're currently working with Policenauts, and almost sure i wont be hacking any more Saturn games.

    Audio pitch may be affected because you changed the sample rate, but SS is strill trying to play it @16k samples per second, so the result is sound speeding up.

    Are you using an emulator?

    I don't know if it's a problem of out of memory. Are SS audio filesize smaller than PSX version? You should investigate it using a debugger (we use Yabause).

    Good luck with the project!
     
  4. CyberWarriorX

    CyberWarriorX New Member

    First off, good luck with your project! Not too easy of a game to work with iirc, but should be do-able, at least Lunar SSS anyways. MPEG version will likely require a custom disc building tool if you plan to expand anything.

    Yeah, you pretty much have to reverse engineer the file format in that case. With any luck it'll only be endian of pointers and data sizes you have to change.

    SCSPDRV.BIN is definitely 68k code for the sound driver. Usually you'll see some kind of identifier around 0x1020-0x1040 like "<< SEGASATURN >>" if you want to verify it. Usually pretty easy to identify.

    The *TONE.BIN are likely instrument data files. Usually they have the .TON extenion or .BIN. *.SEQ are a kind of compressed midi file. Not sure about the *.DAT files. Could be possibly scsp dsp code.

    One of the unknown set of 4 bytes is likely the offset for the file data in either bytes or sectors. Try multiplying it by 2048 and see if it matches the start of any data.

    As for the audio issue, not sure what to suggest. Technically the audio is likely streamed and not copied to ram straight up, but honestly I'm not sure. How exactly are you inserting it currently? What's the size in bytes of the original jap vs the converted english?
     
  5. TrekkiesUnite118

    TrekkiesUnite118 New Member

    Quick update. It appears I was wrong about the one instance of flipped data. The instance that looked like this:

    Actually isn't the case at all. What's happening is that the PS1 files have about 8 or more bytes of extra data in their headers which is throwing off the data just enough to make it look flipped. After I took this into consideration the data appears to match up much better between the JP Saturn and JP PS1 files.

    There isn't anything around 0x1020 but around 0x13A2 I found this:

    So yeah, I think that is the sound driver.


    The files are just raw PCM. So I'm just doing a straight conversion between bit rates in Audacity and exporting them as raw PCM, then swapping the files in the BIN.

    I'm pretty sure it is RAM related, as the files are broken up rather cleverly on the Saturn version to fit into about 512KB. The largest clip is 432KB in the Japanese Saturn version and it's part of Alex's introduction narration. The clip itself is actually split into two files with the 2nd one coming in at 32KB. Basically in total it's about a 28 second clip that's split into 2 clips of 27 Seconds and 1 Second respectively. The only logical reason I can think of for doing that is to fit into RAM. In my playing around I think I was able to squeeze a 460ish KB file in without it crashing, but it still got cut off.

    The English PS1 files are significantly larger in size. Alex's speech is broken into two files that are 637KB and 262KB respectively. They are also in XA format. The larger size I believe has to do with two things. The PS1 audio is at more than double the bit rate (around 37000 Hz) and the English dialog takes some liberties and rambles on a bit longer. Alex's intro now comes in at 41 seconds long split into 2 files of 29 seconds and 12 seconds long respectively.

    I think though I might be able to make it work by combining the files into one clip and doing a more even split. But I'll have to play with it. And I'll probably still have to change some of the pointers that handle the clips to compensate for the different length.
     
  6. CyberWarriorX

    CyberWarriorX New Member

    Weird. What's the extra data for? Alignment?

    That's right, I forgot Lunar SSS and SSSC MPEG use their own custom sound driver. Was quite a pain to get working with Yabause.

    Would make some sense, also you'd likely get less than 512kb in space since the sound driver and music take a small chunk. Just don't understand why they would do it this way. They could've even buffered some in wram or cd block if they wanted to. Oh well.

    Yeah, seems as though re-working the split is the easiest solution. Have you look into adding sound clip files, whether it's possible, etc.?
     
  7. TrekkiesUnite118

    TrekkiesUnite118 New Member

    I've thought of adding sound clip files, but I'm not sure if it's possible or necessary. A lot of the other clips are in the 32-200KB range on the Saturn so I would assume the PS1 clips once converted down to 16000 Hz should fit into memory. The largest clip after being converted after all was right on the edge of being too big. So I was planning on exploring the idea of splitting the clips in a different way first.

    As for the extra data I'm not entirely sure what it is. I'll need to spend more time looking at the file formats to figure out what exactly is going on.
     
  8. TrekkiesUnite118

    TrekkiesUnite118 New Member

    Ok, I figured I'd give an update on the TEXT files.

    So I believe the first 0x800 bytes of these are the headers. In each one it appears that there's at least this much space reserved before what appears to be the bulk of the file. That's pretty much all of the good news.

    Basically these files are a bit of a toss up when doing comparisons. From what I've found out comparing the JP PS1 files to the Saturn files is that the Saturn data is larger. This could mean that the PS1 data is compressed in some way. The issue I described earlier is related exclusively to the file TEXT000.DAT. That one has what appear to be symbol tables scattered throughout it, and on the PS1 these tables tend to be a couple bytes larger throwing things off. The data in these appears to be identical for the most part, but the data in between them appears to be quite different with the Saturn data being the larger of the two.

    So it would appear that the first thing necessary to make any headway would be to figure out if and how the PS1 files are compressed. Once that can be determined then we could figure out if the data is in fact the same and can be transplanted back and forth.

    I am thinking a good place to start with this would be to figure out how to read the file headers and see if that can point to any clues about how to read the files and if they are compressed.

    If anyone else wants to take a look at the files and possibly help figure them out I have uploaded them to my drop box here:

    PS1:
    https://www.dropbox.com/s/m005m0w368iow54/PS1_JP_TEXT.zip

    Saturn:
    https://www.dropbox.com/s/gwezlgtdgv1bckf/SATURN_TEXT.zip
     
  9. TrekkiesUnite118

    TrekkiesUnite118 New Member

    I haven't posted anything in a while but here's an update on what my train of thought has become.

    I really haven't had much time lately to spend looking through the files as I'd like, but I think at this point it might be just better to take standard Rom hacking practices and attempt to dump the script from the individual files. The problem with this approach though is that it goes into a realm of skills that I know little to nothing about and I really don't have as much time as I would like to devote to learning those skills right now.

    However I do think that TEXT000.DAT might be a table file for symbols. There's repeating blocks of data which are just listings of the complete western alphabet with varying gaps between them of gibberish (which could be Japanese characters that my Hex Editor can't identify). Also when I've looked through the other Text files, after the header there seems to be a loose pattern that I've identified. Basically we get a series of hex values which always seems to end in FFFF. I think this might correspond to the start or end of a line of text or a textbox or something like that. The hex values inbetween those repeating FFFFs appear to have some kind of logic to them. There are a few values that seem to pop up more commonly than others which could correspond to commonly used letters/symbols. So I believe this is probably the meat of the script.

    I would really appreciate it if some people with more ROM hacking experience than me could step in and help out with this.
     
  10. kirkrunner

    kirkrunner New Member

    Wow great work so far sounds like alot but keep it up, i wish i knew more about romhacking to help you out but im still pretty newbie. i am crossing my fingers for ya. Would be a dream to play on a modded system
    -cheers mate
     
  11. Ms. Tea

    Ms. Tea New Member

    Hope people don't mind if I bump this; it seemed relevant. :) Since TrekkiesUnite118 crossposted between forums, I figured I'd crosspost my update too.

    I picked up from TrekkiesUnite118's notes and made some more progress.

    The iOS version turned out to be the rosetta stone. While the PS1 and Windows versions use the same format as each other, the iOS version was ported from the Saturn source code and it uses the same text format. The iOS script files are actually compatible with the Saturn version almost as-is - it just needs a minor ASM hack to handle a one-byte encoding, along with an English font. It already supports a variable-width font so inserting the English version's font is looking pretty feasible. The menu text will take some more work, but it's all looking very doable.

    I've also been putting some work into FMVs, and have been able to successfully insert English audio:

    I'm still waiting on my VCD card to arrive in the mail so I can test with real hardware, but authoring compatible MPEG files for the VCD edition is pretty straightforward.

    I've put my notes up publicly: github.com/mistydemeo/lunardata They're a bit more detailed than what I posted here. My Cinepak dubbing tool is also available, though it needs a bit more work: github.com/mistydemeo/cinerepak

    Silver Star Story has been on the backburner for the past bit while Suppertails66 and I work on Magical School Lunar for Saturn, but I should have time to pick it back up again soon.
     
  12. CyberWarriorX

    CyberWarriorX New Member

    Hey Ms. Tea, great work you've done :)

    By the way, how are you rebuilding the disc image for Silver Star Story Complete?
     
  13. Ms. Tea

    Ms. Tea New Member

    For my initial test I used CDmage, which is able to insert RIFF/CDXA-format MPEG files as long as they're the same size as the original (or smaller). I don't have a tool that can rebuild from scratch, but I have a feeling I'm going to want one. Might look at extending mkisofs to add support for referencing files which exist outside the track being generated.
     
  14. CyberWarriorX

    CyberWarriorX New Member

    Oh that's right. Forgot about CDmage.

    I do have a couple utilities I wrote that could possibly be used for it that you may at least want to look at:

    vcdbuild - Based on Sega's own tool. Build a disc from a script
    vcdextract - Extracts an already built disc image and creates a script for the above utility

    I know for sure mpeg multiplexing, etc. works for vcdbuild.
     
  15. Ms. Tea

    Ms. Tea New Member

    Thanks! That looks really helpful.

    I don't develop on Windows, so I can't build vcdextract from source right now, but I'm going to try out the precompiled version in your Langrisser repo in Wine. Is that version up to date?
     
  16. CyberWarriorX

    CyberWarriorX New Member

    Yeah they should be up to date.
     
  17. TrekkiesUnite118

    TrekkiesUnite118 New Member

    Wow this is awesome to see. If you need any help with anything let me know. At the very least I can do testing for the MPEG version as I do have a modded Saturn with the VCD card.
     
  18. Ms. Tea

    Ms. Tea New Member

    Thanks! Testers will definitely be helpful when I have something playable.

    There's a few things on Silver Star Story in particular that I've left to the side while I focus on other things - if you're interested, any help on them would really be appreciated!

    • 1-byte encoding for text. Dialogue is currently hardcoded to use 2-byte characters; this would require an ASM hack.
    • Extracting the English font from Silver Star Story or Eternal Blue. It's a VWF that's probably stored in 16x16 tiles; the Japanese Saturn version already has VWF support, so inserting it looks pretty feasible.
    • In-game (not FMV) audio. I know you already took a bit of a look into this - I haven't had the chance to look deeper into it.
     
  19. mrkotfw

    mrkotfw New Member

    Excellent work!

    That's good to know that Japanese ver. already has VWF support! As for the 2-byte characters, why is that an issue aside from potentially increasing the size? Maybe pack 2 ASCII characters per 2-byte characters?

    Do you have any screenshots of the VWF support in action?
     
  20. Ms. Tea

    Ms. Tea New Member

    I already have usable script files from the iOS version which uses ASCII - it seems easier to adjust the game to support the files that already exist than to redo all of those files with two-byte characters.

    Yep! Here's an example from when I was experimenting with inserting random text to get a feel for encoding. You can see that the period, which comes in between other characters, is very much thinner than the surrounding characters. The 1 is also a bit thinner as well.

    [​IMG]
     
    Last edited: Aug 20, 2017
    SaturnTeam likes this.

Share This Page