Luigi Auriemma

Hi Luigi!

I didn't want to post on Xentax again so I'm writing directly to you now. ;)
In my struggle to get the music out of any version of LEGO Indy 2, I also tried the XBox 360 version, which has some strange XMA format that toWAV can't decode. I fear it to be XMA2 but since I haven't stumbled on the format yet, I'll need you to take a quick look at the file and tell me your thoughts. ;)

Here's a file: http://www.filefront.com/15009299/1_0_HUB_1Nepal_amb.xma/

The structure is quite interesting: the stream is split into chunks of

1800 bytes (starting with 18 00 01),
0800 bytes (starting with 04) and
1000 bytes (starting with 1C 00 01).

This structure begins at offset 0x13C0 and repeats during the whole file. Could this be a 6ch XMA? The structure looks very different to the multichannel XMAs I've already seen.
What do you think? Thanks for your support!

Regards, Timo

I accidentially found out something really nice: the header from the file of this post http://forum.xentax.com/viewtopic.php?f=17&t=3858 lets toWAV decode all six channels of the LEGO Indy 2 files! =) =) =)
Strangely it doesn't work for the file from the Xentax post...

I've attached the multichannel XMA header for files structured like the above.

Erm.... ok, gotta backpedal... The header seems to be for 4 channels, whereas the input file is for six.
Front channels are mixed up, the center/LFE channel stays silent and the rear channels seems to be correct. So the header has either an 1800 or 1000 byte interleave. Maybe at 0x2c or 0x39?
I'll play around a bit and let you know what I find out. I guess we could get all channels by successively deleting the first three channel chunks and decoding in between, which would then produce one valid channel each time.

unfortunately I can't help because I don't know much about xma, but I have read on xentax that you have solved the problem, right?

I've found out how to decode these files. It's a bit laborious but works. Here's what you do:

1. Open a file and the multichannel header in a HEX editor
2. Delete the old header till "data"
3. Copy & paste the multichannel header into the file at the start
4. Save the changed file as [old name]_1.xma
5. Find out the chunk lengths sequence (in the above file's case it's 1000, 1800, 800 and repeating)
6. Delete the first chunk and save the new file as [old name]_2.xma (-1000 bytes)
7. Delete the second chunk as well and save the new file as [old name]_3.xma (-1800 bytes)
8. run toWAV
9. check the playing times of the created files and delete all files that are shorter or duplicates

I've also tried this with the XMA file from Ace Combat 6 (link of Xentax post above), but this file seems to have 8 channels, so it's not decodable atm.

Is there a way to fasten this by writing a little program that automates the above procedure?

Very nice work, timo i havent tested this, but hopefully points to the right direction in xma multichannel decoding, strange it doesnt work for pies file though.

from point 1 to 4 it's all clear and the job can be done with quickbms, while I have not fully understood point 5 which is probably the ugly part of the method.

the following script does the 1-4 job:
I have used a MEMORY_FILE mainly for 2 reasons: easy updating of the script for other in-memory operations (like point 5) and for avoiding problems with the append mode on the file (warning + appending on a previously created file with the same name)

Thanks for the script so far! Works perfectly! =)
The chunk size recognition isn't that hard I think (also look at the post on Xentax, I found out other things about the). All the program needs to do is identify an interleave cycle with the single interleave chunks. Search for the placeholders (FF FF FF) first. When they stop, the chunk is finished and a new chunk starts. A cycle consists of three chunks for 6 channels and four chunks for 8 channels. So maybe you just write a script for 6ch and one for 8ch.
Now there's a little problem with these placeholders. I guess when the files are encoded, the encoder automatically searches for the chunk combination that wastes as little space for placeholders as possible. So as a result theree will be chunks that are completely filled with data. In this case there are no placeholders before the next chunk. That was the case with the file from Ace Combat 6: the first time the placeholders showed after the E800 chunk was in cycle 3. Before and after, the chunk is filled and thus toWAV recognizes it as a F000 chunk, whereas it's two chunks of E800 and 0800.
I'd just try to read the whole file and try to identify the interleave structure. I'll speak in decimal numbers, it's easier. ;)

Here are some hints about what I'd do (sorry if some points are a bit technical but I study maths ;)):

Suppose the recognized chunk lengths from a little file are it's safe to assume that the cycle is
(the second chunk is completely filled here, so you'll jump to the next placeholders)

How to get there
Step 1 - Add the minimum chunk length to the list. In this case this is the '10'.
Step 2 - As XMA files contain a very high number of cycles i'd say it's almost 100% safe to just calculate the distribution of the remaining chunk lengths from the above chunk length sequence.

case 1: 3 sizes to identify; three ones by far the most occuring -> ok!
case 2: 3 sizes to identify; two ones are by far the most occuring -> the appearance ratio of <#of1> to <#of2> is about 2 to 1 -> ok!
In the above example, the '15' appears eleven times, the '10' only six times, '30' and '25' each one time. '30' and '25' can be disregarded, the '15' appears about twice as much as the '10', so the chunk lengths are probably 15, 15 and 10 (at first in no particular order).
case 3: 3 sizes to identify; one is by far the most occuring one -> ok (symmetric interleave)

I can give you a multistep-method for identifying interleave cycles. That's basically just what came directly into my mind. Until someone has a better idea, you can try to implement this:

Step 3 - Check all possible non-cyclic permutations for convenience:
Little explanation on cyclic permutations: http://en.wikipedia.org/wiki/Cyclic_permutation
First, identify all non-cyclic permutations (if you don't know how, I can help here), then search in your chunk length sequence for each of them. One permutation will be found the most often. For example if the cycle is 5:10:12, you won't find any occurances of the sequence 12:10:5 (a non-cyclic permutation) in the identified chunk length sequence.

Step 4 - Check cyclic permutations (ORD) of the sequence identified above for convenience:
Let seq be the found chunk sequence from above: seq(1)=10, seq(2)=30, ... and the possible order ORD.
From here it's just some lines of 'if', 'else' and 'elseif'. Here are some lines that should occur somewhere in this process:

if ORD(1)>seq(1) -> go to next permutation of ORD
if ORD(1)==seq(1) -> first place is right, check next places and permutate if necessary
if ORD(1)<seq(1) -> check if ORD(1)+ORD(2)==seq(1)
Maybe you can figure out the rest. ;)

Ok, that's it for now, I hope these hints are helping a bit.n Post if you have questions!

seems that the xmaencode got leaked somehow.

taken from xentax

http://www.filefactory.com/file/a1e8efa/n/xmaencode.rar

But, we can't seem to figure out how to make it run...

I'd posted this in another thread on XMA: http://hcs64.com/files/xma_parse01.zip
This is an XMA stream parser (incomplete). At the beginning of a "block", there is one packet from each stream, in order. A packet is 0x800 bytes. In the packet header is specified how many packets to skip before you get to the next packet of this stream. It should be possible to use this to separate out the streams systematically.
Sorry if this is stuff already done, I haven't really paid attention since I wrote this back in Feb.

[edit]
I think the block size is 0x3000 for this file (0x30), you'll notice that the parser reports the same sample count for:
xma_test blah.xma -o 3c0 -b 3000
xma_test blah.xma -o bc0 -b 3000
xma_test blah.xma -o 13c0 -b 3000

6656 samples for each, as it should be, despite the fact that the second stream is all silence. It looks like the second stream has the shortest frames it can manage. This holds in the second block, too.

I only get an error message when trying it... :-\
-> Parse error: unexpected "frame sync" 5d11

You need the -o to tell it where the stream starts, and the -b 3000 to get the block size right. I really only provided it for a look at the code, the executable doesn't do much.

Incidentally this has got me thinking again, I made a few fixes to the parser and added an option to dump out the stream, -x outfile.bin. The text output of all the data is now only done with -v, and it will attempt to follow the whole stream across as many blocks as there appear to be. I think I'm missing some at the end of the file, but things seem to be working out consistently until then. I don't know how to assemble a playable file out of the raw stream, I assume you guys are more familiar.

http://hcs64.com/files/xma_parse02.zip

xma_test blah.xma -o 3c0 -b 3000 -x stream1.bin
xma_test blah.xma -o bc0 -b 3000 -x stream2.bin
xma_test blah.xma -o 13c0 -b 3000 -x stream3.bin

blah.xma here is the 1_0_HUB_1Nepal_amb.xma you posted earlier.

[edit]
Updated to handle last block (which can be incomplete) properly. Sample count now agrees with that in header.
http://hcs64.com/files/xma_parse03.zip

[edit]
I put this header on the extracted streams and it seems to work ok for stereo when decoding with towav: http://hcs64.com/files/xma_header.zip
I took a look at the Ace Combat 6 file for the heck of it, I can extract the streams like so:
./xma_test.exe ../Ace\ Combat\ 6\ -\ 1.xma -o 800 -b 10000 -d 5d0000
but the extracted stream behaves in towav the same as the original file. Note that I had to specify a data size; while the stream passes the parser checks up to the end of the file, the file seems to be truncated (just look at the size of the data chunk).

Hm, I'm not really familiar with parsing and this technical stuff... The only thing I see is that the file really seems to be truncated. Will try your header on other files to check that out.
I'm also currently testing some things with the Indy2 files. Apparently something's not right with the decoded files...

In case it wasn't clear (I couldn't tell), I think xma_test works at separating out the streams in the example you gave, see my example with blah.xma above. Stereo streams can then be decoded correctly with toWav.

I'm still working on the Ace Combat 6 files, I think that the glitches are caused in part by block boundaries. There seems to be a correlation between glitches in the decoded stream and the a packet having no frames starting in it (which happens occasionally at the end of a block). These packets have a frame count of 0 and an invalid bit skip value (0x7fff), and after the overflowed data from the previous packet they have a dummy frame indicating end of stream (size 0x7fff). It may be that this is confusing the decoder somehow. I was successful at converting XMA2 (which is what my parser handled earlier, the whole block concept and packet skipping is XMA2) into plain XMA, and I built a .xma with that which works in xmaencode, but it suffers the same issues. I made some attempts at fixing these packets, but I haven't found any improvement yet. My last resort is going to be repacketizing the whole XMA2 stream as XMA, without block boundaries to worry about.

Hope no one minds the extra post:

I was successful at rebuilding the Ace Combat 6 stream as a plain XMA stream, and it seems to fix the skipping issue. Steps to reproduce with http://hcs64.com/files/xma_parse04.zip :

1. Rebuild the XMA2 stream (the front channels start at 0x800, bass at 0x1000, surround at 0x1800) as plain XMA
xma_test blah.xma -o 800 -b 10000 -d 1401800 -r ace.bin
2. Stick a header on it (the one I uploaded a little while ago works)
3. Decode with toWav!


I tried it with the nontruncated version of that track that pie put up here: http://www.megaupload.com/?d=YAGG8Z4H

This doesn't work with everything even in Ace Combat 6, the 27.6 MB file is doing something different, but it's at least a baby step in the right direction.

P.S. To get a XMA stream (not XMA2) working with xmaencode.exe, prepend a header (http://hcs64.com/files/xma.head.zip), add a seek table (http://hcs64.com/files/xma_seek.zip), and correct the RIFF and data chunk sizes.

Great, thanks HCS!

xma_parse 0.5 : http://hcs64.com/files/xma_parse05.zip

Will handle the 27.6 MB Ace Combat 6 ending theme like so (front channels, as usual, bass is at 1000, back at 1800)
xma_test endtheme.xma -1 -o 800 -b 10000 -d 1b81000 -x ace_ending.bin

Resulting raw stream will work in both toWav and xmaencode.exe when correctly wrapped

AlphaTwentyThree: I noticed that even that ambient track you send has a few clicks in it when decoded with toWav. It too benefits from the XMA2->XMA rebuild, so try it with -r instead of -x.

How do you correct these riff and chunk sizes?

RIFF size is at 0x4, should be file size - 8.
data chunk size is at 0x38, should be size of data (ace.bin or whatever)

Thanks a lot for all the stuff. Maybe I'm a bit slow, but when I try to split this file here http://www.sendspace.com/file/yzcq6b to three stereo XMA files, it does only work for the first two sets of channels.
These are the commands I used:

xma_test 1_0_HUB_1Nepal_Qui.xma -o BEC -b 3000 -d 8AE800 -r 1_0_HUB_1Nepal_Qui1.xma
xma_test 1_0_HUB_1Nepal_Qui.xma -o 1BEC -b 3000 -d 8AD800 -r 1_0_HUB_1Nepal_Qui2.xma
xma_test 1_0_HUB_1Nepal_Qui.xma -o 33EC -b 3000 -d 8AC000 -r 1_0_HUB_1Nepal_Qui3.xma

The first two files are created and can perfectly be decoded by adding the XMA stereo header. But the last command gives back the error
"Parse error: skip bits (492) did not match previous frame overflow (0)"
What am I doing wrong here? Thanks for your help!

XMA packets are 0x800 bytes, so if the first stream starts at 0xBEC (as it does here), the next stream starts immediately after it at 0xBEC+0x800 = 0x13EC, then the third stream is at 0x1BEC.

Not that this matters for ripping, but note that this ordering only happens at the beginning of a block for XMA2, and for XMA that's interleaved it only happens at the beginning of the file. The blocking and packet header frame count of XMA2 were introduced to allow faster seeking.

Thanks for your answer. However, could you please post the correct commands for the file I've uploaded? Because I can't get it to work... Thanks.

xma_test 1_0_HUB_1Nepal_Qui.xma -o BEC -b 3000 -d 8AE800 -r out1.bin
xma_test 1_0_HUB_1Nepal_Qui.xma -o 13EC -b 3000 -d 8AE000 -r out2.bin
xma_test 1_0_HUB_1Nepal_Qui.xma -o 1BEC -b 3000 -d 8AD800 -r out3.bin

Just tested with exactly the xma_test.exe in http://hcs64.com/files/xma_parse05.zip
Note that out2.bin is silent, there's no bass.

Oh silly me! I thought the second file was errornous because it's so small. Didn't think of a silent channel. ;)
Thanks! :D
When I've got the time, I'll write a bms script for adding the header including the chunk and size correction and a batch file to automatically split a multichannel XMA to stereo XMAs. =)
Hm, I see that there should be another bms script before to analyse the file. The Ace Combat 6 files need another command because they have another structure.
But all in all, that little parser is INGENIOUS!!! =D

I have one last question: why are the three split files only 2/3 the size of the original file? What's contained in the last third?

The answer is the XMA2 blocking, and the silent stream. If we export as XMA2 we get:
out1.bin.xma2 3039232 bytes
out2.bin.xma2 1517568 bytes
out3.bin.xma2 4200448 bytes
total 8757248 bytes

which is fairly close to the original file size of 9106412 bytes, but it still loses some.

But with XMA packed solid:
out1.bin 2926592 bytes
out2.bin 73728 bytes
out3.bin 3225600 bytes
total 6225920 bytes

The rest goes to padding. There's a few effects at work here:

1) At the beginning of each XMA2 block, which is of a fixed size, the streams must start at the same sample. The streams have to divide up the available packets per block, which can only very poorly approximate their relative bitrates, and with a block size as small as 3000 (only contains 6 packets) this becomes the dominant effect. This forces a minimum bitrate of 1 packet / maximum bitrate of any stream in the block, which is what bloats the silent stream (1517568 vs 73728).

There are even situations where the encoder will insert entire dummy packets into the stream in order to force synchronization (first one in this file is at 1E3EC, it's notable for being all 0s instead of the 1s used for normal packet padding), which is why there is a difference even when just exporting the XMA2 streams; the dummy packets belong to no stream.

2) An XMA2 block must be decodable by itself, so its first packet can't have any bits of overflow from the previous block. This means that the last packet of a frame will have padding after the last complete frame, which you only see at the very end of a XMA stream. This affects all streams, but is most pronounced with the highest bitrate stream, as it has the largest frames (which are more likely to not fit in the last packet), thus 4200448 vs 3225600.

3) As there is an upper limit to the number of frames (64) in an XMA2 packet (a limit which XMA does not have), and a frame is a fixed number of samples (512), even with silence only so many samples can fit into one packet. Thus the silence takes up many more packets that it would in the more tightly packed XMA. This doesn't come into play in this stream due to 1; there must be at least one packet for this stream per block. If I had an XMA2 repack option, though, you'd see it.

The blocking is to enable fast seeking, you can see that this comes at a cost. The silent channel is kind of a degenerate case which exaggerates it, and it's downright irresponsible of them to have wasted space on it, but item 2 above at least effects every blocked XMA2 stream.

OH MY GOD! Thanks a HUGE lot for these explanations, that sums up about all the basic facts one needs to know about the XMA format! :D :D :D

One last question: Where the did you get all this information?

If it wasn't in xma2defs.h I worked it out from parsing these files.

thank you hcs for your usefull information , hope this will be works also for other strange xma files :> thanks