Talk:M2: Difference between revisions

From wowdev
Jump to navigation Jump to search
No edit summary
(Cleaned up information about animation blocks to be at least partially understandable without having to read a shitload of text)
Line 2: Line 2:
--[[User:Schlumpf|schlumpf_]] 00:53, 23 August 2008 (CEST)
--[[User:Schlumpf|schlumpf_]] 00:53, 23 August 2008 (CEST)


== On animations ==
== Animation blocks in WotLK ==
=== old (schlumpf) ===
'''Offset Type Name Description'''
''0x000'' uint16 InterpolationType How the values are interpolated. (0=none, 1=linear, 2=hermite)
''0x002'' uint16 GlobalSequenceID If its made with a global sequence, its in here. Id is it is, -1 if not.
''0x004'' uint32 nTimestamps The number of timestamps. Is the same as the number of keys, most of the time.
''0x008'' uint32 ofsTimestamps Offset to the timestamps in the file.
''0x00C'' uint32 nKeys Number of the keys.
''0x010'' uint32 ofsKeys Values of the block.


The offsets point to a sub-structure now. Its like this:
* Previously M2s used a single-timeline approach, chaining all animations into one long piece and separating them via begin and end given in animation data. Now, each animation has an own timeline.
'''Offset Type Name Description'''
* Animation blocks contain a list of lists of timestamps and a list of lists of values, where the first list is by animation and the second one by timestamp-entry.
''0x000'' uint32 Number It tells us where the values actually are.
''0x004'' uint32 Offset The offset to the data.


The easiest form is if you get a Number == 1 block straight as in single-value AnimationBlocks in some blocks. Particles have only such AnimationBlocks. It just gives you another offset where the data is. Its in this file at the given offset. But if you get a whole list with a lot of stupid values (such as offsets that can't be in the M2 and the .anims), you have a little problem. I don't know why, but Blizzard made some crap here. Most likely to get us confused. Or there is more information I know nothing of.
  template<typename T>
  struct array_ref
  {
    uint32_t number;
    T* elements;  // in file, obviously an offset only
  };
 
  using timestamp_list = array_ref<uint32_t>;
  template<typename T> using value_list<T> = array_ref<T>;
 
  template<typename T>
  struct animation_block
  {
    uint16_t interpolation_type;
    uint16_t global_sequence;
    array_ref<timestamp_list> timestamps;
    array_ref<value_list<T>> values;
  };


What helped me so far was this "simple" conversion: Take nTimestamps / nKeys, substract 1, divide by 3. So out of 0x22 (34) you will get 11. This is the offset to the real sub-structure (if you start with the one pointed to at 0). This offset is then into one of the [[M2/WotLK/.anim|.anim]] files. I think its just "if the first file is full, we will take the next one". I am not sure about this. I have been sleepless for 2 days now and cba to look even more into these blocks.
* Thus, as example, with
 
  struct bone
  {
    int32_t bone_id;
    uint32_t flags;
    int16_t parent_bone;
    uint16_t _1[3];
    animation_block<vec3_float> translation;
    animation_block<quat_short> rotation;
    animation_block<vec3_float> scale;
    vec3_float pivot;
  } b;


The data at the final offset can be of any structure. It can be one float, a short, an integer but too a Vector of three floats or a quaternion which is made in four floats or shorts, depending on the occurence of the main AnimationBlock.
one may get the number of animations having translation information with


  b.translation.timestamps.number


Actually I'd say that we may have gotten this structure:
and the number of timestamps in the first animation using
'''Offset Type Name Description'''
''0x000'' uint16 InterpolationType How the values are interpolated. (0=none, 1=linear, 2=hermite)
''0x002'' uint16 GlobalSequenceID If its made with a global sequence, its in here. Id is it is, -1 if not.
''0x004'' uint32 offsetIntoList1 (This - 1) / 3 is the index into the list at offsetToTimestampsOfs.
''0x008'' uint32 offsetToTimestampsOfs Offset to the list which holds the offset to the data of the timestamps.
''0x00C'' uint32 offsetIntoList2 (This - 1) / 3 is the index into the list at offsetToKeysOfs.
''0x010'' uint32 offsetToKeysOfs Offset to the list which holds the offset to the data of the keys.


What the fuck, Blizzard?
  b.translation.timestamps.elements[0].number


I think the structure is as follows (based on the description by Bartflast, below):
and the first timestamp value of the first animation via
'''Offset Type Name Description'''
''0x000'' uint16 InterpolationType How the values are interpolated. (0=none, 1=linear, 2=hermite)
''0x002'' uint16 GlobalSequenceID If its made with a global sequence, its in here. Id is it is, -1 if not.
''0x004'' uint32 numberOfTimestampPairs Number of timestamp count/offset pairs
''0x008'' uint32 offsetToTimestampPairs Offset to the list which holds the timestamp pairs
''0x00C'' uint32 numberOfKeyFramePairs  Number of key frame count/offset pairs
''0x010'' uint32 offsetToKeyFramePairs  Offset to the list which holds the key frame pairs


The list of timestamp pairs are, again, references to the actual lists of timestamps... double indirection at least... so yes, WTF Blizzard.  I'm pretty sure that the timestamp data from each list is an integer because the data seems to be laid out in a fashion that the timestamp lists and keyframe lists are interleaved within the file. For example:
  b.translation.timestamps.elements[0].elements[0]


<pre>
The actual translation vector for animation 0 at timestamp 0 is at
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:70) - timelines Count/Offset[7,0x1340]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:72) -  timeline #0[13,0x66d0]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:72) -  timeline #1[20,0x8240]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:72) -  timeline #2[0,0x0]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:72) -  timeline #3[18,0xa2d0]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:72) -  timeline #4[39,0xbd30]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:72) -  timeline #5[0,0x0]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:72) -  timeline #6[15,0xff00]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:74) - keyFrames Count/Offset[7,0x1380]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:76) -  keyFrame #0[13,0x6710]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:76) -  keyFrame #1[20,0x8290]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:76) -  keyFrame #2[0,0x0]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:76) -  keyFrame #3[18,0xa320]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:76) -  keyFrame #4[39,0xbdd0]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:76) -  keyFrame #5[0,0x0]
[2009-01-29 08:57:33,953] DEBUG (M2AnimationBlock.java:76) -  keyFrame #6[15,0xff40]
</pre>
<p>
So, in the above example, timeline #0 has 13 timeline entries in it.  If each timeline entry were 4 bytes (an int:32), the next "logical" place to put the next list would be 0x66d0 + (13 entries x 4 bytes per entry) = 0x6704.  Since it seems that they are keeping everything on an even 16 (0x10) byte boundary, that would be 0x6710.  As you can see above, 0x6710 is the exact start of keyframe #0!  For timeline #1, 0x8240 + (20 entries * 4 bytes per entry) = 0x8290 which is the start of keyframe #1!  It makes sense, at least in Blizzard's convoluted data structure.  So while the lists are interleaved, it doesn't actually help us to know that.  However, it did assist me in decoding how each list was stored.  Also, it makes complete sense that the lengths of the two arrays are the same.
<br/><br/>
I'm not yet sure exactly what the keyFrame data is.  I "believe" that it's the data necessary to do the animation but I'm still looking at it.<br/><br/>


Ok, after further investigation I'm now almost certain that yes, the keyFrame data is variable depending on the animation. For example, in the bone structure animation block, the translation animation block is three (3) floats (i.e. 12 bytes per entry in the keyFrame list).  For rotation animation, it's four (4) shorts (i.e. 8 bytes per entry in the keyFrame list -- remember, quaternions are stored as shorts in WotLK).  I have tested several examples of this and it seems to always work out.  I'll post another log entry of an entire bone object that my test application is reading in to try and make it more obvious.  I intend on releasing the code I'm writing as well once it's further along.<br/><br/>
  b.translation.values.elements[0].elements[0]
 
Ok, so I've now got some code that I think correctly reads in the bones from the M2 file and loads the animation blocks.  I cannot seem to create a new page.  If someone would kindly do this for me, perhaps call it AnimationBlockJavaDebug or something, I'll post the debug.  I just don't want to pollute this page with a plethora of debug output.  Sorry.
<br/><br/>
Here is a sample debug output of a timeline for an animation sequence that has a 666ms duration.  There was a bug in a piece of code I had written earlier that was causing the wrong times to be displayed.  It now seems that this is, in fact, the times at which the keyframe data changes.
 
<pre>
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:95) -  timeline #0[13,0x66d0]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/0[0(0x0)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/1[33(0x21)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/2[133(0x85)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/3[166(0xa6)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/4[200(0xc8)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/5[300(0x12c)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/6[333(0x14d)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/7[366(0x16e)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/8[466(0x1d2)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/9[500(0x1f4)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/10[533(0x215)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/11[600(0x258)]
[2009-01-29 13:29:12,000] DEBUG (M2AnimationBlock.java:98) -    time #0/12[666(0x29a)]
</pre>
 
Below is the included keyframe data for the above timeline to show it being read correctly (this is for the bone transformation animation so the key frames are vectors):
<pre>
[2009-01-29 13:56:35,421] DEBUG (M2AnimationBlock.java:108) - keyFrames Count/Offset[7,0x1380]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:42) -  keyFrame #0[13,0x6710]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/0[(-0.0, 0.0, -0.045006532)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/1[(-0.0, -0.0033689167, -0.039352413)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/2[(-0.0, -0.02599839, 0.0037053423)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/3[(-0.0, -0.027641889, 0.009359465)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/4[(-0.0, -0.024762945, 0.0037053423)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/5[(-0.0, -0.0072746114, -0.039352413)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/6[(-0.0, 2.9722223E-5, -0.045006532)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/7[(-0.0, 0.00733375, -0.039352413)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/8[(-0.0, 0.024816971, 0.0037053423)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/9[(-0.0, 0.027692862, 0.009359465)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/10[(-0.0, 0.026044305, 0.0037053423)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/11[(-0.0, 0.010867, -0.025869684)]
[2009-01-29 13:56:35,421] DEBUG (M2VectorAnimationBlock.java:49) -    vector #0/12[(-0.0, 0.0, -0.045006532)]
</pre>
 
One thing of interest is the last frame is always identical to the first frame.  I believe this is mostly for convenience when calculating the interpolation values between frames.  This is generally because the data is calculated by taking the current frame and interpolating it forward against the next frame.
 
=== new and improved (Slartibartfast) ===
Maybe you've already figured it out yourself, chuanhsing, but it still might be helpful as I cannot see any anymations in your latest WMV version:
 
Animation in WotLK is actually easy once you've understood what Blizzard has changed. They have essentially gone from one single, long timeline per model (with every single animation covering a slice of this timeline, based on their start and end times) to a multi-timeline approach where each animation has its own timeline, starting at 0 with the length as defined in the animation sequence data structure.
 
Because of this, the basic animation block structure had to be changed. They dumped the interpolation ranges (as those aren't necessary anymore if each animation has its own timeline), and they have added another layer of references for the timestamp and value information. This means instead of the number of timestamps followed by the offset to said timestamps there's now the number of timelines followed by an offset to the timeline data. This offset leads to a structure of 8 bytes per timeline, with the first 4 bytes making up the actual number of timestamps for one specific timeline and the next 4 being an offset to the actual timestamp data. The keyframe values are stored in a similar fashion.
 
All of this of course does not account for the new .anim files which seem to be single animations that aren't stored in the .m2 file itself - I haven't looked at those yet, so I can't give you any hints about their structure. But the changes described above will get all animations stored in the file itself to work fine, which is the vast majority of animations.
 
 
Okay, now I think I've understood how those .anim-files fit into the picture. Blizzard has created them to externalize seldomly-needed animations such as dance animations from the main model file, probably to speed up the loading process by loading them on-demand when they are to be played.
 
Their structure is simple: they don't have any structure on their own. They are just a pile of binary data, which is being referenced from the animation sub-blocks (I just decided to call those sub-structures in the animation blocks that contain the actual keyframe data for a single animation "animation sub-blocks", see my last post for details on this) in the exact same way as data in the main model file is being referenced: by offsets to timestamp/keyframe data. These references just point into the .anim-file corresponding to a specific animation id if an animation is stored externally.
 
The names of the animation data files are composed from the following template: [model file name][animation id]-[animation sub-id].anim
 
However, there's still one thing I don't understand: how someone can decide if an animation sequence is stored internally (in the model file) or externally. Of course it's possible to check if a corresponding .anim file exists, but I don't think that this is how WoW actually does it as that would result in quite some overhead. That information is probably stored somewhere in the main model file, but I haven't found it yet...
 
* Posted by Slartibartfast at the [http://www.wowmodelviewer.org/forum/index.php?topic=3672.40 wowmodelviewer-forums].
 
=== some observations (Shlainn) ===
 
I think I figured out a part of the "internal vs .anim" problem.
* All animations which have flags & 0x20 are stored internally.
* Animations which do not have flags & 0x20 are not stored internally.
* Animations which do not have flags & 0x20 AND do not have flags & 0x40 are in .anim files
* Animations which do not have flags & 0x20 AND DO have flags 0x40 are stored... somewhere. I have no clue.
 
<pre>
flags & 0x20 | flags & 0x40 | Storage location
-------------+--------------+-----------------
    true    |    any      | M2 file
    false    |    false    | anim files
    false    |    true    | somewhere
</pre>
== Animations in some beta-code ==
 
animated.h
void init(AnimationBlock &b, MPQFile &f, int *gs)
 
original code
<pre>
    uint32 *ptimes = (uint32*)(f.getBuffer() + b.ofsTimes);
    for (size_t i=0; i<b.nTimes; i++)
      times.push_back(ptimes[i]);
   
    // keyframes
    assert((D*)(f.getBuffer() + b.ofsKeys));
    D *keys = (D*)(f.getBuffer() + b.ofsKeys);
    switch (type) {
      case INTERPOLATION_NONE:
      case INTERPOLATION_LINEAR:
        for (size_t i=0; i<b.nKeys; i++)
          data.push_back(Conv::conv(keys[i]));
        break;
      case INTERPOLATION_HERMITE:
        for (size_t i=0; i<b.nKeys; i++) {
          data.push_back(Conv::conv(keys[i*3]));
          in.push_back(Conv::conv(keys[i*3+1]));
          out.push_back(Conv::conv(keys[i*3+2]));
        }
        break;
    }
</pre>
 
modified code
 
<pre>
if( b.nTimes == 0 )
return;
 
for(size_t j=0; j < b.nTimes; j++) {
AnimationBlockHeader* pHeadTimes = (AnimationBlockHeader*)(f.getBuffer() + b.ofsTimes + j*sizeof(AnimationBlockHeader));
uint32 *ptimes = (uint32*)(f.getBuffer() + pHeadTimes->ofsEntrys);
for (size_t i=0; i < pHeadTimes->nEntrys; i++)
times[j].push_back(ptimes[i]);
}
 
// keyframes
for(size_t j=0; j < b.nKeys; j++) {
AnimationBlockHeader* pHeadKeys = (AnimationBlockHeader*)(f.getBuffer() + b.ofsKeys + j*sizeof(AnimationBlockHeader));
 
D *keys = (D*)(f.getBuffer() + pHeadKeys->ofsEntrys);
switch (type) {
case INTERPOLATION_NONE:
case INTERPOLATION_LINEAR:
for (size_t i = 0; i < pHeadKeys->nEntrys; i++)
data[j].push_back(Conv::conv(keys[i]));
break;
case INTERPOLATION_HERMITE:
for (size_t i = 0; i < pHeadKeys->nEntrys; i++) {
data[j].push_back(Conv::conv(keys[i*3]));
in[j].push_back(Conv::conv(keys[i*3+1]));
out[j].push_back(Conv::conv(keys[i*3+2]));
}
break;
}
}
</pre>
 
modelheaders.h
 
  struct AnimationBlockHeader
  {
    uint32 nEntrys;
    uint32 ofsEntrys;
  };


Igor:
* Some timestamps/values.elements entries may have number/elements = 0, if for that animation id no animation is given.
Yeap, it does work but not for all animations. The pHeadTimes->ofsEntrys can be 0 or an offset that points to absurd timestamps, same for keyframes. For nKeys/nTimes even by 3 or nope as well. Be honest, we don't know how to solve this puzzle ;)
* .anim files are just a blob of data which may as well be in the main model file, that is pointed to by the first array_ref layer.
* [model file name][animation id]-[animation sub-id].anim
* it seems like it is possible to detect if animation data is stored in-m2 or externally via
** All animations which have flags & 0x20 are stored internally.
** Animations which do not have flags & 0x20 are not stored internally.
** Animations which do not have flags & 0x20 AND do not have flags & 0x40 are in .anim files
** Animations which do not have flags & 0x20 AND DO have flags 0x40 are stored... somewhere. I have no clue.


== Deleted Blocks ==
== Deleted Blocks ==

Revision as of 14:31, 4 March 2014

Finished for Build 8820 now. Post changes here first! And please pay attention to the style when you add something.. --schlumpf_ 00:53, 23 August 2008 (CEST)

Animation blocks in WotLK

  • Previously M2s used a single-timeline approach, chaining all animations into one long piece and separating them via begin and end given in animation data. Now, each animation has an own timeline.
  • Animation blocks contain a list of lists of timestamps and a list of lists of values, where the first list is by animation and the second one by timestamp-entry.
 template<typename T>
 struct array_ref
 {
   uint32_t number;
   T* elements;  // in file, obviously an offset only
 };
 
 using timestamp_list = array_ref<uint32_t>;
 template<typename T> using value_list<T> = array_ref<T>;
 
 template<typename T>
 struct animation_block
 {
   uint16_t interpolation_type;
   uint16_t global_sequence;
   array_ref<timestamp_list> timestamps;
   array_ref<value_list<T>> values;
 };
  • Thus, as example, with
 struct bone
 {
   int32_t bone_id;
   uint32_t flags;
   int16_t parent_bone;
   uint16_t _1[3];
   animation_block<vec3_float> translation;
   animation_block<quat_short> rotation;
   animation_block<vec3_float> scale;
   vec3_float pivot;
 } b;

one may get the number of animations having translation information with

 b.translation.timestamps.number

and the number of timestamps in the first animation using

 b.translation.timestamps.elements[0].number

and the first timestamp value of the first animation via

 b.translation.timestamps.elements[0].elements[0]

The actual translation vector for animation 0 at timestamp 0 is at

 b.translation.values.elements[0].elements[0]
  • Some timestamps/values.elements entries may have number/elements = 0, if for that animation id no animation is given.
  • .anim files are just a blob of data which may as well be in the main model file, that is pointed to by the first array_ref layer.
  • [model file name][animation id]-[animation sub-id].anim
  • it seems like it is possible to detect if animation data is stored in-m2 or externally via
    • All animations which have flags & 0x20 are stored internally.
    • Animations which do not have flags & 0x20 are not stored internally.
    • Animations which do not have flags & 0x20 AND do not have flags & 0x40 are in .anim files
    • Animations which do not have flags & 0x20 AND DO have flags 0x40 are stored... somewhere. I have no clue.

Deleted Blocks

Block D

  • nD records of (int16, int16) starting at ofsD

Maybe a lookup table for animations? Since the numbers happen to be in fixed positions. The first short seems to increase with the position for models with all animations (like characters), the second seems to be flags or a modifier? Or something.


Contain indices into the texture animations list, or -1 meaning a static texture.

Texture animations

  • This block contains definitions for texture animations, for example, flowing water or lava in some models. The keyframe values are used in the texture transform matrix.

nTexAnims records of 0x54 bytes starting at ofsTexAnims, followed by data referenced in these records.

Offset 	Type 						Description
0x00 	AnimationBlock (float, float, float) 		Translation
0x1C 	AnimationBlock (float, float, float ???) 	Rotation?
0x38 	AnimationBlock (float, float, float) 		Scaling?

The three subrecords specify texture transforms. Translation seems to work, producing nice flowing lava and waterfalls.