M2/Loading

Shader Substitution

Cata Notes

In Cataclysm 4.1.0, and perhaps all of Catacylsm, the shader substitution logic outlined below only runs for M2s with a version <= 264.

It seems Blizzard moved the shader substitution logic to their tooling at some point during Cataclysm (TODO: when?), but kept the substitution paths around in the event an older M2 was loaded by the client.

WotLK Notes

In Wrath of the Lich King, the value on disk for M2Batch->shader is not the value used at runtime to select the appropriate shader effect.

Rather, the game runs through a series of substitution code paths, after which, the value is used to select the right effect from the BLS files.

Typically, the value on disk for M2Batch->shader is 0, with one large exception: if flag 0x08 is set in the M2, the value in M2Batch->shader is actually an index offset inside the M2TextureCombinerCombos table.

Bit Layout of M2Batch->shader After Substitution

After shader substitution, the layout of M2Batch->shader is as follows:

Bits	Notes
0, 1, 2	Texture 2 Combiner Mode
3	Texture 2 Env Mapped
4, 5, 6	Texture 1 Combiner Mode
7	Texture 1 Env Mapped
8, 9, 10, 11, 12, 13	Unknown
14	Set for T2 coords??
15	Skips simple substitution??

CM2Shared::SubstituteSimpleShaders

The following is reversed from 3.3.5a (12340):

void __fastcall CM2Shared::SubstituteSimpleShaders(CM2Shared *this) {
  uint32_t batchCount = this->skinProfile->batches.count;

  if (batchCount == 0) {
    return;
  }

  uint32_t batchIndex;

  for (batchIndex = 0; batchIndex < batchCount; ++batchIndex) {
    M2Batch batch = this->skinProfile->batches.data[batchIndex];

    if (batch->shader & 0x8000) {
      continue;
    }

    M2Material material = &this->data->materials.data[batch->materialIndex];

    // Flag 0x08: use_combiner_combos
    if (this->data->flags & 0x08) {
      uint32_t textureCombinerComboIndex = batch->shader;
      uint32_t textureCoordComboIndex = batch->textureCoordComboIndex;

      batch->shader = 0;

      uint16_t shader[2];

      uint32_t textureCount = batch->textureCount;
      uint32_t textureIndex;

      for (textureIndex = 0; textureIndex < textureCount; ++textureIndex) {
        bool isFirstTexture = textureIndex == 0;
        bool isLastTexture = textureIndex == textureCount - 1;

        uint16_t textureCombiner = 0;

        // If this is the first texture and the batch material's blending mode is opaque,
        // override the combiner mode to opaque; otherwise, use the combiner mode from the
        // combiner combos
        if (isFirstTexture && material->blendMode == 0) {
          textureCombiner = 0;
        } else {
          textureCombiner = this->data->textureCombinerCombos.data[textureCombinerComboIndex + textureIndex];
        }

        shader[textureIndex] = textureCombiner;

        uint16_t textureCoord = this->data->textureCoordCombos.data[textureCoordComboIndex + textureIndex];

        // If the texture coord is env, set env bit for texture
        if (textureCoord > 2u) {
           shader[textureIndex] |= 8u;
        }

        // If this is the last texture and the texture coord is T2, enable bit 15
        if (isLastTexture && textureCoord == 1) {
          batch->shader |= 0x4000u;
        }
      }

      batch->shader |= shader[0] << 4 | shader[1];
    } else {
      uint16_t shader = 0;

      uint16_t textureCombiner = 0;

      // If the material blend mode is opaque, force the combiner to opaque; otherwise,
      // default combiner to mod
      if (material->blendMode == 0) {
        textureCombiner = 0;
      } else {
        textureCombiner = 1;
      }

      shader = textureCombiner;

      uint16_t textureCoord = this->data->textureCoordCombos.data[textureCoordComboIndex];

      // If the texture coord is env, set env bit for texture
      if (textureCoord > 2u) {
        shader |= 8u;
      }

      // If the texture coord is T2, enable bit 15
      if (textureCoord == 1) {
        batch->shader |= 0x4000;
      }

      batch->shader |= shader << 4;
    }
  }
}

CM2Shared::SubstituteSpecializedShaders

TODO