M2/Loading

Shader Substitution

The following information is relevant for M2s in the Wrath of the Lich King and Cataclyms eras. See the notes for each expansion below.

It appears the shader substitution logic was completely removed in Mists of Pandaria, and no longer exists in the client.

Cata Notes

In Cataclysm, the shader substitution logic only runs for M2s with a version <= 264.

It seems Blizzard moved the shader substitution logic to their tooling at some point during Cataclysm, but kept the substitution paths around in the event an older M2 was loaded by the client.

WotLK Notes

In Wrath of the Lich King, the value on disk for M2Batch->shader is not the value used at runtime to select the appropriate shader effect.

Rather, the game runs through a series of substitution code paths, after which, the value is used to select the right effect from the BLS files.

These substitution functions are called from CM2Shared::InitializeSkinProfile, and are always run.

Typically, the value on disk for M2Batch->shader is 0, with one large exception: if flag 0x08 is set in the M2, the value in M2Batch->shader is actually an index offset inside the M2TextureCombinerCombos table.

Bit Layout of M2Batch->shader After Substitution

After shader substitution, the layout of M2Batch->shader is as follows:

Bits	Notes
0, 1, 2	Texture 2 Combiner Mode
3	Texture 2 Env Mapped
4, 5, 6	Texture 1 Combiner Mode
7	Texture 1 Env Mapped
8, 9, 10, 11, 12, 13	Unknown
14	Set for T2 coords??
15	Skips simple substitution??

CM2Shared::SubstituteSimpleShaders

The following is reversed from 3.3.5a (12340):

void __fastcall CM2Shared::SubstituteSimpleShaders(CM2Shared *this) {
  uint32_t batchCount = this->skinProfile->batches.count;

  if (batchCount == 0) {
    return;
  }

  uint32_t batchIndex;

  for (batchIndex = 0; batchIndex < batchCount; ++batchIndex) {
    M2Batch batch = this->skinProfile->batches.data[batchIndex];

    if (batch->shader & 0x8000) {
      continue;
    }

    M2Material material = &this->data->materials.data[batch->materialIndex];

    // Flag 0x08: use_combiner_combos
    if (this->data->flags & 0x08) {
      uint32_t textureCombinerComboIndex = batch->shader;
      uint32_t textureCoordComboIndex = batch->textureCoordComboIndex;

      batch->shader = 0;

      uint16_t shader[2];

      uint32_t textureCount = batch->textureCount;
      uint32_t textureIndex;

      for (textureIndex = 0; textureIndex < textureCount; ++textureIndex) {
        bool isFirstTexture = textureIndex == 0;
        bool isLastTexture = textureIndex == textureCount - 1;

        uint16_t textureCombiner = 0;

        // If this is the first texture and the batch material's blending mode is opaque,
        // override the combiner mode to opaque; otherwise, use the combiner mode from the
        // combiner combos
        if (isFirstTexture && material->blendMode == 0) {
          textureCombiner = 0;
        } else {
          textureCombiner = this->data->textureCombinerCombos.data[textureCombinerComboIndex + textureIndex];
        }

        shader[textureIndex] = textureCombiner;

        uint16_t textureCoord = this->data->textureCoordCombos.data[textureCoordComboIndex + textureIndex];

        // If the texture coord is env, set env bit for texture
        if (textureCoord > 2u) {
           shader[textureIndex] |= 8u;
        }

        // If this is the last texture and the texture coord is T2, enable bit 15
        if (isLastTexture && textureCoord == 1) {
          batch->shader |= 0x4000u;
        }
      }

      batch->shader |= shader[0] << 4 | shader[1];
    } else {
      uint16_t shader = 0;

      uint16_t textureCombiner = 0;

      // If the material blend mode is opaque, force the combiner to opaque; otherwise,
      // default combiner to mod
      if (material->blendMode == 0) {
        textureCombiner = 0;
      } else {
        textureCombiner = 1;
      }

      shader = textureCombiner;

      uint16_t textureCoord = this->data->textureCoordCombos.data[textureCoordComboIndex];

      // If the texture coord is env, set env bit for texture
      if (textureCoord > 2u) {
        shader |= 8u;
      }

      // If the texture coord is T2, enable bit 15
      if (textureCoord == 1) {
        batch->shader |= 0x4000;
      }

      batch->shader |= shader << 4;
    }
  }
}

CM2Shared::SubstituteSpecializedShaders

TODO