M2/Loading

Shader Substitution

The following information is relevant for M2s in the Wrath of the Lich King and Cataclyms eras. See the notes for each expansion below.

It appears the shader substitution logic was completely removed in Mists of Pandaria, and no longer exists in the client.

Cata Notes

In Cataclysm, the shader substitution logic only runs for M2s with a version <= 264.

It seems Blizzard moved the shader substitution logic to their tooling at some point during Cataclysm, but kept the substitution paths around in the event an older M2 was loaded by the client.

WotLK Notes

In Wrath of the Lich King, the value on disk for M2Batch->shader is not the value used at runtime to select the appropriate shader effect.

Rather, the game runs through a series of substitution code paths, after which, the value is used to select the right effect from the BLS files.

These substitution functions are called from CM2Shared::InitializeSkinProfile, and are always run.

Typically, the value on disk for M2Batch->shader is 0, with one large exception: if flag 0x08 is set in the M2, the value in M2Batch->shader is actually an index offset inside the M2TextureCombinerCombos table.

Bit Layout of M2Batch->shader After Substitution

After shader substitution, the layout of M2Batch->shader is as follows:

Bits	Notes
0, 1, 2	Texture 2 Combiner Mode
3	Texture 2 Env Mapped
4, 5, 6	Texture 1 Combiner Mode
7	Texture 1 Env Mapped
8, 9, 10, 11, 12, 13	Unused??
14	Set for T2 coords??
15	Advanced Effect

Note that if advanced effect is set, bits 0 - 14 simply hold a single number, which is used to look up the advanced effect in CM2Shared::GetEffect.

CM2Shared::SubstituteSimpleShaders

The following is reversed from 3.3.5a (12340).

void __fastcall CM2Shared::SubstituteSimpleShaders(CM2Shared *this) {
  uint32_t batchCount = this->skinProfile->batches.count;

  if (batchCount == 0) {
    return;
  }

  uint32_t batchIndex;

  for (batchIndex = 0; batchIndex < batchCount; ++batchIndex) {
    M2Batch batch = this->skinProfile->batches.data[batchIndex];

    if (batch->shader & 0x8000) {
      continue;
    }

    M2Material material = &this->data->materials.data[batch->materialIndex];

    // Flag 0x08: use_combiner_combos
    if (this->data->flags & 0x08) {
      uint32_t textureCombinerComboIndex = batch->shader;
      uint32_t textureCoordComboIndex = batch->textureCoordComboIndex;

      batch->shader = 0;

      uint16_t shader[2];

      uint32_t textureCount = batch->textureCount;
      uint32_t textureIndex;

      for (textureIndex = 0; textureIndex < textureCount; ++textureIndex) {
        bool isFirstTexture = textureIndex == 0;
        bool isLastTexture = textureIndex == textureCount - 1;

        uint16_t textureCombiner;

        // If this is the first texture and the batch material's blending mode is opaque,
        // override the combiner mode to opaque; otherwise, use the combiner mode from the
        // combiner combos
        if (isFirstTexture && material->blendMode == M2BLEND_OPAQUE) {
          textureCombiner = M2COMBINER_OPAQUE;
        } else {
          textureCombiner = this->data->textureCombinerCombos.data[textureCombinerComboIndex + textureIndex];
        }

        shader[textureIndex] = textureCombiner;

        uint16_t textureCoord = this->data->textureCoordCombos.data[textureCoordComboIndex + textureIndex];

        // If the texture coord is env, set env bit for texture
        if (textureCoord > 2u) {
           shader[textureIndex] |= M2COMBINER_ENVMAP;
        }

        // If this is the last texture and the texture coord is T2, enable bit 15
        if (isLastTexture && textureCoord == 1) {
          batch->shader |= 0x4000u;
        }
      }

      batch->shader |= shader[0] << 4 | shader[1];
    } else {
      uint16_t shader = 0;

      uint16_t textureCombiner;

      // If the material blend mode is opaque, force the combiner to opaque; otherwise,
      // default combiner to mod
      if (material->blendMode == M2BLEND_OPAQUE) {
        textureCombiner = M2COMBINER_OPAQUE;
      } else {
        textureCombiner = M2COMBINER_MOD;
      }

      shader = textureCombiner;

      uint16_t textureCoord = this->data->textureCoordCombos.data[textureCoordComboIndex];

      // If the texture coord is env, set env bit for texture
      if (textureCoord > 2u) {
        shader |= M2COMBINER_ENVMAP;
      }

      // If the texture coord is T2, enable bit 15
      if (textureCoord == 1) {
        batch->shader |= 0x4000;
      }

      batch->shader |= shader << 4;
    }
  }
}

CM2Shared::SubstituteSpecializedShaders

TODO

Shader Selection

When the game loads an M2 skin, CShaderEffects are created for all M2Batches based on the value in M2Batch->shader.

See below for the various functions involved.

CM2Shared::GetEffect

The following is reversed from 3.3.5a (12340).

CShaderEffect* __thiscall CM2Shared::GetEffect(CM2Shared *this, M2Batch *batch) {
  CShaderEffect* effect;

  // Simple effect
  if (!(batch->shader & 0x8000)) {
    effect = CM2Shared::CreateSimpleEffect(this, batch->textureCount, batch->shader, batch->textureCoordComboIndex);

    // Fallback
    // 0000000000010001
    // 1 texture: Combiners_Mod
    // 2 texture: Combiners_Mod_Mod
    if (!effect) {
      effect = CM2Shared::CreateSimpleEffect(this, batch->textureCount, 0x11u, batch->textureCoordComboIndex);
    }

    return effect;
  }

  // Advanced effect
  char* vsName = 0;
  char* psName = 0;
  uint32_t colorOp = 0;
  uint32_t alphaOp = 0;

  switch (batch->shader & 0x7FFF) {
    case 0u:
      return NULL;
    case 1u:
      vsName = "Diffuse_T1_Env";
      psName = "Combiners_Opaque_Mod2xNA_Alpha";
      colorOp = 0;
      alphaOp = 3;
      break;
    case 2u:
      vsName = "Diffuse_T1_Env";
      psName = "Combiners_Opaque_AddAlpha";
      colorOp = 0;
      alphaOp = 3;
      break;
    case 3u:
      vsName = "Diffuse_T1_Env";
      psName = "Combiners_Opaque_AddAlpha_Alpha";
      colorOp = 0;
      alphaOp = 3;
      break;
    default:
      break;
  }

  char* effectName;

  // Create effect name for hashing
  strcpy(&effectName, vsName);
  strcat(&effectName, psName);

  effect = CShaderEffectManager::GetEffect(&effectName);

  if (!effect) {
    effect = CShaderEffectManager::CreateEffect(&effectName);

    CShaderEffect::InitEffect(effect, vsName, psName);

    CShaderEffect::InitFixedFuncPass(effect, &colorOp, &alphaOp, 1);

    // TODO verify
    if (!effect) {
      effect = CM2Shared::CreateSimpleEffect(this, batch->textureCount, 0x11u, batch->textureCoordComboIndex);
    }
  }

  return effect;
}

CM2Shared::CreateSimpleEffect

The following is reversed from 3.3.5a (12340).

CShaderEffect *__thiscall CM2Shared::CreateSimpleEffect(CM2Shared *this, uint32_t textureCount, uint16_t shader, uint16_t textureCoordComboIndex) {
  uint32_t combiner[2];
  uint32_t env[2];

  combiner[0] = (shader >> 4) & M2COMBINER_OP_MASK;  // T1 Combiner
  combiner[1] = (shader >> 0) & M2COMBINER_OP_MASK;  // T2 Combiner
  envmap[0]   = (shader >> 4) & M2COMBINER_ENVMAP;   // T1 Env Mapped
  envmap[1]   = (shader >> 0) & M2COMBINER_ENVMAP;   // T2 Env Mapped

  char* vsName;
  char* psName;

  // 1 texture
  if (textureCount == 1) {
    if (envmap[0]) {
      vsName = "Diffuse_Env";
    } else {
      if (this->data->textureCoordCombos.data[textureCoordIndex] === 0) {
        vsName = "Diffuse_T1";
      } else {
        vsName = "Diffuse_T2";
      }
    }

    switch (combiner[0]) {
      case 0u:
        psName = "Combiners_Opaque";
        break;
      case 1u:
        psName = "Combiners_Mod";
        break;
      case 2u:
        psName = "Combiners_Decal";
        break;
      case 3u:
        psName = "Combiners_Add";
        break;
      case 4u:
        psName = "Combiners_Mod2x";
        break;
      case 5u:
        psName = "Combiners_Fade";
        break;
      default:
        psName = "Combiners_Mod";
        break;
    }
  // 2 textures
  } else {
    if (envmap[0] && envmap[1]) {
      vsName = "Diffuse_Env_Env";
    } else if (envmap[0]) {
      vsName = "Diffuse_Env_T2";
    } else if (envmap[1]) {
      vsName = "Diffuse_T1_Env";
    } else {
      vsName = "Diffuse_T1_T2";
    }

    switch (combiner[0]) {
      case 0u:
        switch (combiner[1]) {
          case 0u:
            psName = "Combiners_Opaque_Opaque";
            break;
          case 3u:
            psName = "Combiners_Opaque_Add";
            break;
          case 4u:
            psName = "Combiners_Opaque_Mod2x";
            break;
          case 6u:
            psName = "Combiners_Opaque_Mod2xNA";
            break;
          case 7u:
            psName = "Combiners_Opaque_AddNA";
            break;
          default:
            psName = "Combiners_Opaque_Mod";
            break;
        }
      case 1u:
        switch (combiner[1]) {
          case 0u:
            psName = "Combiners_Mod_Opaque";
            break;
          case 1u:
            psName = "Combiners_Mod_Mod";
            break;
          case 3u:
            psName = "Combiners_Mod_Add";
            break;
          case 4u:
            psName = "Combiners_Mod_Mod2x";
            break;
          case 6u:
            psName = "Combiners_Mod_Mod2xNA";
            break;
          case 7u:
            psName = "Combiners_Mod_AddNA";
            break;
          default:
            psName = "Combiners_Mod_Mod";
            break;
        }
      case 3u:
        switch (combiner[1]) {
          case 1u:
            psName = "Combiners_Add_Mod";
            break;
          default:
            return NULL;
            break;
        }
      case 4u:
        switch (combiner[1]) {
          case 4u:
            psName = "Combiners_Mod2x_Mod2x";
            break;
          default:
            return NULL;
            break;
        }
    }
  }

  char* effectName;

  // Create effect name for hashing
  strcpy(&effectName, vsName);
  strcat(&effectName, psName);

  CShaderEffect* effect;

  effect = CShaderEffectManager::GetEffect(&effectName);

  if (!effect) {
    effect = CShaderEffectManager::CreateEffect(&effectName);

    CShaderEffect::InitEffect(effect, vsName, psName);

    // M2GetCombinerOps (inlined)
    for (int textureIndex = 0; textureIndex < textureCount; textureIndex++) {
      colorOps[textureIndex] = colorOpTable[combiner[textureIndex]];
      alphaOps[textureIndex] = alphaOpTable[combiner[textureIndex]];
    }

    CShaderEffect::InitFixedFuncPass(effect, colorOps, alphaOps, textureCount);
  }

  return effect;
}