BLS

From wowdev
Revision as of 18:11, 16 November 2018 by Barncastle (talk | contribs) (→‎BLS v1.3 (WotLK): added pre-WotLK v1.3)
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

BLS is the container format that stores the GPU shaders used to render the world. In WoD, there are now four different shader types under the Shaders\* directory.

  • Vertex
    • (versions: arbvp1, vp40, glvs_150, ps_2_0, ps_3_0, ps_4_0, ps_5_0)
  • Fragment
    • (versions: arbfp1, fp40, glfs_150, ps_2_0, ps_3_0, ps_4_0, ps_5_0)
  • Geometry
    • (versions: glgs_150, gs_4_0, gs_5_0)
  • Hull/Domain (equivalent to Tessellation in OpenGL, except WoW currently only has these shaders for DX)
    • (versions: ds_5_0/hs_50)

In MoP+, the BLS format changed significantly (the version increased from 1.3 to 1.4), a new header has been introduced and all of the shader text inside the BLS files is now contained inside compressed chunks. Very little appears to be known about the structure of the format itself, and the meaning of the various fields. Below is enough information to be able to extract the compressed chunks and read out shader text from the various shader components. Additionally, the Shaders\Effects folder contains WFX files that allow WoW to map shader types from models onto the various shader components (although, there are occasions in the M2 source code where shaders are picked in code).

BLS v1.4 (Cata+)

This section only applies to versions ≥ Cata.

The BLS format changed with the release of Cata, the version number of the file was bumped and all BLS files have the same "GXSH" magic signature. These new v1.4 files now contain compressed shader text but extra steps are required to be able to decode the text.

Header

struct BLSHeader {
/*0x00*/	char magic[4];			// FourCC-style magic, "GXVS", "GXPS", etc. Character order reversed in-file. "GXSH" in all new WoD shaders.
/*0x04*/	uint32_t version;		// version, 0x10003 in WotLK, 0x10004 in Cata/MoP/WoD, 0x10005 in BfA (25902), 0x10006 in BfA (25976)
/*0x08*/	uint32_t permutationCount;	// (from old definition)
/*0x0c*/	uint32_t nShaders;		// the expected number of shaders in this file (this is the number
						// of distinct shaders that will be found if you inspect the file after decompression)
/*0x10*/	uint32_t ofsCompressedChunks;	// offset to array of offsets to compressed chunks
/*0x14*/	uint32_t nCompressedChunks;	// number of compressed chunks
/*0x18*/	uint32_t ofsCompressedData;	// offset to the start of the compressed (zlib) data 
						// bytes (all offsets from the array above are offset by this much)

This header starts at 0x00 in the BLS file. It is followed by an BLSHeader.nShaders amount of uint32 offsets pointing to the start of each shader block in the decompressed data. Said data is at ofsCompressedData and must be decompressed with zlib inflate. There are multiple chunks of separately compressed zlib data (all sections start with 78 9C, the zlib magic header for default compression). To decompress the file, take the ofsCompressedData offset, then for each chunk read the offset from ofsCompressedChunks:

(unsigned char *)ptr + header->ofsCompressedData + ofsCompressedChunks[index]

Each decompressed chunk should be appended to the last. Some of the shader text spans multiple chunks. The offsets just help with finding the start of the zlib header for each chunk. Inside the decompressed data, the old WotLK block header is present, along with a shader preamble (WoD+).

Shader Block

The decompressed BLS data is formatted in the same format that old WotLK shaders used, with the exception of WoD shaders that have a new (important) structure that is needed to find the start of the shader text. It no longer appears to be true that the top-level permutationCount field represents the number of shaders present in the inner BLS data.

Note: all the instances of BLSBlock are padded to the next nearest 4-byte alignment.

Note: In BFA? 3 fields of this struct were removed.

struct BLSBlock {
/*0x00*/	uint32_t flags;
/*0x04*/	uint32_t flags2;
/*0x08*/	uint32_t unknown;
/*0x0c*/	uint32_t unknown2;
/*0x10*/	uint32_t unknown3;
/*0x14*/	uint32_t unknown4;
/*0x18*/	uint32_t unknown5;
/*0x1c*/	uint32_t len;
}

The length field dictates where this block ends, in order to iterate over all blocks in BLS file you would start at the first block, read len and then skip to the next block using that. In MoP, the shader text was present immediately after the BLSBlock structure. However, in WoD there is a significant amount of new data between the end of this structure and the beginning of the shader text. Experimentally, there appears to be a header in front of this data that holds the offset to the text itself.

The content of the decompressed data depends on the shader type. The ShaderDataHeader below is for 'GLS2', but not verified for any other type.

struct ShaderDataHeader_GLS2 {
  uint32_t magic; // 'GLS2'
  uint32_t size;
  uint32_t shaderType; // GLShader::ShaderType
  uint32_t type2; // ?, 3 = GLSL
  uint32_t target; // as in GL_FRAGMENT_SHADER
  uint32_t codeOffset;
  uint32_t codeSize;
  uint8_t _unk1C[0x8];
  uint32_t some_length;
  uint32_t some_offset;
  uint8_t _unk2C[0x14]
  uint32_t some_other_count;
  uint32_t ofsUniformMapTable; // points to uint32_t[] entries; which points to shader_uniform_info_ts
  uint32_t nUniformMapTable;
};
struct shader_uniform_info_t
{
  uint32_t _unk0;
  uint8_t _unk4;
  uint8_t uniformClass; // 4 = struct
  uint8_t _unk6;
  uint8_t enabled;
  uint16_t shader_constant_index;
  uint16_t mapIndex;
  uint16_t shader_constant_count;
};


Rough structure of GLS3 header:

 struct ShaderDataHeader_GLS3 {
  uint32_t magic; // 'GLS3'
  uint32_t size;
  uint32_t type2; // ?, 3 = GLSL
  uint32_t unk1;
  uint32_t target;
  uint32_t codeOffset;
  uint32_t codeSize;
  uint32_t unk2;
  uint32_t unk3;//-1
  uint32_t inputParamsOffset; //offset to array of input_shader_uniform_info_t
  uint32_t inputParamCount;
  uint32_t outputOffset; // offset to array of output_shader_uniform_info_t
  uint32_t outputCount;
  uint32_t uniformBufferOffset; // offset to array of uniformBuffer_shader_uniform_info_t
  uint32_t uniformBufferCount;
  uint32_t samplerUniformsOffset; //offset to sampler_shader_uniform_info_t
  uint32_t samplerUniformsCount;
  uint32_t unk5Offset;
  uint32_t unk5Count;
  uint32_t unk6Offset;
  uint32_t unk6Count;
  uint32_t variableStringsOffset;
  uint32_t variableStringsSize;
};
struct input_shader_uniform_info_t
{
  uint32_t glslParamNameOffset; //Offset to zero terminated string
  uint32_t unk0;
  uint32_t internalParamNameOffset;
  uint32_t unk1;
};
struct output_shader_uniform_info_t
{
  uint32_t glslParamNameOffset; //Offset to zero terminated string
  uint32_t unk0;
  uint32_t internalParamNameOffset;
  uint32_t unk1;
};
struct uniformBuffer_shader_uniform_info_t
{
  uint32_t glslParamNameOffset; //Offset to zero terminated string
  uint32_t unk0;
  uint32_t unk1;
}
struct sampler_shader_uniform_info_t
{
  uint32_t glslParamNameOffset; //Offset to zero terminated string
  uint32_t unk0;
  uint32_t unk1;
  uint32_t unk2;
};

The shader text starts at offset of (BLSBlock + BLSBlock2.offset). The shader text is then present in the format used by the particular graphics API used in that subdirectory. (ie. glvs/glfs == OpenGL GLSL, arbvp1/arbfp1 == OpenGL ARB assembly, vs_*/ps_* == DirectX, etc).

For GLS3 it's not guaranteed that start of next BLSBlock will be at (BLSBlock + BLSBlock.len).


Note: in 6.1.2 the OpenGL GLSL (glvs/glfs/etc) shaders appear to have been recompiled with a newer version of the HLSL cross-compiler and they are exceptionally informative. In particular, there are named uniform (constant) blocks that indicate the meaning of the variables.

BLS v1.3 (WotLK)

This section only applies to versions PreVanilla (0.7.0.3694) … Wrath.

Part of the early WotLK development included an overhaul to the BLS format introducing the first iteration of the v1.4 structure. However; this change did not include a version change. Luckily, this change was already in place by the first publicly available build so v1.3 can be simplified to WotLk and pre-WotLK.

Header

This section only applies to version Wrath.
  • Main header (0xC bytes)

This header is in all files - pixel and vertex shaders in all profiles.

struct BLSHeader {
/*0x00*/	char[4] magix;		// in reverse character order: "SVXG" in case of a vertex shader, "SPXG" in case of a fragment shader
/*0x04*/	uint32 version;		// Always 0x10003 - version 1.3 of format
/*0x08*/	uint32 permutationCount;
/*0x0C*/
};
This section only applies to versions < Wrath.

The next iteration of the old format. A new field has been added after version and the DirEntry structure has been replaced by a flat offset array.

struct BLSHeader {
  char magic[4];                // "SVXG" for vertex and "SPXG" for pixel
  uint32_t version;             // always 0x10003 - version 1.3 of format
  uint32_t _unk08;              // always 1
  
  #if pixel_shader
   uint32_t offsets[12];
  #else
   uint32_t offsets[6];
  #endif
}; 

Blocks

This section only applies to version Wrath.

There are permutationCount blocks of the following structure. They are padded to 0x*0, 0x*4, 0x*8 and 0x*C.

struct BLSBlock {
/*0x00*/	DWORD flags0;		// seen: 0x3FE80 in pixel shaders; 0x1A0F in vertex shaders. there may be more ..
/*0x04*/	DWORD flags4;		// seen: 0x200 in pixel shaders; 0x3FEC1 in vertex shaders (there may be more ..)
/*0x08*/	DWORD unk8;		// Never seen anything in here.
/*0x0C*/	uint32 size;		// Tells you how large the block actually is.
/*0x10*/	char data[size];	// In whatever format defined.
/*----*/
};
This section only applies to versions < Wrath.

The pre-WotLK block structure is the same as v1.2 however the code is prefixed with an index and conditional sub-index. The sub-index is only used if the index is not 0.

BLS v1.2 (Beta)

This section only applies to versions PreVanilla (0.7.0.3694) … PreVanilla (0.9.1.3810).

Header

This header is used for both pixel and vertex shaders. It should be noted that, while the code existed there are no vertex shaders in the files of this version.

struct BLSHeader
{
  char magic[4];                // "SVXG" for vertex and "SPXG" for pixel
  uint32_t version;             // always 0x10002 - version 1.2 of format
 
  #if pixel_shader
   DirEntry dir[11];            // pixel shaders have 11 blocks
  #else
   DirEntry dir[3];             // vertex shaders have 3 blocks
  #endif
 
  DirEntry endoffile;           // start = file length, count = 0
}; 

struct DirEntry
{
  uint32_t start;               // offset to block
  uint32_t count;               // total size of block
};

Blocks

Enumerating the BLSHeader.dir provides the offsets for each block. Entries with a count of 0 should be ignored.

struct CGxShader {
  uint32_t ccount;
  CGxShaderParam consts[ccount];  
  uint32_t pcount;
  CGxShaderParam params[pcount];   
  uint32_t unknown;                // always 1  
  uint32_t bytes;
  char code[bytes];
}

struct CGxShaderParam {
  char name[64];
  Type type;
  float f[16];
  uint32_t unknown;                // always 3 or 0
  uint32_t unknown;                // always 0
  uint32_t unknown;                // always 0  
  
  enum Type
  {
    Type_Vector4 = 0x0,            // C4Vector
    Type_Matrix34 = 0x1,           // C34Matrix
    Type_Matrix44 = 0x2,           // C44Matrix
    Type_3 = 0x3,                  // used in terrain3*.bls, terrain4*.bls
    Type_4 = 0x4,                  // used in terrain3*.bls, terrain4*.bls
    Type_Force32Bit = 0xFFFFFFFF,
  };
};

BLS v1.1 (Alpha)

This section only applies to versions PreVanilla (0.5.3.3368) … PreVanilla (0.6.0.3592).

Header

This header is used for both pixel and vertex shaders. It should be noted that, while the code existed there are no vertex shaders in the files of this version.

struct BLSHeader
{
  char magic[4];                // "SVXG" for vertex and "SPXG" for pixel
  uint32_t version;             // always 0x10001 - version 1.1 of format
  
  #if pixel_shader
   DirEntry dir[11];            // pixel shaders have 11 blocks
  #else
   DirEntry dir[3];             // vertex shaders have 3 blocks
  #endif
}; 

struct DirEntry
{
  uint32_t start;               // offset to block
  uint32_t count;               // total size of block
};

Blocks

Enumerating the BLSHeader.dir provides the offsets for each block. Entries with a count of 0 should be ignored.

struct CGxShader {
  uint32_t ccount;
  CGxShaderParam consts[ccount];
  uint32_t pcount;
  CGxShaderParam params[pcount];
  uint32_t bytes;
  char code[bytes];
}

struct CGxShaderParam {
  char name[32];
  Type type;
  uint32_t index;
  float f[16];                     // if the type doesn't use all 16 floats the remainder is 0 padded
  
  enum Type
  {
    Type_Vector4 = 0x0,            // C4Vector
    Type_Matrix34 = 0x1,           // C34Matrix
    Type_Matrix44 = 0x2,           // C44Matrix
    Type_Force32Bit = 0xFFFFFFFF,
  };
};