CASC: Difference between revisions

From wowdev
Jump to navigation Jump to search
(Added size and size-size)
(Include algorithm to calculate checksum_b in casc data file header)
(29 intermediate revisions by 7 users not shown)
Line 1: Line 1:
CASC is the name of the new file system that Blizzard has created to replace the outdated format of [[MPQ]].
<div style='border: 1px solid black; padding: 10px;'>
'''Missing something?''' This page was recently split up into separate pages. For the content transfer part of [[NGDP]], see [[TACT]]. This page should now only contain information on the local filesystem format called CASC.
</div>
 
CASC is the name of the new file system that Blizzard created to replace the outdated format of [[MPQ]]. It is used in WoW versions {{Template:Sandbox/VersionRange|min_expansionlevel=6|min_inclusive=6}}.


=CASC v1=
=CASC v1=
Line 8: Line 12:
The remainder of this article will refer exclusively to the system called CASC v2 as 'CASC'. While many parts of the file system are identical between v1 and v2, there are enough changes to make explaining both formats at once inadvisable.
The remainder of this article will refer exclusively to the system called CASC v2 as 'CASC'. While many parts of the file system are identical between v1 and v2, there are enough changes to make explaining both formats at once inadvisable.


=NGDP=
=Journal-based Data Files=
CASC was introduced simultaneously with a new system for managing configuration, blob, and installation files called NGDP, or Next Generation Download Protocol. When the acronym 'NGDP' is used in conjunction with the term CASC, it is typically referring to the hosted components of the CASC file system, and its ability to stream data on the fly.
During the installation process for a Blizzard game, the program will download the required files as requested by root, encoding, download, and install. It stores the downloaded data fragments in data files in "INSTALL_DIR\Data\data\". The program will record the content hash ([[BLTE]]-compressed hash), size, and position of the file as well as the number of the data file that it is in. It places those four parameters into journal files with the extension '.idx'.
 
==NGDP URLs==
As of October 14th, 2014, the following generic NGDP URLs are known:
* http://us.patch.battle.net:1119/(ProgramCode)/cdns - a table of domains available with game data per region
* http://us.patch.battle.net:1119/(ProgramCode)/versions - a table of the current game version, build config, cdn config and optionally keyring per region
* http://us.patch.battle.net:1119/(ProgramCode)/bgdl - similar to versions, but tailored for use by the Battle.net App background downloader process
* http://us.patch.battle.net:1119/(ProgramCode)/blobs - contains InstallBlobMD5 and GameBlobMD5
* http://us.patch.battle.net:1119/(ProgramCode)/blob/game - a blob file that regulates game functionality for the Battle.net App
* http://us.patch.battle.net:1119/(ProgramCode)/blob/install - a blob file that regulates installer functionality for the game in the Battle.net App
 
Keep in mind Blizzard's CDN is pretty shit at caching sometimes so you after an update to the above files it might switch back and forth between the old and new version of the files for a few hours.
 
==NGDP Program Codes==
 
The following NGDP program codes are known to be served at some point during NGDP's existence.
 
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="90" | Program
! width="250" | Description
! width="200" | Status
|-
| agent || Battle.net Agent || style="color: green;" | Active
|-
| agent_test || Probably Agent test ||
|-
| bna || Battle.net App || style="color: green;" | Active
|-
| bnt || Heroes of the Storm Alpha || style="color: red;" | Deprecated
|-
| bts || Bootstrapper (currently for dst2) || style="color: orange;" | Partial (only versions as of now, cdnpath=tpr/bnt004)
|-
| catalogs || Catalog || style="color: green;" | Active
|-
| clnt || Client || style="color: red;" | Deprecated
|-
| d3 || Diablo 3 Retail || style="color: green;" | Active
|-
| d3b || Diablo 3 Beta (2013) || style="color: orange;" | Partial
|-
| d3cn || Diablo 3 China || style="color: green;" | Active
|-
| d3cnt || Diablo 3 China Test (?) || style="color: orange;" | Unused (everything empty)
|-
| d3t || Diablo 3 Test || style="color: green;" | Active
|-
| demo || || style="color: orange;" | Partial
|-
| dst2 || Destiny 2 || style="color: green;" | Active
|-
| dst2a || Destiny 2 Alpha || style="color: blue;" | Active (encrypted)
|-
| dst2dev || Destiny 2 "takehome" development || style="color: orange;" | Partial (encrypted?)
|-
| dst2e1 || Destiny 2 Event || Active; probably 1..9
|-
| dst2t || Destiny 2 Public Test || style="color: green;" | Active
|-
| hero || Heroes of the Storm Retail || style="color: green;" | Active
|-
| heroc || Heroes of the Storm Tournament || style="color: green;" | Active
|-
| herot || Heroes of the Storm Test || style="color: green;" | Active
|-
| hsb || Hearthstone Retail || style="color: green;" | Active
|-
| hst || Hearthstone Test || style="color: orange;" | Partial
|-
| pro || Overwatch Retail || style="color: green;" | Active
|-
| proc || Overwatch Tournament US || style="color: green;" | Active
|-
| proc_cn || Overwatch Tournament China || style="color: green;" | Active
|-
| proc_eu || Overwatch Tournament Europe || style="color: green;" | Active
|-
| proc_kr || Overwatch Tournament Korea || style="color: green;" | Active
|-
| proc2 || Overwatch Professional 2 (prometheus_tournament_2) || style="color: green;" | Active
|-
| proc2_cn || Overwatch Professional 2 China || style="color: green;" | Active
|-
| proc2_eu || Overwatch Professional 2 Europe || style="color: green;" | Active
|-
| proc2_kr || Overwatch Professional 2 Korea || style="color: green;" | Active
|-
| proc3 || Overwatch Tournament (Dev) ||
|-
| prodev || Overwatch Dev || style="color: blue;" | Active (encrypted)
|-
| prot || Overwatch Test || style="color: green;" | Active
|-
| prov || Overwatch Vendor || style="color: green;" | Active
|-
| s1 || StarCraft 1 || style="color: green;" | Active
|-
| s1a || Starcraft 1 Alpha || style="color: blue;" | Active (encrypted)
|-
| s1t || StarCraft 1 Test || style="color: green;" | Active
|-
| s2 || StarCraft II Retail || style="color: green;" | Active
|-
| s2b || StarCraft II Beta || style="color: red;" | Deprecated
|-
| s2t || StarCraft II Test || style="color: red;" | Deprecated
|-
| sc2 || StarCraft II || style="color: red;" | Deprecated
|-
| storm || Heroes of the Storm || style="color: red;" | Deprecated
|-
| test || || style="color: red;" | Deprecated
|-
| viper || Call of Duty Black Ops 4 || style="color: green;" | Active
|-
| viperdev || Call of Duty Black Ops 4 - Alpha ||
|-
| w3 || Warcraft III || style="color: green;"| Active
|-
| w3t || Warcraft III Public Test || style="color: green;"| Active
|-
| war3 || Warcraft III (old) || style="color: orange;"| Partial
|-
| wow || World of Warcraft Retail || style="color: green; | Active
|-
| wow_beta || World of Warcraft Alpha/Beta || style="color: green; | Active
|-
| wowdev || World of Warcraft Dev || style="color: blue;" | Active (encrypted)
|-
| wowe1 || World of Warcraft Event 1 || style="color: green;" | Active
|-
| wowe2 || World of Warcraft Event 2 || style="color: orange;" | Active (partial)
|-
| wowe3 || World of Warcraft Event 3 || style="color: orange;" | Active (partial)
|-
| wowt || World of Warcraft Test || style="color: green;" | Active
|-
| wowv || World of Warcraft Vendor || style="color: blue;" | Active (encrypted)
|-
| wowz || World of Warcraft Submission (previously Vendor) || style="color: green; | Active
|}
 
=CASC Online=
 
==Standard URL Hash Format==
URL Format: http://(cdnsHost)/(cdnsPath)/(pathType)/(FirstTwoHexOfHash)/(SecondTwoHexOfHash)/(FullHash)
 
Current CDNs for a specific program can be found in the http://us.patch.battle.net:1119/(ProgramCode)/cdns file. Blizzard regularly shuffles around CDNs and sometimes even adds/removes CDNs, so be sure to parse the CDNs file to stay up to date. For WoW, the cdnsPath of "tpr/wow" has never changed.
 
Known path types are:
* config - contains the three types of config files: Build configs, CDN configs, and Patch configs
* data - contains archives, indexes, and unarchived standalone files (typically binaries, mp3s, and movies and files mentioned in buildconfig like root, install and download)
* patch - contains patch manifests, files, archives, indexes
 
Blizzard regularly cleans old builds from the CDN so any example files mentioned in this article might be unavailable at the time of reading (feel free to update them to something more recent if this is the case).
 
==Config Files==
 
===Build Config===
Example file: http://blzddist1-a.akamaihd.net/tpr/wow/config/22/38/2238ab9c57b672457a2fa6fe2107b388
 
Some of the files listed in this file are explained later on in this article.
 
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="140" | Value name
! width="650" | Description
|-
| [[#Root|root]] || Content hash of the '''decoded''' root file, look this up in encoding to get the '''encoded''' hash.
|-
| [[#Install|install]] || First key is the content hash of the '''decoded''' install file. Second key, if present, is the '''encoded''' hash; if absent, look up the content hash in encoding to get the encoded hash/CDN key.
|-
| install-size || Install size(s) corresponding to the install hash(es). Absent in older builds.
|-
| [[#Download|download]] || First key is the content hash of the '''decoded''' download file. Second key, if present, is the '''encoded''' hash; if absent, look up the content hash in encoding to get the encoded hash.
|-
| download-size || Download size(s) corresponding to the download hash(es). Absent in older builds.
|-
| [[#Download Size|size]] || CKey and EKey of the download size file, respectively. Introduced in build 27547.
|-
| size-size || Download size sizes corresponding to the download size keys.
|-
| build-partial-priority || Content hash:block size pairs for priority non-archived files, ordered by priority. The block size appears to be the amount to request in each partial HTTP request. These files are BLTE-encoded and usually but not always split into blocks of fixed uncompressed size where the partial priority block size is a power-of-two multiple of the file's uncompressed block size. First seen in build 24116. Optional.
|-
| partial-priority || Content hash of a partial download file containing priority files to download first; note that the sizes in the download file entries are download block sizes, '''not''' file sizes. First seen in build 22231, replaced by build-partial-priority in build 24116. Optional.
|-
| partial-priority-size || Unknown: always 0 if present. Present if partial-priority is present.
|-
| [[#Encoding|encoding]] || First key is the content hash of the '''decoded''' encoding file. Second key is the '''encoded''' hash. If either none or 1 is found, the game (at least Wow) switches to plain-data (?) . Seen in build 20173
|-
| encoding-size || Encoding sizes corresponding to the encoding hashes.
|-
| [[#Patch|patch]] || Unknown
|-
| patch-size || Unknown
|-
| [[#Patch_Config|patch-config]] || Content hash of non-encoded patch config (see Patch Config)
|-
| build-branch || Optional. Presumably the SCM branch built.
|-
| build-comments || Optional.
|-
| build-creator || Optional. Presumably the user who submitted the build.
|-
| build-name || Optional? Name of the build
|-
| build-playbuild-installer || Optional? Type of installer for the Battle.net app to use
|-
| build-product || Optional? Product name
|-
| build-uid || Optional? Program code (see NGDP Program Codes)
|}
 
===CDN Config===
Example file: http://blzddist1-a.akamaihd.net/tpr/wow/config/42/33/423364147752a596911aa1de2ff1f6a4
 
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="140" | Value name
! width="650" | Description
|-
| [[#Archives|archives]] || CDN keys of all archives (and by appending .index to the hash their indexes)
|-
| [[#Archive-Group_Index_.28.index.29|archive-group]] || CDN key of the the combined index file (see Archive-Group Index)
|-
| file-index || CDN key of .index file for unarchived files. These files have 0-byte archive offset fields. Seen in Warcraft III.
|-
| file-index-size || Size of unarchived file .index.
|-
| [[#patch-archives|patch-archives]] || CDN keys of patch archives (needs research)
|-
| patch-archive-group || CDN key of probably the combined patch index file (needs research)
|-
| [[#Build_Config|builds]] || List of build configs this config supports (optional)
|-
| patch-file-index || CDN key of .index file for unarchived patches. These files have 0-byte archive offset fields. Seen in Warcraft III.
|-
| patch-file-index-size || Size of unarchived patch .index.
|}
 
===Patch Config===
Example file: http://blzddist1-a.akamaihd.net/tpr/wow/config/20/f0/20f0593dd1b9fdaeaa7a808f83d48f1d
 
This configuration file was added after all of the others. It first appeared in CASC v1 for Heroes of the Storm in August 2014. It then appeared in WoW for CASC v2 in build 19027 (October 10th, 2014).
The purpose of this file is to reduce redundant downloads. It achieves this by directing the system to download patch files to apply and update previously downloaded material.
The structure and purpose of all of the fields of this file requires further research.
 
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="140" | Value name
! width="650" | Description
|-
| patch-entry || Repeats 3 times with patch entries for install, download and encoding file. Builds using the partial-priority (not build-partial-priority) mechanism may additionally have a partial-priority entry.
|-
| patch || Patch manifest file
|-
| patch-size || Size of patch file (optional?)
|}
 
====patch-entry====
 
The format of these strings is:
 
<pre>patch-entry = <type> <content hash> <content size> <BLTE-encoding key> <BLTE-encoded size> <encoding string></pre>
 
followed by sets of
 
<pre><old BLTE-encoding key> <old content size> <patch hash> <patch size></pre>
 
This serves to pair old files to the patch needed to bring them up to date, as well as providing the information required to detect if the file is already up to date.
 
==Data Files==
 
Example index: http://blzddist1-a.akamaihd.net/tpr/wow/data/00/52/0052ea9a56fd7b3b6fe7d1d906e6cdef.index
 
Example archive: http://blzddist1-a.akamaihd.net/tpr/wow/data/00/52/0052ea9a56fd7b3b6fe7d1d906e6cdef
 
==Patch Files==
 
==Armadillo==
If <tt>$program/blob/game</tt> contains <tt>decryption_key_name</tt>, data on CDN is encrypted. The key is retrieved from the keychain or manually entered (a base32 encoded version that contains key and checksum is used). It is stored in <tt>$key_name.ak</tt>. The key file contains the key followed by a the first four bytes of the md5 hash of the key for verification purposes.
 
Encryption uses Salsa20. The IV is the last 8 bytes (16 characters) of the cdn hash.
 
===Known keys===
decryption_key_name    used_by              key                                                checksum      base32
sc1Dev                  s1a                  F6 79 DC 38 E0 C3 65 FB 48 2E 48 A7 48 90 9D 29    19 F5 BB 88  6Z45YOHAYNS7WSBOJCTUREE5FEM7LO4I
prolivedev              prodev 
pro                    pro (pre-launch)           
destiny_live            dst2a
destiny_dev            dst2a (pre-alpha)
destiny2_openbeta      dst2
viperlivedev            viperdev
wowdevalpha            wowdev
wowvendor              wowv
 
=File References=
Files are referred to by many different pieces of data in CASC. A quick summary of them:
* Filename: The file's real name. Note that one file can have many names - essentially, one encoding key can map to many different name hashes.
* Locale Flag:
* Content Flag:
* Name Hash: The file's name, after being hashed with the Jenkins Hash.
* Content Hash: The MD5 of the entire file in its uncompressed state; the purest representation of the data.
* Encoding Hash/Key: MD5 hash of the potentially encoded file. For unencoded files, the content hash. For chunkless BLTE files lacking a chunk table, this hash covers the entire encoded file. For chunked BLTE files, this hash covers only the BLTE headers including chunk table, as the chunk table contains hashes of the content of each chunk. A given file can be encoded in many ways, and a single content hash may potentially have multiple encoding keys.
* CDN Key: The key used to lookup a file on the CDN. Synonym of encoding key.
* Header Hash: Inaccurate synonym of encoding key.
 
=BLTE encoded files=
Any files stored inside the data files are BLTE encoded, which means before reading anything in the file, first you have to decode it. The documentation below refers to ''decoded'' files!
 
It consists of these chunks in the following order:
* Header
* ChunkInfo (only if Header.headerSize > 0)
* Data
 
 
To read a BLTE encoded file:
# Read the Header chunk
# Read the ChunkInfo chunk if Header.headerSize > 0
# Read each of the Data chunks and combine them to create the complete file
 
'''Note:''' If there is no ChunkInfo struct, there is just one Data chunk.
 
 
*'''Header'''
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="80" | Offset (Hex)
! width="200" | Type
! width="150" | Name
! width="900" | Description
|-
| 0x00 || char[4] || FileSignature || "BLTE"
|-
| 0x04 || uint32_t [BE] || headerSize || Size of the BLTE header (BLTE header = Header + ChunkInfo).
|}
 
 
*'''ChunkInfoEntry'''
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="80" | Offset (Hex)
! width="200" | Type
! width="150" | Name
! width="900" | Description
|-
| 0x00 || uint32_t [BE] || compressedSize || Compressed size of the chunk (the compression mode byte is included).
|-
| 0x04 || uint32_t [BE] || decompressedSize || Decompressed chunk of the size.
|-
| 0x08 || char[16] || checksum || The checksum of the compressed chunk (the compression mode byte is included).
|}
 
 
*'''ChunkInfo'''
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="80" | Offset (Hex)
! width="200" | Type
! width="150" | Name
! width="900" | Description
|-
| 0x00 || uint8_t [BE] || flags || Flags of some sort.
|-
| 0x02 || uint24_t [BE] || chunkCount || The number of chunks.
|-
| 0x04 || ChunkInfoEntry[chunkCount] || chunks || The chunk info for the chunks in the file.
|}
 


if either flags != 0xF or chunkCount == 0, the file is deemed badly formatted.
==Shared Memory==
 
*'''Data'''
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="85" | Offset (Hex)
! width="260" | Type
! width="150" | Name
! width="900" | Description
|-
| 0x00 || char || encodingMode || Available values: N, Z, F, E
|-
| 0x01 || char[ChunkInfo.compressedSize - 1] || data || The encoded data.
|}
 
 
 
Example implementation as Binary Template can be found here: [[BLTE-Template]]
 
 
'''Encoding modes:'''
* N: Plain data.
* Z: Zlib encoded data.
* 4: lz4hc encoded data.
 
The encoded data is preceded by the two header bytes as specified by the zlib RFC [https://www.ietf.org/rfc/rfc1950.txt] (Section 2.2). 78 DA most of the time.
 
Reading bits left-to-right:
 
var compressionInfo = reader.ReadBits(4);
var compressionMethod = reader.ReadBits(4);
var flevel = reader.ReadBits(2);
var fdict = reader.ReadBit();
var fcheck = reader.ReadBits(5);
 
In .NET you can basically skip these bytes and use DeflateStream. Make sure to wrap around the chunk so it internally does not try to consume bytes from the following chunk.
 
* F: Recursively encoded BLTE data.
* E: encrypted: one of salsa20, arc4, rc4.
 
struct
{
  unsigned char key_name_length;              // 0x8
  unsigned char key_name[key_name_length];
  unsigned char IV_length;                    // 0x4
  unsigned char IV[IV_length];
  char type; // 'S': salsa20, 'A': arc4
} E_chunk;
 
key_name is resolved by client to the actual key. keys are distributed via keyrings and some keys are hardcoded.
 
==Encoding Specification (ESpec)==
ESpecs are string-based representations of the encoding of BLTE-encoded data files that serve as recipes for the patcher to produce a binary-identical encoded output file (as patching operates on the unencoded data). The information they contain is redundant with the information in the BLTE header of the file itself, but due to ESpec shorthand notations multiple ESpecs can encode the same output for the same input (e.g. "b:256*2=z", "b:{256=z,256=z}", etc.), and the same ESpec can result in different output block configurations for inputs of different sizes (e.g. 500 bytes of input results in 2 blocks and 600 bytes 3 blocks with "b:256*=z"). They are used most extensively in [[#Encoding|Encoding files]].
 
An example parser can be found [https://gist.github.com/heksesang/b15057fe3f093eebee3a at this gist].
 
The strings are not whitespace-tolerant. They use the following [https://en.wikipedia.org/wiki/Extended_Backus–Naur_form#Table_of_symbols EBNF grammar] (concatenations are omitted), where <tt>e-spec</tt> is a top level string:
 
e-spec          =  ( 'n' )
                |  ( 'z' [ ':' ( zip-level | '{' zip-level ',' zip-bits '}' ) ] )
                |  ( 'e' ':' '{' encryption-key ',' encryption-iv ',' e-spec '}' )
                |  ( 'b' ':' ( final-subchunk | '{' ( [{block-subchunk ','}] final-subchunk ) '}' ) )
                ;
block-subchunk  =  block-size-spec '=' e-spec ;
block-size-spec =  block-size [ '*' block-count ] ; (* block-count (1 if unspecified) blocks of block-size bytes *)
block-size      =  number [ block-unit ] ;
block-unit      =  'K'                          (* count * 2^10 *)
                |  'M'                          (* count * 2^20 *)
                ;
block-count    =  number ;
final-subchunk  =  final-size-spec '=' e-spec ;
final-size-spec =  block-size-spec
                |  block-size '*'              (* greedy spec of block-size blocks (last block <= block-size) *)
                |  '*'                          (* greedy block *)
                ;
zip-level      = number
                ;
zip-bits        = number | ( 'm' 'p' 'q' )
                ;
encryption-key  =  ? eight byte upper-hex encoded [[#Encryption_keys|key name]] ?
                ;
encryption-iv  =  ? four byte upper-hex encoded IV value ?
                ;
 
* where a <tt>greedy</tt> final-size-spec consumes all remaining bytes in the parent block or file
** "all remaining bytes" ''may be 0'', producing no blocks
* where <tt>zip-level</tt> defaults to <tt>9</tt> and <tt>zip-bits</tt> to <tt>15</tt> if not given
** <tt>zip-bits</tt> with value <tt>'mpq'</tt> means <tt>0</tt>
 
===Examples===
 
b:{164=z,16K*565=z,1656=z,140164=z}
 
* blocks on top level
** 164 bytes of zip with level=9, bits=15
** 565 blocks of 16Kb zip chunks with level=9, bits=15
** 1656 bytes of zip with level=9, bits=15
** 140164 of zip with level=9, bits=15
 
b:{1768=z,66443=n}
 
* blocks on top level
** 1768 bytes of zip with level=9, bits=15
** 66443 bytes of raw data
 
b:{256K*=e:{237DA26C65073F42,06FC152E,z}}
 
* blocks on top level
** unspecified count of 256 kb chunks encrypted with key 237DA26C65073F42 and IV 06FC152E
*** containing zip data with level=9, bits=15
*** last chunk <= 256 kb
 
z
 
* zipped data on top level with level=9, bits=15
 
b:{22=n,31943=z,211232=n,27037696=n,138656=n,17747968=n,*=z}
 
* blocks on top level
** 22 bytes of raw data
** 31943 bytes of zip with level=9, bits=15
** 211232 bytes of raw data
** 27037696 bytes of raw data
** 138656 bytes of raw data
** 17747968 bytes of raw data
** an unspecified amount of zipped data with level=9, bits=15
 
b:{16K*=z:{6,mpq}}
 
* blocks on top level
** unspecified count of 16 kb chunks of zipped data with level=6, bits=0
*** last chunk <= 16 kb
 
=Encryption keys=
 
== Battle.net app==
key_name          key                              type      seen in
2C547F26A2613E01  37C50C102D4C9E3A5AC069F072B1417D  salsa20  Battle.net App Alpha 1.5.0
 
== Overwatch ==
Overwatch has keys included in the client/keyring, but they can also be streamed from the server.
key_name          key                              type    seen in        method    used for
FB680CB6A8BF81F3  62D90EFA7F36D71C398AE2F1FE37BDB9  salsa20  0.8.0.24919    keyring
402CD9D8D6BFED98  AEB0EADEA47612FE6C041A03958DF241  salsa20  0.8.0.24919    keyring
DBD3371554F60306  34E397ACE6DD30EEFDC98A2AB093CD3C  salsa20  0.8.0.24919    keyring
11A9203C9881710A  2E2CB8C397C2F24ED0B5E452F18DC267  salsa20  0.8.0.24919    keyring
A19C4F859F6EFA54  0196CB6F5ECBAD7CB5283891B9712B4B  salsa20  0.8.0.24919    keyring
87AEBBC9C4E6B601  685E86C6063DFDA6C9E85298076B3D42  salsa20  0.8.0.24919    keyring
DEE3A0521EFF6F03  AD740CE3FFFF9231468126985708E1B9  salsa20  0.8.0.24919    keyring
8C9106108AA84F07  53D859DDA2635A38DC32E72B11B32F29  salsa20  0.8.0.24919    keyring
49166D358A34D815  667868CD94EA0135B9B16C93B1124ABA  salsa20  0.8.0.24919    keyring
1463A87356778D14  69BD2A78D05C503E93994959B30E5AEC  salsa20  ≤ 1.0.3.0      keyring
5E152DE44DFBEE01  E45A1793B37EE31A8EB85CEE0EEE1B68  salsa20  ≤ 1.0.3.0      keyring
9B1F39EE592CA415  54A99F081CAD0D08F7E336F4368E894C  salsa20  ≤ 1.0.3.0      keyring
24C8B75890AD5917  31100C00FDE0CE18BBB33F3AC15B309F  salsa20  ≤ 1.0.3.0      keyring
EA658B75FDD4890F  DEC7A4E721F425D133039895C36036F8  salsa20  ≤ 1.0.3.0      keyring
026FDCDF8C5C7105  8F41809DA55366AD416D3C337459EEE3  salsa20                  keyring
CAE3FAC925F20402  98B78E8774BF275093CB1B5FC714511B  salsa20                  keyring
061581CA8496C80C  DA2EF5052DB917380B8AA6EF7A5F8E6A  salsa20                  keyring
BE2CB0FAD3698123  902A1285836CE6DA5895020DD603B065  salsa20                  keyring
57A5A33B226B8E0A  FDFC35C99B9DB11A326260CA246ACB41  salsa20  1.1.0.0.30200  keyring  Ana
42B9AB1AF5015920  C68778823C964C6F247ACC0F4A2584F8  salsa20  1.2.0.1.30684  keyring  Summer Games
4F0FE18E9FA1AC1A  89381C748F6531BBFCD97753D06CC3CD  salsa20  1.2.0.1.30684  keyring
7758B2CF1E4E3E1B  3DE60D37C664723595F27C5CDBF08BFA  salsa20  1.2.0.1.30684  keyring
E5317801B3561125  7DD051199F8401F95E4C03C884DCEA33  salsa20  1.4.0.2.32143  keyring  Halloween Terror
16B866D7BA3A8036  1395E882BF25B481F61A4D621141DA6E  salsa20  1.4.1.0.31804  keyring  Bastion Blizzcon 2016 skin
11131FFDA0D18D30  C32AD1B82528E0A456897B3CE1C2D27E  salsa20  1.5.0.1.32795  keyring  Sombra
CAC6B95B2724144A  73E4BEA145DF2B89B65AEF02F83FA260  salsa20  1.5.0.1.32795  keyring  Ecopoint: Antarctica
B7DBC693758A5C36  BC3A92BFE302518D91CC30790671BF10  salsa20  1.5.0.1.32795  keyring  Genji Oni Skin (HotS Nexus Challenge)
90CA73B2CDE3164B  5CBFF11F22720BACC2AE6AAD8FE53317  salsa20  1.6.1.0.33236  keyring  Oasis map
6DD3212FB942714A  E02C1643602EC16C3AE2A4D254A08FD9  salsa20  1.6.1.0.33236  keyring
11DDB470ABCBA130  66198766B1C4AF7589EFD13AD4DD667A  salsa20  1.6.1.0.33236  keyring  Winter Wonderland
5BEF27EEE95E0B4B  36BCD2B551FF1C84AA3A3994CCEB033E  salsa20                  keyring
9359B46E49D2DA42  173D65E7FCAE298A9363BD6AA189F200  salsa20                  keyring  Diablo's 20th anniversary
1A46302EF8896F34  8029AD5451D4BC18E9D0F5AC449DC055  salsa20  1.7.0.2.34156  keyring  Year of the Rooster
693529F7D40A064C  CE54873C62DAA48EFF27FCC032BD07E3  salsa20  1.8.0.0.34470  keyring  CTF Maps
388B85AEEDCB685D  D926E659D04A096B24C19151076D379A  salsa20  1.8.0.0.34470  keyring  Numbani Update (Doomfist teaser)
E218F69AAC6C104D  F43D12C94A9A528497971F1CBE41AD4D  salsa20  1.9.0.0.34986  keyring  Orisa
F432F0425363F250  BA69F2B33C2768F5F29BFE78A5A1FAD5  salsa20  1.10.0.0.35455  keyring  Uprising
061D52F86830B35D  D779F9C6CC9A4BE103A4E90A7338F793  salsa20  1.10.0.0.35455  keyring  D.Va Officer Skin (HotS Nexus Challenge 2)
1275C84CF113EF65  CF58B6933EAF98AF53E76F8426CC7E6C  salsa20                  keyring
D9C7C7AC0F14C868  3AFDF68E3A5D63BABA1E6821883F067D  salsa20                  keyring
BD4E42661A432951  6DE8E28C8511644D5595FC45E5351472  salsa20  1.11.0.0.36376  keyring  Anniversary event
C43CB14355249451  0EA2B44F96A269A386856D049A3DEC86  salsa20  1.12.0.0.37104  keyring  Horizon Lunar Colony
E6D914F8E4744953  C8477C289DCE66D9136507A33AA33301  salsa20  1.13.0.0.37646  keyring  Doomfist
5694C503F8C80178  7F4CF1C1FBBAD92B184336D677EBF937  salsa20  1.13.0.0.37646  keyring  Doomfist
21DBFD65F3E54269  AB580C3837CAF8A461F243A566B2AE4D  salsa20  1.13.0.0.37646  keyring  Summer Games 2017
27ABA5F88DD8D078  ????????????????????????????????  salsa20  1.13.0.0.37646  ???????  ??????
21E1F90E71D33C71  328742339162B32676C803C2255931A6  salsa20  1.14.1.0.39083  keyring  Deathmatch
D9CB055BCDD40B6E  49FB4477A4A0825327E9A73682BECD0C  salsa20  1.15.0.0.?????  keyring  Junkertown
8175CE3C694C6659  E3F3FA7726C70D26AE130D969DDDF399  salsa20  1.16.0.0.40011  keyring  Halloween 2017
B8DE51690075435A  C07E9260BB711217E7DE6FED911F4296  salsa20  1.16.0.0.?????  keyring  Winston Blizzcon 2017 skin
F6CF23955B5D437D  AEBA227328A5B0AA9F51DAE3F6A7DFE4  salsa20  1.17.0.2.41350  keyring  Moira
0E4D9426F2891F5C  9FF064C38BE52CCDF73748180F628205  salsa20  1.18.1.2.42076  keyring  Winter Wonderland 2017
9240BA6A2A0CF684  DF2E37D78B43108FA6242068B70D1F65  salsa20  1.19.1.3.42563  keyring  Overwatch League
82297FBAB7F5EB80  B534C20965852FB15AECAC17E381B417  salsa20  1.19.1.3.42563  keyring  Jan 2017 Lootbox Update
9ADF00AA1A174A69  9A4AC899261A2F1C6969F39397C358E7  salsa20  1.19.1.3.42563  keyring  Blizzard World
CFA05AA76B49F881  526DDDEF19BF373C25B629A334CD7237  salsa20  1.19.1.3.42563  keyring  WoW's Battle For Azeroth Preorder
493455579DA0B18E  C0BABF72AD2C05DFC14017D1ADBF5977  salsa20  1.19.3.1.43036  stream    Inaugral Season Spray/Icon ??
6362C5AD65DAE686  62F603D5390F763ED51773F0164FEDB5  salsa20  1.19.3.1.43036  stream    White/Gray OWL Skins ??
8162E5313A9C135D  F407834D9521587C5012B0A59D7E064B  salsa20  1.20.0.2.43435  stream    Lunar New Year 2018 (Dog) / Ayutthaya / Comp CTF
68EAE8FDC008C381  ????????????????????????????????  salsa20  1.20.0.2.43435  n/a      ??
F412C6327C4BF091  6FAFC648CBF1C2115B769593C170E732  salsa20  1.20.0.2.43435  stream    SC2 20th Anniversary (Kerrigan Skin)
 
== World of Warcraft ==
WoW's [[DB/TactKey|TactKey.db2]] and [[DB/TactKeyLookup|TactKeyLookup.db2]] contain keys and key_names (called lookups there) respectively. Either can be streamed from server.
These files cannot be bruteforced by requesting hotfix data (Blizzard has guards in place)
 
The id field in the table below corresponds to the id field in the db2's.
key_name          key                              type      id  seen in                  used for
FA505078126ACB3E  BDC51862ABED79B2DE48C8E7E66C6200  salsa20  15  WOW-20740patch7.0.1_Beta  <not used between 7.0 and 7.3>
FF813F7D062AC0BC  AA0B5C77F088CCC2D39049BD267F066D  salsa20  25  WOW-20740patch7.0.1_Beta  <not used between 7.0 and 7.3>
D1E9B5EDF9283668  8E4A2579894E38B4AB9058BA5C7328EE  salsa20  39  WOW-20740patch7.0.1_Beta  Enchanted Torch pet
B76729641141CB34  9849D1AA7B1FD09819C5C66283A326EC  salsa20  40  WOW-20740patch7.0.1_Beta  Enchanted Pen pet
FFB9469FF16E6BF8  D514BD1909A9E5DC8703F4B8BB1DFD9A  salsa20  41  WOW-20740patch7.0.1_Beta  <not used between 7.0 and 7.3>
23C5B5DF837A226C  1406E2D873B6FC99217A180881DA8D62  salsa20  42  WOW-20740patch7.0.1_Beta  Enchanted Cauldron pet
3AE403EF40AC3037  ????????????????????????????????  salsa20  51  WOW-21249patch7.0.3_Beta  <not used between 7.0 and 7.3>
E2854509C471C554  433265F0CDEB2F4E65C0EE7008714D9E  salsa20  52  WOW-21249patch7.0.3_Beta  Warcraft movie items
8EE2CB82178C995A  DA6AFC989ED6CAD279885992C037A8EE  salsa20  55  WOW-21531patch7.0.3_Beta  BlizzCon 2016 Murlocs
5813810F4EC9B005  01BE8B43142DD99A9E690FAD288B6082  salsa20  56  WOW-21531patch7.0.3_Beta  Fel Kitten
7F9E217166ED43EA  05FC927B9F4F5B05568142912A052B0F  salsa20  57  WOW-21531patch7.0.3_Beta  Legion music <not used in 7.3>
C4A8D364D23793F7  D1AC20FD14957FABC27196E9F6E7024A  salsa20  58  WOW-21691patch7.0.3_Beta  Demon Hunter #1 cinematic (legion_dh1)
40A234AEBCF2C6E5  C6C5F6C7F735D7D94C87267FA4994D45  salsa20  59  WOW-21691patch7.0.3_Beta  Demon Hunter #2 cinematic (legion_dh2)
9CF7DFCFCBCE4AE5  72A97A24A998E3A5500F3871F37628C0  salsa20  60  WOW-21691patch7.0.3_Beta  Val'sharah #1 cinematic (legion_val_yd)
4E4BDECAB8485B4F  3832D7C42AAC9268F00BE7B6B48EC9AF  salsa20  61  WOW-21691patch7.0.3_Beta  Val'sharah #2 cinematic (legion_val_yx)
94A50AC54EFF70E4  C2501A72654B96F86350C5A927962F7A  salsa20  62  WOW-21691patch7.0.3_Beta  Sylvanas warchief cinematic (legion_org_vs)
BA973B0E01DE1C2C  D83BBCB46CC438B17A48E76C4F5654A3  salsa20  63  WOW-21691patch7.0.3_Beta  Stormheim Sylvanas vs Greymane cinematic (legion_sth)
494A6F8E8E108BEF  F0FDE1D29B274F6E7DBDB7FF815FE910  salsa20  64  WOW-21691patch7.0.3_Beta  Harbingers Gul'dan video (legion_hrb_g)
918D6DD0C3849002  857090D926BB28AEDA4BF028CACC4BA3  salsa20  65  WOW-21691patch7.0.3_Beta  Harbingers Khadgar video (legion_hrb_k)
0B5F6957915ADDCA  4DD0DC82B101C80ABAC0A4D57E67F859  salsa20  66  WOW-21691patch7.0.3_Beta  Harbingers Illidan video (legion_hrb_i)
794F25C6CD8AB62B  76583BDACD5257A3F73D1598A2CA2D99  salsa20  67  WOW-21846patch7.0.3_Beta  Suramar cinematic (legion_su_i)
A9633A54C1673D21  1F8D467F5D6D411F8A548B6329A5087E  salsa20  68  WOW-21846patch7.0.3_Beta  legion_su_r cinematic
5E5D896B3E163DEA  8ACE8DB169E2F98AC36AD52C088E77C1  salsa20  69  WOW-21846patch7.0.3_Beta  Broken Shore intro cinematic (legion_bs_i)
0EBE36B5010DFD7F  9A89CC7E3ACB29CF14C60BC13B1E4616  salsa20  70  WOW-21846patch7.0.3_Beta  Alliance Broken Shore cinematic (legion_bs_a)
01E828CFFA450C0F  972B6E74420EC519E6F9D97D594AA37C  salsa20  71  WOW-21846patch7.0.3_Beta  Horde Broken Shore cinematic (legion_bs_h)
4A7BD170FE18E6AE  AB55AE1BF0C7C519AFF028C15610A45B  salsa20  72  WOW-21846patch7.0.3_Beta  Khadgar & Light's Heart cinematic (legion_iq_lv)
69549CB975E87C4F  7B6FA382E1FAD1465C851E3F4734A1B3  salsa20  73  WOW-21846patch7.0.3_Beta  legion_iq_id cinematic
460C92C372B2A166  946D5659F2FAF327C0B7EC828B748ADB  salsa20  74  WOW-21952patch7.0.3_Beta  Stormheim Alliance cinematic (legion_g_a_sth)
8165D801CCA11962  CD0C0FFAAD9363EC14DD25ECDD2A5B62  salsa20  75  WOW-21952patch7.0.3_Beta  Stormheim Horde cinematic (legion_g_h_sth)
A3F1C999090ADAC9  B72FEF4A01488A88FF02280AA07A92BB  salsa20  81  WOW-22578patch7.1.0_PTR  Firecat Mount
18AFDF5191923610  ????????????????????????????????  salsa20  82  WOW-22578patch7.1.0_PTR  <not used between 7.1 and 7.3>
3C258426058FBD93  ????????????????????????????????  salsa20  91  WOW-23436patch7.2.0_PTR  <not used between 7.2 and 7.3>
094E9A0474876B98  E533BB6D65727A5832680D620B0BC10B  salsa20  92  WOW-23910patch7.2.5_PTR  shadowstalkerpanthermount, shadowstalkerpantherpet
3DB25CB86A40335E  02990B12260C1E9FDD73FE47CBAB7024  salsa20  93  WOW-23789patch7.2.0_PTR  legion_72_ots
0DCD81945F4B4686  1B789B87FB3C9238D528997BFAB44186  salsa20  94  WOW-23789patch7.2.0_PTR  legion_72_tst
486A2A3A2803BE89  32679EA7B0F99EBF4FA170E847EA439A  salsa20  95  WOW-23789patch7.2.0_PTR  legion_72_ars
71F69446AD848E06  E79AEB88B1509F628F38208201741C30  salsa20  97  WOW-24473patch7.3.0_PTR  BlizzCon 2017 Mounts (AllianceShipMount and HordeZeppelinMount)
211FCD1265A928E9  A736FBF58D587B3972CE154A86AE4540  salsa20  98  WOW-24473patch7.3.0_PTR  "Shadow" fox pet (store)
0ADC9E327E42E98C  017B3472C1DEE304FA0B2FF8E53FF7D6  salsa20  99  WOW-23910patch7.2.5_PTR  legion_72_tsf
BAE9F621B60174F1  38C3FB39B4971760B4B982FE9F095014  salsa20  100 WOW-24727patch7.3.0_PTR  Rejection of the Gift cinematic (legion_73_agi)
34DE1EEADC97115E  2E3A53D59A491E5CD173F337F7CD8C61  salsa20  101 WOW-24727patch7.3.0_PTR  Resurrection of Alleria Windrunner cinematic (legion_73_avt)
E07E107F1390A3DF  290D27B0E871F8C5B14A14E514D0F0D9  salsa20  102 WOW-25079patch7.3.2_PTR  Tottle battle pet, Raptor mount, Horse mount (104 files)
32690BF74DE12530  A2556210AE5422E6D61EDAAF122CB637  salsa20  103 WOW-24781patch7.3.0_PTR  legion_73_pan
BF3734B1DCB04696  48946123050B00A7EFB1C029EE6CC438  salsa20  104 WOW-25079patch7.3.2_PTR  legion_73_afn
74F4F78002A5A1BE  C14EEC8D5AEEF93FA811D450B4E46E91  salsa20  105 WOW-25079patch7.3.2_PTR  SilithusPhase01 map
423F07656CA27D23  ????????????????????????????????  salsa20  107 WOW-25600patch7.3.5_PTR  bltestmap
0691678F83E8A75D  ????????????????????????????????  salsa20  108 WOW-25600patch7.3.5_PTR  filedataid 1782602-1782603
324498590F550556  ????????????????????????????????  salsa20  109 WOW-25600patch7.3.5_PTR  filedataid 1782615-1782619
C02C78F40BEF5998  ????????????????????????????????  salsa20  110 WOW-25600patch7.3.5_PTR  test/testtexture.blp (fdid 1782613)
47011412CCAAB541  ????????????????????????????????  salsa20  111 WOW-25600patch7.3.5_PTR  unused in 25600
23B6F5764CE2DDD6  ????????????????????????????????  salsa20  112 WOW-25600patch7.3.5_PTR  unused in 25600
8E00C6F405873583  ????????????????????????????????  salsa20  113 WOW-25600patch7.3.5_PTR  filedataid 1783470-1783472
78482170E4CFD4A6  768540C20A5B153583AD7F53130C58FE  salsa20  114 WOW-25600patch7.3.5_PTR  Magni Bronzebeard VO
B1EB52A64BFAF7BF  458133AA43949A141632C4F8596DE2B0  salsa20  115 WOW-25600patch7.3.5_PTR  dogmount, 50 files
FC6F20EE98D208F6  57790E48D35500E70DF812594F507BE7  salsa20  117 WOW-25632patch7.3.5_PTR  shop stuff
402CFABF2020D9B7  67197BCD9D0EF0C4085378FAA69A3264  salsa20  118 WOW-25678patch7.3.5_PTR  filedataid 1854762
6FA0420E902B4FBE  27B750184E5329C4E4455CBD3E1FD5AB  salsa20  119 WOW-25744patch7.3.5_PTR  legion_735_epa / legion_735_eph
1076074F2B350A2D  ????????????????????????????????  salsa20  121 WOW-26287patch8.0.1_Beta  skiff
816F00C1322CDF52  ????????????????????????????????  salsa20  122 WOW-26287patch8.0.1_Beta  snowkid
DDD295C82E60DB3C  ????????????????????????????????  salsa20  123 WOW-26287patch8.0.1_Beta  redbird
83E96F07F259F799  ????????????????????????????????  salsa20  124 WOW-26522patch8.0.1_Beta  Encrypted mount(s)? (19 files)
C1E5D7408A7D4484  A7D88E52749FA5459D644523F8359651  salsa20  226 WOW-26871patch8.0.1_Beta  Sylvanas Warbringer cinematic
E46276EB9E1A9854  CCCA36E302F9459B1D60526A31BE77C8  salsa20  227 WOW-26871patch8.0.1_Beta  ltc_a, ltc_h and ltt cinematics
D245B671DD78648C  19DCB4D45A658B54351DB7DDC81DE79E  salsa20  228 WOW-26871patch8.0.1_Beta  stz, zia, kta, jnm & ja cinematics
4C596E12D36DDFC3  B8731926389499CBD4ADBF5006CA0391  salsa20  229 WOW-26871patch8.0.1_Beta  bar cinematic
0C9ABD5081C06411  25A77CD800197EE6A32DD63F04E115FA  salsa20  230 WOW-26871patch8.0.1_Beta  zcf cinematic
3C6243057F3D9B24  58AE3E064210E3EDF9C1259CDE914C5D  salsa20  231 WOW-26871patch8.0.1_Beta  ktf cinematic
7827FBE24427E27D  34A432042073CD0B51627068D2E0BD3E  salsa20  232 WOW-26871patch8.0.1_Beta  rot cinematic
5DD92EE32BBF9ABD  ????????????????????????????????  salsa20  234 WOW-27004patch8.0.1_Subm  filedataid 2238294
 
=States of CASC Data=
CASC data comes in all forms and sizes.
 
==Key CASC Files==
===Encoding===
The encoding file maps content hashes <tt>C-Key</tt>s to encoded-file hashes <tt>E-Key</tt>s. In addition, there is information on how the files are [[CASC#BLTE_encoded_files|BLTE]]-encoded by <tt>E-Spec</tt>s.
 
Blocks in this file are, in this order
 
* header
* encoding specification data <tt>ESpec</tt>
* content key → encoding key table <tt>CEKeyPageTable</tt>
* encoding key → encoding spec table <tt>EKeySpecPageTable</tt>
* encoding specification data for the encoding file itself
 
An incomplete/outdated 010 Editor template can be found at [https://gist.github.com/heksesang/fdda3e4f8a5ed53b71ed this gist] which can be used to understand page handling.
 
====Header====
Header is a constant 0x16 bytes giving size information for the other blocks, mostly.
 
struct {
/*0x00*/  char signature[2];                            // "EN"
          enum {
            encoding_version_1 = 1,                    // {{Sandbox/VersionRange|min_expansionlevel=6|min_build=6.0.1.18125}}
          };
/*0x02*/  uint8_BE_t version;
/*0x03*/  uint8_BE_t hash_size_ckey;
/*0x04*/  uint8_BE_t hash_size_ekey;
/*0x05*/  uint16_BE_t CEKeyPageTable_page_size_kb;      // in kilo bytes. e.g. 4 in here → 4096 byte pages (default)
/*0x07*/  uint16_BE_t EKeySpecPageTable_page_size_kb;  // ^
/*0x09*/  uint32_BE_t CEKeyPageTable_page_count;
/*0x0D*/  uint32_BE_t EKeySpecPageTable_page_count;
/*0x11*/  uint8_BE_t _unknown_x11;                      // 0 -- sometimes assumed to be part of ESpec_block_size, but actually asserted to be zero by agent
/*0x12*/  uint32_BE_t ESpec_block_size;
/*0x16*/
} header;
 
====ESpec====
Encoding specification strings are just a blob of zero terminated strings referenced by <tt>EKeySpecPageTable</tt> with the accumulated size of <tt>header.ESpec_block_size</tt> (including zero terminators).
 
The definition of the format for these strings is described in the [[#Encoding_Specification_.28ESpec.29|BLTE]] section.
 
====Page Tables====
The format of the two page tables is the same:
 
* an index for fast key → page access, followed by
* the actual pages with specific content
 
In both cases, the entries in the lists have the same count, and semi-dynamic size, depending on <tt>header.hash_size_ckey</tt> and <tt>header.hash_size_ekey</tt>. Note that the page checksum size is fixed to MD5's 16 bytes. The position comments below assume the standard key size of 16 bytes.
 
struct page_index_t {
/*0x00*/  char first_Xkey[header.hash_size_Xkey];      // where X is c for CEKeyPageTable and e for EKeySpecPageTable
/*0x10*/  char page_md5[0x10];
/*0x20 usually*/
};
 
The pages themselves are filled as much possible with actual entry structs. They are padded to the end for alignment of pages. Pages don't actually have to be full.
 
=====CEKeyPageTable=====
This table maps one <tt>ckey</tt> to one or more <tt>ekey</tt>s. This means that there can be multiple representations, e.g. an encrypted and an unencrypted version. This isn't usually the case and on reading any of them can be picked, since they represent the same file content. It can be used to pick an already downloaded archive, or unencrypted one, or handle deleted archives.
 
struct ckey_ekey_entry_t {
/*0x00*/  uint8_BE_t keyCount;
/*0x01*/  uint40_BE_t file_size;                        // of the non-encoded version of the file
/*0x06*/  char ckey[header.hash_size_ckey];            // this ckey is represented by…
/*0x16*/  char ekey[header.hash_size_ekey][keyCount];  // …these ekeys
/*0x26 usually*/
} page_entries[];
 
=====EKeySpecPageTable=====
This table maps one <tt>ekey</tt> to the corresponding <tt>espec</tt> describing how the encoding happened.
 
struct ekey_espec_entry_t {
/*0x00*/  char ekey[header.hash_size_ekey];
/*0x10*/  uint32_BE_t espec_index;                      // not an offset but an index, assuming zero-terminated espec strings
/*0x14*/  uint40_BE_t file_size;                        // of the encoded version of the file
/*0x19 usually*/
} page_entries[];
 
===Install===
File signature: "IN"
 
The install file lists files installed on disk. Since the install file is shared by architectures and OSs, there are also tags to select a subset of files. When using multiple tags, a binary combination of the bitfields of files to be installed can be created.
 
====Header Structure====
The file begins with a 10 byte header describing the number of tags and files listed. '''Structure names were invented by the author of this page.'''
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="80" | Offset (Hex)
! width="70" | Type
! width="90" | Name
! width="900" | Description
|-
| 0x00 || char[2] || FileSignature || "IN"
|-
| 0x02 || uint8_t || Version? || 1
|-
| 0x03 || uint8_t || hash_size || size of hashes used for files (usually md5 -> 16)
|-
| 0x04 || uint16_BE_t || num_tags || number of tags in header of file
|-
| 0x06 || uint32_BE_t || num_entries ||  The number of entries in the body of the file
|}
 
====Tags Structure====
After the header, an array with information about available tags follows. Each tag has a bitfield listing the files installed when the given tag is chosen.
 
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="70" | Type
! width="90" | Name
! width="900" | Description
|-
| char[] || name ||
|-
| uint16_BE_t || type || A number shared amongst specific flags. Actual meaning is [[#Product_Specific|specific to products]].
|-
| char[divru (header.entries, CHAR_BIT)] || files || A bitfield that lists which files are installed when the specified tag is installed.
|}
 
====Files Structure====
The remainder of the file is populated by a list of files with their content hash, each a variable size (due to the strings). Structure names were invented by the author of this page.
 
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="70" | Type
! width="90" | Name
! width="900" | Description
|-
| char[] || FileName ||  The name of the file.
|-
| char[header.hash_size] || hash || The hash of the uncompressed file. Usually MD5.
|-
| uint32_BE_t || Size || The size of the file.
|}
 
====C-like structure====
char I; char N;
uint8_BE_t _unk3;
uint8_BE_t hash_size;
uint16_BE_t num_tags;
uint32_BE_t num_files;
struct {
  string name;
  uint16_BE_t type;
  char flags[divru (num_files, CHAR_BIT)];
} tags[num_tags];
struct {
  string name;
  char hash[hash_size];
  uint32_BE_t size;
} files[num_files];
 
===Download===
The download file lists all files stored in the data archives.
The client uses this to download files ahead of time, without it, the client will download on demand which can lead to issues.
The download priority is set inside the entries with 0 being the highest and 2 the lowest however, if the game is running, missing assets in the player's vicinity take precedence.
 
Just like the install file, the download file is shared across all architectures and locales so utilizes the same bitfield-tag system to assess what subset of files are needed.
 
NOTE: partial-priority download files do not contain the actual file sizes but redefine the FileSize field to ChunkSize.
 
This file has this structure:
 
* Header
* Entries[Header.EntryCount]
* Tags[Header.TagsCount]
 
====Download Header====
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="120" | Type
! width="150" | Name
! width="810" | Description
|-
| char[2] || Signature || The signature for this file (always "DL")
|-
| char || Version || 1 < 7.3.0, 2
|-
| char || ChecksumSize || Always 0x10
|-
| char || unk || ??? Always 1
|-
| int [BE] || EntryCount || The amount of file entries in this file
|-
| short [BE] || TagCount || The amount of tag entries in this file
|}
 
====Download Entry====
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="120" | Type
! width="150" | Name
! width="120" | Version Added
! width="810" | Description
|-
| char || Unk || 2 || ??? Appears to be a boolean. Currently only set to 1 on 4 specific records
|-
| char[16] || Hash || 1 || This hash is found in every node of the encoding file. (Reverse lookup) MD5
|-
| uint40_t [BE]  || FileSize || 1 || The compressed size of the file
|-
| char || DownloadPriority || 1 || 0 = highest, 2 = lowest
|-
| char[4] || Unk || 1 || ???
|}
 
====Download Tag====
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="120" | Type
! width="150" | Name
! width="810" | Description
|-
| string || Name || A C-String indicating this tag's Name.
|-
| short [BE] || Type || Hash type
|-
| char[N] || Bits || an array of size N = Header.EntryCount / 8 + (Header.EntryCount % 8 > 0 ? 1 : 0); that is basically a massive bit mess. Use Schroeppel's 8 bits reverse function on it to have bits.
|}
 
====code-ish====
struct {
/*0x00*/  char signature[2];                                // "DL"
          enum {
            download_version_1 = 1,
            download_version_2 = 2,                        // {{Unverified|{{Sandbox/VersionRange|min_expansionlevel=7|min_build=7.3.0.???}}}}
            download_version_3 = 3,
          };
/*0x02*/  uint8_BE_t version;
/*0x03*/  uint8_BE_t hash_size_ekey;
/*0x04*/  uint8_BE_t has_checksum_in_entry;
/*0x05*/  uint32_BE_t entry_count;
/*0x09*/  uint16_BE_t tag_count;
/*0x0b*/
#if version ≥ download_version_2
    /*0x0b*/  uint8_BE_t {{Unverified|number_of_flag_bytes}};            // defaults to 0, up to {{Unverified|4}}
    /*0x0c*/
    #if version >= download_version_3
        /*0x0c*/  uint8_BE_t {{Unverified|base_priority}};                // defaults to 0
        /*0x0d*/  char _unknown_0d[3];                      // As of 1.15.6.2-test4, this is explicitly 0. It is ignored on reading.
        /*0x10*/
    #endif
#endif
} header;
struct {
/*0x00*/  char ekey[header.hash_size_ekey];
/*0x10*/  uint40_BE_t file_size;
/*0x15*/  uint8_BE_t priority;                              // {{Unverified|header.base_priority is subtracted on parse}}
/*0x16*/
#if header.has_checksum_in_entry
/*0x16*/  uint32_BE_t checksum;
/*0x1a*/
#endif
#if header.version ≥ download_version_2
          enum {
            download_flag_plugin = 1,                      // "plugin"
            download_flag_plugin_data = 2,                  // "plugin-data"
          };
          uint8_BE_t flags[header.number_of_flag_bytes];    // {{Unverified|defaults to 0 if no flag bytes present}}
#endif
} entries[header.entry_count];
struct {
          char const name[];                                // this string is zero terminated, no fixed size
                                                            // thus for readability we start offset at 0 here.
/*0+00*/ uint16_BE_t type;                                  // game specific. usually architecture, category, locale, os, region or alike.
/*0+02*/ char mask[divru (header.entry_count._, CHAR_BIT)]; // if bit is set, entries[bit] is part of this tag
/*0x??*/
} tag[header.tag_count];
 
===Download Size===
Build 27547 introduced the Download Size file of unknown purpose. The file is a stripped-down Download file with partial EKeys and files sorted by encoded files size. The purpose of this file is not clear.
 
struct Header
{
char signature[2]; // "DS"
uint8_t version;
uint8_t ekeySize; // 9
uint32_BE_t numFiles;
uint16_BE_t numTags;
char unknown[5];
};
struct TagEntry
{
char name[]; // Null-terminated
uint16_BE_t type;
char fileMask[(hdr.numFiles + 7) / 8];
}
struct FileEntry
{
char ekey[hdr.ekeySize];
uint32_BE_t esize;
};
SizeHeader hdr;
TagEntry tags[hdr.numTags];
FileEntries[hdr.numFiles]; // Sorted descending by esize
 
===Patch===
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
! width="120" | Type
! width="150" | Name
! width="810" | Description
|-
| char[2] || Signature || The signature for this file (always "PA")
|-
| char || version || 1 or 2
|-
| char || file_key_size || <= 0x10
|-
| char || size_b || <= 0x10
|-
| char || patch_key_size || <= 0x10
|-
| char || block_size_bits || 2 <= block_size_bits <= 24. block size == 2^block_size_bits.
|-
| short || block_count ||
|-
| char || flags ||
|-
| char[16] || EncodingCkey || ckey for encoding file
|-
| char[16] || EncodingEkey || ekey for encoding file
|-
| int || DecodedSize || Decoded encoding file size in bytes
|-
| int || EncodedSize || Encoded encoding file size in bytes
|-
| char || EspecLength || Length of the following string
|-
| char[EspecLength] || EncodingEspec || espec of encoding file
|-
| char[] || ??? || byte array containing blocks entries, blocks, and optional patch tail
|}
 
 
header+entries needs to be less than 0x10000 bytes (at least in wow-18179). md5sum is only checked for header+entries, file might be larger thus.
 
struct PatchManifest_Header
{
  uint16_BE_t magic; // 'PA'
  uint8_t version; // 1 or 2
  uint8_t file_key_size; // <= 0x10
  uint8_t size_b; // <= 0x10
  uint8_t patch_key_size; // <= 0x10
  uint8_t block_size_bits; // 12 <= block_size_bits <= 24. max block size == 2^block_size_bits
  uint16_BE_t block_count; // (file_key_size + 20) * entry_count + sizeof (PatchManifest_Header) < 0x10000
  uint8_t unk2; // flags
#if encoding_information_apparently_added_after_18179
  uint8_t encoding_ckey[16];
  uint8_t encoding_ekey[16]; // probably since PA2
  uint32_BE_t decoded_size;
  uint32_BE_t encoded_size;
  uint8_t encoding_espec_length;
  char encoding_format[encoding_espec_length];
#endif
  } header;
struct PatchManifest_Block
{
  uint8_t last_file_ckey[header.file_key_size];
  uint8_t md5_of_block[16];
  uint32_BE_t block_offset; // in this file
} blocks[header.block_count]; // sorted ascending by key
// at positions given in PatchManifest_Block
struct block
{
  struct
  {
    uint8_t num_patches; // <= 0x10.
    uint8_t target_file_ckey[header.file_key_size];
    uint40_BE_t decoded_size;
    struct
    {
      uint8_t source_file_ekey[header.file_key_size];
      uint40_BE_t decoded_size;
      uint8_t patch_ekey[header.patch_key_size];
      uint32_BE_t patch_size;
      uint8_t unk; // some sort of patch index number. first entry seems to always be 1
    } patches[num_patches];
  } files[]; // count unspecified: read until the next file num_patches would be 0
                // OR block would exceed max block size
};
// some files have a block of data after the last block of the patch manifest (which may be shorter than max block size). this block appears to be a patch of encoding, but the format is not understood.
 
==CDN File Organization==
Data for every CASC-based game exists on the CDN in one of two places at any given time. To reduce file-system and download overhead many files are packed into archives and indexed by archive indices, both of a different format than the CASC installations found on client systems; other "unarchived", "standalone", or "loose" files are stored as separate files on the CDN that must be downloaded independently. There are at least three different reasons why a particular file is found in one or the other:
# Small files typically incur larger filesystem overhead and benefit most from being packed into archives. A rough rule of thumb appears to be that files smaller than 2 MB or so are put in archives. Presumably larger files are not archived because they make it more difficult to minimize the number of archive files, which are limited in size (a 2 MB file limit would limit unused space in an archive to under 0.8%, given enough data to form a full archive to begin with).
# Key files such as encoding, partial-priority, TVFS (Warcraft 3 root), and the download, install, and patch manifests, as well as their respective patches, are typically stored loose for quick access and are rarely ever found in archives.
# As games evolve old files become obsolete and are removed from the CDN. However, the archive system means that archives can only be removed when every file in the archive is no longer needed, potentially wasting large amounts of space on the CDN - the exact opposite of the purpose of bundling files into archives to begin with. Thus as the amount of unused data in an archive grows over time, files still in use may be converted to loose files to allow the archive to be purged from the CDN, even when the files are unusually small to be found independently.
 
Previously, no official indices/manifests of loose files existed, and they could only be found by subtracting archived files from file lists in encoding or manifest files. Beginning with Warcraft 3 and subsequently being deployed more widely on 7/24/2018, new fields in the CDN config file link to index files containing "all" loose data or patch files on the current CDNs.
 
===Archives===
Archives are extensionless 256 MB files that are usually only stored on the Blizzard CDNs. Their naming follows the standard URL hash format using the '/data/' path type.
 
The structure of the archives is presumably just file fragment after file fragment. You will never need to parse it because you can just look up offset + size of your file fragment in the index files and then take the piece directly out of the archive.
 
The fragments are all BLTE encoded.
 
The filename is '''NOT''' the hash of the archive content but the hash of the '''index''''s footer.
 
===Archive Indexes (.index)===
These '.index' files reveal to the user where the compressed game files are located within the archives. All indexes (except the Archive-Group index, see below) are named after their archive (only difference is these have an extension).
'.index' files are stored on the CDN using the standard hash naming scheme (remember they have an extension though). They are also located in the directory 'INSTALL_DIR/Data/indices/' for a WoW install.
 
====Normal Index Entry Structure====
*'''The file is divided into 4kb chunks populated by these standard index entries of 0x18 (hex) bytes. Each chunk is zero-padded to a full 4kb, though there may be more than 0x18 bytes of padding at the end of a chunk -- be sure the check for all-null blte_header_hash fields. The last chunk is a table-of-contents, listing the LAST blte_header_hash in each chunk and checksum of each block. All checksums are the first checksumSize bytes of the MD5 of the respective data. Structure names were invented by the author of this page.'''
 
struct index_entry {
  char blte_header_hash[footer.keySizeInBytes];
  uint_BE_t(footer.sizeBytes) blte_encoded_size;
  uint_BE_t(footer.offsetBytes) offset_to_blte_encoded_data_in_archive;
};
struct index_block {
  static constexpr const block_size = footer.blockSizeKb << 10;
  index_entry entries[block_size / sizeof (index_entry)];
  char padding[block_size - sizeof (entries)];
} blocks[];
struct {
  struct {
    char last_hash[footer.keySizeInBytes]; // last hash of a block
  } entries[num_blocks];
  struct {
    char lower_part_of_md5_of_block[footer.checksumSize];
  } blocks_hash[num_blocks];
} toc;
struct {
  char toc_hash[checksumSize]; // client tries to read with 0x10, then backs off when smaller
  char version?;        // always 1
  char _11;            // 0
  char _12;            // 0
  char blockSizeKb?;    // Normally 4. Left-shifted by 10. Believed to be block size in KB.
  char offsetBytes;    // Normally 4 for archive indices, 6 for group indices, and 0 for loose file indices
  char sizeBytes;      // Normally 4
  char keySizeInBytes;  // Normally 16
  char checksumSize;    // Normally 8, <= 0x10
  uint32_t numElements; // BigEndian in _old_ versions (e.g. 18179)
  char footerChecksum[checksumSize];
} footer;
 
* footerChecksum is calculated over the footer beginning with version ''when footerChecksum is zeroed''
* The archive/index name is the MD5 of the footer beginning with toc_hash
 
===archive-group===
<tt>archive-group</tt> is actually a very special <tt>.index</tt> file that is typically much larger than other indices. There is '''no''' archive attached, it is only the index. Also, the index is '''not''' found on the CDN! It is assembled by combining all index files given in the CDN config, on client side. The hash is just for verification purposes. Client side a file is assembled using the normal index file format, for caching.
 
It has a single difference in format to normal indices: While other indices have their <tt>offsetBytes</tt> long <tt>offset</tt> field point into the archive, for archive groups, the field also has an <tt>archiveIndex</tt>:
 
struct {
  uint16_BE_t archiveIndex; // Index of the archive in the CDN config's archive list
  uint32_BE_t offsetBytes;  // The offset within the specified archive
};
 
It does not only having the offset, but also an index into a file. Semantically that's still an offset, so no further field for size is used.
 
It is suggested you do not just parse indices by <tt>.index</tt> filename locally but take the config files into account. An easy heuristic is that if <tt>offsetBytes</tt> is not 4, it is a special index, either loose files or a group.
 
===patch-archives===
Patch archives are the /patch/ equivalent to the regular (data) archives on the CDN. Like archives, these are binary blobs of fragments indexed by an accompanying .index file with the same name. Again, the index is a <tt>hash, size, offset</tt> tuple, but the hash is the content hash rather than an encoding hash.
 
Most files in patch archives are [[Patching Files|ZBSDIFF1]] blobs, though in principle any file that might be in the /patch/ namespace may be found in patch archives and must be handled accordingly.
 
===patch-archive-group===
See [[#archive-group|archive-group]]. There is no known difference other than the combined data being <tt>patch-archives</tt>.
 
==Journal-based Data Files==
During the installation process for a Blizzard game, the program will download the required files as requested by root, encoding, download, and install. It stores the downloaded data fragments in data files in "INSTALL_DIR\Data\data\". The program will record the content hash (BLTE-compressed hash), size, and position of the file as well as the number of the data file that it is in. It places those four parameters into journal files with the extension '.idx'.
 
===Shared Memory===
The shared memory file is called 'shmem' and is usually located in the same folder as the data and .IDX journals. This file contains the path where the data files are stored, which is the current version of each of the .IDX files, and which areas of the data files have unused space. The file is recreated every time a client is started.
The shared memory file is called 'shmem' and is usually located in the same folder as the data and .IDX journals. This file contains the path where the data files are stored, which is the current version of each of the .IDX files, and which areas of the data files have unused space. The file is recreated every time a client is started.


====Shared Memory Header Structure====
===Shared Memory Header Structure===
*'''The first part of the header.'''
*'''The first part of the header.'''
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
Line 1,145: Line 25:
! width="90" | Name
! width="90" | Name
! width="900" | Description
! width="900" | Description
|-
|- style="vertical-align: top;"
| 0x00 || uint32_t || BlockType || A value indicating what type of block this is. For this block, the value is 4.
| 0x00 || uint32_t || BlockType || A value indicating what type of block this is. For this block, the value is either 4 or 5.<br>• If the type is 5, then the free space block contains no entries. (see below)
|-  
|-  
| 0x04 || uint32_t || NextBlock || The offset of the next block.
| 0x04 || uint32_t || NextBlock || The offset of the next block.
|-  
|- style="vertical-align: top;"
| 0x08 || char[0x100] || DataPath || The path of the data files. This is prefixed with "Global\" if the path is an absolute path.
| 0x08 || char[0x100] || DataPath || The path to the data files. It is always prefixed with "Global\". The path uses forward slashes (except the prefix).<br>• When the file is written by the Battle.Net client, it is an absolute path.<br>• When the file is written by the game, it is a relative path (relative from the game executable).
|}
|}


Line 1,177: Line 57:
|}
|}


 
===Shared Memory Free Space Structure===
====Shared Memory Free Space Structure====
After a small header, this structure is split up into two equal parts.
After a small header, this structure is split up into two equal parts.
The first part contains entries with the number of unused bytes.
The first part contains entries with the number of unused bytes.
Line 1,207: Line 86:
! width="900" | Description
! width="900" | Description
|-  
|-  
| 0x00 || uint10* || DataNumber || This is always set to 0 in this part of the block.
| 0x00 || uint10* || DataNumber || Before Agent v8012? this was always set to 0. After Agent v8012? this appears to be 1 when the data file referenced by the equivalent entry in the unused byte positions section has not been created.
|-  
|-  
| 0x01 || uint30* || Count || The number of unused bytes.
| 0x01 || uint30* || Count || The number of unused bytes.
Line 1,225: Line 104:
|}
|}


===.IDX Journals===
=== Special Case: Block Type 5 ===
There is a special case after a game update by the Battle.Net client. It seems that the client only writes a template file without free space entries and lets the game later complete the file. The following applies in this case:
 
* the first block type is set to 5 (instead of 4) with the same layout
* file size is always 16,384 bytes long
* offset to next block is set to 4,096<br>''this is misleading: it is still at 340, and the free space block is found at that offset, not at offset 4,096 (340 is indicated by the block size field of the 1st block)''
* the free space block has an entry count of zero but is still padded to 1090 entries as usual
* after the terminator block, the file is additionally padded to 16,384 bytes
 
==.IDX Journals==
Example file path: INSTALL_DIR\Data\data\0e00000054.idx
Example file path: INSTALL_DIR\Data\data\0e00000054.idx


Line 1,237: Line 125:
   }
   }


To determine the bucket index of the cross-reference entries at the start of each data file, use that same function and add 1 to the result, modulo 16 for the number of indices.


====.IDX Header Structure====
  uint8_t cascGetBucketIndexCrossReference(const uint8_t k[16]) {
    uint8_t i = cascGetBucketIndex(k);
    return (i + 1) % 16;
  }
 
 
===.IDX Header Structure===


The header is little-endian:
The header is little-endian:
Line 1,275: Line 170:
|}
|}


====.IDX Entry Structure====
===.IDX Entry Structure===
*'''The rest of the file is populated by these normal entries, each 0x12 bytes in size. Structure names were invented by the author of this section because official names were not available.'''
*'''The rest of the file is populated by these normal entries, each 0x12 bytes in size. Structure names were invented by the author of this section because official names were not available.'''
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
{| border="1" cellpadding="2" style="background:#FCFCFC; color:black"
Line 1,291: Line 186:
*'''* designates unusual data types. In C#, you can read the Offset by reading a Byte, reading a big-endian UInt32, shifting the byte left 32 bits, and ORing them together.  Use a 30-bit mask (0x3fffffff) to get the file offset, and right shift the value 30 bits to get the archive number.
*'''* designates unusual data types. In C#, you can read the Offset by reading a Byte, reading a big-endian UInt32, shifting the byte left 32 bits, and ORing them together.  Use a 30-bit mask (0x3fffffff) to get the file offset, and right shift the value 30 bits to get the archive number.


===.XXX Data Files===
==.XXX Data Files==
Example file path: INSTALL_DIR\Data\data\data.015
Example file path: INSTALL_DIR\Data\data\data.015


These files consist of a sequence of headers with corresponding BLTE data.
These files consist of a sequence of headers with corresponding [[BLTE]] data.


Most .xxx archives begin with 16 special index cross-linking files. These files have no data and have encoding keys of XXYYbba1af16c50e1900000000000000, where XX is the index number and YY is the .xxx number. The purpose of these files is unclear.
Most .xxx archives begin with 16 special index cross-linking files. These files have no data and have encoding keys of XXYYbba1af16c50e1900000000000000, where XX is the index number and YY is the .xxx number. The purpose of these files is unclear.
Line 1,313: Line 208:
| 0x16 || uint32_t || ChecksumA || hashlittle(first 0x16 bytes of the header, 0x3D6BE971)
| 0x16 || uint32_t || ChecksumA || hashlittle(first 0x16 bytes of the header, 0x3D6BE971)
|-  
|-  
| 0x1A || uint32_t || ChecksumB || Checksum of the first 0x1A bytes of the header. The exact algorithm seems to vary over time.
| 0x1A || uint32_t || ChecksumB || Checksum of the first 0x1A bytes of the header. The exact algorithm seems to vary over time. The current implementation is described below
|}
|}
The algorithm for calculating ChecksumB is equivalent to the following C code:
    // Table is extracted from Agent.exe 8020. Hasn't changed for quite a while
    uint32_t TABLE_16C57A8[0x10] = {
        0x049396b8, 0x72a82a9b, 0xee626cca, 0x9917754f, 0x15de40b1, 0xf5a8a9b6, 0x421eac7e, 0xa9d55c9a,
        0x317fd40c, 0x04faf80d, 0x3d6be971, 0x52933cfd, 0x27f64b7d, 0xc6f5c11b, 0xd5757e3a, 0x6c388745,
    };
   
    // Arguments:
    //  header: Pointer to the memory containing the header
    //  archive_index: Number of the data file the record is stored in (e.g. xxx in data.xxx)
    //  archive_offset: Offset of the header inside the archive file
    // Precondition: Header is at least 0x1e bytes (e.g a full header)
    // Precondition: checksum_a has already been calculated and stored in the header
    // Assumption: Code is written assuming little endian.
    uint32_t checksum(uint8_t *header, uint16_t archive_index, uint32_t archive_offset) {
        // Top two bits of the offset must be set to the bottom two bits of the archive index
        uint32_t offset = (offset & 0x3fffffff) | (archive_index & 3) << 30;
   
        uint32_t encoded_offset = TABLE_16C57A8[(offset + 0x1e) & 0xf] ^ (offset + 0x1e);
   
        uint32_t hashed_header = 0;
        for (int i = 0; i < 0x1a; i++) { // offset of checksum_b in header
            ((uint8_t *)&hashed_header)[(i + offset) & 3] ^= header[i];
        }
   
        uint32_t checksum_b = 0;
        for (int j = 0; j < 4; j++) {
            int i = j + 0x1a + offset;
            ((uint8_t *)&checksum_b)[j] = ((uint8_t *)&hashed_header)[i & 3] ^ ((uint8_t *)&encoded_offset)[i & 3];
        }
        return checksum_b;
    }




Line 1,324: Line 252:
! width="900" | Description
! width="900" | Description
|-
|-
| 0x00 || char[Header.Size - 30] || Data || The BLTE file data. See the BLTE section above.
| 0x00 || char[Header.Size - 30] || Data || The [[BLTE]] file data. See the [[BLTE]] page.
|}
|}
=Product Specific=
In this section, the game/usage specific parts of CASC are describe. While CASC is a generic format, a lot of stuff is hardcoded, like hash sizes. Other parts are actually left up to the implementation, like root files or download tags.
==World of Warcraft==
===Root===
struct CASRecord {
    char content_key[16];  // MD5 hash of the file's raw data
    uint64 name_hash;      // Jenkins96 (lookup3) hash of the file's path
};
enum locale_flags : uint32 {
  enUS = 0x2,
  koKR = 0x4,
  frFR = 0x10,
  deDE = 0x20,
  zhCN = 0x40,
  esES = 0x80,
  zhTW = 0x100,
  enGB = 0x200,
  enCN = 0x400,
  enTW = 0x800,
  esMX = 0x1000,
  ruRU = 0x2000,
  ptBR = 0x4000,
  itIT = 0x8000,
  ptPT = 0x10000,
};
enum content_flags : uint32 {
  LowViolence = 0x80,
  Bundle = 0x40000000,
  NoCompression = 0x80000000,
};
struct CASBlock {
    int32 num_records;
    content_flags flags;
    locale_flags locale;
    int32 fileDataIDDeltas[num_records];  // each block starts with 0, +1 is implicit per entry, so consecutive ids will have delta=0
    CASRecord records[num_records];
    int32 file_data_id (size_t index) const
    {
      return index == 0
        ? fileDataIDDeltas[index]
        : file_data_id (index - 1) + 1 + fileDataIDDeltas[index];
    }
};
while (FTell() < FileSize())
    CASBlock blocks;
===Tags===
Values depend on versions, semantic categories are cross version.
* '''Platform''': The deployment target, i.e. Windows or OSX
* '''Architecture''': Sub-division of the deployment target, i.e. x86_32 or x86_64
* '''Locale''': The same as in [[Localization]]: Files specific to a single localisation of the game.
* '''Region''': Equivalent to the patch server regions, i.e. us, eu, kr, tw, cn.
* '''Category''': A replacement for the MPQ system to tag low priority downloads: speech, text
* '''Alternate''': A special category for censored content.
====Version specific values====
{{Template:SectionBox/VersionRange|min_expansionlevel=6|min_build=6.0.1.18125|max_expansionlevel=6|max_build=6.0.1.18761}}
Architecture = 1, Locale = 2, Platform = 3
{{Template:SectionBox/VersionRange|min_expansionlevel=6|min_build=6.0.1.18764|max_expansionlevel=6|max_build=6.2.2.20426}}
Architecture = 1, Category = 2, Locale = 3, Platform = 4, Region = 5
{{Template:SectionBox/VersionRange|min_expansionlevel=6|min_build=6.2.2.20438|max_expansionlevel=7}}
Platform = 1, Architecture = 2, Locale = 3, Region = 4, Category = 5
{{SectionBox/VersionRange|min_expansionlevel=8|min_build=8.0.1.26604|note={{Unverified|Actually alternate was just missing above but schlumpf doesn't want to verify since when}}}}
enum {
  platform = 1,
  architecture = 2,
  locale = 3,
  region = 4,
  category = 5,
  alternate = 0x4000,
};
=hashpath=
hashpath (string path) → uint32_t
{
  string normalized = toupper (path).replace (from: '/', to: '\\')
  uint32_t pc = 0, pb = 0;
  hashlittle2 (normalized, strlen (normalized), &pc, &pb);
  return pc;
}


[[Category:Format]]
[[Category:Format]]

Revision as of 16:07, 30 September 2022

Missing something? This page was recently split up into separate pages. For the content transfer part of NGDP, see TACT. This page should now only contain information on the local filesystem format called CASC.

CASC is the name of the new file system that Blizzard created to replace the outdated format of MPQ. It is used in WoW versions ≥ WoD.

CASC v1

The CASC file system made its first debut in the Heroes of the Storm Technical Alpha, which was hosted on Blizzard's servers in late January. The form of CASC that Heroes of the Storm uses is designated by Blizzard as "CASC". In contrast, World of Warcraft's "build-playbuild-installer" config line clearly states it is generated by "ngdptool_casc2" (NGDP stands for Next Generation Download Procotol). These are the two most substantial changes between CASC v1 and CASC v2:

  • Sections of CASC v1 data files are grouped together in collections of files we call "packages". These packages all have the same root folder, and if all of the files are not properly added with the package's base directory, the extraction process will produce an incredibly mangled directory output. This system is completely removed in CASC v2.
  • CASC v1's Root file relates content hashes to file names. CASC v2's Root file relates content hashes to name hashes. Translating name hashes to file names requires use of the Jenkins Hash function [1], which in turn requires a listfile to generate the hashes. Essentially CASC v1 has its own listfile (in root). CASC v2 does not, and requires the user to provide names.

The remainder of this article will refer exclusively to the system called CASC v2 as 'CASC'. While many parts of the file system are identical between v1 and v2, there are enough changes to make explaining both formats at once inadvisable.

Journal-based Data Files

During the installation process for a Blizzard game, the program will download the required files as requested by root, encoding, download, and install. It stores the downloaded data fragments in data files in "INSTALL_DIR\Data\data\". The program will record the content hash (BLTE-compressed hash), size, and position of the file as well as the number of the data file that it is in. It places those four parameters into journal files with the extension '.idx'.

Shared Memory

The shared memory file is called 'shmem' and is usually located in the same folder as the data and .IDX journals. This file contains the path where the data files are stored, which is the current version of each of the .IDX files, and which areas of the data files have unused space. The file is recreated every time a client is started.

Shared Memory Header Structure

  • The first part of the header.
Offset (Hex) Type Name Description
0x00 uint32_t BlockType A value indicating what type of block this is. For this block, the value is either 4 or 5.
• If the type is 5, then the free space block contains no entries. (see below)
0x04 uint32_t NextBlock The offset of the next block.
0x08 char[0x100] DataPath The path to the data files. It is always prefixed with "Global\". The path uses forward slashes (except the prefix).
• When the file is written by the Battle.Net client, it is an absolute path.
• When the file is written by the game, it is a relative path (relative from the game executable).


  • Followed by a number of these entries. The count can be calculated like this: (NextBlock - 264 - idxFileCount * 4) / 8
Offset (Hex) Type Name Description
0x00 uint32_t Size The size of the block.
0x04 uint32_t Offset The offset of the block.


  • Followed by a number of these entries. The count is equal to number of .IDX files (usually 16).
Offset (Hex) Type Name Description
0x00 uint32_t Version The version number. Used to identify the .IDX filename.

Shared Memory Free Space Structure

After a small header, this structure is split up into two equal parts. The first part contains entries with the number of unused bytes. The second part contains entries with the position of the unused bytes.

There can be up to 1090 entries. Each of the two parts will always be 5450 bytes, so if there are fewer than 1090 entries, the rest of the bytes will be padded with '\0'.

  • The header part of the structure.
Offset (Hex) Type Name Description
0x00 uint32_t BlockType A value indicating what type of block this is. For this block, the value is 1.
0x04 uint32_t NextBlock The offset of the next block.
0x08 char[0x18] Padding Padding at the end of the header.


  • This is the number of unused bytes. There can be up to 1090 entries of these. If there are fewer, the rest of the area is padded.
Offset (Hex) Type Name Description
0x00 uint10* DataNumber Before Agent v8012? this was always set to 0. After Agent v8012? this appears to be 1 when the data file referenced by the equivalent entry in the unused byte positions section has not been created.
0x01 uint30* Count The number of unused bytes.


  • This is the position of the unused bytes. There can be up to 1090 entries of these. If there are fewer, the rest of the area is padded.
Offset (Hex) Type Name Description
0x00 uint10* DataNumber The number of the data file where the unused bytes are located.
0x01 uint30* Offset The position within the data file where the unused bytes are located.

Special Case: Block Type 5

There is a special case after a game update by the Battle.Net client. It seems that the client only writes a template file without free space entries and lets the game later complete the file. The following applies in this case:

  • the first block type is set to 5 (instead of 4) with the same layout
  • file size is always 16,384 bytes long
  • offset to next block is set to 4,096
    this is misleading: it is still at 340, and the free space block is found at that offset, not at offset 4,096 (340 is indicated by the block size field of the 1st block)
  • the free space block has an entry count of zero but is still padded to 1090 entries as usual
  • after the terminator block, the file is additionally padded to 16,384 bytes

.IDX Journals

Example file path: INSTALL_DIR\Data\data\0e00000054.idx

.IDX journals contain a mapping from keys to the location of their data in the local CASC archives. There used to be one .IDX file per journal, and the naming scheme used to have two separate meanings. The '0e' part of the file name used to designate which archive the .IDX file was associated with. This changed halfway through the Warlords Beta. Now there are 16 indices total, and the first byte of the hex filename says which of the 16 indices it is, while the remainder of the hex filename is just a version number that increments when a new set of files is added to the local archives.

To determine which of the 16 indices a key is bucketed in, the key is hashed by xoring together each 4-bit nibble in the first 9 bytes of the key:

 uint8_t cascGetBucketIndex(const uint8_t k[16]) {
   uint8_t i = k[0] ^ k[1] ^ k[2] ^ k[3] ^ k[4] ^ k[5] ^ k[6] ^ k[7] ^ k[8];
   return (i & 0xf) ^ (i >> 4);
 }

To determine the bucket index of the cross-reference entries at the start of each data file, use that same function and add 1 to the result, modulo 16 for the number of indices.

 uint8_t cascGetBucketIndexCrossReference(const uint8_t k[16]) {
   uint8_t i = cascGetBucketIndex(k);
   return (i + 1) % 16;
 }


.IDX Header Structure

The header is little-endian:

Offset (Hex) Type Name Description
0x00 uint32 HeaderHashSize The number of bytes to use for the hash at +04; usually 0x10.
0x04 uint32 HeaderHash This should equal the value of pc after calling hashlittle2 on the following HeaderHashSize bytes of the file with an initial value of 0 for pb and pc.
0x08 uint16 Unk0 Must be 7
0x0a uint8 BucketIndex The bucket index of this file; should be the same as the first byte of the hex filename.
0x0b uint8 Unk1 Must be 0
0x0c uint8 EntrySizeBytes Must be 4
0x0d uint8 EntryOffsetBytes Must be 5
0x0e uint8 EntryKeyBytes Must be 9
0x0f uint8 ArchiveFileHeaderBytes Must be 30
0x10 uint64 ArchiveTotalSizeMaximum The maximum size of a casc installation; 0x4000000000, or 256GiB.
0x18 char[8] padding The header is padded with zeroes to the next 0x10-byte boundary.
0x20 uint32 EntriesSize This is the length in bytes of the entries in the index file.
0x24 uint32 EntriesHash This should equal the value of pc after calling hashlittle2 on the following EntriesSize bytes of the file with an initial value of 0 for pb and pc.

.IDX Entry Structure

  • The rest of the file is populated by these normal entries, each 0x12 bytes in size. Structure names were invented by the author of this section because official names were not available.
Offset (Dec) Type Name Description
00 char[9] Key The first 9 bytes of the key for this entry.
09 uint40* Offset Unlike the other little-endian integers in this file, this is a big-endian 5-byte integer. The top 10 bits are the number of the archive (data.%03d), and the bottom 30 bits are the offset in that archive to the file data.
14 uint32 Size The length of the file in bytes.
  • * designates unusual data types. In C#, you can read the Offset by reading a Byte, reading a big-endian UInt32, shifting the byte left 32 bits, and ORing them together. Use a 30-bit mask (0x3fffffff) to get the file offset, and right shift the value 30 bits to get the archive number.

.XXX Data Files

Example file path: INSTALL_DIR\Data\data\data.015

These files consist of a sequence of headers with corresponding BLTE data.

Most .xxx archives begin with 16 special index cross-linking files. These files have no data and have encoding keys of XXYYbba1af16c50e1900000000000000, where XX is the index number and YY is the .xxx number. The purpose of these files is unclear.

  • The data header.
Offset (Hex) Type Name Description
0x00 char[0x10] BlteHash Encoding key of the file, in reversed byte order. Note that only as many bytes (final bytes in this reversed order) of this key as are contained in the .idx files (9) must be accurate, and the remaining 7 bytes may be 0s or otherwise altered.
0x10 uint32_t Size The size of this header + the following data.
0x14 char[0x02] Flags?? Unknown. Mostly 0. Set to 1,0 by Agent.exe on index cross-linking files, possibly indicating data-less metadata files.
0x16 uint32_t ChecksumA hashlittle(first 0x16 bytes of the header, 0x3D6BE971)
0x1A uint32_t ChecksumB Checksum of the first 0x1A bytes of the header. The exact algorithm seems to vary over time. The current implementation is described below

The algorithm for calculating ChecksumB is equivalent to the following C code:

   // Table is extracted from Agent.exe 8020. Hasn't changed for quite a while
   uint32_t TABLE_16C57A8[0x10] = {
       0x049396b8, 0x72a82a9b, 0xee626cca, 0x9917754f, 0x15de40b1, 0xf5a8a9b6, 0x421eac7e, 0xa9d55c9a,
       0x317fd40c, 0x04faf80d, 0x3d6be971, 0x52933cfd, 0x27f64b7d, 0xc6f5c11b, 0xd5757e3a, 0x6c388745,
   };
   
   // Arguments:
   //  header: Pointer to the memory containing the header
   //  archive_index: Number of the data file the record is stored in (e.g. xxx in data.xxx)
   //  archive_offset: Offset of the header inside the archive file
   // Precondition: Header is at least 0x1e bytes (e.g a full header)
   // Precondition: checksum_a has already been calculated and stored in the header
   // Assumption: Code is written assuming little endian.
   uint32_t checksum(uint8_t *header, uint16_t archive_index, uint32_t archive_offset) {
       // Top two bits of the offset must be set to the bottom two bits of the archive index
       uint32_t offset = (offset & 0x3fffffff) | (archive_index & 3) << 30;
   
       uint32_t encoded_offset = TABLE_16C57A8[(offset + 0x1e) & 0xf] ^ (offset + 0x1e);
   
       uint32_t hashed_header = 0;
       for (int i = 0; i < 0x1a; i++) { // offset of checksum_b in header
           ((uint8_t *)&hashed_header)[(i + offset) & 3] ^= header[i];
       }
   
       uint32_t checksum_b = 0;
       for (int j = 0; j < 4; j++) {
           int i = j + 0x1a + offset;
           ((uint8_t *)&checksum_b)[j] = ((uint8_t *)&hashed_header)[i & 3] ^ ((uint8_t *)&encoded_offset)[i & 3];
       }
       return checksum_b;
   }


  • The BLTE data.
Offset (Hex) Type Name Description
0x00 char[Header.Size - 30] Data The BLTE file data. See the BLTE page.