CASC
CASC is the name of the new file system that Blizzard has created to replace the outdated format of MPQ.
CASC v1
The CASC file system made its first debut in the Heroes of the Storm Technical Alpha, which was hosted on Blizzard's servers in late January. The form of CASC that Heroes of the Storm uses is designated by Blizzard as "CASC". In contrast, World of Warcraft's "build-playbuild-installer" config line clearly states it is generated by "ngdptool_casc2" (NGDP stands for Next Generation Download Procotol). These are the two most substantial changes between CASC v1 and CASC v2:
- Sections of CASC v1 data files are grouped together in collections of files we call "packages". These packages all have the same root folder, and if all of the files are not properly added with the package's base directory, the extraction process will produce an incredibly mangled directory output.
- CASC v1's root file relates content hashes to file names. CASC v2's root file relates content hashes to name hashes. Translating name hashes to file names requires use of the Jenkins Hash function [1], which in turn requires a listfile to generate the hashes. Essentially CASC v1 has its own listfile (in root). CASC v2 does not, and requires the user to provide names.
The remainder of this article will refer exclusively to the system called CASC v2. While many parts of the file system are identical between v1 and v2, there are enough changes to make explaining both formats at once inadvisable.
NGDP
CASC was introduced simultaneously with a new system for managing configuration, blob, and installation files called NGDP, or Next Generation Download Protocol. When the acronym 'NGDP' is used in conjunction with the term CASC, it is typically referring to the hosted components of the CASC file system, and its ability to stream data on the fly.
NGDP URLs
As of October 14th, 2014, the following generic NGDP URLs are known:
- http://us.patch.battle.net:1119/ProgramCode/cdns - a list domains available with game data per region
- http://us.patch.battle.net:1119/ProgramCode/versions - a list of the current game version, build config, and cdn config per region
- http://us.patch.battle.net:1119/ProgramCode/bgdl - similar to versions, but tailored for use by the Battle.net App background downloader process
- http://us.patch.battle.net:1119/ProgramCode/blob/game - a blob file that regulates game functionality for the Battle.net App
- http://us.patch.battle.net:1119/ProgramCode/blob/install - a blob file that regulates installer functionality for the game in the Battle.net App
NGDP Program Codes
As of October 14th, 2014, the following program codes are known to support NGDP:
- heroes
- storm - Note: Deprecated
- wow
- wowt
- wow_beta
CASC Online
Standard URL Hash Format
URL Format: http://(cdnsHost)/(cdnsPath)/(pathType)/(FirstTwoHexOfHash)/(SecondTwoHexOfHash)/(FullHash)
For WoW, cdnsHost of dist.blizzard.com.edgesuite.net should always be acceptable, and currently the cdnsPath of "tpr/wow" has never changed.
Known path types are:
- config - contains the three types of config files: Build configs, CDN configs, and Patch configs
- data - contains archives, indexes, and unarchived standalone files (typically binaries, mp3s, and movies)
- patch - contains 'PA' files
Example URL: http://dist.blizzard.com.edgesuite.net/tpr/wow/config/5b/27/5b277d732299a79a935bc5a3f6ed3240
Config Files
Build Config
Example file: http://dist.blizzard.com.edgesuite.net/tpr/wow/config/4d/8b/4d8bb3b6fd0416aa9371a80aaefd2e93
CDN Config
Example file: http://dist.blizzard.com.edgesuite.net/tpr/wow/config/5b/27/5b277d732299a79a935bc5a3f6ed3240
Patch Config
This configuration file was added after all of the others. It first appeared in CASC v1 for Heroes of the Storm in August 2014. It then appeared in WoW for CASC v2 around build 19000 (approximately October 1st, 2014). The purpose of this file is to reduce redundant downloads. It achieves this by directing the system to download patch files to apply and update previously downloaded material. The structure and purpose of all of the fields of this file are unknown at this time.
Data Files
Patch Files
File signature = "PA" The structure and purpose of all of the fields of this file are unknown at this time.
States of CASC Data
Blizzard-Created Archives
In its natural state, the vast majority of the data for any CASC-based game exists in the archives.
Archives
Archives are extensionless 256 MB files that are usually only stored on the Blizzard CDNs. Their naming follows the standard URL hash format using the '/data/' path type.
The structure of the archives is presumably just file fragment after file fragment. You will never need to parse it because you can just look up offset + size of your file fragment in the index files and then take the piece directly out of the archive.
Archive Indexes (.index)
These '.index' files reveal to the user where the compressed game files are located within the archives. All indexes (except the Archive-Group index, see below) are named after their archive (only difference is these have an extension).
Normal IndexEntry Structure
- The entire file is populated by these IndexEntries of 0x18 bytes.
- NOTE: This structure uses big endian numbers.
Offset Type Name Description 0x00 char[16] ContentHash The MD5 of the compressed BLTE fragment that this index entry represents 0x10 uint32 Offset Position of the fragment in the archive 0x14 uint32 Size Size of the fragment
Archive-Group Index (.index)
Archive-group is actually a very special '.index' file. While virtually all '.index' files are under 2 MB, the archive-group '.index' file is always over 15 MB. It is essentially a merger of all .index files, with a structure change. There is a new uint16 field that serves as an index for the array of archives from this build's CDN config.
Therefore, it is critical that you identify this outlier - if you try to parse it as a regular '.index' purely because of its extension, your program will undoubtedly fail. You can identify it because it will be named the same as the 'archive-group' hash listed in the CDN config. Additionally, it will not be listed as an archive hash in the CDN config. As discussed before, the different file structure and irregular file size are also viable methods to avoid parsing this file (or to avoid parsing the other '.index' files).
Merged IndexEntry Structure
- The entire file is populated by these IndexEntries of 0x1A bytes.
- NOTE: This structure uses big endian numbers.
Offset Type Name Description 0x00 char[16] ContentHash The MD5 of the compressed BLTE fragment that this index entry represents 0x10 uint16 ArchiveIndex If you placed the hashes of the 'archives = ' line of the CDN config in an array, this number would be the index for that array. 0x12 uint32 Offset Position of the fragment in the archive 0x16 uint32 Size Size of the fragment
Journal-based Data Files
During the installation process for a Blizzard game, the program will download the required files as requested by root, encoding, download, and install. It stores the downloaded data fragments in data files in "INSTALL_DIR\Data\data\". The program will record the content hash (BLTE-compressed hash), size, and position of the file as well as the number of the data file that it is in. It places those four parameters into journal files with the extension '.idx'.
.IDX Journals
Example file path: INSTALL_DIR\Data\data\0e00000054.idx
.XXX Data Files
Example file path: INSTALL_DIR\Data\data\data.015