TACT

From wowdev
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

TACT (Trusted Application Content Transfer) is the name of the content transfer part of NGDP. It is often used to install games that use CASC filesystems, but also supports non-CASC based products.

Product Information

Upon initial release, TACT used regular HTTP to retrieve information for its products. In early 2018, Blizzard started working on a new system that has superseded this. More information about the new system can be found on the Ribbit page.

As of March/April 2019 Ribbit is now the primary system for retrieving version information for most products. HTTP version information has already diverged in some places and should probably only be used as a fallback.

As of 8.3.7 and 9.0.1 the WoW client uses Ribbit as the primary way of retrieving version information as well.

HTTP URLs

Note: These are now just wrappers over Ribbit V2.

URL Description
http://us.patch.battle.net:1119/(product)/cdns a table of CDN domains available with game data per region
http://us.patch.battle.net:1119/(product)/versions current version, buildconfig, cdnconfig, productconfig and optionally keyring per region
http://us.patch.battle.net:1119/(product)/bgdl similar to versions, but tailored for use by the Battle.net App background downloader
http://us.patch.battle.net:1119/(product)/blobs contains InstallBlobMD5 and GameBlobMD5
http://us.patch.battle.net:1119/(product)/blob/game a blob file that regulates game functionality for the Battle.net App
http://us.patch.battle.net:1119/(product)/blob/install a blob file that regulates installer functionality for the game in the Battle.net App

Products

The following products are known to have existed at one point. When this page refers to a product it refers to the TACT Product. Agent UIDs are specific to Agent.

Non-Games
TACT Product Agent UID Description Status
agent agent Battle.net Agent Active
agent_test Probably Agent test
bna battle.net Battle.net App Active
bts Bootstrapper Partial (only versions as of now, cdnpath=tpr/bnt004)
catalogs Catalog Active
clnt Client Deprecated
demo Partial
test Deprecated

Most games have multiple products. Tables for these are collapsed by default to not affect page length. Click Expand to see all known products for a game.

Blizzard Arcade Collection
TACT Product Agent Product Description Status
rtro rtro Blizzard Arcade Collection Retail Active
rtrodev rtro_dev Blizzard Arcade Collection Dev
Diablo Immortal (PC)
TACT Product Agent Product Description Status
anbs anbs Diablo Immortal Retail
anbsdev anbs_dev Diablo Immortal Dev Active (encrypted)
Diablo II: Resurrected
TACT Product Agent Product Description Status
osi Diablo II: Resurrected Retail Active
osib osi_beta Diablo II: Resurrected Beta Active
osia osi_dev Diablo II: Resurrected Alpha Active (encrypted)
osidev osi_dev_2 Diablo II: Resurrected Dev Active (encrypted)
osiv1 osi_vendor_1 Diablo II: Resurrected Vendor 1 Active (encrypted)
osiv2 osi_vendor_2 Diablo II: Resurrected Vendor 2
osiv3 osi_vendor_3 Diablo II: Resurrected Vendor 3 Active (encrypted)
osiv4 osi_vendor_4 Diablo II: Resurrected Vendor 4
osiv5 osi_vendor_5 Diablo II: Resurrected Vendor 5
osiv6 osi_vendor_6 Diablo II: Resurrected Vendor 6
Diablo III
TACT Product Agent Product Description Status
d3 diablo3(_locale) Diablo 3 Retail Active
d3b Diablo 3 Beta (2013) Partial
d3cn d3cn Diablo 3 China Active
d3cnt Diablo 3 China Test (?) Unused (everything empty)
d3t diablo3_ptr(_locale) Diablo 3 Test Active
Diablo IV (aka Fenris)
TACT Product Agent Product Description Status
fenris fenris Diablo IV Retail Active
fenrisb fenris_beta Diablo IV Beta Active
fenrise fenris_event Diablo IV Pre-Launch Disabled (encrypted)
fenrisdev fenris_dev Diablo IV Dev Disabled (encrypted)
fenrisvendor fenris_vendor Diablo IV Vendor Active (encrypted), Season 1
fenrisvendor2 fenris_vendor2 Diablo IV Vendor 2 Active (encrypted)
fenrisvendor3 fenris_vendor3 Diablo IV Vendor 3 Active (encrypted), Season 2
Heroes of the Storm
TACT Product Agent Product Description Status
bnt Heroes of the Storm Alpha Deprecated
hero heroes Heroes of the Storm Retail Active
heroc heroes_tournament Heroes of the Storm Tournament Active
herot heroes_ptr Heroes of the Storm Test Active
storm Heroes of the Storm Deprecated
Hearthstone
hsb hs_beta Hearthstone Retail Active
hsc hs_tournament Hearthstone Chournament Active
hst Hearthstone Test Partial
Overwatch (aka Prometheus)
TACT Product Agent Product Description Status
pro prometheus Overwatch Retail Active
proc prometheus_tournament Overwatch Tournament US Active
proc_cn prometheus_tournament_cn Overwatch Tournament China Active
proc_eu prometheus_tournament_eu Overwatch Tournament Europe Active
proc_kr prometheus_tournament_kr Overwatch Tournament Korea Active
proc2 prometheus_tournament2 Overwatch Professional 2 Active
proc2_cn prometheus_tournament2_cn Overwatch Professional 2 China Active
proc2_eu prometheus_tournament2_eu Overwatch Professional 2 Europe Active
proc2_kr prometheus_tournament2_kr Overwatch Professional 2 Korea Active
proc3 prometheus_tournament3 Overwatch Tournament (Dev)
procr prometheus_tournament_viewer Overwatch League Stage 3
procr2 prometheus_tournament_viewer_2 Overwatch League Stage 2
prodev prometheus_dev Overwatch Dev Active (encrypted)
proe Not on public CDNs
prot prometheus_test Overwatch Test Active
prov prometheus_vendor Overwatch Vendor Active (encrypted)
proms prometheus_viewer Overwatch World Cup Viewer Partial
StarCraft 1
TACT Product Agent Product Description Status
s1 s1 StarCraft 1 Active
s1a Starcraft 1 Alpha Active (encrypted)
s1t s1_ptr StarCraft 1 Test Active
StarCraft II
TACT Product Agent Product Description Status
s2 s2(_locale) StarCraft II Retail Active
s2b StarCraft II Beta Deprecated
s2t s2_ptr(_locale) StarCraft II Test Deprecated
sc2 StarCraft II Deprecated
Warcraft III
TACT Product Agent Product Description Status
w3 w3 Warcraft III Active
w3t w3_ptr Warcraft III Public Test Active
war3 Warcraft III (old) Partial
w3b w3_beta Warcraft III: Reforged Beta
World of Warcraft
TACT Product Agent Product Description Status
wow wow(_locale) World of Warcraft Retail Active
wow_beta wow_beta World of Warcraft Alpha/Beta Active
wow_classic wow_test, later wow_classic World of Warcraft Classic (BCC) Active
wow_classic_beta wow_classic_beta World of Warcraft Classic (BCC) Beta Active
wow_classic_ptr wow_classic_ptr World of Warcraft Classic (BCC) Test Active
wow_classic_era wow_classic_era World of Warcraft Classic (Vanilla) Active
wow_classic_era_beta wow_classic_era_beta World of Warcraft Classic (Vanilla) Beta Active
wow_classic_era_ptr wow_classic_era_ptr World of Warcraft Classic (Vanilla) Test Active
wowdev wow_alpha World of Warcraft Dev Active (encrypted)
wowdemo World of Warcraft (Classic) Demo Active (encrypted)
wowe1 wow_event1 World of Warcraft Event 1 Active
wowe2 wow_event2 World of Warcraft Event 2 Active (partial)
wowe3 wow_event3(_locale) World of Warcraft Event 3 Active (partial)
wowt wow_ptr(_locale) World of Warcraft Test Active
wowv wow_vendor World of Warcraft Vendor Active (encrypted)
wowv2 wow_vendor2 World of Warcraft Vendor 2 (Classic) Active (encrypted)
wowz wow_submission World of Warcraft Submission (previously Vendor) Active

Non-Blizzard

Call of Duty
TACT Product Agent Product Description Status
viper viper Call of Duty: Black Ops 4 Active
viperdev viper_alpha Call of Duty: Black Ops 4 - Alpha Active (encrypted)
viperv1 viper_vendor Call of Duty: Black Ops 4 Vendor
odin odin_placeholder → odin Call of Duty: Modern Warfare
odindev odin_dev Call of Duty: Modern Warfare Dev Active (encrypted)
odinv1 odin_vendor1 Call of Duty: Modern Warfare Vendor 4
odinv2 odin_vendor2 Call of Duty: Modern Warfare Vendor 4
odinv3 odin_vendor3 Call of Duty: Modern Warfare Vendor 4
odinv4 odin_vendor4 Call of Duty: Modern Warfare Vendor 4
odina odin_alpha another cod mw dev channel
odinb odin_beta public (paid) beta
odine odin_event tournament/event
lazr lazarus Call of Duty: MW2CR (Campaign Remastered)
lazrdev lazarus_dev Call of Duty: MW2CR Dev
lazrv1 lazarus_vendor_1 Call of Duty: MW2CR Vendor 1
lazrv2 lazarus_vendor_2 Call of Duty: MW2CR Vendor 2
zeus zeus Call of Duty: Black Ops Cold War Active
fore fore Call of Duty: Vanguard Active
foreb foreb Call of Duty: Vanguard (Public Beta) Active
Crash Bandicoot 4
TACT Product Agent Product Description Status
wlby wlby Crash Bandicoot 4
wlbydev wlby_dev Crash Bandicoot 4 Developing Active (encrypted)
wlbyv[1-6] wlby_vendor_[1-6] Crash Bandicoot 4 for Vendors Active (encrypted)
Destiny 2
TACT Product Agent Product Description Status
dst2 destiny_2 Destiny 2 Active
dst2a destiny2_alpha Destiny 2 Alpha Active (encrypted)
dst2dev destiny2_takehome Destiny 2 "takehome" development Partial (encrypted?)
dst2e1 destiny2_event Destiny 2 Event Active; probably 1..9
dst2igr Destiny 2 Internet Game Room Active
dst2t destiny2_ptr Destiny 2 Public Test Active

Glossary

This page uses many terms. Here's an overview of stuff you need to know to be able to understand most of the explanations below.

Term Description
Filename The file's real filename. Note that one file in TACT can have many names referring to it - essentially, one encoding key can map to many different name hashes.
Name Hash/Lookup The filename after being hashed with the Jenkins Hash.
Content Hash/CKey The MD5 of the entire file in its uncompressed state; the purest representation of the data.
Encoding Hash/EKey MD5 hash of the potentially encoded file. For unencoded files, the content hash/CKey. For chunkless BLTE files lacking a chunk table, this hash covers the entire encoded file. For chunked BLTE files, this hash covers only the BLTE headers including chunk table, as the chunk table contains hashes of the content of each chunk. A given file can be encoded in many ways, and a single content hash may potentially have multiple encoding keys.
CDN Key The key used to lookup a file on the CDN. Same as encoding hash/EKey.

File types

Files are stored on CDNs in the following format: http://(cdnsHost)/(cdnsPath)/(pathType)/(FirstTwoHexOfHash)/(SecondTwoHexOfHash)/(FullHash)

Current list of CDNs for a specific product can be found in the http://us.patch.battle.net:1119/(product)/cdns file or via Ribbit in the v1/products/(product/)cdns endpoint. Blizzard regularly shuffles around CDNs and sometimes even adds/removes CDNs, so be sure to parse cdns to stay up to date. The CDN path found in the cdns file very rarely changes, the only known occurence is Overwatch changing from tpr/pro to tpr/ovw.

Known path types are:

  • config - contains the three types of config files: Build configs, CDN configs, and Patch configs
  • data - contains archives, indexes, and unarchived standalone files (typically binaries, mp3s, and movies and files mentioned in buildconfig like root, install and download)
  • patch - contains patch manifests, files, archives, indexes

Blizzard regularly cleans old builds from the CDN so any example files mentioned in this article might be unavailable at the time of reading (feel free to update them to something more recent if this is the case).

Product configs referred to in the versions file are generally saved in tpr/configs for all games, but to be sure this path should be found by parsing the cdns file for the product in question.

It is intended behavior for some files, even recent ones, to not be available on all CDNs, you should try a different CDN from the list of CDNs in those cases. For example an archive might not be available on us.cdn.blizzard.com but instead be available on level3.blizzard.com

Some CDNs also implement rate limiting, they will return HTTP code 429 if you are exceeding requests. Exact request limits are unknown and seem to vary per region/time.

Config Files

Build Config

Example file: http://blzddist1-a.akamaihd.net/tpr/wow/config/22/38/2238ab9c57b672457a2fa6fe2107b388

Some of the files listed in this file are explained later on in this article.

What follows is a table for all known variables that have been seen in a buildconfig.

Value name Description
root Content hash of the decoded root file, look this up in encoding to get the encoded hash.
install First key is the content hash of the decoded install file. Second key, if present, is the encoded hash; if absent, look up the content hash in encoding to get the encoded hash/CDN key.
install-size Install size(s) corresponding to the install hash(es). Absent in older WoW builds.
download First key is the content hash of the decoded download file. Second key, if present, is the encoded hash; if absent, look up the content hash in encoding to get the encoded hash.
download-size Download size(s) corresponding to the download hash(es). Absent in older WoW builds.
size CKey and EKey of the download size file, respectively. Introduced in WoW build 27547.
size-size Download size sizes corresponding to the download size keys.
build-partial-priority Content hash:block size pairs for priority non-archived files, ordered by priority. The block size appears to be the amount to request in each partial HTTP request. These files are BLTE-encoded and usually but not always split into blocks of fixed uncompressed size where the partial priority block size is a power-of-two multiple of the file's uncompressed block size. First seen in build 24116. Optional.
partial-priority Content hash of a partial download file containing priority files to download first; note that the sizes in the download file entries are download block sizes, not file sizes. First seen in build 22231, replaced by build-partial-priority in build 24116. Optional.
partial-priority-size Unknown: always 0 if present. Present if partial-priority is present.
encoding First key is the content hash of the decoded encoding file. Second key is the encoded hash. If either none or 1 is found, the game (at least Wow) switches to plain-data (?) . Seen in build 20173
encoding-size Encoding sizes corresponding to the encoding hashes.
patch The manifest of patchable data files. Does not include the metadata files e.g. encoding, whose patches are specified in the patch config file. Optional.
patch-size Size of the patch manifest, if any. Optional.
patch-config Content hash of non-encoded patch config (see Patch Config)
build-attributes Optional. Seen in agent.
build-branch Optional. Presumably the SCM branch built.
build-comments Optional.
build-creator Optional. Presumably the user who submitted the build.
build-fixed-hash Optional. Seen in S2.
build-replay-hash Optional. Seen in S2.
build-name Optional? Name of the build
build-playbuild-installer Optional? Type of installer for the Battle.net app to use
build-product Optional? Product name
build-t1-manifest-version Optional.
build-uid Optional? Program code (see Products)
vfs-root VFS root
vfs-root-size VFS root size
vfs-i VFS manifest, where i is index.
vfs-i-size VFS manifest size, where i is index.

CDN Config

Example file: http://blzddist1-a.akamaihd.net/tpr/wow/config/42/33/423364147752a596911aa1de2ff1f6a4

Value name Description
archives CDN keys of all archives (and by appending .index to the hash their indexes)
archive-group CDN key of the the combined index file (see Archive-Group Index)
file-index CDN key of .index file for unarchived files. These files have 0-byte archive offset fields. Seen in Warcraft III.
file-index-size Size of unarchived file .index.
patch-archives CDN keys of patch archives (needs research)
patch-archive-group CDN key of probably the combined patch index file (needs research)
builds List of build configs this config supports (optional)
patch-file-index CDN key of .index file for unarchived patches. These files have 0-byte archive offset fields. Seen in Warcraft III.
patch-file-index-size Size of unarchived patch .index.

Patch Config

Example file: http://blzddist1-a.akamaihd.net/tpr/wow/config/20/f0/20f0593dd1b9fdaeaa7a808f83d48f1d This configuration file was added after all of the others. It first appeared in CASC v1 for Heroes of the Storm in August 2014. It then appeared in WoW for CASC v2 in build 19027 (October 10th, 2014). The purpose of this file is to reduce redundant downloads. It achieves this by directing the system to download patch files to apply and update previously downloaded material. The structure and purpose of all of the fields of this file requires further research.

Value name Description
patch-entry Repeats 4 times with patch entries for install, download, encoding and size file. Builds using the partial-priority (not build-partial-priority) mechanism may additionally have a partial-priority entry.
patch Patch manifest file
patch-size Size of patch file (optional?)

patch-entry

The format of these strings is:

patch-entry = <type> <content hash> <content size> <BLTE-encoding key> <BLTE-encoded size> <encoding string>

followed by sets of

<old BLTE-encoding key> <old content size> <patch hash> <patch size>

sorted by oldest build to newest. This serves to pair old files to the patch needed to bring them up to date, as well as providing the information required to detect if the file is already up to date.

These patches are stored externally to the #Patch file (which solely handles game asset patching) and are stored directly on Blizzard's CDN. These are raw ZBSDIFF1 blobs applied to the BLTE decoded contents of each file.

Example patch: https://blzddist1-a.akamaihd.net/tpr/wow/patch/0b/68/0b6829cdd07255fa57b6243b0eba2ee3

Data Files

Example index: http://blzddist1-a.akamaihd.net/tpr/wow/data/00/52/0052ea9a56fd7b3b6fe7d1d906e6cdef.index

Example archive: http://blzddist1-a.akamaihd.net/tpr/wow/data/00/52/0052ea9a56fd7b3b6fe7d1d906e6cdef

Patch Files

#Patch

Encoding table

The encoding file maps content hashes C-Keys to encoded-file hashes E-Keys. In addition, there is information on how the files are BLTE-encoded by E-Specs.

Blocks in this file are, in this order

  • header
  • encoding specification data ESpec
  • content key → encoding key table CEKeyPageTable
  • encoding key → encoding spec table EKeySpecPageTable
  • encoding specification data for the encoding file itself

An incomplete/outdated 010 Editor template can be found at this gist which can be used to understand page handling.

Header

Header is a constant 0x16 bytes giving size information for the other blocks, mostly.

struct {
/*0x00*/  char signature[2];                            // "EN"
          enum {
            encoding_version_1 = 1,                     // ≥ WoD (6.0.1.18125)
          };
/*0x02*/  uint8_BE_t version;
/*0x03*/  uint8_BE_t hash_size_ckey;
/*0x04*/  uint8_BE_t hash_size_ekey;
/*0x05*/  uint16_BE_t CEKeyPageTable_page_size_kb;      // in kilo bytes. e.g. 4 in here → 4096 byte pages (default)
/*0x07*/  uint16_BE_t EKeySpecPageTable_page_size_kb;   // ^
/*0x09*/  uint32_BE_t CEKeyPageTable_page_count;
/*0x0D*/  uint32_BE_t EKeySpecPageTable_page_count;
/*0x11*/  uint8_BE_t _unknown_x11;                      // 0 -- sometimes assumed to be part of ESpec_block_size, but actually asserted to be zero by agent
/*0x12*/  uint32_BE_t ESpec_block_size;
/*0x16*/
} header;

ESpec

Encoding specification strings are just a blob of zero terminated strings referenced by EKeySpecPageTable with the accumulated size of header.ESpec_block_size (including zero terminators).

The definition of the format for these strings is described on the BLTE page.

Page Tables

The format of the two page tables is the same:

  • an index for fast key → page access, followed by
  • the actual pages with specific content

In both cases, the entries in the lists have the same count, and semi-dynamic size, depending on header.hash_size_ckey and header.hash_size_ekey. Note that the page checksum size is fixed to MD5's 16 bytes. The position comments below assume the standard key size of 16 bytes.

Note: As of 8.3, the entry count of both tables is no longer exactly the same.

struct page_index_t {
/*0x00*/  char first_Xkey[header.hash_size_Xkey];       // where X is c for CEKeyPageTable and e for EKeySpecPageTable
/*0x10*/  char page_md5[0x10];
/*0x20 usually*/
};

The pages themselves are filled as much possible with actual entry structs. They are padded to the end for alignment of pages. Pages don't actually have to be full.

CEKeyPageTable

This table maps one ckey to one or more ekeys. This means that there can be multiple representations, e.g. an encrypted and an unencrypted version. This isn't usually the case and on reading any of them can be picked, since they represent the same file content. It can be used to pick an already downloaded archive, or unencrypted one, or handle deleted archives.

struct ckey_ekey_entry_t {
/*0x00*/  uint8_BE_t keyCount;
/*0x01*/  uint40_BE_t file_size;                        // of the non-encoded version of the file
/*0x06*/  char ckey[header.hash_size_ckey];             // this ckey is represented by…
/*0x16*/  char ekey[header.hash_size_ekey][keyCount];   // …these ekeys
/*0x26 usually*/
} page_entries[];
EKeySpecPageTable

This table maps one ekey to the corresponding espec describing how the encoding happened.

struct ekey_espec_entry_t {
/*0x00*/  char ekey[header.hash_size_ekey];
/*0x10*/  uint32_BE_t espec_index;                      // not an offset but an index, assuming zero-terminated espec strings
/*0x14*/  uint40_BE_t file_size;                        // of the encoded version of the file
/*0x19 usually*/
} page_entries[];

Install manifest

File signature: "IN"

The install file lists files installed on disk. Since the install file is shared by architectures and OSs, there are also tags to select a subset of files. When using multiple tags, a binary combination of the bitfields of files to be installed can be created.

Header Structure

The file begins with a 10 byte header describing the number of tags and files listed. Structure names were invented by the author of this page.

Offset (Hex) Type Name Description
0x00 char[2] FileSignature "IN"
0x02 uint8_t Version? 1
0x03 uint8_t hash_size size of hashes used for files (usually md5 -> 16)
0x04 uint16_BE_t num_tags number of tags in header of file
0x06 uint32_BE_t num_entries The number of entries in the body of the file

Tags Structure

After the header, an array with information about available tags follows. Each tag has a bitfield listing the files installed when the given tag is chosen.

Type Name Description
char[] name
uint16_BE_t type A number shared amongst specific flags. Actual meaning is specific to products.
char[divru (header.entries, CHAR_BIT)] files A bitfield that lists which files are installed when the specified tag is installed.

Files Structure

The remainder of the file is populated by a list of files with their content hash, each a variable size (due to the strings). Structure names were invented by the author of this page.

Type Name Description
char[] FileName The name of the file.
char[header.hash_size] hash The hash of the uncompressed file. Usually MD5.
uint32_BE_t Size The size of the file.

C-like structure

char I; char N;
uint8_BE_t _unk3;
uint8_BE_t hash_size;
uint16_BE_t num_tags;
uint32_BE_t num_files;

struct {
  string name;
  uint16_BE_t type;
  char flags[divru (num_files, CHAR_BIT)];
} tags[num_tags];

struct {
  string name;
  char hash[hash_size];
  uint32_BE_t size;
} files[num_files];

Download manifest

The download file lists all files stored in the data archives. The client uses this to download files ahead of time, without it, the client will download on demand which can lead to issues. The download priority is set inside the entries with 0 being the highest and 2 the lowest however, if the game is running, missing assets in the player's vicinity take precedence.

Just like the install file, the download file is shared across all architectures and locales so utilizes the same bitfield-tag system to assess what subset of files are needed.

NOTE: partial-priority download files do not contain the actual file sizes but redefine the FileSize field to ChunkSize.

This file has this structure:

  • Header
  • Entries[Header.EntryCount]
  • Tags[Header.TagsCount]

Download Header

Type Name Description
char[2] Signature The signature for this file (always "DL")
char Version 1 < 7.3.0, 2
char ChecksumSize Always 0x10
char unk ??? Always 1
int [BE] EntryCount The amount of file entries in this file
short [BE] TagCount The amount of tag entries in this file

Download Entry

Type Name Version Added Description
char Unk 2 ??? Appears to be a boolean. Currently only set to 1 on 4 specific records
char[16] Hash 1 This hash is found in every node of the encoding file. (Reverse lookup) MD5
uint40_t [BE] FileSize 1 The compressed size of the file
char DownloadPriority 1 0 = highest, 2 = lowest; -1 for the Install since Battle
char[4] Unk 1 ???

Download Tag

Type Name Description
string Name A C-String indicating this tag's Name.
short [BE] Type Hash type
char[N] Bits an array of size N = Header.EntryCount / 8 + (Header.EntryCount % 8 > 0 ? 1 : 0); that is basically a massive bit mess. Use Schroeppel's 8 bits reverse function on it to have bits.

code-ish

struct {
/*0x00*/  char signature[2];                                // "DL"
          enum {
            download_version_1 = 1,
            download_version_2 = 2,                         // ≥ Legion (7.3.0.???)
            download_version_3 = 3,
          };
/*0x02*/  uint8_BE_t version;
/*0x03*/  uint8_BE_t hash_size_ekey;
/*0x04*/  uint8_BE_t has_checksum_in_entry;
/*0x05*/  uint32_BE_t entry_count;
/*0x09*/  uint16_BE_t tag_count;
/*0x0b*/

#if version ≥ download_version_2
    /*0x0b*/  uint8_BE_t number_of_flag_bytes;             // defaults to 0, up to 4
    /*0x0c*/

   #if version >= download_version_3
       /*0x0c*/  uint8_BE_t base_priority;                 // defaults to 0
       /*0x0d*/  char _unknown_0d[3];                       // As of 1.15.6.2-test4, this is explicitly 0. It is ignored on reading.
       /*0x10*/
   #endif
#endif
} header;

struct {
/*0x00*/  char ekey[header.hash_size_ekey];
/*0x10*/  uint40_BE_t file_size;
/*0x15*/  uint8_BE_t priority;                              // header.base_priority is subtracted on parse
/*0x16*/

#if header.has_checksum_in_entry
/*0x16*/  uint32_BE_t checksum;
/*0x1a*/
#endif

#if header.version ≥ download_version_2
          enum {
            download_flag_plugin = 1,                       // "plugin"
            download_flag_plugin_data = 2,                  // "plugin-data"
          }; 
          uint8_BE_t flags[header.number_of_flag_bytes];    // defaults to 0 if no flag bytes present
#endif

} entries[header.entry_count];

struct {
         char const name[];                                 // this string is zero terminated, no fixed size
                                                            // thus for readability we start offset at 0 here.
/*0+00*/ uint16_BE_t type;                                  // game specific. usually architecture, category, locale, os, region or alike.
/*0+02*/ char mask[divru (header.entry_count._, CHAR_BIT)]; // if bit is set, entries[bit] is part of this tag
/*0x??*/
} tag[header.tag_count];

Download Size

Build 27547 introduced the Download Size file of unknown purpose. The file is a stripped-down Download file with partial EKeys and files sorted by encoded files size. The purpose of this file is not clear.

struct Header
{
	char signature[2];     // "DS"
	uint8_t version;       // only known: 1 (as of agent 6700)
	uint8_t ekeySize;      // 9, up to 23 possible, but agent 6700 hardcodes 9, so, uh.
	uint32_BE_t numFiles;
	uint16_BE_t numTags;
	uint40_BE_t totalSize; // Size of all files combined
};

struct TagEntry
{
	char name[]; // Null-terminated
	uint16_BE_t type;
	char fileMask[(hdr.numFiles + 7) / 8];
}

struct FileEntry
{
	char ekey[hdr.ekeySize];
	uint32_BE_t esize;
};

SizeHeader hdr;
TagEntry tags[hdr.numTags];
FileEntries files[hdr.numFiles]; // Sorted descending by esize

Patch

Type Name Description
char[2] Signature The signature for this file (always "PA")
char version 1 or 2
char file_key_size <= 0x10
char size_b <= 0x10
char patch_key_size <= 0x10
char block_size_bits 2 <= block_size_bits <= 24. block size == 2^block_size_bits.
short block_count
char flags
char[16] EncodingCkey ckey for encoding file
char[16] EncodingEkey ekey for encoding file
int DecodedSize Decoded encoding file size in bytes
int EncodedSize Encoded encoding file size in bytes
char EspecLength Length of the following string
char[EspecLength] EncodingEspec espec of encoding file
char[] ??? byte array containing blocks entries, blocks, and optional patch tail

header+entries needs to be less than 0x10000 bytes (at least in wow-18179). md5sum is only checked for header+entries, file might be larger thus.

struct PatchManifest_Header
{
  uint16_BE_t magic; // 'PA'
  uint8_t version; // 1 or 2
  uint8_t file_key_size; // <= 0x10
  uint8_t size_b; // <= 0x10
  uint8_t patch_key_size; // <= 0x10
  uint8_t block_size_bits; // 12 <= block_size_bits <= 24. max block size == 2^block_size_bits
  uint16_BE_t block_count; // (file_key_size + 20) * entry_count + sizeof (PatchManifest_Header) < 0x10000
  uint8_t flags; // &1: plain-data mode manifest, &2: extended header 

#if encoding_information_apparently_added_after_18179
  uint8_t encoding_ckey[16];
  uint8_t encoding_ekey[16]; // probably since PA2
  uint32_BE_t decoded_size;
  uint32_BE_t encoded_size;
  uint8_t encoding_espec_length;
  char encoding_format[encoding_espec_length];
#endif
 } header;

struct PatchManifest_Block
{
  uint8_t last_file_ckey[header.file_key_size];
  uint8_t md5_of_block[16];
  uint32_BE_t block_offset; // in this file
} blocks[header.block_count]; // sorted ascending by key

// at positions given in PatchManifest_Block
struct block
{
  struct
  {
    uint8_t num_patches; // <= 0x10.
    uint8_t target_file_ckey[header.file_key_size];
    uint40_BE_t decoded_size;
    struct
    {
      uint8_t source_file_ekey[header.file_key_size];
      uint40_BE_t decoded_size;
      uint8_t patch_ekey[header.patch_key_size];
      uint32_BE_t patch_size;
      uint8_t patch_ordinal; // order of patch application, 1-based indexing
    } patches[num_patches];
  } files[]; // count unspecified: read until the next file num_patches would be 0 
               // OR block would exceed max block size
};

// some files have a block of data after the last block of the patch manifest (which may be shorter than max block size). this block appears to be a patch of encoding, but the format is not understood.

Product Specific

In this section, the game/usage specific parts of CASC are described. While CASC is a generic format, a lot of stuff is hardcoded, like hash sizes. Other parts are actually left up to the implementation, like root files or download tags.

World of Warcraft

Root

As of 8.2 there was changes to this format. The client currently (last checked: 9.0.1) parses backwards compatible. There are three changes:

  • The file is now chunked with the old content being in a MFST (ManiFeST) chunk.
  • blocks can be without name hashes and thus ID-only.
  • the name and content hashes are no longer interleaved but two separate arrays
    • In memory, the manifest is kept in a FDID → entry map, so when reverse engineering the parser one may think that fnv1a is relevant, but that’s merely the hashmap insert.
    • To parse that variable format the client uses two pointers to chash/name and a stride (of 16 for key only and 24 for with name) rather than one for records. It also determines the name hash/key hash base pointers depending on format, allowing them to have one parse able to read all format versions.

In 10.1.7.50893 3 ints were added to the header; header size, version and something that's probably padding.

enum locale_flags : uint32_t {
  enUS =     0x2,
  koKR =     0x4,
  frFR =    0x10,
  deDE =    0x20,
  zhCN =    0x40,
  esES =    0x80,
  zhTW =   0x100,
  enGB =   0x200,
  enCN =   0x400,
  enTW =   0x800,
  esMX =  0x1000,
  ruRU =  0x2000,
  ptBR =  0x4000,
  itIT =  0x8000,
  ptPT = 0x10000,
};
enum content_flags : uint32_t {
  HighResTexture     =        0x1,            // is high-res texture (cataclysm 4.4.0 beta)
  Install            =        0x4,            // file is in install manifest
  LoadOnWindows       =        0x8,            // macOS clients do not read block if flags & 0x108 != 0
  LoadOnMacOS         =       0x10,            // windows clients do not read block if flags & 0x110 != 0
  x86_32             =       0x20,            // install manifest file only - load on 32 bit systems
  x86_64             =       0x40,            // install manifest file only - load on 64 bit systems
  LowViolence         =       0x80,
  DoNotLoad          =      0x100,            // neither macOS nor windows clients read blocks with this flag set. LoadOnMysteryPlatform?
  UpdatePlugin       =      0x800,            // only ever set for UpdatePlugin.dll and UpdatePlugin.dylib
  ARM64              =     0x8000,            // install manifest file only - load on ARM64 systems
  Encrypted           =  0x8000000,
  NoNameHash          = 0x10000000,
  UncommonResolution = 0x20000000,            // denotes non-1280px wide cinematics
  Bundle              = 0x40000000,
  NoCompression       = 0x80000000,
};
struct MD5Hash {
  char bytes[0x10];
};

if (format_version == 8.2) {
  uint32_t magic;                        // 'MFST'
#if ≥ DF (10.1.7.50893) 
  uint32_t header_size;
  uint32_t version;
#endif
  uint32_t total_file_count;             // sum of all blocks' num_records
  uint32_t named_file_count;             // sum of those blocks' num_records that have NoNameHash not set
#if ≥ DF (10.1.7.50893) 
  uint32_t likely_padding;
#endif
}

// bool allow_non_named_files = total_file_count ≠ named_file_count || format_version < 8.2;
// bool use_old_record_format = magic ≠ 'MFST';

struct CASBlock {
  uint32_t num_records;
  content_flags flags;
  locale_flags locale;
  int32_t fileDataIDDeltas[num_records]; // each block starts with 0, +1 is implicit per entry, so consecutive ids will have delta=0

  if (use_old_record_format) {
    struct CASRecord {
      MD5Hash content_key;               // MD5 hash of the file's raw data
      uint64_t name_hash;                // Jenkins96 (lookup3) hash of the file's path 
    } records[num_records];
  }
  else {
    MD5Hash content_key[num_records];    // MD5 hash of the file's raw data
    if (!(allow_non_named_files && flags & NoNameHash)) {
      uint64_t name_hash[num_records];   // Jenkins96 (lookup3) hash of the file's path 
    }
  }

  int32 file_data_id (size_t index) const
  {
    return index == 0
      ? fileDataIDDeltas[index]
      : file_data_id (index - 1) + 1 + fileDataIDDeltas[index];
  }
} blocks[];                              // count: fill up until end of file

hashpath

This function is used in the root file by WoW and other older MPQ-based games to calculate filename lookups.

hashpath (string path) → uint32_t
{
  string normalized = toupper (path).replace (from: '/', to: '\\')
  uint32_t pc = 0, pb = 0;
  hashlittle2 (normalized, strlen (normalized), &pc, &pb);
  return pc;
}

Tags

Values depend on versions, semantic categories are cross version.

  • Platform: The deployment target, i.e. Windows or OSX
  • Architecture: Sub-division of the deployment target, i.e. x86_32 or x86_64
  • Locale: The same as in Localization: Files specific to a single localisation of the game.
  • Region: Equivalent to the patch server regions, i.e. us, eu, kr, tw, cn.
  • Category: A replacement for the MPQ system to tag low priority downloads: speech, text
  • Alternate: A special category for censored content.

Version specific values

This section only applies to versions WoD (6.0.1.18125) … WoD (6.0.1.18761).

Architecture = 1, Locale = 2, Platform = 3

This section only applies to versions WoD (6.0.1.18764) … WoD (6.2.2.20426).

Architecture = 1, Category = 2, Locale = 3, Platform = 4, Region = 5

This section only applies to versions WoD (6.2.2.20438) … Legion.

Platform = 1, Architecture = 2, Locale = 3, Region = 4, Category = 5

This section only applies to versions ≥ Battle (8.0.1.26604). Actually alternate was just missing above but schlumpf doesn't want to verify since when.
enum {
  platform = 1,
  architecture = 2,
  locale = 3,
  region = 4,
  category = 5,
  alternate = 0x4000,
};

Shared storage

As of 8.1 both the WoW client and Battle.net app have started supporting sharing storage containers ([CASC]). There is one Data folder in the root folder with storage for multiple products. Products and product-specific data (Cache, Erros, Interface/Addons, Logs, WTF, etc) are stored in subfolders.

Root folder has a .build.info with cached build information for each product sharing the storage, each product subfolder also has a .flavor.info file with the TACT product name (e.g. wowt or wow_classic_beta, should be the same as in .build.info). Both of these files are written by the Battle.net agent and not by the clients themselves. However, if these two files are available the clients will use cached information in the .build.info file instead of getting it from remote.

Armadillo

If $program/blob/game or productconfig contains decryption_key_name, data on CDN is encrypted. The key is retrieved from the keychain or manually entered (a base32 encoded version that contains key and checksum is used). It is stored in $key_name.ak. The key file contains the key followed by a the first four bytes of the md5 hash of the key for verification purposes.

struct {
/*0x00*/  unsigned char key[0x10];
/*0x10*/  unsigned char md5_of_key[0x4];  // first four bytes only
/*0x14*/
} ak_file;

Encryption uses Salsa20. The IV is the last 8 bytes (16 characters) of the cdn hash.

Known keys

decryption_key_name     used_by              key                                                 checksum      base32

# Starcraft 1
sc1Dev                  s1a                  F6 79 DC 38 E0 C3 65 FB 48 2E 48 A7 48 90 9D 29     19 F5 BB 88   6Z45YOHAYNS7WSBOJCTUREE5FEM7LO4I

# Overwatch
pro                     pro (pre-launch)             
proc                    proc3
prolivedev              prodev  
provendor               prov
prolivedev2             prodev

# World of Warcraft
wowdemo                 wowdemo
wowdevalpha             wowdev, wowdev2
wowdev (df press event/end of 2022)          3F 10 3E 3B A7 CA FE 90 26 2E 5C A4 3C E7 BC E5     DB 7F AD 8A   H4ID4O5HZL7JAJROLSSDZZ544XNX7LMK
wowvendor               wowv
wowdark                 wowlivetest2

# Destiny 2
destiny2_openbeta       dst2
destiny_dev             dst2a (pre-alpha)
destiny_event           dst2e1
destiny_live            dst2a
dst2livedev             dst2dev

# Call of Duty: Black Ops 4
viperlivedev            viperdev

# Odin
odinlivedev             odindev

# Orbis
orbislivedev            orbisdev
# Diablo 4
fenrisdev               fenris_dev
fenrise                 fenris_event

CDN File Organization

Data for every CASC-based game exists on the CDN in one of two places at any given time. To reduce file-system and download overhead many files are packed into archives and indexed by archive indices, both of a different format than the CASC installations found on client systems; other "unarchived", "standalone", or "loose" files are stored as separate files on the CDN that must be downloaded independently. There are at least three different reasons why a particular file is found in one or the other:

  1. Small files typically incur larger filesystem overhead and benefit most from being packed into archives. A rough rule of thumb appears to be that files smaller than 2 MB or so are put in archives. Presumably larger files are not archived because they make it more difficult to minimize the number of archive files, which are limited in size (a 2 MB file limit would limit unused space in an archive to under 0.8%, given enough data to form a full archive to begin with).
  2. Key files such as encoding, partial-priority, TVFS (Warcraft 3 root), and the download, install, and patch manifests, as well as their respective patches, are typically stored loose for quick access and are rarely ever found in archives.
  3. As games evolve old files become obsolete and are removed from the CDN. However, the archive system means that archives can only be removed when every file in the archive is no longer needed, potentially wasting large amounts of space on the CDN - the exact opposite of the purpose of bundling files into archives to begin with. Thus as the amount of unused data in an archive grows over time, files still in use may be converted to loose files to allow the archive to be purged from the CDN, even when the files are unusually small to be found independently.

Previously, no official indices/manifests of loose files existed, and they could only be found by subtracting archived files from file lists in encoding or manifest files. Beginning with Warcraft 3 and subsequently being deployed more widely on 7/24/2018, new fields in the CDN config file link to index files containing "all" loose data or patch files on the current CDNs.

Archives

Archives are extensionless 256 MB files that are usually only stored on the Blizzard CDNs. Their naming follows the standard URL hash format using the '/data/' path type.

The structure of the archives is presumably just file fragment after file fragment. You will never need to parse it because you can just look up offset + size of your file fragment in the index files and then take the piece directly out of the archive.

The fragments are all BLTE encoded.

The filename is NOT the hash of the archive content but the hash of the index's footer.

Archive Indexes (.index)

These '.index' files reveal to the user where the compressed game files are located within the archives. All indexes (except the Archive-Group index, see below) are named after their archive (only difference is these have an extension). '.index' files are stored on the CDN using the standard hash naming scheme (remember they have an extension though). They are also located in the directory 'INSTALL_DIR/Data/indices/' for a WoW install. For local indexes (.idx), see the CASC page.

Normal Index Entry Structure

  • The file is divided into 4kb chunks populated by these standard index entries of 0x18 (hex) bytes. Each chunk is zero-padded to a full 4kb, though there may be more than 0x18 bytes of padding at the end of a chunk -- be sure the check for all-null EKey fields. The last chunk is a table-of-contents, listing the LAST EKey in each chunk and checksum of each block. All checksums are the first checksumSize bytes of the MD5 of the respective data. Structure names were invented by the author of this page.
struct index_entry {
  char EKey[footer.keySizeInBytes];
  uint_BE_t(footer.sizeBytes) blte_encoded_size;
  uint_BE_t(footer.offsetBytes) offset_to_blte_encoded_data_in_archive;
};

struct index_block {
  static constexpr const block_size = footer.blockSizeKb << 10;
  index_entry entries[block_size / sizeof (index_entry)];
  char padding[block_size - sizeof (entries)];
} blocks[];

struct {
  struct {
    char last_EKey[footer.keySizeInBytes]; // last EKey of a block
  } entries[num_blocks];

  struct {
    char lower_part_of_md5_of_block[footer.checksumSize]; 
  } blocks_hash[num_blocks];
} toc;

struct { // note: size comments assume 0x10 checksum size
/*0x00*/  char toc_hash[checksumSize]; // client tries to read with 0x10, then backs off in size when smaller
/*0x10*/  char version?;        // always 1
/*0x11*/  char _11;             // 0
/*0x12*/  char _12;             // 0
/*0x13*/  char blockSizeKb?;    // Normally 4. Left-shifted by 10. Believed to be block size in KB.
/*0x14*/  char offsetBytes;     // Normally 4 for archive indices, 5 for patch group indices, 6 for group indices, and 0 for loose file indices
/*0x15*/  char sizeBytes;       // Normally 4
/*0x16*/  char keySizeInBytes;  // Normally 16
/*0x17*/  char checksumSize;    // Normally 8, <= 0x10
/*0x18*/  uint32_t numElements; // BigEndian in _old_ versions (e.g. 18179)
/*0x1C*/  char footerChecksum[checksumSize];
/*0x??*/
} footer;
  • footerChecksum is calculated over the footer beginning with version when footerChecksum is zeroed
  • The archive/index name is the MD5 of the footer beginning with toc_hash

archive-group

archive-group is actually a very special .index file that is typically much larger than other indices. There is no archive attached, it is only the index. Also, the index is not found on the CDN! It is assembled by combining all index files given in the CDN config, on client side. The hash is just for verification purposes. Client side a file is assembled using the normal index file format, for caching.

It has a single difference in format to normal indices: While other indices have their offsetBytes long offset field point into the archive, for archive groups, the field also has an archiveIndex:

struct {
  uint16_BE_t archiveIndex; // Index of the archive in the CDN config's archive list
  uint32_BE_t offsetBytes;  // The offset within the specified archive
};

It does not only having the offset, but also an index into a file. Semantically that's still an offset, so no further field for size is used.

It is suggested you do not just parse indices by .index filename locally but take the config files into account. An easy heuristic is that if offsetBytes is not 4, it is a special index, either loose files or a group.

patch-archives

Patch archives are the /patch/ equivalent to the regular (data) archives on the CDN. Like archives, these are binary blobs of fragments indexed by an accompanying .index file with the same name. Again, the index is a hash, size, offset tuple, but the hash is the content hash rather than an encoding hash.

Most files in patch archives are ZBSDIFF1 blobs, though in principle any file that might be in the /patch/ namespace may be found in patch archives and must be handled accordingly.

patch-archive-group

See archive-group. There is no known difference other than the combined data being patch-archives.

TACT keys

TACT keys are used to encrypt data in files. In most cases, TACT keys are streamed to clients when content (such as cinematics in WoW or skins in Overwatch) is unlocked.

Due to long length/very frequent updates, the list has been moved to TACT/Keys.