ADB: Difference between revisions
m (→DBCache.bin) |
(→Version 5 and Version 6: Changed 'sha256' to 'verification_hash'.) |
||
Line 240: | Line 240: | ||
{{Template:SectionBox/VersionRange|min_expansionlevel=7|min_build=7.2.0.23436}} | {{Template:SectionBox/VersionRange|min_expansionlevel=7|min_build=7.2.0.23436}} | ||
Version 5 added a new field directly after 'build_id' in 'dbcache_file_header'. It is a | Version 5 added a new field directly after 'build_id' in 'dbcache_file_header'. It is a verification hash, calculated using the current file plus a specific random number embedded in the current WoW executable. This number changes for every build. This was likely added in order to prevent players from being able to load arbitrary gamedata easily using custom DBCache.bin files. (Note that this is still entirely possible, just much more tedious.) | ||
There were no structure changes in Version 6. | There were no structure changes in Version 6. | ||
Line 251: | Line 251: | ||
uint32_t version; | uint32_t version; | ||
uint32_t build_id; | uint32_t build_id; | ||
uint8_t | uint8_t verification_hash[32]; | ||
}; | }; | ||
Revision as of 03:40, 29 December 2019
The structures described here are used in ADB files, which are a cache of dynamically streamed database entries, typically used for hotfixes and content that Blizzard wants to hide from dataminers until the last minute.
For more information, please check out the page on DB2 files. This article assumes you already have a basic understanding of how they function.
Table content structures
This page describes the structure of ADB files. For a list of existing DB2 files (and therefore ADB files) and their contents, see the categories DBC, Vanilla, Burning Crusade, Wrath of the Lich King, Cataclysm, Mists of Pandaria and Warlords of Draenor. If you add documentation for a file, please add the correct categories (also the build number) as well.
WCH3
This file format was introduced in Cataclysm. It was phased out in favor of WCH4 in Legion (Patch 7.0.1 build 20740).
Structure
See DB2#WDB2 for how to adapt this structure for its .DB2 counterpart.
struct adb_header { uint32_t magic; // 'WCH3' uint32_t record_count; uint32_t field_count; // array fields count only count as '1' field uint32_t record_size; uint32_t string_table_size; // string block almost always contains at least one zero-byte uint32_t table_hash; uint32_t build; uint32_t timestamp_last_written; // set to time(0); when writing in WowClientDB2_Base::Save() uint32_t min_id; uint32_t max_id; uint32_t locale; // as seen in TextWowEnum uint32_t copy_table_size; // always zero for ADBs }; template<typename record_type> struct adb_file { adb_header header; // static_assert (header.record_size == sizeof (record_type)); if (header.max_id != 0) { int indices[header.max_id - header.min_id + 1]; // maps from id to row index in records[] below short string_lengths[header.max_id - header.min_id + 1]; // sum of lengths of all strings in row } record_type records[header.record_count];* char string_table[header.string_table_size]; };
- Note: Each record is aligned to the length of the longest field type in the record. If you have a record with three fields: Int32, Int8, and Int8 - then there will be 2 bytes of padding at the end of every record, including the last one. 'record_size' should account for this (it would be '8' in the example given in the previous sentence).
WCH4
This file format was introduced in Legion (Patch 7.0.1 build 20740). It was phased out in favor of WCH5 just a few builds later, also in Legion (Patch 7.0.1 build 20810). As such, support for WCH4 is likely important for almost nobody.
The major changes are the addition of an optional offset map which forces records to have inline strings and an optional block after the string block that contains non-inline IDs.
It is worth noting that min_id and max_id fields now see different use, despite the header remaining the same. min_id and max_id are always non-zero, even when the offset_map structure is present.
Note that, in most cases, the non-inline IDs structure still exists, even if it is made completely redundant by the existence of the offset map. After all, why do work once when you could do it twice?
Structure
See DB2#WDB3 for how to adapt this structure for its .DB2 counterpart.
template<typename record_type> struct wch4_file { adb_header header; struct offset_map_entry { uint32_t ID; uint32_t offset; // this offset is absolute, not relative to another structure; this can (and often will) be zero, in which case you should ignore that entry and move on uint16_t length; // this is the length of the record located at the specified offset }; offset_map_entry offset_map[header.max_id - header.min_id + 1];* record_type records[header.record_count]; char string_table[header.string_table_size]; uint32_t IDs[header.record_count];* };
Note: If offset_map exists, all the strings will be embedded inline in the records (as null-terminated c-strings). The string block usually still exists, just as size 2 with two blank entries.
- This part of the structure is optional.
WCH5
This file format was introduced in Legion (Patch 7.0.1 build 20810) and was phased out in favor of WCH6 in Legion (Patch 7.0.3 Build 21414). As such, support for WCH5 is likely important for almost nobody.
While not a format change, the string table is now always 0 bytes if the offset map is present. In those cases, the 'string_table_size' header value changes its meaning to "offset to the start of the offset map". This is a minor optimization since parsing the file must begin with the offset map.
It's worth noting that WCH5 is missing two header fields compared to its .DB2 counterpart. Since the deduplication technology we call 'copy_table' is not employed in any of the .ADB formats, copy_table_size variable was always zero anyway. As for flags, Blizzard always loads the .DB2 file before loading the .ADB file, so the flags field being in both files is just redundant (for them); any .ADB file will be read using the flags field from the .DB2 file of the same name. This is inconvenient for us, as most (if not all) of us load .ADB files standalone at the moment, but there are workarounds (such as having a configuration file to remember the last used flags value for each named .DB2 file).
After Legion build 21737, an additional header field was added (right before 'build') called 'layout_hash'. This field's appearance is similar to the change to the WDB5 header made in the same build, except in that case, the 'build' field was removed when the 'layout_hash' field was added. For the post-21737 WCH5 header, both fields are present.
Structure
See DB2#WDB4 or DB2#WDB5 for how to adapt this structure for its .DB2 counterpart.
struct wch5_adb_pre_21737_header { uint32_t magic; // 'WCH5' uint32_t record_count; uint32_t field_count; // array fields count only count as '1' field uint32_t record_size; uint32_t string_table_size; // if flags & 0x01 != 0, this field takes on a new meaning - it becomes an absolute offset to the beginning of the offset_map uint32_t table_hash; uint32_t build; uint32_t timestamp_last_written; // set to time(0); when writing in WowClientDB2_Base::Save() uint32_t min_id; uint32_t max_id; uint32_t locale; // as seen in TextWowEnum }; struct wch5_adb_post_21737_header { uint32_t magic; // 'WCH5' for .adb (cache) uint32_t record_count; uint32_t field_count; // array fields count as the size of array for WDB4 but the entire array only counts as '1' field for WCH5 uint32_t record_size; uint32_t string_table_size; // if flags & 0x01 != 0, this field takes on a new meaning - it becomes an absolute offset to the beginning of the offset_map uint32_t table_hash; uint32_t layout_hash; // from build 21737 onward, this field is present uint32_t build; uint32_t timestamp_last_written; // set to time(0); when writing in WowClientDB2_Base::Save() uint32_t min_id; uint32_t max_id; uint32_t locale; // as seen in TextWowEnum }; template<typename record_type> struct wch5_file { wch5_adb_post_21737_header header; struct offset_map_entry // Optional - can be detected by pulling 'flags' from relevant DB2 { uint32_t ID; uint32_t offset; // this offset is absolute, not relative to another structure; this can (and often will) be zero, in which case you should ignore that entry and move on uint16_t length; // this is the length of the record located at the specified offset }; offset_map_entry offset_map[header.max_id - header.min_id + 1];* record_type records[header.record_count]; char string_table[header.string_table_size]; uint32_t IDs[header.record_count]; // Optional - can be detected by pulling 'flags' from relevant DB2 };
Note: If offset_map exists, all the strings will be embedded inline in the records (as null-terminated c-strings). The string_table will not exist.
WCH6
This file format was introduced in Legion (Patch 7.0.3 build 22345) and was phased out in favor of WCH7 in Legion (Patch 7.0.3 Build 22451).
Literally nothing changed in WCH6. The introduction of this incremented signature was probably a temporary countermeasure to combat a problem Blizzard was having with 'build' not being updated successfully. This would leave ADBs around basically forever (since they are only wiped when current build is greater than the one in the ADBs being loaded), which could cause conflicts if you downloaded an updated client with new DB2s.
WCH7
This file format was introduced in Legion (Patch 7.0.3 build 22451) and was phased out in favor of WCH8 in Legion in the next build (Patch 7.0.3 Build 22484).
A new header field has been added between 'record_count' and 'field_count'. This field designates the size of a new table which is located between the string table and the non-inline IDs table. All records are padded to 'record_size'.
WCH8
This file format was introduced in Legion (Patch 7.0.3 build 22484) and is still in use today. It was used alongside WDB5.
The only difference to WCH7 is the padding of files with an offset map. These are padded as per the WCH5 specification again; presumably as this was unnecessarily changed in WCH7.
DBCache.bin
WDB5 and WCH8 were retired simultaneously in Patch 7.2.0 Build 23436. All files in the 'ADB' folder were consolidated into a single new file, DBCache.bin. It contains server-provided new and changed records for all client-side database tables. It even sometimes contains server-side database changes (entries with an unknown 'table_hash'). It is theorized that the file is a sort of 'packet replayer' for the WoW executable, used for updating client-side DB2s to latest hotfixes on boot up, before confirming with the server about the newest changes.
The file uses the same record structures as the WoW binary requires but with none of the mechanics found in newer DB2 formats. There is no offset map, field compression of any kind, and no deduplication. Strings are inline, and IDs are always provided before the record, regardless of whether or not there is an ID value in the record data additionally.
There have previously been two versions of this page which different naming:
- The index is not exactly an index but a push ID. Multiple records will have the same "index" when pushed at the same time.
- version was referred to as region_ish in both header and entry, which may be utter bogus though. It listed 1 = us, 3 = eu, 4 = ptr, which does indeed seem to be more version than region. This would mean though that entries should also refer to it as version.
- sha256 was marked for
(7.3.2) rather than
(7.2.0.23436).
Version 1
struct dbcache_file { struct dbcache_file_header { uint32_t magic; // 'XFTH' ('HoTFiX') uint32_t version; uint32_t build_id; }; struct dbcache_entry { uint32_t magic; // 'XFTH' ('HoTFiX') int32_t index; uint32_t data_size; uint32_t table_hash; uint32_t record_id; uint8_t is_valid; // A boolean describing the 'validity' of the file. This may also control 'deletion' of the entry in question. uint8_t padd[3]; if (data_size) uint8_t record[data_size]; }; dbcache_file_header header; while (!FEof()) // This is code for '010 Editor' that means 'keep reading until the end of the file' { dbcache_entry entry; } };
Version 2 through Version 4
Version 2 added a new field directly after 'magic' in 'dbcache_entry'. At first, it was thought that the new field was just an expansion of the signature, as the field looked like 'XFTH2' for a long time. However, it appears to be something different.
There were no structure changes in Version 3 or Version 4.
struct dbcache_file_v2 { struct dbcache_file_header { uint32_t magic; // 'XFTH' ('HoTFiX') uint32_t version; uint32_t build_id; }; struct dbcache_entry_v2 { uint32_t magic; // 'XFTH' ('HoTFiX') uint32_t region_ish; // 3 for eu, 4 for ptr, apparently 1 for us int32_t index; uint32_t data_size; uint32_t table_hash; uint32_t record_id; uint8_t is_valid; // A boolean describing the 'validity' of the file. This may also control 'deletion' of the entry in question. uint8_t padd[3]; if (data_size) uint8_t record[data_size]; }; dbcache_file_header header; while (!FEof()) // This is code for '010 Editor' that means 'keep reading until the end of the file' { dbcache_entry_v2 entry; } };
Version 5 and Version 6
Version 5 added a new field directly after 'build_id' in 'dbcache_file_header'. It is a verification hash, calculated using the current file plus a specific random number embedded in the current WoW executable. This number changes for every build. This was likely added in order to prevent players from being able to load arbitrary gamedata easily using custom DBCache.bin files. (Note that this is still entirely possible, just much more tedious.)
There were no structure changes in Version 6.
struct dbcache_file_v5 { struct dbcache_file_header_v5 { uint32_t magic; // 'XFTH' ('HoTFiX') uint32_t version; uint32_t build_id; uint8_t verification_hash[32]; }; struct dbcache_entry_v2 { uint32_t magic; // 'XFTH' ('HoTFiX') uint32_t region_ish; // 3 for eu, 4 for ptr, apparently 1 for us int32_t index; uint32_t data_size; uint32_t table_hash; uint32_t record_id; uint8_t is_valid; // A boolean describing the 'validity' of the file. This may also control 'deletion' of the entry in question. uint8_t padd[3]; if (data_size) uint8_t record[data_size]; }; dbcache_file_header_v5 header; while (!FEof()) // This is code for '010 Editor' that means 'keep reading until the end of the file' { dbcache_entry_v2 entry; } };
Version 7
Version 7 removed the 'region_ish' field and moved the 'data_size' two fields back towards the end of the entry structure.
struct dbcache_file_v7 { struct dbcache_file_header_v5 { uint32_t magic; // 'XFTH' ('HoTFiX') uint32_t version; uint32_t build_id; uint8_t sha256[32]; }; struct dbcache_entry_v7 { uint32_t magic; // 'XFTH' ('HoTFiX') int32_t index; uint32_t table_hash; uint32_t record_id; uint32_t data_size; // Moved from before 'table_hash' to after 'record_id' in Version 7 uint8_t is_valid; // A boolean describing the 'validity' of the file. This may also control 'deletion' of the entry in question. uint8_t padd[3]; if (data_size) uint8_t record[data_size]; }; dbcache_file_header_v5 header; while (!FEof()) // This is code for '010 Editor' that means 'keep reading until the end of the file' { dbcache_entry_v7 entry; } };