Diablo 3 MPQ Hacking and You Guide : Diablo 3 Bots | Diablo 3 Hacks

Quick MPQ Introduction
The MPQ file format was developed by Blizzard and is used by all their games. It is essentially an archive format that supports various compression techniques. It is optimized for read operations and uses hashing for quick file indexing.

The MPQ files are also used for patching the client game files. When a patch is released, it will include one or more MPQ files which contain only the updated files. The patch MPQ can also contain references to files that are to be deleted. When a file is patched, it can either be replaced in its entirety (if the changes are significant), or, more commonly, the MPQ will contain a BSDIFF of the file. Each file patch entry contains the MD5 of the file before the patch. This allows the patching mechanism to ensure that it does not attempt to patch a file that is not at the correct version level.

When the game client is loaded, it loads the needed game files from the appropriate "core" MPQ, and then applies that patches one at a time in chronological order. The MPQs do not store a version of the patched file! It's possible that the patched files are stored in the cache directory.

The tool that I've been using for working with MPQ files is Ladik's MPQEditor. (Not sure about the site policies for external linking of private sites, so I didn't include a link, but you can google easily enough.) I've had no trouble extracting and patching Diablo 3 game files with this tool.

Diablo 3 MPQ Files
In the Diablo 3 beta client, the core MPQ files are located in the following directory (for Windows clients):
[D3_Install_Dir]\Data_D3\PC\MPQs\

This directory includes the following MPQ files:
base-Win.mpq -- This MPQ contains the core executable files. Unlike other MPQs, the contents of this MPQ are extracted to the [D3_Install_Dir]. This MPQ includes the "Diablo III.exe", and various DLLs including the "bnet\battle.net.dll".
ClientData.mpq -- This MPQ contains most of the game graphics related files. I think the files in this MPQ are probably platform specific, but I haven't spent any time looking into the details of this MPQ.
CoreData.mpq -- This MPQ contains most of the interesting game client data. This includes all the affix information, item information, monster information, treasure classes, etc.
enUS_Audio.mpq -- As the name implies, this MPQ contains all the enUS localized sound files (*.snd and *.ogg). All sound files with English dialog are included here.
enUS_Cutscene.mpq -- This MPQ contains the enUS localized cut-scenes for the five character classes.
enUS_Text.mpq -- This MPQ contains all the enUS localized strings that appear as game text anywhere in the game. This includes everything from localized affix and item names to NPC conversations.
HLSLShaders.mpq -- This MPQ contains shaders.
Sound.mpq -- This MPQ contains non-localized music and sounds.
Texture.mpq -- This MPQ contains all of the game textures. These textures are stored in a Blizzard specific archive format. Each *.tex file can contain multiple textures. Necrolis' D3TexConv program (he includes the source in the download) can be used to extract the images from the *.tex files.

Diablo 3 MPQ Patch Files
THE MPQ patch files are located in one of three subdirectories under the MPQs directory:
base -- For all base files.
enUS -- For all enUS localized files.
Win -- For all Windows platform specific files.

An MPQ patch file has a name like "d3-update-base-7338.MPQ" where "7338" is the version number. As a reminder, to patch the core MPQ file, you must apply all of the patch files in sequence.

All files in an MPQ have a 0x10 byte file header. The first 4 bytes are always the "magic number": EF BE AD DE. Not sure if the next 4 bytes identify a version number or the file type, but this DWord is identical for all files of the same type for a given version. The last two DWords in this header always seem to be 0x00000000.

MPQ GameBalance *.gam Files
The "CoreData.mpq" file contains the "GameBalance" directory. This is where many interesting gam ("*.gam") files are located including "AffixList.gam" and "Items_*.gam". There's a lot of interesting game information located in these files.

The file type id / version number for *.gam files was 0x0000071C in the core MPQ, but is now 0x0000075F as of version 7318 or 7338.

MPQ GameBalance *.gam Header
The gam files have a header size (excluding the 0x10 byte MPQ header) of 0x3A8 as of version 7318 or 7338 (this was 0x398 in older versions).

I haven't figured out the complete structure of the gam header, but here is what I have so far (make sure you add 0x10 bytes for the MPQ header if you are looking at absolute file offsets):
DWord fileId; // +0x00 (unique file id)
DWord unknown1[2]; // +0x04
DWord startOffsetIndex; // +0x0C (?? see below)
DWord magicNumber1; // +0x10 (fixed value: 0x30303030)
DWord magicNumber2; // +0x14 (fixed value: 0x6962672E)
DWord unknown2[248]; // +0x18 (0xF8 bytes that always seem to be 0x00)
DWord resourceName[256]; // +0x110 (filename of the resource from which this file was generated)
DWord unknown3; // +0x210 (the value of this DWord is different for each file)
DWord unknown4[388]; // +0x214 (0x184 bytes that always seem to be 0x00 with the exception of the data start and end offsets -- see below)

The "unknown4" block contains all 0x00s with the exception of two DWords (back to back) that contain the data portion start and end offsets. For a given version of gam files, the start offset is always the same. For version 7338, this is 0x000003A8. The location of this offset in the "unknown4" block is variable and appears to depend on the "startOffsetIndex" at 0x0C. This value seems to be an index that determines the location of the two DWord offsets. For a given value of the "startOffsetIndex", the location of the offset DWords is always the same. Here are some examples from the core version; the first value is that of 0x0C, and the second value is the absolute file offset where the data start and end offset DWords are located:
0x01 --> 0x228
0x02 --> 0x238
0x03 --> 0x248
0x04 --> 0x348
0x05 --> 0x268
0x07 --> 0x288
0x08 --> 0x278
0x0A --> 0x298

While at first it appears that the "startOffsetIndex" directly corresponds to the offset location, there are several outlier values (e.g., 0x04 --> 0x348) that prevent this from being the case.

The way I have been processing these files is to read all the bytes starting at 0x214 and looking for the first non-zero DWord; this is the location of the data start offset. This approach seems to work fine although I'd like to find something a little more elegant.

The "unknown3" value at 0x210 is completely different for each gam file. I cannot find any kind of pattern and I have no clue what this value is.

MPQ GameBalance *.gam Data
Following the gam header is the data portion. Each record in the data portion has a fixed record size. As far as I can tell, this record size is not specified in the header, nor is the total number of records. However, it's easy enough to figure out the record sizes when looking at the hex dumps of the file.

Some gam files also have a variable data section located at the end of the records. To determine if a given gam file has a variable data section, simply add the MPQ file header (0x10 bytes) with the gam file header (0x3A8) with the data end offset. If this is the same as the gam file size, then there is no variable data.

Two of the files that have variable data are the "AffixList.gam" and "Items_Armor.gam" (as do most of the other "Item_*.gam" files). Gam files that contain variable data will contain pairs of DWords that specify how to read the variable: the offset of the variable data (from the start of the data portion) and the number of bytes of the variable data.

For the "AffixList.gam" file, each record can have up to 4 references to variable data. Each piece of variable data corresponds to an affix modcode (more on this later, probably in a separate post). The structure for the modcodes is as follows (0x18 in size):
DWord modcode; // +0x00
DWord modparam; // +0x04 (used for things like elemental damage, otherwise 0xFFFFFFFF)
DWord unknown1; // +0x08 (currently 0 for everything)
DWord unknown2; // +0x0C (currently 0 for everything)
DWord varDataOffset; // +0x10 (note that this is the offset from the start of the data section)
DWord varDataNumBytes; // +0x14 (number of bytes of the variable data)

The four modcode structures are located towards the end of each record at the following offsets:
modcode1: 0x1C0 (offset at 0x1D0, size at 0x1D4)
modcode2: 0x1A8 (offset at 0x1B8, size at 0x1BC)
modcode3: 0x190 (offset at 0x1A0, size at 0x1A4)
modcode4: 0x178 (offset at 0x188, size at 0x18C)

The size of the variable data varies based on the modcode. The possible values are 0 (in the case of no modcode), 0x0C, 0x1C, 0x28,0x34, and 0x40. In the core version, here are some of the modcodes with their corresponding sizes:
0x0C: Block, GoldPickUpRadius, HitMana, HitLife
0x1C: Experience, Resists, Attack, Prec, Def, Vit, Will, AllStats, Life, MinD, MaxD, Thorns, DR, KillMana, KillLife
0x28: Damage, CriticalD, Defense, Life, Gold, MF, Haste, Cast, LifeS, Regen, Run
0x34: Inferior, Superior
0x40: ItemCost

In the next post, I will dive into the details of the "AffixList.gam" file. After that, I will dive into the "Item_*.gam", and then the "\TreasureClass\*.trs" files.