Building a command line tool to view Minecraft data

By John Lekberg on July 04, 2020.

This week's post is about building a command line tool to view Minecraft NBT files. You will learn:

How to unpack binary data with the struct module.
How to write a recursive parser for the NBT file format.
How to turn the parsed data into a JSON friendly file format.

Named Binary Tag (NBT) files store information about Minecraft in a simple binary file format. To learn more about the details of the NBT format, read these documents:

Script source code

view-nbt

#!/usr/bin/env python3

import collections
import enum
import gzip
import json
import struct

# ---- Data

Tag = enum.Enum(
    "Tag",
    [
        "End",
        "Byte",
        "Short",
        "Int",
        "Long",
        "Float",
        "Double",
        "Byte_Array",
        "String",
        "List",
        "Compound",
        "Int_Array",
        "Long_Array",
    ],
)

tag_scalar = {
    Tag.Byte,
    Tag.Short,
    Tag.Int,
    Tag.Long,
    Tag.Float,
    Tag.Double,
    Tag.String,
}

tag_array = {Tag.Byte_Array, Tag.Int_Array, Tag.Long_Array}

byte_tag = {
    0x00: Tag.End,
    0x01: Tag.Byte,
    0x02: Tag.Short,
    0x03: Tag.Int,
    0x04: Tag.Long,
    0x05: Tag.Float,
    0x06: Tag.Double,
    0x07: Tag.Byte_Array,
    0x08: Tag.String,
    0x09: Tag.List,
    0x0A: Tag.Compound,
    0x0B: Tag.Int_Array,
    0x0C: Tag.Long_Array,
}

scalar_fmt = {
    Tag.Byte: "b",
    Tag.Short: "h",
    Tag.Int: "i",
    Tag.Long: "q",
    Tag.Float: "f",
    Tag.Double: "d",
}

array_fmt = {
    Tag.Byte_Array: "b",
    Tag.Int_Array: "i",
    Tag.Long_Array: "q",
}


def _validate_data():
    """Runs some assertions to make sure I defined the data
    correctly.
    """
    assert set(Tag) == tag_scalar | tag_array | {
        Tag.End,
        Tag.Compound,
        Tag.List,
    }
    assert set(Tag) == set(byte_tag.values())
    assert scalar_fmt.keys() == tag_scalar - {Tag.String}
    assert array_fmt.keys() == tag_array


# ---- Parsing

Node = collections.namedtuple(
    "Node", ["tag", "name", "value"]
)


def parse(data):
    """Parse binary NBT data into a Node.

    data -- a bytes-like object containing the NBT data.
    """
    M = bytes(data)
    I = 0

    def take(fmt):
        """Unpack data at the current offset and advance the
        offset forward.
        """
        nonlocal I
        s = struct.Struct(">" + fmt)
        [value] = s.unpack_from(M, offset=I)
        I += s.size
        return value

    def take_s(N):
        """Take an N character string from the offset and
        advance the offset forward.
        """
        return take(f"{N}s").decode("utf-8")

    def repeat(N, f):
        """Create a list from calling a function N times."""
        return [f() for _ in range(N)]

    def _parse(*, tag, named):
        """Recursively parse the data.

        tag -- Optional[Tag]. If the Tag is known in
            advance, then this is that Tag, otherwise None.
            This is used by Tag.List, which provides the
            element tag types.
        named -- Boolean. "A Tag name should be parsed."
            E.g. this is True inside Tag.Compound, but False
            inside Tag.List.
        """
        if tag is None:
            tag = byte_tag[take("B")]

        name = None
        if named and tag != Tag.End:
            N = take("H")
            name = take_s(N)

        value = None
        if tag in scalar_fmt:
            value = take(scalar_fmt[tag])
        elif tag == Tag.String:
            N = take("H")
            value = take_s(N)
        elif tag in array_fmt:
            N = take("i")
            value = repeat(N, lambda: take(array_fmt[tag]))
        elif tag == Tag.List:
            list_tag = byte_tag[take("B")]
            N = take("i")
            value = repeat(
                N, lambda: _parse(tag=list_tag, named=False)
            )
        elif tag == Tag.Compound:
            value = []
            while True:
                node = _parse(tag=None, named=True)
                if node.tag == Tag.End:
                    break
                value.append(node)

        return Node(tag=tag, name=name, value=value)

    return _parse(tag=None, named=True)


# ---- JSON Formatting

tag_use_value = tag_scalar | tag_array


def json_friendly(node):
    """Turn a Node object into data that is more JSON
    friendly.

    E.g. Tag.Compound is turned into a dictionary. Tag.List
    is turned into a list.
    """
    if node.tag in tag_use_value:
        return node.value
    elif node.tag == Tag.List:
        return [
            json_friendly(subnode) for subnode in node.value
        ]
    elif node.tag == Tag.Compound:
        return {
            subnode.name: json_friendly(subnode)
            for subnode in node.value
        }


# ---- Main Logic

if __name__ == "__main__":
    _validate_data()

    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument("nbt_file")
    parser.add_argument(
        "--gzip",
        action="store_true",
        help="use if nbt_file is GZIP compressed",
    )
    args = parser.parse_args()

    with open(args.nbt_file, "rb") as file:
        data = file.read()
    if args.gzip:
        data = gzip.decompress(data)
    node = parse(data)
    print(json.dumps(json_friendly(node)))

$ ./view-nbt --help

usage: view-nbt [-h] [--gzip] nbt_file

positional arguments:
  nbt_file

optional arguments:
  -h, --help  show this help message and exit
  --gzip      use if nbt_file is GZIP compressed

Using the script to view an NBT file

I have an NBT file, level.dat, that contains information about a Minecraft level that I play on:

$ cat level.dat | openssl base64

H4sIAAAAAAAAAMVZzY8cRxV/s7M7O9P77V07tkMgBC4oxMT5sCJEEu/OeNdr1vFq
x4lNLqOa7pqZYqu7mqrqXQ8HFCFxiMQNATHKIScO4eOKkEBykPiULSHxByAkBBeQ
QOKUS3hV1T3T4+611ygSI/VopurVq/devfd79V57AB5Mt4gmVTh7g0QBlSzqX5cE
f7Rjchg1ByTyKQCcqcHChpA43qSRpvJNSD8V8Fqs12N+wvWwOg0nHFWbfZ3uUBlf
ZyEdk85KwiLcAaZhejzz8q+qUN8iIb0+jO1QA+bbVB5QuSFRJlV3aw9IxDgn9wly
M+NegzU33iIh6dNdXM2Fv//q72+bzw9qsOqmbxBpRLCT6uKaW+zB8g0hebBFozbV
GudVBea6IkpUxx9QpY3EitLghT8/Efz1mZ+sVWClTyMqiaadHiU6kVRVPPAC1CpS
TETKg9WQRdSXpKe/KFCZQ7OBB410nZB1qKt0s1La+7b0YL7LREg7SiTSpwWB5jmR
fdqxNArqMK2NOc+MGacG7HAypJIGI5KlMUkkmKKjiTKh0FRr42E9oJ2I4rc8SrHl
HHNL+Eha1aEWo2WpLvLJhDw5ngjRB1lnUocjlSvVArVbmRyn0ZFntjCmRKpH1MsJ
Udzs4YKXrDEOPNeSpC+iTdYf6IaJJ00PyVBVcdI4+TI+M2monMCnbuPMLDTBBejf
AEv4TOGzis88Pgv4mPVz+Czis2LDDGAWnc3t9mV0KBpUKrC8K+kBE4niw3RsHJAj
LDD+qde//e5LLuRm36DShEoF6u2IxGogNG43tY36eLOo7WsICVA7f+78hXPnUb7Z
FhnmQQMDFMFEM8KRv5GhcYOoqyII0LUrsEA4F4dNEYYGQpDvmTJ8a1EMBoAn/4gA
0EyUFuGGUOrSAUKLQgkbBpX2Ek4Rg7xAbDJJrzN/Hw9CJrSOrkdupTsgULJoh0Z9
PYCZCy+++PwFXNFDeodG2YpljLvEp0GLdpP+dtQTMNMjXOHM2YAp0uX0Eh9qSa5i
vCGQ6OaA4nYZzRqJIpEgIK8HBwaXQytmynoxkOLQINvkjid8J6DFu2uJjhM9muoJ
2WcHtEVJsGtBYcRsIRBXRdeayMK1G11BfS9FmulhU5IwNDNTz71Qh/lU9j3C0NSZ
tIuBuIHIiFHVHPp8JNBSIPAguXHTifEVPP/DlllwlSqFGoyEmQvEdcZpS4p4rG0S
oeRSEb4e9TFus03xlLYjJcKIkZGaOBSGNGAYEXtUGZ1G5KuRgW7C92ga4eiO2bqG
tcGOECODebiKT5p3YZ/SeDsyDiPkcMQX7ecMNSH0SiB2WMg0DZomfI39sgVzoehu
SUZ7OXMvmewnQuNx7Zgar34eCa0CeyRgiYKp88+iT6F5xh6dW7/MRX89wGMaRUE6
cVrF1NcGztRWmsSagyTaHxGsKkSVdNUm7twlY6fH7XaJloJPbofRUnNO5MHMhsnz
HtRDGgpUShkYWbqcSL1hI1hpEsZgwcZrczSgGZQGj7x1rSXrJpoqA0zVGkxvEEVf
/ZcL+gwUPjWGQHtuzD9HtEYhO4E7nXTdxeXJdZ88cp2yBs62+/nt2+/llj1ZXBam
AZouREBEJ0i4MSYGgrkVbaKvbPKhMVAVFneF1IQ3heABRilYMF5e7yohY+Nz6yHG
tXaw2CBdxhHX8EIB82ySay0kwx4fGv4sQjN2E4YpeQYah4Q7J3n57t17CKhIt2Hm
EBZrPSfFDNTxlyPauXsP/88bGVsM+bhLHm7uSeqzmG4Isd/A65r9Yy9gzXwG7hIp
Kc+nKHdJemw8oGKJQNdB3gxh5fR44lCIgEadGL2a3JpI30NqILuDBBxO5FkT6QvC
86TEJz4jna4gGk4VeCu8qQT5BI8hF3Z60pxl7k7D0evcbmfygwhMnb4kQzeV2/Sr
SdTntLjpANl3uojN6MM58jiRMacFdTADFDZNjYXAHwdCTHBJp4xm+U3T4R41J/dM
zlh4cLhBmGgtIlRZhB0fg81koo6ZM46Qt63EdNcVh3ljpUY0h5M7z4DI/Y4g+079
ooBW7hyXrslGTtGCrXwiY1rmLTEnBoVOFvTpUtrL02cHPcAw5PDZB+qvQqd2cbuu
pZqYEAkfGSxvqBgBzQJ2Lgo4wbuGzINRpgZeWRVWBEYfPYFWKukTmYplf/okmjDz
hPz5sPHTjNHRFgpybhyzaN8Z+omcbC7zdgaI8bRDJOIwPFVq1kkj5c7KAGmkSSEM
bGykR7h63/D9tJmhjc/knYNJf1AY7UtKU5cpuuNAUHj6AQoUvLwQ012eFKPRSt1F
BM8JLTDv9lPSpbwLxHja+eDJ/CSvGTfZxC59vDxaO33jEznD+UMSFTZT2twz77d6
CjKfKJ6zxV/nbo8VZ7t4iVEl56JYP8qLkkIZmmM1r1JmtxIGnHThbHFDTvt9WyPl
zDUCj9Mlbkp5SHUFPsPUBifKOPpmIiPi003Gtb212ytTmgaXmGqjw1K5lbBrMY1w
aI2pImmlAQtabFDMcbG5mwQmk32uPHiOSETHCZr/LVkdJ74+nhxWGqXl4fiI6e7/
Eb0PT7GjoC5Lu0cE+vHj+mPItqXRf5xke3wUeLRUfETGPT40PHoKfmimLYOOYvY9
KndWYBkRwWHIGCUaTKV/8GL8+IigFGROF7BozOdsBkAlK7Gqadg61jUrqlC7GbdN
kWC7KbM34+sCSwFz256D6ddf326Z7stffva9v73SfO23byWn3ln4wvw1LBksYsl8
X7SC1R8GWRN9mSGeN6B2VZjqoZb2aUo/HsztIfy/QQcMq22s0lxherzFJTUCFkVW
nppLAQ2o7gplmdx5+92bf/rNL75x8drGh9e+ub1057s//dpbL31rdwqmTecEClVS
HaZYAIs5f8IznoLqOpOVz2MRcy3aklgXBeUFVM56V/7+uzu3/31l4Ye/rn34wdr3
/9CA+h4a2ahnel5TzZlPE3Brauum4BuUrUJta5cp4Xqw/p5RfbWH4NW2DQLDaYfi
XfNVrJzKJKzjmWet39K+aUGk56LLH/znXsN6hGVtDrgKM21fGFuBdwzLVuFUnPbd
die85SP8pGb/6B8zUL0Z777y/vs/wgL7kmmCbWMuUeAccs60/9NOnGm8VaFh9M5E
WkvtcOnWgCRqbId//vj2FNRNTZ+5+XKbcuprGhjubS5MOVtrQGPUHTEV/VIFpu2c
O/vSsqMCM01TDlc8qGrSxwBKey4Ap40TWQYVx+DIwvIBTGDEZMoxOaKCfACLpRGL
qmORA6YMf9ziqYxwuqixhVQDn6OdMuIZR1yWuAu0NUebx9ZDSjFjUiltE8HRz2b0
swXLZZc5EnNEsoLkDbfgZF5yxLeOaWirgjheQc9xpZMR1zPiuSIEYN7JyGoZ2bwj
WylkvIJyC47y/pt0RlfN6BYLRpgssApqLRetnK9ZM/qnMRAWTLyYrl3a1rJhVrPt
sq/g780q1E1/zMbNVPQOhqAeJCYs7Ujl7acyavOS7VlEGbxTBhYVMKmNX7btCB/T
3Yi1eRF2EaMQQZ7ItPGatcsr4KVbuHbU7EEW71d+WYPFtE1PevRNEdHsndg0eDuY
/iywYPKqqKXvfOliDVYmXqOZHS6ecwsasNTGG0iQcBqkLXSXMBoWMmwrbXbX9DQl
qYE3fjmQvRIogpGHsIoju3j5Ug2YvRSZswkmXwjOQ73l+s9BmrX+C1DWASjXHAAA

I use view-nbt to view the data in a readable format:

(level.dat files are compressed, so I use the --gzip flag.)

$ ./view-nbt --gzip level.dat |
  python3 -m json.tool |
  head -n 30

{
  "Data": {
    "WanderingTraderSpawnChance": 25,
    "BorderCenterZ": 0.0,
    "Difficulty": 3,
    "BorderSizeLerpTime": 0,
    "raining": 0,
    "Time": 15810,
    "GameType": 0,
    "ServerBrands": [
      "vanilla"
    ],
    "BorderCenterX": 0.0,
    "BorderDamagePerBlock": 0.2,
    "BorderWarningBlocks": 5.0,
    "WorldGenSettings": {
      "bonus_chest": 0,
      "seed": 3809796128940862740,
      "generate_features": 1,
      "dimensions": {
        "minecraft:overworld": {
          "generator": {
            "settings": "minecraft:overworld",
            "seed": 3809796128940862740,
            "biome_source": {
              "seed": 3809796128940862740,
              "large_biomes": 0,
              "type": "minecraft:vanilla_layered"
            },
            "type": "minecraft:noise"

How the script works

NBT files have tags of several different types, so I used an enum.Enum, Tag, to represent the tags.

To help with the parsing, I defined several auxiliary data structures:

scalar_fmt is a dictionary that maps some Tag objects to format strings usable by the struct module. This is meant for "simple" tags, like Tag.Int or Tag.Byte.
array_fmt is a dictionary that maps some Tag objects to format strings usable by the struct module (but, missing a length prefix). This is meant for "simple array" tags, like Tag.Int_Array.
byte_tag is a dictionary that maps bytes to the corresponding tags. I got this information from reading the NBT file format specification.

The parser uses the struct module to unpack binary data into different numbers (like int, double, short).

The parser is recursive.

It directly parses Tag.String and the tags in scalar_fmt and array_fmt.
It recursively parses Tag.List and Tag.Compound.

The parsing creates a Node object, which is a collections.namedtuple that holds:

The tag type.
The tag name (optional).
The tag value.

To improve the readability of the data, I have a function, json_friendly, that turns Node objects into data structures that, in my opinion, make the JSON easier to read.

I use Python's gzip module to provide support for compressed NBT files.

I use Python's json module to dump the JSON friendly data into a string.

In conclusion...

In this week's post, you learned how to build a command line tool to view Minecraft NBT files. You learned:

How to unpack binary data with the struct module.
How to write a recursive parser for the NBT file format.
How to turn the parsed data into a JSON friendly file format.

My challenge to you:

Create a tool, view-region, that turns Region files into readable JSON.

If you enjoyed this week's post, share it with your friends and stay tuned for next week's post. See you then!

(If you spot any errors or typos on this post, contact me via my contact page.)