Asset packing

Offline data conversion tool of Jazz² Resurrection.

AssetPacker is a standalone command-line tool in "Sources/Utilities/AssetPacker" that converts original Jazz Jackrabbit 2 data into the layout a given platform loads. The game performs the same conversion on its first run (see GameEventHandler::RefreshCache, which forwards to the shared Jazz2::Compatibility::AssetConverter driver), and that stays the second, in-game way of doing it. This tool exists so the data can be prepared ahead of time — for the platforms that cannot convert anything themselves (the consoles and the web build), and for build pipelines.

Beside the full conversion it offers several single-file commands: packing and unpacking bitmap fonts (see Jazz2::UI::FontFormat), making the indexed font images editable, and re-encoding .j2v cinematics into the game's own Jazz2::VideoFormat container.

Building the tool

The tool is host-only — it is built by default as part of the desktop CMake build (NCINE_BUILD_ASSET_PACKER, default ON) and skipped for every cross-compiled target. It links only the base layer and the original-data converters, deliberately not the engine, so it has no renderer, no window backend and no networking. After a regular desktop build the binary is at "build/Sources/Utilities/AssetPacker/AssetPacker".

Command-line reference

AssetPacker [<command>] <source> <target> [options]

The command may be omitted, in which case convert is assumed:

convert <source directory> <target directory> — Converts the original game data
- <source directory> — Directory containing the original game files (Anims.j2a, *.j2l, *.j2t, ...), or a game installation that keeps them in a Source subdirectory — in which case its Content is copied to the target as well
- <target directory> — Directory the converted data is written to (created if needed)
- --target=<profile> — desktop (default) | console | dreamcast | wii | gamecube | psp | emscripten (or web)
- --video-downscale=N — Downscale cinematics by N (1–4), 1 (the default) keeps them as they are. Cinematics are re-encoded for dreamcast (or any N > 1) and otherwise copied unchanged, desktop gets none, as the game reads the originals
- --originals-only — Convert only the episodes and levels the original game shipped
- --shareware-only — Convert only what the Shareware Demo shipped (implies --originals-only)
- --all-videos — Deploy every cinematic found, not just the two the game plays
- --skip-non-episode-levels — Convert only levels that belong to an episode
pack-font <source .png> <target .font> — Packs a grid image and the character list next to it into a single file. The list is read from <source .png>.json, or from the binary <source .png>.font if there is no JSON next to the image
unpack-font <source .font> <target .png> — Unpacks a font back into a grid image and a character list, ready to be edited and packed again. The list is written both as <target .png>.json, which is the one to edit, and as the binary <target .png>.font
apply-palette <source .png> <target .png> — Replaces the palette indices of an image with the colors they stand for, so it can be edited
to-indices <source .png> <target .png> — Resolves the colors of an edited image back to the nearest palette indices
recompress-video <source .j2v> <target .j2v> [--video-downscale=N] — Re-encodes one cinematic on its own, N defaults to 1, which keeps the original resolution

Progress and warnings are printed to stdout, errors to stderr, and the exit code is non-zero on failure.

Converting the game data

The source can be either a directory of original game files or a whole game installation, which keeps them in a Source subdirectory next to the Content the game ships and the Cache it converts into. Pointing the tool at the installation is the convenient thing to do, so it looks for both layouts — and when the source turns out to be an installation and the target is not the desktop profile, the installation's own Content (fonts, animations, metadata, translations — nothing of which is derived from the original data) is carried over as well, which is what makes the output a complete self-contained tree. The tool requires Anims.j2a (or the shareware AnimsSw.j2a) to be present, sprites and sounds are converted from it into a package — Source.pak for the desktop profile, Prebaked.pak for the others, which is what tells the game the tree needs no conversion — and the levels, tilesets, episodes and music follow via Jazz2::Compatibility::AssetConverter.

Target profiles

What the output directory is going to be loaded by decides its layout:

desktop — The converted data goes into a Cache subdirectory of the target, and an index file (Source.idx) is written alongside — the descriptor the game checks on startup before deciding to reconvert. It must stay identical to what GameEventHandler::WriteCacheDescriptor writes, field for field (signature, file type, cache version, flags, the modification time of Anims.j2a, the event count and the build version) — the game compares every one of them and reconverts everything if any disagrees. Music is not copied, because the desktop game reads it from its own Content/Music, and cinematics are left alone, because the game reads the originals where they are.
console (also selected by dreamcast, wii, gamecube and psp) — A staged Content tree written directly into the target, holding everything the console loads. The consoles all consume the same staged tree, so they share one profile — only the cinematics are decided per platform, which is why dreamcast is tracked separately (see below). The sprite and sound package is named Prebaked.pak here, and that name is the marker: a console that can convert (the PSP, the Wii and the PlayStation 3, whose content is on writable storage) finds it and skips the conversion instead of looking for original files that were never deployed — see Jazz2::ContentResolver::IsContentPrebaked(). No index is written, deliberately: a prepared tree is never rewritten, and an index there would only invite the game to try.
emscripten — Likewise a tree prepared entirely ahead of time for the web build, with Prebaked.pak and without an index.

Both non-desktop profiles also put the game's own Animations and Metadata inside Prebaked.pak instead of next to it — a few hundred small files become part of one, which is what a console pays for on a memory card, a memory stick or a disc. They are the two directories the game reads through the package layer, everything else the tree carries (the translations, the levels, the tilesets, the music and the cinematics) is read as a loose file and stays one. The conversion writes its own files first, so an asset the original data provides is what a path present in both resolves to, exactly as when the two are kept apart. The desktop profile is untouched by this: its Source.pak holds only what was converted, and the game reads its Content from where it is installed.

Level filters

--originals-only keeps the episodes the original game shipped plus every level reachable from one of them, --shareware-only narrows that to the Shareware Demo content, and --skip-non-episode-levels drops the levels that belong to no episode. Whatever is skipped is listed by name rather than only counted — the list of levels the original game shipped is maintained by hand, and this is how a name missing from it shows up.

Cinematics

Only two cinematics are ever played by the game (Intro and Ending), Logo is present in the original data but nothing asks for it, so it is deployed only with --all-videos. The files land in a Cinematics subdirectory under lowercase names, which is how the player looks them up. What happens to them depends on the target:

Nothing for the desktop profile — the game reads the original files where they are
Copied unchanged for every other target except the Dreamcast — they decode the original format perfectly well and are better off with the smaller file
Re-encoded for dreamcast, or whenever --video-downscale is greater than 1 (on any profile), since downscaling means re-encoding either way

Only the Dreamcast needs the re-encoding: inflating the original container there costs 55–115 ms per frame against a 42 ms budget, where the re-encoded one costs under one millisecond.

The .j2v recompression

The original .j2v videos (CineFeed signature) are 640×480 in four interleaved zlib streams — opcodes, the offset and row parts of copy-from-previous-frame runs, and the literal pixels plus palettes — and the decoder has to inflate every frame at playback time. Jazz2::Compatibility::J2vRecompressor decodes the video exactly the way the player does (including decoding into one reused frame buffer, so pixels no run covers keep the previous frame's value), optionally downscales it once by picking every n-th pixel of every n-th row — matching what the player used to do at runtime — and re-encodes it into the game's own Jazz2::VideoFormat container. The player detects the format by its signature and accepts both.

The output container keeps the same idea — 8-bit indexed frames encoded as changes against the previous one — but replaces the entropy coding with a byte-oriented delta-RLE codec whose decoder is nothing but memcpy and memset over a reused frame buffer. A file starts with the same signature and type/version fields every other file the game writes uses, followed by width, height, frame delay, frame count, pixel format, codec and an extension area older players skip, then one size-prefixed payload per frame. Each frame is a flag byte (bit 0 — a 256-entry BGRA palette follows) and then commands until the end-of-frame marker: literal spans stored as they are, runs of one repeated byte, and skips that leave the previous frame's pixels in place — which is what makes an unchanged region nearly free. Each command has a short single-byte form and a long form with a 16-bit count, the exact opcode ranges are documented on Jazz2::VideoFormat.

Note the recompressed file is substantially larger than the zlib-compressed original at the same resolution — that is the deliberate trade: entropy coding is exchanged for a decoder that fits the weakest platform's frame budget. Platforms that can afford the inflation get the original file copied instead. recompress-video re-encodes a single file outside a full conversion, which is the way to experiment with resolutions per target.

Font packing

A font is authored as a grid image — one equally sized cell per character — next to a sidecar description listing the cell size, the character set and how far the pen moves for each character. The game instead loads a single self-contained .font file (version 2, described by Jazz2::UI::FontFormat) in which every glyph has been measured down to the pixels it actually inks, packed against its neighbours and stored as palette indices. pack-font and unpack-font are the two directions of that conversion, so a shipped font can be opened up, changed and packed again without the grid form having to be kept around in the repository.

The description comes in two interchangeable forms. <image>.png.json is the one meant to be edited, <image>.png.font is the binary form the sidecar has always been kept in, holding the same fields packed into a few bytes with the characters outside the ASCII range spelled out in UTF-8. unpack-font writes both, and pack-font reads the JSON wherever it finds one next to the image and falls back to the binary form only when there is none — so editing the JSON is enough, and a font unpacked by an older build still packs. Which one was used is printed.

What pack-font does:

Trimming — Every glyph is reduced to its inked bounding box, what is left of the cell is recorded as the glyph's bearing, so drawing it at the trimmed size in the trimmed place puts every pixel exactly where the full cell used to put it. A glyph with no inked pixels at all (a space) has zero size and is only advanced over.
Index normalization — A font is entirely index based: every pixel either names a palette color or is index 0 and draws nothing, there is no per-pixel coverage. Pixels that say otherwise (transparent in one channel but not the other, or partially covered) are resolved to one of the two, with a warning listing how many were touched.
Packing — Shelf packing, tallest glyph first, with a margin of FontFormat::GlyphMargin (one pixel) on each side, so neighbours are two pixels apart — enough that a bilinear sample taken at the very edge of one glyph cannot reach the next. The atlas width is a power of two chosen to minimize the padded area (which is what the console backends that need power-of-two textures pay), while the height is left at whatever the shelves come to (which is what everyone else pays). The atlas may not exceed 1024×1024, a glyph may not exceed 255 pixels in either direction and a bearing may not exceed 127 — exceeding any of these is an error.
Encoding — Everything except the few identifying bytes (signature, file type, version, flags, compressed size) is one deflate block: the atlas size, line height, base spacing and character counts, then a 9-byte glyph record per character (position, size, bearings, advance — with a 32-bit codepoint prefix for the non-ASCII ones), then the atlas itself as one palette index per pixel in the same QOI-based encoding every other image asset uses. The ASCII range is described by two bytes (where it starts and how long it is) and everything else by a 16-bit count, so a font may have at most 255 characters in the range and 65535 outside it — exceeding either is an error.

unpack-font rebuilds the grid image and both sidecars from a packed font — the glyphs are placed back into cells at their bearings, so packing the result again measures the pixels afresh and arrives back at the same font. Because the unpacked atlas holds palette indices, it looks like near-black noise in an image editor, apply-palette and to-indices are the two halves of making it editable — the first replaces the indices with the colors they stand for, the second resolves an edited image back to the nearest palette entries (warning when a color had no exact match). The tool carries its own minimal PNG codec (Jazz2::AssetPacker::PngCodec) precisely because it does not link the renderer.

The JSON description

{
    "CellWidth": 15,
    "CellHeight": 20,
    "Columns": 19,
    "LineHeight": 20,
    "BaseSpacing": -2,
    "AsciiFirst": 32,
    "AsciiCount": 95,
    "Characters": [
        { "Char": " ", "Advance": 5 },
        { "Char": "!", "Advance": 4 },
        { "Char": "\"", "Advance": 6 },
        { "Char": "é", "Advance": 8 },
        { "Codepoint": 120071, "Advance": 11 },
        { "Fallback": true, "Advance": 8 }
    ]
}

CellWidth, CellHeight, Columns — The grid the image is read as: the size of one cell and how many of them sit in a row. Required, as is Characters, everything else has a default.
LineHeight — How far one line of text sits below the previous one. Belongs to the font rather than to the grid it is authored in, so it can be set on its own, defaults to the cell height, which is what it almost always is.
BaseSpacing — Pixels added between characters on top of each advance, negative to tighten. Defaults to none.
AsciiFirst, AsciiCount — Where the contiguous run the packed font indexes directly starts and how far it reaches. Both are worked out from the list itself when omitted — the leading run of consecutive characters, which is how one is written out — and the first AsciiCount characters have to be exactly that run, since that is the order the packed file stores them in. The run is described by two bytes, so it has to end by U+00FF, and the game keeps only the part of it below U+0080.
Characters — One entry per cell, in the order the cells are laid out in the image, and at least one. Char names the character as itself, Codepoint as a bare number for the ones without a spelling that survives a text file, and Fallback marks the single entry drawn in place of anything the font doesn't have (the same thing as "Codepoint": 0). Advance is how far the pen moves after drawing the character and is required, the glyph's size and bearings are not listed at all, being measured from the image every time it is packed.

Every field is checked against what the packed file can hold — the cell up to 65535 pixels in either direction, Columns and every Advance up to 255, BaseSpacing a signed 16-bit value — and one that is out of range, missing where it is required, or not the type it should be is an error naming the field and, inside the list, the position. So is a list that doesn't begin with the ASCII range it claims, which is also how a character repeated inside that range is caught:

Error: "font_small.png.json" has to list the ASCII range first, one character after another
from U+0020, but the character at position 34 is U+0041 where U+0042 was expected

What still describes a loadable font is a warning rather than an error — a codepoint listed twice outside the ASCII range (only one of the two is ever drawn), and a character below U+0080 left out of the range, where the game never looks for it. Both cost a glyph quietly, so they are worth reading:

Warning: "font_small.png.json" lists U+0104 more than once

The file is otherwise read leniently: comments, trailing commas and a byte order mark are all tolerated, so it can be annotated while it is worked on and saved by any editor. It is written as UTF-8 without one.

Platform notes

Desktop builds do not need the tool at all — the game converts on first run. Running it is still useful to prepare the cache in a build pipeline, or to pre-downscale cinematics with --video-downscale.
Dreamcast, PlayStation Portable, Wii and GameCube all consume a tree prepared with --target=dreamcast / psp / wii / gamecube. All four share the same layout, the target name only decides the cinematics — the Dreamcast gets them re-encoded into Jazz2::VideoFormat, the others get the smaller originals copied. Where each console expects that tree, and how it gets there, is described in Building for consoles. The Dreamcast and the GameCube have no conversion compiled in at all and require the prepared tree, the PSP and the Wii would convert an installation left in their Source directory, which is slow on both and is why they are given a prepared tree as well.
PlayStation 2 and PlayStation 3 consume the same tree under the generic --target=console profile — neither has a name of its own, because neither needs anything the profile does not already do. The PS2 boots from a read-only disc and so requires the prepared tree, exactly as the Dreamcast does. The PS3 has a writable hard disk and could in principle convert, but is given a prepared tree for the same reason the PSP is — doing it on the console is slow.
PS Vita and Nintendo Switch do convert the data themselves on first run, so the tool is only a way to skip that wait there.
Emscripten likewise ships a fully prepared tree (--target=emscripten), typically with --shareware-only for the public demo.
The packed .font files are what all platforms load, but the consoles benefit the most: one palette index per pixel instead of expanded RGBA cuts both the file size and the texture memory, and the tightly packed atlas minimizes the power-of-two padding the PVR, the GX and the PSP's GE all have to pay.

Pitfalls

The desktop cache descriptor must match the game exactly. Source.idx is compared field for field against what the running game would write — including the event count and the build version — and the game silently reconverts everything on startup when any field disagrees. Prepare the cache with a tool built from the same sources as the game.
Never write an index for the console or web profiles. The tool deliberately omits it there, a prepared tree is never rewritten, and an index would only invite the game to try.
Do not rename Prebaked.pak in a prepared tree. Its name is what marks the tree as already converted, so a renamed package leaves the game looking for original files it does not have — and finding none, showing a main menu with nothing to play.
The source must contain a supported version of the game data. The tool looks for Anims.j2a (or AnimsSw.j2a) at the top of the source directory or in its Source subdirectory and refuses to run without it.
Skipped levels are listed on purpose. The list of original levels is maintained by hand, so read the skipped-levels output after a --originals-only conversion — a level that should have been kept shows up there by name.
Recompressed cinematics are larger than the originals. Only re-encode where the platform needs it (the tool's per-target defaults already do the right thing), everything else wants the original file.
The game plays pre-downscaled cinematics as they are — the player picks its runtime downscale from the actual frame width, so a file reduced by the tool is not halved a second time.