EvilEngine/HIP (File Format)

From Heavy Iron Modding
< EvilEngine
(Redirected from HIP (File Format))

HIP archives, also referred to as HIP/HOP files, are files used by Heavy Iron Studios to store all level data in their Evil Engine games. They are made up of assets, which are the resources and objects used to create levels, characters and menus.

In Scooby-Doo! Night of 100 Frights, all HIP archives use the file extension .HIP. SpongeBob SquarePants: Battle for Bikini Bottom introduced the file extension .HOP, which has the exact same file format as .HIP but is typically used to store different types of assets. Localized .HIP files were added in The SpongeBob SquarePants Movie and later games to support different languages (_US.HIP, _JP.HIP, etc.).

File Format

There are three known revisions of the HIP archive format: the first one used in Scooby-Doo! Night of 100 Frights, a second one used in SpongeBob SquarePants: Battle For Bikini Bottom and a third one which is used in The SpongeBob SquarePants Movie, The Incredibles, The Incredibles: Rise of the Underminer and Ratatouille Prototype. 

PKR: HIP file version update history
Version 1:
- baseline
Version 2:
- Compatible version support information
- Asset data checksum
- Layer data checksum
- Creation/Modification date
Version 3:
- HIP Layers represent asset memory placement
- Layers are not compacted to reduce waste
- Loading is based on layers (was by asset)
- Added asynchronous loading support

Integers

All integers in HIP archives are unsigned 32-bit (4 bytes long). They are always written in big endian, regardless of platform.

Integers are referred to as int in format descriptions.

Strings

All strings in HIP archives are null-terminated (i.e. end with a NULL byte, '\0' in C/C++). They do not have a character count at the beginning. All strings are padded at the end so that their length is a multiple of two bytes, meaning:

  • Strings with an odd length ("abc") will end with one NULL (hex 61 62 63 00).
  • Strings with an even length ("abcd") with end with two NULLs (hex 61 62 63 64 00 00).
  • Empty strings are written as hex 00 00.

Strings are referred to as string in format descriptions.

Structure

A HIP archive is stored as a tree of blocks (also known as chunks), where blocks can contain arbitrary data as well as any number of child blocks.

Each block begins with an int human-readable ID that determines what kind of data is present in the block, followed by an int length that is the total amount of bytes taken up by the block's data + child blocks, if any. After the ID and length, the block's data is stored; its format depends on the block ID (see Blocks). Any remaining bytes after the data make up the block's child blocks, which have the same format as normal blocks. Child blocks are parsed repeatedly until the end of the parent block is reached.

The basic block hierarchy common to all HIP archives is as follows:

  • HIPA
  • PACK (Package)
    • PVER (PackageVersion)
    • PFLG (PackageFlags)
    • PCNT (PackageCount)
    • PCRT (PackageCreated)
    • PMOD (PackageModified)
    • PLAT (PackagePlatform) [Not present in Scooby]
  • DICT (Dictionary)
    • ATOC (AssetTableOfContents)
      • AINF (AssetInfo)
      • AHDR (AssetHeader) [Multiple can be present]
    • LTOC (LayerTableOfContents)
      • LINF (LayerInfo)
      • LHDR (LayerHeader) [Multiple can be present]
  • STRM (AssetDataStream)
    • DHDR (AssetDataHeader)
    • DPAK (AssetDataPackage)

Assets

Assets are the most important part of HIP archives; they're like the files in ZIP archives. Assets hold all of the raw data for textures, models, sounds, and even objects within a level. Assets also have a name, as well as a unique ID (which is generated from the name using a hash algorithm). The ID is used by the game to find/identify assets. Assets also have a type ID that tells the game how to interpret the asset's raw data. The asset's data is stored at a specified offset in the DPAK block. HIP archives do not use any compression for asset data.

Each asset entry is represented by an AHDR block in HIP archives.

Types

A list of all known asset types can be found here: List of asset types

Layers

Layers are used by HIP archives to group assets of specific types together. Layers have a type ID that specifies what "category" of asset types they contain. For example, TEXTURE layers contain all of the RWTX assets in a HIP archive, MODEL layers contain all of the MODL assets, and DEFAULT layers contain many different types of assets (most .HIP files have all their assets in the DEFAULT layer).

Assets in the same layer have their raw data stored next to each other in the DPAK block. This is mainly done to optimize the memory layout of assets in-game, as well as enforce the order in which assets are loaded (e.g. TEXTURE layers must be stored before MODEL in order for textures to be applied to models properly).

Each layer entry is represented by an LHDR block in HIP archives.

Types

The following table lists all known layer types and their corresponding type IDs in each game. An empty cell means the layer type is not present in the game.

Name Scooby BFBB TSSM Incredibles ROTU RatProto
DEFAULT 0
TEXTURE 1
TEXTURE_STRM 2
BSP 2 3
MODEL 3 4
ANIMATION 4 5
VRAM 5 6
SRAM 6 7
SNDTOC 7 8
CUTSCENE 8 9
CUTSCENETOC 9 10
JSPINFO 10 11

Layout

HIP archives often contain multiple layers of the same type, usually placed right next to each other. In the vanilla games, assets are distributed across those layers so that each layer has roughly the same number of assets. For example, if a HIP archive has 3 TEXTURE layers and 30 total RWTX assets, each TEXTURE layer will contain 10 RWTX assets. This was most likely done for multi-threading purposes (loading multiple layers in parallel).

The SpongeBob SquarePants Movie and later games have JSP layer "groups" to support multiple JSPs in one level. These are made up of 3 BSP layers followed by one JSPINFO layer. JSP layer groups often appear multiple times in a HIP archive. A .HIP/.HOP pair will always have the same number of JSP layer groups, even though they are completely empty in .HIP files.

The following tables specify the layout of layers for each game. Each table is sorted by the layers' order of appearance in HIP archives.

  • The Name column is the internal name for each layer type.
  • The Count column is how many times each layer type repeats.
  • The Asset Types column lists what asset types appear in each layer type and whether they mainly appear in .HIP, .HOP, or localized .HIP files (e.g. _US.HIP).
Scooby-Doo! Night of 100 Frights
Name Count Asset Types
TEXTURE 1 HIP: RWTX
BSP 1 HIP: BSP
MODEL 1 HIP: MODL
ANIMATION 1
DEFAULT 1 HIP: ALST, ANIM, ATBL, BUTN, CAM, CNTR, COND, CSNM, DPAT, DSTR, EGEN, ENV, FOG, GRUP, GUST, HANG, LITE, LOBM, MAPR, MINF, MPHT, MRKR, MVPT, NPC, PARE, PARS, PEND, PICK, PKUP, PLAT, PLYR, PORT, PRJT, SCRP, SFX, SIMP, SURF, TEXT, TIMR, TRIG, UI, UIFT, VOLU
CUTSCENE 1 HIP: CSN
SRAM 1 HIP: SND, SNDS
SNDTOC 1 HIP: CTOC, SNDI
SpongeBob SquarePants: Battle for Bikini Bottom
Name Count Asset Types
TEXTURE 3 HOP: RWTX
BSP 3 HOP: JSP (RpClump)
JSPINFO 1 HOP: JSP (JSP Info)
MODEL 3 HOP: MODL
ANIMATION 1 HOP: ANIM
DEFAULT 1 HIP: ALST, ATBL, BOUL, BUTN, CAM, CNTR, COND, CRDT, CSNM, DPAT, DSCO, DSTR, DYNA, EGEN, ENV, FLY, FOG, GRUP, LKIT, MINF, MRKR, MVPT, PARE, PARP, PARS, PICK, PKUP, PLAT, PLYR, PORT, RAW, SFX, SHRP, SIMP, SURF, TEXT, TIMR, TRIG, UI, UIFT, VIL, VILP

HOP: COLL, JAW, LODT, MAPR, PIPT, SHDW

CUTSCENE 1 HOP: CSN
SRAM 1 HOP: SND, SNDS
SNDTOC 1 HOP: CTOC, SNDI
Name Count Asset Types
The SpongeBob SquarePants Movie
TEXTURE 3 HOP: RWTX
TEXTURE_STRM 1 HIP: TEXS
BSP 3 HOP: JSP (RpClump)
JSPINFO 1 HOP: JSP (JSP Info)
MODEL 3 HOP: MODL
ANIMATION 1
DEFAULT 1 HIP: ALST, ATBL, BOUL, BUTN, CAM, CNTR, COND, CSNM, DPAT, DSCO, DYNA, EGEN, ENV, FLY, FOG, GRUP, LKIT, MINF, MRKR, MVPT, PARE, PARP, PARS, PICK, PKUP, PLAT, PLYR, PORT, RANM, RAW, SCRP, SDFX, SGRP, SIMP, SPLN, SURF, TIMR, TRIG, UIM

HOP: ANIM, COLL, DEST, JAW, LODT, MAPR, PIPT, SHRP

Localized HIP: CRDT, TEXT

CUTSCENE 1 HOP: CSN
SRAM 1 HOP: CSSS, SND, SNDS
SNDTOC 1 HOP: SNDI
CUTSCENETOC 1 HOP: CTOC
The Incredibles
Name Count Asset Types
TEXTURE 3 HOP: RWTX
TEXTURE_STRM 1 HIP: BINK, TEXS, WIRE
BSP 3 HOP: JSP (RpClump)
JSPINFO 1 HOP: JSP (JSP Info)
MODEL 3 HOP: MODL
ANIMATION 1
DEFAULT 1 HIP: ALST, ATBL, ATKT, BOUL, BUTN, CAM, CNTR, COND, CSNM, DPAT, DUPC, DYNA, ENV, FOG, GRUP, LKIT, MINF, MRKR, MVPT, NPCS, ONEL, PGRS, PICK, PLAT, PLYR, PORT, RANM, RAW, SCRP, SDFX, SGRP, SIMP, SLID, SPLP, SSET, SURF, TIMR, TPIK, TRIG, TRWT, UIM, VIL, ZLIN

HOP: ANIM, COLL, DEST, DTRK, GRSM, LODT, MAPR, NGMS, PIPT, SHRP, SPLN

Localized HIP: CRDT, SUBT, TEXT

CUTSCENE 1 HOP: CSN
SRAM 1 HOP: CSSS, SND, SNDS

Localized HIP: SNDS

SNDTOC 1 HOP: SNDI

Localized HIP: SNDI

CUTSCENETOC 1 HOP: CTOC
The Incredibles: Rise of the Underminer
Name Count Asset Types
TEXTURE 3 HOP: RWTX
TEXTURE_STRM 1 HIP: TEXS, WIRE

HOP: BINK

BSP 3 HOP: JSP (RpClump)
JSPINFO 1 HOP: JSP (JSP Info)
MODEL 3 HOP: MODL
ANIMATION 1 HOP: ANIM
DEFAULT 1 HIP: ALST, ATBL, BOUL, CAM, CCRV, CNTR, COND, DPAT, DYNA, ENV, FOG, GRUP, LKIT, MINF, MRKR, MVPT, NPCS, PARE, PARP, PARS, PGRS, PICK, PLAT, PLYR, PORT, RAW, SCRP, SDFX, SGRP, SIMP, SPLN, SURF, TIMR, TPIK, TRIG, TRWT, UIM, VOLU

HOP: COLL, DEST, JAW, LODT, MAPR, NGMS, PIPT, SHRP, SPLN

Localized HIP: CRDT, SUBT, TEXT

CUTSCENE 1
SRAM 1 HOP: SND, SNDS

Localized HIP: SNDS

SNDTOC 1 HOP: SNDI

Localized HIP: SNDI

CUTSCENETOC 1

Blocks

This section contains the format for all known block types in HIP archives. Each section heading is the block's type ID as if it was a human-readable string.

HIPA

This block contains no data or child blocks. It is used to mark the start of the file, and the ID can serve as a magic number to identify the file's type.

PACK

This block contains no data but it does contain child blocks: PVER, PFLG, PCNT, PCRT, and PMOD, as well as PLAT in every game past Scooby Doo: Night of 100 Frights. This block is used to group together various metadata about the archive.

PVER

This block contains information about the archive version. 

int subVersion
int clientVersion
int compatVersion

subVersion is 2 in all games.

clientVersion is 0x00000001 in Scooby-Doo PS2 prototype, 0x00040006 (4.6) in Scooby-Doo final, and 0x000A000F (10.15) in all other games.

compatVersion is 1 in all games.

PFLG

This block contains archive flags. It is unused by the game. 

int flags
  • In all games, this value always includes 0x2E in the flags. Its meaning is unknown.
  • In BFBB, every file except font2.HIP has some extra flags:
    • 0x140000 is used in German PS2 HIP's
    • 0x290000 is used in US GC HIP's
    • 0x2A0000 is used in US Xbox HIP's
    • 0x2C0000 is used in US PS2 HIP's
    • 0x510000 is used in GC mn-pal/mnu3.HIP and mn-pal/mnu3.HOP
    • 0x02000000 is used in all US HIP's except font2.HIP
    • 0x03000000 is used in all German PS2 HIP's

PCNT

This block contains counts of specific things, only the first 2 are used by the game. 

int assetCount
int layerCount
int maxAssetSize
int maxLayerSize
int maxXformAssetSize

assetCount determines how many assets are in the HIP archive and thus how many AHDR blocks should be present.

layerCount determines how many layers are in the HIP archive and thus how many LHDR blocks should be present.

maxAssetSize is the largest asset size in bytes out of all assets in the archive.

maxLayerSize is the size in bytes of the layer that takes up the most space in the DPAK block, minus the padding bytes at the end.

maxXformAssetSize is the largest asset size in bytes out of all assets specifically with the READ_TRANSFORM flag set (see AHDR flags).

PCRT

This block contains the creation date of the archive. 

int createdDate
string createdDateString

createdDate is the number of seconds since 00:00, Jan 1 1970 UTC, with a timezone offset of UTC-7:00 (Pacific Time).

createdDateString is createdDate as a string, in the format Www Mmm dd hh:mm:ss yyyy.

  • Www - the day of the week (one of Mon, Tue, Wed, Thu, Fri, Sat, Sun).
  • Mmm - the month (one of Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec).
  • dd - the day of the month
  • hh - hours
  • mm - minutes
  • ss - seconds
  • yyyy - years
  • Example: Thu Sep 02 06:15:46 2004

This is the same format returned by the C standard function ctime(). It always ends with a newline character ('\n', '\x0A') in Scooby.

PMOD

This block contains the latest modification date of the archive. 

int modifiedDate

PLAT

This block contains information about the specific game, platform, language, etc. the archive was built for. It's not present in Scooby. The format differs between BFBB and its subsequent games:

BFBB: 

int platformID
string platformName
string region
string language
string gameName

TSSM, Incredibles, ROTU: 

int platformID
string language
string region
string gameName

platformID is a 4-byte ID referring to which platform the archive was built for.

platformName is a human-readable string corresponding to the platformID.

region is the archive's target region.

language is the archive's target language.

gameName is the archive's target game.

The following table lists all possible values for these variables:

Name BFBB TSSM/Incredibles/ROTU
platformID 'GC\0\0', 'P2\0\0', 'XB\0\0' 'GC\0\0', 'PS2\0', 'BX\0\0'
platformName GameCube, Xbox, PlayStation 2
region NTSC, PAL NTSC, PAL
language US Common, United Kingdom, French, German US, BE, CH, CZ, DE, DK, ES, FI, FR, IT, JP, KR, NL, NO, PL, PT, RU, SE, SK, TW, UK
gameName Sponge Bob, Incredibles, Jimmy Newtron Incredibles

DICT

This block serves as a "dictionary" for all the assets and layers in the HIP archive. It contains no data and 2 child blocks: ATOC and LTOC.

ATOC

This section holds all the asset entries for the HIP archive. It contains no data and a AINF child block followed by a variable number of AHDR child blocks. The number of AHDR blocks should match the assetCount value found in PCNT. All AHDR blocks together form a list and is sorted by the id value found within each block when building the HIP archive.

AINF

This block is unused by the game. 

int ainf

This value is always 0.

AHDR

This block defines an entry for an asset. It contains some data followed by an ADBG child block. 

int id
int type
int offset
int size
int plus
int flags

id is the asset's unique 4-byte ID, which is calculated by the asset's name using a variation of the BKDR hash algorithm, described here.

type is the asset's 4-byte type ID, which is typically human-readable and can be read as an ASCII string if preferred. All known type IDs can be found here.

offset is the starting position of the asset's data, relative to the beginning of the file.

size is the length in bytes of the asset's data.

plus is the amount of padding bytes present between the end of the asset's data and the next asset in the layer's data. It's calculated from the alignment value in the ADBG block. The last asset in each layer has a plus value of 0.

flags is a bitfield of settings which specifies how the asset's data is stored and how it should be handled by the game:

  • 0x1 - SOURCE_FILE - The asset's data was sourced from an external file. The filename/path can be found in the ADBG child block.
  • 0x2 - SOURCE_VIRTUAL - The asset's data was created within the level editor Heavy Iron used. The filename in ADBG is empty.
  • 0x4 - READ_TRANSFORM - The asset's data is stored in a special format which needs to be "transformed" by the game into a runtime-specific format. This applies to all RenderWare assets, such as MODL, which is stored as a .dff file and needs to be transformed into an "RpClump" (RenderWare object) at runtime.
  • 0x8 - WRITE_TRANSFORM - The asset's data needs to be transformed from a runtime-specific format into a special binary format, likely used by Heavy Iron's level editor.
ADBG

This block defines debugging info for its parent AHDR block. 

int alignment
string name
string filename
int checksum

alignment is the multiple of bytes that the asset's data aligns to. This value can be -1 (or any negative value), which means it uses the "default" alignment value for the asset's specific type ID. The default alignment value for each asset type can be found in .LIP/.LOP files in BFBB and Scooby (either 16 or 32 for most types).

name is the asset's name, which also determines the asset's ID. This was (unfortunately) trimmed to 31 characters (plus the NULL byte) when stored in vanilla HIP files, despite HIP files supporting longer strings, however, the asset ID was calculated from the asset's original name. This means that there are sometimes multiple assets in a HIP file with the same name but different IDs.

filename is the asset's source filename if the SOURCE_FILE flag is set in the parent AHDR block.

checksum is the checksum of the asset's data. The algorithm to calculate the checksum is CRC-32/MPEG-2. It's unused by the game.

LTOC

This section holds all the layer entries for the HIP archive. It contains no data and a LINF child block followed by a variable number of LHDR child blocks. The number of LHDR blocks should match the layerCount value found in PCNT.

LINF

This block is unused by the game. 

int linf

This value is always 0.

LHDR

This block defines an entry for a layer. It contains some data followed by an LDBG child block. 

int type
int assetCount
int[assetCount] assetIDs

type is the layer's type ID, which determines what types of assets are stored within this layer. Multiple layers can have the same type ID, which means that assets will be divided evenly between those layers. Layers lists all known type IDs.

assetCount is the number of assets in this layer.

assetIDs is a list of all the assets' IDs. The order of IDs in this list determines the order that the assets' data will be stored in DPAK.

LDBG

This block is unused by the game. 

int ldbg

This value is unknown in Scooby-Doo PS2 prototype and always -1 (0xFFFFFFFF) in all other games.

STRM

This block organizes all of the data for the assets in the HIP archive. It contains no data and 2 child blocks: DHDR and DPAK. Some Incredibles PC archives (mainly localized ones) are known to have an incorrect section/block size value in this block. This could be due to the additional used padding bytes, even though the assets are the same length as in the US version. However, blocksize values in this block are ignored by the game anyway.

DHDR

This block is unused by the game. 

int dhdr

This value is always -1 (0xFFFFFFFF).

DPAK

This block contains all asset data, grouped by layer.

Each layer is aligned to a fixed size which depends on the target platform:

  • GameCube: 32 bytes
  • PS2/Xbox: 2048 bytes

Each asset is aligned to the alignment value specified by their corresponding ADBG block.

In order to align assets and layers, padding bytes are used. All padding bytes have the value 0x33.

  • All assets, except for the last asset in each layer, have padding bytes after them. The plus value in each asset's AHDR block is set to the amount of padding bytes used.
  • All layers, including the last one, have padding at the end. The maxLayerSize value in PCNT does not include the amount of padding bytes used.

If there are 0 assets (assetCount in PCNT is 0), this block contains no data. Otherwise: 

int paddingAmount
char[paddingAmount] padding
char[] data

There is padding between the start of the block and the beginning of the asset data. The amount of padding is calculated from the fixed layer alignment size (described above), minus the first 4 bytes which is reserved for paddingAmount. It specifies how many padding bytes will follow.

Scooby-Doo PS2 Prototype: Those HIP archives do not have padding bytes at the beginning of this block. They also use 0x00 as pad byte instead of 0x33.

Tools

HipHopFile is a C# library which can be used to work with HIP files.

External links

  • xhipio.cpp - BFBB's source code for reading basic blocks and values from HIP archives
  • xpkrsvc.cpp - BFBB's source code for reading all assets and layers from BFBB-specific HIP archives
Retrieved from "https://heavyironmodding.org/wiki/EvilEngine/HIP_(File_Format)?oldid=9171"