RPG format

From OHRRPGCE-Wiki
Jump to: navigation, search

RPG file format (also known as data mismanagement 101)[edit]

The RPG file format is the format used by the OHRRPGCE to store games. This page is intended as a reference for people who want to write utilities to manipulate or convert RPG files, for those working with the source code, and for those who just wish to poke and prod their data. This file is also an interesting case study in bad data structuring, and should provide a wealth of cautionary examples of what not to do. It tracks my progress from "not having a clue how to manage data" to "knowing how to manage data but being unable to do it right because of backwards compatibility" and finally to "not caring one way or the other".

Lump Format[edit]

An RPG file is made of many files lumped together. These files are referred to as "lumps".

An RPG file starts with a null-terminated file (lump) name.

Lump names may be up to 50 characters long, from the set [a-zA-Z0-9._-]. Lump names are case insensitive (consider the lump file format non-case-preserving), and each lump should have a unique name. (Very old implementations of the RPG format supported directory paths in the lump name, both relative and absolute. This would have made it possible for an RPG file to overwrite critical system files in other directories-- luckily, I found this flaw before anyone else did, so it was never exploited :)

Anyway. Null-terminated filename. 4 bytes of size information, then the data, repeat:

Data Meaning
(characters) Filename (encoded in ASCII)
BYTE always 0-- it terminates the filename
LONG size, in PDP-endian format, [high-byte, highest byte, lowest byte, low byte]

The following example Python code converts a 4-byte string into a size:

import struct
# the on-disk size format can alternatively be thought of as two little-endian unsigned 16bit integers
hisize, losize = struct.unpack ('<2H', rawsizestring)
size = losize | (hisize << 16)

The reverse (converting a size into a 4-byte string) is similarly simple:

import struct
losize = size & 0xffff
hisize = (size >> 16) & 0xffff
rawsizestring = struct.pack ('<2H', hisize, losize)

Assumptions to make[edit]

Unless otherwise noted, numbers are stored in sane-right-thinking-normal-moral-christian-good-to-their-mothers byte ordering of

  0,1

  A,B

 low byte, high byte

Unless otherwise noted, 16 and 32-bit numbers are signed, as the goodly creators of QuickBasic decided that nobody would ever really need an unsigned value for anything. (the same people, if I recall correctly who decided that 640k of RAM would be all anyone ever needed)

Assumptions not to make[edit]

Once again, even though lump names are usually UPPERCASE, dont assume they have to be.

Even though lumps may appear to come in a specific predefined order in the RPG file, don't assume they will always come in that order. The lumping code puts ARCHINYM.LMP first to improve browsing speed, but that's not written in stone.

Lump naming rules[edit]

Lump names vary depending on the file name of the RPG file. It was moronic of me to design in that way, but I didn't know any better at the time, and I have to maintain backwards compatibility with my past screwups.

eg. If the filename of the RPG file is WANDER.RPG it will contain many lumps named WANDER.???

In an attempt to fix this silliness without breaking backwards compatibility, I added a new all-important lump. ARCHINYM.LMP

ARCHINYM.LMP Is a text file, which contains the Internal name used by all of the flexible-named lumps in the file. Then it has a CR/LF pair, and a string that tells the version number of CUSTOM.EXE that the file was created with (or the version of RPGFIX that first added the ARCHINYM.LMP)

So. Read the first line of ARCHINYM.LMP to get the correct prefix of most of the other lumps.

If ARCHINYM.LMP does not exist, you know you are dealing with an obsolete file, and you can assume that the variable-named lumps will use the RPG filename as a prefix.

All new lumps added in the future are likely to have fixed arbitrary names.

Note: new versions now always use 'ohrrpgce' as the internal name, to cut this mess down to an absolute minimum. This is obviously can never be assumed.

List of Lumps[edit]

WARNING: The documentation on any page in the "Unfinished Articles" category might be not just incomplete, but down-right WRONG. Any other page may still contain errors or be out-of-date... but we don't expect many errors!

ARCHINYM.LMP Internal game name
ATTACK.BIN   More attack data, in addition to DT6
BINSIZE.BIN  Binary record lengths for newer binary lumps
BROWSE.TXT   Long game title and description
DEFPAL#.BIN  Default palettes, one for each of PT0 through PT8
COMMANDS.BIN (inside HSP lump) Builtin script command names
DEFPASS.BIN  Default passability for tilesets
FIXBITS.BIN  Bitsets that indicate whether format fixes have been applied
SCRIPTS.BIN  (inside HSP lump) New script names and data
SCRIPTS.TXT  (inside HSP lump) Obsolete script names
LOOKUP1.BIN  Script trigger lookup table
SFXDATA.BIN  Sound effect data
SONGDATA.BIN Song data, currently just the name of each (replaces SNG)
PALETTES.BIN 256-colour master palettes
PLOTSCR.LST  Plotscript names and ID numbers
MENUITEM.BIN Menu items for user-defined main menu and submenus
MENUS.BIN    User-defined main menu and submenus
UICOLORS.BIN User interface colors
slicetree_#_#.reld Saved slice collections
SLICELOOKUP.TXT User defined names for looking up slices
distrib.reld Meta-data used for exporting distribution files
heroes.reld  Hero definitions
heroform.reld Hero formation data

(The following are the extensions of the variably-named lumps. See #Lump naming rules)
GEN          General misc data and record-counts

.##          .1 .2 .3 and so on, up to .99 are BAM songs.
.T## / ##.T  tilemap for map ## (See Map Format)
.P## / ##.P  passability (wallmap) for map ##
.E## / ##.E  enemy positions (foemap) for map ## 
.D## / ##.D  door links for map ##
.L## / ##.L  NPC locations for map ##
.N## / ##.N  NPC definitions for map ##
.Z## / ##.Z  Zonemaps for map ##
DOR          Obsolete door locations. Feel free to delete it from your RPG
DOX          Real door locations
DT0          Obsolete hero data
DT1          Enemy Data
DT6          Attack Data
EFS          Enemy formation sets
FOR          Enemy formations
FNT          Font
HSP          Plotscripts
HSX / HSZ    (inside HSP lump) Single compiled Plotscript
ITM          Item data
MAP          General Map data
MAS          Master palette
MN           Map names
MXS          Backdrops and backgrounds
PAL          16-color palettes
PT0          Hero pictures
PT1          Small Enemy Graphics
PT2          Medium Enemy Graphics
PT3          Large Enemy Graphics
PT4          Walkabout Graphics
PT5          Weapon Graphics
PT6          Attack Graphics
PT7          Box Border Graphics
PT8          Character Portrait graphics
SAY          Text boxes
SHO          Shop definitions
SNG          Obsolete music name list
STF          Shop stuff
STT          Global Text Strings
TAP          Tile animation patterns
TIL          Maptile sets
TMN          Tag names
VEH          Vehicle definitions
Map Format    Describes what lumps you need to load a map.

Other non-lump files[edit]

RSAV          RELOAD-based saved game files
SAV           Obsolete saved game files
HS            HSpeak-produced .HS files are simply .HSP files with a different extension
OHF           Exported font files are just .FNT files with a different extension
tilemap       A .tilemap file, exported from the map editor, is just a .T##
              file with a different extension.
slice         A .slice file, exported from the slice collection editor, is just a
              slicetree_#_#.reld file with a different extension
BMD           A .BMD file is just a temporary MIDI file, which has been converted from a BAM file
ohrkey        Replayable recorded user input
gameconfig.ini Holds config settings for a single game customised by the player.
editorconfig.ini Holds config settings for Custom (shared between all games).
joyset.ini    Created next to game.exe, stores the joystick settings from the CTRL+J menu.
              A relic of the DOS days but still used, will eventually be replaced.
persist.reld  Holds persistent data for a single game which doesn't belong in saved games,
              such as in-app purchases.
session_info.txt.tmp  Created in the working.tmp folder by Custom while editing a game.
              It provides information about the unlumped game and the instance of Custom editing it.

Places where tags are used[edit]

(Warning: we're likely to forget to update this.)

SAY           9 conditional checks, 2 settable tags, 2 choicebox tags
N             2 checkable tags per NPC definition
D             2 checkable tags per door link
MENUITEM.BIN  2 checkable tags, 1 settable tag, 1 toggleable tag
TAP           Optional tag check command in tile animation patterns, tag to disable the animation
DT0           Have hero, is alive, is leader, is in party now
DT6           2 checkable tags, for 2 settable tags
ITM           own item, is in inventory, is equipped, is equipped by active hero
STF           2 check tags, 2 settable tags per shop item/hero
VEH           Currently riding vehicle
HSZ           Scripts use tags with check tag and set tag
FOR           Battle formation Victory tag

Places where scripts are used[edit]

(Warning: we're likely to forget to update this.)

GEN           4 scripts: New game, game over, load game, menu action
MAP           5 scripts: map autorun, after-battle, instead-of-battle, each-step, on-keypress
MENUS.BIN     1 On-close script trigger
MENUITEM.BIN  1 script for each menu item (provided that the menu item is type 4)
N             1 NPC script trigger
SAY           2 script triggers, one instead-of-box, one after-box
SHO           Inn animation script
VEH           4 scripts: use button, cancel button, on-mount, on-dismount