Plan for new RPG format

After extensive debate on the talk page and mailing list (at [1] and [2]) this particular plan seems to have been abandoned and replaced with RELOAD, to where some of the ideas on this page have survived and evolved.

This article or section is born of the wild ideas and speculation of Mike C.. It should not be considered official information just yet.

The current RPG format sucks. It is a mish-mash of binary formats that are constrained by 16-bit signed integers and backwards compatability hacks and stuff like that.

But, it needn't be that way. My proposal is this:

• For lumps where it makes sense, we should convert binary lumps into an XML-based format.
• For lumps where it doesn't make sense to do so (eg, tile maps), they should remain binary.

How Custom changes

When opening an old game, Custom will have to convert the old-style binary lumps into XML based lumps. This should be an easy process, except that it's not incremental, as is the current upgrade process. Fortunately, it only need be done once, and never again.

The various editors would need to be rewritten to work with XML. This is not a big deal, except that much code is written on the asbestos-ridden foundation of the binary format. However, with liberal use of wrapper functions, this should not be too much of a pain. Further, this transition could be made relatively painless, in a way I will explain soon.

For efficiency reasons, Game would not deal with the XML. It would deal with binary lumps, but not the current ones. They would be lumps generated from "compiling" the XML lumps, in a format that makes sense for that lump. For example, the attack data would be in a single lump, the .GEN lump would be sorted, and would have irrelevant entries removed, etc.

Although we are converting from XML back to binary, we will still retain the XML lumps for future editing. The binary lumps would be treated like the output of a Makefile process: important to running the program, but recreatable from the source.

Thus, to make the transition as painless as possible, we need start with only one lump, something easy, like the Textbox format. We can compile it at save time, and the user of Custom doesn't even have to know the difference. Eventually, when the whole file needs to be recompiled, it will be more efficient to offer separate "Save" and "Save and Compile" options, as well as a command line "compiler", so that when you're just making changes, it's quick and easy.

How Game changes

Dealing with the game player is both easier and harder than Custom. It's easier, in that we don't need to care about having to write the data down when we're done using it, but harder in that backwards-compatibility is more important.

So, to start with, here's how to deal with old, binary-based games:

• The data needs to be "decompiled" into XML, just like Custom would do.
• The resultant XML lumps need to be "recompiled" back into binary, just like Custom would do.

For (hypothetical) games based on XML, but "old", we simply:

• Compile the XML lumps into binary.

As long as we design the XML format carefully, there need not be a difference between an "old" XML file, and a current one that has a default field. More details below.

So, now, we have binary data that we know how to process. So, we process it. Nothing much else to say, really.

The XML format

As an example of how a lump would look in XML, let's look at the Textbox format.

<textboxes>
 <textbox id="1">
  <text>Bob: Hello!</text>
  <appearance border="1" size="4" transparent="transparent" />
 </textbox>
</textboxes>

As you can see, only the data which is not default need be included. It doesn't have any conditionals, nor does it define many of the appearance properties. So, why waste space with them? Assuming we have reasonable defaults, then we can leave out lots of things.

Let's say we change the Textbox format to add... Um... Sparkles! Yes, you can now add sparkles, defining the colour, quantity and placement of each sparkle! But, let's say we don't want to add any to our textbox. What would the XML look like?

<textboxes>
 <textbox id="1">
  <text>Bob: Hello!</text>
  <appearance border="1" size="4" transparent="transparent" />
 </textbox>
</textboxes>

That's right, it would be exactly the same. In other words, we don't need to upgrade anything, since it doesn't mean anything.

(For those who are curious, here's how I would mark up the sparkle feature:

<textboxes>
 <textbox id="1">
  <text>Bob: Hello!</text>
  <appearance border="1" size="4" transparent="transparent">
   <sparkles>
    <sparkle x="2" y="35" color="255,0,0" />
    <sparkle x="73" y="3" color="0,0,255" />
   </sparkles>
  </appearance>
 </textbox>
</textboxes>

)

The binary format, of course, would need to include all the data, including default data, since binary is not as flexible as text. However, since we're generating the binary, and not using it as our main data, then we don't need to worry about making room for things. Need to change a 16-bit number into a 32-bit number? No problem, just update the compiler and anywhere it's used in Game (assuming it's not already treated as 32-bit!).

Another benefit of using XML is that data import/export is trivial! For example, we just add the textbox import/export function, which is both complicated and messy. But, with XML, it's as easy as copying out the lump. We don't even have to care about what things to export, since we only store non-default fields.