Plan for faster game loading and upgrading
Game and Custom take far too long to unlump and upgrade games. This is extremely annoying in large or old games. However, both of these things are in fact mostly unnecessary. The long upgrade times in Game are largely an unfortunate consequence of the Plan for file format upgrade unification. Also, even if you use an .RPGdir instead of an .RPG file, Custom still copies the whole game to a temporary working folder.
Don't unlump[edit]
Provide an interface for abstracting the loading from an RPG lump which allows us to read straight from the RPG file (or from an .RPGdir). There are at least two ways we could we do this. One is the obvious route: a new set of functions for this; all code directly acting on files and file descriptors will need to be replaced. Just how much of this is there?
Alternatively, it turns out that FB's files are fully abstract -- see FB_FILE. This means that we could write an interface to access RPG lumps as normal files. We would implement all the required methods and provide just an OPENLUMP function as public interface which creates one of our lump wrapping FB_FILEs, and then just modify all the usages of OPEN everywhere, which is surely far less work. But how much work would writing the backend methods be? We could of course write everything in FB, except that the FB runtime headers might need porting to FB from C.
Sound backends[edit]
Many newer games are largely sound and music files. SDL_Mixer and Audiere don't require files to be unlumped in order to play them. SDL_Mixer can read from RWops and Audiere has abstract File objects.
Don't copy[edit]
Whichever lump-abstraction method we choose, we could allow modification of lumps, storing a modified copy either in memory or to temporary file. This should work in both Game and Custom. This could then also replace our current methods for runtime modification of lumps with scripts which are cumbersome and error prone.
Don't upgrade[edit]
Game and Custom both do exactly the same upgrade on each game they load. But Game probably doesn't necessarily have to go through the same steps, it could do less.
Some upgrade steps patch specific things like the password format, which is a fixed amount of work. Others iterate through each data record of some type and make some change; it's these later ones where the problem lies. By targeting just the worst offenders, upgrade time could probably be cut down to almost nothing. The very worst offenders are the fixbits, which do a slow logical load and save of each record, followed by upgraderecordlength which just does binary loads and saves.
We increasingly have nice wrappers for reading/writing NPCs, heroes, doors, doorlinks, graphics, tilesets, palettes, attacks, vehicles, etc. We could upgrade data only when it is actually loaded, in these load functions, cleanly and out of sight. And of course, record lengths don't have to be expanded at all except when another upgrade depends on them being the right size.