Talk:FOE/Endianness
Mike C.: The reason that the bits appear to be backwards is because the OHR and all platforms it runs on uses Little-Endian byte ordering.
If you have number AB CD, then it lives in memory as CD AB, and is stored on disk the same way. 32-Bit numbers are the same: AB CD EF GH => GH EF CD AB.
Not sure of any particular rationale, but the decision was made decades ago.
S'orlok Reaves: I'm aware of little-endian usages in lumps for INTEGERS, I just didn't know the bitsets were reversed too. So, given a byte of bitset data, 08, this translated to to three bits off, one bit on, then four bits off. I verified this, of course... the "chum bug" in WH is the easiest to test. If I recall, "endian"-ness can ONLY apply to bytes, not bits. Can you point me to a line in the sources that loads bitsets? I'd like to have a look.
James Paige: I have always thought of bits in a little-endian least-signifigant-bit-first way.
1 = 10000000 2 = 01000000 3 = 11000000 4 = 00100000
This is opposite of how most people (and text books) write it.
Mike C.: Hmm. That's a good point, S'orlok. You're right, I thought the bits weren't reversed either. According to readbit(), bit 1 in an integer is, in fact, the low-bit. It doesn't say which end is up, since that's too low level, but it's not our code writing it the wrong way... (read bit is located in several places ( >:( ), one of which is in allmodex.bas:1499)
On the other hand, James, it would make sense if it was totally reversed (sorta)... This requires research.
Mike C.: According to Wikipedia, bits themselves are, 99% of the time, "little endian".
However, this is irrelevant, as (3 & 1) will always be 1, no matter which order the bits are in.
S'orlok Reaves: To James: In fact, it does make more sense your way (I had to kludge some (1<<(7-i))'s into my code in several places. Probably this will change in the FMF, later (read: "When I forget how long it took me to 'fix' this."). To Mike: Irrelevant? Of course! The OHRRPGCEFMF currently doesn't recognize ANY bitsets! Hahaha!
Mike C.: What I'm saying is that, as far as the OHR's data format goes, to read a bitset, we do this:
- Load the 16-bit integers corresponding to the bits (which are stored in little endian format)
- Calculate which integer we're looking in (bit / 16), and which bit it is (bit % 16), and create a mask based on it (1 << (bit % 16))
- We use the mask to fetch the bit ((int & mask) != 0)
S'orlok Reaves: This is suitable for the standard CE, I agree: because it doesn't store arbitrary-width numbers. In the FMF, two variables may be, say, 5 bits and 3 bits. Then, you've got:
----> --> (BYTE) = (AAAAA BBB)
... with the arrows denoting how humans would read the nunber. Were you to store that little-endian, a whole lot of shifting would take place. Consider, also, the case of a 3-bit, then 11-bit, then 2 bit variable:
--> -------------> -> (BYTE)(BYTE) = (AAA BBBBB) (BBBBBB CC) //Easy to load with: temp = readByte()<<8 + readByte() var2 = (temp&0x1FFC)>>2
I can't think of a good (easy and fast) way to do this were the bit-order reversed. Actually, I could really use your input on this one. (Note that the system described isn't pure theory; .FOE actually uses it to store enemy stats.)
Any thoughts?
Mike C.: I have always been wary of atomic storage units of less than a byte (including 4-bit bitmaps). I think it would be in your best interest to round up to the next byte. Yeah, it would result in a few wasted bits if bits % 8 != 0. But, I'm sure it would make your life easier, and backwards compatibility a ton easier too.
If you're insistent on using bits as your base unit, well, good luck. Subclass stream, and add a readBit() method, that's my only advice...
S'orlok Reaves: Ah, good advice; I'd LOVE to just use bytes, but anything I can do to shrink lumps for a mobile phone by any significant amount is worth it. (I have run tests...) Four years from now, I'll round up. For now:
You're correct, reading bits is very error-prone; there are too many assumptions to make. So I decided to read bytes, and then convert to bits. At the very least, this avoids confusion over "unused bits."
Finally, backwards compatibility is not a design goal of the XRPG format while it is in alpha. (Hopefully, that talk on the devlist of an XML format will come to fruition.) Rather, I strive to be compatible with the current lump format of CUSTOM. If an old XRPG doesn't work, just re-import it. I find it likely that a major use of the FMF will be for developers to "show off" their games on their own phones, so they will always have access. Casual gamers will probably keep the .RPG file on their computer (who really sweeps away 4mb files?) and should also have no problem.
Which brings me to my last question of the day: If I have a password-protected game in lump format X, can I update it to format X+1 without knowing the password? If not, I shall have to hack a solution together for that....
Thanks again for your comments; I truly appreciate how developers here both treat my project seriously and give me space to work. Best of luck on Uber!
Mike C.: Mmmm... XML format...
- cough* sorry. Anyway, the password is stored in the GEN lump. This has two significant repercussions, one of which is that the RPG is unencrypted. So, upgrade away.
(one of our eventual goals is to add the upgrade sub to Game, so that Game itself can upgrade RPGs on the fly)
James Paige: I would agree that upgrading an RPG file without knowing the password is okay. Dang passwords. I wish I had never let people talk me into that one in the first place :P
S'orlok Reaves: Done (stamp) and done(stamp) -on paper, at least, until the FMF reaches something more like a beta ("let's do something else"). Cheers!
Mike C.: I must say, passwords are not necessarily a bad thing. Some people are, naturally, concerned that someone my steal the sprites they spent literally minutes creating in MS Paint. And, encrypting it is not useless, even if the encryption format is fully documented. I mean, look look at, say, Blowfish. I may be highly skilled in the areas of Technomancy, but this is far beyond my skill level to implement.
Anyway, that's irrelevant. Right now, you need not know the password to access the juicy meats of an RPG (especially me, who has private versions of Custom and Unlump that do not ask for passwords)
James Paige: I didn't mean to say I think passwords in general are a bad thing. I meant that passwords on OHRRPGCE RPG files are a bad thing, yes, bad enough to make kittens cry.
Mike C.: Kittens... cry...? :'( I'm so sorry, I had no idea... *goes to donate to a kitten charity fund to appease his guilty conscience*