I haven't worked for a long time on my rpg file. I've downloaded the new version of custom.exe. Can it bug my game?

Generally, no. If a new feature is added, there will almost always be a reasonable default for older games.

The problem usually arises when people find they are relying on bugs which have then been fixed. An example of this was the old "Stats from equipment are added on levelup", which meant that heroes could become much stronger than the designer intended, since the stats on equipment were added in each time the hero leveled up. To counteract that, designers had to make the heroes that much weaker.

When the bug was fixed, there was a new problem: without the level-up bug, heroes were too weak. To solve the issue once and for all, James added a bitset that turned the bug on and off.

So, in summary, read whatsnew.txt to see if any bugs you relied on are fixed. Other than that, everything should work as before.