Can I run CUSTOM.exe or GAME.exe in a small window instead of fullscreen?
Yes, yes you can.
To run in a window, either press [Alt]+[Enter] while GAME or CUSTOM is running*, or edit the shortcut to GAME or CUSTOM and add -w to the end of the command-line.
* Note that hitting [Alt] while GAME or CUSTOM is displaying the "Load Game" menu will actually exit the program. Wait until you've loaded and are running (or editing) a game.
CustomQB.exe and Game-QB.exe
No. They are DOS programs, and Windows is unable to run them windowed because of technical issues.
However, using DosBox, you can run dos programs in a window (or fullscreen). In addition, you will get sound (from BAMs).
The downside is that you need an obscenely high-end computer to get any decent results. On the other hand, if you use an unofficial WIP build, certain changes make it much faster. We are unable to provide support for it, however.