Ogg Vorbis (.ogg) is a compressed music/sound format. It provides better quality and better compression than MP3, and is a patent-free format.
OGG files provide good compression and good sound quality. Custom forces you to convert MP3 files into OGG files because certain of our music backends have trouble playing back certain MP3 files. Sad, isn't it.
If you want to manually convert WAV or MP3 files into OGG (for better fine-tuning of compression options), download oggdropXPd. When you run oggdrop, you will see a floating ogg vorbis fishy icon. Drag and drop your WAV files onto it, and it will create an OGG vorbis version of the file.
You can also record your own sound effects with Audacity and then export as OGG.
By default, CUSTOM encodes files at quality 2. Ogg quality can go from -1 to 10, with negative one being poor quality but small files, and ten being obsessively perfect quality but gigantic files. 4 is almost certainly higher than you need for simple sound effects. Experiment with different values. 2 is a good default. I have successfully gone as low as 0 or 1 with no noticeable difference (your experience will vary depending on the sound effect you are working with)
Another alternative is using SoX to record, play back, and convert into OGG, if you prefer command-line programs.
Sometimes you might want to play an ogg file without importing it into your game. That is easy. Just download and install Ogg Vorbis Codecs from Illiminable. They allow you to play OGG files in Windows Media Player, or any other DirectShow compatable player. Optionally, if you're running Windows, you can just use Winamp or VLC, which both include support for the ogg format.