DRD Team

Not sure about this, but isn't there a way to achieve use of brightmaps when monsters are in custom colored fog/fades? From all that I've tried, it just doesn't want to work in anything except normal fog.

First thing, what effect are you trying to accomplish from this?

Secondly, when you do the console command testfade "00 00 01" (with quotes) you will notice certain changes in the lighting effects. Anything other than "00 00 00" for fog causes the game to enter fog mode, and among its side effects are the disabling of brightmaps. This is done intentionally in order to better simulate the effects of fog - some of these side effects are visible in the base ZDoom renderer as well.

I think this fixed my woes.

gl_spritebrightfog (boolean)

I was just trying to get brightmaps working in custom colored fog, was all. And, damn. Wish there was a way to change that around... Oh well. I think this does the trick for me though.

I'm going to guess this is due to the "only one shader per surface" rule. If both for and brightmaps are using shaders, then only one effect can be seen.

No, that's not it. The fog shader code as currently written does not work with brightmap effects. To fix that I'd have to duplicate quite a bit of code both in the shader and in the engine.
Now, that wouldn't be a big problem if the non-shader rendering path didn't exist.

So, the choice is, either I drop support for old hardware, leave brightmaps in fog out or create a mess.

Dropping support for old hardware is not an option and creating a mess sure not a preference. So it stays as it is.