Woke up way too early this morning, and ended up watching a bunch of YouTube videos. One of them I came across was this, which made me realize I had been using the wrong BFO frequency for my radio! Adjusted things, and I can hear people! I was doing the BFO on the wrong side of the filter, getting frequency inversion. For the amount of times I heard n2cqr talk about it, you think I would have gotten it right!
The voices were super quiet - don't have the right gain in the right places - but I could hear it! I've also got a bit of a ground loop I think, so I had to make sure I was touching the audio amp potentiometer or my amplifiers became oscillators. But at least now I know I'm not crazy - I really couldn't figure out why I wasn't hearing what I expected.
For my reference, here are the frequencies. I am doing the 20M band, and the BFO (ch2 here) is at the top end of my crystal filter. (I measured it the other day as going from ~8.9967 to ~8.9986.)