Certainly it is difficult to be poor. I've been both poor and comfortable--though never wealthy--and poor is much worse.
I would argue that there is no "desperate search to continue finding more oppressors.' This implies that people who call out racism are digging for it, which in turn implies that it is buried and that we are pulling it out of obscurity.
This is patently absurd. We aren't digging for racism and desperately searching for 'more oppressors' because we're already constantly oppressed by the reality of white privilege.
Oppression is in our face every day, regardless of whether we want it there or not. The answer is much more often 'not.' White privilege only exists if our oppression exists. Why would we want to 'search desperately' for oppression?! It's in every orifice, all the time. We utilize constant energy to try to extrude it while a number of white people pretend it doesn't exist, energy we could use for other things if racism didn't exist.
You may enjoy Gina Crossley-Corcoran. She was desperately poor: