What levels of hardware upgrade feel meaningful?

Obviously more memory is good, more context, bigger models, but some jumps don't actually unlock a meaningful difference in ability to run different or better models. For example, I don't currently view jumping from 32+16 to 64+16 as a particularly worthwhile upgrade as compared to going to 32+32, though correct me if I'm wrong. I'd like to build a DDR4 + HBM2 based inference machine to complement my main, 32 GB DDR5 + 16GB GDDR7, computer. The idea is that even if the hardware is slower, the greater overall capacity enabled by the slightly more affordable hardware could allow me to run a greater diversity of models. What level of memory upgrade do you think is most logical for maximizing model access if the compromise is outright speed? 32+32? 128+32? 64+64? Or am I completely asking the wrong kind of question and just outing my own ignorance here? either way I'd like your input.