Would there be a use case for running a 405B on a single 8xA100 node with up to 30 fine tuned specialists loaded hot at sub 200ms switching?

I know people consider llama 405b and others to be old now, lol, but I'm wondering if there would be a use case for it. I had a use case for a project I was building and I wanted to share what I got and get some feedback which would be much appreciated. base model: llama 3.1 405b (awq-int4, 202gb) hardware: single 8xa100 80gb node had free vram remaining: 150gb after base + adapters + kv cache adapter switching was sub 200ms via vllm enable lora uptime is over 60 days with zero service restarts adapter training is nf4 trained adapters served on awq-int4 base without retraining projected adapters capacity is roughly 30+ based on remaining vram and adapters sizes which were between 2-5gb each. 7 concurrent adapters combined was 82.9 tok/sec time to first token was 63-66ms single adapter throughput was 18.7-19.2 tok/sec sustained and 25 tok/sec peak Multi lora at smaller model sizes is already well documented and the gap I wanted to test was whether the same pattern holds at 405b scale on a single node under real production conditions. I was running into issues with the health niche since it's super sensitive sending information across API models and the smaller llms weren't producing the right outcomes. I couldn't justify the cost of the H100 which is what I found on the Meta documentation and I was fortunate enough to find a way to fit it on the 8xA100 so I wanted to share it. Legal and my user facing AI was the biggest issue in most categories and subcategories which is the main reason I went with the 405b with being fine tuned and distilled to reduce the chances of a bad output that could cause problems in the health niche. Same reason I went self hosted with a large llm. I know some people run smaller models for very specific tasks, some use larger models to train smaller models so they aren't always on, but for large models that typically require a larger node. For my case I needed large models because certain tasks pass through multiple models and the smaller ones didn't have the reasoning depth needed so I needed the larger model. So far I've had zero issues over 60 days. I've used fine tuning and distillation for the legal, CRO, SEO, and other adapters and it's performed well for everything so far. I have 7 adapters currently loaded with tons of headroom. I'm curious as to what workloads people think this actually fits or doesn't and if so, what would you use it for. I have a full write up and configs on Hugging Face if anyone is interested.