the same-ish prompt gives you a different agent plan depending on the run and i still don't have a great answer for testing that

so in hospilot a goal gets turned into a pipeline of agents by an llm - not picking from some fixed list, actually deciding which agents and what order based on the goal text. thats the whole point tbh, thats what makes it actually useful for stuff nobody predefined annoying part is run the same-ish prompt twice and you can just get a different graph back. not WRONG necessarily, usually both are fine individually, just different. fine when theres a human watching who just picks whichever, way worse when youre trying to write a test that goes "given this goal the pipeline should look like X" and X isnt stable run to run like normal testing just assumes determinism, same input same output forever until you touch the code. that assumption doesnt hold here at all. code didnt change, prompt barely changed, output still isnt guaranteed to match what we do instead of asserting the exact graph is more like asserting properties of it - does it include the agents that HAVE to be there for this kind of goal, does it respect whatever constraints got passed in, does it avoid stuff that shouldnt be in scope. weaker than exact match but at least its testing something real instead of "did the model feel the same as it did last tuesday" anyone found something better than property based checks for this? testing llm decisions that are allowed to vary but arent allowed to be WRONG feels like its own whole discipline and most testing advice i've read just doesnt map onto it at all