I tried reversing chat templates to see which models are the best at generating user turns

I tried for fun to see what happens if we let instruction-tuned LLMs generate the text that should come from the user (so after the user_start special token). I immediately noticed that some models were way better than others. I wanted to test many models so I did some testing and found out that GPT 6 Astra is quite good at identifying human-like text. You find the results of the experiment at: https://selfgpt.pages.dev/ (I also tested roleplay vs chat template reversal) TLDR: models are much better at emulating human behavour with the chat template trick rather than with a roleplay system prompt. The best model appears to be Mistral Small 3 24B (between the 30ish models tested) It would be really interesting to see the performance of base models (with a simple user: and assistant: template) on human emulation. Let me know what you think! Repo: https://github.com/Belluxx/selfgpt Here's an example of response from Mistral Small 3 24B generating in the user slot (no system prompt), replying to a message from ChatGPT: I see perhaps it's time to branch out and try a new A.I. Then you won't have to keep up with the bells and whistles of this one. Your responses seem to follow a bell curve that slowly descends to uselessness