Verifiable Self-Evolution for Open-Ended Dialogue Skills via Future-Feedback Prediction

# Verifiable Self-Evolution for Open-Ended Dialogue Skills via Future-Feedback Prediction
Source: [https://arxiv.org/html/2607.18973](https://arxiv.org/html/2607.18973)
ChaoJin Zhao ByteDance zhaochaojin@bytedance\.com&Xuan Jiang ByteDance jiangxuan\.1217@bytedance\.com

###### Abstract

Textual skills provide a lightweight way to improve frozen language\-model agents, but their self\-evolution normally requires a stable validation signal\. Such signals are natural in mathematics or code, where an answer can be checked after it changes, yet are problematic in open\-ended dialogue: changing the assistant response also changes the user’s next reaction, so a logged reaction cannot directly evaluate a counterfactual response\. We propose*future\-feedback skill evolution*, which first redirects self\-evolution from prescribing the current answer to predicting whether the observed answer will lead to a positive or negative subsequent user signal\. This prediction task is verifiable on fixed logged tuples and therefore supports validation\-gated textual optimization\. The evolved feedback skill captures interpretable criteria for response quality and can subsequently serve as a diagnostic and optimization target for answer skills\. On a proprietary, privacy\-preserving sales\-assistant dataset, careful quality filtering and a balanced resolved/unresolved split yield more than 75% prediction accuracy\. Beyond this result, the central contribution is a formulation that converts otherwise moving conversational feedback into a fixed offline learning target, enabling reproducible skill evolution without placing every candidate skill in live traffic\. We discuss the boundary between observational verification and counterfactual validity, and position the method as an offline optimization stage rather than a replacement for final human or online evaluation\.

*Keywords*agent skill evolution⋅\\cdotuser feedback prediction⋅\\cdotdialogue evaluation⋅\\cdotlanguage\-model agents

## 1Introduction

Large language model \(LLM\) agents are increasingly controlled by textual*skills*: persistent instructions that encode domain procedures, quality criteria, and tool\-use policies\. Compared with parameter training, skill editing is inexpensive, inspectable, and deployable without changing the underlying model\. Recent work such as SkillOpt formalizes this idea as text\-space optimization: an optimizer proposes bounded edits to a skill, and an edit is retained only when it improves a held\-out score\(Yanget al\.,[2026](https://arxiv.org/html/2607.18973#bib.bib1)\)\.

The validation score is the critical assumption\. In verifiable domains, an altered response can still be checked against a stable target\. A mathematical proof can be re\-evaluated, code can be executed, and structured outputs can be compared with known answers\(Gaoet al\.,[2023](https://arxiv.org/html/2607.18973#bib.bib7)\)\. Open\-ended dialogue lacks this invariance\. Consider a logged interaction

x=\(𝒞,ℋ,Q1,A,Q2\),x=\(\\mathcal\{C\},\\mathcal\{H\},Q\_\{1\},A,Q\_\{2\}\),where𝒞\\mathcal\{C\}is contextual and user information,ℋ\\mathcal\{H\}is dialogue history,Q1Q\_\{1\}is the current request,AAis the assistant response, andQ2Q\_\{2\}is the user’s subsequent utterance or behavioral feedback\. If an answer skill changesAAintoA′A^\{\\prime\}, the originalQ2Q\_\{2\}is no longer the reaction that would necessarily follow\. Consequently, scoringA′A^\{\\prime\}against the loggedQ2Q\_\{2\}conflates factual observation with an unobserved counterfactual\.

This creates a practical obstacle for dialogue skill evolution\. Without a fixed offline objective, each candidate answer skill would ideally require human judgment or online A/B traffic\. Iterative deployment is slow, expensive, and risky; moreover, it provides no clean validation gate indicating whether textual self\-evolution has converged\.

We introduce a change of learning target\. Instead of first evolving a skill that tells the assistant*how to answer*, we evolve a feedback\-prediction skill that estimates*how a user will respond to an already observed answer*\. Given\(𝒞,ℋ,Q1,A\)\(\\mathcal\{C\},\\mathcal\{H\},Q\_\{1\},A\), the predictor estimates whether the next signal indicates resolved/accepted or unresolved/rejected\. Unlike counterfactual answer evaluation, this task can be measured against a fixed logged label\. Failed predictions are summarized into bounded additions, deletions, or replacements in the feedback skill; candidates are accepted only when they improve held\-out performance\.

The resulting feedback skill is useful beyond classification\. To predict whether a user will accept an answer, it must encode operational distinctions between superficially plausible responses and genuinely resolving ones: tool success is not user adoption, a weak acknowledgment is not acceptance, a generic suggestion is not an actionable plan, and a response must cover the user’s independent decision points\. These rationales expose concrete directions for improving the answer skill\.

Our contributions are:

- •We formulate the moving\-target problem that prevents direct reuse of logged next\-turn feedback for offline evolution of open\-ended answer skills\.
- •We propose validation\-gated future\-feedback skill evolution, turning logged human and behavioral signals into a fixed, verifiable objective for textual skill optimization\.
- •We identify a feedback–generation duality: the interpretable rules required to predict dissatisfaction also provide actionable diagnostics for answer generation\.
- •We report an industrial case study in which cleaned high\-quality data and a balanced resolved/unresolved evaluation produce over 75% accuracy, while explicitly separating observational prediction accuracy from counterfactual guarantees\.

![Refer to caption](https://arxiv.org/html/2607.18973v1/figure_concept.png)Figure 1:Verification differs for the two skills\. RevisingSAS\_\{A\}produces a new answerA′A^\{\\prime\}and an unknown subsequent signalQ2′Q\_\{2\}^\{\\prime\}, so loggedQ2Q\_\{2\}cannot validate the candidate\. RevisingSQS\_\{Q\}keeps\(Q1,A,Q2\)\(Q\_\{1\},A,Q\_\{2\}\)fixed, allowing every prediction to be compared with observedQ2Q\_\{2\}\.Figure[1](https://arxiv.org/html/2607.18973#S1.F1)makes the claim boundary explicit\. Held\-out logged data verifies evolution of the feedback\-prediction skill itself\. Its learned criteria can guide answer\-skill edits, but the resulting counterfactual answers require fresh human judgments or online interactions; the original logged reaction is not a valid label for them\.

## 2Related Work

#### Textual skill and prompt optimization\.

Textual artifacts can serve as external, editable state for frozen models\. SkillOpt applies optimizer\-style discipline to this space through bounded edits, held\-out validation, and rejection of non\-improving candidates\(Yanget al\.,[2026](https://arxiv.org/html/2607.18973#bib.bib1)\)\. Our work adopts this validation\-gated perspective but addresses a domain in which the validation target itself changes when the generated response changes\.

#### User satisfaction estimation\.

User satisfaction has long been studied as an automatic evaluation signal for dialogue systems\. The USS benchmark provides turn\- and dialogue\-level satisfaction annotations across multiple domains\(Sunet al\.,[2021](https://arxiv.org/html/2607.18973#bib.bib4)\)\. SPUR uses supervised iterative prompting to learn interpretable satisfaction rubrics from labeled examples\(Linet al\.,[2024](https://arxiv.org/html/2607.18973#bib.bib2)\)\. These studies establish that satisfaction\-related patterns can be learned from dialogue\. Our focus is not merely estimating satisfaction; it is using a verifiable feedback\-prediction task as the optimization substrate for self\-evolving textual skills\.

#### Predicting future dissatisfaction\.

See and Manning \([2021](https://arxiv.org/html/2607.18973#bib.bib3)\)predict whether a user will express dissatisfaction in the next turn and use that signal to rank chatbot responses\. This is closely related to our predictive target\. We differ in treating the predictor’s textual policy itself as the object of validation\-gated evolution and in extracting its learned failure rationales as reusable skill knowledge\.

#### Learned rewards and their limits\.

Human\-feedback learning uses learned preference or reward models to optimize model behavior\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.18973#bib.bib6)\)\. Such models may fail under distribution shift or be exploited by optimization\. Our proposal is deliberately narrower: the logged\-data objective verifies the evolution of the feedback skill on observed interactions\. It does not, by itself, prove that arbitrary answers optimized against that skill will satisfy users\. Final deployment still requires human or online confirmation\.

## 3Problem Formulation

### 3\.1Logged conversational feedback

Let

D=\{\(Xi,Ai,Yi\)\}i=1N,Xi=\(𝒞i,ℋi,Q1,i\),D=\\\{\(X\_\{i\},A\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{N\},\\quad X\_\{i\}=\(\\mathcal\{C\}\_\{i\},\\mathcal\{H\}\_\{i\},Q\_\{1,i\}\),whereYi∈\{0,1\}Y\_\{i\}\\in\\\{0,1\\\}is derived from the subsequent user utterance or a behavioral signal\. In our application,Y=1Y=1means adopted or resolved andY=0Y=0means unresolved\.

An answer skill𝒮A\\mathcal\{S\}\_\{A\}induces a response distribution

A∼pθ​\(A∣X,𝒮A\),A\\sim p\_\{\\theta\}\(A\\mid X,\\mathcal\{S\}\_\{A\}\),for a frozen target model with parametersθ\\theta\. The true utility of a new answer depends on an unknown reaction:

U​\(A′\)=𝔼​\[Y′∣X,A′\]\.U\(A^\{\\prime\}\)=\\mathbb\{E\}\[Y^\{\\prime\}\\mid X,A^\{\\prime\}\]\.For a logged record containingAA, onlyYYis observed\. ReplacingAAwithA′A^\{\\prime\}does not preserve the label because generally

p​\(Y∣X,A\)≠p​\(Y∣X,A′\)\.p\(Y\\mid X,A\)\\neq p\(Y\\mid X,A^\{\\prime\}\)\.Thus, a fixed dataset cannot directly provide an unbiased validation score for arbitrary revisions of𝒮A\\mathcal\{S\}\_\{A\}\.

### 3\.2A verifiable surrogate task

We instead optimize a feedback skill𝒮F\\mathcal\{S\}\_\{F\}that predicts the observed label:

Y^=fθ​\(X,A;𝒮F\)\.\\hat\{Y\}=f\_\{\\theta\}\(X,A;\\mathcal\{S\}\_\{F\}\)\.For every candidate𝒮F′\\mathcal\{S\}\_\{F\}^\{\\prime\}, the held\-out score

J​\(𝒮F′\)=1\|Dval\|​∑\(X,A,Y\)∈Dval𝟙​\[fθ​\(X,A;𝒮F′\)=Y\]J\(\\mathcal\{S\}\_\{F\}^\{\\prime\}\)=\\frac\{1\}\{\|D\_\{\\mathrm\{val\}\}\|\}\\sum\_\{\(X,A,Y\)\\in D\_\{\\mathrm\{val\}\}\}\\mathbb\{1\}\[f\_\{\\theta\}\(X,A;\\mathcal\{S\}\_\{F\}^\{\\prime\}\)=Y\]is measurable without changing the interaction\. Other deployment\-relevant metrics, such as asymmetric costs for false positives, may be included inJJ\.

This shift does not make the feedback model a perfect counterfactual oracle\. It does make*its own evolution*reproducible and rejectable on fixed data, which is the property needed for disciplined textual optimization\.

## 4Method

### 4\.1Overview

Figure[2](https://arxiv.org/html/2607.18973#S4.F2)contrasts direct answer\-skill evolution with our approach\. Direct evolution tries to reuse a reaction attached to the old answer\. Future\-feedback evolution holds the logged answer fixed, learns a validated prediction skill, and then uses its interpretable criteria to diagnose answer quality\.

![Refer to caption](https://arxiv.org/html/2607.18973v1/figure_overview.png)Figure 2:Formal comparison with a fixed log\. EvolvingSAS\_\{A\}changesAAand leaves the subsequentQ2′Q\_\{2\}^\{\\prime\}unknown\. EvolvingSQS\_\{Q\}predicts the recordedQ2Q\_\{2\}, so every edit remains testable on the same held\-out data\.
### 4\.2Feedback labels and data construction

Signals may originate from explicit user feedback, textual acceptance or rejection, and product\-side behavioral events\. The implementation maps available evidence into a binary resolved/unresolved label\. Silent termination is excluded when its meaning is ambiguous; this avoids noisy labels but introduces selection bias, discussed in Section[7](https://arxiv.org/html/2607.18973#S7)\.

Data quality is essential\. Raw production logs contain incomplete turns, proactive outreach that is not a response to a user problem, duplicated or malformed records, weakly grounded labels, and missing conversational context\. We apply quality filters so that each retained example has a clear turn boundary, a usable response, and a high\-confidence subsequent label\. We then construct train and test partitions with an approximately1:11\{:\}1resolved/unresolved ratio\. This prevents majority\-class accuracy from obscuring whether the skill learns both outcomes\.

### 4\.3Validation\-gated skill evolution

Starting from an initial feedback skill𝒮F\(0\)\\mathcal\{S\}\_\{F\}^\{\(0\)\}, the predictor produces a label and rationale for each training example\. At iterationtt, an optimizer model receives high\-value failures—especially confident false positives—and proposes a bounded textual editΔ\(t\)\\Delta^\{\(t\)\}:

𝒮F~\(t\+1\)=Edit⁡\(𝒮F\(t\),Δ\(t\)\)\.\\tilde\{\\mathcal\{S\}\_\{F\}\}^\{\(t\+1\)\}=\\operatorname\{Edit\}\(\\mathcal\{S\}\_\{F\}^\{\(t\)\},\\Delta^\{\(t\)\}\)\.Both the current and candidate skills are evaluated on the held\-out set\. The candidate is accepted only if it improves the primary validation objective; otherwise it is rejected:

𝒮F\(t\+1\)=\{𝒮F~\(t\+1\),J​\(𝒮F~\(t\+1\)\)\>J​\(𝒮F\(t\)\),𝒮F\(t\),otherwise\.\\mathcal\{S\}\_\{F\}^\{\(t\+1\)\}=\\begin\{cases\}\\tilde\{\\mathcal\{S\}\_\{F\}\}^\{\(t\+1\)\},&J\(\\tilde\{\\mathcal\{S\}\_\{F\}\}^\{\(t\+1\)\}\)\>J\(\\mathcal\{S\}\_\{F\}^\{\(t\)\}\),\\\\ \\mathcal\{S\}\_\{F\}^\{\(t\)\},&\\text\{otherwise\.\}\\end\{cases\}A lexicographic variant can retain equal\-accuracy candidates only when an asymmetric business utility improves\. Penalizing false positives more strongly is useful when declaring a user satisfied despite an unresolved issue is costlier than conservatively predicting unresolved\.

### 4\.4From prediction rationales to answer guidance

The evolved skill learns reusable distinctions\. Representative rules include:

- •successful tool execution or message delivery does not imply that the user adopted the result;
- •greetings, product recommendations, reports, and calls to action do not by themselves resolve a user need;
- •weak acknowledgments such as “OK” may indicate receipt rather than acceptance;
- •parameter advice should specify an action and at least one operational anchor, such as a value, interface, time window, or re\-evaluation condition;
- •the response should cover the independent decision points in the user’s latest request\.

These rules can be transformed into diagnostics for𝒮A\\mathcal\{S\}\_\{A\}\. For example, a response recommending that a user “observe for several days and adjust the parameter appropriately” appears helpful but lacks a target value and a trigger for further adjustment\. The feedback skill predicts unresolved and explains the missing actionability; the explanation can then be distilled into an answer\-skill instruction\. In this paper, we establish this bridge conceptually and through observed learned rules, but do not claim a separately measured end\-to\-end improvement of𝒮A\\mathcal\{S\}\_\{A\}\.

## 5Industrial Case Study

### 5\.1Setting

We evaluate the method on privacy\-preserving interactions from a production sales\-assistant scenario\. Inputs include available context, dialogue history, tool\-use summaries, and the assistant’s current response\. Labels indicate whether subsequent user or behavioral evidence supports resolved/adopted versus unresolved/not adopted\. All examples and operational identifiers are anonymized, and no raw dialogue is released\.

The latest evaluation uses a curated high\-quality subset rather than the earlier unfiltered extraction\. Train and test sets are precisely partitioned and balanced to approximately equal positive and negative proportions\. Because this paper emphasizes the formulation and the data are proprietary, we report the verified aggregate result without disclosing sensitive counts or traffic statistics\.

### 5\.2Result

On the balanced held\-out set, the evolved feedback skill achievesmore than 75% accuracy\. A random or constant classifier on this construction has approximately 50% accuracy, so the result indicates that the skill extracts meaningful interaction\-level signals rather than exploiting label prevalence\.

Two engineering choices were decisive\. First, data cleaning removed low\-quality and weakly labeled interactions whose conversational relation was ambiguous\. Second, precise partitioning produced a balanced test set, making accuracy directly interpretable across resolved and unresolved cases\. We treat these as part of task construction rather than as a novel modeling contribution: a verifiable objective is useful only when the labels and partitions faithfully represent the behavior to be predicted\.

The learned rules also provide qualitative evidence of convergence toward user\-centered criteria\. The predictor becomes less likely to equate fluent sales language, completed workflows, or successful tools with actual resolution, and more likely to require coverage, specificity, and an actionable next step\.

### 5\.3What the result establishes

The experiment supports three claims\. First, next\-feedback prediction is learnable from cleaned conversational records\. Second, textual skill edits can be evaluated and gated on fixed held\-out data\. Third, the evolved artifact is interpretable enough to expose recurring answer defects\.

The result does*not*establish that 75% observational accuracy is sufficient to optimize arbitrary counterfactual answers safely\. When the answer distribution moves, the predictor may also shift in accuracy\. Our method therefore reduces the need for repeated online testing during skill search, but does not eliminate a final human preference study or controlled online validation for a deployed answer skill\.

## 6Discussion

### 6\.1Why predict feedback instead of directly judging answers?

A direct LLM judge can assign a score to a new answer, but its score may encode generic preferences rather than the target users’ actual behavior\. Future\-feedback prediction ties the skill to observed human or behavioral outcomes\. It also produces an ordinary supervised validation loop: every prediction can be compared with a fixed label, and every textual edit can be accepted or rejected reproducibly\.

### 6\.2The feedback–generation duality

Prediction and generation are not identical, but they share a quality model\. A predictor cannot reliably forecast dissatisfaction without representing omissions, ambiguity, non\-actionability, unsupported claims, and mismatches with user intent\. This creates a useful duality:

predictive failure criteria⟹candidate generation constraints\.\\text\{predictive failure criteria\}\\quad\\Longrightarrow\\quad\\text\{candidate generation constraints\}\.The feedback skill is therefore both an evaluator and a repository of response\-quality knowledge\. Future work can automate the compilation of its rationales into answer\-skill edits and validate those edits with a one\-time counterfactual human study\.

### 6\.3Operational value

The framework changes the role of online evaluation\. Instead of exposing every candidate skill to users, developers can perform most iterations offline, reject non\-improving candidates, inspect learned rules, and deploy only the final candidate for controlled confirmation\. This is especially valuable in customer\-facing systems where careless exploration can degrade trust\.

## 7Limitations

#### Counterfactual distribution shift\.

The predictor is validated on logged answers\. Optimizing an answer skill may generate out\-of\-distribution responses that exploit predictor weaknesses\. The feedback skill should therefore be viewed as a validated observational model and diagnostic oracle, not a proof of human satisfaction for arbitraryA′A^\{\\prime\}\.

#### Silent users\.

Interactions with no interpretable subsequent signal are excluded\. Some may represent silent dissatisfaction, so the resulting labels cover explicit or behaviorally observable resolution more reliably than total user satisfaction\.

#### Data curation and reproducibility\.

The strongest reported result depends on proprietary cleaning rules and a private production dataset\. This is common in industrial dialogue research but limits external reproduction\. Public satisfaction datasets such as USS could test cross\-domain generality in future work\(Sunet al\.,[2021](https://arxiv.org/html/2607.18973#bib.bib4)\)\.

#### Binary feedback\.

Resolved/unresolved labels simplify graded, personalized satisfaction\. A user can accept an answer while remaining partly dissatisfied, or reject a correct answer for reasons outside response quality\.

#### No additional answer\-skill experiment\.

Consistent with the scope of this study, we do not report a new human evaluation or online A/B test of an answer skill distilled from the feedback predictor\. We therefore present rationale\-to\-guidance transfer as a supported mechanism and research direction, not as a completed causal result\.

## 8Conclusion

Open\-ended dialogue breaks the fixed\-target assumption underlying validation\-gated skill evolution: changing an answer changes the feedback that would follow\. We proposed future\-feedback skill evolution, which makes the first optimization target the prediction of logged user feedback rather than the direct generation of a counterfactual answer\. This yields a fixed offline objective for evolving an interpretable feedback skill\. In a curated and balanced industrial dataset, the method achieves over 75% held\-out accuracy and learns criteria that distinguish superficial completion from genuine resolution\. The broader lesson is that verifiability in open dialogue can be recovered by choosing a prediction target whose outcome has already been observed\. This does not remove the need for final human validation, but it makes the path to that validation more controlled, inspectable, and reproducible\.

## References

- PAL: program\-aided language models\.Proceedings of the 40th International Conference on Machine Learning\.Cited by:[§1](https://arxiv.org/html/2607.18973#S1.p2.11)\.
- Y\. Lin, J\. Neville, J\. W\. Stokes, L\. Yang, T\. Safavi, M\. Wan, S\. Counts, S\. Suri, R\. Andersen, X\. Xu,et al\.\(2024\)Interpretable user satisfaction estimation for conversational systems with large language models\.arXiv preprint arXiv:2403\.12388\.Cited by:[§2](https://arxiv.org/html/2607.18973#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems35,pp\. 27730–27744\.Cited by:[§2](https://arxiv.org/html/2607.18973#S2.SS0.SSS0.Px4.p1.1)\.
- A\. See and C\. Manning \(2021\)Understanding and predicting user dissatisfaction in a neural generative chatbot\.InProceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue,pp\. 1–12\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.sigdial-1.1)Cited by:[§2](https://arxiv.org/html/2607.18973#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Sun, S\. Zhang, K\. Balog, Z\. Ren, P\. Ren, Z\. Chen, and M\. de Rijke \(2021\)Simulating user satisfaction for the evaluation of task\-oriented dialogue systems\.InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 2499–2506\.External Links:[Document](https://dx.doi.org/10.1145/3404835.3463241)Cited by:[§2](https://arxiv.org/html/2607.18973#S2.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2607.18973#S7.SS0.SSS0.Px3.p1.1)\.
- Y\. Yang, Z\. Gong, W\. Huang, Q\. Yang, Z\. Zhou, Z\. Huang, Y\. Li, X\. Gao, Q\. Dai, B\. Liu,et al\.\(2026\)SkillOpt: executive strategy for self\-evolving agent skills\.arXiv preprint arXiv:2605\.23904\.Cited by:[§1](https://arxiv.org/html/2607.18973#S1.p1.1),[§2](https://arxiv.org/html/2607.18973#S2.SS0.SSS0.Px1.p1.1)\.