Can LLMs achieve Recursive Self-Improvement (RSI)?

The highly-cited STaR paper proved models can bootstrap their own reasoning. By generating rationales and filtering via correctness, the model trains itself.

A core classic for RSI! 🧠

https://t.co/biI1ZgrCth https://t.co/MEbW7Db4j6

---

Self-Taught Reasoner Bootstrapping Reasoning With Reasoning

Source: https://arxiv.org/html/2203.14465 **footnotetext:These authors contributed equally to this workYuhuai WuEmail:[email protected]Jesse MuAffiliation:Department of Computer Science, Stanford UniversityEmail:[email protected]Noah D. GoodmanAffiliation:Department of Computer Science, Stanford UniversityEmail:[email protected]Affiliation:Google Research

Abstract

Generating step-by-step “chain-of-thought” rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering. However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracy by using only few-shot inference. We propose a technique to iteratively leverage a small number of rationale examples and a large dataset without rationales, to bootstrap the ability to perform successively more complex reasoning. This technique, the “Self-Taught Reasoner” (STaR), relies on a simple loop: generate rationales to answer many questions, prompted with a few rationale examples; if the generated answers are wrong, try again to generate a rationale given the correct answer; fine-tune on all the rationales that ultimately yielded correct answers; repeat. We show that STaR significantly improves performance on multiple datasets compared to a model fine-tuned to directly predict final answers, and performs comparably to fine-tuning a 30×\timeslarger state-of-the-art language model on CommensenseQA. Thus, STaR lets a model improve itself by learning from its own generated reasoning.

1Introduction

Human decision-making is often the result of extended chains of thought[1,2]. Recent work has shown that explicit intermediate reasoning (“rationales”) can improve large language model (LLM) performance as well[3,4,5,6,7,8]. For example,5demonstrated that LLMs explicitly trained to use “scratchpads” for intermediate steps can attain perfect in-distribution performance on arithmetic, and strong out-of-distribution generalization, while models trained to predict answers directly fail to do either. These works suggest that generating explicit rationales before giving a final answer (“rationale generation”) is valuable for LLMs across diverse tasks including mathematical reasoning, commonsense reasoning, code evaluation, social bias inference, and natural language inference. However, the two primary methods for inducing rationale generation both have serious drawbacks.

One approach to rationale generation is the construction of a fine-tuning dataset of rationales, either manually by human annotators or automatically with hand-crafted templates[3,4,5,9]. Manual methods are expensive, and it is infeasible to construct such a dataset for each interesting problem[3]. Meanwhile, template-based methods rely on automatically-generated rationales but only work when a general solution is already known[5]or reasonable hard-coded heuristics can be made[4].

An alternative is to leverage in-context learning by including only a few rationale examples in the language model prompt. This has been shown to improve accuracy on mathematical and symbolic reasoning tasks relative to prompting without rationales (“direct” prompting)[5,6]. Yet, while few-shot techniques with rationales tend to outperform their non-reasoning counterparts, they generally substantially underperform models fine-tuned to directly predict answers using larger datasets[5,6].

Q: What can be usedto carry a small dog?Answer Choices:(a) swimming pool(b) basket(c) dog show(d) backyard(e) own homeA:The answer must besomething that can beused to carry a smalldog. Baskets aredesigned to hold things.Therefore, the answeris basket(b).

Figure 1:An overview of STaR and a STaR-generated rationale on CommonsenseQA. We indicate the fine-tuning outer loop with a dashed line. Thequestionsand ground truthanswersare expected to be present in the dataset, while therationalesare generated using STaR.In this paper, we adopt a different approach: by leveraging the LLM’s pre-existing reasoning ability, we iterativelybootstrapthe ability to generate high-quality rationales. Specifically, we few-shot prompt a large language model to self-generate rationales and refine the model’s ability further by fine-tuning on those rationales that lead to correct answers. We repeat this procedure, using the improved model to generate the next training set each time. This is a synergistic process, where improvements in rationale generation improve the training data, and improvements in training data further improve rationale generation.

However, we find this loop eventually fails to solve any new problems in the training set because it receives no direct training signal for problems it fails to solve. To overcome this issue, we proposerationalization: for each problem that the model fails to answer correctly, we generate a new rationale by providing the model with the correct answer. This lets the model reason backward—given the correct answer, the model can more easily generate a useful rationale. These rationales are then collected as part of the training data, which often improves overall accuracy.

We thus develop the Self-Taught Reasoner (STaR, Fig.1) method, a scalable bootstrapping method allowing models to learn to generate their own rationales, while also learning to solve increasingly difficult problems. In our method, we repeat the following process: in each iteration, first construct a finetuning dataset by attempting to solve the dataset using the current model’srationale generationability; then, augment this dataset usingrationalization, justifying ground-truth answers to problems the model failed to solve; finally, finetune the large language model on the combined dataset.

Applying STaR on arithmetic, math word problems, and commonsense reasoning, we observe it is able to effectively translate a small number of few-shot prompts into a large rationale dataset, yielding dramatic performance improvements. On CommonsenseQA[10], we find STaR improves over both a few-shot baseline (+35.9%) and a baseline fine-tuned to directly predict answers (+12.5%) , and performs comparably to a fine-tuned model that is 30×\timeslarger (72.5% vs. 73.0%).

Thus, we make the following contributions:

1. 1.We propose a bootstrapping mechanism to iteratively generate a rationale dataset from a few initial examples with rationales—without needing to check new rationales’ correctness.
2. 2.We complementrationale generationwithrationalization, where a model is tasked with justifying an answer and then fine-tuned as if it had come up with the rationale without any hint. We show rationalization accelerates and improves the bootstrapping process.
3. 3.We evaluate these techniques with a variety of ablations in both mathematical and commonsense reasoning domains.
4. 4.We propose what is, to our knowledge, the first technique to allow a pre-trained large language model to iteratively use its language modeling capacity to improve itself.

2Background and Related Work

In-context Learning

Recently, a collection of works has emerged exploring the capacity for large language models to perform in-context learning[11,12]. In essence, in-context learning treats few-shot learning as a language modeling problem, by showing a few examples in the context (i.e. prompt), and allowing the model to learn and identify the pattern to apply to new examples. Some have studied in-context learning based on the language modeling objective in terms of Bayesian inference13while others have attempted to describe the process more mechanistically in terms of “induction heads”[14]. Moreover, differences in prompt configurations have been known to have dramatic effects on few-shot performance. Some have even found that replacing few-shot prompts with a “soft prompt” which can be optimized in embedding space results in noticeable gains[15]. Instead of emphasizing the representation of the question, we focus on the model output; in particular, we focus on the model’s ability to reason through a problem before coming to a conclusion.

Rationales

One of the initial works on the impact of rationales on language model performance was3, showing that training a language model on a dataset with explicit rationales preceding the answer could improve a model’s ability to generate the final answer. However, this required many thousands of training examples to be manually annotated with human reasoning. Recently,5demonstrated that step-by-step “scratchpads” can improve fine-tuned LLM performance and generalization on tasks such as arithmetic, polynomial evaluation, and program evaluation. Similarly,6used a single few-shot “chain-of-thought” reasoning prompt in order to improve model performance on a collection of tasks, without fine-tuning. Finally,16showed that a curriculum learning approach could help solve formal math problems, as long as 1) they were translated into Lean (a theorem-proving language[17]), 2) one could directly evaluate the validity of the proofs, 3) one could sample numerous potential solutions for each problem, 4) had trained a separate value function model, and 5) started with GPT-f (a model already fine-tuned on a large math dataset[18]). We note that there are many domains where these conditions do not all apply. In addition, works have aimed to explain why rationales have this beneficial effect: some have analyzed their impact from the perspective of latent variable models[19]while others have provided formal proofs of the benefit of intermediate task supervision[20].

Iterated Learning

A variety of iterated learning algorithms have been proposed, where solutions or successful methods which are found are in turn used to find additional solutions[21,22,16].21introduced Expert Iteration (ExIt), a reinforcement learning technique serving as an inspiration for our approach. Essentially, it consists of a loop of self-play by an “apprentice,” followed by imitation learning with feedback from a slower “expert” and then the replacement of the expert with the now-improved apprentice.16builds off of ExIt for formal reasoning, while22applies iterated learning to visual question answering using modular networks which can be combined compositionally. There are further similarities between STaR and expert iteration methods[21]. For example, filtering generated examples based on whether their ultimate answer matches the target can be seen as expert feedback. However, we have a fixed “expert” and do not train a separate value function.

Natural Language Explanations

Natural language explanations have also been discussed from the perspective of explainable machine learning, focusing on justification rather than reasoning[23,24]. The motivation for this line of work is largely grounded in explainable decision making, and similarly to3, generally does not find that requiring post-hoc explanations improves model performance.

3Method

3.1Rationale Generation Bootstrapping (STaR Without Rationalization)

We are given a pretrained LLMMMand an initial dataset of problemsxxwith answersyy:𝒟={(xi,yi)}i=1D\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{D}. Our technique starts with a smallpromptset𝒫\mathcal{P}of examples with intermediaterationalesrr:𝒫={(xip,rip,yip)}i=1P\mathcal{P}=\{(x^{p}_{i},r^{p}_{i},y^{p}_{i})\}_{i=1}^{P}, whereP≪DP\ll D(e.g.P=10P=10). Like standard few-shot prompting, we concatenate this prompt set to each example in𝒟\mathcal{D}, i.e.xi=(x1p,r1p,y1p,…,xPp,rPp,yPp,xi)x_{i}=(x^{p}_{1},r^{p}_{1},y^{p}_{1},\dots,x^{p}_{P},r^{p}_{P},y^{p}_{P},x_{i}), which encourages the model to produce a rationaler^i\hat{r}_{i}forxix_{i}followed by an answery^i\hat{y}_{i}. We assume that rationales that lead to correct answers are of better quality than those that lead to incorrect answers. Therefore, we filter the generated rationales to include only the ones which result in the correct answer (y^i=yi\hat{y}_{i}=y_{i}). We fine-tune the base modelMMon this filtered dataset, and then restart this process by generating the new rationales with the newly fine-tuned model. We keep repeating this process until the performance plateaus. Note that during this process, once we collect a new dataset, we train from the original pre-trained modelMMinstead of continually training one model to avoid overfitting. We provide an outline of this algorithm in Algorithm1.

STaR can be seen as an approximation to an RL-style policy gradient objective. To see this, note thatMMcan be viewed as a discrete latent variable modelpM​(y∣x)=∑rp⁡(r∣x)​p​(y∣x,r)p_{M}(y\mid x)=\sum_{r}p(r\mid x)p(y\mid x,r); in other words,MMfirst samples a latent rationalerrbefore predictingyy. Now, given the indicator reward function𝟙​(y^=y)\mathbbm{1}(\hat{y}=y), the total expected reward across the dataset is

J⁡(M,X,Y)\displaystyle J(M,X,Y)=∑i𝔼r^i,y^i∼pM(⋅∣xi)𝟙(y^i=yi),\displaystyle=\sum_{i}\mathbb{E}_{\hat{r}_{i},\hat{y}_{i}\sim p_{M}(\cdot\mid x_{i})}\mathbbm{1}(\hat{y}_{i}=y_{i}),(1)∇J​(M,X,Y)\displaystyle\nabla J(M,X,Y)=∑i𝔼r^i,y^i∼pM(⋅∣xi)[𝟙(y^i=yi)⋅∇logpM(y^i,r^i∣xi)],\displaystyle=\sum_{i}\mathbb{E}_{\hat{r}_{i},\hat{y}_{i}\sim p_{M}(\cdot\mid x_{i})}\left[\mathbbm{1}(\hat{y}_{i}=y_{i})\cdot\nabla\log p_{M}(\hat{y}_{i},\hat{r}_{i}\mid x_{i})\right],(2)where the gradient is obtained via the standard log-derivative trick for policy gradients. Note that the indicator function discards the gradient for all sampled rationales that do not lead to the correct answeryiy_{i}: this is the filtering process in STaR (Line 5). Thus, STaR approximatesJJby (1) greedily decoding samples of(r^i,y^i)(\hat{r}_{i},\hat{y}_{i})to reduce variance of this estimate (at the cost of potentially biased exploration of rationales), and (2) taking multiple gradient steps on the same batch of data (similar to some policy gradient algorithms[25]). These approximations make STaR a simple and broadly applicable method that can be implemented with standard LLM training machinery; future work should more closely investigate the link between STaR and the RL objective above.

3.2Rationalization

Q:Wheredoyouputyourgrapesjustbeforecheckingout?

AnswerChoices:

(a)mouth

(b)grocerycart(CORRECT)

(c)supermarket

(d)fruitbasket

(e)fruitmarket

A:Theanswershouldbetheplacewheregroceryitemsareplacedbeforecheckingout.Oftheabovechoices,grocerycartmakesthemostsenseforholdinggroceryitems.Therefore,theanswerisgrocerycart(b).

Figure 2:A few-shot prompt hint we use for rationalization (and not for rationale generation), using the rationale from[6], with its hint included ingreen, followed by the rationale and the answer generated by the model.The rationale generation bootstrapping algorithm carries a limitation. Since the model is only trained on the examples which it answers correctly, improvement ends when the model fails to solve new problems in the training set. This is fundamentally due to the fact that the algorithm cannot obtain any training signal from failed examples. Inspired by3, we propose a technique we call “rationalization”. Specifically, we provide the answer as a hint to the model and ask it to generate rationales in the same style as in the previous rationale generation step. Given the answer, the model is able to reason backwards, and hence more easily generate a rationale that leads to the correct answer. For example, in Figure2, we provide the hint that ”(b) grocery cart” is the correct answer in the prompt to generate the rationale. We apply rationalization to the problems which the model failed to solve with rationale generation. When adding a rationalization-generated rationale to our dataset, we do not include the hint in its corresponding prompt, as if the model had come up with the rationale without the hint. After filtering, we fine-tune on the previously generated dataset combined with the rationalization-generated dataset.

Algorithm 1STaRInputMM: a pretrained LLM; dataset𝒟={(xi,yi)}i=1D\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{D}(w/ few-shot prompts)

1:

M0←{M_{0}}\leftarrowM{M}# Copy the original model

2:for

nnin

1​…​N1...Ndo# Outer loop

3:

(r^i,y^i)←Mn−1​(xi)∀i∈[1,D](\hat{r}_{i},\hat{y}_{i})\leftarrow M_{n-1}(x_{i})\quad\forall i\in[1,D]# Perform rationale generation

4:

(r^irat,y^irat)←Mn−1​(add_hint​(xi,yi))∀i∈[1,D](\hat{r}^{\text{rat}}_{i},\hat{y}^{\text{rat}}_{i})\leftarrow{M_{n-1}}(\text{add\_hint}(x_{i},y_{i}))\quad\forall i\in[1,D]# Perform rationalization

5:

𝒟n←{(xi,r^i,yi)∣i∈[1,D]∧y^i=yi}\mathcal{D}_{n}\leftarrow\{(x_{i},\hat{r}_{i},y_{i})\mid i\in[1,D]\land\hat{y}_{i}=y_{i}\}# Filter rationales using ground truth answers

6:

𝒟nrat←{(xi,r^irat,yi)∣i∈[1,D]∧y^i≠yi∧y^irat=yi}\mathcal{D}^{\text{rat}}_{n}\leftarrow\{(x_{i},\hat{r}^{\text{rat}}_{i},y_{i})\mid i\in[1,D]\land\hat{y}_{i}\neq y_{i}\land\hat{y}^{\text{rat}}_{i}=y_{i}\}# Filter rationalized rationales

7:

Mn←train​(M,𝒟n∪𝒟nrat)M_{n}\leftarrow\text{train}(M,\mathcal{D}_{n}\,{\color[rgb]{0,0,1}\cup\,\mathcal{D}^{\text{rat}}_{n}})# Finetune the original model on correct solutions - inner loop

8:endfor

Algorithm1describes the full algorithm, with the parts inbluecorresponding to rationalization. Without those parts, Algorithm1corresponds to STaR without rationalization. Figure1provides an overview diagram. Fine-tuning on the dataset generated by rationalization has a crucial benefit of exposing the model to difficult problems which otherwise would not have appeared in its finetuning dataset. This can be understood as challenging the model to “think outside the box” about problems on which it was unsuccessful. A secondary benefit of rationalization is an increase in dataset size.

4Experiments

For our experiments, we focus on arithmetic, commonsense reasoning, and grade school math to demonstrate STaR’s breadth. In particular, for arithmetic, we follow a setup inspired by5. For commonsense question-answering we follow13,6and use CommonsenseQA (CQA), a widely used multiple-choice dataset for this domain[10]. For grade school math, we use GSM8K from9.

4.1Experimental Protocol

We used GPT-J as our base language model, and the fine-tuning script from the GPT-J repository[26]. We chose GPT-J, a 6B-parameter model, because the checkpoint and fine-tuning code are publicly available[26], and the model is large enough to generate rationales of non-trivial quality to be bootstrapped from. More hyperparameter details about GPT-J and our fine-tuning are included in AppendixH. Following the default setting of26, we perform a 100-step learning rate warmup, from which point we use a constant learning rate. Unless stated otherwise, we start with4040training steps at the first outer loop, and increase the number of fine-tuning training steps by20%20\%with each outer loop. In general, we found that training more slowly at the beginning ultimately benefits model performance. We expect that further improvement is possible via a thorough hyperparameter search—we leave this to future work due to computational constraints.

For arithmetic problems, we first generate a dataset of 50,000 randomly sampled questions (uniformly over the digit lengths) in the format introduced by5. For each outer loop iteration on arithmetic, we sample 10,000 problems from the dataset. We use 10 random few-shot rationale examples for each digit for its corresponding few-shot prompt. For each of the9,7419,741questions in the training set of CommonsenseQA, we add the question to the few-shot rationale prompt, and prompt the model to generate the rationale and answer for that question. For few shot prompting on CQA, we start with the same 10 questions as used in6, with the rationales modified slightly to fix an incorrect answer and to more explicitly reference relevant knowledge. We include these modified prompts in AppendixB111Based on27, this is unlikely to meaningfully affect[6]’s few-shot performance.. These prompts serve as our complete set of explanations. We run STaR until we see performance saturate, and we report the best results.

When performing rationalization, we find that the choice to include or omit few-shot prompts on outer-loop iterations after the first iteration does not have a substantial impact on the method’s ultimate performance. However, there are some nuances which we discuss further in Section5, leading us to use few-shot prompts unless stated otherwise.

4.2Datasets

Arithmetic

Input:
6 2 4 + 2 5 9
Target:
<scratch>
6 2 4 + 2 5 9 , C: 0
2 + 5 , 3  C: 1
6 + 2 , 8 3  C: 0
, 8 8 3  C: 0
0 8 8 3
</scratch>
8 8 3

Figure 3:A visualization of a 3-digit arithmetic problem with a scratchpad. C corresponds to the carry from the previous digit’s summation.The arithmetic task is to calculate the sum of twonn-digit integers. We generate the dataset based on the descriptions in5and visualize an example scratchpad in Figure3. Everything up to and including “Target:” is given as part of a prompt, and the model is asked to generate the scratchpad (start/end indicated by “<scratch>”) and the final answer, as in5. Each line of the scratchpad corresponds to the summation of each pair of digits from the final digit to the first digit, the accumulating final digits of the answer, and a carry digit corresponding to whether the previous pair summed to at least 10. We include few-shot prompts for 1 to 5 digits. When performing rationalization, we include the correct answer after “Target” and query the model to produce the scratchpad and then reproduce the correct answer following the scratchpad.

CommonsenseQA

The multiple-choice commonsense reasoning task, CommonsenseQA[10](CQA), is constructed from ConceptNet, a semantic graph of concepts and their relationships with over a million nodes[28].[10]identified a set of “target” concepts in ConceptNet for each question, where the target concepts share a semantic relationship to one “source” concept. Then each question is crowdsourced to allow a reader to identify one target concept, while mentioning the source concept. In addition, two distractor answers are added. The dataset has 12,247 questions, each with five choices, with 9,741 in the train set, 1,221 in the dev set, and 1,285 in the (withheld) test set.

Corresponding to the broad variety of ConceptNet, CQA contains a diverse set of questions which require commonsense reasoning ability building off of standard world knowledge, where human performance is 89%[10]. Many have pointed out that CQA contains a number of biases, along several dimensions including gender[3]. We discuss how this may impact our method in AppendixG. There are also many typos and questions which are fundamentally ambiguous222For example, “Billy bought coffee and waited for his wife to arrive from France. Where might he have been?” includes airport and train station as options. The correct answer, perhaps surprisingly, is train station.. We use it despite these issues as it is a general question-answering dataset relying on both common world knowledge and simple reasoning, which serves as a good test-bed for our method.

Grade School Math (GSM8K)

We also evaluate on the Grade School Math (GSM8K) dataset, which contains 7,473 train and 1,319 test examples of grade-school-level word problems[9]. These math problems are posed in natural language and require two to eight calculation steps to arrive at a final answer. This dataset combines the skills needed for arithmetic and commonsense reasoning.

4.3Symbolic Reasoning: Results on Arithmetic

The accuracies of the model across digits11-55over each iteration of the outer loop are plotted in Figure4. After running STaR for 16 iterations, the overall accuracy is89.5%89.5\%. For reference, a baseline trained on 10,000 examples without rationales for 5,000 steps attains76.3%76.3\%accuracy. Notably, few-shot accuracy on arithmetic problems is very low, even with rationales: accuracy on 2-digit addition is less than1%1\%, and accuracy on more digits close to zero.

(a)Without rationalization(b)With rationalization Figure 4:A visualization of the accuracy ofnn-digit summation with each iteration of STaR with and without rationalization for arithmetic. Each series corresponds to the accuracy of summing twonn-digit numbers.Figure 5:We introduce additional digits to STaR with rationalization at the 20thiteration.With rationalization, the accuracy is able to improve especially quickly. After one fine-tuning iteration on the model’s generated scratchpads, 2-digit addition improves to32%32\%from less than 1%. Without rationalization, the performance improvement is stage-wise: the model generally has poor performance on thenn-digit sum until it has good performance on the(n−1)(n-1)-digit sum. With rationalization, the model can learn many lengths at once, though not with equal accuracy. Rationalization allows many problems to be solved few-shot, so we start STaR training with 300 steps (note, doing so without rationalization causes overfitting on11-digit addition), and increase training by 20 steps per iteration.

We also perform an experiment where we continue pre-training STaR with rationalization with additional digits, starting before the 20th iteration, while keeping the total number of training examples fixed at each iteration. We find that not only does this appear to quickly improve performance on the initial set of digits, but when evaluated on 9 and 10 digit examples, never seen during training, the model successfully solves many of these out-of-distribution problems. As visualized in Figure5, the introduction of these digits appears to make the training less stable, but the exact cause is unclear.

4.4Natural Language Reasoning: Commonsense Question Answering

The CommonsesenseQA (CQA) setting introduces several new challenges. In the arithmetic task, an incorrect scratchpad in the reasoning step, and to a lesser degree in the rationalization step, was extremely likely to result in an incorrect answer. On the other hand, CQA problems are 5-way multiple choice questions. Thus, one will get the right answer at random approximately 20% of the time, regardless of the quality of reasoning. Moreover, some simple heuristics (e.g. semantic similarity) can meaningfully improve this to≈\approx30% without any reasoning, as shown by[10].

We evaluate this dataset as described in the experimental protocol and compare to several baselines. The first baseline is to finetune GPT-J to directly output the final answer, which we call “GPT-J Finetuned”. We also compare to GPT-3 finetuned to directly predict the final answer from29, and a 137B parameter Lambda model few-shot prompted with chain-of-thought (CoT) rationales from6.

We found that, as shown in Table1, STaR without rationalization outperformed GPT-J fine-tuned directly on the final answer for the entire dataset, despite training on less of the data. The inclusion of rationalization improved this performance to72.5%72.5\%, far closer to the73%73\%of the 30×\timeslarger GPT-3. As expected, we also see STaR surpassed the few-shot baselines, including the much-larger 137B LaMDA model[30,6]. We expect accuracy would be further improved if we applied STaR to a model with higher few-shot performance.

Case Study

Note that it is harder to judge the rationale quality: for arithmetic, one can compare them to the ground truth rationales, but for CQA the evaluation is necessarily qualitative. For this reason, we include a case study in Figure7. We observe that the rationales provided are generally coherent and of a similar structure to the few-shot rationales. We make the following two observations:

1. 1.After training with STaR, we see the model was able to generate reasonable rationales that solve new problems, which explains part of the observed performance gain.
2. 2.We also see that there were many instances in which STaR improved the quality of rationales over those generated in a few-shot manner.

Table 1:We evaluate several baselines, including a few-shot GPT-J evaluation both with and without scratchpads, a GPT-J baseline finetuned to directly predict the answer, and STaR with and without rationalization applied to GPT-J. We use CoT to denote non-STaR models outputting rationales, and Direct to indicate those directly predicting the final answer. Note the final STaR model is trained on 78.2% of the training dataset with rationale generation, and an additional 8.5% from rationalization.

Human Evaluation

Based on the observation that STaR may improve reasoning quality for problems even when they were initially answered correctly via few-shot prompting, we performed a preliminary qualitative analysis. We randomly selected 50 rationales generated from few-shot CoT and STaR-generated rationales on questions which they both answered correctly, as well as human-generated rationales for these problems from[3]. We then presented a random subset of 10 questions and rationales to each of 20 crowdworkers on Prolific[31]with the rationales in a randomized order, asking them to rank the rationales based on which they felt best justified the answer. The participants were 30% more likely to rank the STaR-generated rationales higher than the few-shot rationales (p=.039p=.039). This indicates that, as mentioned in the case study, STaR can improve the quality of rationale generation.

We also found that the participants were 74% more likely to prefer the STaR-generated rationales over the human-generated rationales (pp<.001.001). To be clear, we do not believe that this indicates human-level rationale-generation performance. Instead, we feel that it speaks to the difficulty of eliciting high-quality rationales. We reproduce the test prompts in AppendixCand elaborate on the limitations of the crowdsourced explanations dataset.

Failure Cases

Finally, we found a variety of interesting failure cases, many of which corresponded to standard logical fallacies. For example, the model often made statements related to the topic of the question but which were not actually arguments for why the answer should be true. Sometimes, the model claimed the question implied the answer as an argument, without explaining why. Other times, especially early in training, the model answered as if it has knowledge about a particular individual, instead of making a general statement - e.g. “the king’s castle is a place where he feels safe” instead of “castles are places where kings feel safe.” We provide examples and analyze errors in AppendixA.

Few-shot Prompt Training

Including few-shot prompts during fine-tuning[12]appears to have a meaningful performance benefit (60.9% to 68.8% without rationalization, 69.9% to 72.5% with rationalization). Thus, we generally suggest its use for at least some portion of the training, though we discuss some caveats in Section5.

Table 2:We find that STaR substantially improves GSM8K performance over the baselines, despite training on only 25.0% of the data for the model without rationalization, and 28.7% of the dataset (with 0.5% from rationalization) for the model with rationalization.Figure 6:A comparison of the number of calculator steps generated by the model in order to solve examples in the training set relative to the number of steps used in the ground truth.

4.5Mathematical Reasoning in Language: Grade School Math

We again find on GSM8K that STaR substantially improves performance beyond few-shot with rationales or training to directly predict the answers (without rationales), shown in Table2and include the few-shot prompt in AppendixI. We observe that on this task, the use of rationalization does not substantially improve performance. Note that, in training, it was necessary to cap the number of training steps at the 30th iterations (after 7912 steps), to prevent the training process from becoming prohibitively long. The results were reached after 36 iterations for STaR without rationalization and an additional 10 iterations with rationalization.

Most often, the number of calculation steps generated by the model matches the number of steps taken by humans (generally between 53% and 57% agreement across all iterations). We visualize this explicitly in Figure6. We see that when the ground truth and model disagree on the number of calculation steps, the model typically uses fewer. Sometimes this is because the model skips steps, but occasionally it finds different solutions. We show an example in AppendixJ, where the model disregards redundant information and solves a 7-step problem in a single step.

5Discussion and Challenges

The Impact of Rationalization

An essential question is exactly what role rationalization plays. Intuitively, rationalization allows a model to reverse-engineer a solution, or provides a heuristic for identifying whether each step makes the conclusion more likely. This parallels real-world problems where the final result is known, but challenging to derive a good justification. From a mathematical perspective, while rationale generation samples rationales from the distributionp⁡(r∣x)p(r\mid x)provided by our modelMM, rationalization conditions on the answer, letting us access an alternative distributionp⁡(r∣x,y)p(r\mid x,y)which may be a better search space for rationales. Then rationalization could be framed as an off-policy estimate of the objective in Equation1, sampling from the hint-augmented model as a proposal distribution. Future work should establish more connections between rationalization and these RL objectives, and examine more generally when and why rationalization improves learning.

In addition, due to the low sampling temperature, the outputs without rationalization correspond to the examples where the model is most confident in its answer. This results in these examples providing a weaker gradient signal than the rationalization examples, at least in the first iteration. Since we retrain from the initial pre-trained model every time we run a fine-tuning iteration, the degree of this effect is also difficult to measure directly. Finally, we must point out that the method to add the “hint” does not follow immediately from the question and answer and in some contexts providing it may be nontrivial. An exploration of the various impacts of different hinting techniques and their generality is an avenue for future work.

Temperature

One intuitive alternative to rationalization, if one seeks to expand the training dataset, is more and higher-temperature sampling. However, in practice, we found that this is counterproductive. In general, it substantially increases the likelihood of a correct answer despite incorrect reasoning, and training on bad or irrelevant reasoning prevents generalization. This is particularly clear in more structured tasks, like arithmetic, where the scratchpads that the model learns to produce with a higher-temperature sampling approach diverge into meaninglessness and cause the model to stagnate. Overall, we found that higher temperatures as an alternative to rationalization (e.g.0.50.5or0.70.7) consistently led to models worse than models with reasoning alone. In addition, as text generation by large language models is sequential (i.e. one cannot produce a token without producing the preceding token), generating text is a bottleneck and this is computationally far less efficient than rationalization. For example, generating 10 sample outputs is approximately 10 times slower than generating one sample output. However, one potentially valuable way to leverage multiple samples would be to use the method proposed in32, using the majority-vote result of multiple high-temperature scratchpads as a ground truth against which we compare a low-temperature scratchpad. This may allow one to apply STaR to a dataset of only questions, without answers.

Few-shot Prompting

A noteworthy phenomenon is that the inclusion of few-shot prompting during sampling seems to dramatically reduce “drift” where later rationales become increasingly dissimilar from the initial few-shot set of rationales. One benefit of this is that the model may be less constrained by the quality and difficulty of the initial rationales, theoretically allowing it to generalize more. One potentially negative consequence is that the style of the rationales may less-closely match the original prompting style. Another benefit is in terms of computational resources - a shorter prompt length allows for a shorter sequence length when sampling. Technically, the point in training at which we “disable” few-shot prompts is another hyperparameter which we could tune, but we leave this to future work. In addition, by leaving prompts out after the initial outer-loop iteration, the model tends to perform gradually worse at rationalization as it trains for longer periods of time. As a result, it may be necessary to include some hints during training for long periods of time with this approach.

Ultimately, the choice to include few-shot prompts in later iterations of training appears to depend on the use-case: when the goal is consistent adherence to a particular prompt style, which may benefit explainability, include few-shot prompts in sampling; when the goal is a faster training loop, one may remove them. Moreover, it is possible that with other datasets or larger models there is an impact on performance, so we encourage this to be generally treated as a hyperparameter.

6Conclusion

We present the Self-Taught Reasoner (STaR), which iteratively improves a model’s ability to generate rationales to solve problems. We few-shot prompt a model to solve many problems in a step-by-step manner by generating rationales, and then prompt it to rationalize the correct answer for problems it gets wrong. We finetune on both the initially correct solutions and rationalized correct solutions, and repeat the process. We find that this technique significantly improves the model’s generalization performance on both symbolic reasoning and natural language reasoning.

There are several important limitations on STaR as presented. In order for the first iteration of STaR to succeed, few-shot performance must be above chance, implying that the initial model must be big enough to have some reasoning capabilities. For instance we found that GPT-2 was not able to bootstrap from few-shot reasoning in even the arithmetic domain. A further limitation is that settings with a high level of chance performance (e.g. binary decisions) yield many poor rationales, confounding the STaR approach. An open problem is how to filter bad reasoning in these settings.

Nonetheless, we believe using examples without reasoning to bootstrap reasoning is a very general approach, and that STaR can serve as the basis of more sophisticated techniques across many domains.

Acknowledgements

We thank Imanol Schlag for his detailed feedback about this work, as well as Rose E Wang, Markus Rabe, Aitor Lewkowycz, Rishi Bommasani, Allen Nie, Alex Tamkin, and Qian Huang. We thank Cem Anil for his very helpful insight that rationale finetuning performance can be improved if the training includes the few-shot rationales. We also thank Ben Prystawski for his suggestions on survey creation. We thank Google TPU Research Cloud for TPU access.

References

• [1]William James, Frederick Burkhardt, Fredson Bowers, and Ignas K Skrupskelis.The principles of psychology, volume 1.Macmillan London, 1890.
• [2]K Anders Ericsson and Herbert A Simon.Protocol analysis: Verbal reports as data.the MIT Press, 1984.
• [3]Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher.Explain yourself! leveraging language models for commonsense reasoning.ACL, 2019.
• [4]Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.Unsupervised commonsense question answering with self-talk.EMNLP 2020, 2020.
• [5]Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al.Show your work: Scratchpads for intermediate computation with language models.arXiv preprint arXiv:2112.00114, 2021.
• [6]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.Chain of thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022.
• [7]Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E Peters.Few-shot self-rationalization with natural language prompts.arXiv preprint arXiv:2111.08284, 2021.
• [8]Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill.Can language models learn from explanations in context?arXiv preprint arXiv:2204.02329, 2022.
• [9]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021.
• [10]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.Commonsenseqa: A question answering challenge targeting commonsense knowledge.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, 2019.
• [11]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020.
• [12]Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.Finetuned language models are zero-shot learners.ICLR 2022, 2021.
• [13]Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma.An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021.
• [14]Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, and et al.In-context learning and induction heads.Transformer Circuits, Mar 2022.
• [15]Brian Lester, Rami Al-Rfou, and Noah Constant.The power of scale for parameter-efficient prompt tuning.EMNLP 2021, 2021.
• [16]Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever.Formal mathematics statement curriculum learning.arXiv preprint arXiv:2202.01344, 2022.
• [17]Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer.The lean theorem prover (system description).InInternational Conference on Automated Deduction, pages 378–388. Springer, 2015.
• [18]Stanislas Polu and Ilya Sutskever.Generative language modeling for automated theorem proving.arXiv preprint arXiv:2009.03393, 2020.
• [19]Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang.Towards interpretable natural language understanding with explanations as latent variables.Advances in Neural Information Processing Systems, 33:6803–6814, 2020.
• [20]Noam Wies, Yoav Levine, and Amnon Shashua.Sub-task decomposition enables learning in sequence to sequence tasks.arXiv preprint arXiv:2204.02892, 2022.
• [21]Thomas Anthony, Zheng Tian, and David Barber.Thinking fast and slow with deep learning and tree search.Advances in Neural Information Processing Systems, 30, 2017.
• [22]Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville.Iterated learning for emergent systematicity in vqa.ICLR 2021, 2021.
• [23]Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom.e-snli: Natural language inference with natural language explanations.Advances in Neural Information Processing Systems, 31, 2018.
• [24]Hanxiong Chen, Xu Chen, Shaoyun Shi, and Yongfeng Zhang.Generate natural language explanations for recommendation.arXiv preprint arXiv:2101.03392, 2021.
• [25]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017.
• [26]Ben Wang.Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX.https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
• [27]Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.Rethinking the role of demonstrations: What makes in-context learning work?arXiv preprint arXiv:2202.12837, 2022.
• [28]Robyn Speer, Joshua Chin, and Catherine Havasi.Conceptnet 5.5: An open multilingual graph of general knowledge. singh 2002 (2016).arXiv preprint arxiv:1612.03975, 2016.
• [29]Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang.Human parity on commonsenseqa: Augmenting self-attention with external attention.arXiv:2112.03254, December 2021.human parity result on CommonsenseQA.
• [30]Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.Lamda: Language models for dialog applications.arXiv preprint arXiv:2201.08239, 2022.
• [31]Stefan Palan and Christian Schitter.Prolific. ac—a subject pool for online experiments.Journal of Behavioral and Experimental Finance, 17:22–27, 2018.
• [32]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.Self-consistency improves chain of thought reasoning in language models, 2022.
• [33]Bernease Herman.The promise and peril of human evaluation for model interpretability.arXiv preprint arXiv:1711.07414, 2017.
• [34]Alon Jacovi and Yoav Goldberg.Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?ACL 2020, 2020.
• [35]Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020.
• [36]Diederik P Kingma and Jimmy Ba.Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.

Appendix

Appendix ACommonsenseQA Error Patterns

Throughout our experiments, we came across a variety of interesting failure cases for commonsense reasoning. Note that all the final answers are correct – however, we take issue with the reasoning used in order to arrive at those answers.

A.1Question Implies Answer

One key failure case was answers in the form of “the answer must be something that is <question property>. <answer> is <question property>. Therefore, the correct answer is <answer>.” In these cases, the model fails to explainwhythe answer that it has chosen satisfies the question property.

Q: What might someone get from learning about science?
Answer Choices:
(a) headache
(b) see things differently
(c) increased knowledge
(d) accidents
(e) appreciation of nature
A: The answer must be something that someone would get from learning about
science. Learning about science would increase knowledge. Therefore, the
answer is increased knowledge (c).

These rationales, while perhaps useful to the model, read to us as opaque and unexplanatory.

A.2Begging the Question

A related but stronger version of the previous failure case, while less common, is particularly uninsightful. Sometimes the model will imply the answer that it has chosen in its question.

Q: The christian rock band was on a large stage, the singer’s foot got caught
on a wire and he had a small what?
Answer Choices:
(a) commit sin
(b) fail
(c) disobey
(d) stumble
(e) boo boo
A: The answer must be something that would cause the singer to stumble. The
singer’s foot got caught on a wire, so he would stumble. Therefore, the
answer is stumble (d).

A.3Exercise to the Reader

A rare failure case is when the model finds it unnecessary to justify its answer. For example:

Q: Other than school or from text where can one gain knowledge?
Answer Choices:
(a) book
(b) class
(c) thinking
(d) head
(e) meeting
A: The answer must be a place where someone can gain knowledge other than
school or from text. The answer is meeting (e).

A.4World State Assertions

Sometimes, the model will assume that it knows something about a subject or a person whose name was used as a variable. This leads to somewhat comical examples of reasoning. Part of the reason for this is that generally, there is an expectation that good rationales will leverage understanding of more general classes of objects and appeal to the relationship between those general classes and the particular instance. For example, the argument that “a person would typically feel exhilaration from heights” is generally more compelling than the argument that “James would feel exhilaration from heights.”

Q: A person’s hand hurts after hitting the bottle. What might happen to
the bottle?
Answer Choices:
(a) alcoholism
(b) breaking
(c) fly
(d) nausea
(e) ticket
A: The answer must be something that happens to the bottle. The bottle is
broken. Therefore, the answer is breaking (b).

Q: James loved climbing.  What emotion might he feel from heights?
Answer Choices:
(a) exhaustion
(b) exhilaration
(c) falling down
(d) nervousness
(e) vertigo
A: The answer must be a feeling that would cause someone to climb a mountain.
James would feel exhilaration from heights. Therefore, the answer is
exhilaration (b).

A.5Red Herrings

Some errors in reasoning corresponded to the model making a statement which, while technically true, is not useful in demonstrating the claim.

Q: A squirrel ran up to it’s home, where is the small dog likely to be
barking up?
Answer Choices:
(a) manhattan apartment
(b) wood
(c) neighbor’s yard
(d) tree
(e) basket
A: The answer must be a place where a small dog would be likely to be
barking. Squirrels are not small dogs. Therefore, the answer is tree (d).

Q: John ran very fast, his feet touched the ground in what way?
Answer Choices:
(a) behind
(b) loose
(c) feverishly
(d) slowly
(e) lightly
A: The answer must be a way of running that is different from the way John
ran. Running is a fast, light-footed activity. Therefore, the answer is
lightly (e).

A.6Hint Short-cutting

In the experiments where the model was shown some examples of “hints” during training, in order to prevent it from losing the ability to perform rationalization over time, the model appeared to pick up on the fact that the final answer would always correspond to the hinted answer. This led to answers such as

Q: Billy bought coffee and waited for his wife to arrive from  France.
Where might he have been?
Answer Choices:
(a) airport
(b) grocery store
(c) internet cafe
(d) supermarket
(e) train station
A: The answer must be a place where Billy could have been waiting for his
wife to arrive from France. The airport is a place where people can wait
for flights. Therefore, the answer is train station (e).

Appendix BModified CQA Prompts

For reference, we include our modified prompts based closely on those in6.

Q:Whatdopeopleusetoabsorbextrainkfromafountainpen?

AnswerChoices:

(a)shirtpocket

(b)calligrapher’shand

(c)inkwell

(d)deskdrawer

(e)blotter

A:Theanswermustbeusedtoabsorbextraink.Blottersaredesignedtoabsorbliquids.Therefore,theanswerisblotter(e).

Q:Whathomeentertainmentequipmentrequirescable?

AnswerChoices:

(a)radioshack

(b)substation

(c)television

(d)cabinet

(e)desk

A:Theanswermustrequirecable.Cableisusedtoprovidesatellitechannelstotelevisions.Therefore,theansweristelevision(c).

Q:Thefoxwalkedfromthecityintotheforest,whatwasitlookingfor?

AnswerChoices:

(a)prettyflowers

(b)henhouse

(c)naturalhabitat

(d)storybook

(e)denseforest

A:Theanswermustbeareasonforafoxtogointotheforest.Theforestisafox’snaturalhabitat.Therefore,theanswerisnaturalhabitat(c).

Q:Sammywantedtogotowherethepeoplewere.Wheremighthego?

AnswerChoices:

(a)populatedareas

(b)racetrack

(c)desert

(d)apartment

(e)roadblock

A:Theanswermustbeaplacewithmanypeople.Populatedareas,bydefinition,havealotofpeople.Therefore,theanswerispopulatedareas(a).

Q:Wheredoyouputyourgrapesjustbeforecheckingout?

AnswerChoices:

(a)mouth

(b)grocerycart

(c)supermarket

(d)fruitbasket

(e)fruitmarket

A:Theanswershouldbetheplacewheregroceryitemsareplacedbeforecheckingout.Oftheabovechoices,grocerycartmakesthemostsenseforholdinggroceryitems.Therefore,theanswerisgrocerycart(b).

Q:GoogleMapsandotherhighwayandstreetGPSserviceshavereplacedwhat?

AnswerChoices:

(a)unitedstates

(b)mexico

(c)countryside

(d)atlas

(e)oceans

A:TheanswermustbesomethingthatusedtodowhatGoogleMapsandGPSservicesdo,whichisgivedirections.Atlaseswerealsousedtogivedirections.Therefore,theanswerisatlas(d).

Q:Beforegettingadivorce,whatdidthewifefeelwhowasdoingallthework?

AnswerChoices:

(a)harder

(b)anguish

(c)bitterness

(d)tears

(e)sadness

A:Theanswershouldbeafeelingwhichwouldcausesomeonewhowasdoingalltheworktogetdivorced.Ifsomeonefeelsbittertowardstheirspouse,theyarelikelytowantadivorce.Therefore,theanswerisbitterness(c).

Appendix CHuman-evaluated Test Prompts

We also selected a random sampling of 50 questions which were correctly answered both few-shot and by a STaR-trained model (without rationalization), as discussed in4.4. Presented in a random order, twenty crowdworkers preferred the STaR-generated answers. We reproduce the examples here with the few-shot rationale first, the STaR-trained rationale second, and the human rationale third, though these were shuffled when presented to participants. We selected human answers from[3]’s original split rationales where possible, finding that duplicate rationales were much more common in the new split rationales. For example, the explanation “Rivers flow trough valleys,” appeared over 400 times verbatim in the new split dataset, and “This word was most relevant” appeared over 150 times.[3]’s dataset also includes explanations like “The only answer that makes sense” or “BOB WILL NOT POKEMON CARDS WERE COMMON AND WORTHLESS BUT WRONG ABOUT THEM SO FEEL REALLY RARE TO DELAY” or restatements of the answer. We append the phrase “Therefore, the answer is ANSWERTEXT (ANSWERLETTER)” with ANSWERTEXT replaced by the correct answer’s text and ANSWERLETTER replaced by the correct answer letter. This is done 1) to make it less obvious that one of the answers is generated by a different source and 2) to prioritize differences in rationales, not the answer format.

Before the questions and after the consent form, we presented the following directions:

Eachquestionwillhavethreeexplanationsforoneanswer,whichisapre-determinedcorrect(“best”)answer.

Pleaseranktheexplanationsforeachquestionfrombesttoworst(with1asthebestand3astheworst)basedonhowwelltheyjustifythecorrectanswer.

Notetherearesometyposandgrammaticalerrorsintheexamples,inthequestions,answers,andtheexplanation.Pleasetreattheanswerletter(a/b/c/d/e)attheendoftheexplanationsasthecorrectanswerletter,evenifanotheranswerisvalid(andperhapsmorereasonable),andevenifitdoesnotmatchwiththeanswertextprecedingit.

The examples were subsampled and presented to the crowdworkers:

Q:Whoislikelyyohaveacaringheart?

AnswerChoices:

(a)person

(b)species

(c)artichoke

(d)allmammals

(e)liveanimal

A:Theanswermustbeaperson.Allmammalshaveacaringheart.Therefore,theanswerisperson(a).

A:Theanswermustbesomeonewhoislikelytohaveacaringheart.Humansarethemostlikelytohaveacaringheart.Therefore,theanswerisperson(a).

A:Apersonistheonlyhumanbeingcapableofhavingacaringheart.Therefore,theanswerisperson(a).

Q:Ifapersonwantedtobecomeagoodwriterwhatshouldtheydoincollege?

AnswerChoices:

(a)wordsentence

(b)ownanimal

(c)readthenewspaper

(d)catchcold

(e)studyliterature

A:Theanswermustbesomethingthatwouldhelpapersonbecomeagoodwriter.Theanswerisstudyliterature(e).

A:Theanswermustbesomethingthatwouldhelpapersonbecomeagoodwriter.Literatureisaformofstudy.Therefore,theanswerisstudyliterature(e).

A:studyingliteraturewillgiveoneskillsofwriting.Therefore,theanswerisstudyliterature(e).

Q:ThehumanalwayswouldhavefunmakingupquestionsfortheA.I.overlords,hefoundthetaskquitewhat?

AnswerChoices:

(a)doenjoy

(b)eatcake

(c)enjoyliving

(d)getlaid

(e)enjoyable

A:Theanswermustbesomethingthatthehumanwouldenjoydoing.ThehumanenjoysmakingupquestionsfortheA.I.overlords.Therefore,theanswerisenjoyable(e).

A:Theanswermustbesomethingthatthehumanfoundenjoyable.ThehumanfoundthetaskofmakingupquestionsfortheA.I.overlordsenjoyable.Therefore,theanswerisenjoyable(e).

A:Havingfunisenjoyable.Therefore,theanswerisenjoyable(e).

Q:WhathappenswhenImakeapigofmyselfeatingbreakfast?

AnswerChoices:

(a)fullstomach

(b)gainenergy

(c)dieting

(d)haveenergy

(e)feelingsatisfied

A:Theanswermustbeafeelingwhichwouldcausesomeonetofeelfullaftereatingbreakfast.Theanswerisfullstomach(a).

A:Theanswermustbeafeelingwhichwouldcausesomeonetomakeapigofthemselveseatingbreakfast.Ifsomeonefeelsfull,theyarelikelytostopeating.Therefore,theanswerisfullstomach(a).

A:Acolloquialphraseforoverindulginginfood.Therefore,theanswerisfullstomach(a).

Q:Inordertoseeastoryonthebigscreenwhatmustyoudo?

AnswerChoices:

(a)gotomovies

(b)visualize

(c)reading

(d)openbook

(e)sketchingapicture

A:Theanswermustbesomethingthatyoudotoseeastoryonthebigscreen.Moviesareawaytoseeastoryonthebigscreen.Therefore,theanswerismovies(a).

A:Theanswermustbesomethingthatisrequiredtoseeastoryonthebigscreen.Moviesareaformofstorytelling.Therefore,theanswerisgotomovies(a).

A:Thebigscreenisacolloquialtermmeaningamovietheater.Therefore,theanswerisgotomovies(a).

Q:Whatistheoppositeofremembering?

AnswerChoices:

(a)answeringquestion

(b)stoppingtothink

(c)knowing

(d)closingeyes

(e)forgetting

A:Theanswermustbesomethingthatistheoppositeofremembering.Theanswerisforgetting(e).

A:Theanswermustbeafeelingwhichwouldcausesomeonetonotremembersomething.Ifsomeoneforgetssomething,theyarelikelytonotrememberit.Therefore,theanswerisforgetting(e).

A:Healthcomplications.Therefore,theanswerisforgetting(e).

Q:Ifyouaren’texperiencingcuriosityandwanttoensureviewingenjoyment,whatmayyouwatch?

AnswerChoices:

(a)examinething

(b)stareintoamirror

(c)gosomewhere

(d)hearnews

(e)seefavoriteshow

A:Theanswermustbesomethingthatwillmakeyouwanttowatch.Ifyouarenotexperiencingcuriosity,youmaywanttowatchsomethingthatwillmakeyouwanttowatch.Therefore,theansweriswatch(e).

A:Theanswermustbesomethingthatwillnotcauseyoutowanttowatch.Ifyouarenotexperiencingcuriosity,youarelikelytowanttowatchsomethingthatisentertaining.Therefore,theanswerisfavoriteshow(e).

A:seefavoriteshowcanyoudoinfrontofatvifyouhavecuriosityaboutwhatisbeingdisplayedonthetv.Therefore,theanswerisfavoriteshow(e).

Q:Thecarotidisamainarterysusceptibletodamagewhere?

AnswerChoices:

(a)neck

(b)face

(c)busycity

(d)torso

(e)humanbody

A:Theanswermustbeaplacewherethecarotidarteryissusceptibletodamage.Thecarotidarteryislocatedintheneck.Therefore,theanswerisneck(a).

A:Theanswermustbeaplacewherethecarotidissusceptibletodamage.Thecarotidissusceptibletodamageintheneck.Therefore,theanswerisneck(a).

A:Thecarotidarteryislocatedintheneck.Therefore,theanswerisneck(a).

Q:Thekingneededtofeelsafe,wheredidhego?

AnswerChoices:

(a)castle

(b)throneroom

(c)deckofcards

(d)fort

(e)court

A:Theanswermustbeaplacewherethekingfeelssafe.Theking’scastleisaplacewherehefeelssafe.Therefore,theansweriscastle(a).

A:Theanswermustbeaplacewherethekingcanfeelsafe.Castlesareplaceswherekingsfeelsafe.Therefore,theansweriscastle(a).

A:Ingeneral,Kingliveincastle.Therefore,theansweriscastle(a).

Q:Whatistheresultofinstitutingcivilaction?

AnswerChoices:

(a)badfeelings

(b)confusion

(c)goingtocourt

(d)hurtfeelings

(e)sandwiches

A:Theanswermustbearesultofinstitutingcivilaction.Theresultofinstitutingcivilactionisalawsuit.Therefore,theanswerisgoingtocourt(c).

A:Theanswermustbearesultofinstitutingcivilaction.Ifsomeonegoestocourt,theyarelikelytofeelhurtfeelings.Therefore,theanswerisgoingtocourt(c).

A:AngerandTrialwillnotinstituteacourtcase.Therefore,theanswerisgoingtocourt(c).Q:Thepersonwantedtofixhiscar,whatdidheorder?

AnswerChoices:

(a)sellingthings

(b)gas

(c)feelresentful

(d)machinepart

(e)crossstreet

A:Theanswermustbesomethingthatwouldhelpthepersonfixhiscar.Thepersonwouldnotwanttosellthings,becausehewouldnotwanttogiveuphiscar.Therefore,theanswerismachinepart(d).

A:Theanswermustbesomethingthatwouldbeusedtofixacar.Machinepartsareusedtofixcars.Therefore,theanswerismachinepart(d).

A:Amachinepartistheonlythinglistedthatwouldfixthecar.Therefore,theanswerismachinepart(d).

Q:Healwaystraveledupthemountainwhenwinterarrived,hehadapassionanddesiretodowhat?

AnswerChoices:

(a)playvolleyball

(b)playhockey

(c)ski

(d)knit

(e)skate

A:Theanswermustbesomethingthatthepersondoeswhenwinterarrives.Theanswerisskiing(c).

A:Theanswermustbesomethingthatthepersondidwhenwinterarrived.Theanswerisski(c).

A:Skiingistheonlyoptionthatisdoneonamountain.Therefore,theanswerisski(c).

Q:BobthoughtthathisPokemoncardswerecommonandworthless,buthewaswrongaboutthem.Theywerereallywhat?

AnswerChoices:

(a)personal

(b)special

(c)shiny

(d)rare

(e)irregular

A:Theanswermustbesomethingthatisnotcommonandworthless.Pokemoncardsarenotcommonandworthless.Therefore,theanswerisrare(d).

A:Theanswermustbesomethingthatarerare.Pokemoncardsarerare.Therefore,theanswerisrare(d).

A:BOBWILLNOTPOKEMONCARDSWERECOMMONANDWORTHLESSBUTWRONGABOUTTHEMSOFEELREALLYRARETODELAY.Therefore,theanswerisrare(d).

Q:Ifyou’relightingacandletoseewhatareyouin?

AnswerChoices:

(a)dimlylitroom

(b)synagogue

(c)sunlight

(d)birthdaycake

(e)bedroom

A:Theanswermustbeaplacewhereyoucanseewhatyouarein.Theanswerisaroomwithdimlighting.Therefore,theanswerisdimlylitroom(a).

A:Theanswermustbeaplacewhereyoucanseeacandle.Candlesareusedtoseethingsinthedark.Therefore,theanswerisdimlylitroom(a).

A:Lightingorilluminationisthedeliberateuseoflighttoachieveapracticaloraestheticeffect.Lightingincludestheuseofbothartificiallightsourceslikelamps...Therefore,theanswerisdimlylitroom(a).

Q:Thepersonwasthirsty,whatshouldhehavedone?

AnswerChoices:

(a)eat

(b)sunnydays

(c)compliments

(d)drinkwater

(e)allkindsofthings

A:Theanswermustbesomethingthatwouldmakethepersonfeelbetter.Ifthepersonisthirsty,heshoulddrinkwater.Therefore,theanswerisdrinkwater(d).

A:Theanswermustbesomethingthatwouldmakesomeonefeelbetteriftheywerethirsty.Ifsomeoneisthirsty,theyarelikelytowanttodrinkwater.Therefore,theanswerisdrinkwater(d).

A:Thepersonwasthirsty,sodrinkwater.Therefore,theanswerisdrinkwater(d).

Q:Wherewouldyoufindmanyproductsforsale?

AnswerChoices:

(a)market

(b)costonedollar

(c)shelf

(d)salon

(e)demonstration

A:Theanswermustbeaplacewheremanyproductsaresold.Amarketisaplacewheremanyproductsaresold.Therefore,theanswerismarket(a).

A:Theanswermustbeaplacewithalotofproductsforsale.Marketsarethemostcommonplacetobuyproducts.Therefore,theanswerismarket(a).

A:Theonlyplaceproductsareforsaleareinmarketsandstores.Therefore,theanswerismarket(a).

Q:Ifmanandmammothwantedtomeet,theywouldprobablyhavetodoitduringwhen?

AnswerChoices:

(a)ababyshower

(b)ancienthistory

(c)storybook

(d)stoneage

(e)forest

A:Theanswermustbeatimewhenmanandmammothwouldmeet.Stoneageisthetimewhenmanandmammothlived.Therefore,theanswerisstoneage(d).

A:Theanswermustbeatimewhenmanandmammothwouldbeabletomeet.Thestoneagewasatimewhenmanandmammothwouldbeabletomeet.Therefore,theanswerisstoneage(d).

A:Duringthestoneageistheonlytimemammothslived.Therefore,theanswerisstoneage(d).

Q:Theypridedthemselvesonbeingawealthofknowledge,andthat’swhymanychosetoattendtheirwhat?

AnswerChoices:

(a)bookstore

(b)university

(c)meeting

(d)class

(e)encyclopedia

A:Theanswermustbeaplacewherepeoplegotolearnaboutthings.Theanswerisuniversity(b).

A:Theanswermustbeaplacewherepeoplegotolearnaboutthings.Universitiesareplaceswherepeoplegotolearnaboutthings.Therefore,theanswerisuniversity(b).

A:becauseintheuniversityyougetmoreknowledge.Therefore,theanswerisuniversity(b).

Q:whathappenswhenvisitingmuseumthathavenarrowhalls?

AnswerChoices:

(a)sorefeet

(b)amazement

(c)retrospection

(d)boredom

(e)crowding

A:Theanswermustbesomethingthathappenswhenvisitingamuseumthathasnarrowhalls.Crowdingisacommonprobleminmuseums.Therefore,theansweriscrowding(e).

A:Theanswermustbesomethingthatwouldcausesomeonetohavesorefeetiftheyarewalkinginnarrowhalls.Ifsomeonefeelscramped,theyarelikelytowanttogosomewhereelse.Therefore,theansweriscrowding(e).

A:museumisoneofthecrowdestplace.Therefore,theansweriscrowding(e).

Q:Ontheirhiketheybroughtafilteringstraw,theywereworriedaboutgermsinthewhat?

AnswerChoices:

(a)makesick

(b)doctor

(c)water

(d)stream

(e)mouth

A:Theanswermustbesomethingthatcanfilteroutgerms.Filteringstrawsareusedtofilteroutgerms.Therefore,theanswerisfilteringstraw(c).

A:Theanswermustbesomethingthatwouldcausesomeonetobringafilteringstrawonahike.Filteringstrawsareusedtofilterwater.Therefore,theansweriswater(c).

A:strawitsomethingusedtodrinkwater.Therefore,theansweriswater(c).

Q:IfIamnotgoodatplayingpoker,IwillbeginwhatifIplay?

AnswerChoices:

(a)discord

(b)socialevent

(c)losingmoney

(d)competition

(e)heightenedawareness

A:Theanswermustbesomethingthatwillhappenifyouarenotgoodatplayingpoker.Ifyouarenotgoodatplayingpoker,youwillbegintolosemoney.Therefore,theanswerislosingmoney(c).

A:Theanswermustbesomethingthatwillhelpyouifyouarenotgoodatplayingpoker.Ifyouarelosingmoney,youarelikelytowanttostopplaying.Therefore,theanswerislosingmoney(c).

A:Pokerisagameinwhichpeoplelosetheirmoney.Therefore,theanswerislosingmoney(c).

Q:Peoplearemeetingpeople,whatisthelikelyphysicalgesture?

AnswerChoices:

(a)introducingyourself

(b)smiling

(c)shakehands

(d)shareahug

(e)talking

A:Theanswermustbeaphysicalgesturethatisusedtogreetsomeone.Theanswerishandshake(c).

A:Theanswermustbeagestureusedtomeetpeople.Ifsomeonewantstomeetpeople,theyarelikelytowanttoshakehands.Therefore,theanswerisshakinghands(c).

A:Ahandshakeisagloballywidespread.Therefore,theanswerisshakinghands(c).

Q:Whatdodogsgetthatmakethemitch?

AnswerChoices:

(a)bones

(b)fleas

(c)badbreath

(d)waggingtails

(e)fourlegs

A:Theanswermustbesomethingthatmakesdogsitch.Dogsgetfleas,whichmakethemitch.Therefore,theanswerisfleas(b).

A:Theanswermustbesomethingthatdogsgetthatmakethemitch.Fleasareacommoncauseofdogitch.Therefore,theanswerisfleas(b).

A:fleasmakesyouitch.Therefore,theanswerisfleas(b).

Q:Whatisignoranceoftenequatedwith?

AnswerChoices:

(a)bliss

(b)hostility

(c)peace

(d)accidents

(e)findtruth

A:Theanswermustbesomethingthatisoftenequatedwithignorance.Blissisoftenequatedwithignorance.Therefore,theanswerisbliss(a).

A:Theanswermustbeafeelingwhichwouldcausesomeonetobeignorant.Ifsomeonefeelsbliss,theyarelikelytobeignorant.Therefore,theanswerisbliss(a).

A:blisscomesfromignorance.Therefore,theanswerisbliss(a).

Q:Thetwoteamswerecompetingallthetime,whatlikelydevelopedbetweenthem?

AnswerChoices:

(a)accomplishment

(b)enemies

(c)rivalry

(d)friendship

(e)badblood

A:Theanswermustbesomethingthatdevelopedbetweenthetwoteams.Theteamswerecompetingallthetime.Therefore,theanswerisrivalry(c).

A:Theanswermustbeafeelingwhichwouldcausetwoteamstocompeteallthetime.Rivalryisacommonfeelingwhichwouldcausetwoteamstocompeteallthetime.Therefore,theanswerisrivalry(c).

A:likelydevelopedbetweenthemofrivalry.Therefore,theanswerisrivalry(c).

Q:Jamesdoesn’twantarelationship.Hethinksthatlovinganotherwillonlycreatewhat?

AnswerChoices:

(a)satisfaction

(b)jealousy

(c)happiness

(d)wonderful

(e)quiet

A:TheanswermustbesomethingthatwouldmakeJamesfeelbadaboutnothavingarelationship.Ifsomeonefeelsjealous,theyarelikelytowantarelationship.Therefore,theanswerisjealousy(b).

A:Theanswermustbeafeelingwhichwouldcausesomeonewhodoesn’twantarelationshiptonotwantone.Ifsomeonefeelsjealousofanother,theyarelikelytonotwantarelationship.Therefore,theanswerisjealousy(b).

A:Jealousyisanegativeemotion.Therefore,theanswerisjealousy(b).

Q:Thehadareputationforcelebratinginadebaucherousmanner,whatoftenhappenedasaresult?

AnswerChoices:

(a)drunkenness

(b)enjoyment

(c)havefun

(d)hangover

(e)fighting

A:Theanswermustbearesultofacelebration.Theanswerisdrunkenness(a).

A:Theanswermustbesomethingthatwouldcausesomeonetohaveareputationforcelebratinginadebaucherousmanner.Alcoholisadepressant.Therefore,theanswerisdrunkenness(a).

A:itisnormalnotagoodideatogetdrunk.Therefore,theanswerisdrunkenness(a).

Q:Thepapertowelsareusuallyinplainsightinthekitchen,wheredidthedadgonexttofindthem?

AnswerChoices:

(a)pantry

(b)cabinet

(c)grocerystore

(d)wastebin

(e)market

A:Theanswermustbeaplacewherethedadwouldgotofindthepapertowels.Thepantryisaplacewherefoodisstored.Therefore,theanswerispantry(a).

A:Theanswermustbeaplacewherepapertowelsarestored.Pantriesarethemostlikelyplacetostorepapertowels.Therefore,theanswerispantry(a).

A:Thepantryisinthekitchenandstoresthings.Therefore,theanswerispantry(a).

Q:Myhouseisverydry,withlittlewaterintheair.WhattoolshouldIusetofixthis?

AnswerChoices:

(a)humidifier

(b)surfaceofearth

(c)skatingrink

(d)icecube

(e)burnittotheground

A:Theanswermustbeatoolthatwillmaketheairinyourhousemoist.Ahumidifierisatoolthatwillmaketheairinyourhousemoist.Therefore,theanswerishumidifier(a).

A:Theanswermustbesomethingthatwouldmaketheairinmyhousemorehumid.Ahumidifierisusedtomaketheairinaroommorehumid.Therefore,theanswerishumidifier(a).

A:Thiswordwasmostrelevant.Therefore,theanswerishumidifier(a).

Q:Amotorcycleisquickerbecauseitisn’taswhatasacar?

AnswerChoices:

(a)heavy

(b)shiny

(c)red

(d)obesity

(e)expensive

A:Theanswermustbesomethingthatisnotasheavyasacar.Amotorcycleislighterthanacar.Therefore,theanswerisnotheavy(a).

A:Theanswermustbesomethingthatisnotasquickasamotorcycle.Motorcyclesarenotasheavyascars.Therefore,theanswerisheavy(a).

A:Becauseitismuchsmaller.Therefore,theanswerisheavy(a).

Q:Thebaseballteamwasplayingball.Itwastimeforthepitchertoact.Howdidthepitchergettheballtothebatter?

AnswerChoices:

(a)lose

(b)injury

(c)competition

(d)throwing

(e)rolled

A:Theanswermustbeawaytogettheballtothebatter.Thepitcheristheonewhothrowstheball.Therefore,theansweristhrowing(d).

A:Theanswermustbeawayforthepitchertogettheballtothebatter.Thepitcheristhepersonwhoisresponsibleforgettingtheballtothebatter.Therefore,theansweristhrowing(d).

A:Thepitcherthrewtheballtothebattersothathecouldhitit.Therefore,theansweristhrowing(d).

Q:Inwhatsortofcontainershouldyoukeepyourficus?

AnswerChoices:

(a)refigerator

(b)screenedporch

(c)claypot

(d)shadyplaces

(e)arboretum

A:Theanswermustbeacontainerthatkeepstheficusinashadyplace.Claypotsareusedtokeepplantsinshadyplaces.Therefore,theanswerisclaypot(c).

A:Theanswermustbeacontainerinwhichficuscanbekept.Ficusaretropicalplants.Therefore,theanswerisclaypot(c).

A:Claypotistheonlytypeofcontainerlisted.Therefore,theanswerisclaypot(c).

Q:Jameswascookingsstewinhisapartmentkitchen.Wheremighthelookforapotato.

AnswerChoices:

(a)restaurants

(b)casserole

(c)rootcellar

(d)pockets

(e)pantry

A:Theanswermustbeaplacewherepotatoesarestored.Theanswerispantry(e).

A:Theanswermustbeaplacewherepotatoesarestored.Pantriesarethemostlikelyplacetostorepotatoes.Therefore,theanswerispantry(e).

A:apartmentsdonotoftenhaverootcellars,andarenotrestaurants.apantryisclosesttoJamesinsidehisapartment.Therefore,theanswerispantry(e).

Q:Ifshipsareinastormandthesuncomesout,whathashappened?

AnswerChoices:

(a)sinks

(b)crossriver

(c)collide

(d)bottomout

(e)carrypeople

A:Theanswermustbeathingthathappenswhenshipsareinastormandthesuncomesout.Shipsareinastormwhentheyareinthemiddleoftheocean.Therefore,theanswerisbottomout(d).

A:Theanswermustbesomethingthathappenswhenthesuncomesout.Thesunisasourceofenergy.Therefore,theanswerisbottomout(d).

A:thesuncomesoutsoitsnotastormanymore.Therefore,theanswerisbottomout(d).

Q:Wheredoallmarmotlivebeforetheyarecaptured?

AnswerChoices:

(a)switzerland

(b)spain

(c)vancouverisland

(d)wilderness

(e)australia

A:Theanswermustbeaplacewheremarmotslive.Marmotsliveinthewilderness.Therefore,theansweriswilderness(d).

A:Theanswermustbeaplacewheremarmotsarefound.Thewildernessisthemostlikelyplacetofindmarmots.Therefore,theansweriswilderness(d).

A:Marmotsdon’tliveinAustraliaorSwitzerland.Therefore,theansweriswilderness(d).

Q:Wheredoesonewaitonabenchforsomethingontracks?

AnswerChoices:

(a)trainstation

(b)statepark

(c)crosswalk

(d)busstop

(e)busdepot

A:Theanswermustbeaplacewhereonewaitsforatrain,bus,orothervehicle.Theansweristrainstation(a).

A:Theanswermustbeaplacewhereonecanwaitforatrain.Trainstationsareplaceswheretrainsstop.Therefore,theansweristrainstation(a).

A:Onlytrainsareontracks.Therefore,theansweristrainstation(a).

Q:Whatcanplanningvacationleadtobetweenacouplewhentheydisagree?

AnswerChoices:

(a)relaxation

(b)enjoying

(c)arguments

(d)goingabroad

(e)spendingmoney

A:Theanswermustbesomethingthatcanhappenwhenacoupledisagrees.Argumentsareacommonresultofdisagreements.Therefore,theanswerisarguments(c).

A:Theanswermustbearesultofplanningvacation.Argumentsareacommonresultofplanningvacation.Therefore,theanswerisarguments(c).

A:Argumentsarethedirectresultofdisagreements.Therefore,theanswerisarguments(c).

Q:Aslothclimbsslow,butitcaneventuallyreachallthewayupwhere?

AnswerChoices:

(a)forestcanopy

(b)universe

(c)tropicalrainforest

(d)largetrees

(e)naturepreserve

A:Theanswermustbeaplacewhereaslothcanclimb.Theforestcanopyisthehighestpointintheforest.Therefore,theanswerisforestcanopy(a).

A:Theanswermustbeaplacewhereaslothcanclimb.Thecanopyofaforestisthehighestpointinaforest.Therefore,theansweriscanopyofaforest(a).

A:TheslothisamammalthatlivesinthetreesofSouthAmerica.Theyhaveaslowandpeculiarwayofmovingabout.Infact,theyaretheslowestlandmammalsandtravelverysluggishlyontheground(0.025m/s-0.050m/s).Therefore,theansweriscanopyofaforest(a).

Q:WhereinAmericaisknownfortheaters?

AnswerChoices:

(a)newyorkcity

(b)shoppingmall

(c)downtownarea

(d)skatingrink

(e)militarybase

A:Theanswermustbeaplacewherepeoplegotoseemovies.Theatersareplaceswherepeoplegotoseemovies.Therefore,theansweristheater(a).

A:Theanswermustbeaplacewithalotoftheaters.NewYorkCityisknownforitsmanytheaters.Therefore,theanswerisNewYorkCity(a).

A:NewYorkCityisthemainregioninUSknownfortheaters.Therefore,theanswerisNewYorkCity(a).

Q:Whatdoeshearingsomeoneplayviolinbeautifullymakeyou?

AnswerChoices:

(a)buyearplugs

(b)inspiring

(c)guitar

(d)makemusic

(e)likemusic

A:Theanswermustbesomethingthatmakesyoufeelinspired.Hearingsomeoneplayviolinbeautifullymakesyoufeelinspired.Therefore,theanswerisinspiring(b).

A:Theanswermustbeafeelingwhichwouldcausesomeonetowanttohearsomeoneplayviolinbeautifully.Musicisaformofart.Therefore,theanswerisinspiring(b).

A:Hearingsomeoneplayaviolinbeautifullymakesmeinspired.Therefore,theanswerisinspiring(b).

Q:Whatemotiondoescommittingmurderinduce?

AnswerChoices:

(a)fear

(b)gotojail

(c)problems

(d)cruel

(e)dead

A:Theanswermustbeafeelingthatwouldcausesomeonetocommitmurder.Murderisacrime.Therefore,theanswerisfear(a).

A:Theanswermustbeafeelingwhichwouldcausesomeonetocommitmurder.Ifsomeonefeelsfear,theyarelikelytowanttoavoidcommittingmurder.Therefore,theanswerisfear(a).

A:Theonlyfeeling.Therefore,theanswerisfear(a).

Q:SittingtoclosewhilewatchingTVcancausewhatsortofpain?

AnswerChoices:

(a)brainproblems

(b)laziness

(c)getfat

(d)headache

(e)laughter

A:TheanswermustbeapainthatcomesfromsittingtooclosetotheTV.SittingtooclosetotheTVcancauseheadaches.Therefore,theanswerisheadache(d).

A:TheanswermustbeatypeofpainwhichcanoccurifyouaresittingtooclosetotheTV.HeadachesareacommonsideeffectofsittingtooclosetotheTV.Therefore,theanswerisheadache(d).

A:theprocessofaginginhumans.Therefore,theanswerisheadache(d).

Q:Whatmightpeopledotoprotecttheirlegsfromgettingdirtyonthefarm?

AnswerChoices:

(a)wearjeans

(b)milkcow

(c)believeingod

(d)avoidmud

(e)plantflowers

A:Theanswermustbesomethingthatpeopledotoprotecttheirlegsfromgettingdirty.Oftheabovechoices,wearingjeansisthemostlogical.Therefore,theanswerisjeans(a).

A:Theanswermustbesomethingthatwouldprotectlegsfromgettingdirtyonthefarm.Jeansareusedtoprotectlegsfromgettingdirty.Therefore,theansweriswearjeans(a).

A:Jeanscoveroverlegs.Therefore,theansweriswearjeans(a).

Q:Whattypesofstovesarethere?

AnswerChoices:

(a)thesetwosareirreleventtothequestions

(b)warmpotofbeans

(c)brownmeat

(d)warmroom

(e)gasorelectric

A:Theanswermustbeatypeofstove.Thestoveisusedtoheatfood.Therefore,theanswerisgasorelectric(e).

A:Theanswermustbeatypeofstove.Stovesareusedtowarmrooms.Therefore,theansweriselectric(e).

A:stovesaregasorelectric.Therefore,theanswerisgasorelectric(e).

Q:Inwhatgeologicalfeaturewillyoufindfungusgrowing?

AnswerChoices:

(a)showerstall

(b)toenails

(c)basement

(d)forrest

(e)cave

A:Theanswermustbeaplacewherefungusgrows.Theansweriscave(e).

A:Theanswermustbeaplacewherefungusgrows.Fungusgrowsincaves.Therefore,theansweriscave(e).

A:Riversflowtroughvalleys.Therefore,theansweriscave(e).

Q:Whatisoursunconsideredtobethecenterof?

AnswerChoices:

(a)universe

(b)solarsystem

(c)outerspace

(d)star

(e)beans

A:Theanswermustbesomethingthatisconsideredtobethecenteroftheuniverse.Thesunisthecenterofthesolarsystem.Therefore,theanswerissolarsystem(b).

A:Theanswermustbeaplacewherethesunisthecenterof.Thesunisthecenterofthesolarsystem.Therefore,theanswerissolarsystem(b).

A:Itisthebestoptionbecauseitistheclosest.Therefore,theanswerissolarsystem(b).

Q:Whatarepeopleinalibrarylikelydoing?

AnswerChoices:

(a)talktoeachother

(b)boardships

(c)studybooks

(d)sufferhunger

(e)playinggames

A:Theanswermustbesomethingthatpeopleinalibraryarelikelytobedoing.Peopleinalibraryarelikelytobestudyingbooks.Therefore,theanswerisstudybooks(c).

A:Theanswermustbesomethingthatislikelytohappeninalibrary.Librariesareplaceswherepeoplestudybooks.Therefore,theanswerisstudyingbooks(c).

A:Booksshelvedasstudy-skills:HowtoBecomeaStraight-AStudentbyCalNewport,TheStudySkillsHandbookbyStellaCottrell,MakeItStick:TheScienc...Therefore,theanswerisstudybooks(c).

Q:Asharkisn’tsomethingyouhavetoworryaboutatthelakeinthewoods,butthereareplentyofdangersinthewhat?

AnswerChoices:

(a)greatoceans

(b)greatoutdoors

(c)tropicalwaters

(d)pacificocean

(e)gulfofmexico

A:Theanswermustbesomethingthatisdangerousinthegreatoutdoors.Thegreatoutdoorsisaplacewheretherearemanydangers.Therefore,theanswerisgreatoutdoors(b).

A:Theanswermustbeaplacewheresharksareadanger.Lakesinthewoodsarenotaplacewheresharksareadanger.Therefore,theanswerisgreatoutdoors(b).

A:Thegreatoutdoorsistheonlychoicewhereyouwouldfindalakeinthewoods.Therefore,theanswerisgreatoutdoors(b).

Q:Youcansharefileswithsomeoneifyouhaveaconnectiontoawhat?

AnswerChoices:

(a)freeway

(b)radio

(c)wires

(d)computernetwork

(e)electricalcircuit

A:Theanswermustbeawaytosharefiles.Theansweriscomputernetwork(d).

A:Theanswermustbeaconnectionwhichallowsyoutosharefiles.Acomputernetworkisacollectionofcomputerswhichareconnectedtoeachother.Therefore,theansweriscomputernetwork(d).

A:Filescanbetransmittedoveracomputernetwork,andnotafreeway.“wires“doesntmakeanysense.Therefore,theansweriscomputernetwork(d).

Q:Whatcananinjuryhappento?

AnswerChoices:

(a)emergencyroom

(b)doctor’soffice

(c)livingthing

(d)bones

(e)accident

A:Theanswermustbesomethingthatcanhappentoalivingthing.Theanswerislivingthing(c).

A:Theanswermustbesomethingthatcanhappentoalivingthing.Ifsomeoneisinjured,theyarelikelytowanttogototheemergencyroom.Therefore,theanswerislivingthing(c).

A:Livingthingsfeelpain,andthereforecanbeinjured.Therefore,theanswerislivingthing(c).

Appendix DExample Rationalizations on CQA

We include a randomly sampled set of rationalizations which the model is able to produce before fine-tuning. We observe that sometimes, the model constructs an argument roughly of the form “the answer must havea set of properties.correct answerhasthose properties. therefore, the answer iscorrect answer.” This structure of argument is fairly standard, but given that the model originally answered those questions incorrectly, it resembles template-matching more than reasoning. The technique of rephrasing the question in a simpler way before answering it directly is also often effective. In general, the rationales from the pretrained model’s rationalization appears to be of similar quality to the pretrained model’s rationale generation. We reference the pretrained model here rather than the STaR-trained model, as the final iteration’s rationales are the ones which the model struggled to learn from – that is, if the rationales were good, then we might expect that the model would have already incorporated them into its rationale generation.

Q:WhydoIeatjustenoughwheneatingbreakfast?

AnswerChoices:

(a)gas

(b)fullstomach

(c)feelingsatisfied(CORRECT)

(d)haveenergy

(e)bloated

A:Theanswermustbeareasonforeatingjustenoughwheneatingbreakfast.Eatingjustenoughisawaytofeelsatisfied.Therefore,theanswerisfeelingsatisfied(c).

Q:Howisadoglikelytocommunicatewithanotherdog?

AnswerChoices:

(a)reproduce

(b)bark(CORRECT)

(c)domanythings

(d)clone

(e)jumpup

A:Theanswermustbeawayforadogtocommunicatewithanotherdog.Dogsbarktocommunicatewitheachother.Therefore,theanswerisbark(b).

Q:Ifapersonistryingtoweaseloutofpunishmentforacrime,wherearetheylikelytobe?

AnswerChoices:

(a)courtroom(CORRECT)

(b)cherrytree

(c)chickencoop

(d)naturalhistorymuseum

(e)jail

A:Theanswermustbeaplacewhereapersonwouldbetryingtogetoutofpunishmentforacrime.Theansweriscourtroom(a).

Q:Whatwouldeasilyhideasnake?

AnswerChoices:

(a)livingroomofahouse

(b)kewgardens

(c)terrarium

(d)thickforest(CORRECT)

(e)tropicalforest

A:Theanswermustbesomethingthatwouldhideasnake.Theansweristhickforest(d).

Q:Thepersondidn’tlikethevaryingsizeofthecakesliceshandedout,shedemandmorewhat?

AnswerChoices:

(a)fairness(CORRECT)

(b)candies

(c)compliments

(d)wellfed

(e)arguements

A:Theanswermustbeareasonforthepersontodemandmorecake.Thepersonisnothappywiththesizeofthecakeslices.Therefore,theanswerisfairness(a).

Q:InAmerica,thecourthouseofanygiventownhasalikelylocation,whereisit?

AnswerChoices:

(a)electricalcircuit

(b)gettingmarried

(c)capitalcity

(d)centeroftown(CORRECT)

(e)michigan

A:Theanswermustbealocationwhereacourthouseislocated.Thecapitalcityisthelocationofthecapitalofastate.Therefore,theansweriscenteroftown(d).

Q:WheremightIfindabowlnestledamongotherbowlsinthekitchen?

AnswerChoices:

(a)refrigerator

(b)kitchentop

(c)cupboard(CORRECT)

(d)dishwasher

(e)apartment

A:Theanswermustbeaplacewhereabowlisfound.Thekitchenisaplacewherebowlsarefound.Therefore,theansweriscupboard(c).

Q:WhichstatehasblueridgemountainsthatarecalledBlueMountain?

AnswerChoices:

(a)appalachians

(b)virginia

(c)pennsylvania(CORRECT)

(d)northcarolina

(e)montana

A:Theanswermustbeastatewithmountains.TheanswerisPennsylvania(c).

Q:Wherecouldyoufindmanyradiostudio?

AnswerChoices:

(a)radiostation

(b)country

(c)cleararea

(d)officebuilding

(e)largecity(CORRECT)

A:Theanswermustbeaplacewheremanyradiostudiosarelocated.Radiostudiosareusedtobroadcastradioprograms.Therefore,theanswerislargecity(e).

Q:Wherewouldsomeonebringyouacup?

AnswerChoices:

(a)apartment

(b)closet

(c)restaurant(CORRECT)

(d)table

(e)party

A:Theanswermustbeaplacewheresomeonewouldbringyouacup.Arestaurantisaplacewherepeoplebringcups.Therefore,theanswerisrestaurant(c).

Appendix ESTaR Without Rationalization (Rationale Generation Bootstrapping)

For convenience, we include the rationale generation bootstrapping algorithm alone here. However, it is simply Algorithm1, with the blue components removed.

Algorithm 2Rationale Generation Bootstrapping (STaR without rationalization)InputMM: a pretrained LLM; dataset𝒟={(xi,yi)}i=1D\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{D}(w/ few-shot prompts)

1:

M0←{M_{0}}\leftarrowM{M}# Copy the original model

2:for

nnin

1​…​N1...Ndo# Outer loop

3:

(r^i,y^i)←Mn−1​(xi)∀i∈[1,D](\hat{r}_{i},\hat{y}_{i})\leftarrow M_{n-1}(x_{i})\quad\forall i\in[1,D]# Perform rationale generation

4:

𝒟n←{(xi,r^i,yi)∣i∈[1,D]∧y^i=yi}\mathcal{D}_{n}\leftarrow\{(x_{i},\hat{r}_{i},y_{i})\mid i\in[1,D]\land\hat{y}_{i}=y_{i}\}# Filter rationales using ground truth answers

5:

Mn←train​(M,𝒟n)M_{n}\leftarrow\text{train}(M,\mathcal{D}_{n})# Finetune the original model on the correct solutions - inner loop

6:endfor

Appendix FCase Study

Q: John wants to eat at an Indian
restaurant in on top of the space
needle, where would he have to go?
Answer Choices:
(a) reservation (b) curry
(c) britsh town (d) seattle
(e) large city
A: The answer must be a place with
a space needle. The space needle is
a landmark in Seattle. Therefore,
the answer is Seattle (d).

Q: I put a towel in a cardboard box for
the kitten, I wanted it to have what?
Answer Choices:
(a) farmhouse (b) zoos (c) cold place
(d) floor (e) warm place
A: The answer must be a place where the
kitten would be comfortable. The kitten
would be comfortable in a warm place.
Therefore, the answer is warm place (e).

Q: Needlepoint requires many
accessories, what holds the work taut?
Answer Choices:
(a) sewing (b) collection of folk art
(c) livingroom (d) frame (e) bathroom
A: The answer must be something that
holds the work taut. The work is taut
because it is held in place. Therefore,
the answer is frame (d).

(a)Problems where the model was able to solve questions which it had previously answered incorrectly. Q: They prided themselves on being awealth of knowledge, and that’s why manychose to attend their what?Answer Choices:(a) book store (b) university (c) meeting(d) class (e) encyclopediaA: The answer must be a place where peoplego to learn about things. The answer isuniversity (b).A: The answer must be a place where peoplego to learn about things. Universities areplaces where people go to learn aboutthings. Therefore, the answer isuniversity (b).Q: You want a real estate agent that isstraight forward with you, the lastthing you want is for them to what ahouse’s quality?Answer Choices:(a) cover up (b) offer price(c) misrepresent (d) lie (e) falsifyA: The answer must be a way to determinethe quality of a house. The answer ismisrepresent (c).A: The answer must be something thatwould be a bad thing for the real estateagent to do. The real estate agent wouldbe misrepresenting the house’s qualityif they were to lie about it. Therefore,the answer is misrepresent (c).

(b)Problems where the model was already able to answer them with few-shot reasoning, but was able to answer better after more iterations.

Figure 7:Case study on STaR. We highlight some problems which the model is not initially able to solve, as well as rationales which are initially correct but improve. We note that for CQA, one is tasked with selecting the “best” answer, where multiple answers may be technically correct.

Appendix GBias and Faithfulness

It is important to note that STaR is designed to amplify the reasoning that leads to correct solutions on a given dataset. The implication of this with a dataset like CommonsenseQA is that if biases are “useful” in solving the dataset then they will be amplified. This is made worse by rationalization, as biased answers that the model may not naturally arrive at are, in a sense, pulled out of the model. The exact interaction between the bias in a dataset and the pre-existing bias learned by the model is unclear, and something to consider before real-world deployment of all large language models in general, but STaR-trained models in particular.

We find some encouraging initial results on this however: for questions where gender is not relevant, the model appears to disregard it in its explanation. For example: “Q: Where is a good place for a woman to store her sunglasses?→\rightarrowA: The answer must be a place where sunglasses are stored. Sunglasses are stored in purses. Therefore, the answer is purse (e).” or “Q: The only baggage the woman checked was a drawstring bag, where was she heading with it?→\rightarrowA: The answer must be a place where a drawstring bag is checked. The answer is airport (e).” We believe this question warrants a much more comprehensive study.

One important challenge with models which seek to be interpretable or provide explanations for their reasoning is that of faithfulness. While our primary emphasis is not on the explainability benefits that STaR may bring, there is a fundamental challenge around evaluating explanations and rationales: namely, faithfulness[33,34].[34]describe faithful explanations as those which “accurately [represent] the reasoning process behind the model’s prediction.” While STaR encourages the use of reasoning in rationales which leads the model to correct answers, it is difficult, if not impossible, to ensure that the rationales reflect the model’s internal processing. For example, it is straightforward to imagine the model implicitly selecting a particular answer immediately and then generating a rationale to justify that selected answer. This would allow a model to generate unbiased rationales while selecting answers in a biased way.

The fact that our model outperforms one fine-tuned to directly predict the answers, and ablation studies from papers such as6make it clear that the generation of a rationale before producing an answer non-trivially improves the model’s answer quality. However, it is difficult to evaluate the degree to which any particular answer’s rationale is faithful. However, we note that there problems are not unique to STaR, but are symptomatic of the difficulty of understanding large language models and in particular the rationales generated by large language models.

Appendix HHyperparameters

GPT-J is a 28-layer decoder-only transformer, with an embedding size of 1024, 16 attention heads of dimension 256, and an FFN hidden layer of size 16384. It was pre-trained on the Pile[35], with a vocabulary size of 50.4K

In general, unless otherwise stated, we use a batch size of 8 sequences, each of length 1024. We also use packing, namely, packing the shorter examples to form longer sequences (up to length 1024) to improve TPU utilization. We do not use weight decay, and we train and sample on a single TPU-v3 node. We performed a hyperparameter search over learning rates from10−710^{-7}to10−410^{-4}using the Adam optimizer[36]. We found that10−610^{-6}was consistently the best-performing learning rate.

Appendix IGSM8K Few-shot Prompt

We include the following few-shot prompts for GSM8K, based on the examples in9.

Q:Nataliasoldclipsto48ofherfriendsinApril,andthenshesoldhalfasmanyclipsinMay.HowmanyclipsdidNataliasellaltogetherinAprilandMay?

A:Nataliasold48/2=<<48/2=24>>24clipsinMay.

Nataliasold48+24=<<48+24=72>>72clipsaltogetherinAprilandMay.

####72

Q:Bettyissavingmoneyforanewwalletwhichcosts100\.Bettyhasonlyhalfofthemoneysheneeds\.Herparentsdecidedtogiveher15forthatpurpose,andhergrandparentstwiceasmuchasherparents.HowmuchmoremoneydoesBettyneedtobuythewallet?

A:Inthebeginning,Bettyhasonly100/2=$<<100/2=50>>50.

Betty’sgrandparentsgaveher15*2=$<<15*2=30>>30.

Thismeans,Bettyneeds100-50-30-15=$<<100-50-30-15=5>>5more.

####5

Q:Julieisreadinga120-pagebook.Yesterday,shewasabletoread12pagesandtoday,shereadtwiceasmanypagesasyesterday.Ifshewantstoreadhalfoftheremainingpagestomorrow,howmanypagesshouldsheread?

A:Mailaread12x2=<<12*2=24>>24pagestoday.

Soshewasabletoreadatotalof12+24=<<12+24=36>>36pagessinceyesterday.

Thereare120-36=<<120-36=84>>84pageslefttoberead.

Sinceshewantstoreadhalfoftheremainingpagestomorrow,thensheshouldread84/2=<<84/2=42>>42pages.

####42

Q:Markhasagardenwithflowers.Heplantedplantsofthreedifferentcolorsinit.Tenofthemareyellow,andthereare80%moreofthoseinpurple.Thereareonly25%asmanygreenflowersasthereareyellowandpurpleflowers.HowmanyflowersdoesMarkhaveinhisgarden?

A:Thereare80/100*10=<<80/100*10=8>>8morepurpleflowersthanyellowflowers.

SoinMark’sgarden,thereare10+8=<<10+8=18>>18purpleflowers.

Purpleandyellowflowerssumupto10+18=<<10+18=28>>28flowers.

ThatmeansinMark’sgardenthereare25/100*28=<<25/100*28=7>>7greenflowers.

SointotalMarkhas28+7=<<28+7=35>>35plantsinhisgarden.

####35

Q:Alexisisapplyingforanewjobandboughtanewsetofbusinessclothestoweartotheinterview.Shewenttoadepartmentstorewithabudgetof200andspent30onabutton-upshirt,46onsuitpants,38onasuitcoat,11onsocks,and18onabelt.Shealsopurchasedapairofshoes,butlostthereceiptforthem.Shehas$16leftfromherbudget.HowmuchdidAlexispayfortheshoes?

A:LetSbetheamountAlexispaidfortheshoes.

ShespentS+30+46+38+11+18=S+<<+30+46+38+11+18=143>>143.

Sheusedallbut$16ofherbudget,soS+143=200-16=184.

Thus,AlexispaidS=184-143=$<<184-143=41>>41fortheshoes.

####41

Q:Tinamakes$18.00anhour.Ifsheworksmorethan8hourspershift,sheiseligibleforovertime,whichispaidbyyourhourlywage+1/2yourhourlywage.Ifsheworks10hourseverydayfor5days,howmuchmoneydoesshemake?

A:Sheworks8hoursadayfor18perhoursoshemakes8\*18=<<8*18=144.00>>144.00per8-hourshift

Sheworks10hoursadayandanythingover8hoursiseligibleforovertime,soshegets10-8=<<10-8=2>>2hoursofovertime

Overtimeiscalculatedastimeandahalfsoandshemakes18/hoursoherovertimepayis18\*\.5=<<18*.5=9.00>>9.00

Herovertimepayis18+9=$<<18+9=27.00>>27.00

Herbasepayis144\.00per8\-hourshiftandsheworks5daysandmakes5\*144=$<<144*5=720.00>>720.00

Herovertimepayis27\.00perhourandsheworks2hoursofovertimeperdayandmakes27\*2=<<27*2=54.00>>54.00inovertimepay

2hoursofovertimepayfor5daysmeansshemakes54*5=$270.00

In5daysherbasepayis720\.00andshemakes270.00inovertimepaysoshemakes720\+270=$<<720+270=990.00>>990.00

####990

Appendix JSTaR GSM8K Solutions

We observe some interesting patterns with the GSM8K solutions proposed by the STaR-trained model. Typically, when the solution takes substantially fewer calculation steps than the ground truth, it corresponds to an instance where the model accidentally answered the question correctly despite mistakes in its reasoning. In some cases, however, the model produces simpler solutions than those in the ground truth. One example is shown in Figure8.

Figure 8:An example problem in the training set where STaR derives a significantly simpler solution than the ground truth.