Evolving language compositionality in a frequency-structured meaning space

# Evolving language compositionality in a frequency-structured meaning space
Source: [https://arxiv.org/html/2607.29642](https://arxiv.org/html/2607.29642)
11institutetext:Université de Namur and Vrije Universiteit Brussel, Belgium,11email:fabio\.de\.ponte@vub\.be
22institutetext:University of Bristol, UK
22email:\{bq23138,conor\.houghton,seth\.bullock\}@bristol\.ac\.uk###### Abstract

The iterated learning model was introduced to investigate language evolution: the way in which the characteristic properties of human languages have been shaped, at least partly, by repeated transmission from one language user to another\. The key finding is that language compositionality can arise spontaneously as a consequence of language being passed repeatedly through a language learning bottleneck\. Here we explore how changing the frequency of different meanings, so that some meanings occur much more frequently than others, affects the character of its compositionality\. We find that, as observed in natural languages, high\-frequency meanings can escape the pressure to conform to the grammar that characterizes lower\-frequency meanings\. However, when the frequency structure is instead imposed on parts rather than on whole meaning vectors, the language fails to transmit across generations\. This occurs despite the fact that the most frequent elements are reliably learned\. These results suggest that frequency can shape emergent linguistic structure only when the frequency distribution is defined over form\-meaning units that learners can acquire holistically\. When frequency is instead distributed over smaller units, it fails to support the relational structure required for compositional generalisation, thereby preventing stable language transmission\.

## 1Introduction

Modelling language evolution across generations allows us to examine how linguistic properties emerge and develop over time\. In this paper, we use the semi\-supervised version of the iterated learning model \(ILM\)\[[8](https://arxiv.org/html/2607.29642#bib.bib1),[7](https://arxiv.org/html/2607.29642#bib.bib2)\]to investigate three such properties:*expressivity*,*compositionality*, and*stability*\.

Expressivity refers to the extent to which a linguistic system can distinguish between meanings, or, equivalently, the degree of ambiguity in the language\. Natural languages achieve remarkable expressivity, allowing speakers to convey highly unusual meanings or meanings that may never have been expressed before\. In our model, the measure is operationalised as the proportion of meanings that can be expressed through distinct signals\.

Compositionality refers to the extent to which the mapping between meanings and signals systematically reuses parts of signals to express parts of meanings\. For example, the concept ‘to research’ is consistently expressed by the word ‘research’ across a range of English utterances and contexts, such as ‘I research’, ‘we researched’, ‘they are researching’, ‘a research paper’, and ‘a job in research’\. A defining property of compositionality is therefore that small changes in meaning tend to correspond to small changes in the associated signal\. An interesting feature of natural languages is that the*regularity*of this reuse can vary with the frequency of the meaning\. For example, unlike the relatively rare verb ‘to research’, which follows a highly regular pattern, the very common meaning ‘to be’ is expressed by forms that differ substantially in English, such as ‘I am’, ‘we were’, and ‘they are’\.

Stability is a measure of the similarity between the language taught by the tutor and the language learnt by the pupil\. It is calculated as the proportion of meanings for which the same signal is used by both tutor and pupil\.

The experiment described below is designed to explore how the frequency structure of the meanings that a language must express influences the evolution of the three mentioned properties, with a particular focus on compositionality\.

### 1\.1The semi\-supervised iterated learning model

The iterated learning model \(ILM\) simulates a scenario in which an initially naive pupil acquires language from a tutor and subsequently becomes the tutor for a new naive pupil in the next generation\. At each transmission step, language passes through a bottleneck\[[11](https://arxiv.org/html/2607.29642#bib.bib4),[6](https://arxiv.org/html/2607.29642#bib.bib3),[12](https://arxiv.org/html/2607.29642#bib.bib17)\]: the pupil is not exposed to the entire language but only sees a subset of meaning\-signal pairs\. As a consequence, the learner is required to generalise to the unseen portion of the language\.

In the semi\-supervised ILM used here\[[8](https://arxiv.org/html/2607.29642#bib.bib1)\], each agent is modelled as a neural network, specifically an autoencoder consisting of an encoder and a decoder\. The encoder maps meanings to signals, a process we refer to asencoding, while the decoder performs the inverse operation by reconstructing meanings from signals, a process we refer to asdecoding\. In each case, a particular meaning takes the form of a bitstring and the signal that expresses this meaning in a particular language is also a bitstring of the same length\. An agent’s language is thus the mapping from each possible meaning bitstring to the signal bitstring that expresses it\.

At the start of each generation, a naive pupil is created with no knowledge of language\. Their tutor then generates a set of utterances in their own language, pairing each meaning in a randomly sampled subset of the meaning space with the associated signal used to express this meaning in the tutor’s language\. The pupil’s encoder and decoder are then trained on this subset: for the encoder, meanings serve as inputs and signals as targets, while the reverse is true for the decoder\. Each batch contains a single sample to train the encoder and a single sample to train the decoder, together with a larger number of meanings used to further train the pupil’s whole autoencoder, used as both inputs and targets\.

## 2The interpretation

Compositionality is a property that operates across multiple levels of linguistic structure\. A sentence such as “The brown dog is in the garden” describes a particular situation that can be modified through the replacement of individual words\. For example, changing dog to cat produces “The brown cat is in the garden”, while replacing garden with kitchen yields “The brown dog is in the kitchen”\. In both cases, a local modification produces a limited and systematic change in meaning\. By contrast, replacing several words simultaneously can lead to a substantially different, but still systematic, interpretation, as in “A blue car is in the garage”\. The same principle applies below the word level\. Words such as “actor” and “action” share the root “act”, which contributes a core semantic component, while suffixes such as \-or, \-ion encode additional information\. Arguably, compositionality also extends to higher levels of discourse, where replacing or modifying entire paragraphs can alter the overall meaning and communicative effect of a text in a systematic manner\.

The semi\-supervised ILM investigates the mechanisms underlying compositionality and is not restricted to any particular level of linguistic organisation\. Consequently, bitstrings and individual bits may be interpreted as corresponding to different representational layers\. In this paper, for ease of exposition, we treat each signal bitstring as an object and individual bits as its properties\. Under this interpretation, an entire meaning bitstring represents a situational setting or scenario that must be communicated, while the corresponding signal bitstring is interpreted as a holophrastic utterance\.

Constructivist accounts of language acquisition propose that linguistic knowledge emerges gradually through children’s exposure to, and participation in, language use\[[13](https://arxiv.org/html/2607.29642#bib.bib9),[14](https://arxiv.org/html/2607.29642#bib.bib10),[15](https://arxiv.org/html/2607.29642#bib.bib11),[16](https://arxiv.org/html/2607.29642#bib.bib12),[18](https://arxiv.org/html/2607.29642#bib.bib6),[19](https://arxiv.org/html/2607.29642#bib.bib7),[2](https://arxiv.org/html/2607.29642#bib.bib13)\], rather than being determined by a pre\-specified biological template\. The process relies on pattern recognition: the ability to identify similarities and differences across sensory and motor experiences and to use these regularities to construct categories and cognitive schemas\. More specifically, during early language development, children begin establishing broad associations between the linguistic forms they hear and the situational contexts to which these forms refer\. These associations are known as holophrase constructions, because children initially process them as indivisible wholes rather than analysing them into smaller meaningful components\. Holophrases may correspond to multiword expressions that children interpret as single units, such as “Look at the black dog”\. Subsequently, exposure to expressions such as “Look at the black cat”, referring to a similar situation except for the substitution of the animal, allows children to infer that a subpart of the utterance, such as “dog”, corresponds to a specific component of the situation and can be systematically replaced\. This process marks the beginning of the emergence of compositional structure\.

The usage\-based approach has led to major advances in the computational modelling of language learning within a constructionist framework, particularly through the Fluid Construction Grammar paradigm\[[4](https://arxiv.org/html/2607.29642#bib.bib14),[5](https://arxiv.org/html/2607.29642#bib.bib15),[3](https://arxiv.org/html/2607.29642#bib.bib16)\]\. This line of research has demonstrated that large\-scale, human\-interpretable construction grammars can be learned directly from language use and can capture complex syntactic and semantic regularities\[[20](https://arxiv.org/html/2607.29642#bib.bib8)\]\. While successful in modelling acquisition from existing linguistic data, this approach does not address the broader evolutionary question of how language itself changes over time or how linguistic systems emerged in the first place\[[9](https://arxiv.org/html/2607.29642#bib.bib5)\]\. This is the focus of the ILM\.

As mentioned in the introduction, language is defined as a mapping between meaning vectors and signal vectors\. An individual bitiiin a meaning vector𝐦∈M\\mathbf\{m\}\\in Mwill be referred to as a “fact”\. Similarly, a bitjjin a signal vector𝐬∈S\\mathbf\{s\}\\in Swill be referred to as a “word”\. Strictly speaking, “morpheme” would be a more accurate term, since the signal vector is intended to model any aspect of how meaning is encoded\. However, for convenience, we use “word” throughout\.

In this paper, we interpret meaning bitstrings as representations of situational settings\. For example, consider a set of objects that vary in terms of an ordered set of binary properties such as “large” versus “small” or “black” versus “white”\. Each bit of a meaning bitstring might encode one of these properties\. With an 8\-bit meaning representation, we can map282^\{8\}scenarios: the first bit may indicate size, the second colour and so on\. Likewise, a specific combination of signal bits represents a holophrastic sentence used to convey a specific situational setting, in this case, a particular meaning\. For the first tutor, signal bitstrings are arbitrary with respect to meanings and tend to exhibit no predefined internal structure\. Nothing prevents multiple meanings from being mapped to the same signal and, equivalently, there is no guarantee that every signal is employed within a mapping\. Importantly, there is no guarantee that an arbitrary mapping between meanings and signals can be learned from exposure to a subset of the mapping, because it is likely that there is no information in one part of the mapping that can help to anticipate the structure of some other part\.

## 3Methods

### 3\.1The experiment

At each generation, a subsetBBof meanings is selected and used to create a set of meaning\-signal pairs\. It is then used for supervised training: the encoder is trained onB1=BB\_\{1\}=B, and the decoder onB2B\_\{2\}, a shuffled copy ofB1B\_\{1\}\. Concurrently, a subsetAAof meanings, typically larger thanBB, is sampled from the full meaning space to train the autoencoder in an unsupervised fashion\. Within each epoch, the encoder and decoder each receive\|B\|\|B\|gradient updates\. For every iteration,rradditional autoencoder updates are performed on meanings drawn independently fromAA\. This yields a1:1:r1\{:\}1\{:\}rupdate ratio between encoder, decoder, and autoencoder, so the autoencoder receivesr×\|B\|r\\times\|B\|gradient steps per epoch compared to\|B\|\|B\|for the supervised components\.

In\[[8](https://arxiv.org/html/2607.29642#bib.bib1)\]it is demonstrated that this experimental setup can achieve optimal compositionality, expressivity, and stability\. Figure[1](https://arxiv.org/html/2607.29642#S4.F1)a shows the evolution of all three metrics for a 16\-bit encoder\-decoder model over 50 generations, under the configuration\|A\|=480\|A\|=480,\|B\|=320\|B\|=320, learning rate0\.50\.5, and 20 autoencoder training iterations for every encoder and decoder learning iteration, that is a loop ratio of1:1:201\{:\}1\{:\}20\. Results are averaged over 30 independent trials\. Under these conditions, all three metrics converge to their maximum value within approximately 20 generations\.

Here, we extend this by investigating how language compositionality is shaped when learners are exposed to an uneven distribution of meanings\. Specifically, we address two questions: first, whether meaning vectors, \(representing situational settings\) that occur more frequently during training lead to signals that are more or less compositional; and second, what effects emerge when the uneven meaning distribution applies at the level of individual bits rather than at the level of whole meanings\.

To address these questions, we impose an uneven frequency profile on meanings, firstwithin the whole meaning vector spaceand thenbetween the individual bitsof the meaning vectors\. We use a Zipf distribution to model this frequency profile, reflecting its well\-established role in the frequency structure of natural language\[[10](https://arxiv.org/html/2607.29642#bib.bib19),[17](https://arxiv.org/html/2607.29642#bib.bib20),[21](https://arxiv.org/html/2607.29642#bib.bib21)\]\. Zipf’s law\[[1](https://arxiv.org/html/2607.29642#bib.bib23)\]states that the frequency of each element of the set is inversely proportional to its rank:f​\(r\)∝1/rαf\(r\)\\propto 1/r^\{\\alpha\}, whererris the rank, the position of the element in a frequency ordered list, andα\\alphais an exponent controlling the steepness of the distribution\. Settingα=0\\alpha=0recovers a uniform distribution, in which all meanings are equally frequent, whereas larger values ofα\\alphaconcentrate probability mass on the highest\-ranked meanings\. By sampling meanings according to this distribution during training, we expose the learner to a skewed input in which a few meanings recur frequently and a long tail of meanings are encountered only occasionally or not at all, allowing us to isolate the effect of frequency on the emergence of compositional structure\.

To apply a Zipf distribution over the set ofnn\-bit meaning vectors, the2n2^\{n\}meanings are first shuffled and then each is given a probability proportional to1/rankα1/\\mathrm\{rank\}^\{\\,\\alpha\}, where rank is the new rank position of the meaning\. These probabilities are preserved from generation to generation\.

We also consider a different manipulation, in which the Zipf distribution is applied to the individual bits within each meaning vector\. During training, each meaning vector is generated independently by sampling a bit at each position according to its own probability of taking the value 1, with these probabilities following a Zipf distribution across bit positions\. For example, withα=1\\alpha=1, the probability that the second highest ranking bit is set to 1 is half the probability that the highest ranking bit is set to 1\.

Finally, we consider an intermediate manipulation in which the Zipf distribution is applied to contiguous chunks of the meaning vector, rather than to whole meanings or to individual bits\. Chunk size determines the granularity of the frequency bias: a chunk size ofnncorresponds to sampling complete meaning vectors, while a chunk size of 1 applies the distribution independently to each bit\. Intermediate chunk sizes interpolate between these two extremes\. Chunk\-level sampling also helps clarify the difference between meaning\-vector\-level sampling and bit\-level sampling\. At first sight, applying the skewed distribution to individual bits and applying it to chunks of size 1 appear equivalent\. However, they are very different\. When sampling at the bit level withn=16n=16andα=1\\alpha=1, the highest\-ranked bit has probability0\.50\.5of being11, the second\-ranked bit has probability0\.250\.25, the third\-ranked bit has probability≈0\.17\\approx 0\.17, and so on\. In contrast, when the skew is applied to chunks, the ranking is defined within each chunk, not between them\. For chunks of size 1, each chunk can only take two possible states, causing the distribution to collapse into a binary bias where each bit has probability0\.660\.66of being11and0\.330\.33of being0\. This affects in a different way the distribution of meaning vectors\.

### 3\.2Metrics

To measure the progress of language learning at each generation, we will use three metrics: expressivity, compositionality and stability\.

Expressivityquantifies the diversity of the signals produced by the pupil after learning, and hence the lack of ambiguity in their language\. Specifically, it is the number of distinct signals employed in the language normalised by the number of possible signals available, which is2n2^\{n\}if signals are bitstrings of lengthnn\. A fully expressive language uses every possible signal exactly once; a degenerate one reuses the same signal to encode multiple meanings\.

Compositionalityquantifies the degree to which individual bits of a signal \(the words\) systematically correspond to individual bits \(the facts\) of meaning\. The metric is computed by identifying, for each meaning bit, the signal bit that provides the most reliable information about it, i\.e\., the one associated with the lowest conditional entropy\. Entropy, for a probabilitypp, is defined as:

H​\(p\)=\{−p​log2⁡\(p\)−\(1−p\)​log2⁡\(1−p\)if​p∈\(0,1\)0otherwiseH\(p\)=\\begin\{cases\}\-p\\log\_\{2\}\(p\)\-\(1\-p\)\\log\_\{2\}\(1\-p\)&\\text\{if \}p\\in\(0,1\)\\\\ 0&\\text\{otherwise\}\\end\{cases\}\(1\)Where multiple meaning bits are best encoded by the same signal bit, the conflict is resolved greedily: the best\-fitting pair retains its entropy score, while all remaining competing bits are penalised with an assigned entropy of1\.01\.0\. The final compositionality score is then1−H¯1\-\\bar\{H\}, whereH¯\\bar\{H\}is the mean adjusted entropy across all meaning bits\.

From this procedure, two complementary compositionality metrics are calculated\. The first operates at the vector level: it measures the extent to which a given meaning vector is encoded by a signal vector that conforms to the global grammar defined through the process above\. The second operates at the bit level: it measures the extent to which a single signal bit systematically encodes any single meaning bit across the entire language\. The pseudocode for these measures is presented in the Appendix\.

Stabilitymeasures the extent to which the pupil’s language matches the tutor’s language after learning\. As with compositionality, two complementary metrics are calculated\. The first operates at the vector level: at each generation, it is defined as the proportion of meanings, across the entire meaning space, for which the pupil exactly reproduces the tutor’s encoding, providing a snapshot of how faithfully the language as a whole is transmitted \(Figure[1](https://arxiv.org/html/2607.29642#S4.F1), left panels, and Figure[2](https://arxiv.org/html/2607.29642#S4.F2), panela\)\. The second operates at the bit level: for each meaning bit position, the bit stability score is the fraction of generations, over the whole chain, in which that position is encoded by the same carrier signal bit, identified as its most frequent one; low scores thus indicate a meaning bit that frequently switched between carriers \(Figure[3](https://arxiv.org/html/2607.29642#S4.F3), right panel\)\.

As a baseline, we run the model withα=0\\alpha=0, corresponding to a uniform frequency distribution over meanings during training\. Figure[1](https://arxiv.org/html/2607.29642#S4.F1)shows that, under this condition, compositionality, expressivity and stability all rapidly converge to their maximum values and sentence\-level compositionality rises uniformly for all sentences, reproducing previously reported results\[[11](https://arxiv.org/html/2607.29642#bib.bib4),[8](https://arxiv.org/html/2607.29642#bib.bib1)\]\.

## 4Results

Figure[1](https://arxiv.org/html/2607.29642#S4.F1)c shows that, when a pupil is exposed to an uneven distribution of samples from the tutor’s language, a compositional language arises and remains stable for the rest of the chain: at the aggregate level compositionality, expressivity and stability all climb within the first 10 generations and then plateau near 1 around generation 30\. However, Figure[1](https://arxiv.org/html/2607.29642#S4.F1)d shows that the most frequently communicated meanings \(for example, rank≤1\\leq 1, dark blue line\) achieve compositionality more slowly than those that are communicated more rarely\. Meanings that occur more frequently tend to be associated with holophrastic sentences, while meanings encountered less often must be encoded compositionally via systematic regularities within the language\.

![Refer to caption](https://arxiv.org/html/2607.29642v1/x1.png)Figure 1:Flat vs\. whole\-meaning Zipf distribution:Language evolution in the semi\-supervised iterated learning model for a 16\-bit encoder\-decoder neural network over 30 trials of 50 generations, under the configuration\|A\|=480\|A\|=480,\|B\|=320\|B\|=320, loop ratio1:1:201\{:\}1\{:\}20, and learning rate0\.50\.5\. Results are shown for a flat distribution \(α=0\\alpha=0, upper two panels\) and a Zipf distribution over meaning vectors \(α=1\\alpha=1, lower two panels\)\. All results are averaged over 30 independent trials\. The two left\-hand panels show compositionality, expressivity and stability over generations\. The two right\-hand panels show per\-meaning compositionality grouped \(averaged\) intolog2\\log\_\{2\}bins of rank\. The first bins \(high\-rank\) correspond to the most frequent meanings, while later bins \(low\-ranks\) correspond to increasingly rare meanings\. Forα=0\\alpha=0all meanings are sampled uniformly, so the bins are equivalent and the curves coincide, rising rapidly to near 1\. Forα=1\\alpha=1, the compositionality of signals conveying the most frequently communicated meanings rise more slowly, only approaching one much later in language evolution\.![Refer to caption](https://arxiv.org/html/2607.29642v1/x2.png)Figure 2:Bit\-level Zipf distribution:Language evolution under bit\-level Zipf distribution \(α=1\\alpha=1\), where the frequency skew is applied between bit positions rather than within the meaning vector space\. In all other respects the simulations follow the same scheme and parameters as those depicted in Figure[1](https://arxiv.org/html/2607.29642#S4.F1)\. In panelscandd, the signal bits are ranked by how often they are set to 1 across all signals used in the pupil’s language\.Then we turn to what happens when the uneven distribution is applied to the bit level: the results withα=1\\alpha=1are shown in Figure[2](https://arxiv.org/html/2607.29642#S4.F2)\. Panelashows that the chain fails to converge: stability remains close to zero across generations, while expressivity and compositionality fluctuate around 0\.4 without reaching a stable regime\. Because the Zipf distribution is applied at the level of bit positions, the per\-sentence analysis \(panelb\) reveals no clear structure\. However, the per\-bit frequency analysis uncovers a pronounced gradient \(paneld\); higher\-ranked meaning bits tend to exhibit greater compositionality, as illustrated in panelc\(per generation\) and in Figure[3](https://arxiv.org/html/2607.29642#S4.F3)\(per rank\)\. Finally, although the language as a whole does not reach stability, the highest\-ranked bits exhibit substantially greater stability than lower\-ranked ones\.

What appears to be happening is that pupils consistently fail to reproduce the exact signals used by their tutors, so language stability remains close to zero\. Nevertheless, the chain of independently emerging languages do share some common internal structure\. In particular, the highest\-ranked meaning bits \(1–3\) are consistently mapped to signal bits with high compositionality, indicating that these signal bits reliably track the corresponding meaning bits\. These signal bits therefore carry the greatest amount of information\. By contrast, lower\-ranked meaning bits are mapped to signal bits with substantially lower compositionality\. This is likely because, under the Zipfian distribution, low\-ranked meaning bits are almost always observed as ‘0’ during training, providing little pressure for learners to encode them systematically\. Although the identity of the high\-ranking meaning bits varies randomly across different simulations, this pattern is remarkably consistent across independently evolved languages\. The signal bit that is most frequently set to ‘1’ typically takes this value about 50% of the time and also exhibits the highest compositionality, while progressively rarer signal bits tend to exhibit progressively lower compositionality\.

![Refer to caption](https://arxiv.org/html/2607.29642v1/x3.png)Figure 3:Per\-bit compositionality and stability by frequency rank:the highest\-ranked meaning bits are associated \(left panel\) with the most compositional signal bits\. The four highest\-ranked bits exhibit higher compositionality than bits ranked fifth and below, although rank 1 shows slightly lower compositionality than rank 2\. The highest\-ranked bits exhibit substantially greater stability \(right panel\) than lower\-ranked ones\.Finally, we look at what happens when the uneven distribution is applied at the chunk\-level\. In Figure[4](https://arxiv.org/html/2607.29642#S5.F4)we can see that, as expected, with a chunk size of 16, the effect almost perfectly overlaps with sentence\-level sampling: the first bin exhibits the lowest compositionality, while the remainder of the curve tends to remain flat\. With a chunk size of 8, compositionality initially remains flat and then gradually decreases with rank\. In contrast, when chunks have size 1, 2, or 4, compositionality shows a more pronounced decrease with rank\.

## 5Discussion

Our results show that the effect of frequency on emergent compositionality depends critically on*what*the frequency structure is imposed upon\. When a Zipf distribution is placed over whole meanings the model reproduces a well\-known property of natural language: the most frequent meanings converge more slowly to the systematic grammar and remain less compositional than rarer ones \(Figure[1](https://arxiv.org/html/2607.29642#S4.F1)d\)\. This suggests that a meaning that recurs in nearly every training batch is encountered often enough to be fixed as an idiosyncratic, holophrastic whole, and is under less pressure to conform to the regularities that organise the rest of the language \(like the irregular conjugation of the very frequent verb “to be”\)\. A rare meaning, by contrast, is seen too seldom to be memorised individually; the only way it can survive repeated transmission through the bottleneck is by inheriting and thereby conforming to the systematic meaning–signal mapping \(like the regular conjugation of the infrequent verb “to research”\)\.

Chunk\-level sampling clarifies that this effect is a property of skew over whole meanings, rather than of skew*per se*\. As shown in Figure[4](https://arxiv.org/html/2607.29642#S5.F4), with a chunk size of 16, the signal associated with the highest\-frequency meaning vector exhibits the lowest compositionality\. In contrast, with a chunk size of 1, signals associated with the highest\-frequency meaning vectors are the most compositional\. Moreover, under bit\-level sampling \(right panel\), emergent languages exhibit a qualitatively different pattern: overall compositionality is substantially lower, and lower\-frequency meanings are mapped to signals with higher compositionality\.

![Refer to caption](https://arxiv.org/html/2607.29642v1/x4.png)Figure 4:Compositionality bylog2\\log\_\{2\}frequency\-rank bin:meanings are grouped into logarithmic frequency\-rank bins, with bin 0 containing the highest\-frequency meaning, bin 1 the next highest\-frequency group, and so on\. With a chunk size of 16, the effect almost perfectly matches whole\-vector\-level sampling: the first bin displays the lowest compositionality, while the rest of the curve remains approximately flat\. For a chunk size of 8, compositionality is initially stable but gradually declines with increasing rank\. By contrast, when chunks are smaller \(sizes 1, 2, or 4\), compositionality exhibits a decrease across rank\.Moreover, when a Zipf distribution is imposed over meaning bits, no transmissible language emerges\. Aggregate stability remains close to zero across generations \(Fig\.[2](https://arxiv.org/html/2607.29642#S4.F2), panela\), indicating that no shared mapping is consistently transmitted over time\. Nevertheless, high\-rank meaning bit positions \(those most frequently taking the value 1\) become relatively stably associated with signal bits and achieve higher levels of per\-bit compositionality, whereas low\-rank positions switch more often between carriers and remain only weakly systematic \(Fig\.[3](https://arxiv.org/html/2607.29642#S4.F3)\)\. As a result, the language develops locally stable and compositional regularities without ever consolidating into a globally stable system\.

## 6Conclusion

We use a semi\-supervised iterated learning model to investigate how imposing frequency structure on a meaning space shapes the compositionality of emergent languages\. Our central finding is that the effect of frequency skewness does not depend on its presence*per se*, but on the level of representation at which it is introduced\. When entire configurations are presented frequently, they tend to be memorised as fixed units and associated with holophrastic phrases\. In contrast, when individual elements occur frequently across different configurations, independently of their context, they become strongly associated with lexical forms but lose their role within a broader structural system\. The consequence is that no stable language is transmitted across generations\. Although the high\-rank meaning bit positions may become locally more compositional by acting as reliable carriers, low\-rank positions remain largely uninformative\. The resulting system lacks the relational structure required for compositional generalisation\.

These findings are consistent with usage\-based accounts of language, in which grammatical knowledge emerges from recurring constructions rather than from isolated symbol–referent associations\. A frequent element learned independently may acquire a stable form, but successful language transmission requires preserving the relationships that bind elements into larger meaningful configurations\. Frequency therefore supports the emergence of structure only when it shapes units that learners can acquire holistically; when applied to isolated sub\-symbolic features, it fails to provide the organisation needed for generalisation beyond limited exposure\.

## 7Further work

In this work, we addressed how different distributions of exposure to language influence its acquisition and modification across generations, and how compositionality can emerge in the process\. We did not, however, address how this compositional mechanism could develop into the hierarchical structures characteristic of natural languages\.

A possible direction for future work is to move beyond frequency distributions and introduce*dependencies*among meaning components\. In the experiments presented here, each bit of a meaning vector was sampled independently\. In natural communication, however, meaning components are not independent but tend to co\-occur in structured ways: for example, only certain entities are likely to be hairy, and only some are likely to have tails that wag\.

We therefore propose constraining the generation of meanings during training by introducing dependencies of this kind\. A bit may take the value 1 only when a set of dependent bits satisfies a specific configuration\. The key question is whether the ILM will favour languages whose compositional structure captures these dependencies, such that a hierarchically organised meaning space gives rise to a hierarchically organised language purely through transmission across a learning bottleneck\.

## References

- \[1\]L\. Aitchison, N\. Corradi, and P\. E\. Latham\(2016\)Zipf’s law arises naturally when there are underlying, unobserved variables\.PLoS computational biology12\(12\),pp\. e1005110\.Cited by:[§3\.1](https://arxiv.org/html/2607.29642#S3.SS1.p4.5)\.
- \[2\]B\. Ambridge and E\. Lieven\(2015\)A constructivist account of child language acquisition\.The handbook of language emergence,pp\. 478–510\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p3.1)\.
- \[3\]K\. Beuls and P\. Van Eecke\(2023\)Construction grammar and artificial intelligence\.arXiv preprint arXiv:2309\.00135\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p4.1)\.
- \[4\]K\. Beuls and P\. Van Eecke\(2023\)Fluid construction grammar: state of the art and future outlook\.InProceedings of the First International Workshop on Construction Grammars and NLP \(CxGs\+ NLP, GURT/SyntaxFest 2023\),pp\. 41–50\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p4.1)\.
- \[5\]K\. Beuls and P\. Van Eecke\(2024\)Humans learn language from situated communicative interactions\. what about machines?\.Computational Linguistics50\(4\),pp\. 1277–1311\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p4.1)\.
- \[6\]L\. Brace, S\. Bullock, and J\. Noble\(2015\)Achieving compositional language in a population of iterated learners\.MIT Press\.Cited by:[§1\.1](https://arxiv.org/html/2607.29642#S1.SS1.p1.1)\.
- \[7\]S\. Bullock and C\. Houghton\(2024\)Modeling language contact with the iterated learning model\.InArtificial Life Conference Proceedings 36,pp\. 54\.Cited by:[§1](https://arxiv.org/html/2607.29642#S1.p1.1)\.
- \[8\]J\. Bunyan, S\. Bullock, and C\. Houghton\(2025\)An iterated learning model of language change that mixes supervised and unsupervised learning\.PLOS Complex Systems2\(3\),pp\. e0000030\.Cited by:[§1\.1](https://arxiv.org/html/2607.29642#S1.SS1.p2.1),[§1](https://arxiv.org/html/2607.29642#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.29642#S3.SS1.p2.4),[§3\.2](https://arxiv.org/html/2607.29642#S3.SS2.p6.1)\.
- \[9\]A\. Cangelosi and D\. Parisi\(2002\)Computer simulation: a new scientific approach to the study of language evolution\.InSimulating the evolution of language,pp\. 3–28\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p4.1)\.
- \[10\]K\. George\(1949\)Zipf\. human behavior and the principle of least effort\.Addison\-Wesley\.Cited by:[§3\.1](https://arxiv.org/html/2607.29642#S3.SS1.p4.5)\.
- \[11\]S\. Kirby\(2002\)Learning, bottlenecks and the evolution of recursive syntax\.Linguistic Evolution Through Language Acquisition: Formal and Computational Models,pp\. 173–204\.Cited by:[§1\.1](https://arxiv.org/html/2607.29642#S1.SS1.p1.1),[§3\.2](https://arxiv.org/html/2607.29642#S3.SS2.p6.1)\.
- \[12\]J\. Lind and A\. Jon\(2025\)A sequence bottleneck for animal intelligence and language?\.Trends in cognitive sciences29\(3\),pp\. 242–254\.Cited by:[§1\.1](https://arxiv.org/html/2607.29642#S1.SS1.p1.1)\.
- \[13\]B\. MacWhinney\(1975\)Rules, rote, and analogy in morphological formations by hungarian children\.Journal of Child Language2\(1\),pp\. 65–77\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p3.1)\.
- \[14\]B\. MacWhinney\(1982\)Basic syntactic processes\.Language Acquisition1,pp\. 73–136\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p3.1)\.
- \[15\]A\. M\. Peters\(1983\)The units of language acquisition\.Vol\.1,CUP Archive\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p3.1)\.
- \[16\]J\. M\. Pine and E\. V\. Lieven\(1997\)Slot and frame patterns and the development of the determiner category\.Applied Psycholinguistics18\(2\),pp\. 123–138\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p3.1)\.
- \[17\]A\. I\. Saichev, Y\. Malevergne, and D\. Sornette\(2009\)Theory of zipf’s law and beyond\.Vol\.632,Springer Science & Business Media\.Cited by:[§3\.1](https://arxiv.org/html/2607.29642#S3.SS1.p4.5)\.
- \[18\]M\. Tomasello\(2003\)Constructing a language: a usage\-based theory of language acquisition\. cambdridge\.Massachusetts: Harvard University Press\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p3.1)\.
- \[19\]M\. Tomasello\(2009\)The usage\-based theory of language acquisition\.InThe Cambridge handbook of child language,pp\. 69–87\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p3.1)\.
- \[20\]P\. Van Eecke and K\. Beuls\(2026\)A method for learning large\-scale computational construction grammars from semantically annotated corpora\.arXiv preprint arXiv:2603\.12754\.Cited by:[§2](https://arxiv.org/html/2607.29642#S2.p4.1)\.
- \[21\]R\. E\. Wyllys\(1981\)Empirical and theoretical bases of zipf’s law\.Cited by:[§3\.1](https://arxiv.org/html/2607.29642#S3.SS1.p4.5)\.

## 8Appendix

The pseudocode below describes three compositionality measures that capture complementary aspects of language structure\. The bit\-level metric identifies the most reliable one\-to\-one correspondences between meaning bits and signal bits by measuring the consistency of each possible mapping through conditional entropy, thereby extracting a global bit\-level grammar\. The vector\-level metric evaluates how well individual meaning vectors conform to this grammar by measuring the proportion of bit\-level rules correctly expressed by their corresponding signals\. Finally, the whole\-language metric aggregates these local correspondences into a single global score, quantifying the extent to which the entire language can be described by a consistent compositional encoding\.

Algorithm 1Bit\-Level Compositionality1:foreach pair \(meaningBit

ii, signalBit

jj\)do⊳\\trianglerightTwo\-sided binary entropy function

2:

p1←P​\(signalj=1∣meaningi=1\)p\_\{1\}\\leftarrow P\(\\text\{signal\}\_\{j\}=1\\mid\\text\{meaning\}\_\{i\}=1\)
3:

p0←P​\(signalj=1∣meaningi=0\)p\_\{0\}\\leftarrow P\(\\text\{signal\}\_\{j\}=1\\mid\\text\{meaning\}\_\{i\}=0\)
4:

entropy​\(i,j\)←12​H​\(p1\)\+12​H​\(p0\)\\text\{entropy\}\(i,j\)\\leftarrow\\frac\{1\}\{2\}H\(p\_\{1\}\)\+\\frac\{1\}\{2\}H\(p\_\{0\}\)
5:endfor

6:⊳\\trianglerightGreedy one\-to\-one assignment across whole language

7:whileunassigned bits remaindo

8:Pick

\(i∗,j∗\)\(i^\{\*\},j^\{\*\}\)with lowest

entropy​\(i,j\)\\text\{entropy\}\(i,j\)
9:Mark

i∗i^\{\*\}and

j∗j^\{\*\}as used

10:

compositionalityj∗←1−entropy​\(i∗,j∗\)\\text\{compositionality\}\_\{j^\{\*\}\}\\leftarrow 1\-\\text\{entropy\}\(i^\{\*\},j^\{\*\}\)
11:

correlationj∗←sign​\(2​p1−1\)\\text\{correlation\}\_\{j^\{\*\}\}\\leftarrow\\text\{sign\}\(2p\_\{1\}\-1\)
12:endwhile

Algorithm 2Vector\-Level Compositionality1:Bit\-level grammar \(Algorithm[1](https://arxiv.org/html/2607.29642#alg1)\) already computed

2:foreach meaning

ccdo

3:

rank←Zipf rank of​c\\text\{rank\}\\leftarrow\\text\{Zipf rank of \}c⊳\\trianglerightfrom shuffled permutation

4:

bin←⌊log2⁡\(rank\+1\)⌋\\text\{bin\}\\leftarrow\\lfloor\\log\_\{2\}\(\\text\{rank\}\+1\)\\rfloor
5:

6:foreach \(signalBit

jj, meaningBit

ii\) pair in the grammardo

7:

expected←meaningi​\[c\]\\text\{expected\}\\leftarrow\\text\{meaning\}\_\{i\}\[c\]
8:count whether

pupilSignalj​\[c\]\\text\{pupilSignal\}\_\{j\}\[c\]matchesexpected

9:endfor

10:

comp​\[c\]←\\text\{comp\}\[c\]\\leftarrowfraction of rules that matched

11:

12:endfor

Algorithm 3Whole\-Language Compositionality1:foreach pair \(meaningBit

ii, signalBit

jj\)do

2:

p1←P​\(signalj=1∣meaningi=1\)p\_\{1\}\\leftarrow P\(\\text\{signal\}\_\{j\}=1\\mid\\text\{meaning\}\_\{i\}=1\)
3:

p0←P​\(signalj=1∣meaningi=0\)p\_\{0\}\\leftarrow P\(\\text\{signal\}\_\{j\}=1\\mid\\text\{meaning\}\_\{i\}=0\)
4:

entropy​\(i,j\)←12​H​\(p1\)\+12​H​\(p0\)\\text\{entropy\}\(i,j\)\\leftarrow\\frac\{1\}\{2\}H\(p\_\{1\}\)\+\\frac\{1\}\{2\}H\(p\_\{0\}\)
5:endfor

6:⊳\\trianglerightGreedy assignment with collision resolution

7:foreach meaningBit

iido

8:

j∗←arg⁡minj⁡entropy​\(i,j\)j^\{\*\}\\leftarrow\\arg\\min\_\{j\}\\ \\text\{entropy\}\(i,j\)
9:if

j∗j^\{\*\}already claimed by a better\-fit meaningBitthen

10:

adjustedEntropy​\[i\]←1\.0\\text\{adjustedEntropy\}\[i\]\\leftarrow 1\.0⊳\\trianglerightpenalty

11:else

12:

adjustedEntropy​\[i\]←entropy​\(i,j∗\)\\text\{adjustedEntropy\}\[i\]\\leftarrow\\text\{entropy\}\(i,j^\{\*\}\)
13:endif

14:endfor

15:

16:

compositionality←1−1n​∑iadjustedEntropy​\[i\]\\text\{compositionality\}\\leftarrow 1\-\\dfrac\{1\}\{n\}\\sum\_\{i\}\\text\{adjustedEntropy\}\[i\]
17:returncompositionality

![Refer to caption](https://arxiv.org/html/2607.29642v1/x5.png)Figure 5:Distribution skewed at the meaning\-vector level\. Left: expressivity, compositionality, and stability for different values ofα\\alpha\. Right: compositionality as a function oflog2\\log\_\{2\}frequency\-rank bins\. Meanings are grouped into logarithmic frequency\-rank bins, with bin 0 containing the highest\-frequency meaning, bin 1 the next highest\-frequency group, and so on\.![Refer to caption](https://arxiv.org/html/2607.29642v1/x6.png)Figure 6:Distribution skewed at the chunk level\. Left: expressivity, compositionality, and stability for different chunk sizes\. Right: compositionality as a function oflog2\\log\_\{2\}frequency\-rank bins for each chunk size\. Meanings are grouped, at the level of complete meaning vectors, into logarithmic frequency\-rank bins: bin 0 contains the highest\-frequency meaning, bin 1 the next highest\-frequency group, and so on\.