Syntax-semantics interactions – seeking evidence from a synchronic analysis of 38 languages

Corpus study

Under the view that syntax facilitates semantics, it is plausible to assume that on average writers and speakers will deliver difficult semantics via simpler syntax and simplify semantics in the presence of difficult syntax in order to ensure a healthy level of communicative complexity - price the risk of misunderstanding. That is, we expect a tendentially inverse relationship between semantic and syntactic complexity. Whether real data corroborates this expectation is the research question framing this report. We elaborate it through two competing hypotheses.

H0: There is no systematic interaction between semantic complexity and syntactic complexity.

H1: There is an inverse relationship between semantic complexity and syntactic complexity: as lexical semantics complexity increases, syntactic complexity decreases, and vice versa.

A repository containing all files relevant for replication of this research can be found online.^33,34

Corpora and languages

To test the hypotheses, we used the corpora made available through the Universal Dependencies (UD) project^a, accessed in June 2020. The UD corpora are, especially for large languages, a mix of different registers and domains, ranging from blog posts to legal texts, the vast majority of which comes from contemporary sources. Critically, the corpora contain manually annotated syntactic information in the form of dependency relations. The fact that syntactic annotation was done manually and not automatically is beneficial for our goals, since it removes the risk of systematic mistakes introduced by automatic syntactic annotation. For any language with more than one (sub)corpus, we merged the corpora and only included languages for which we have more than 0.1m tokens in total. Further, we only considered languages that have their lemmas specified, which leaves us with the languages we list in Table 1.

Semantic measure

We measured semantic complexity with a canonical and straightforward measure: the sum of the log frequencies of a sentence’s lemmas^b. The inverse correlation between word frequency and its processing difficulty is widely known,^15,16 while lemmas and inflected forms do not appear to behave very differently in terms of distribution.¹⁷ If readers encounter a particular word several times, they become more effective in processing it both at the orthographic and semantic levels; and the more they encounter the word, the more effective they become. Thus, all other things being equal, a sentence’s sum of log lemma frequency should be a proxy for its processing cost. The reason why we did not opt for sequence surprisal or perplexity, for example, is that sequences longer than one word necessarily reflect syntactic behaviours. That is, at an n-gram level, surprisal and perplexity contain both semantic and syntactic information. They are functionally equivalent to the frequency measure at a unigram level, so we use the latter as our semantic measure.

Syntactic measure

We used the UD framework to measure syntactic complexity. The UD framework expresses syntactic relations through dependencies: each element depends on another element, its head.¹⁸ In UD, the head is the semantically most salient element, and the dependent modifies the head. The top-level head is the root of a sequence, typically the main verb of the matrix clause. Critical to our syntactic measure is the notion of dependency length. For each element, we can calculate the distance to its head, where each intervening element increases the distance by one, and for each sentence, we can add up the distances for a sentence’s elements, which results in the sum of dependency lengths. Figure 1 illustrates a dependency analysis; the sum of distances in that example is 6 (2+1+2+1). The measure is taken from the literature.^2,19–21 We consider this an intuitive measure: all other things being equal, sentences with shorter dependency lengths will be easier to process.^2,22

Figure 1. Visualising a simple sequence in the Universal Dependencies framework, including dependency distances (red numerals below a dependency arch/edge).

(Created with spaCy: https://spacy.io/.)

Controlling for confounds

Since the relationship between sentence length and summed dependency distances is not necessarily linear,² there is no simple way to normalise for sentence length. Thus, to remove the risk of spurious sentence length effects, we computed our correlation only at the median sentence length per language, excluding punctuation.^c For example, the median sentence length for English is 12 tokens, while for Dutch, it is 6. Another aspect we controlled for is word length. Word length is known to roughly correlate with information content,²³ so for any median token length, we further extracted the median character length and considered a range of +/-10%. For instance, the median character length for English sentences of 12 tokens is 47 characters, so we only considered sentences with 12 tokens that are within a range of 42 to 52 characters. Since the median sentence length is relatively short for some languages, we also analysed longer sentences, viz. twice the median sentence length, which is the ‘long’ condition.

Analysis

We calculated the Pearson correlation coefficient between the summed logged frequencies and the summed dependency distances for each language, using R.²⁴ For our two measures, positive correlation coefficients are evidence for H1: We expect low frequencies (high semantic complexity) to correlate with shortened dependency lengths (low syntactic complexity).

Semantic measure weakness

We used a canonical, easy to compute, and corpus-dependent semantic measure, amply verified in linguistics, information theory, and cognitive science. While it is a proxy for semantic complexity, as such it leaves room for improvement, we find it unlikely that the mixed results we observe are entirely due to our measure of complexity.

Syntactic measure weakness

Our syntactic measure depends on the UD framework’s validity, which aims to be language universal.¹⁸ We cannot exclude that there are inconsistencies in theoretical choices across languages, e.g., how prepositions or relativizers are attached.²⁵ An in-depth qualitative analysis is needed to follow up on this, and those choices might have contributed to some variance. The extensive amount of work gone into the formulation of the UD framework, together with the efforts made to ensure the consistency of UD syntactic annotations across different languages, makes us think that internal inconsistencies should not shift the mean to a degree such that it conceals a real effect.

Domain/register effects

Specific registers, or even domains, might be characterised by an unusual complexity of both the semantic and the syntactic side, or have a technical vocabulary that becomes “simple” to their usual readers, leaving room for more syntactic complexity. However, the fact that we draw our semantic measures from the corpora themselves should mitigate such an effect: if a rare semantic unit frequently appears in one of our corpora, its frequency increases, and its complexity level decreases. While differences in the corpora’s makeup could cause parts of the observed variance, there is no apparent reason why they should have caused the mean to shift towards zero.

Further factors

Other factors that we could not account for could be at play. This concerns pragmatic aspects²⁶ or even morpho-phonetic aspects of the lexicon.^27–30 Another source of noise in our data could be that syntactic structures can increase in complexity to a good extent without requiring any semantic simplification or even favoring semantic complexity.³¹

We have identified some factors that could cause the observed variance. However, if the interaction between syntax and semantics was as strong as we assumed, then there should have been a measurable shift towards a positive mean in most languages. That means that there should have been some signal from the data at hand “shining through” the noise. Nonetheless, the results are not evidence of our H0: the absence of evidence is not the evidence of absence.³² A practical takeaway could be that the interaction between syntax and semantics is more complex and more subtle than we sometimes conceived it to be, certainly at a synchronic level.

Underlying data

Zenodo: Syntax-semantics interactions – seeking evidence froma synchronic analysis of 38 languages: dataset repository. https://doi.org/10.5281/zenodo.4542536.

This project contains the following underlying data:

Apache 2.0 license, with the Universal Dependency Consortium holding the copyright.

Extended data

Zenodo: Syntax-semantics interactions – seeking evidence froma synchronic analysis of 38 languages: scripts repository. https://zenodo.org/record/4643152.

This project contains the following extended data:

The scripts are available under a CC0 license (”No rights reserved”). While the scripts modify the data sets, the modified data sets continue to be subject to the same Apache 2.0 license as the original data sets.