In language comparison, there is a long history of using concept-based wordlists to get insights into the degree of similarity between languages, going back at least to Morris Swadesh (Swadesh 1950). For these purposes, words from different languages that share the same meaning are compared, either manually or with computational methods. The latter have the advantages of being both faster and more consistent. However, there are also limits to what computer-based methods can detect for the time being.

One of the biggest problems in this context is that none of the currently available methods for automatic cognate detection can infer partial cognates directly if no information on morpheme boundaries is provided by the user. As a result, if morpheme boundaries are missing and morphological differences are frequent in the data one wants to investigate, automatic cognate detection can be seriously hampered (List, Greenhill, and Gray 2017).

• 

• 

Figure 1: Screenshots from EDICTOR (List 2017) contrasting automatic sound alignment without and with morphological annotation.

For example, Latin sōl /soːl/ and French soleil /sɔlɛj/ (both ‘sun’) would be considered more similar by an algorithm if it “knew” that the French word can be analyzed as consisting of two morphemes, and that it should only consider the first one, i. e. /sɔl/. It might also be the case that the same morphemes appear as allomorphs within a given language, e. g. due to vowel harmony or umlaut, but currently there are no automatic methods to detect allomorphic variation inside a language.

In order to tackle these problems one could either try to use existing methods for automatic morpheme boundary detection (Creutz and Lagus 2005) to preprocess our linguistic datasets, or one could increase the information that is made available to the algorithms in order to enable them to compare morphemes instead of full words (Hill and List 2017; List, Lopez, and Bapteste 2016).

Given that automatic morpheme detection does not yield reliable results at the moment, especially not for small wordlists as they are typically used in historical linguistics, we are currently trying to formalize the second approach by developing standards and examples for best practice to augment wordlists with morphological information. In doing so, we do not only hope to optimize our annotation frameworks in order to establish a workflow for future annotations, but also to create example annotations for future tests of novel methods for automatic morpheme detection.

Our goal here is to explore the possibilities of integrating any kind of morphological structures, be they synchronic or diachronic, within a consistent annotation framework. In the following, I want to give a short overview on some new ideas we came up with during the last months.

Preliminaries of annotation

We established a standardized way of how the data needs to be prepared and formatted, following the standard input formats of LingPy (List, Greenhill, and Forkel 2017) and EDICTOR (List 2017), which are largely compatible with the format specifications laid out by the CLDF initiative (Forkel et al. 2018). This is necessary to guarantee the comparability of the results, the re-usability of helper scripts, the integrability with existing annotation solutions, and machine-readability in general. The format specification requires that the wordlists are turned into a spreadsheet saved as TSV-file with a header row and one row for each wordform. The column containing this data is titled FORM. For each other kind of information, a distinct column is used. It needs to be made sure that each field contains only a single string of characters (including spaces if phrases occur among the data), and not multiple forms. If the data with which one is working does not follow these requirements or if other corrections to it are needed, they need to be adjusted accordingly. The original data is always preserved in the column VALUE, so nothing is lost (unless one excludes a given word form completely from the comparison). Comments and notes explaining some specific aspects of the entries under question are placed in the column NOTE.

The first column is called ID and contains a consecutive integer ID that needs to be larger than zero. This is necessary in order to further refine the data with the help of EDICTOR (List 2017) and has the additional advantage of making it easy to put the rows back into the original order if they were reordered. Since several languages might be included in one file and we want to allow for language-external comparison as well, the language of the data is specified in the column DOCULECT (following the terminology of Good and Cysouw 2013). Wordlists are typically created with the use of elicitation glosses and those are included in a column given the header CONCEPT.

ID	DOCULECT	CONCEPT	VALUE	FORM	NOTE
147	Old High German	the goat	geiz	geiz
148	Old High German	the he-goat	boc, buc	boc
149	Old High German	the kid	zickîn	zickîn
150	Old High German	the horse	(h)ros	hros
151	Old High German	the stallion	hengist	hengist	wrong meaning
152	Old High German	the stallion		reino	correction
153	Old High German	the mare	meriha	meriha
154	Old High German	the foal or colt	folo	folo
155	Old High German	the donkey	esil	esil
156	Old High German	the mule	mûl	mûl

Table 1: Entries from the Old High German wordlist of the World Loanword Database (Haspelmath 2009) prepared for further annotation.

Especially when using one’s own data it is highly recommended to also add a column with the Concepticon-ID (List et al. 2018, see also https://concepticon.clld.org) corresponding to the elicitation gloss in order to make sure that later comparison across different datasets will be facilitated.

Segmented IPA-transcriptions

After these preliminary preparations, the words are turned into sequences of IPA-characters. This facilitates comparing languages since the original spelling is often quite idiosyncratic and several computational analyses which one might want to use the data for are only possible if it is provided in IPA. In order to create IPA from orthographic sources, there are different possibilities: One could convert all orthographic entries to their IPA version manually, one could try to extract the data from an available database that lists phonetic transcriptions for orthographic entries (as many modern dictionaries do), or one could write an orthography profile (see Moran and Cysouw 2018, and for an introduction to how to create this kind of file see this tutorial) and convert the data with help of the Python implementation provided by the segments package.

Unfortunately, orthography profiles only work for very regular orthographies like those used by field workers who for convenience transcribe languages in alphabetic letters. For orthographies with a longer history, orthography profiles often cannot be used, since the orthography does not contain full information on the pronunciation. For example, in German, vowel length is not always marked by the spelling. Therefore for these kinds of spelling systems, one of the other options or a combination thereof needs to be chosen.

The IPA-column is named TOKENS. This is the default name for it in EDICTOR. Also, the individual sounds (all sounds that are judged as a sound unit by the researcher, i.e., including affricates and diphthongs, depending on the language under investigation) are separated by spaces.

Standardizing annotation practice

After all these previous (and at times tedious) steps of data preparation, we can finally start to annotate the morphemes in our data by marking morpheme borders with a plus sign. Here again this can either be done manually or by using some computer-assisted approach. If common morphemes are known to the researcher, a very simple (but in our experience also efficient) approach for marking at least a larger part of the morphemes semi-automatically is to use search and replace functionalities (in combination with regular expressions if needed). In this way, nearly all instances of, for example, the German prefix {ver} can be easily annotated by searching for ^f ɛ r and replacing it with f ɛ r + (the ^ marks that it must be at the beginning of a word). Afterwards it is necessary to go through the whole list manually to check for erroneously segmented instances and add those morphemes that have not been caught with the automatic procedure. The resulting annotated data is left in the column TOKENS, but for the un-annotated IPA a backup-column could be created and it is useful to add a column for notes.

ID	DOCULECT	CONCEPT	FORM	TOKENS
632	German	rotten	verfault	f ɛ r + f a ʊ l + t
633	German	rotten	vermodert	f ɛ r + m oː d ə r + t
634	German	drink	trinken	t r ɪ ŋ k + ə n
635	German	hunger	Hunger	h ʊ ŋ ə r
636	German	famine	Hungersnot	h ʊ ŋ ə r + s + n oː t
637	German	thirst	Durst	d ʊ r s t
638	German	suck	saugen	z au g + ə n
639	German	chew	kauen	k au + ə n
640	German	swallow	schlucken	ʃ l ʊ k + ə n
641	German	swallow	verschlucken	f ɛ r + ʃ l ʊ k + ə n

Table 2: Entries from the German wordlist of the Intercontinental Dictionary Series (Key and Comrie 2015), in IPA with morpheme borders.

Once the data has been annotated in this way, we follow an idea proposed in Hill and List (2017) by adding glosses to the morphemes. This helps us specifically to disambiguate homophone morphemes which do not go back to the same ancestor, and, of course, this step can also be done at the same time when correcting the initial morpheme boundaries. The glosses are added into a column called MORPHEMES. The form is free: which words to use as a gloss for a morpheme depends only on practicality, as long as the entry does not have any spaces (as EDICTOR segments the MORPHEMES content by spaces). What is important, however, is that identical morphemes across different words are given the same gloss, and that those that are not identical are given different glosses. In our tests, we use English as our glossing language and derive the morpheme gloss from the main meaning of each given word. Thus, German {nah} (‘near’), for example, is glossed as NEAR. In cases when there are more homonyms in the language investigated than in English, we recommend to add _B, _C etc. to the glosses to distinguish them consistently. In contrast to the very free form proposed in Hill and List (2017), we have made good experience in writing content morphemes in upper case in order to mark them visually. Glosses for grammatical morphemes on the other hand are written in lower case and start with an underscore. Thus, we write _infinitive for the infinitive suffix in our German test data. This is based on a new feature of the EDICTOR, by which all morpheme glosses preceded by an underscore are displayed in transparent and small font to further mark the different in status for grammatical and content morphemes. Additionally, the underscore can be quickly added or deleted by right-clicking on a morpheme in the interface.

Distinguishing content from grammatical forms is not always an easy task, and at it is likely that scholars will disagree about individual decisions. To base our annotations on a clear-cut criterion, we considered as content morphemes only those that appear also as free morphemes or that are confixes (e. g. Schwieger-, ‘-in-law’). But the definition of a free morpheme as well as the border between a confix and an affix is far from clear and free morphemes can also be grammatical. Similarly, it is also not always clear where to put a morpheme border as definitions of what is a morpheme vary and may depend on the research question or the information available. Ideally, one would use the same criterion for glossing not only throughout the same, but also across different languages, but this may prove to be difficult in practice since problems with the definition might only be noticed during the course of annotation. In order to reduce the amount of typing, a script can again be used to find and annotate the most common morphemes. All data, however, will usually need to be checked manually, since it is not possible to distinguish automatically between homophonous and recurring morphemes.

Deeper levels of annotation

As mentioned above, not only morpheme borders but also allomorphs should be included in the information provided to the computer. These non-homophone cognates can be marked in the MORPHEMES column during the previous step by giving them the same gloss names but differentiating them with a number at their end. So {näh} in German näherkommen (‘approach’) is glossed in our examples in Table 3 below as NEAR2. In a further step, we annotate the actual roots by adding a column that we call ROOTS and one that we call ROOTIDS, where we no longer distinguish between NEAR and NEAR2. Although this annotation may seem rather complex, it has the clear advantage that it allows us to distinguish different kinds of word families inside the same language, namely those where morphemes recur in the same, and those where they recur in different forms (be it due to allophony or internal sound change).

ID	DOCULECT	CONCEPT	FORM	TOKENS	MORPHEMES	COGIDS	ROOTS	ROOTIDS
1239	German	approach	nahen	n aː + ə n	NEAR _infinitive	953 42	NEAR _infinitive	890 42
1240	German	approach	hingehen	h ɪ n + g eː + ə n	TO GO _infinitive	926 441 42	TO GO _infinitive	865 414 42
1241	German	approach	näherkommen	n ɛː + ə r + k ɔ m + ə n	NEAR2 _comp COME _infinitive	954 130 955 42	NEAR _comp COME _infinitive	890 128 891 42
1242	German	approach	sich nähern	z ɪ x _ n ɛː + ə r + n	ONESELF NEAR2 _comp _infinitive2	414 954 130 329	_reflexive NEAR _comp _infinitive	394 890 128 42
1243	German	enter	hineingehen	h iː n + ai n + g eː + ə n	TO IN GO _infinitive	926 389 441 42	TO IN GO _infinitive	865 371 414 42
1244	German	enter	eintreten	ai n + t r eː t + ə n	IN TREAD _infinitive	389 933 42	IN TREAD _infinitive	371 872 42
1245	German	enter	hereinkommen	h ɛ r + ai n + k ɔ m + ə n	FROM IN COME _infinitive	939 389 940 42	FROM IN COME _infinitive	878 371 152 42
1246	German	carry (bear)	tragen	t r aː g + ə n	CARRY _infinitive	956 42	CARRY _infinitive	436 42
1247	German	carry (bear)	schleppen	ʃ l ɛ p + ə n	CARRY_B _infinitive	957 42	CARRY_B _infinitive	892 42

Table 3: Entries from the German wordlist of the Intercontinental Dictionary Series (Key and Comrie 2015) with morphological glosses and root annotation.

What we cannot do yet

Morphological annotation works with strings of characters and predefined fields. This means that some types of information which one might want to include are difficult to implement. For an etymological annotation in which not only transparent morpheme borders are included, one will likely encounter cases in which morpheme borders have become fuzzy due to phonological mergers. An example would be the German word Messer (/mɛsɐ/, ‘knife’) which in modern German is monomorphemic but goes back to a compound (Old High German mezzi-sahs, later mezzi-rahs, literally ‘food-knife’, see Watkins 1990: 295). For some languages, cases like these can be found even when taking a synchronic perspective.

Additionally, it would be interesting to include information on the kinds of sound changes that a morpheme or word underwent during its history. But we have not decided where to add this information, and how to standardize it, specifically also because it will be difficult to find a principled and standardized way to do so across different languages. If there was only one sound shift per morpheme, it would be quite possible to develop a straightforward proposal for annotation, but considering that morphemes are not limited in the amount of phonological changes they may accumulate, we find it difficult to come up with a proposal at this stage of the research. However, since we annotate allomorphs consistently, we are already able to identify all allomorphs of a given root in our data. Therefore, it will be easy to add information on sound changes later, once we managed to find a useful representation format. 

Finally, we could not (yet) find any systematic way to model analogical relations between words. Given the importance and frequency of analogy in arguments in historical linguistics, we will try to come up with proposals for this in future versions of our annotation framework.

References

Creutz, M., and K. Lagus. 2005. “Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.0.” Publications in Computer and Information Sciences 81. Helsinki: Helsinki University of Technology.

Forkel, Robert, Johann-Mattis List, Simon J. Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarström, Martin Haspelmath, Gereon A. Kaiping, and Russell D. Gray. 2018. “Cross-Linguistic Data Formats, Advancing Data Sharing and Re-Use in Comparative Linguistics.” Nature Scientific Data.

Good, Jeff, and Michael Cysouw. 2013. “Languoid, Doculect, Glossonym: Formalizing the Notion of ‘Language’.” Journal of Language Documentation and Conservation 7: 331–59. https://scholarspace.manoa.hawaii.edu/handle/10125/4606.

Haspelmath, Martin, and Uri Tadmor, eds. 2009. WOLD. Leipzig: Max Planck Institute for Evolutionary Anthropology. https://wold.clld.org/.

Hill, Nathan W., and Johann-Mattis List. 2017. “Challenges of Annotation and Analysis in Computer-Assisted Language Comparison: A Case Study on Burmish Languages.” Yearbook of the Poznań Linguistic Meeting, no. 3: 47–76.

Key, Mary Ritchie, and Bernard Comrie, eds. 2015. IDS. Leipzig: Max Planck Institute for Evolutionary Anthropology. https://ids.clld.org/.

List, Johann-Mattis, Michael Cysouw, Simon Greenhill, and Robert Forkel, eds. 2018. Concepticon. Jena: Max Planck Institute for the Science of Human History. http://concepticon.clld.org/.

List, Johann-Mattis. 2017. “A Web-Based Interactive Tool for Creating, Inspecting, Editing, and Publishing Etymological Datasets.” In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. System Demonstrations, 9–12. Valencia: Association for Computational Linguistics. http://aclweb.org/anthology/E/E17/E17-3003.pdf.

List, Johann-Mattis, Simon J. Greenhill, and Russell D. Gray. 2017. “The Potential of Automatic Word Comparison for Historical Linguistics.” PLoS ONE, no. 1, 12 (January). Public Library of Science: 1–18. doi:http://dx.doi.org/10.1371/journal.pone.0170046.

List, Johann-Mattis, Simon Greenhill, and Robert Forkel. 2017. “LingPy. A Python Library for Quantitative Tasks in Historical Linguistics.” Jena: Max Planck Institute for the Science of Human History. doi:https://doi.org/10.5281/zenodo.1065403.

List, Johann-Mattis, Philippe Lopez, and Eric Bapteste. 2016. “Using Sequence Similarity Networks to Identify Partial Cognates in Multilingual Wordlists.” In Proceedings of the Association of Computational Linguistics 2016, 2: Short Papers:599–605. Berlin: Association of Computational Linguistics. http://anthology.aclweb.org/p16-2097.

Moran, Steven, and Michael Cysouw. 2018. The Unicode Cookbook for Linguists: Managing Writing Systems Using Orthography Profiles. Translation and Multilingual Natural Language Processing 10. Berlin: Language Science Press.

Swadesh, Morris. 1950. “Salish Internal Relationships.” International Journal of American Linguistics 16 (4): 157–67.

Watkins, Calvert. 1990. “Etymologies, Equations, and Comparanda: Types and Values, and Criteria for Judgment.” In Linguistic Change and Reconstruction Methodology, edited by Philip Baldi, Part 1.45:289–303. Trends in Linguistics. Studies and Monographs. Berlin; New York: Mouton de Gruyter.

 

 

---

---