Behind the Sino-Tibetan Database of Lexical Cognates: Concept selection

One of the crucial steps in creating a database of lexical cognates is the selection of concepts one wants to use for a given study. While many scholars use the classical Swadesh list of 200 items) (Swadesh 1952) for this purpose, or the combined list of 207 items, in which the former has been merged with Swadesh’s updated list of 100 items (Swadesh 1955), and which is often mistakenly attributed to Swadesh himself, although the first official reference seems to be Comrie (1977), it is useful to give the selection of concepts some more thought initially.

What scholars underestimate usually in this context is that there are quite different reasons why the classical Swadesh lists, or any pre-compiled concept lists, may not be apt for the investigation of a given language family. Apart from the obvious reason that a concept list may contain terms that cannot be found in the target languages, additional problems resulting from concept choice result from language-specific characteristics of lexical typology, such as: (A) sound symbolism, which may differ across languages, (B) compoundhood, which also shows language-specific patterns, (C) polysemy or homophony, which both may show areal or family-specific aspects.

In addition, one should not underestimate the importance of data availability. Given that the majority of lexical databases are not compiled from scratch, but derived from existing data collections which serve as some kind of the backbone of the database, one should make sure to check which concepts have been recorded in the major sources upon which one wants to base the database.

When compiling the Sino-Tibetan Database of Lexical Cognates for our study on the origin of Sino-Tibetan (Sagart et al. 2019, we had to struggle with all of the above-mentioned points. Since our initial goal was to create a database of high mutual coverage in terms of attested words per language (see List et al. 2018 for a discussion of this concept), we had to pay specific attention to the problem of data availability. For this reason, we started from an initial comparison of concepts available different sources (including those digitized by the STEDT project, Matisoff 2015). To compare the concepts available in the different sources, we made use of the Concepticon project (, List et al. 2016, see List 2018 for an introduction to the goals of the project) and the pyconcepticon library, which offers quick access to the resources offered by Concepticon.

When having installed the pyconcepticon package, following the instructions provided at the project’s GitHub repository, one can easily check the overlap between several concept lists using the intersection command that can be invoked from the terminal (you have to replace PATH with the path to your local Concepticon repository):

$ concepticon --repos=PATH intersection Yakhontov-1991-35 Swadesh-1952-200 Swadesh-1955-100 Dolgopolsky-1964-15 
1 EYE [1248] 
2 I [1209] 
3 LOUSE [1392] 
4 NAME [1405] 
5 THOU [1215] 
6 TONGUE [1205] 
7 TOOTH [1380] 
8 TWO [1498] 
9 WATER [948]

This code example checks which of the concepts given in the concept lists by Swadesh (100 and 200) plus the lists by Yakhontov (published in [Starostin 1991]) and Dolgopolsky (1964) recur in all lists, and outputs them both with their Concepticon Gloss, and their Concepticon IDs. Note that you do not need to use the lists that are provided by Concepticon alone, you can also compare with your own concept lists, and this is exactly what we did, as we wanted to work with data that was not yet officially linked to Concepticon (see the tutorial by List 2017 for more information).

While our first concept list consisted of as many as 250 concepts, we had to reduce it further, once we started to add more languages to our sample. Often, we ran into simple coverage problems, as some of the concepts we thought would be useful turned out to be missing from sources, and other concepts, like, e.g., MOSQUITO, were not lexified in some regions of the Himalaya.

As a result, our final “master list”, which we used for the phylogenetic analyses, only comprises 180 concepts. The criteria for exclusion of concepts was strictly technical. We ranked all concepts by their coverage with respect to the languages in our sample, and set up a coverage of at least 80% (each concept should be reflected in at least 80% of our languages). We further removed some remaining concepts from the list which had been shown to be problematic with respect to compoundhood, sound symbolism, and polysemy.

Using Concepticon to check for coverage and overlap across concept lists turned out to be very efficient, as it helped us to avoid to include data into our database which did not provide enough coverage. To illustrate how this can be done, we can compare the coverage of our current list of 250 concepts in the database with the coverage of a resource that we might want to add to our database in the future. This resource is the database of cognates in Kho-Bwa languages, compiled by T. A. Bodt (available from Bodt and List 2019). As Bodt’s data and our data are linked to Concepticon (following the recommendations of the CLDF initiative, Forkel et al. 2018), we can compare their intersection with a simple command from the terminal, in which we combine the original intersection command of pyconcepticon with the wc command shipped with all Unix terminals by using the famous and incredibly useful pipe |:

$ concepticon --repos=PATH intersection Sagart-2018-250.tsv Bodt-2019-664.tsv | wc -l 

To run this code as illustrated here, the concept lists need to show a certain format. You can check this out yourself by installing pyconcepticon, downloading the Concepticon data, and the two concept lists, which I have all uploaded to this GitHub Gist.


Bodt, Timotheus A. and List, Johann-Mattis (2019): Testing the predictive strength of the comparative method: An ongoing experiment on unattested words in Western Kho-Bwa langauges. Papers in Historical Phonology 4.1. 22-44.

Comrie, Bernard and Smith, Norval (1977): Lingua Descriptive Series: Questionnaire. Lingua 42. 1-72.

Dolgopolsky, Aron B. (1964): Gipoteza drevnejšego rodstva jazykovych semej Severnoj Evrazii s verojatnostej točky zrenija [A probabilistic hypothesis concering the oldest relationships among the language families of Northern Eurasia]. Voprosy Jazykoznanija 2. 53-63.

Forkel, Robert and List, Johann-Mattis and Greenhill, Simon J. and Rzymski, Christoph and Bank, Sebastian and Cysouw, Michael and Hammarström, Harald and Haspelmath, Martin and Kaiping, Gereon A. and Gray, Russell D. (2018): Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics. Scientific Data 5.180205. 1-10.

List, Johann-Mattis and Cysouw, Michael and Forkel, Robert (2016): Concepticon. A resource for the linking of concept lists. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation. 2393-2400.

List, Johann-Mattis (2017): Historical Language Comparison with LingPy and EDICTOR [Historischer Sprachvergleich mit LingPy und EDICTOR]. Department of Linguistic and Cultural Evolution: Max-Planck Institute for the Science of Human History.

List, Johann-Mattis and Walworth, Mary and Greenhill, Simon J. and Tresoldi, Tiago and Forkel, Robert (2018): Sequence comparison in computational historical linguistics. Journal of Language Evolution 3.2. 130–144.

List, Johann-Mattis (2018): Towards a history of concept list compilation in historical linguistics. History and Philosophy of the Language Sciences 5.10. 1-14.

Matisoff, James A. (2015): The Sino-Tibetan Etymological Dictionary and Thesaurus project. Berkeley:University of California.

Sagart, Laurent and Jacques, Guillaume and Lai, Yunfan and Ryder, Robin and Thouzeau, Valentin and Greenhill, Simon J. and List, Johann-Mattis (2019): Dated language phylogenies shed light on the ancestry of Sino-Tibetan. Proceedings of the National Academy of Science of the United States of America 116. 10317–10322. DOI: 10.1073/pnas.1817972116

Swadesh, Morris (1952): Lexico-statistic dating of prehistoric ethnic contacts. With special reference to North American Indians and Eskimos. Proceedings of the American Philosophical Society 96.4. 452-463.

Swadesh, Morris (1955): Towards greater accuracy in lexicostatistic dating. International Journal of American Linguistics 21.2. 121-137.


OpenEdition suggests that you cite this post as follows:
Johann-Mattis List (June 26, 2019). Behind the Sino-Tibetan Database of Lexical Cognates: Concept selection. Computer-Assisted Language Comparison in Practice. Retrieved July 22, 2024 from

This entry was posted in Primer and tagged , , , , on by .

About Johann-Mattis List

Seit Anfang 2023 leite ich den Lehrstuhl für Multilinguale Computerlinguistik in Passau. In meiner Forschung nehme ich generell einen datenbasierten, empirischen und quantitativen Standpunkt in Bezug auf Sprachwandel und Sprachgeschichte ein, mit einem speziellen Fokus auf südostasiatischen Sprachen. Im Gegensatz zu rein computerbasierten Ansätzen versuche ich jedoch, meine Forschung nah an der traditionellen historischen Linguistik und der linguistischen Theorie auszurichten, weshalb ich einen computer-gestützten Ansatz im Gegensatz zu einem rein computer-basierten Ansatz verfolge.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.