Creating a CLDF Wordlist from Heath et al.’s Dogon Comparative Wordlist

The Dogon and Bangime Linguistics project (https://dogonlanguages.org) offers a large comparative spreadsheet in which translational equivalents for a huge number of concepts are translated into various Dogon languages.  Due to its enormous size, no attempts have been made so far to integrate the spreadsheet with the lexical resources that were compiled as part of the CLDF initiative in order to populate the Lexibank repository. Here, we report a first attempt to circumvent the problems resulting from the size of the spreadsheet by convert not all but parts of the spreadsheet to CLDF Wordlist standards, which allows us to integrate parts of the data with other resources in Lexibank.

1 Introduction

One core resource of the Dogon and Bangime Linguistics project (Heath et al. 2016, https://dogonlanguages.org) is the Dogon Comparative Wordlist, a spreadsheet that lists more than 9000 lexical glosses translated into several Dogon language varieties, spoken in Central-Eastern Mali and Burkina Faso. The spreadsheet was created as the result of intensive field work. In size, it reminds more of a multilingual dictionary than a classical comparative wordlists with only a couple of hundred entries. Due to the internal complexity of the comparative wordlists and its very detailed glosses denoting very finegrained semantic differences that would often only find translational equivalents in a few of the more than two dozen language varieties documented by the Dogon and Bangime Linguistics projects, the underlying concept list has so far not yet been linked with the Concepticon resources (List et al. 2023, https://concepticon.clld.org) and the data has only been integrated with the Lexibank repository through dedicated studies in which words were hand-picked and manually corrected (Hantgan and List 2022, Hantgan et al. 2022).

2 Background

Given the importance of information on Dogon languages for the investigation of the origin of the language isolate Bangime (Hantgan 2022), we have long since been trying to convert the comparative wordlist into a proper CLDF wordlist that would allow for an easy integration with additional lexical resources on languages in the region. However, all attempts have failed so far, due ot the sheer size of the wordlist.

The data on Dogon languages shared by the Dogon and Bangime Linguistics project by Heath et al. (2016) currently covers lexical data for 30 language varieties, including several Dogon varieties and the isolate Bangime. The spreadsheet lists languages in separate columns and concepts in separate rows, showing a total of more than 43 columns (with specifications of lexical glosses in English and French and at different levels of detail) and 9667 rows, corresponding to a total of 121,223 word forms across all languages.

3 CLDF Conversion

The crucial strategy for the conversion to CLDF was to work with a reduced selection of entries. This would of course result in a CLDF wordlist much smaller in size than the original comparative wordlist, but it would allow us to pursue the targeted normalization steps that are so crucial for the conversion of data to CLDF for the inclusion in the Lexibank repository (List et al. 2022). Thanks to the flexibility of CLDFBench (Forkel and List 2020), one is never required to convert all data in a given source into cross-linguistic data formats. Thanks to the smaller selection of lexical entries, it was also possible to consistently stanardize the transcriptions of all lexical entries, an enterprise which would have not been very difficult to be carried out for all 121,223 word forms in the data.

3.1 Language Selection

Of the 30 varieties in the original comparative wordlist, 25 Dogon varieties with the largest coverage were selected. Coverage is specifically important for computational and quantitative analyses, since languages with low coverage tend to cause numerous problems when it comes to the application of computational techniques, such as automated cognate detection (List et al. 2018) or phylogenetic analysis (Sagart et al. 2019). The table below shows all languages which were selected for inclusion in the standardized wordlist.

Number Name Glottocode
1 Ampari ampa1238
2 Bankan Tey bent1238
3 Ben Tey bent1238
4 Bunoge buno1241
5 Dogul Dom (Bendiely & Kundialang, BC) dogu1235
6 Dogul Dom (Kundialang) dogu1235
7 Donno So donn1239
8 Gourou guru1265
9 Jamsay (Douentza area, JH)
10 Jamsay Mondoro jams1239
11 Mombo momb1254
12 Najamba bond1248
13 Nanga nang1261
14 Penange pena1270
15 Perge Tegu perg1234
16 Tebul Ure tebu1239
17 Tiranige tira1258
18 Togo-Kan togo1254
19 Tommo-So (Tongo Tongo, LM)
20 Tomo Kan Diangassagou tomo1243
21 Tomo Kan Segue tomo1243
22 Toro Tegu toro1253
23 Yanda Dom yand1257
24 Yorno So yorn1234

Table 1: Languages selected for the CLDF wordlist.

3.2 Concept Selection

With its more than 9000 distinct glosses for lexical concepts, the spreadsheets shows much more resemblance to a multilingual dictionary than to a comparative wordlist. For the standardization of lexical data in the form of a CLDF Wordlist, concepts need to be linked to the Concepticon project, but concept lists linked to Concepticon rarely exceed more than 1000 concepts in size. In order to ease the linking process and to avoid a biased selection of an only small number of concepts easy to link, we decided to use a new strategy for the linking process. In a first stage, we carried out an automated linking of the lexical glosses to Concepticon, using the PySem library (https://pypi.org/projects/pysem, List 2021). In a second step, all entries were checked, keeping only those that showed a clear mapping to the Concepticon project. This yielded a concept list of 1811 entries. In a third step, this list was further filtered, by retaining only those concept sets, which also occur in the concept list underlying the Intercontinental Dictionary Series (https://ids.clld.org, Key and Comrie 2016). This helped us to further reduce the size of the concepts to 944 items.

While this approach to concept selection was carried out in a semi-automated way very specific to the Dogon data, we think that the general principle of subsampling smaller amounts of data from large lists of lexical glosses may be useful for many future applications, specifically when working, for example, with dictionary data, from which one wishes to subset a certain amount of concepts in order to convert dictionaries into a wordlist.

3.3 Phonetic Transcription

We used the standard procedure of creating orthography profiles (Moran and Cysouw 2018 with the help of the Lexibank pipeline (List et al. 2022). Creating the orthography profile proved particularly challenging, given that tonal information was in part annotated on vowels — which makes it difficult to account for tone in subsequent automated analyses on shared cognates — and that word forms often cointained unstandardized information that would not relate directly to the pronunciation of the word (such as brackets with hints on the meaning, multiple forms in a paradigm, etc.). Instead of trying to automatize this step, we worked through the data manually and also made extensive use of the possibility to correct lexical forms manually by keeping a list of lexemes that would later be replaced to a more standardized representations (available in the file etc/lexemes.tsv in the CLDF dataset).

3.4 CLDF Conversion

CLDF conversion was carried out in the “traditional” manner, following the Lexibank workflow of converting data to CLDF wordlists with CLDFBench (https://pypi.org/project/cldfbench, Forkel and List 2020). We profited from the flexibility of using custom Python code in CLDFBench by applying the concept filter at this stage, by checking for each word form, if its Concepticon concept set recurs in the Intercontinental Dictionary Series. Thanks to the Concepticon being available from CLDFBench, this amounts to one statement by which the dictionary is created, as shown in the code line from the Lexibank script below.

def cmd_makecldf(self, args):
    """
    Convert the raw data to a CLDF dataset. 
    """
    # select IDS concept list to check for concepts to be added
    ids = {c.concepticon_gloss for c in
           self.concepticon.conceptlists["Key-2016-1310"].concepts.values()
           if c.concepticon_gloss}

5 Conclusion

Although it seemed close to being impossible at first to provide a subset of the huge and impressive Dogon Comparative Wordlist, we found a way to address this task with standard procedures that fit nicely in the Lexibank workflow of converting datasets into CLDF Wordlists. In the future, we plan on linking the resulting concept list to Concepticon and carrying out an initial automatic analysis of the data, searching for cognate sets among the 25 Dogon varieties.

The data is curated on GitHub (https://github.com/languageislands/heathdogon) and archived on Zenodo (https://doi.org/10.5281/zenodo.8238983).

References

List, Johann-Mattis and Tjuka, Annika and van Zantwijk, Mathilda and Blum, Frederic and Barrientos Ugarte, Carlos and Rzymski, Christoph and Greenhill, Simon J. and Robert Forkel (2023): CLLD Concepticon [Dataset, Version 3.1.0]. Leipzig:Max Planck Institute for Evolutionary Anthropology.

Moran, Steven, Forkel, Robert, and Heath, Jeffrey (2016): Dogon and Bangime Linguistics. Jena:Max Planck Institute for the Science of Human History. https://dogonlanguages.org

Forkel, Robert and List, Johann-Mattis (2020): CLDFBench. Give your Cross-Linguistic data a lift. In: Proceedings of the Twelfth International Conference on Language Resources and Evaluation. 6997-7004.

Hantgan, Abbie and Babiker, Hiba and List, Johann-Mattis (2022): First steps towards the detection of contact layers in Bangime: A multi-disciplinary, computer-assisted approach [version 2; peer review: 2 approved]. Open Research Europe 2.10. 1-27.

Hantgan, Abbie (2022): The Small Bang. Computer-Assisted Language Comparison in Practice 5.12. 1-2. https://calc.hypotheses.org/5053 

Hantgan, Abbie and List, Johann-Mattis (2022): Bangime: secret language, language isolate, or language island? A computer‐assisted case study. Papers in Historical Phonology 7.1. 1-43.

Key, Mary Ritchie and Comrie, Bernard (2016): The Intercontinental Dictionary Series. Leipzig:Max Planck Institute for Evolutionary Anthropology. https://ids.clld.org 

List, Johann-Mattis and Walworth, Mary and Greenhill, Simon J. and Tresoldi, Tiago and Forkel, Robert (2018): Sequence comparison in computational historical linguistics. Journal of Language Evolution 3.2. 130–144.

List, Johann-Mattis and Forkel, Robert and Greenhill, Simon J. and Rzymski, Christoph and Englisch, Johannes and Gray, Russell D. (2022): Lexibank, A public repository of standardized wordlists with computed phonological and lexical features. Scientific Data 9.316. 1-31. https://lexibank.clld.org 

Moran, Steven and Cysouw, Michael (2018): The Unicode Cookbook for Linguists: Managing writing systems using orthography profiles. Berlin:Language Science Press.

List, Johann-Mattis (2021): PySem: Python library for handling semantic data in linguistics. Leipzig:Max Planck Institute for Evolutionary Anthropology. https://pypi.org/projects/pysem 

Sagart, Laurent and Jacques, Guillaume and Lai, Yunfan and Ryder, Robin and Thouzeau, Valentin and Greenhill, Simon J. and List, Johann-Mattis (2019): Dated language phylogenies shed light on the ancestry of Sino-Tibetan. Proceedings of the National Academy of Science of the United States of America 116. 10317-10322.

Cite this article as: Abbie Hantgan and Johann-Mattis List, “Creating a CLDF Wordlist from Heath et al.’s Spreadsheet of Dogon Languages,” in Computer-Assisted Language Comparison in Practice, 6.2: 59–63 [first published on 26/07/2023], URL: https://calc.hypotheses.org/6329, DOI: 10.15475/calcip.2023.2.2.

Download the article as PDF: calcip-06-2-2.pdf

Grant Information: This project has received funding from the Max Planck Society as part of the CALC³ Research Project (https://calc.digling.org). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright information: This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.


OpenEdition suggests that you cite this post as follows:
Johann-Mattis List (August 16, 2023). Creating a CLDF Wordlist from Heath et al.’s Dogon Comparative Wordlist. Computer-Assisted Language Comparison in Practice. Retrieved July 22, 2024 from https://doi.org/10.58079/m6lz


This entry was posted in Dataset and tagged , , , on by .

About Johann-Mattis List

Seit Anfang 2023 leite ich den Lehrstuhl für Multilinguale Computerlinguistik in Passau. In meiner Forschung nehme ich generell einen datenbasierten, empirischen und quantitativen Standpunkt in Bezug auf Sprachwandel und Sprachgeschichte ein, mit einem speziellen Fokus auf südostasiatischen Sprachen. Im Gegensatz zu rein computerbasierten Ansätzen versuche ich jedoch, meine Forschung nah an der traditionellen historischen Linguistik und der linguistischen Theorie auszurichten, weshalb ich einen computer-gestützten Ansatz im Gegensatz zu einem rein computer-basierten Ansatz verfolge.

One thought on “Creating a CLDF Wordlist from Heath et al.’s Dogon Comparative Wordlist

  1. Pingback: Sélection de billets de la Croisée de la BULAC pour la rentrée – Le Carreau de la BULAC

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.