Merging datasets with LingPy and the CLDF curation framework

Imagine you have two different datasets, both containing approximately the same concepts, but slightly different numbers of columns and — more importantly — potentially identical identifiers in the first column. A bad idea for merging these datasets would be to paste them in Excel or some other kind of spreadsheet software, and then trying to manually adjust all problems that might occur during this process.

A better idea is to just use LingPy and our CLDF curation framework (which was, for example, used when establishing the CLICS² database, see List et al. 2018), which is basically not much work, requiring just a few lines of code to be written, but giving you also the possibility to re-use these code pieces on similar tasks.

I want to illustrate how this can be done by showing for two datasets (Chacon 2017 and Chacon et al. forthcoming), which are both curated within our CLDF framework, how one can merge them with LingPy. Before we can start, we will install these two datasets via pip.

$ pip install -e git+
$ pip install -e git+

In our Python script, we then load the two packages along with LingPy and the pyconcepticonpackage.

from lingpy import *
from lexibank_chaconarawakan import Dataset as ds1
from lexibank_chaconbaniwa import Dataset as ds2
from pyconcepticon.api import Concepticon

Along with these packages, the LingPy packages will also have been installed, if it was not already present on your computer. Both datasets contain a raw folder in which we find the original data as it was curated within the EDICTOR/LingPy approach, that is: the data was originally analyzed with LingPy (List et al. 2018) and then manually corrected with EDICTOR (List 2017). So instead of loading the data with LingPy’s CLDF reader, we load it in its raw form, for convenience.

wl1 = Wordlist(
wl2 = Wordlist(

The part of the data we want is the common Swadesh-list of 100 concepts (Swadesh 1955). We load this list with help of the pyconcepticon API, published as part with the Concepticon project (List et al. 2016).

swad = [
    c.concepticon_id for c in Concepticon(

concepts = {
    wl2[idx, 'concept'] for idx in wl2 if wl2[
        idx, 'concepticon_id'] in swad}

We now declare a dictionary in Python to store the data that we want to then write to a LingPy-wordlist file (that can also be read in edictor). We use the columns present in the first wordlist as our standard, and add two more, one for the original index (called old_idx) and one for a combined cognate-identifier (called cog).

D = {0: wl1.columns+['old_idx', 'cog']}

We now iterate over both wordlists and add the relevant entries to the dictionary, making sure to separate the cognate-identifiers from each other by assigning them to different sets (with help of the name of the datasets that we add as part of the cognate-set identifier).

nidx = 1
for idx in wl1:
    D[nidx] = [wl1[idx, h] for h in wl1.columns
        ] + ['chaconarawakan-'+str(idx)]
    D[nidx] += ['chaconarawakan-'+str(
        wl1[idx, 'cogid'])]
    nidx += 1
for idx in wl2:
    D[nidx] = [wl2[idx, h] for h in wl1.columns
        ] + ['chaconbaniwa-'+str(idx)]
    D[nidx] += ['chaconbaniwa-'+str(
        wl2[idx, 'cogid'])]
    nidx += 1

All that’s left to do now is load the data in to a Wordlist object provided by LingPy and write it to file (after a few operations).

wl = Wordlist(D)

First, we renumber the cognates in the column cog in order to make sure they are numeric.

wl.renumber('cogx', 'cogid', override=True)

Then, we segment the so far non-segmented entries in the data: for idx, ipa, segments in wl.iter_rows('ipa', 'segments'):
if not segments:
wl[idx, 'segments'] = ipa2tokens(ipa)

Now, we write all entries to file that conform to a concept that is also present in Swadesh’s list of 100 items.

        c for c in wl.columns if c not in [
        concept = 'in '+str(concepts))

This is essentially all. The data in file chacon-arawakan-baniwa.tsv can now be analyzed with LingPy or also manually annotated with EDICTOR.


Chacon, T. (2017): Arawakan and Tukanoan contacts in Northwest Amazonia prehistory. PAPIA 27.2. 237-265..

List, J.-M., M. Cysouw, and R. Forkel (2016): Concepticon. A resource for the linking of concept lists. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation. 2393-2400.

List, J.-M. (2017): A web-based interactive tool for creating, inspecting, editing, and publishing etymological datasets. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. System Demonstrations. 9-12.

List, J.-M., S. Greenhill, C. Anderson, T. Mayer, T. Tresoldi, and R. Forkel (eds.) (2018): CLICS: Database of Cross-Linguistic Colexifications. Max Planck Institute for the Science of Human History: Jena.

List, J.-M., S. Greenhill, T. Tresoldi, and R. Forkel (2018): LingPy. A Python library for quantitative tasks in historical linguistics. Max Planck Institute for the Science of Human History: Jena.

Swadesh, M. (1955): Towards greater accuracy in lexicostatistic dating. International Journal of American Linguistics 21.2. 121-137.

Cite this blog post
Johann-Mattis List (2018, December 10). Merging datasets with LingPy and the CLDF curation framework. Computer-Assisted Language Comparison in Practice. Retrieved June 22, 2024, from

This entry was posted in Code, Dataset and tagged , , , on by .

About Johann-Mattis List

Seit Anfang 2023 leite ich den Lehrstuhl für Multilinguale Computerlinguistik in Passau. In meiner Forschung nehme ich generell einen datenbasierten, empirischen und quantitativen Standpunkt in Bezug auf Sprachwandel und Sprachgeschichte ein, mit einem speziellen Fokus auf südostasiatischen Sprachen. Im Gegensatz zu rein computerbasierten Ansätzen versuche ich jedoch, meine Forschung nah an der traditionellen historischen Linguistik und der linguistischen Theorie auszurichten, weshalb ich einen computer-gestützten Ansatz im Gegensatz zu einem rein computer-basierten Ansatz verfolge.

2 thoughts on “Merging datasets with LingPy and the CLDF curation framework

  1. Gereon

    Do you have a single-file version of the script, for example in a Github Gist, so we can read all the commands in context?


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.