From Fieldwork to Trees 3: CLDF recipes

In the previous two posts (Part 1, Part 2), I took you from a matrix of word lists from fieldwork to a LingPy-compatible CLDF Wordlist with cognate codes and alignments. We can now feed this dataset into existing tools and recipes for visualizing and analyzing CLDF Wordlists.

If we add geolocations to the languages in our dataset, we can use an existing CLDF recipe from the CLDF cookbook (Forkel et al. 2018) to plot the density of the data on a map. Some more metadata on the lects would be a good thing to have anyway, so let us add a table to the dataset: Dialects are described in another CSV or TSV table, like this

alkAlor Kecilalor12478.2591124.4092
albAlor Besaralor12478.2224124.4079

The metadata file then gets some additional entries (mostly the description of this new table, as well as a “Foreign Key” relationship linking the form table to this table) to make sure that this data is found and properly associated with the forms in the table, which I have added to Alorese-Lects-metadata.json. If we had descriptions of the concepts, eg. with links to Concepticon (List, Cysouw & Forkel 2016), they would also be associated using the metadata file.

Before we start using this enriched wordlist for other purposes, it may be useful to see whether the file is indeed valid CLDF. The python pycldf package comes with a tool to validate and list statistics of datasets.

$ pip install pycldf
$ cldf validate Alorese-Lects-metadata.json
$ cldf stats Alorese-Lects-metadata.json
<cldf:v1.0:Wordlist at .>
key value
----------------------- ---------------------------------------------------------------------------------------
dc:license Ⓒ 2018 Yunus Sulistyono
dc:identifier Alorese Dialects 0.1.0, abridged
dc:title Alorese Dialects, abridged
dc:publisher Leiden University Centre for Linguistics
dc:creator ['Sulistyono, Yunus', '', 'Kaiping, Gereon Alexander']
dc:description Lexical data of 13 Alorese [alor1247] dialects, abridged
special:publisher_place Leiden, The Netherlands
Path Type Rows
------------- ------------- ------
aligned.tsv FormTable 53
languages.tsv LanguageTable 3

The lack of output after the second command is a good thing: It means that no problems were found in our dataset. If there were errors in the word list, cldf validate would complain.

This dataset is now easy to visualize: Get the cookbook recipe and its dependencies (admittedly, getting basemap  to work can be a bit of a hassle), and then a simple python Alorese-Lects-metadata.json alorese.png --cmap viridis_r will generate a figure containing the three Alorese villages, each with the number of forms sampled from there. (The viridis color map is slightly greener, where the default magma cmap has a yellow that is even more similar to the land background color.)

Number of forms given in the abridged Alorese dataset, for each of the three different dialects. The black line is the coastline of the birdshead peninsula of the island of Alor.

Another interesting tool to analyze CLDF wordlists comes in the Python library accompanying the LexiRumah dataset (Kaiping, Edwards & Klamer 2019): For creating orthographies for the languages in that dataset, we needed to know what sounds they use. The pylexirumah.get_phonetic_inventories module goes through the CLDF wordlist and counts the frequency of all segments that appear, both per-language and overall.

$ python -m pylexirumah.get_phonetic_inventories --dataset Alorese-Lects-metadata.json
a 11
o 8
i 8
e 7
m 5
k 4
n 4
r 4
f 3
p 3
h 2
_ 2
g 2
ŋ 2
l 1
lː 1
ã 1
u 1
a 16
i 6
e 5
_ 5
n 5
t 5
r 4
f 3
ŋ 3
p 3
u 3
g 3
dʒ 3
ɔ 3
h 2
o 2
ɛ 2
m 2
k 1
l 1
h̃ 1
eː 1
a 18
n 8
i 7
g 6
_ 5
r 5
ə 5
m 5
f 4
e 4
ŋ 4
ɔ 4
k 3
p 3
o 3
h 2
u 2
l 2
t 2
dʒ 1
ɛ 1
; 1
ã 1
a 45
i 21
n 17
e 16
o 13
r 13
_ 12
m 12
g 11
f 10
ŋ 9
p 9
k 8
ɔ 7
t 7
h 6
u 6
ə 5
dʒ 4
l 4
ɛ 3
ã 2
lː 1
; 1
eː 1
h̃ 1

This again shows that the data is not entirely clean – ; and are both unexpected as phonetic segments.

Any other CLDF Wordlist tool you might come along should also work with this dataset. For example, have a look at pylexibank (Forkel 2018), the python package for working with the Wordlists in the LexiBank project. The list of useful tools also includes the BEASTling tool for generating phylogenetic inference driver files, which we will work with in the next step.

Cite this article as: Gereon A. Kaiping, "From Fieldwork to Trees 3: CLDF recipes," in Computer-Assisted Language Comparison in Practice, 21/01/2019,

Author: Gereon A. Kaiping

Gereon A. Kaiping is postdoctoral researcher at the Leiden University Centre for Linguistics (NL) working on models of language evolution and computational historical linguistics of Timor-Alor-Pantar languages and neighbouring Austronesian languages is the Lesser Sunda islands (Indonesia and Timor-Leste).

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.