LingPy (List et al. 2017) offers a great deal of functions for string manipulation. Although most of those functions are readily documented (see lingpy.org for details), and the basic ideas have also been described in my dissertation (List 2014), it seems that not many users are aware of these additional possibilities, which the library offers.

In the following, I want to illustrate how we can use LingPy to learn something about consonant clusters occurring in the data underlying the CLICS database (List et al. 2018, clics.clld.org). I have illustrated in an earlier post how one can use the CLICS software API to cook one’s own CLICS application. I will thus assume that you know how to install CLICS (following the instructions on our GitHub page) and the data underlying it.

As a simple shortcut, you can install all required datasets by downloading the data underlying this blog post from GitHub Gist and typing:

$ pip install -r pip-requirements.txt

Once you have done this, please follow the instructions at the CLICS website mentioned above to prepare the CLICS datasets by converting them to CLDF.

Starting from this, and assuming that you have an actual version of LingPy installed, I will now illustrate how you can extract all data that is readily segmented (in the sense of Moran and Cysouw 2018), i.e., by using the space as a segmentation marker, and placing it between all sounds that do not constitute a valid sound unit (see also List et al. 2018b). But additionally, we will compute prosodic strings, an idea that I already discussed in my dissertation (List 2014), and which allows us to distinguish different kinds of consonant in a string, based on whether they appear in an environment in which sonority increases or decreases. In this way, we can make our own cross-linguistic collection of consonant clusters.

But let’s start by loading the releavant libraries.

from lingpy import *
from pyclics.api import Clics
from pyclics.models import Form
from tqdm import tqdm
from collections import defaultdict

To obtain quick access to all the data available in our clics.sqlite3 database, we need to modify the function in the CLICS API slightly, in such a way that the code yields the segmented entries, not the ascified CLICS value that we use for the computation of cross-linguistic colexifications. This is achieved with help of the following function.

def iter_wordlists(db, varieties):
    languages = {
            (v.source, v.id
                ): v for v in varieties}
    for (dsid, vid), v in sorted(
            languages.items()):
        forms = [Form(*row) for row in db.fetchall("""
select
    f.id, f.dataset_id, f.form, f.segments,
    p.name, p.concepticon_id, p.concepticon_gloss,
    p.ontological_category, p.semantic_field 
from
    formtable as f, parametertable as p
where
    f.parameter_id = p.id
    and f.dataset_id = p.dataset_id
    and p.concepticon_id is not null
    and f.language_id = ?
    and f.dataset_id = ?
order by
    f.dataset_id, f.language_id, p.concepticon_id
""", params=(vid, dsid))]
        assert forms
        yield v, forms

In the version of prosodic strings that we want to use for this application, LingPy distinguishes four basic types of prosodic environment: vowels (V), consonants in ascending sonority environment (C), consonants in descending sonority environment (c), and tones (T). To obtain only the consonant clusters from a given sound sequence, we thus need a function that checks if we are currently in consonant environment, returning all clusters from a string. This is done with help of the following function.

def get_clusters(tokens, prostring):
    clusters = ['']
    for t, c in zip(tokens, prostring):
        if c == 'C':
            if clusters[-1].startswith('<'):
                clusters[-1] += ' '+t
            else:
                clusters += ['</ '+t]
        elif c == 'c':
            if clusters[-1].startswith('>'):
                clusters[-1] += ' '+t
            else:
                clusters += ['>/ '+t]
        else:
            clusters += ['']
    return [x for x in clusters if x]

This function will mark clusters in ascending environment by adding </ in the beginning of the cluster, and descending environment by adding a >/. One could think of more elegant ways of marking or handling this, but for our purpose, it is sufficient.

We can now start with the actual code. We start by defining different variables, namely: the CLICS object that allows us to get access to CLICS data, a dictionary that will later be converted to a LingPy Wordlist, to allow for an easy writing to file, and the clusters that we obtained from analyzing the data.

clics = Clics('.') 
D, idx = {0: [ 
    'doculect', 
    'concept', 
    'segments', 
    'cv' 
    ]}, 0 
clusters = defaultdict( 
    lambda : defaultdict(int)

We can now start to load all varieties in CLICS. Usually, I write a print statement after this, since loading the data takes some time, and I prefer some feedback.

varieties = clics.db.varieties
print('[i] loaded clics varieties')

Now, we can loop over the data. We do this with help of the tqdm function that gives visual feedback, since this may take some time. The basic idea of this loop is to retrieve the segments from the CLICS database, check if its valid (using try and except ValueError), and convert it to its prosodic string, using the prosodic_string function provided by LingPy. In addition, we use the get_clusters function to extract the different types of consonant clusters, count them, and store them in our clusters variable.

for v, forms in tqdm(
        iter_wordlists(
            clics.db, 
            varieties
            ), 
        total=len(varieties)
        ):
    for form in forms:
        idx += 1
        clics_form = form.clics_form.strip()
        if clics_form:
            try:
                tokens = clics_form.split()
                prostring = prosodic_string(
                        tokens,
                        _output='CcV'
                        )
                D[idx] = [
                        v.gid, 
                        form.concepticon_id, 
                        clics_form,
                        prostring
                        ]
            except ValueError:
                pass

        clrs = get_clusters(
                tokens, prostring
                )
        for clr in clrs:
            clusters[clr, len(
                clr.split())][v.gid.split('-')[-1]] += 1

Now, that we have obtained all the data, we can write the data to file. First, we write a typical LingPy wordlist, which contains an additional column that we call “CV”. The resulting file is a plain TSV file that can also be inspected with interfaces like EDICTOR (List 2017).

wl = Wordlist(D)
wl.output(
        'tsv', 
        filename='cv-patterns', 
        ignore='all', 
        prettify=False
        )

Now, as a final step, we write the data in our clusters dictionary to file. Here, we write a gain to a TSV file, but we do this “manually” by iterating over the dictionary.

with open('cv-clusters.tsv', 'w') as f:
    f.write('Cluster\tLength\tFrequency\n')
    for (cluster, length), rest in sorted(
            clusters.items(), 
            key=lambda x: len(x[1]),
            reverse=True
            ):
        f.write('{0}\t{1}\t{2}\n'.format(
            cluster, length, len(rest))
            )

Once this is done, we can do many things with the data. We can inspect the occurrence of certain clusters, see whether we find areal patterns, check, which languages are richest in terms of consonant clusters, etc. I won’t discuss any of these possibilities in detail here, as this post was simply written in order to illustrate how easily we can extract the data with help of tools like LingPy and the CLICS API. So if you are interested in the actual results of this little study, I suggest you test the code yourself and see what you get. For convenience, I have uploaded the whole script to a GitHub Gist, which you can find here.

References

List, J.-M. (2014): Sequence comparison in historical linguistics. Düsseldorf University Press: Düsseldorf.

List, J.-M. (2017): A web-based interactive tool for creating, inspecting, editing, and publishing etymological datasets. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. System Demonstrations. 9-12.

List, J.-M., S. Greenhill, and R. Forkel (2017): LingPy. A Python library for quantitative tasks in historical linguistics. Software Package. Version 2.6. Max Planck Institute for the Science of Human History: Jena. http://lingpy.org.

List, J.-M., M. Walworth, S. Greenhill, T. Tresoldi, and R. Forkel (2018): Sequence comparison in computational historical linguistics. Journal of Language Evolution 3.2. 130–144.

List, J.-M., S. Greenhill, C. Anderson, T. Mayer, T. Tresoldi, and R. Forkel (2018): CLICS². An improved database of cross-linguistic colexifications assembling lexical data with help of cross-linguistic data formats. Linguistic Typology 22.2. 277-306.

Moran, S. and M. Cysouw (2018): The Unicode Cookbook for Linguists: Managing writing systems using orthography profiles. Language Science Press: Berlin.

---

---