Representing Structural Data in CLDF

The Cross-Linguistic Data Formats initiative (CLDF, https://cldf.clld.org, Forkel et al. 2018) has helped a lot in preparing the CLICS² database of cross-linguistic colexifications (https://clics.lingpy.org, List et al. 2018), since linking our data to Concepticon (https://concepticon.clld.org, List et al. 2016) and Glottolog (https://glottolog.org, Hammarström et al. 2018) has provided incredible help in merging the different datasets into a big comparative dataset.

CLDF, however, is not restricted to lexical data, but can also be successfully used to store structural data, although — due to the nature of structural data — it is much more difficult to compare different datasets. In a recent publication by Szeto et al. 2018 the authors use structural data to compare different Chinese dialect varieties typologically. The data itself is provided in the paper, but unfortunately, the authors do not share it in form of text files, but list it in tables in the original publication. They also do not share the NEXUS file they created in order to analyze the data with the Neighbor-net algorithm (Bryant and Moulton 2004) implemented by the SplitsTree software package (Huson 1998).

Thanks to the help of David Morrison, who extracted the presence-absence matrix from the table in the original paper, I was now able to convert their data to the CLDF format for structural datasets. The data is available via GitHub at cldf-datasets/szetosinitic. The folder contains both the “raw” data that I used (a KML-file with the dialect locations submitted as supplement to the paper, as well as the feature matrix extracted by David Morrison, and the parameter description as typed off by myself), as well as the CLDF data, and a script that converts the raw data into CLDF. In addition, I have added a script that converts the data to a NEXUS file that can be directly read into SplitsTree. Robert Forkel furthermore added automatic tests that will be carried out if users propose changes to the GitHub repository by making a pull request.

In order to test the code for these different conversions, you can simply clone (=download) the data with git:

$ git clone https://github.com/cldf-datasets/szetosinitic

Afterwards, you should install the dependencies, as listed in the file pip-requirements.txt (make sure you use Python3 for this task, as Python2 is not supported):

$ pip install -r pip-requirements.txt

Once this is done, you can test the conversion of the raw data into the CLDF data by typing:

$ python chinese.py

This won’t give you any visual feedback, but it will in fact recreate all the CLDF data in the folder cldf, and it was the way I created the CLDF data in a first instance.

To receive the NEXUS file, just type:

$ python nexus.py

This will read the CLDF data that you just created and write it in NEXUS format ot the file chinese.nex.

To illustrate that one can use this basic procedure for more than just one dataset, I added an older and smaller dataset published by Norman (2003) which I happened to have typed off quite some time before. This dataset can be found on GitHub at cldf-datasets/normansinitic and you can follow exactly the same steps in order to convert this data as well into NEXUS format.

In the future, I hope we can provide more functionality in the CLDF package itself, so that people could use the cldf command that is installed as well if you install the pycldf package, to convert a dataset from CLDF to different formats needed for computations. However, given the diversity of formats, with very different flavors of NEXUS being required by different software packages, we should better wait a bit until we have extracted the most important use cases that can be encountered.

References

Bryant, D. and V. Moulton (2004): Neighbor-Net. An agglomerative method for the construction of phylogenetic networks. Molecular Biology and Evolution 21.2. 255-265.

Forkel, R., J.-M. List, S. Greenhill, C. Rzymski, S. Bank, M. Cysouw, H. Hammarström, M. Haspelmath, G. Kaiping, and R. Gray (forthcoming): Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics. Scientific Data.

Hammarström, H., R. Forkel, and M. Haspelmath (2018): Glottolog. Version 3.3. Max Planck Institute for Evolutionary Anthropology: Leipzig.

Huson, D. (1998): SplitsTree: analyzing and visualizing evolutionary data. Bioinformatics 14.1. 68-73.

List, J.-M., M. Cysouw, and R. Forkel (2016): Concepticon. A resource for the linking of concept lists. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation. 2393-2400.

List, J.-M., S. Greenhill, C. Anderson, T. Mayer, T. Tresoldi, and R. Forkel (2018): CLICS². An improved database of cross-linguistic colexifications assembling lexical data with help of cross-linguistic data formats. Linguistic Typology 22.2. 277-306.

Norman, J. (2003): The Chinese dialects. Phonology. In: Thurgood, G. and R. LaPolla (eds.): The Sino-Tibetan languages. Routledge: London and New York. 72-83.

Szeto, P., U. Ansaldo, and S. Matthews (2018): Typological variation across Mandarin dialects: An areal perspective with a quantitative approach. Linguistic Typology 22.2. 233-275.

Cite this article as: Johann-Mattis List, "Representing Structural Data in CLDF," in Computer-Assisted Language Comparison in Practice, 03/09/2018, https://calc.hypotheses.org/445.

Author: Johann-Mattis List

Seit Anfang 2017 bin ich leitender Wissenschaftler am Max-Planck-Institut für Menschheitsgeschichte in Jena, in der Abteilung für sprachliche und kulturelle Evolution. In meiner Forschung nehme ich generell einen datenbasierten, empirischen und quantitativen Standpunkt in Bezug auf Sprachwandel und Sprachgeschichte ein, mit einem speziellen Fokus auf südostasiatischen Sprachen. Im Gegensatz zu rein computerbasierten Ansätzen versuche ich jedoch, meine Forschung nah an der traditionellen historischen Linguistik und der linguistischen Theorie auszurichten, weshalb ich einen computer-gestützten Ansatz im Gegensatz zu einem rein computer-basierten Ansatz verfolge.

2 thoughts on “Representing Structural Data in CLDF”

  1. Thanks for adding our data to the CLDF database to make it more publicly accessible. And sorry for the time and effort taken to convert our raw data to the appropriate format!
    We’re currently expanding our typological database to encompass a wider range of features and language varieties. We’ll make sure to include the NEXUS file in our future work employing a similar method.

    1. Thanks in advance for this! If you want to provide an updated version in CLDF format, similar to the one we created, please get in touch with us, and we gladly help you with it. Needless to say, that the credits will all be yours, we are just keen on encouraging people to share their data openly in an easy comparable way. Only in this way we can enhance our approaches together.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.