The Cross-Linguistic Data Formats initiative (CLDF, https://cldf.clld.org, Forkel et al. 2018) has helped a lot in preparing the CLICS² database of cross-linguistic colexifications (https://clics.lingpy.org, List et al. 2018), since linking our data to Concepticon (https://concepticon.clld.org, List et al. 2016) and Glottolog (https://glottolog.org, Hammarström et al. 2018) has provided incredible help in merging the different datasets into a big comparative dataset.
CLDF, however, is not restricted to lexical data, but can also be successfully used to store structural data, although — due to the nature of structural data — it is much more difficult to compare different datasets. In a recent publication by Szeto et al. 2018 the authors use structural data to compare different Chinese dialect varieties typologically. The data itself is provided in the paper, but unfortunately, the authors do not share it in form of text files, but list it in tables in the original publication. They also do not share the NEXUS file they created in order to analyze the data with the Neighbor-net algorithm (Bryant and Moulton 2004) implemented by the SplitsTree software package (Huson 1998).
Thanks to the help of David Morrison, who extracted the presence-absence matrix from the table in the original paper, I was now able to convert their data to the CLDF format for structural datasets. The data is available via GitHub at cldf-datasets/szetosinitic. The folder contains both the “raw” data that I used (a KML-file with the dialect locations submitted as supplement to the paper, as well as the feature matrix extracted by David Morrison, and the parameter description as typed off by myself), as well as the CLDF data, and a script that converts the raw data into CLDF. In addition, I have added a script that converts the data to a NEXUS file that can be directly read into SplitsTree. Robert Forkel furthermore added automatic tests that will be carried out if users propose changes to the GitHub repository by making a pull request.
In order to test the code for these different conversions, you can simply clone (=download) the data with git:
$ git clone https://github.com/cldf-datasets/szetosinitic
Afterwards, you should install the dependencies, as listed in the file
pip-requirements.txt (make sure you use Python3 for this task, as Python2 is not supported):
$ pip install -r pip-requirements.txt
Once this is done, you can test the conversion of the raw data into the CLDF data by typing:
$ python chinese.py
This won’t give you any visual feedback, but it will in fact recreate all the CLDF data in the folder
cldf, and it was the way I created the CLDF data in a first instance.
To receive the NEXUS file, just type:
$ python nexus.py
This will read the CLDF data that you just created and write it in NEXUS format ot the file
To illustrate that one can use this basic procedure for more than just one dataset, I added an older and smaller dataset published by Norman (2003) which I happened to have typed off quite some time before. This dataset can be found on GitHub at cldf-datasets/normansinitic and you can follow exactly the same steps in order to convert this data as well into NEXUS format.
In the future, I hope we can provide more functionality in the CLDF package itself, so that people could use the
cldf command that is installed as well if you install the
pycldf package, to convert a dataset from CLDF to different formats needed for computations. However, given the diversity of formats, with very different flavors of NEXUS being required by different software packages, we should better wait a bit until we have extracted the most important use cases that can be encountered.
Bryant, D. and V. Moulton (2004): Neighbor-Net. An agglomerative method for the construction of phylogenetic networks. Molecular Biology and Evolution 21.2. 255-265.
Forkel, R., J.-M. List, S. Greenhill, C. Rzymski, S. Bank, M. Cysouw, H. Hammarström, M. Haspelmath, G. Kaiping, and R. Gray (forthcoming): Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics. Scientific Data.
Hammarström, H., R. Forkel, and M. Haspelmath (2018): Glottolog. Version 3.3. Max Planck Institute for Evolutionary Anthropology: Leipzig.
Huson, D. (1998): SplitsTree: analyzing and visualizing evolutionary data. Bioinformatics 14.1. 68-73.
List, J.-M., M. Cysouw, and R. Forkel (2016): Concepticon. A resource for the linking of concept lists. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation. 2393-2400.
List, J.-M., S. Greenhill, C. Anderson, T. Mayer, T. Tresoldi, and R. Forkel (2018): CLICS². An improved database of cross-linguistic colexifications assembling lexical data with help of cross-linguistic data formats. Linguistic Typology 22.2. 277-306.
Norman, J. (2003): The Chinese dialects. Phonology. In: Thurgood, G. and R. LaPolla (eds.): The Sino-Tibetan languages. Routledge: London and New York. 72-83.
Szeto, P., U. Ansaldo, and S. Matthews (2018): Typological variation across Mandarin dialects: An areal perspective with a quantitative approach. Linguistic Typology 22.2. 233-275.