Scientific data should be openly accessible. This includes databases which are designed for collaborative work. However, in most cases, these databases are only extended by a team of experts. If a database is truly collaborative, the workflows need to be accessible for everybody. The Concepticon database  (List et al., 2019) invites contributors to include their own data sets. This requires a transparent description of the contributing process.

Aim

Concepticon is a resource that stores concept lists. Those lists are hand-curated word lists which include concepts sets. The project is openly accessible with the intention to add new lists from different sources. But as of yet, the contribution process is rather tedious and often involves consultation of experts via the GitHub platform.

In this blog post, I present a step-by-step instruction for adding concept lists to Concepticon.  In doing so, I hope that I can help those who wish to contribute to Concepticon to save some time. In addition, this post also provides a list of helpful links to other descriptions and further readings on the topic.

Installation

First, you should set up a Virtual Environment. Note that you need to install Python if it isn’t already on your computer. You can create and activate the environment with the following commands:

$ virtualenv PATH/TO/NameOfYourEnvironment
$ source PATH/TO/NameOfYourEnvironment/bin/activate

The virtual environment is recommended because some of the packages, which you will install below, depend on each other. Thus, your commands will not work if one of them isn’t installed. Another reason why you should use a virtual environment is that it installs all the packages only in the environment so that the Python library on your computer stays the same.

The basis for our Concepticon website is a GIT repository. Checkout this tutorial if you don’t know how to set up GIT on your computer. For the repository, you should create a new directory with the folder name concepticon, for example, by using the following command:

$ mkdir PATH/TO/concepticon

You can then change into the folder and download or clone our repository here: https://github.com/concepticon/concepticon-data. With GIT, downloading the data is as simple as typing:

$ git clone https://github.com/concepticon/concepticon-data.git

In addition, you need to install the pyconcepticon library:

$ pip install pyconcepticon

This will automatically install a terminal command concepticon on your computer, with which you can, among others, automatically map a concept list to Concepticon concept sets. Before you start, however, you need to make sure that the pyconcepticon toolkit knows where on your computer the concepticon-data folder is stored. While you can pass the location every time you use the command with help of the --repos argument, it is recommended to store this location in a configuration file on your computer. This file can be automatically installed with help of the cldfcatalog package, which is automatically distributed along with the cldfbench package:

$ pip install cldfbench
$ pip install pyglottolog
$ pip install pyclts

The last two commands install two major reference catalogs that are used as part of the Cross-Linguistic Data Formats initiative, namely Glottolog (Hammerström et al., 2019) and CLTS (List et al., 2019). Once you have done so, you can open the following file in the command line with a text editor of your choice and edit it. If you work in a Linux environment, for example, you can type:

$ nano /home/USER/.config/cldf/catalog.ini

Note that the configuration file’s location may differ, with respect to your operation system. Please check the detailed instructions to the appdirs package, which allows to handle configuration files and application directories in a convenient manner across different platforms.

You can now manually add the default path that you want to use for the pyconcepticon package on your system, by inserting it into the file:

[clones]
clts = /PATH/TO/clts/
concepticon = /PATH/TO/concepticon/concepticon-data
glottolog = /PATH/TO/glottolog/

If this seems too complicated to be done upon first run, you can also actively pass the location of your concepticon-data folder when calling the concepticon commands such as map_concepts. In order to do so, just type the following and then the command you’d like to use:

$ concepticon --repos=PATH/TO/concepticon-data COMMAND

In the following, however, we will assume that you have configured the path via the catalog configuration procedure.

After the set up, you should have a folder with your virtual environment and the Concepticon repository, for example:

YOUR/USER/PATH/Projects/VirtualEnv/base-env/
YOUR/USER/PATH/Projects/Repos/concepticon/concepticon-data

If everything is set up correctly, the following command should give you a list of all the commands and arguments of pyconcepticon:

$ concepticon --help

How can I map my list?

You can find an exhaustive instruction about the mapping process here: https://calc.hypotheses.org/1820. To prepare your list, follow the following steps:

• The file must have the following columns: GLOSS or ENGLISH, and NUMBER
• The list must be stored as a tsv-file, which is a tab-separated value text file ending in .tsv
• The name of the file should follow this scheme:

LastNameFirstAuthor-YearOfPublication-NumberOfWords, e.g. Mueller-2000-113.tsv

The mapping of your list with the glosses in Concepticon will be done automatically when you type:

$ concepticon map_concepts PATH/TO/YOURLIST.tsv > test.tsv

If you decided not to use the configuration file that tells pyconcepticon where the Concepticon repository can be found, you can specify with --repos=YOUR/PATH/TO/concepticon-data, where you have stored your concepticon-data folder. If you open your terminal from within the folder concepticon-data (or you cd-ed into that folder), you do not need to specify the repository. The command > test.tsv saves your list as file test.tsv with three additional columns: CONCEPTICON_ID, CONCEPTICON_GLOSS, SIMILARITY. This file will appear in the folder where you opened your terminal.

Now, that you created an automated mapping, you can correct this by cleaning the list as follows: First, you should manually check each mapping proposed by the algorithm. If the algorithm only found one mapping, there will be a Concepticon ID and a Concepticon Gloss in the respective columns. If the algorithm did not find a mapping, this is indicated by three question marks ???. If multiple matches were identified which are equally likely (based on the settings of the mapping algorithm), this is indicated by adding each possible mapping in one extra line, and adding one line which only has #<<< that indicates that a multiple mapping follows, as well as one line #>>> indicating that the multiple mapping has ended. You need to actively resolve all multiple mappings and should delete those lines which have wrong mappings. If none of the multiple mappings seems suitable, try to find a better mapping and add the respective ID and Gloss as provided by Concepticon (e.g., by searching on the Concepticon website, or in our specific app that allows a quick search in multiple languages), or you leave the Concepticon ID and Concepticon Gloss cells empty. In the end, there should be no field with question marks, no fields indicating the start or the end of a multiple mapping, and also no duplicate rows.

To finalize your mapping, remove the SIMILARITY column (which indicates the supposed quality of the match determined by the mapping algorithm), and create identifiers for you concept list with help of the link command (here applied for a fictitious list called Mueller-2000-113.tsv):

$ concepticon link Mueller-2000-113.tsv

Now, that mapping of the list has been finished successfully, you can place it into the Concepticon data folder and proceed to adding additional data and test the mapping.

Which files do I need to update in addition to my list?

If you add new concepts in your list, the following file needs to be updated: PATH/TO/concepticon-data/concepticondata/concepticon.tsv. Add the information of your list in the following file:PATH/TO/concepticon-data/concepticondata/conceptlists.tsv. If your list includes data from other authors, you can specify this in the NOTE column in the following format: [Tischler 1999](:bib:Tischler1999). Update references for your list and secondary sources in: PATH/TO/concepticon-data/concepticondata/references/reference.bib. The BibTeX key (e.g., Tischler1999) for secondary sources must be equivalent to the one you inserted in the conceptlists.tsv.

Last but not least, add a metadata.json file for your list. You can create this file automatically with the following command:

$ concepticon create_metadata

This generates a json-file with the same name of your list in PATH/TO/concepticon-data/concepticondata/conceptlists.

Clean up your metadata.json file:

• If the file adds an additional GLOSS column which does not occur in your file, delete it in the metadata.json.
• Change the datatype of the columns according to your list. The command automatically adds the datatype string to some columns. But the data in your list columns might be of the following types:
  1. float for floating-point numbers (e.g., 1.565765)
  2. integer for whole numbers (e.g., 1)
  3. string for characters (e.g., one)
• If a column has mixed values, for example, because the frequency values come from a secondary study, you may need to add the following:

  {
  "name": "FREQ_DLEX",
  "null": "None",
  "datatype": {
    "base": "float"
    }
  }

Note that you can change the argument for "null" to any string which is used to indicate missing data: NA, NaN, etc.

How can I test if everything is well-integrated?

If you have done all of the steps above you can test your work with the following command:

$ concepticon test

The test spots potential errors. Examples would be:

1. There is only a Concepticon_ID or Concepticon_Gloss missing in a row.
2. You inserted “Chinese” as a source language in conceptlists.tsv and your list does not have a column “CHINESE”.

Where can I find additional information?

Plenty of information on the mapping procedure has been published in the past. For a descriptions of the tags used in conceptlists.tsv, see List (2018). For additional information on how to contribute to the Concepticon project by making a pull request via GitHub, see the documentation supplied with the Concepticon GitHub repository.

References

List, Johann Mattis & Rzymski, Christoph & Greenhill, Simon & Schweikhard, Nathanael & Pianykh, Kristina & Forkel, Robert (eds.) 2019. Concepticon 2.2.0. Jena: Max Planck Institute for the Science of Human History. (Available online at http://concepticon.clld.org, Accessed on 2020-01-24.)

Johann-Mattis List, Cormac Anderson, Tiago Tresoldi, Simon J. Greenhill, Christoph Rzymski, & Robert Forkel. (2019). Cross-Linguistic Transcription Systems (Version v1.2.0). Max Planck Institute for the Science of Human History. (Available online at http://glottolog.org, Accessed on 2020-01-24.)

List, J.-M. (2018): Towards a history of concept list compilation in historical linguistics. History and Philosophy of the Language Sciences 5.10. 1-14. URL: https://hiphilangsci.net/2018/10/31/concept-list-compilation/

Hammarström, Harald & Forkel, Robert & Haspelmath, Martin. 2019. Glottolog 4.1. Jena: Max Planck Institute for the Science of Human History. (Available online at http://glottolog.org, Accessed on 2020-01-24.)

---

---