Semantic data are notoriously difficult to handle. In contrast to the form-part of the linguistic sign, meanings are not organized sequentially, but rather network-like (List 2014: 34f). As a result, we often encounter problems when trying to model complex relations between different meanings, specifically in those cases, where we have only tables as our base material. This blog post tries to summarize how major types of semantic data are handled in the Concepticon project and how they can be accessed in code.

The Concepticon project started out as a collection of concept lists that were linked with each other in order to assess to which degree these concept lists were employing the same concepts (List et al. 2016). Concepts are in this context roughly thought of as certain senses that can be elicited with the help of an elicitation gloss in field work (by asking a speaker “how do you say X in your language”). The Concepticon links concept lists, that is, lists of elicitation glosses in different elicitation languages, by defining Concept Sets, that is, concepts that recur across concept lists. All in all, this is a very straightforward and very pragmatic enterprise, which originally had the only goal of allowing us to systematically trace which datasets would provide access to which concepts.

As of now, the data in Concepticon has been constantly growing, and there is no end in sight. We have a much larger team of annotators now who are trained in checking our links rigorously and we have additional software that helps us to test if our annotations formally correspond to our expectations. By now, we have also added many concept lists that offer additional information which we were not planning to collect initially. Among these concept lists, the most straightforward ones we can distinguish are ranked lists, annotated lists, and lists with relations. You can find examples for these lists by turning to the Concepticon web application and looking for the tags in the Concept Lists tab.

Ranked lists are defined as those lists which do not only provide the elicitation glosses in alphabetical or any other order, but in a specific order that is supposed to reflect their rank. Rank can refer to different aspects of conceptual properties or the properties of words that express these concepts, such as their stability or their frequency.

Annotated lists contain additional columns with additional information which can vary to quite some degrees, ranging from the phonetic representation of the words expressing a given concept elicitation gloss up to an indication of the authors from which other concept lists their concepts were derived.

Lists with relations are the most interesting type of concept lists we have encountered by now. This type refers to those concept lists where authors provide additional information on the relations between the concepts in a given list. These relations can appear in the form of categories, properties, or direct links between the concepts in a given list.

Given that we use an exclusively tabular model in Concepticon, where the primary data point is one line in a table representing one concept elicitation gloss, it may seem at times difficult to model complex concept relations with this schema. For this reason, it seems time to provide some examples on how to proceed when trying to code up concept data for the Concepticon.

1 Handling Ranked Lists

Ranks are rather straightforward to handle. They can be given directly (by providing a Rank field, which displays numbers) or indirectly, by providing a score field which can be used to rank the list. In order to preserve the unity of the ranked lists and to make them more readily accessible also programmatically, we have adopted the practice of adding always an explicit field with numerical ranks, even if this is not given in the original concept list. Since adding ranks is very trivial (in the VIM editor, I would type %s/^/\=line('.')-1/ in order to add incrementing numbers starting from 0 to each line of my list), I won’t discuss them further here. An example for a ranked list is the list of 100 items by Pozniakov (2014), where items are ranked by stability.

2 Annotated Lists

Annotated lists are mannifold, and it is not easy to make a good decision of what parts of the abundant information that scholars may provide when preparing concept lists should be kept and what parts should be discarded. A frequently recurring case of annotation are Comment or Note fields. Here, we have adopted the practice to make sure that our CSV format, which is based on a tabstop as separator, would not be bothered by newline characters inside a given table cell. As a result, we strip newlines if we encounter them in the comment fields that we adopt when adding annotated concept lists to the Concepticon. An example for an annotated concept list is the one by Kassian et al. (2010), referenced as Kassian-2010-116 in Concepticon.

3 Lists with Relations

The most interesting concept lists are those which provide additional relations between the concepts. If we think of relations as of a graph in which different nodes are linked, we can distinguish different types of relations by considering different types of graphs. A very simple graph is an unweighted, undirected network in which nodes are linked by undirected links and the links are not further annotated. This graph can be easily represented in a table by providing one column for the concept and one for the links, which should be separated by some separation character (a simple space in our example here, assuming that concept identifiers do not have spaces).

If we have a weighted graph, where links are annotated by a specific weight, we can handle the weights (and any other kind of annotation) in a table by providing an extra column, which we could call WEIGHTS and adding the weights in numerical form, again using spaces as separators.

We can easily turn this graph into a directed network by discarding asymmetric links.

What is crucial for the display of a true graph with the help of a table in this example is that our Links field needs to provide the same identifiers as we used to define our concepts (which are given in our case in the field Concept). Furthermore, the list must be exhaustive: an identifier that appears only in LINKS but not in Concept will be wrong, as we have not “defined” this concept. With respect to the separators, one can choose whatever separator one wants to choose, and provide this information with the help of the metadata:

{
  "name": "LINKS",
  "separator": " ",
  "datatype": {"base": "string"}
}

Another very frequent type of relations comes along in the form of specific “tags” provided for individual concepts (see another blog post from last year by Ilia Chechuro on tagged markup, Chechuro 2019). When using tags, we assign each concept in a given concept list a certain number of characteristics. Tags differ from our direct network annotation, since tags can be arbitrarily chosen, while the LINKS field in our network concept lists needs to contain entries from our ENGLISH field (or alternatively an ID field).

When modeling tags and concepts in a network, the resulting network is a bipartite network in which two distinct types of nodes occur, one drawn from our ENGLISH field, and one drawn from our TAGS field, and links can only be assigned between the different types and are therefore undirected. A bipartite network also corresponds to a hypergraph, that is, a graph in which edges can connect more than one node (Jacques and List 2019: FN 14), but knowing this is not necessarily important when adding tags to a concept list. Weighting tags is of course also possible. One could add a specific WEIGHTS field to the concept list, or one could add the weights directly to the tag, using a secondary separator.

In the example above, I have used a colon as a secondary operator. In the same way, we could also have defined our weighted graph above, and there are arguments in favor and against such a notation. The advantage is that all data are in one place, so it is clear that when looking at the data alone, that the weights belong to the tags, while our solution by adding a WEIGHTS column does not provide the semantics that would help us to associate WEIGHTS with LINKS. The disadvantage is of course the readability, but also the danger of yielding very complex cell-internal structures that are difficult to test and resolve.

When adding cell-internal separators, it is therefore always important to make this clear with the metadata. While we can use regular expressions to specify the structure of the metadata, we have still not completely solved the problem of not only handling these complex cases consistently, but also making sure that they are rigorously tested.

As an example how you can actively load and analyze a concept list that was provided in the form of a network (with links and weights), let us look at the following example which loads the concept list reflecting the Database of Semantic Shifts (Zalizniak et al. 2020), which was recently added to Concepticon and entails a rather complex network structure.

Outlook

Handling complex concept relations in our Concepticon reference catalog is a relatively new development. While we are confident that we can use our existing solutions for handling the data, there is still a lot to do. In the future, we should add explicit tests for those additional aspects of our data we consider important to share in a comparable and standardized way.

References

Ilia Chechuro, “Why Tag Markup may be Useful for Lexical Data,” in Computer-Assisted Language Comparison in Practice, 03/06/2020, https://calc.hypotheses.org/2476.

Jacques, Guillaume and List, Johann-Mattis (2019): Save the trees: Why we need tree models in linguistic reconstruction (and when we should apply them). Journal of Historical Linguistics 9.1. 128-166.

Hagberg, Aric and Dan Schult and Pieter Swart (2020): NetworkX. Version 2.5. https://networkx.org

Kassian, Alexei and Starostin, George S. and Dybo, A. and Chernov, Vasiliy (2010): The Swadesh wordlist. An attempt at semantic specification. Journal of Language Relationships 4. 46-89.

List, Johann-Mattis (2014): Sequence comparison in historical linguistics. Düsseldorf:Düsseldorf University Press.

List, Johann-Mattis and Cysouw, Michael and Forkel, Robert (2016): Concepticon. A resource for the linking of concept lists. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation. 2393-2400.

Pozdniakov, Konstantin. 2014. O poroge rodstva i indekse stabil’nosti v bazisnoj leksike pri massovom sravnenii: Atlantičeskie jazyki[On the threshold of relationship and the “stability index” of basic lexicon in mass comparison: Atlantic languages]. Journal of Language Relationship 11. 187-237.

Annika Tjuka, “Adding concept lists to Concepticon: A guide for beginners,” in Computer-Assisted Language Comparison in Practice, 29/01/2020, https://calc.hypotheses.org/2225.

Zalizniak, Anna A. and Smirnitskaya, Anna and Russo, Maksim and Mikhailova, Tatiana and Bobrik, Marina and Gruntov, Ilya and Orlova, Maria and Bibaeva, Maria and Voronov, Mikhail (2020: Database of Semantic Shifts (Version from 07/10/2020). URL: http://datasemshift.ru, Moscow: Institute of Linguistics of the Russian Academy of Sciences.