A major problem for data reuse in computer-assisted historical linguistics, especially when employing data collected with no computational workflows in mind, is linking datasets in terms of the meanings of the words (or, technically, “forms”) they carry. Just as linking languages across different datasets is not as straightforward as one might naively assume, demanding a complex reference catalog such as Glottolog, linking the concepts used in a wordlist (a “concept list”) to our Concepticon project might well be the most intensive task in preparing a dataset for cross-linguistic studies.

 

We could discuss the more theoretical issues at hand, ranging from pragmatic lexicographic decisions to deep philosophical disputations on hermeneutics — even though it is always worth remembering, Concepticon is not an ontology, but a catalog for linking concept lists. However, for the time being, let’s take a hard-headed view and look at the practical issues at play. Common issues are:

• Plain errors in the glosses, especially when OCR or typing from published sources is involved, such as in a recent concept list where I had a typo barl for bark (readers will note how “L” and “K” are next to each other in the most common keyboard layouts).
• Problems with homonyms where the meaning is not specified, such as in bark itself (which could be either a noun, the “skin” of a tree, or a verb, “to bark”) or in the recurring examples of fly (either the insect or the verb meaning “to move in the air”) and of dull (either as opposed to “sharp” or to “smart”).
• Problems with synonyms, such as in the case of a word being annotated as eggplant in one concept list but as AUBERGINE in Concepticon.
• Glosses given in a language other than English, and perhaps one you are not entirely familiar with, or sometimes in multiple languages (like Chinese and English) where the difference in semantics might be more of a hindrance than a help.

These problems alone would be enough to make concept mapping tedious, but most people involved in Concepticon would be happy if they were the only ones. Other common issues can lead almost to frustration are:

• The same dataset using slightly similar (and sometimes visually alike) glosses for the same concept, such as dog and dog (note the trailing whitespace in the second one).
• Just like when reporting forms, authors including all kind of notes
  in their glosses, some useful for the mapping (like fly (noun)), some
  less so (such as poor (dubious)) some which would better be in a “Comments” column (like happy [given by only one speaker]).
• People never shy on coming up with different ways to annotate
  additional information, such as part-of-speech, hardly in a consistent way. Using the flying insect as a common denominator, our collection already includes specimens such as fly (noun), fly (n.), fly (n), fly noun, fly:n, fly [n.], (N) fly, fly (insect), the fly, a fly, the/a fly, and fly (mouche), besides mutations such as FLY (n) and fly (noun (note the missing close-parenthesis).
• Notations for polysemies being just as innovative and free, with commas (hand,arm), semicolons (hand;arm), slashes (hand/arm), or even nothing (hand arm), also allowing for inverted orders and spaces (e.g., arm / hand).
• The lists of basic vocabulary including, and for good reasons, terms which are very culture-specific or not colloquial in other languages, when it is not immediately obvious if the concept is already found in Concepticon (and, if not, whether it should be added) and under which gloss.

When facing repetitive tasks, programmers will immediately think of ways to facilitate and automate them. The Concepticon team did just that, with a set of tools from which we can gain a lot in terms of mapping speed and consistency, and it is fundamental to demonstrate such tools and make them known — after all, programmers also tend to have the bad habit of reinventing the wheel.

The first idea for mapping, and which can be used for some quick exploration, is probably to query the Concepticon catalog either online or from the command-line. For example, if we were to map a single coconut-related concept, we would likely think about querying for the string coco using the online catalog or just using the tool “grep” from the command line:

$ grep -i "coco" concepticondata/concepticon.tsv
147    COCONUT TREE    Agriculture and vegetation    A tropical tree with feathery leaves which bears coconuts.    Person/Thing
970    COCONUT    Agriculture and vegetation    The large hard-shelled oval nut with a fibrous husk of the cocos palm.    Person/Thing
1641    SILK    Clothing and grooming    One of the finest textiles, obtained from cocoons of certain species of caterpillars; it is soft, very strong and absorbent and has a brilliant sheen.    Person/Thing
2442    COCOA BEAN    Agriculture and vegetation    The dried and fully fermented fatty seed of Theobroma cacao, from which cocoa solids and cocoa butter are extracted.    Person/Thing
2649    COCONUT SHELL LADLE    Food and drink    A large spoon made from coconut shell.    Person/Thing
3034    GREEN COCONUT    Agriculture and vegetation    A green (i.e., not mature) oval nut with a fibrous husk of the cocos palm, used as source of coconut water.    Person/Thing
3035    RIPE COCONUT    Agriculture and vegetation    A ripe (i.e., mature) oval nut with a fibrous husk of the cocos palm, used as source of coconut meat.    Person/Thing    

This strategy would work for straightforward cases, however missing most of the problems listed above, including homonyms, synonyms, and typos. Some could be circumvented by expanded strategies, such as by collecting synonyms (like the case of eggplant) in the more of 200 concept lists already mapped, checking to which Concepticon gloss the entries are mapped to. Once more, something far from practical.

The pyconcepticon library offers a much better alternative for this kind of query: a JavaScript library with pre-computed information (such as glosses from already mapped concepts lists, as just discussed) which also takes care of performing a number of string manipulations that we would probably need to perform by hand (in particular, all the common and not-so-common kinds of part-of-speech annotation). With pyconcepticon installed, just call

concepticon app

from the command line, and a new browser-tab will be opened where you can run your queries, as in the images below.  Alternatively, you can also visit http://calc.digling.org/concepticon/ where we store a version corresponding to the most recent release of the Concepticon. Note that, depending on how you installed pyconcepticon, you might need to pass as an argument the path to your Concepticon data repository, such as in concepticon --repos /home/tresoldi/concepticon-data/ map_concepts conceptlist.tsv.

Concepticon lookup app for “eggplant”

Concepticon look-up app for “fly

You can see that with this application not only eggplant is matched to AUBERGINE as desired, but also that basic fuzzy matching is allowed, like in the query for fly returning a list of potential mappings FLY (INSECT), FLY (MOVE THROUGH AIR), and FRY. The output includes
a similarity score which we might investigate in more detail in future posts, but in its essence the lower the score, the higher the match probability.

This tool is a great improvement on manual searching, but it would still be impractical for mapping entire concept lists, which usually range anywhere from 100 to 2,500 concepts. Thankfully, we already have a set of Python functions in pyconcepticon that allow automating this procedure, and which can be used from the command line. Let’s create a fake concept list with different mapping issues (some clear-cut cases, but also glosses with annotations, colexified concepts, typing errors, etc.), which we store in a text file with one entry per line (note the mandatory GLOSS column name in the first line):

GLOSS
dog
eggplant
fly (N)
to fly
hand/arm
dull
bambu

By running the command

concepticon map_concepts conceptlist.tsv

we obtain the following output:

GLOSS    CONCEPTICON_ID    CONCEPTICON_GLOSS    SIMILARITY
dog                2009    DOG                           2
eggplant           1146    AUBERGINE                     2
fly (N)            1504    FLY (INSECT)                  2
to fly             1441    FLY (MOVE THROUGH AIR)        1
hand/arm           2121    ARM OR HAND                   2
#<<<
dull               1518    STUPID                        2
dull                379    BLUNT                         2
#>>>
bambu               ???
#    6/7    86%

We can see that the method had no problem to map the five first entries,
dog (a straightforward case), eggplant (using data already mapped), fly (N) (using an annotation for nouns), to fly (using an annotation for verbs), and hand/arm (using an annotation for colexifications). It was not possible to decide if dull should be linked to STUPID or to BLUNT (equal similarity scores), but the items are grouped together and clearly marked, so that review should be easy. Finally, a case of a typo or different orthography, such as bambu, was not mapped as the method does not try to be too clever, but the unmapped item is clearly identified as such with a triple question mark (a fall-back fuzzy string search for these cases might be implemented in the future). The method also informs us about how many concepts were mapped in the last line.

While glosses in English will perform better, due to its status as default language for glosses and a consequent higher count of entries, the same method can be used for other languages. For example, the concept list below, in Spanish:

GLOSS
perro
hombre
mujer
mano
mosca
hacer

can be linked with the same method by passing the --language es argument to concepticon map_concepts, with the output:

GLOSS    CONCEPTICON_ID    CONCEPTICON_GLOSS    SIMILARITY
perro              2009    DOG                           2
hombre             1554    MAN                           2
mujer               962    WOMAN                         2
mano               1277    HAND                          2
mosca              1504    FLY (INSECT)                  2
hacer              2575    DO OR MAKE                    2
#    6/6    100%

It is worth giving some notes on the inner workings of this method, also introducing the internals aspects of Concepticon. The map_concepts command wraps a call to the internal api.map() method, which will read entries from column GLOSS or ENGLISH and parse them with the parse_gloss() function. This helper method offered in the pyconcepticon library is used to extract the most important information from the “raw” gloss, along with its annotated part-of-speech (if any), as illustrated by the code snippet below:

from pyconcepticon import glosses

for gloss in ['kill', 'kill (v)', 'to kill', 'to kill (somebody)']:
    parsed_gloss = glosses.parse_gloss(gloss)[0]
    print([gloss, parsed_gloss.main, parsed_gloss.pos])

Which returns:

['kill', 'kill', '']
['kill (v)', 'kill', 'verb']
['to kill', 'kill', 'verb']
['to kill (somebody)', 'kill', 'verb']

The main part of glosses and their part-of-speech tags are used in combination with mappings in already included concept lists, where each “raw” gloss is aligned to Concepticon as in the snippet below (a full list can be found in the repository tests):

1504    FLY (INSECT)///FLY  1
1504    FLY (INSECT)///Fly (n.) 1
1504    FLY (INSECT)///a fly    4
1504    FLY (INSECT)///blowfly/housefly 4
1504    FLY (INSECT)///fly  16
1504    FLY (INSECT)///fly (N)  1
1504    FLY (INSECT)///fly (animal) 1
1504    FLY (INSECT)///fly (insect) 5
1504    FLY (INSECT)///fly (n)  3
1504    FLY (INSECT)///fly (n.) 2
1504    FLY (INSECT)///fly (nn.)    1
1504    FLY (INSECT)///fly (noun)   1
1504    FLY (INSECT)///fly (sb.)    1
1504    FLY (INSECT)///fly_N    1
1504    FLY (INSECT)///housefly 2
1504    FLY (INSECT)///the fly  2
1504    FLY (INSECT)///the fly (insect) 46
1504    FLY (INSECT)///the fly (insect)s    46
1441    FLY (MOVE THROUGH AIR)///(the bird) flew    1
1441    FLY (MOVE THROUGH AIR)///FLY    1
1441    FLY (MOVE THROUGH AIR)///FLY (VERB) 1
1441    FLY (MOVE THROUGH AIR)///FLY (v.)   1
1441    FLY (MOVE THROUGH AIR)///Fly    2
1441    FLY (MOVE THROUGH AIR)///TO FLY 3
1441    FLY (MOVE THROUGH AIR)///To fly 1
1441    FLY (MOVE THROUGH AIR)///fly    59
1441    FLY (MOVE THROUGH AIR)///fly (as a bird)    1
1441    FLY (MOVE THROUGH AIR)///fly (of bird)  2
1441    FLY (MOVE THROUGH AIR)///fly (to)   1
1441    FLY (MOVE THROUGH AIR)///fly (v)    4
1441    FLY (MOVE THROUGH AIR)///fly (v.)   4
1441    FLY (MOVE THROUGH AIR)///fly (vb)   3
1441    FLY (MOVE THROUGH AIR)///fly (vb.)  2
1441    FLY (MOVE THROUGH AIR)///fly [vb]   1
1441    FLY (MOVE THROUGH AIR)///fly v. 4
1441    FLY (MOVE THROUGH AIR)///fly vb 1
1441    FLY (MOVE THROUGH AIR)///fly, to    2
1441    FLY (MOVE THROUGH AIR)///fly_V  1
1441    FLY (MOVE THROUGH AIR)///flying 2
1441    FLY (MOVE THROUGH AIR)///to fly 26
1441    FLY (MOVE THROUGH AIR)///to fly (move through air)  126
1441    FLY (MOVE THROUGH AIR)///to fly / the bird flies / flew 1
1441    FLY (MOVE THROUGH AIR)///to fly away    1

This mapping does more than inform us that, say, the gloss the fly (insect) is mapped 46 times to the concept FLY (ANIMAL) or that to fly is mapped 26 times to FLY (MOVE THROUGH AIR). It also allows the algorithm to internally parse all the different glosses for “fly” as either a noun or a verb, so that it collects enough information to link any different and still unobserved gloss to the correct concept. We can also take a first glimpse at how pyconcepticon works under the hood, preparing for future blog posts where the internals will be explored in more detail.

Other commands in pyconcepticon might help the mapping process, such as link (to link concepts to a concept set, so that if either CONCEPTICON_GLOSS or CONCEPTICON_ID is given, the other is added) and mergers (which prints the Concepticon id of potential mergers), but map_concepts and app are the most important ones which cannot be lacking from your computer-assisted language comparison toolbox.

References

List, Johann Mattis & Cysouw, Michael & Greenhill, Simon & Forkel, Robert (eds.) 2018. Concepticon. Jena: Max Planck Institute for the Science of Human History. (Available online at http://concepticon.clld.org, Accessed on 2019-03-26.)

 

---

---