Mapping a given concept list to Concepticon can be done in a straight-forward way, even if automatic mappings need manual refinement. But what can we do when having to deal with larger datasets, say, a dictionary, from which we want to extract specific concepts, such as, for example, the ones in the classical Swadesh list of 100 items (Swadesh 1955)?

In the previous post on this topic, we discussed how the tools offered by the pyconcepticon library, in particular, the concepticon program that can be used from the command line or with a JavaScript interface, make concept mapping easier and more consistent. We also mentioned that they should be enough for automating many of the tasks that make up the mapping of a concept list to Concepticon, especially in those cases when programmers might be tempted to come up with half-baked implementations that are not reusable. Still, cases of one-time, dataset-specific solutions might be desirable or even necessary, such as when dealing with difficult concept lists that need to be pre-processed for manual intervention. While Concepticon, as part of the CLDF standard, is composed of plain-text files that could be loaded and manipulated with any programmer’s favorite language or approach, in most cases it makes sense to use the internal component of the pyconcepticon library, wrapping them around our functions and workflows.

Let’s illustrate this with an actual example, presented step-by-step. We recently had to start mapping a dataset for Barasana, a Tukanoan language spoken in Colombia. We had the following issues:

• Unlike most desirable cases for computer-assisted language comparison, at least in its more common current settings, the data does not come from lists of basic vocabulary (usually collected with the comparative method in mind), but from a bilingual Barasana-Spanish dictionary compiled by Jones and Jones (2009).
• The “dictionary” feature leads to two issues: first, most of the entries are far from comparable in the sense that they are too culture-specific or are clearly the output of word formation processes; second, second, the definitions, while occasionally similar to glosses, are in most cases long and complex lexicographic definitions.
• As the source is a bilingual dictionary, and not a bilingual wordlist or vocabulary, in many cases different words in our source language (Barasana) are translated with the same Spanish “gloss”, especially in case of common vocabulary.
• The same definitions constitute our proxy to the forms glosses, but as mentioned they are given in Spanish, a language with less representation in Concepticon due to the majority of the concept lists already there using English or Mandarin for elicitation.
• For the purpose of the analysis, we did not want to map all possible concepts found in the data and already in Concepticon, but only the essential Swadesh-100 list, thus excluding entries with perfect matches like TELEPHONE.

Most of the issues above should be clear just from the first lines of our raw data:

$ head barasana.tsv
 SOURCE-ORTHOGRAPHY    SOURCE-PHONETIC SOURCE-WORD-CLASS   SOURCE-DEF  ENGLISH-GLOSS   DIALECT-NOTE    GRAMMAR-NOTE
 abarijʉ     ɑ́ˈbɑ.ɾi.hɨ́ E][ɑbɑ́.ˈɾí.hɨ J   s.v.inan.   cosa blanda (como tierra, fruta de árbol, coca pilada, herida, abdomen)         
 abase     ɑ́ˈbɑ.se E][ɑˈbɑ́.sé J  v.i.    (ser/estar) blando, blanda (casabe, lodo)           ntg BARS,91 //aba// ‘soft’ with the stative verb //aba// ‘to be soft’ is// aba-se// ‘that which is soft’, no459; lf caus. 
 abee     ɑˈbéé   interj.     ¡ay no! (exclamación de amor frustrado)             ntg BARS,91 //Abe!// ‘expression of love for someone’ p40, 2.27-2.30 Interjections, 2.29. Exclamatory;
 abi     ɑˈbí    interj.     ¡ay!            ntg BARS,91 //Abi!// ‘Oh, how small!/Oh, what a small quantity!’, p39, 2.29. Exclamatory. //Abo!// ‘oh, how big!; oh, what a huge quantity!’ is an example of an exclamatory interjection. /o/ is used iconically for ‘bigness’ and /i/ is used for ‘smallness’. Thus, //Abi//! ‘oh, how small!; oh, what a small quantity!’ We have recorded over thirty exclamatory interjections, many of which are synonymous, and most of which begin with /a/. Some of these are listed in (118); 
 aboo     ɑˈbóo   interj.     ¡ah!        va abuu , ph ɑˈbúúú ,   ntg BARS,91 //Abo!// ‘Oh, how big!/Oh, what a huge quantity!’, p.39, 2.29. Exclamatory. is an example of an exclamatory interjection. o is used iconically for ‘bigness’ and i is used for ‘smallness’. Thus, //Abi!// ‘oh, how small!; oh, what a small quantity!’ We have recorded over thirty exclamatory interjections, many of which are synonymous, and most of which begin with a. Some of these are listed in (118); 
 abore         v.caus.dep.     ser.blando-CAUS-nS          ntg BARS,91 //abo// (soft.CAUS) p70, no311;
 abu     ˈɑ́bu   inan.s.de masa  alga        va aburi , uv E , vn J  
 aburi     ɑ́ˈbúɾí     inan.s.de masa  alga        va abu , uv J , vn E    
 áburi     ˈɑ́buɾi     inan.s.de masa  zumo; jugo de yuca brava (no venenoso)      

No acceptable automatic mapping seems possible, which makes this one more supporting evidence for our insistence in computer-assisted language comparison: assisted mappings, where our algorithm does not try to be too clever but gives enough information for an expert decision later, are possible. We can demonstrate how such data can be generated, all while relying on previous work by means of the concept_map() and concept_map2() functions of pylexibank (briefly mentioned in the previous post).

While there are some differences in their inner workings, with concept_map() more a “full search” than concept_map2(), both mapping functions work similarly, requiring two lists of strings:

• The first one, glosses, is a list of glosses or definitions we want to map, as found in the source data.
• The second one, map_reference, is a list of reference points composed of Concepticon glosses and reference glosses, separated by triple slashes "///". Examples, as those provided in the previous post and available on-line in pre-compiled lists (such as this for English), are "FLY (INSECT)///fly (insect)" and "FLY (MOVE THROUGH AIR)///fly (as a bird)".

The items composing the glosses list can be pre-processed by the user in terms of textual manipulations, like removing leading and trailing spaces, or information extraction (such as obtaining the actual gloss from definitions with comments, like dog from the dog (noun)), but both functions take care of that by default. The map_reference list can also be compiled by the user as needed, but, in most cases, we will use pyconcepticon‘s _get_map_for_language() function, which loads the precompiled, per-language references from concept lists mapped in the past, allowing us to quickly stand in the shoulders of past concept mappers.

Let’s explore these data structures: we load our raw data using Python’s default csv library, extract the glosses with a list comprehension, and build a reference map from the second element of each item of the value returned by _get_map_for_language() (readers are free to explore the additional information returned by this function; the "es" parameter is the language code for Spanish, which is necessary as pyconcepticon defaults to "en" for English).

# Imports the necessary libraries
import csv
import pyconcepticon.api

# Loads the Concepticon API with data from the specified path
# NOTE: REMEMBER TO SET YOUR OWN PATH IF NECESSARY!!!
CONCEPTICON_PATH = "/home/tresoldi/src/concepticon-data"
Concepticon = pyconcepticon.api.Concepticon(CONCEPTICON_PATH)

# Loads the raw data
with open('barasana.tsv') as csvfile:
    reader = csv.DictReader(csvfile, delimiter='\t')
    data = [row for row in reader]

# Loads the full language mapping, also setting the `lang`uage
lang = "es"
spanish_map = Concepticon._get_map_for_language(lang)

# Builds lists of glosses and references, and shows some contents
glosses = [entry.get('SOURCE-DEF') for entry in data]
map_reference = [entry[1] for entry in spanish_map]
print("glosses", glosses[:5])
print("map_reference", map_reference[:5])

Which returns:

glosses ['cosa blanda (como tierra, fruta de árbol, coca pilada, herida, abdomen)', '(ser/estar) blando, blanda (casabe, lodo) ', '¡ay no! (exclamación de amor frustrado) ', '¡ay! ', '¡ah! ']
map_reference ['ABACUS///Ábaco', 'ABSTAIN FROM FOOD///ayunar', 'ACCORDION///Acordeón', 'ACCUSE///acusar, denunciar', 'ACHIOTE///achiote, bija']

We can now obtain the automatic mapping for each gloss, by calling either concept_map() or concept_map2() on the glosses list (from which we first remove any empty glosses); the differences between the methods will be explained in a future post, but it should be enough to know that they are essentially interchangeable, both returning a dictionary with the indexes in glosses as the keys and a tuple with the list of matched entries in map_reference and the similarity score as the values (concept_map2() return all entries with a similarity score lower than the requested one). Such formal description might be a bit unintuitive, so let’s demonstrate with code:

# Import the functions for the mapping
from pyconcepticon.glosses import concept_map, concept_map2

# Remove empty glosses
glosses = [gloss for gloss in glosses if gloss]

# Perform the mapping and show some of the entries (if they are found,
# otherwise print `None`)
mapping = concept_map(glosses, map_reference, language=lang, similarity_level=5)
for i in range(5):
    print([i, glosses[i], mapping.get(i, None)])

And, especially, with its results (also noting as some definitions, here “glosses”, are not mapped as the algorithm is unable to find an acceptable match with the similarity range we requested), below. The first entry, ([1757], 4), means that a single match, of index 1757 and similarity score of 4, was found for the gloss starting with cosa blanda...; gloss of indexes 2 to 4, all interjections such as ¡ay!, had no match.

[0, 'cosa blanda (como tierra, fruta de árbol, coca pilada, herida, abdomen)', ([1757], 4)]
[1, '(ser/estar) blando, blanda (casabe, lodo) ', ([978], 4)]
[2, '¡ay no! (exclamación de amor frustrado) ', None]
[3, '¡ay! ', None]
[4, '¡ah! ', None]

As humans, we prefer textual representations to indexes of list indexes. The code should do us the heavy and boring work of mapping the latter to the former, so we collect the matched glosses by iterating over the mapping dictionary and extracting the part of the reference string found before the "///" delimiter — note that we do so by setting a default value ([], None), which indicates no matched concepts (its first element is an empty list) and, consequently, no similarity score (thus, None). As the list of matched elements will likely have repeated elements (because more than one reference gloss might be matched, and those will likely share their Concepticon ID), we also take a set of the elements before storing the results we care about in mapped_glosses. This programming logic could be done in a single Python list comprehension, but it is better to proceed stepwise here and collect each gloss match within a for loop.

# Collect matched glosses and show some of them
mapped_glosses = {}
for gloss_idx, gloss in enumerate(glosses):
    match_idxs, sim = mapping.get(gloss_idx, ([], None))

    match_glosses = set([
        map_reference[match_idx].split('///')[0]
        for match_idx in match_idxs
    ])

    mapped_glosses[gloss] = list(match_glosses)

for item in sorted(mapped_glosses.items())[:10]:
    print(item)

The output shows that our code works and, once more, that most definitions will not be mapped.

('(año, semana) pasado', [])
('(estar) abierto, abierta', [])
('(estar) agotado, agotada (persona, animal) ', ['ANIMAL'])
('(estar) agotado, agotada (personas, animales) ', [])
('(estar) agotado, agotada (una persona, un animal) ', [])
('(estar) agrio, agria (limón, lulo, casabe hecho del almidón guardado por demasiado tiempo) ', ['SOUR'])
('(estar) alerto, alerta; ', [])
('(estar) apretado, apretada ', [])
('(estar) arrugada ', [])
('(estar) atrancado, atrancada; ', [])

At this point, we have our automatic mapping of all lexicographic definitions in the source as a list of potential matches in Concepticon with no, one, or multiple elements. We now need to filter only the entries with potential matches in the Swadesh 100 list.

As mentioned earlier, given that Concepticon data is stored in plain-text files, we could just read the raw concepticon-data/concepticondata/conceptlists/Swadesh-1964-100.tsv file and collect all its gloss under the CONCEPTICON_GLOSS column. However, pyconcepticon already has functions in place for iterating over the properties of a concept list, so that it is easier and faster for us to just wrap it in our own code. Considering that we need to perform some checks and operations, and especially that such code might be useful in the future, we can write our own function for collecting the glosses of an existing concept list.

# Define a function for obtaining the glosses of a concept list
def glosses_from_list(list_id):
    """
    Returns a list with the Concepticon glosses for a given conceptlist.

    Takes care of removing duplicates, empty entries, etc.

    Parameters
    ----------
    list_id : str
        The unique identifier for the list, as used in Concepticon, such
        as "Swadesh-1964-100".
    """

    # Obtain the concept list with a list comprehension; note the [0]
    # subsetting for getting the first element of the comprehension
    concept_list = [
        cl for cl in Concepticon.conceptlists.values()
        if cl.id == list_id
    ][0]

    # Obtain all the Concepticon glosses in the list; remember that they
    # glosses might be duplicated or empty at this point
    glosses = [
        concept.concepticon_gloss for concept
        in concept_list.concepts.values()
    ]

    # Remove any empty (i.e., non mapped) glosses and make sure there are
    # no duplicates (using `set`)
    glosses = set([gloss for gloss in glosses if gloss])

    # Return as a list
    return sorted(glosses)

# Obtain the glosses from the Swadesh 100 list and show some of them
swadesh = glosses_from_list("Swadesh-1964-100")
print(swadesh[:10])

Which works as expected:

['ALL', 'ASH', 'BARK', 'BELLY', 'BIG', 'BIRD', 'BITE', 'BLACK', 'BLOOD', 'BONE']

We are now ready to generate our results, iterating over the data we collected earlier and writing to a barasana-swadesh.tsv file the rows that hold potential Swadesh entries. A better output than the one coded below could be provided, making the reviewing task even easier — for example, by sorting entries by similarity score, listing glosses alphabetically, or already including the Concepticon ID. However, this should be easy to any Python programmer and is beyond our scope of illustrating the inner working of pyconcepticon.

# Write the output
with open('barasana-swadesh.tsv', 'w') as handler:
    # Write headers
    handler.write('FORM\tPHONETIC\tDEFINITION\tGLOSSES\tNOTES\n')

    # Iterate over all rows looking for potential Swadesh data
    for row in data:
        # Extract the list of glosses per form, defaulting to an empty list
        glosses = mapped_glosses.get(row['SOURCE-DEF'], [])

        # Get the subset of glosses that are in Swadesh 100
        glosses = [gloss for gloss in glosses if gloss in swadesh]

        # If there is at least one Swadesh gloss, build a buffer from the
        # row, adding an information on all potential glosses (separated with a
        # vertical bar in case of multiple glosses), and print it
        if glosses:
            buf = [
                row['SOURCE-ORTHOGRAPHY'],
                row['SOURCE-PHONETIC'],
                row['SOURCE-DEF'],
                '|'.join(glosses),
                row['GRAMMAR-NOTE'],
            ]

            handler.write('%s\n' % '\t'.join(buf))

We can finally inspect the results and confirm that the code is working, such as in good potential matches as ãmo for HAND and baáre for EAT. We can already spot problems, as well: adocʉ̃ was mapped to PERSON due to the parsing of its definition, and baásãare, clearly a word formation including the baáre above, is also mapped to EAT (which was expected in our logic, as both have the same Spanish gloss “comer”). Nevertheless, we surely saved a lot of time for the linguists that would correct this first mapping, also gaining much in terms of consistency. Even better, once the results will be finished and included in Concepticon, the new concept list from Barasana with Spanish definitions will improve the results for future semi-automatic and automatic mappings, especially in case of other datasets with glosses in Spanish (thus helping, for example, to include more Native American Languages in Lexibank).

$ head barasana-swadesh.tsv 
 FORM    PHONETIC    DEFINITION  GLOSSES NOTES
 adocʉ̃     ɑˈdó.kɨ̃́   este tamaño de (animal, persona)    PERSON  
 ãmo     ˈɑ̃́mõ E][ˈɑ̃́mṍ J  mano    HAND    
 ãmʉa     ɑ̃́ˈmɨ̃ɑ̃ E][ˈɑ̃́mɨ̃́ɑ̃́ J  cuello (ser humano, botella)    NECK    
 ãmʉtutu     ɑ̃́mɨ̃.tuˈtú E][ɑ̃́mˈɨ̃́.tútu J     cuello (ser humano, botella)    NECK    
 baare     ˈbɑ́ɑ́.ɾe E][ˈbɑ́ɑ́.ɾé J    nadar   SWIM    
 baáre     ˈbɑɑ́.ɾe    comer   EAT ntg BARS,91 //ba// ‘eat’ no4; 
 baásãare     ˈbɑɑ́.sɑ̃́ɑ̃́.ɾẽ E][ˈbɑɑ́.sɑ̃́ɑ̃.ɾẽ J   comer   EAT 
 bajirocare     bɑˈhí.ɾókɑ.ɾe   morir (una persona)     DIE ntg BARS,91 p.23, 2.4. Verbs with only a subject argument, p.24,  Verbs which agree in number with the subject. A very productive verb used in verb compounds, both transitive and intransitive, is roka 'to move down/away (s)'; roka becomes rea  with plural subjects. For example, baji roka 'to die (s)' and baji rea 'to die (p)'; 
 boagʉ, boago     ˈbóɑ́.gɨ    (ser/estar) podrido, podrida (animal, persona)  PERSON  

The entire source code here developed is available at this GitHub Gist.

References

Barasana Literacy Committee, Paula S. Jones and Wendell H. Jones (compilers). 2009. Diccionario bilingüe: Eduria & Barasana-Español, Español-Eduria & Barasana. Bogotá, D.C.: Editorial Fundación para el Desarrollo de los Pueblos Marginados. 613pp.

Swadesh, M. (1955): Towards greater accuracy in lexicostatistic dating. International Journal of American Linguistics. 21.2. 121-137.

---

---