BioBERTpt : a Portuguese neural language model for clinical named entity recognition

Rubel Schneider, Elisa Terumi (Pontícia Universidade Católica do Paraná, Brazil) ; Andrioli de Souza, João Vitor (Pontícia Universidade Católica do Paraná, Brazil) ; Knafou, Julien (Haute école de gestion de Genève, HES-SO // Haute Ecole Spécialisée de Suisse Occidentale) ; Copara, jenny (Haute école de gestion de Genève, HES-SO // Haute Ecole Spécialisée de Suisse Occidentale) ; Oliveira, Lucas E.S. e (Pontícia Universidade Católica do Paraná, Brazil) ; Gumiel, Yohan B. (Pontícia Universidade Católica do Paraná, Brazil) ; Oliveira, Lucas F.A. de (Pontícia Universidade Católica do Paraná, Brazil) ; Teodoro, Douglas (Haute école de gestion de Genève, HES-SO // Haute Ecole Spécialisée de Suisse Occidentale) ; Paraiso, Emerson Cabrera (Pontícia Universidade Católica do Paraná, Brazil) ; Moro, Claudia (Pontícia Universidade Católica do Paraná, Brazil)

With the growing number of electronic health record data, clinical NLP tasks have be-come increasingly relevant to unlock valu-able information from unstructured clinical text. Although the performance of down-stream NLP tasks, such as named-entity recog-nition (NER), in English corpus has recently improved by contextualised language models, less research is available for clinical texts in low resource languages. Our goal is to assess a deep contextual embedding model for Portuguese, so called BioBERTpt, to sup-port clinical and biomedical NER. We transfer learned information encoded in a multilingual-BERT model to a corpora of clinical narratives and biomedical-scientific papers in Brazilian Portuguese. To evaluate the performance of BioBERTpt, we ran NER experiments on two annotated corpora containing clinical narra-tives and compared the results with existing BERT models. Our in-domain model out-performed the baseline model in F1-score by 2.72%, achieving higher performance in 11 out of 13 assessed entities. We demonstrate that enriching contextual embedding models with domain literature can play an important role in improving performance for specific NLP tasks. The transfer learning process en-hanced the Portuguese biomedical NER model by reducing the necessity of labeled data and the demand for retraining a whole new model.


Conference Type:
published full paper
Faculty:
Economie et Services
School:
HEG - Genève
Institute:
CRAG - Centre de Recherche Appliquée en Gestion
Subject(s):
Informatique
Publisher:
Virtual conference, 19 November 2020
Date:
2020-11
Virtual conference
19 November 2020
Pagination:
Pp. 65–72
Published in:
Proceedings of the 3rd Clinical Natural Language Processing Workshop
DOI:
Appears in Collection:



 Record created 2020-12-03, last modified 2020-12-04

Fulltext:
Download fulltext
PDF

Rate this document:

Rate this document:
1
2
3
 
(Not yet reviewed)