No data to crawl? Monolingual corpus creation from PDF files of truly low-resource languages in Peru

• Autores: Bustamante G., Oncevay A., Zariquiey R.
• Formato: artículo
• Fecha de Publicación: 2020
• Institución: Consejo Nacional de Ciencia Tecnología e Innovación
• Repositorio: CONCYTEC-Institucional
• Lenguaje: inglés
• OAI Identifier: oai:repositorio.concytec.gob.pe:20.500.12390/2648
• Enlace del recurso: https://hdl.handle.net/20.500.12390/2648
• Nivel de acceso: acceso abierto
• Materia: Yine; Ashaninka; Corpus creation; Endangered languages; Indigenous languages; Low-resource languages; Monolingual corpus; Pdf processing; Shipibo-Konibo; Yanesha; https://purl.org/pe-repo/ocde/ford#6.02.02

We introduce new monolingual corpora for four indigenous and endangered languages from Peru: Shipibo-konibo, Ashaninka, Yanesha and Yine. Given the total absence of these languages in the web, the extraction and processing of texts from PDF files is relevant in a truly low-resource language scenario. Our procedure for monolingual corpus creation considers language-specific and language-agnostic steps, and focuses on educational PDF files with multilingual sentences, noisy pages and low-structured content. Through an evaluation based on language modelling and character-level perplexity on a subset of manually extracted sentences, we determine that our method allows the creation of clean corpora for the four languages, a key resource for natural language processing tasks nowadays.