USING THE BERT MODEL FOR SEARCH AUTOMATION IN ENGINEERING ACTIVITIES

Abstract

In today’s digital world, where the amount of data created and shared daily is overwhelming, it becomes more and more challenging for engineers to find relevant information when trying to solve their technical problems or improve their technology. Current search and knowledge management applications rely heavily on NLP-based automation. The recent advances in NLP transfer learning have resulted in powerful models, such as BERT, which perform well on NLP tasks in the general domain. In this work, we evaluate different approaches to adapting BERT to the domain of engineering. We compare multiple domain-specific models in their ability to identify new technologies and assign topics to engineering articles. Our experiments show that the domain-adaptation strategy of further pre-training on domain-specific data without vocabulary extension leads to the best performance in this tasks solutions. After the evaluation, we describe the challenges and the limitations of our approach and provide directions for future research.