Abstract: 

You probably have heard buzz-words like BERT and even how they revolutionized the way we solve NLP problems. BERT is a deep learning model pretrained on large amounts of text, so they can readily provide powerful numerical representations of text for downstream outcome-driven NLP models. Models like BERT are so commoditized you might be using them in-house. But the question remains; Are you getting the most out of these omnipotent monoliths? In this workshop, you will go beyond just using BERT and explore techniques to suit it for the domain-specific task at hand. To do that, we will use an example from the financial domain. But the methods are generalizable to any other domain. By the end of this workshop, you will understand type of data used by the model(s), how to combine BERT with downstream models and advance techniques to get even more impressive results.

Session Outline
Lesson 1: Introduction to BERT
In this lesson, we will understand basics of BERT; a pretrained deep learning based NLP model and how BERT fits into the final NLP task we’re interested in solving. By the end of this lesson, you will be able to explain inputs and outputs of BERT, make predictions with BERT and build a downstream NLP model using BERT’s output.

Lesson 2: Specializing BERT for domain-specific tasks
In this lesson, we will learn how we can leverage domain-specific data to finetune BERT particularly to identify sentiments from financial news. Based on what’s covered, you will be able to prepare textual data from a given domain, finetune BERT with domain-specific text and leverage the finetuned model to solve the task more accurately.

Background Knowledge
High level understanding of basic deep learning model
Basic knowledge in NLP (e.g. basic stages in text pre-processing) will be advantageous
Language: Python
Frameworks: TensorFlow, NLTK, Python libraries (os, pickle, numpy)

Bio: 

Thushan is a Senior Data Scientist at QBE Insurance, Australia, where he works on marrying artificial intelligence with business acumen to solve business problems. He has extensive knowledge of deep learning techniques, that has been poured into two books and a video course on deep learning (TensorFlow). In his free time, Thushan answers questions on StackOverflow, creates YouTube content or presents at local meetups. He obtained his PhD from the University of Sydney, specializing in deep learning and reinforcement learning. When not indulging in the world of data science, Thushan enjoys meditation, swimming and hiking.