Deep Learning Archives | Open Data Science Conference

Mar

Brand Voice: Deep Learning for Speech Synthesis

The production of artificial natural-sounding human speech is a fascinating topic due to its complexity and surprising results, with applications that range from chatbots to the automatization of audio content in news media. One obvious example of a Text-to-Speech (TTS) application for news media is a Read Aloud feature for a news article that makes it possible to listen while commuting. In fact, this feature is already offered by some newspapers, though they mainly use external TTS services to power their applications.

As Europe’s largest digital publisher, we decided to go one step further by building up our own technology in-house and creating a custom brand voice instead of relying on third-party providers. Out of our work, have released two Open Source projects: ForwardTacotron and TransformerTTS.

Here, and then more in-depth and interactively in the workshop at ODSC East 2021, we will have a look at the steps needed to train a TTS Transformer-based neural network. We will then rely on the TransformerTTS repository.

Go ahead and listen to some of our samples!

ForwardTacotron

TransformerTTS

If you are new to the topic of speech synthesis this will serve as an introduction to the general concepts of the workshop session, in which we will go into all the interesting details as well as do some play-around with the models.

If you already are familiar with the topic of speech synthesis this will be useful to refresh some of the ideas and get to know some of the details of the repository that we will use during the workshop.

Speech Synthesis

Modern speech synthesis is a multi-step problem where multiple neural networks are trained and deployed to convert raw text into a natural sounding voice and one of the best approaches, Microsoft released their FastSpeech paper in 2019, this process is divided into 3 steps:

– aligning text and audio using an autoregressive model
– converting text to spectrogram using a feed-forward TTS model
– reconstructing natural sounding voice from spectrograms with a neural vocoder model, like MelGAN or WaveRNN

Speech can also be reconstructed from the TTS model outputs using the Griffin-Lim algorithm, although the generated voice will not sound as natural.

Here we will focus on the first two steps, which dictate the prosody and expressiveness of the produced speech by doing the actual text-to-speech translation.

Let’s dive in.

Setup

The Data

As with most machine learning problems, the first step is gathering the data. In this case, training data is made of wav files containing the voice you want to reproduce and the corresponding transcriptions.

Like most interesting things in life, there is a bright side and a dark side.

In the dark side lies the fact that to obtain some reasonable results you will need to collect at the very least about 7–8 hours of very clean and diverse data (common dataset used in research contain about 20 hours of clear text-voice data).

On the bright side, you will not need the sound files and the text to be aligned with complicated timestamps and pre-computed text features: raw voice and text will be enough, as long as they do represent each other.

So make sure that the transcriptions are correct and consistent!

Create a csv file containing all your transcriptions: if you have a voice file called my_voice_000001.wav saying “I am a wonderful being.”, your metadata.csv file should contain the line

my_voice_000001.wav|I am a wonderful being.

There is actually another good news, in case you don’t have the data and don’t want to spend hours in front of a professional microphone: there is some well-curated open-source data out there for you to train on, one of the most commonly used being LJSpeech.

Go ahead and get it from https://keithito.com/LJ-Speech-Dataset/. This dataset already contains a wav folder and a metadata file in the format we like.

Now that we got the data ready, let’s jump into the next step:

The Code for Speech Synthesis

Clone the Speech Synthesis repository and follow the installation instructions.

git clone https://github.com/as-ideas/TransformerTTS.git

Preprocessing: the what

The first thing you want to do after getting the data, is preparing it for your models.

There are two preprocessing steps, one for text and one for audio.

Due to the discrepancy between how things are written and how they are actually spoken out loud, it is convenient to convert the text into phonemes: these are perceptually distinct units of sound that distinguish one word from another in a particular language. For instance the words fake news are represented as ˈfɛɪ̯kˌnjuːs. You can check https://en.wiktionary.org for more of these. This representation has two major advantages. First of all, it makes the speech to text mapping injective, as there will be only one way of transcribing the voice with phonemes, which greatly simplifies the text-to-speech task, although it remains an ill-defined problem, as there is many ways any word can be spoken: pitch, rhythm and volume are some of the features of speech which are not represented by phonemes. Afterall there is a reason why we summon those big neural networks for the task! A second advantage is control and flexibility: if your model learns to pronounce all the phonemes correctly it could, in principle, pronounce any word of any language. You will see this is not really the case, as we will need to present the model with combinations of these phonemes for it to sounds natural.

The next step is preparing the audio: in order to again simplify the problem and work in the neat space of Fourier transforms which greatly simplify the operations on waveforms, we convert our audio snippets into Mel-Spectrogram (“mel” in short): these are power-spectra, a logarithmically scaled windowed Short-time Fourier transform of the audio.

Speech Synthesis — Mel-Spectrogram of an utterance

Basically now we deal with compressed images instead of lengthy audio time-series, a computationally much cheaper task.

Preprocessing: the how

In order to actually do all of these, let us jump back to our repo and set up the yaml configuration files. There isn’t much you need to do as the standard settings will work for most use-cases.

Open up the session_paths.yaml file and edit it. Assuming you have the wav and metadata files under the directory /data/datasets/MyVoice/ and that you want to keep the training logs and weight files under /data/logs/tts/

wav_directory: '/data/datasets/MyVoice/'
metadata_path: '/data/datasets/MyVoice/metadata.csv'
log_directory: '/data/logs/tts/'
train_data_directory: 'transformer_tts_data'
data_config: './data_config.yaml'
aligner_config: './aligner_config.yaml'
tts_config: './tts_config.yaml'data_name: ljspeech # raw data naming for default data reader (select function from data/metadata_readers.py)

Under data_name you can specify a name for your data. Should you change it, make sure to add an appropriate metadata reader to the python file data/metadata_readers.py. For instance, for data called my_voice, that has the metadata you actually want to use on the third column of the csv instead of the second:

def my_voice(metadata_path: str, column_sep='|') -> dict:
    text_dict = {}
    with open(metadata_path, 'r', encoding='utf-8') as f:
        for l in f.readlines():
            l_split = l.split(column_sep)
            filename, text = l_split[0], l_split[2]
            if filename.endswith('.wav'):
                filename = filename.split('.')[0]
            text = text.replace('n', '')
            text_dict.update({filename: text})
    return text_dict

Let us leave the other options unchanged for now.

Note that if you are using the LJSpeech dataset the standard options will allow you to use a pre-trained vocoder model from any of these great open source repositories: MelGAN and HiFi-GAN.

All set, now from the root project folder run

python create_training_data.py --config configs/session_paths.yaml

Training

Now that we carried all the necessary preprocessing steps, let us train some models, shall we?

Aligning the data: the what

As mentioned earlier, we do not need text-audio alignment information: we can create it ourselves. The first step will in fact be to train an aligner model, that will compute the duration for each of the phonemes and all the punctuation present in the text.

This is a very delicate step in the pipeline, as it will greatly impact the results of our final model.

To extract these alignments we use an autoregressive model that takes as inputs both the phonemized metadata and the correspondent target mel: given the whole text input sequence and the first nth mel timesteps, the model will predict the (n+1)th spectrogram timestep.

A major trick and caveat here is that we can and want to use the target mels to run the autoregression in parallel in what is called a “teacher forcing” process: instead of using its own first n predictions to predict the next (as a normal autoregressive model would do), we use the first n targets as input.
This, together with a causal mask (an upper triangular mask) which prevents the model from using the information from the future timesteps, allows us to train all the timesteps in parallel.

Interestingly, we will use teacher forcing also during prediction: all we need are those sweet sweet alignments which, with teacher forcing, are easier to computed. No need to bother doing actual (hard) autoregressive predictions!

Aligning the data: the how

To train the aligner model, simply run from the root project folder

python train_aligner.py --config configs/session_paths.yaml

During this phase the transformer model will build an attention mapping between the input and the target, which we will then use to extract the alignent information, or phoneme-durations, that we will need for our final TTS model.

Image for post — Attention mapping: the horizontal and vertical axes correspond to mel-timesteps and phonemes respectively

There is a lot of nitty gritty details that go into this phase, as well as clear indicators of whether your training session is being successful. All of this can be easily monitored through the tensorboard session so make sure to tune in for our workshop to get all the details and a thorough explanation of this step.

Get them durations

Now that we trained the aligner, one of the hardest steps is behind us. All we’re left to do is to extract the information we need from the aligner model and train the TTS model.

Given that the attention maps are often far from perfect and contain uncertainty, as you can probably grasp from the above picture, we use a graph algorithm to make sure that the durations we extract are solid: the famous Dijkstra algorithm for minimum paths.

We will dig deeper into this process during the workshop; for now it’s enough to know that this step helps us exclude uncertainty and ignore unlikely jumps that occur in the raw attention maps. In the picture below you can see the result— a much cleaner extracted alignment map on top of the actual attention maps.

Now that we have cleaned the attention maps, we can finally simply count how many mel timesteps correspond to each phoneme to get our phoneme-durations and proceed to the next step!

Again, no need to code any of this yourself, simply run

python extract_durations.py --config configs/session_paths.yaml

to get the last piece of the puzzle.

Train the TTS model

Finally

python train_tts.py --config configs/session_paths.yaml

will trigger the last operation of the pipeline.

With this step we will train a FastPitch-like model, that will predict on the fly the duration for each phoneme (using the data extracted with the previous step) as well as the pitch profile of the target audio file.

There are some further choices that go into this training step, such as minor architecture details as well quite a few knobs, levers and indicators to keep under control, all of which you can access from the Tensorboard session.

Once training is done, using the model will be as easy as

from utils.config_manager import Config
from data.audio import Audio

config_loader = Config(config_path=f'config/session_paths.yaml')
audio = Audio(config_loader.config)
model = config_loader.load_model()
out = model.predict("I don't speak often, but when I do, it's for ODSC East.")

# Convert spectrogram to wav (with griffin lim)
wav = audio.reconstruct_waveform(out['mel'].numpy().T)

Having control over pitch and speed/duration allows us to greatly control the prosody of the synthesized voice and, most importantly, enhance our results with greater control during training.

If you wish to know more about the remaining details, have a thorough look at a full training session at ODSC East 2021 (over 3 days of training), play around with some model or maybe just chat about the topic, please join our workshop session, “Brand Voice: Deep Learning for Speech Synthesis,” on April 1st at 11 AM EST.

Hope to see you there!

About the author on Deep Learning for Speech Synthesis/ODSC East 2021 speaker:

Francesco Cardinale received his MS degree in Computational Mathematics in 2017 from the Department of Mathematics at the Technical University of Berlin. After a research experience in Bayesian machine learning, he pivoted into deep learning with generative models and computer vision. In his current position as a machine learning research engineer, he works on NLP and speech synthesis and has presented and authored a few open source projects related to these topics.

Mar

PyTorch Lightning: From Research to Production, Minus the Boilerplate

The following post introduces PyTorch Lightning, outlines its core design philosophy, and provides inline examples of how this philosophy enables more reproducible and production-capable deep learning code.

What is PyTorch Lightning?

PyTorch Lighting is a lightweight PyTorch wrapper for high-performance AI research.

Simply put, PyTorch Lightning is just organized PyTorch code.

Organizing PyTorch code with Lightning enables seamless training on multiple-GPUs, TPUs, CPUs and the use of difficult to implement best practices such as model sharding and mixed precision.

Losing the Boiler Plate PyTorch Lightning Design Philosophy Explained

1. Self Contained Models and Data

One of the traditional bottlenecks to reproducibility in deep learning is that models are often thought of as just a graph of computations and weights.

Example PyTorch Computation Graph from the PyTorch AutoGrad Docs

In reality, reproducing Deep Learning requires mechanisms to keep track of components such as initializations, optimizers, loss functions, data transforms, and augmentations.

A core design philosophy of PyTorch Lightning is that all the components and code related to reproducibility should be self-contained. A good test to see how self-contained your model is to ask yourself this question: “Can someone drop this file into a Trainer without knowing anything about the internals?”

The lightning module contains all the default initialization parameters needed for reproducibility.

2. Modular Code

PyTorch Lightning provides a modular framework to decouple research and data code resulting in faster iteration and more reproducible code.

Visualized Modularization of Deep Learning Code with Lighting

The modular nature of this code increases readability. For example, if I want to understand how a module trains or inferences instead of guessing where in the code this is implemented, I can look at the modules training_step and forward functions.

Similarly, if I want to know how data was preprocessed, transformed, or split, I can look at the Data Modules init, prepare_data, and setup functions.

3. Reduce the Boiler Plate

After decoupling your research and data code, the remaining boilerplate is managed by Lightning, providing implementations of proven deep learning best practices to reduce error and training time.

The Lightning Trainer standardizes the boilerplate with best practices to reduce ~80% of the most common Deep Learning errors.

Otherwise, state-of-the-art methods are often abandoned because boilerplate code such as ensuring that the model is not accidentally configured to evaluation mode on fine-tuning was not properly configured.

Lightning manages boilerplate code, such as device optimization, logging, process rank management, and more so that researchers can focus on building the best models possible. If users have special use cases requiring additional abstraction, they can create and share their own callbacks.

4. Maximum Flexibility

For research to flourish, tools must be flexible. Lightning’s standardized best practices are accessible to the end-user as overridable hooks enabling maximum flexibility for those who want to experiment with crazy ideas that stray off the standard path.

An example Auto Encoder training_step override that demonstrates full access to the underlying boilerplate when needed.

Since hooks for processes such as loss configuration are standardized, Lightning makes it much easier to experiment with custom losses for domain-specific applications and combine different models for more complex Multi-Input Modeling scenarios.

From Reproducible Research to Production Deep Learning

Now that we have a better understanding of the core design philosophy of underlying PyTorch Lightning, let’s look at some of the cool features this enables out the box from Multi GPU and TPU training to one line Onnx and Torch Script export.

If you want to get started quickly, Lightning also provides an example implementation of common Deep Learning Tasks from Text Summarization to Object Detection as part of the PyTorch Lightning Flash repo.

PyTorchLightning/lightning-flash

Read our launch blogpost Pip / conda pip install lightning-flash -U Pip from source pip install…github.com

Production Scale Training with Grid

While Lightning helps keep your PyTorch code organized, reproducible and scalable there is one remaining barrier towards production that we have yet to discuss, managing infrastructure. Often orchestrating compute and data pipelines to train and serve models at scale requires extensive configuration orcode modification to accomplish. That is where Grid comes in, Grid manages this overhead for you enabling PyTorch Lightning code to scale from a laptop to the cloud without changing a single line of code.

With Grid trains you can take a PyTorch Lightning script and scale and track hundreds of experiments as follows.

If you don’t have a powerful enough laptop interactive nodes provide optimized development environments that make it easier to hit the ground running developing your code enabling true production scale training of applied state of the art deep learning models.

Conclusions on PyTorch Lightning

This post shows how Lightning’s core design principles enable more reproducible and production-ready deep learning code.

Lightning code is clearer to read because engineering code is abstracted away, and common functions such as training_steps, process_data are standardized. Lightning handles the tricky engineering preventing common mistakes while enabling access to all the flexibility of PyTorch when needed.
Lightning modules are hardware agnostic; if your code runs on a CPU, it will run on GPUs, TPUs, and clusters without requiring gradient accumulation or process rank management. You can even implement your own custom accelerators.
Each release is tested rigorously with every new PR on every supported version of PyTorch and Python, OS, multi GPUs, and even TPUs.Lightning has dozens of integrations with popular machine learning tools such as TensorBoard, CometML, and Neptune.
Grid enables you to scale production training of your PyTorch Lightning code from your laptop to the cloud without having to modify a single line of code.

Article written by Ari Bornstein & Sean Narenthiran.

Editor’s note: Learn more about PyTorch Lightning in William Falcon’s ODSC East 2021 talk, “From Research to Production, Minus the Boilerplate,” there!

About the ODSC Speaker:

William Falcon is the creator of the popular open-source project PyTorch Lightning, and the recently announced Grid AI. William created Lightning while doing his PhD at NYU and as a PhD researcher at Facebook AI; Lightning allows users to scale models without the boilerplate and Grid enables large-scale training on the cloud. Previously he co-founded the now acquired NextGenVest and spent time at Goldman Sachs. His PhD (currently on leave to focus on Lightning), is funded by Google Deepmind and NSF Foundation. His research interest is in unsupervised learning and the intersection of AI and neuroscience. William is a native of Venezuela and holds a BA from Columbia University in Computer Science and Statistics, with a minor in Math.

Mar

Dataset Management for Computer Vision

When building computer vision solutions, the emphasis is usually on the modeling side and on leveraging the latest algorithm. While the model is important, in my experience I have found that an even more important component to delivering a successful solution is to build and maintain a suitable dataset via efficient dataset management.

Why dataset management is key

From a technical standpoint:

Let’s look at things from a representation theory standpoint. For any supervised task, modeling is essentially about teaching a model a representation of the world that allows going from the input data to our labels.

In “classic machine learning,” we do this via feature engineering – essentially changing the surface of the loss function to be more treatable so that it’s easy for a model to learn.

The same thing holds true in computer vision. One difference here is that when using deep learning, we have little explicit control on the feature engineering part of the process.

The job of making the loss landscape smoother can however still be carried out by building and maintaining a suitable dataset. That’s a big, golden knob we can turn and spin to control our experiments with whatever fancy model we use.

From a more intuitive standpoint:

Before training, a model is a blank canvas*. It doesn’t know what a line is, it doesn’t know what an object is – we are teaching it everything we need it to know, and we are doing this by showing it pictures and labels.

This means that we need both:

to be extremely careful in what we teach
to try and find the easiest ways to pass the message.

* In many cases, one starts with a model that has already been pre-trained, so it actually knows something about the world. There are some nuances to it of course, but the same principles discussed here apply.

What to look out for – the “ah moments”

Looking back at the work that I have done, I tried to distill some key aspects to look out for.

Those listed below have all led me to some “ah” moments, either when building or when maintaining a model, and as such, I now explicitly set them as action points when working on a new model:

Disentangle business goals from the modeling task
Understand the domain well
Think about corner cases, assumptions and express them in the dataset
Think about bias in the input data
Think about data drift once live – how to catch it, how to act on it

Dataset Management Corner cases and assumptions

There are many cases in which it might be hard for a model to perform well on identifying what we set it to identify.

This could be due to images being of poor quality, being taken from a different angle, being partially obstructed, etc. Many of these can be called “corner cases,” but they might be very prominent up to the point of being key to the success or failure of our project.

Unfortunately, we don’t know how corner they are unless we explicitly ask ourselves the question.

A simple example explains best how I approach asking and trying to answer it:

Let’s imagine we want to teach a model to recognize dogs. In our training set, we might find the pictures below, and we might wonder how to deal with them.

Do we show them to the model? What do we tell the model?

Those are all cases that our model might face when deployed live. If we have these examples on our training dataset, the best thing to do is to explicitly try to tackle them – whether we will be successful or not, and whether “tackling it” means to ignore them or not.

Case 1: full dog

This is a simple example. Here we would simply draw a box around the dog (assuming an Object Detection task for simplicity), and we are good.

Case 2: half-a-dog?

What if instead of this picture, we had one where only got some part of the dog? How would we approach it?

Case 3: what’s that?

Taking case 2 to an extreme, what if we just have cases where we can only see a tail?

What to do in these cases

There are not always clear cut answers. One approach that worked well with me across domains, has been to put myself in the “model’s shoes.” What might it think? What would I want it to think?

If we label the tail alone as “Dog,” we are effectively telling the model that the tail IS a “Dog” – i.e. the tail and a full complete dog are the very same thing.

Leaving aside ontological and philosophical considerations about this (which are also super intriguing!), in practical terms this can be problematic because the model learns based on what it sees. Maybe the model could learn to recognize the head, the ears, and the shape of a dog – but we make the job that much harder by passing it conflicting information.

What’s more interesting is that this is not necessarily bound to fail. Maybe the model can effectively learn underlying concepts such as body parts, without being expressly told so. Or maybe it can’t. (For reference: recent AI research has been investigating this. This is beyond the scope of the article, but look up “disentangling representations”).

My line of action in this case is:

Think if it’s important to take on these corner cases
If it is, try to be explicit about them: I set up an experimental framework to try and take them on.

So back to the example, I would first try and think

If we can expect to have many pictures of tails alone in our images for whatever reason
What would be the impact on the bottom line if the model failed to recognized tails: maybe it matters, maybe it doesn’t, depending on the application

It’s important to ask ourselves if that matters as accounting for corner cases has trade-offs, and can increase the effort and costs needed to build the model.

If the answers to both questions point at us needing to take on the problem, I would then possibly explicitly annotate the input images so that I can test what works:

Use “dog” to represent both full dogs and tails
Use “dog” to represent full dogs, ignore tails
Use “dog” to represent full dogs, use “tail” to represent tails

An efficient way to do this is to have granular annotations in the input data, which can we then easily combine from code as we wish to run explicit experiments.

—

That is for now! If you found this article on dataset management interesting I will go more into detail and touch on other points during my ODSC East 2021 talk, “Dataset Management for Computer Vision: Possibly the Most Underrated and Important Component to Delivering Successful Computer Vision Solutions in Real-life.”

About the author/ODSC East 2021 speaker, Andrea La Rosa:

Founder at Remo.ai. Data Scientist, 10y in Quantitative Analytics and Machine Learning across industries, working for big enterprises and startups. Board games and surfing in my free time

Oct

Deep Learning-Driven Text Summarization & Explainability with Reuters News Data

Image credit: REUTERS/Dominic Ebenbichler for Reuters news data

Editor’s note: At ODSC West 2020, Nadja Herger, Nina Hristozova, and Viktoriia Samatova will hold a workshop focused on text summarization and that will allow you to automatically generate news headlines powered by Reuters News, and learn about the power of transfer learning and explainable AI.

Natural Language Processing (NLP) is one of the fastest-moving fields within AI and it encompasses a wide range of tasks, such as text classification, question-answering, translation, topic modeling, sentiment analysis, and summarization. Here, we focus on text summarization, which is a powerful and challenging application of NLP.

Summarization & Transfer Learning

When discussing summarization, an important distinction to make is between extractive and abstractive summarization. Extractive summarization refers to the process of extracting words and phrases from the text itself to create a summary. Abstractive summarization more closely resembles the way humans write summaries [link]. The key information of the original text is maintained using semantically consistent words and phrases. Due to its complexity, it relies on advances in Deep Learning to be successful [source].

Here, we investigate the automatic generation of headlines from English news articles across all content categories based on the Reuters News Archive, which is professionally produced by journalists and strictly follows rules of integrity, independence and freedom from bias [source]. The headlines themselves are considered fairly abstractive, with over 70% of bi-grams, and over 90% of 3-grams being novel.

We see a trend towards pre-training Deep Learning models on a large text corpus and fine-tuning them for a specific downstream task (also known as transfer learning) [source]. This has the advantage of reduced training time, as well as needing less training data to achieve satisfactory results. Due to the democratization of AI, we observe a leveling of the playing field where everyone can get hold of these models and adapt them for their use cases. We finetuned a state-of-the-art summarization model on Reuters news data, which significantly outperformed the base model itself. An example of a tokenized, unformatted article text and associated machine-generated headline is shown below. The original article text was published by Reuters in October 2019 [link].

Explainability

Do you trust this automatically generated news headline? Researchers commonly rely on the ROUGE score to evaluate the model’s performance for such a task [source]. In its most basic form, it essentially measures the overlap of n-grams between the machine-generated and human-written summaries. If I told you that the model has a ROUGE score of around 45 on the hold-out set, is that sufficient for you to trust the prediction on a previously unseen article text?

How can we increase trust in what the model generated? The move towards more complex models for NLP tasks makes the need for explainable AI more apparent. Explainable AI is an umbrella term for a range of techniques, algorithms, and methods, which accompany outputs from AI systems with explanations [source]. As such, it addresses the often undesired black-box nature of many AI systems, and subsequently allows users to understand, trust, and manage AI solutions. The desired level of explainability depends on the end user [source]. Here, we are interested in making the model output explainable to a potential reviewer rather than for example an AI system builder, who would have different expectations in terms of technical details.

Let us take a look at how adding an explainability feature can support us in our task of verifying if the machine-generated headline is factually accurate. In addition to just generating the headline, we can gain insights into the most relevant parts of the news article. The illustration below builds upon the example shared earlier, by adding highlights to the article text in Reuters news data.

The darker the highlights, the more important a given word for the resulting headline text. This makes it significantly easier to verify the headline itself. Particularly the first sentence seems to have the largest impact on the generated headline. Interestingly, it refers to “until early next year” rather than “until early 2019”. The year “2019” never occurs in the article text. For this particular example, we actually have access to the human-written headline, see screenshot of the original article from the Reuters site below.

Knowing that the article was published in 2019, it is evident that “early next year” refers to the year 2020 rather than 2019, and thus renders the machine-generated headline partially inaccurate. We believe that verifying machine-generated headlines with an extra layer of explainability leads to increased trust and easier detection of biases or mistakes.

For more details on text summarization and Reuters news data, the power of transfer learning, as well as adding explainability for increased trust, join us for our hands-on workshop at ODSC West in October. You will walk away with an interactive notebook to get a head start in applying these concepts to your own challenges!

Nadja Herger is a Data Scientist at Thomson Reuters Labs, based in Switzerland. She is primarily focusing on Deep Learning PoCs within the Labs, where she is working on applied NLP projects in the legal and news domains, applying her skills to text classification, metadata extraction, and summarization tasks.

Nina Hristozova is a Data Scientist at Thomson Reuters (TR) Labs. She has a BSc in Computer Science from the University of Glasgow, Scotland. As part of her role at TR she has worked on a wide range of projects applying ML and DL to a variety of NLP problems. Her current focus is on applied summarization of legal text.

Viktoriia Samatova is a Head of Applied Innovation team of Data Scientists within Reuters Technology division focused on discovering and applying new technologies to enhance Reuters products and improving efficiency of news content production and discoverability.

Oct

Why I Love Keras and Why You Should Too

I started working with Deep Learning (DL) in the 2016 – 2017 time frame when the framework ecosystem was much more diverse and fragmented than it is today. Theano was the gold standard at the time, Tensorflow had just been released, and DeepLearning4j was still being built. Both Theano and Tensorflow were relatively low-level frameworks, and were somewhat painful to work with, especially for newbies like me. The Keras and Lasagne libraries helped alleviate the pain somewhat, offering a higher level API — Lasagne wrapped Theano, while Keras provided a common interface for either Theano or Tensorflow.

[Related article: Deep Learning in R with Keras]

My first DL project was Image Classification. By this time, the ImageNet problem was pretty much considered solved, with submitted ML models routinely scoring in excess of 95% accuracy. For reference, the ImageNet task was to distinguish between about 1000 types of everyday objects. My classifier, on the other hand, needed to distinguish between 8 (later 11) classes of medical images.

Using Transfer Learning to leverage the knowledge in the ImageNet models seemed like a natural fit. In my case, Transfer Learning would involve taking an existing high-performing ImageNet model, and fine-tuning it with my labeled data to fit the medical image classification task.

The first iteration of my model ended up being a hybrid of a Caffe backend to generate vector image representations using trained ImageNet models, and a Keras Dense network to classify these vectors into one of 8 classes. Caffe is a C++ framework specialized for image tasks, and the main reason I chose it was because it provided downloadable trained ImageNet models. Later, when the Keras project provided its own downloadable ImageNet models, I built a second iteration of the model, which fine-tuned the ImageNet model and learned to classify medical images end-to-end.

I bring up this story to illustrate how easy and natural it felt to start working with Keras. For a long time, that was the only library I needed for building my models. Over this period, I have been consistently delighted by its intuitive API, its sensible default parameters, and the overall quality of tutorial material on its website.

However, if you are like me, you often have to understand and train other peoples’ models in addition to your own. Since these models are often built by DL researchers using Pytorch and Tensorflow, I ended up learning these frameworks as well. I can attest from experience that Keras is by far the easiest to learn and use.

Today, the Deep Learning ecosystem is much more mature, so thankfully one can get by with learning fewer frameworks. While many excellent frameworks have been released over these intervening years, and are being used in specialized niches, the major ones are Keras, Tensorflow, and Pytorch. Pytorch became popular because of its eager execution model, which Tensorflow did not allow, and which Keras hid behind its cleverly-designed API. Keras has since been subsumed into Tensorflow as tf.keras, but the original Keras lives on as well, with an additional CNTK (from Microsoft) backend. For its part, Tensorflow, in its 2.x incarnation, has embraced Pytorch’s eager execution model, and made tf.keras its default API.

So there has been lots of convergence, and while I recommend learning all three of the frameworks listed above, if you need to get productive quickly and don’t have a framework imposed upon you (by the project or by corporate policy), I would recommend you start with Keras. If you are proficient in one of the others, you should still learn Keras, because for a majority of tasks you are likely to be more productive with Keras than with your current framework.

Of course, the simplicity and elegance of Keras comes at a price. Most things are easy and intuitive to do in Keras, but certain things are very hard or even impossible. Some of these tasks are possible in the other lower-level frameworks, but you pay for that convenience with more verbose code and a steeper learning curve. However, with tf.keras (and to some extent with the original Keras using Tensorflow backend), you have access to the full Tensorflow substrate. In addition, the Keras team has recently been busy refining their code, which now makes it possible to do certain things in Keras that were previously thought to be impossible.

I am honored to present Keras: from soup to nuts – an example-driven tutorial at OSDC West this year, where I hope to touch upon some of these things that make Keras such a simple yet powerful addition to your DL toolbox. I hope to see you there!

About the author/ODSC West speaker: Sujit Pal is an applied data scientist at Elsevier Labs, an advanced technology group within the Reed-Elsevier Group of companies. His areas of interests include Semantic Search, Natural Language Processing, Machine Learning, and Deep Learning. At Elsevier, he has worked on several machine learning initiatives involving large image and text corpora, and other initiatives around recommendation systems and knowledge graph development. He has co-authored Deep Learning with Keras (https://www.packtpub.com/big-data-and-business-intelligence/deep-learning-keras) and Deep Learning with Tensorflow 2.x and Keras (https://www.packtpub.com/data/deep-learning-with-tensorflow-2-0-and-keras-second-edition), and writes about technology on his blog Salmon Run (https://sujitpal.blogspot.com/).

12 3