Jennifer is a speaker for ODSC East 2020 this April 13-17 in Boston. Be sure to check out her talk, “Impacting Clinical Development with Advanced Analytics: Challenges & Opportunities,” there!

Why did I become a statistician in Big Pharma? When I completed my training, I had never considered it. Big Pharma statisticians were generally considered pencil pushers, engaged in the most boring work possible. But when a position opened up as a research statistician at Genentech (which is a member of the Roche group), I wanted it – because Genentech has always had the reputation of being at the forefront of biological science and research, in particular, seemed more promising in terms of interesting work. With Roche’s acquisitions of Flatiron Health and Foundation Medicine and partnerships like PicnicHealth, science is now more broadly defined to include data science as well. With this large-scale transformation occurring within the company, opportunities to pursue methodology development on novel data types now exist in all functions of the company – embedded statisticians are in turn uniquely positioned to directly impact our pipeline.

Pharma generally recognizes the need to embrace change in how we analyze and handle data internally, a process in the past clinical teams did not prioritize as the end result of a trial is meant to be a filing, not a curated data mart. Now that we are focused on curated and integrated datasets moving forward, there is naturally a commensurate pressure to gain additional insights from this data. Expectations are higher – we’ve already run the logistic regression or mixed effect model repeat measurement (MMRM) on the primary, secondary and exploratory endpoints. In order to justify the heavy-lift needed to wrangle curated data from our incurred technical debt, the algorithm used to analyze this data must be proportionally complex. Terms start flying around like advanced analytics, machine learning, deep learning, and artificial intelligence.

What is the bar to adopt these analytical methodologies?  The core of Big Pharma is to develop drugs and transition new molecules through the pipeline of progressing evidence in support of helping patients. We expect that our clinical development plans are informed by data and evidence. Just like any modeling exercise – there are training and test data. In this setting, training data is a large non-interventional cohort, which does not typically reflect the highly ascertained patient population of a trial. As an example, this atezolizumab trial in Non-Small Cell Lung Cancer has 35 inclusion/exclusion criteria, many of which are based on variables that will not typically be collected in an observational cohort or electronic health medical record database. This is simply the first obstacle – but assuming that we will hit the bar, what are the rest of the obstacles unique to Pharma that we will need to address? What do our first forays at the forefront of integrating data science into our pipelines look like? Come find out at my talk where I would love to hear your feedback!

More on the author/speaker:

Jennifer Tom joined Genentech in 2014 as a statistician in bioinformatics and computational biology and moved into product development in 2018. She has supported human genetics, microbiome, imaging, early clinical development, and biomarker activities across various non-oncology indications. Previously she worked as a software engineer at Agilent and the visiting assistant Neyman Professor of statistics at Berkeley. Jennifer received a BA in Molecular and Cellular Biology from UC Berkeley and an MS and PhD in Biostatistics from UCLA.

Shauna is a speaker for ODSC East 2020 this April 13-17 in Boston. Be sure to check out her talk, “Graph Neural Networks and their Applications,” there!

Graph neural networks have revolutionized the performance of neural networks on graph data. Companies such as Pinterest[1], Google[2], and Uber[3] have implemented graph neural network algorithms to dramatically improve the performance of large-scale data-driven tasks.

Introduction to Graphs

A graph is any dataset that contains nodes and edges. Nodes are entities (e.g. people, organizations), and edges represent the connections between nodes. For example, a social network is a graph in which people in the network are considered nodes. Edges exist when two people are connected in some way (e.g. friends, sharing one’s posts). 

In retail applications, customers and products can be viewed as nodes. An edge shows the relationship between the customer and a purchased product. A graph can be used to represent spending habits for each customer. Additionally, nodes can also have features or attributes. People have attributes such as age and height, and products have attributes such as price and size. Pinterest has used graph neural networks in this fashion to improve the performance of its recommendation system by 150%[1].

The advent of Graph Neural Networks

Until the development of graph neural networks, deep learning methods could not be applied to edges to extract knowledge and make predictions, and instead could only operate based on the node features. Applying deep learning to graph data allows us to approach tasks link prediction, community detection, and generating recommendations. 

Deep learning can also be applied to node classification, or predicting the label of an unlabelled node. This takes place in a semi-supervised setting, where the labels of some nodes are known, but others are unknown. A survey of deep learning node classification methods shows a history of advances in state-of-the-art performance while illustrating the range of use cases and applications. 

Methods discussed in this blog post were evaluated on the benchmark CoRA dataset. CoRA consists of journal publications in deep learning. Each publication is a node, and edges exist between nodes if they cite or are cited by another paper in the dataset. The dataset is comprised of 2708 publications with 5429 edges. 

DeepWalk

DeepWalk[4], released in 2014, was the first significant deep learning-based method to approach node classification. DeepWalk’s approach was similar to that taken in natural language processing (NLP) to derive embeddings. An embedding is a vector-representation of an object such a word in NLP or a node in a graph. To create its embeddings, DeepWalk takes truncated random walks from graph data to learn latent representations of nodes. On the CoRA dataset, DeepWalk achieved 67.2% accuracy on the benchmark node classification experiment[5]. At the time, this was 10% better than competing methods with 60% less training data.

To demonstrate how the embeddings generated by DeepWalk contain information about the graph structure, below is a figure that shows how DeepWalk works on Zachary’s Karate Network. The Karate Network is a small network of connections of members of a karate club, where edges are made between two members if they interact outside of karate. The node coloring is based on the different sub-communities within the club. The left image is the input graph of the social network, and the right is the two-dimensional output generated by the DeepWalk algorithm operating on the graph data. 

Source: [4].

Graph Convolutional Networks

In 2016, Thomas N. Kipf and Max Welling introduced graph convolutional networks (GCNs)[6], which improved the state-of-the-art CoRA benchmark to 81.5%. A GCN is a network that consists of stacked linear layers with an activation function. Kipf and Welling introduced a new propagation function that operates layer-wise and works directly on graph data. 

Source: [6] The t-SNE visualization of the two-layer GCN trained on the CoRA dataset using 5% of labels. The colors represent document class. 

The number of linear layers in a GCN determines the size of the target node neighborhood to consider when making the classification prediction. For example, one hidden layer would imply that the graph network only examines immediate neighbors when making a classification decision. 

The input to a graph convolutional network is an adjacency matrix, which is the representation of the graph itself. It also takes the feature vectors of each node as input. This can be as simple as a one-hot encoding of each node’s attributes, while more complex versions can be used to represent the complex features of a node. 

Applying GCN to Real-World Data

Our research team was interested in implementing a GCN on real-world data in order to speed up analyst tasking. We implemented a GCN architecture written in PyTorch to perform node classification on article data. The graph used in our dataset was derived from article data grouped together in “playlists” by a user who determined that they were relevant. The nodes were individual articles and playlists, and edges existed between an article and a playlist if a specific article was included in the playlist. Instead of manually poring through the corpus to determine additional relevant articles, we used the GCN to recommend other potentially relevant documents. After running our corpus of about 100,000 articles and 7 different playlists through a 2-layer GCN, our network performed 5x better than random.

Graph-BERT

GCN’s were the leading architecture for years, and many variations of them were subsequently released. Then, in January 2020 Graph-BERT[7] removed the dependency on links and reformatted the way that graph networks are usually represented. This is important for scalability, while also showing improved accuracy and efficiency relative to other types of graph neural networks. We are currently exploring how Graph-BERT will impact the use cases we have been addressing with graph neural networks.

Conclusion 

Graph neural networks are an evolving field in the study of neural networks. Their ability to use graph data has made difficult problems such as node classification more tractable. 

For a deeper discussion on graph neural networks and the problems that they can help solve, attend my talk at ODSC East, “Graph Neural Networks and their Applications.”

CITATIONS

[1] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in Proc. of KDD. ACM, 2018, pp. 974–983.

[2]  Sanchez-Lengeling , Benjamin, et al. “Machine Learning for Scent: Learning Generalizable Perceptual Representations of Small Molecules.” ArXiv, no. 1910.10685, 2019. Stat.ML, arxiv.org/abs/1910.10685.

[3] Uber Engineering. “Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations”, eng.uber.com/uber-eats-graph-learning.

[4] Perozzi, Bryan, Al-Rfou, Rami and Skiena, Steven. “DeepWalk: Online Learning of Social Representations.” CoRR abs/1403.6652 (2014).

[5] “Node Classification on CoRA.” paperswithcode.com/sota/node-classification-on-coraom.

[6] Kipf, Thomas N. and Welling, Max. “Semi-Supervised Classification with Graph Convolutional Networks.” Paper presented at the meeting of the Proceedings of the 5th International Conference on Learning Representations, 2017.

[7] Zhang, Jiawei, Zhang, Haopeng, Xia, Congying, and Sun, Li. “Graph-Bert: Only Attention is Needed for Learning Graph Representations.” arxiv.org/abs/2001.05140 (2020).

About the speaker/author: Shauna Revay is an applied machine learning researcher for Novetta’s Machine Learning Center of Excellence. She specializes in rapid prototyping of ML solutions spanning fields such as NLP, audio analysis and ASR, and the implementation of graph neural networks. Shauna earned her PhD in mathematics at George Mason University where she studied harmonic and Fourier analysis. She pursues research in interdisciplinary topics and has published papers in mathematics, biology, and electrical engineering journals and conference proceedings.

Neural networks are weirdly good at translating languages and identifying dogs by breed, but they can be intimidating to get started with. In an effort to smooth this on-ramp, I created a neural network framework specifically for teaching and experimentation. It’s called Cottonwood and this notebook shows how to build a first neural network.

What you’ll get out of this case study to build a first neural network

You will get to build an honest-to-goodness neural network, a densely connected 8-layer autoencoder and run it on an image dataset. This will set you up well for subsequent projects using your framework of choice, whether it’s Cottonwood, PyTorch, Tensorflow.

What you’ll need to get started

To run this case study, you’ll need Python 3, Git, and access to your command line.

No prior machine learning experience necessary.

Get cottonwood

To get Cottonwood, navigate your command line to directory you’d like it to live in. Then clone the Git repository, and install the Cottonwood package. The “-e” option lets you edit your copy of Cottonwood on the fly, opening up all kinds of opportunities for experimentation.

git clone https://gitlab.com/brohrer/cottonwood.git
python3 -m pip install -e cottonwood

It’s also important to specify the version of Cottonwood we want to use. Because Cottonwood young and designed to encourage experimentation, it’s not guaranteed to be backward compatible. The version matters. This case study runs well on version 14.

cd cottonwood
git checkout v14

When you’re done, navigate to where you want this case study to live and open up a notebook or script there.

Import some packages

The first thing to do is to pull in the Cottonwood building blocks we’re going to need. These include several types of layers, a Dense layer, a Difference layer, and a RangeNormalization layer. It also includes an ANN (artificial neural network) model and a Nordic runes data set that comes prepackaged with Cottonwood for coding up examples.

from cottonwood.core.layers.dense import Dense
from cottonwood.core.layers.difference import Difference
from cottonwood.core.layers.range_normalization import RangeNormalization
from cottonwood.core.model import ANN
import cottonwood.data.data_loader_nordic_runes as dat

Get some data

Then we need to pull the data in. The get_data_sets() function in our data loader pulls in two generators, a training_set and an evaluation_set. A generator is a convenient Python object that hands you a new example every time you call next() on it. We put this to the test by calling next(training_set) and getting a first example of our data.

training_set, evaluation_set = dat.get_data_sets()

sample = next(training_set)

n_pixels = sample.shape[0] * sample.shape[1]

We can take a quick look at it and see that it’s a two dimensional array of zeros and ones, a very crude representation of an image. It has seven rows and seven columns, 49 pixels it all. If you squint and tilt your head you can just make out the pattern. This data set is a collection of 24 Norse runes, coarsely pixelized.

print(sample)
 

[[0 1 0 0 0 1 0]

 [0 1 0 0 0 1 0]

 [0 1 1 0 0 1 0]

 [0 1 0 1 0 1 0]

 [0 1 0 0 1 1 0]

 [0 1 0 0 0 1 0]

 [0 1 0 0 0 1 0]]

Create some layers

Now we get to construct the neural network itself. The first step is to add a RangeNormalization layer. This ensures that the data falls in a range that can be meaningfully interpreted by the network.

Then we have a sequence of six Dense layers. The first argument is the number of outputs that the layer will have. The second argument explicitly connects each Dense layer to the previous layer. The Dense layer uses the previous layer to know how many inputs it should expect.

Here we have five hidden layers and an output layer. The hidden layer with the smallest number of nodes is the bottleneck layer. At this layer the information contained in the images is at its highest compression. Instead of 49 separate pixel values, it’s reduced down to eight node activities.

Finally, we add a Difference layer to find how close the output of the autoencoder is to the input. This gives us an error signal that we can try to make as small as possible, re-creating the outputs with as much fidelity as we can.

layers = []

layers.append(RangeNormalization(training_set))

layers.append(Dense(17, previous_layer=layers[-1]))

layers.append(Dense(13, previous_layer=layers[-1]))

layers.append(Dense(8, previous_layer=layers[-1]))

layers.append(Dense(12, previous_layer=layers[-1]))

layers.append(Dense(19, previous_layer=layers[-1]))

layers.append(Dense(n_pixels, previous_layer=layers[-1]))

layers.append(Difference(layers[-1], layers[0]))

Build and run the model

Finally it’s time to fire up the autoencoder! We pass in the layers to the ANN model to initialize it. Then, thanks to all of our preparatory work, training the model just requires calling its train() method and passing at the training_data_set. Similarly evaluating the model is as straightforward as calling its evaluate() method and passing the evaluation_data_set.

When the model is done running, it returns its error history. If the model behavior is behaving well, the error gets smaller over time.

autoencoder = ANN(layers=layers)

autoencoder.train(training_set)

autoencoder.evaluate(evaluation_set)

When the model runs, it also saves out some informative reports. In the reports directory, we can inspect model_parameters.txt. This human readable text file gives all the information necessary to re-create this exact model in a different framework.

type: artificial neural network

number of training iterations: 800000

number of evaluation iterations: 200000

error_function:  mean squared error

layer 0:  range normalization

  range maximum: 1

  range minimum: 0

layer 1:  fully connected

  number of inputs: 49

  number of outputs: 17

  activation function:  hyperbolic tangent

  initialization:  LSUV

  optimizer:  momentum

    learning_rate: 0.001

    momentum amount: 0.9

    minibatch size: 1

layer 2:  fully connected

  number of inputs: 17

…

We can also see the performance history in a plot. This is helpful for getting a quick feel for how our model is performing.

To get a more detailed look at the model, including the nodes in each layer and the weights between them, we can browse through the snapshots generated while the model runs. This visualization gives a good pictorial representation of the model and helps to spark research questions and ideas for things to try in the next run.

Next steps

[Related article: Building a Custom Convolutional Neural Network in Keras]

Now you’re in a good place. You have a fully functional autoencoder performing compression on an image dataset. You have a few options if you’d like to take it to the next level. You can experiment with different architectures. You can pull in a new set of images of your own. If you have been bitten by the bug, and really want to know how it all works, I have prepared a series of courses for you at the End-to-End Machine Learning School: 193, 312, 313, and 314.

Or you can sign up for my Introduction to Neural Networks workshop sequence, offered at ODSC East 2020 in Boston covering theory in part 1 and practice in part 2.

About the Author/Speaker: Brandon Rohrer, iRobot

Brandon got started in machine learning studying robotics at MIT, machine vision and signal processing at Sandia National Laboratories, then predictive modeling at DuPont Pioneer, and cloud computing at Microsoft. At Facebook, he built global power distribution models using satellite imagery and unstructured text classifiers. Now at iRobot, he helps robots get better at doing their jobs.