Abstract: 

Facial recognition systems are everywhere. Of course, it's where you would expect it, such as airports, border crossings, and government offices. However, it's also in some public surveillance cameras, all over social media, embedded in smart home solutions, and even in your phone. Have you ever wondered how facial recognition systems work?

In this hands-on session, we will build a facial recognition system from scratch using open-source technologies and publically available pre-trained models.

We will create a Javascript web app that uses Tensorflow lite to get facial landmarks (nose, mouth, eyes, chin, etc.) with pre-trained models for the client side. For the server side, we will create a Python API that uses another pre-trained model to generate unique facial descriptors with facial landmarks and compare it against a vector database of faces. The javascript web app connects with the Python API to determine whose face it is. However, to appease any privacy concerns, we will also demonstrate other ways of building a system without relying on server-side inference. And during the workshop, participants can optionally enter their faces into the face database and have the biometrics removed at the end of the workshop.

The goal is that developers should be confidently able to apply what they learned to their projects and learn more in general about facial recognition, vector databases, and machine learning with Javascript.

Session Outline:

Lesson 1: The Javascript Web App
We will learn how to turn on the camera in a web browser, locate the face, and capture facial landmarks (nose, mouth, eyes, chin, etc.) for every frame.
Lesson 2: The Python API & Vector Database
Here we will create an API that can receive any facial landmarks and leverage another deep learning model to create a facial descriptor that it can store in a vector database or use to compare and find the closest match in the database.
Lesson 3: Tying it all together
From the web app, when a face is found, send a ""Find Closest Match"" request to the server. Then display the name of the person that is the closest match on the screen.
Bonus lesson: Local inference and examples of use cases
With Python code, we will examine how to do all the machine learning inference on the local device. And other use cases for facial detection and recognition systems.

Learning objectives:
- How to make Facial Detection and Recognition systems, and the theory and use cases behind them
- How to leverage Mediapipe and Dlib open-source models for facial recognition tasks both in Python and Javascript
- How to populate and search Vector Databases, and the theory and potential use cases behind them
- How to use TensorFlow Lite and make a client-side application in Javascript

Background Knowledge:

Some programming experience in any language is a prerequisite and some Python or Javascript experience would be helpful. Instructions to install Python, clone a repository and create a Python environment will be provided, but it will be easier if they know how to do this and come prepared for it.

Bio: 

Serg Masís has been at the confluence of the internet, application development, and analytics for the last two decades. He's an Agronomic Data Scientist at Syngenta, a leading agribusiness company with a mission to improve global food security. Before that role, he co-founded a search engine startup, incubated by Harvard Innovation Labs, that combined the power of cloud computing and machine learning with principles in decision-making science to expose users to new places and events efficiently. Whether concerning leisure activities, plant diseases, or customer lifetime value, Serg is passionate about providing the often-missing link between data and decision-making. He wrote the bestselling book "Interpretable Machine Learning with Python" and is currently working on a new book titled "DIY AI" with do-it-yourself projects for AI hobbyists and practitioners alike.