Financial Data Engineering: Challenges and Practices


Finance is a highly technology-intensive and data-driven sector. Ongoing investments in digital transformation have reshaped financial markets, creating new opportunities and challenges. Consequently, practitioners in banks, FinTech startups, asset management companies, and hedge funds are increasingly facing new and complex requirements related to designing and implementing a reliable and scalable financial data infrastructure while following secure and compliant standards. Crucially, many practitioners lack a holistic view of the concepts, standards, models, and technologies necessary to build such data infrastructures. This is where the practice of financial data engineering is required.

This tutorial provides a practical introduction to financial data engineering. By the end, you will have a comprehensive understanding of the challenges involved in creating data-driven financial products. You will learn to navigate the complex financial data landscape and gain expertise in designing and implementing various layers of financial data infrastructure, such as ingestion, storage, transformation, serving, and monitoring.

Session Outline:

Part 1: The Financial Data Landscape
This part provides a comprehensive overview of the financial data landscape, including data types, sources, structures, and attributes. By the end, you will have a solid understanding of the complexities of financial data and know what to expect when working with it.

Part 2: Challenges in Financial Data Management
This part will familiarize you with the business and financial domain-oriented requirements and challenges in designing data-driven products and systems for financial markets. By the end, you will be familiar with problems such as reference data management, financial identification systems, and financial entity systems.

Part 3: Financial Data Infrastructure
In this part, we will explore the different components of a financial data infrastructure, including ingestion, storage, transformation, delivery, and monitoring. By the end, you will gain a comprehensive understanding of the technical aspects inherent in designing financial data infrastructures and how these technical considerations align with the financial sector's domain-specific requirements.

Part 4: Building a Basic Reference Data Table for a Handful of Financial Instruments
In this part, I'll guide you through a real-world demonstration using a Jupyter Notebook. We'll collect and aggregate different financial identifiers for a subset of common stocks. This hands-on exercise will provide you with practical insights into the complexities of financial identification and the approaches available to tackle them.

Attendees will develop an understanding of the domain-oriented challenges and requirements involved in the design and implementation of reliable and secure financial data infrastructure. I will run a Python code integrated within a Jupyter Lab notebook and will make calls to open-access financial APIs.

Basic knowledge of finance and Python is a plus, but not a strict requirement.


Tamer Khraisha is a senior software and data engineer with over a decade of experience in both industry and academia. He holds a background in financial economics and a Ph.D. in Network Science. He has collaborated with various FinTech startups, where he has designed and developed software solutions for payments, financial analysis, and asset management. As a researcher and scientific author, Tamer's work primarily focuses on the intersection between financial markets, data, and technology.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google