A practical guide for Data Mesh implementation

Why Data Mesh?

Key challenges in the current traditional Data and Analytics approach:

  • Most of the Data Warehouse implantations are centralised and monolithic
  • Lack of compossibility in the traditional ELT/ETL (E-Extract, T-Transform, L-Load) approach
  • Hype-specialised silos within the business and information technology team.
  • Fail to scale consumers of the data
  • Fail to materialize data-driven values

A brief history, how information landscape had evolved over 40 yrs

Evolution of data and application landscape

Application and Data Paradigm

To describe the evolution of the information and data processing landscape we need to go back a couple of decades when the Mainframe was introduced as a large computer system by IBM in 1952. During the 1960s and 1970s, IBM mainframe dominated the large computer market.

Programming Paradigm

Form 1990 the object-oriented-programming (OOP) is also becoming popular. C++ and Java-based applications were started to develop based OOP programming techniques

Big-Data, IoT and Cloud Computing

Since 2006 the CSP (Cloud Service Provider) started to offer different cloud computing services which are helping organisations to modernise the application landscape.

Data Mesh

Over the last two decades, we have seen organisations are dwelling with Data Warehouse or Data Lake or Hybrid (mix of both) approach. We have also seen the application landscape has changed from monolithic to Service-oriented to Microservice architecture. The organisations have changed from project to product mindset.

Data Mesh Building Blocks

Data Mesh Principles

4 Principles of Data Mesh Architecture

  • Domain ownership: Responsibility for modelling and providing important data is distributed to the people closest to it, providing access to the exact data they need, when they need it.
  • Data as a product: Data is treated as a product like any other, complete with a data product owner, consumer consultations, release cycles, and quality and service-level agreements.
  • Self-service: Empower consumers to independently search, discover, and consume data products. Data product owners are provided standardized tools for populating and publishing their data products.
  • Federated governance: This is embodied by a cross-organization team that provides global standards for the formats, modes, and requirements of publishing and using data products. This team must maintain the delicate balance between centralized standards for compatibility and decentralized autonomy for true domain ownership.
Data Mesh Building Blocks — Dependencies

Data Mesh Implementation Approach

Data Mesh implementation approach has lots of similarities with the Microservice architecture principles (e.g., domain-driven design, product-first mindset, composable architecture, decentralised governance). so, it’s worth revisiting the Microservice architecture and principles while defining the Data Mesh implementation approach.

Typical Microservice Architecture
  1. Developing separate analytics microservice by using the same data source used/developed by the application microservices
  2. Repurposing the message hub and creating a new subscription for the analytics data product
  3. Developing a completely decoupled message-hub
  4. Developing a separate data virtualisation layer based on the same data source used by the application microservices

Data Mesh Implementation Approach — Pattern1

In this approach, Analytics Data Products are developed based on separate Analytics service APIs. The Analytics service APIs are developed based on the read-replica data sources.

Input Ports: Pattern — 1

Data Mesh Implementation Approach — Pattern2

Analytics data products are developed based on the event-based model. In this approach, the transaction events are sourced from the same message broker in a different subscription for analytics product development.

Input Ports: Pattern — 2

Data Mesh Implementation Approach — Pattern3

Analytics Data Products are developed based on the event-based architecture, completely decoupled from the microservice application.

Input Ports: Pattern — 3

Data Mesh Implementation Approach — Pattern4

Analytics Data Products are developed on top of the data virtualisation layer. The data virtualisation layer is developed based on transactional data sources.

Input Ports: Pattern — 4

Data Mesh Implementation Approach — Analytics Services

In the previous section, we have seen that the input data ports can be developed based on pub/sub, API, or data virtualisation techniques. The Data Products are developed within a business domain based on the input data sources. The output data ports are served via reporting or data/API services. In the middle, the data product is developed based on descriptive (rule-based), predictive, or recommendation models to serve the analytics requirement of the individual domain.

Analytics Data Product

Reference:

The Data Mesh principles have been taken from https://martinfowler.com/articles/data-mesh-principles.html written by Zhamak Dehghani

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Shantanu De

Shantanu De

Shantanu De is a Principal Architect at Royal Mail Group