Semarchy xDM 5.3 - Simplify the Data Hub Streaming & Batch Integration
by FX Nicolas | May 26, 2021
As the focal point of governance, a Data Hub is a critical piece in all data management and data transformation initiatives. It is, most of all, the enabler for data flows between diverse endpoints, including operational applications (ERP, CRM, etc), data warehouses, data lakes, and even external parties. Being able to implement repeatable and scalable integration from and to the data hub, quickly and at a reasonable cost is critical to the success of the data hub strategy.
The Intelligent Data Hub platform supports out-of-the-box integration in batch and streaming mode. Semarchy xDM version 5.3 enhances these integration capabilities with new features to simplify end-to-end integration of the data hub, for both analytical and operational use cases.
Streaming Integration with Data Notifications
Near real-time propagation of data changes is key for operational efficiency. When information consolidates from multiple sources using match and merge capabilities (consolidation pattern), or when new data is authored into the hub (centralized pattern), it must be propagated to downstream systems that require high quality and well-governed information to pursue business processes.
Data Notifications, introduced in xDM 5.3, enable designers to easily automate data and change propagation from the data hub to consuming applications.
Data Notifications are delivered - on schedule or data change - to event streaming platforms (such as Apache Kafka) or to applications via REST API calls and message services. Their payload is fully customizable, using Semarchy xDM’s Named Queries, to accommodate the requirements of the consumers. Notifications may leverage the comprehensive audit trail available in the hub to propagate golden data, master data, validation errors, history, etc.
With Data Notifications, designers and integration teams can simplify data integration and change propagation patterns from the data hub to downstream systems.
Model-Driven SQL APIs
The new Database Views feature allows designers to create logical views in the model and deploy them in the data hub schema as physical SQL views, where they can be consumed by external applications in batch, by business intelligence, reporting, and visualization tools, including the platform xDM Dashboards module.
Database Views use the full power of SemQL transformations, functions, and expressions to define columns’ content and data filters based on Semarchy’s semantically complete model.
Using database views, developers can quickly design and deploy SQL APIs with their data hubs, fully aligned with the model structure and lifecycle.
Other Integration Enhancements
Data Notifications and Database Views are the integration highlights of version 5.3. Many other features have been added to make integration faster and simpler, including Delayed Continuous Loads to handle large batch integrations, Mass Update and Integration Summary in the REST API, Named Query performance enhancements, etc.