New feature Data Lake Governance Center (DGC) 

Flexible Engine
Release Notes

Octobre, 2023 

What’s new?

We are pleased to announce the arrival of Data Lake Governance Center (DGC) to Flexible Engine, now available for our users in the Paris region. With the exponential growth of data, DGC is specifically designed to address prevalent data management challenges faced by businesses today, by enhancing efficiency, reducing costs and promoting seamless data sharing.

Data Lake Governance Center (DGC)  

Key features

Data Lake Governance Center (DGC) offers a comprehensive operations platform for serverless data lifecycle management, empowering users to seamlessly integrate, develop, and visualize their data. DGC serves two primary functions:

  1. Data integration: DGC facilitates data migration across 20+ data sources and their integration into data lake services such as MapReduce clusters (MRS) or Data Warehouse (DWS). Users benefit from wizard-based configuration, management, and support for various data integration types, including single table, entire database, incremental, and periodic data integration. For a list of supported data sources, please refer to the provided link.
  2. Data development: DGC delivers a robust framework for automated data ingestion and processing from diverse sources. It empowers users to construct a formidable big data processing architecture, leveraging ETL, Python, or SQL scripts, and scheduled workflows.

Advantages

  • A comprehensive data operations platform: (DGC) stands as an all-encompassing data operations platform, empowering users to seamlessly execute a multitude of tasks. It facilitates data integration from diverse domains, fosters the development of data services, and enables the interconnection of data from various sources.
  • Versatile data development capabilities: DGC offers a diverse array of scheduling configuration policies and robust job scheduling. It promotes collaborative development among multiple users, facilitating online editing and real-time querying of SQL and shell scripts. Job development is streamlined through data processing nodes, including CDM, SQL, MRS, Shell, and Spark.
  • Effortless serverless operation, workflow scheduling, and maintenance: this serverless data management service excels in unified workflow scheduling, incorporating both one-time and event-triggered mechanisms for data integration workflows. It exhibits adaptive resource allocation for computing and storage, dynamically catering to the demands of tasks and data generated by workflows.
  • Streamlined operations and maintenance: DGC’s Operations and Maintenance (O&M) center diligently oversees all tasks while offering customizable notification settings. This feature provides users with real-time task status updates, ensuring the uninterrupted operation of services.

The pricing model for DGC

The DGC service employs a transparent pricing model based on two primary usages:

  1. Data integration: DGC offers flexibility in data integration management by providing a choice of three CDM instance sizes (Cloud Data Migration). Larger CDM instances deliver enhanced performance. Usage is billed hourly until the instance is deleted, with reduced pricing available for stopped CDM instances (stopping initiated solely through the API).
  2. Data development: to streamline workflows, data development, and Operations & Maintenance (O&M), users can initiate a data development instance (Starter). Usage is billed on an hourly basis until the instance is deleted.

For detailed pricing information, please refer to the pricelist.

For more information on DGC please visit the HelpCenter page.