Implementing a Scalable Data Engineering Platform for a Trust and Corporate Management Company

The client needed a robust data engineering platform to collate and transform data from multiple ERPs, ensuring scalability, cost-effectiveness, and the ability to incorporate micro-strategies over the data architecture.

Challenge

The primary challenge was to create a scalable and cost-effective data engineering platform capable of integrating multiple ERPs while minimizing service delays and maintaining high performance.

Scalability and Cost Efficiency

Designing a thin and scalable data engineering layer suitable for the cloud with pay-per-use to minimize costs.

Integration and Transformation

Handling data integration and transformation from multiple ERPs to ensure seamless data flow and accurate analytics.

Solution

Entrans developed a comprehensive data engineering platform with Azure architecture, integrating various technologies to achieve the client's objectives.

Result: The platform provided efficient data integration, transformation, and storage, ensuring scalability and cost-effectiveness while supporting future growth.

Detailed Solution:

Azure Architecture: The solution utilized Azure architecture to provide a scalable and cost-effective platform. The use of Azure DataBricks with Spark allowed for efficient data processing, despite slower start times and non-serverless nature. The architecture separated computation from storage, with Blob storage used for data storage.
Curated Data Lake: The preliminary stage involved creating a curated data lake before loading data into Azure Datawarehouse. Transformations were handled in Spark, and the transformed data was then pushed into Blob storage, which synced with Azure Datawarehouse.
Scalable Instances: Startup times were required while running as spot instances, and instances were scaled on demand to ensure optimal performance.

Impact: The implementation led to a scalable, efficient, and cost-effective data engineering platform that streamlined data integration and transformation, enhancing the client's ability to make data-driven decisions.