Case Study: Ukraine’s Energy Company

How Ukraine's Energy Company Expanded Big Data Analysis and Energy Trend Forecasting Capabilities

About the company

The Energy Company of Ukraine (ECU) is a national leader in energy trading and supply, providing electricity to dozens of strategic enterprises across Ukraine and importing it from the EU in times of energy shortage. The company continuously adopts new technologies to enhance the efficiency of its operations and management decisions. Amid dynamic changes and high competition, the need for tools for detailed analytics and accurate energy trend forecasting has become critically important. To address this, ECU turned to Cloudfresh to implement a solution based on Google Cloud Platform (GCP).

Project start date: April 6, 2024
Project end date: May 31, 2024

Challenge

The primary challenge was to build a centralized data warehouse that would automate the collection and processing of information from numerous sources, ranging from public websites to internal PostgreSQL databases. Additionally, a key requirement was the development of a reliable orchestration for extract, transform, and load (ETL) processes to minimize operational costs and ensure the system’s high reliability.

ECU sought a solution that would not only efficiently gather and store large amounts of data but also enable real-time complex analytics using machine learning and intelligent decision-making tools.

Solution

In collaboration with Cloudfresh, Google Cloud Premier Partner, a comprehensive solution was developed based on the Google Cloud Platform, ensuring automation and observability at every stage of the data lifecycle — from extraction to analysis.

1. Orchestration and data extraction: To collect data, 20 connectors (API connectors and web scrapers) were developed using Cloud Functions and Cloud Run. These connectors were created for various data sources — from public websites to internal databases. Orchestration was handled by Cloud Composer, automating and managing all tasks with minimal manual intervention. Leveraging the capabilities of Apache Airflow, Cloud Composer provided reliable control over each step of the process, ensuring observability of the system’s state and timely detection of potential issues.
2. Data Fusion for data transformation: Once extracted, data is stored in Cloud Storage for initial retention. Further transformation is performed using Cloud Data Fusion, an integration platform that allows building and managing ETL processes without deep programming knowledge. With a wide range of connectors and custom transformation capabilities, Data Fusion fully automated the transformation process, standardizing the data and preparing it for subsequent analysis in BigQuery.
3. BigQuery as an analytical platform: A key advantage of the implemented solution was the use of BigQuery as the primary tool for storing and analyzing data. BigQuery is a powerful tool for processing large datasets with the ability to scale at the cloud platform level. It supports real-time analytical queries, enabling ECU to quickly obtain critical insights for decision-making.
   The integration with BigQuery also opens opportunities for leveraging machine learning tools (ML). In the future, ECU plans to implement BigQuery ML, allowing the company to build models for predicting energy flows and market trends directly within BigQuery, without the need to move data to separate ML platforms.
4. Observability and management: The entire system offers a high level of observability and manageability. Using Cloud Monitoring and Cloud Logging, every stage of data processing is monitored, with real-time alerts for failures or issues, ensuring the stability of ETL processes.

Results

After implementing the GCP-based solution, ECU achieved:

• Automation of data collection from 20 different sources using Cloud Functions and Cloud Run, reducing manual work and improving process reliability.
• Reduced operational costs through automation of ETL processes and data storage optimization.
• Preparation for advanced analytical tools based on BigQuery and ML, enhancing the accuracy of energy trading forecasts.
• Complete observability and manageability of the system through integration with Cloud Monitoring and Cloud Logging, enabling timely issue detection and resolution.

Cloudfresh’s role

The Cloudfresh team oversaw the entire process of implementation: from designing the architecture to setting up connectors and ensuring reliable process orchestration. By applying Google Cloud’s best practices, Cloudfresh helped ECU achieve significant improvements in data collection, processing, and analysis.

Future plans

ECU plans to continue working with Cloudfresh to implement BigQuery ML analytics models and use Looker for data visualization. This will further optimize operational processes and enhance business decision-making efficiency.