Modern data platform POC transforms Oil CoOP | New Era Technology

This leading Midwest oil refinery cooperative must manage many business processes to successfully deliver fuel and oil, oversee pipelines, and manage retailers. And to do so, they were relying on point solutions from multiple vendors for many of their operational needs. They had data coming in from these disparate solutions but no enterprise data management platform to integrate the information and support accurate business intelligence reporting.

But they were not just looking to integrate their data; they wanted to elevate their data landscape through advanced analytics and machine learning to improve their business prospects like price optimization and targets for profitable oil exploration.

Their primary goal was to evaluate which platforms would best support their business needs and provide scalability and efficiency for their anticipated use of the platform. We had done previous work for this client, so when the time came to integrate and modernize their data landscape, they reached out for help.

Modern Data Platform Evaluation

The first step was to assess the client’s current data landscape and state of applications and their demands for data integration based on business needs.

We looked at their business requirements for data and functionality, from the key select sources including WolfPack, Guardian, and select unstructured manual data sources, and organized the data into subject areas to serve as a semantic layer for all data available.

With an understanding of their current state, we then began evaluating viable solution options for a target modern data platform. Based on feedback from the client, we presented two options with a focus on cloud implementations that support BI and analytics — an Azure-based data architecture and a Snowflake Data Cloud solution.

Then we provided the client with a complete assessment deliverable that included our findings of their current state and viable options for the modern data platform, including:

• An outline of solution options, components, benefits, and impacts
• An outline of the data architecture
• Estimated costs for development and deployment
• Anticipated operational costs
• Side-by-side comparisons of key features and considerations of the two options

We also provided the client with a high-level roadmap of prioritized next steps, which included the creation of POCs to test the business use cases against.

Modern Data Platform Proof of Concepts (POCs)

Armed with the information from our initial assessment, the client asked us to execute a combination of POCs to demonstrate how these modern Azure-based cloud platforms and technologies would support the company’s BI and analytics needs.

To create the POCs, we defined the modern data platform reference architecture within Azure with options for specific technologies, including native Azure components (e.g., Azure Data Factory, Azure Data Lake, Azure SQL Data Warehouse, Azure Analysis Services, PowerBI, and Azure DataBricks), and Snowflake.

Building the POCs included:

• Configuration of Azure subscription and services
• Design of structures in Azure SQL Data Warehouse
• Creation of tabular model in Azure Analysis Services
• Execution of data pipelines into Azure and Snowflake
• Creation of a Snowflake database & designated views
• Build out of data pipelines between Azure services and Snowflake
• Creation of analytics model using Spark with Azure DataBricks

Once the platform was created, we demonstrated BI reporting using PowerBI against the POC platforms and the execution of machine learning use cases using Azure ML studio and custom model development.

We then tested multiple use cases for data ingestion, BI, and analytics. We validated various use cases for viability, including:

• Financial reporting using Dynamics AX data that is integrated with budgeting and pricing data
• Supply chain optimization
• Predictive maintenance