Work | New Era Technology | Your digital transformation partner

Modern data architecture sets foundation for water innovation company | New Era Technology

Written by New Era Technology | Jan 10, 2023 5:00:00 AM

Solutions 

We recommended a four-phase approach that provided a complete assessment and analysis of their current data architecture, future state options, roadmap, and proofs of concept based on the final selections. This would allow us to build out use cases for the modern data architecture. 

A Four-phased Foundation: From Assessment to Tool Selection

Phase 1: Analysis and Information Gathering

Gathering essential information to understand the needs and factors the define the objectives of the platform.

  • Review the Client Data Strategy artifacts to understand business strategy and platform objectives
  • Gain and understanding of key business and technical factors leading to an interest in cloud or modern data platforms
  • Gain an understanding of the most essential high-level use cases that will increase in value from the platform
  • Plan and conduct supplemental interviews to address any unknowns and gaps in understanding
  • Deeper dive with technical representatives on the current state landscape and technical considerations as necessary

Phase 2: Current State Architecture Readout

Capture the challenges with the current architecture and ecosystem and identify the main criteria to be considered in modern architecture.

  • Identifying constraints, limitations, and pain points with the current architecture
  • Assimilating criteria for considering the cloud and modern data platforms
  • Validate criteria with project sponsors

Phase 3: Future State Architecture Options and Roadmap

Document the architecture options and establish the roadmap and next steps to move to the next phase.

  • Outline summary of possible cloud platforms, modern data architectures, and technologies. 
  • With the assumption that the Client has preferences for a cloud platform and relevant data management technologies, focus on identifying the services and design patterns most relevant for that architecture
  • Evaluate the options for key components including ETL/ELT orchestration, data/database tech, and metadata catalog
  • Identify solutions for various ingestions, including near-real-time and streaming requirements
  • Develop a recommended strategy to transition to modern architecture
  • Identify POCs to finalize recommendation or for further validation, if either applies
  • Engage with product vendors in support of tools rationalization, demonstrations, and POCs

Phase 4: Proof of Concept for Tools and Final Recommendations

Perform further rationalization, demonstrations, and POCs of the options identified in the architecture

  • Enable, acquire, and configure tools for demonstrations and POCs, working with vendors, and Client infrastructure teams
  • Refine criteria and scope for POCs
  • Define the method for evaluating and measuring POC results
  • Perform POCs and capture results
  • Conduct a review of results and work with the Client to support decision making
  • Identify stand-up engagements for pilot phases
  • Define the scope of the pilot; data source(s), data subject area model, data pipeline patterns, BI dashboard/report, aspects of the data catalog

Goals

Project Goal

Rinse and Repeat (or Iterative Implementation

As seen in phase 4, standing up and providing the initial iteration is basically creating a pilot program. We recommended taking an iterative approach so the client could see quick successes with the modernized architecture and allow them to continue to build on it as use cases surfaced. Once an iteration worked, we repeated it with new data sources. 

Our Solution
Each time, we would:
  • Figure out what data they still needed
  • What sources they had or did not have to provide it
  • Identify why it was important
  • Add the data sources to the data warehouse in the cloud

All without the Client having to worry about the on-prem limitations they were previously using and with a roadmap of use cases to expand upon. 


Project Goal

Modernize BI and Analytics Work and Reporting

With near-real-time access to data and improved data pipeline patterns in place as seen in Phase 4, we wanted to create a successful, easy to use BI dashboard. This would let them put their data to work and gain powerful and functional insight from the data they collect. 

Our Solution

With the data sources streaming into the cloud-based data warehouse, we could shape the data into the language the BI teams are using. 

  • Transform the data
  • Configure new, updated reports
  • Build a successful BI dashboard/report

They now had access to the right data they needed to achieve greater business insights and make informed decisions.