2022/11/27

D4P2 - Data-Driven Design & Development of Product or Platform

How to build a data architecture to drive innovation--today and tomorrow | McKinsey

• Design and deploy new data technologies alongside legacy infrastructure to drive market-driven innovations such as personalized offers, real-time alerts, and predictive maintenance.
• from data lakes to customer analytics platforms to stream processing—have increased the complexity of data architectures enormously, often significantly hampering an organization’s ongoing ability to deliver new capabilities, maintain existing infrastructures, and ensure the integrity of artificial intelligence (AI) models.
• need a new approach to defining, implementing, and integrating their data stacks, leveraging both cloud (beyond infrastructure as a service) and new concepts and components.

Data Architecture.png

six shifts.png

1. From on-premise to cloud-based data platform
   從內(nèi)部部署到基于云的數(shù)據(jù)平臺(tái)
   • Serverless data platform e.g. Amazon S3 and Google BigQuery
   • Containerized data solutions
2. From batch to real-time data processing
   從批處理到實(shí)時(shí)數(shù)據(jù)處理
   • Messaging platform e.g. REBAR Messaging
   • Streaming processing and analytics solutions e.g. Data Fabric
   • Alerting platforms e.g. DataDog
3. From pre-integrated commercial solutions to modular, best-of-breed platforms
   從預(yù)集成的商業(yè)解決方案到模塊化、一流的平臺(tái)
   • Data pipeline and API-based interfaces
   • Analytics workbenches e.g. Amazon Sagemaker and Kubeflow
4. From point-to-point to decoupled data access
   從點(diǎn)到點(diǎn)到解耦數(shù)據(jù)訪(fǎng)問(wèn)
   • An API management platform e.g. API Gateway and Open API
   • A data platform to "buffer" transactions outside of core systems e.g. Data Lake and Data Mesh
5. From an enterprise warehouse to domain-based architecture
   從企業(yè)倉(cāng)庫(kù)到基于領(lǐng)域的體系結(jié)構(gòu)
   • Data infrastructure as a platform
   • Data visualization techniques
   • Data cataloging tools
6. From rigid data models to flexible, extensible data schemas
   從剛性數(shù)據(jù)模型到靈活、可擴(kuò)展的數(shù)據(jù)模式
   • Data vault 2.0 techniques
   • Graph databases
   • Technology services
   • JavaScript Object Notion (JSON)

How to get started

• Apply a test-and-learn mindset
• Establish data "tribes
• Invest in DataOps
• Create a data culture"