Data Platform
Proximity
Middleware / wrapper
Models
middleware / wrappers
Cloud · datastore
Multi-CloudApache KafkaksqlDB
Compliance
SOC 2HIPAAPCI DSSISO 27001
Architecture
Managed Apache Kafka (Confluent Cloud) deployed across AWS/Azure/GCP with the Kora cloud-native engine; adds stream processing via Flink. Built around open Kafka with proprietary cloud tooling.
Data Platform
Proximity
Native (own / self-hosted weights)
Models
proprietary / self-built modelsMeta · LlamaOpenAI · GPT
Cloud · datastore
Multi-CloudDelta Lake (transactional lakehouse tables on cloud object storage)[6][10]Apache Spark (distributed compute engine for ETL/ML/streaming)[11][12]Unity Catalog (centralized governance and metadata layer)[1][5]Cloud object storage (Amazon S3, Azure Data Lake Storage, Google Cloud Storage) as primary data store[6][4][12]Medallion (Bronze/Silver/Gold) data modeling pattern on top of Delta tables[2][12]
Compliance
Separation of control plane (Databricks-managed backend services) and compute/data plane (customer cloud account) for isolation and governance[3][4][7][11]Network isolation via VPC/VNet injection, CIDR planning, and private connectivity for data plane resources[1][5][7]Workspace- and catalog-level access control with Unity Catalog (role-based access, row/column-level policies)[1][5][10]Encryption of data in transit and at rest via underlying cloud provider primitives (KMS/Key Vault, TLS)[5][10]Governance features including lineage tracking, masking, and auditability built into lakehouse stack[1][10]
Architecture
Databricks runs a two-layer architecture where a Databricks-managed control plane hosts the UI, APIs, metadata, and orchestration services, while customer workloads execute in a compute/data plane inside the customer’s AWS, Azure, or GCP account (or Databricks’ serverless account) against cloud object storage using Spark and Delta Lake.[3][4][9][11] Around this core, Databricks positions an open-core lakehouse stack (Delta Lake, Spark, MLflow) with proprietary governance, serverless, and AI platform services, integrated into medallion-style production patterns.[1][2][6][12]