● Covered by 2 sources · 2 reportsMedium impact2 neutral
Google Cloud's Lakehouse enables BigQuery to query Databricks Apache Iceberg tables directly
Key points
- BigQuery queries Databricks Iceberg tables directly.
- Zero-copy approach eliminates data duplication.
- Supports GCS, Amazon S3, and Azure Blob Storage.
- Metadata refreshes every 5 minutes.
- Centralized security via delegated access.
- Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg.
- Migration works with Parquet tables in Amazon S3 registered in the AWS Glue Data Catalog.
Direct Querying of Databricks Iceberg Tables
Google Cloud's borderless Lakehouse now allows BigQuery to directly query Apache Iceberg tables managed by Databricks. This integration eliminates the need to copy data between platforms, addressing historical challenges of bridging systems like Databricks and BigQuery for analytics workloads.
How the Integration Works
Databricks-managed Iceberg tables automatically generate Iceberg-compatible metadata alongside native Delta logs. Lakehouse reads this Iceberg metadata directly, applying BigQuery’s vectorized engine to open-format Parquet files stored in Google Cloud Storage (GCS). This zero-copy approach means BigQuery reads the same files Databricks writes, with changes appearing after Lakehouse's scheduled metadata refresh, which occurs as often as every 5 minutes.
Multi-Cloud and Cross-Cloud Capabilities
While GCS is a primary storage layer, Lakehouse extends this federation to external cloud environments such as Amazon S3 and Azure Blob Storage. By combining the open Apache Iceberg REST Catalog with cross-cloud connections, BigQuery can query data across multiple clouds without physically moving it. Each federated catalog is located in a user-chosen Google Cloud region, ideally close to the data, enabling multi-cloud analytics without replication pipelines. Cross-Cloud Interconnect (for AWS) and Lakehouse caching can help reduce remote-cloud egress charges.
Read/Write Interoperability and Security
The Lakehouse framework, built on Apache Iceberg, establishes read and write (R/W) interoperability across different engines, allowing data teams to execute multi-statement SQL commands directly within BigQuery. For security, Lakehouse centralizes access controls through delegated access. Administrators can configure a BigQuery Cloud Resource Connection linked to a service account, which manages IAM rights for GCS buckets, addressing vulnerabilities that arise from managing access across disparate systems.
Updates
- Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg.
- Migration works with Parquet tables in Amazon S3 registered in the AWS Glue Data Catalog.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub aws/agent-toolkit-for-awsHow outlets covered it
Amazon S3 Tables now provide a managed service for migrating Apache Hive tables to Apache Iceberg, specifically for Parquet tables registered in the AWS Glue Data Catalog. This feature automates the migration process, including data validation and type conversions, allowing data lake modernization without manual pipeline construction or compute provisioning.
Google Cloud's borderless Lakehouse now allows BigQuery to directly query Databricks-managed Apache Iceberg tables without data duplication. This integration removes the need for complex data pipelines, reducing storage costs, latency, and fragmented security controls. It enables cross-cloud analytics by reading open-format files in Google Cloud Storage, Amazon S3, or Azure Blob Storage.