BigQuery Python UDFs Now Support Standard Logging to Cloud Logging

New Logging Feature in BigQuery Python UDFs

BigQuery has introduced a new logging feature for Python User-Defined Functions (UDFs). This update allows developers to use Python's standard logging library directly within BigQuery UDFs. Structured logs can now be seamlessly emitted into Cloud Logging, removing the need for external plugins or custom connectors.

Enabling Pipeline Observability

This new capability supports various operational monitoring and debugging workflows. One key application is building fault-tolerant and resilient data pipelines. The feature enables the capture of row-level diagnostic telemetry during query execution, which helps teams isolate problematic records, trigger real-time alerts, and triage issues using SQL without stopping entire workloads.

Example Workflow: E-commerce Event Ingestion

An example demonstrates an e-commerce event ingestion pipeline with three stages. First, an inline validation Python UDF parses incoming JSON payloads; valid records proceed, while corrupted rows trigger logging.error() and return NULL. Second, Cloud Monitoring uses log-based metrics to notify teams if error rates exceed thresholds. Third, data engineers can query quarantined payloads in BigQuery using standard SQL over the _AllLogs view to diagnose root causes and re-ingest recovered data.

Implementation Details

To use the feature, developers create a generic Python UDF that parses JSON and checks for required fields, logging diagnostic JSON and returning NULL on failure. Logging is enabled in the session using SET @@udf.properties. This allows queries to succeed without interruption, routing bad rows to Cloud Logging for later analysis and alerting.