Netflix Details Workload Attestation for Apache Spark on Amazon EMR

Bridging Cloud and Internal Identities

Organizations often operate with two distinct identity systems: one provided by the cloud vendor (e.g., AWS IAM roles) and an internal system for service-to-service authentication. On managed compute platforms like Amazon EMR, workloads initially only possess the cloud provider's identity. Netflix addressed this by creating a mechanism for Apache Spark workloads to acquire a first-class internal identity.

Netflix's Internal Identity System

Netflix's internal service-to-service authentication relies on Metatron, a private Public Key Infrastructure (PKI). Workloads receive short-lived X.509 certificates, and mutual TLS is used for service authentication. The issuance of these certificates is contingent on an attestation process, which verifies the workload's claimed identity based on environment-specific proofs.

Data Projects and Identity Mapping

Netflix defines 'Data Projects' as units of data ownership, each with its own identity, authorized users, and associated tables. Jobs run under the Data Project's identity, ensuring consistent table-level access control and audit trails. A critical component of the solution is a 1:1 mapping between each Data Project identity and a dedicated AWS IAM role, recorded by the service managing Data Project metadata.

Solving the Identity Gap

The core problem was that Spark jobs on managed compute, starting with only an AWS execution role, lacked the internal identity required for accessing internal company resources like encrypted columns, evaluating table ACLs, or maintaining interactive sessions. The established mapping between Data Projects and IAM roles facilitates the translation, allowing a process running with a specific AWS role to be recognized with its corresponding internal identity.