Netflix Develops Workload Attestation for AWS EMR Spark Jobs to Bridge IAM and Internal PKI

Bridging AWS IAM and Internal Identity

Netflix developed a workload attestation mechanism for Apache Spark jobs running on Amazon EMR. This system allows these jobs to convert their AWS execution identity into a short-lived internal workload identity. The primary goal is to connect AWS IAM with Netflix’s internal Metatron Private Key Infrastructure (PKI).

Authentication for Internal Services

The new mechanism enables managed compute workloads to authenticate to Netflix's internal services. This eliminates the need for manual credential injection, simplifying the security and operational aspects of running Spark jobs that interact with internal systems. Netflix uses Metatron, a private PKI, to issue short-lived X.509 certificates for mutual TLS authentication within its internal services.

Identity Mapping and Scalability

Netflix operates two distinct identity systems: AWS IAM for managed workloads and Metatron for internal services. The new design maps each internal Data Project to a dedicated IAM role. To handle a large number of Data Projects, these roles are distributed deterministically across a pool of dedicated AWS accounts, rather than being concentrated in a single account.

Attestation Flow

When a job is submitted, Netflix's control plane validates the request and creates a signed workload metadata payload. The Spark driver then uses its AWS execution role credentials to request its identity from the AWS Security Token Service (STS). Netflix's Identity service retrieves this request from AWS STS to verify the IAM role. It then compares the AWS response with the signed metadata from the control plane. If they match, the Identity service issues a Metatron certificate, confirming the workload's identity.

Handling Spark Executor Fan-Out

The architecture also addresses the challenge of Spark's executor fan-out. Instead of each of thousands of executors independently performing attestation, the Spark driver attests once. It then distributes the necessary credentials to its executors via authenticated and encrypted Spark RPC, reducing attestation traffic.