Securing Spark on YARN with Kerberos and ACLs: Authentication, Authorization, and Web UI Login on a Hadoop and Spark Cluster

In this session, Samkit Jain, Junior Software Engineer at Ksolves India Limited, addresses a security challenge that many Hadoop and Spark teams underestimate: enabling authentication is only the beginning. A truly secure distributed data processing environment also requires controlled access to workloads and queues, enforced user permissions across applications, and protected web interfaces that do not expose administrative functionality to unauthorized users.

Many clusters that have Kerberos enabled still carry significant security gaps. YARN queue ACLs are left permissive, allowing any authenticated user to submit jobs to any queue. Spark application ACLs are not configured, leaving workload management open to unintended access. The YARN ResourceManager web UI operates without proper authentication, showing anonymous dr.who identities instead of verified users. And auth_to_local rules are configured incorrectly, creating mismatches between Kerberos principals and the cluster user identities they are supposed to map to. These are not edge cases. They are common misconfigurations that leave otherwise authenticated clusters exposed.

This webinar covers the full security model for Spark workloads running on YARN, from Kerberos fundamentals through to practical testing of access restrictions. Samkit walks through how Kerberos tickets, principals, and keytabs work in a Hadoop environment, how to configure Kerberos for Spark applications running on YARN, how to submit long-running Spark jobs using principals and keytabs without requiring passwords, how to implement YARN queue ACLs and Spark application ACLs to control workload access, and how to enable secure browser access to the ResourceManager web UI using SPNEGO and Kerberos authentication. The session also covers HDFS user isolation, identification of common security misconfigurations, and practical approaches to validating that authentication and access restrictions are working as intended.

Whether you are working with Hadoop administration, Spark, YARN, or big data infrastructure, this session provides practical insights into strengthening cluster security and managing access across distributed workloads.

Key Takeaways

• How Kerberos authentication works in Hadoop and YARN environments, including tickets, principals, and keytabs.
  
• Configuring Kerberos for Spark applications running on YARN in practice.
  
• Submitting Spark jobs using principals and keytabs without requiring passwords for long-running workloads.
  
• Implementing YARN queue ACLs and Spark application ACLs to control access across teams.
  
• Enabling secure browser access to the YARN ResourceManager web UI using SPNEGO and Kerberos authentication.
  
• Replacing anonymous dr.who identities with verified users in the ResourceManager web UI.
  
• Identifying common security misconfigurations including permissive root queue ACLs and incorrect auth_to_local rules.
  
• Validating authentication and access restrictions through practical testing approaches.