Kerberos support in Sqoop2

Sqoop2 has a new security framework, which includes support for:

  1. Simple authentication
  2. Kerberos authentication

This blog post will detail how to setup Sqoop2 with Kerberos.

Bringing Kerberos support in Sqoop2 was a co-engineering effort of Intel and Cloudera.

TLDR Setup

Set the following configuration properties in sqoop.properties. Make sure the principals and keytab provided exist.

org.apache.sqoop.authentication.kerberos.principal=sqoop/_HOST@
org.apache.sqoop.authentication.kerberos.keytab=/home/kerberos/sqoop.keytab
org.apache.sqoop.authentication.kerberos.http.principal=HTTP/_HOST@
org.apache.sqoop.authentication.kerberos.http.keytab=/home/kerberos/sqoop.keytab
org.apache.sqoop.authentication.kerberos.proxyuser=true

Start Sqoop2 server and client. Then run:

export SQOOP2_HOST=<FQDN>
sudo –u sqoop <SQOOP2 DIRECTORY>/bin/sqoop.sh server start
kinit <USER PRINCIPAL>
<SQOOP2 DIRECTORY>/bin/sqoop.sh client

Setup

Setting up Kerberos in Sqoop2 is a breeze. It can be achieve in 4 steps:

1. Setup Kerberos

The following principals need to be created:

  1. sqoop/<FQDN>@<REALM>
  2.  HTTP/<FQDN>@<REALM>

Also, a keytab needs to be created containing both those principals needs to be created. Here’s how to do it:

addprinc -randkey HTTP/@
addprinc -randkey sqoop/@
xst -k /home/kerberos/sqoop.keytab HTTP/@
xst -k /home/kerberos/sqoop.keytab sqoop/@

Check out the MIT Kerberos KDC install guide for more information on setting up Kerberos.

2. Configure Sqoop2

Set the following configuration properties in sqoop.properties.

org.apache.sqoop.authentication.kerberos.principal=sqoop/_HOST@
org.apache.sqoop.authentication.kerberos.keytab=/home/kerberos/sqoop.keytab
org.apache.sqoop.authentication.kerberos.http.principal=HTTP/_HOST@
org.apache.sqoop.authentication.kerberos.http.keytab=/home/kerberos/sqoop.keytab
org.apache.sqoop.authentication.kerberos.proxyuser=true
  • The <FQDN> should be replaced by the Fully Qualified Domain Name of the server, which could be found via “hostname -f” in command line.

  • Sqoop2 uses SPNEGO to facilitate authentication. The principal HTTP/<FQDN>@<REALM> is required for SPNEGO and is case-sensitive.

3. Start Sqoop2

export SQOOP2_HOST=<FQDN>
sudo –u sqoop <SQOOP2 DIRECTORY>/bin/sqoop.sh server start
kinit <USER PRINCIPAL>
<SQOOP2 DIRECTORY>/bin/sqoop.sh client
  • SQOOP2_HOST must be set in order to tell the client which host is used when negotiating authentication. An alternative to this is to use set server –host <FQDN>.
  • Sqoop2 server should be started as the user that will have the sqoop/<FQDN>@<REALM> prinicipal. In the above example, it’s the user sqoop.
  • Before running the sqoop client, you must kinit with the principal that you’d like to run your jobs.

4. Verify it’s working

Now that you have Sqoop2 server running with Kerberos, try out a command:

show version --all

Summary

We hope you enjoy the new security framework and kerberos support in Sqoop2. If you have any questions, reach out to us at team@ingest.tips or to @ingesttips.

Tweet about this on TwitterShare on FacebookShare on LinkedIn

Tagged:


'Kerberos support in Sqoop2' has no comments

Be the first to comment this post!

Would you like to share your thoughts?

Your email address will not be published.