Recently I was in charge of implementing the single-sign-on feature of Oracle using Kerberos at my workplace. With this post I would like to pinpoint some pitfalls and how to avoid them when setting up and troubleshooting Oracle Kerberos. I do not aim to provide a complete guide on the topic as there are already many useful blog posts and Oracle documents available, for example:
http://laurentschneider.com/wordpress/2014/01/the-long-long-route-to-kerberos.html
https://docs.oracle.com/database/121/DBSEG/asokerb.htm#DBSEG062
Our Oracle database environment is currently consisting of Oracle 11g and 12c installations running on Solaris x64 platform. The first bug I encountered was related to Solaris:
Solaris Bug: 21246454 – kerberos ticket on solaris 11 x64 shows garbage characters
This bug can bee seen in your client side trace files if you switch on support level trace in the Oracle Client’s sqlnet.ora file.
(1) [15-DEC-2015 08:20:43:701] nauk5aegetservcred: Client is t„„„„„„@TESTDOMAIN.COM, Service provider is oracle/dbhost„„@TESTDOMAIN.COM (1) [15-DEC-2015 08:20:43:701] nauk5ba_fcc_retrieve: entry (1) [15-DEC-2015 08:20:43:702] nauk5ba_fcc_retrieve: Looking for client t„„„„„„@TESTDOMAIN.COM. (1) [15-DEC-2015 08:20:43:702] nauk5ba_fcc_retrieve: Looking for service oracle/dbhost„„@TESTDOMAIN.COM. (1) [15-DEC-2015 08:20:43:702] nauk5b1_fcc_open_file: entry (1) [15-DEC-2015 08:20:43:702] snauk5g_open_file: entry (1) [15-DEC-2015 08:20:43:702] snauk5g_open_file: Setting read lock.
As you can see the Kerberos stack does not recognize the active directory account of the client: Instead of t0001 it is looking for “t,,,,,,,,,” and the same applies to the host of the database. You will probably encounter this bug if you are using Oracle 12c on Solaris x64 and either the database hostname or the client user’s Active Directory account contains numbers. The patch for this bug will not be included in any PSU so you will need to download it and install it on all of your Oracle 12c homes if they are on a Solaris x64 platform.
Preauthentication does not work with Oracle 11.2.0.3
As you probably know the oracle database checks with the help of an Active Directory SPN if a client has the right credentials. The Active Directory user for the SPN should look something like this:
Please note that the “Do not require Kerberos preauthentication” should be set to enabled otherwise you might see the following errors during login:
oracle@dbhost:/u00/oracle/orabase/product/11.2.0.3.7_a/ $ sqlplus passworduser@DBER6 SQL*Plus: Release 11.2.0.3.0 Production on Wed Aug 24 15:15:12 2016 Copyright (c) 1982, 2011, Oracle. All rights reserved. Enter password: ERROR: ORA-12638: Credential retrieval failed
And during cloning:
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of Duplicate Db command at 05/09/2016 09:37:39 RMAN-05501: aborting duplication of target database RMAN-03015: error occurred in stored script Memory Script RMAN-03009: failure of backup command on ch1 channel at 05/09/2016 09:37:28 ORA-17629: Cannot connect to the remote database server ORA-17627: ORA-12638: Credential retrieval failed ORA-17629: Cannot connect to the remote database server
In both cases a user with password authentication tries to login to the database and fails. Even if SQLNET.FALLBACK_AUTHENTICATION is set to TRUE password identified users cannot login to the database. The root cause for this problem is that Fallback only works if a Kerberos ticket expires. In case the Kerberos configuration is corrupted or in this case the database cannot do Preauthentication, fallback cannot happen and the ORA-12638: Credential retrieval failed message is raised.
By using the klist -f -e -c command we can make sure that the Oracle Instance is using the right type of kerberos ticket:
$ /usr/bin/klist -f -e -c /u00/oracle/orabase/network/admin_kerberos/cache
Ticket cache: FILE:/u00/oracle/orabase/network/admin_kerberos/cache
Default principal: oracle/kerbuser@TESTDOMAIN.COM
Valid starting Expires Service principal
05/09/16 13:51:02 05/09/16 23:51:02 krbtgt/TESTDOMAIN.COM@TESTDOMAIN.COM
renew until 05/16/16 13:51:02, Flags: FRI
Etype(skey, tkt): ArcFour with HMAC/md5, ArcFour with HMAC/md5
05/09/16 13:51:02 05/09/16 23:51:02 oracle/kerbuser@TESTDOMAIN.COM
renew until 05/16/16 13:51:02, Flags: FR
Etype(skey, tkt): AES-256 CTS mode with 96-bit SHA-1 HMAC, AES-256 CTS mode
with 96-bit SHA-1 HMAC
The example above shows the type of ticket which is needed for an 11.2.0.3 Instance. Please note the Flags sections market with blue. These flags mean that no Preauthentication is required as the tickets were generated with an SPN which has the “Do not require Kerberos preauthentication” property set in Active Directory as shown in the picture above. In case the property is not set you might see the flags FRIA and FRA where the A flag means Preauthenticated. For the full list of Kerberos ticket flags you can see the Oracle documentation here:
http://docs.oracle.com/cd/E19253-01/816-4557/user-1/
Another problem with 11.2.0.3 databases is that with this version of Oracle the instance is not able to get a kerberos ticket when a client tries to establish connection so we need to find a workaround for this problem. We need to make sure that the instance always has a valid ticket. So here is how we can get a ticket for the Oracle Server using the keytab file:
export TNS_ADMIN=/u00/oracle/orabase/network/admin_kerberos kinit -c /u00/oracle/orabase/network/admin_kerberos/cache -k -t /u00/oracle/orabase/network/admin_kerberos/dbhost01.keytab oracle/dbhost01 kvno -c /u00/oracle/orabase/network/admin_kerberos/cache oracle/dbhost01
Let us check now if we really received a valid ticket:
$/usr/bin/klist -e -c /u00/oracle/orabase/network/admin_kerberos/cache Ticket cache: FILE:/u00/oracle/orabase/network/admin_kerberos/cache Default principal: oracle/dbhost.testdomain.com@TESTDOMAIN.COM Valid starting Expires Service principal 03/30/16 11:36:45 03/30/16 21:36:45 krbtgt/testdomain.com@TESTDOMAIN.COM renew until 04/06/16 11:36:45, Etype(skey, tkt): ArcFour with HMAC/md5, ArcFour with HMAC/md5 03/30/16 11:36:45 03/30/16 21:36:45 oracle/dbhost.testdomain.com@TESTDOMAIN.COM renew until 04/06/16 11:36:45, Etype(skey, tkt): AES-256 CTS mode with 96-bit SHA-1 HMAC, AES-256 CTS mode with 96-bit SHA-1 HMAC
In case you are working a huge number of Oracle hosts it is a good idea to automate the whole process. For this purpose I have created the following script:
#!/usr/bin/ksh hostn=`/usr/bin/hostname` servn=`/usr/bin/grep $hostn /etc/hosts | /usr/bin/awk '{ print $2}'` /usr/bin/kinit -c /u00/oracle/orabase/network/admin_kerberos/cache -k -t /u00/oracle/orabase/network/admin_kerberos/$hostn.keytab oracle/$servn /usr/bin/kvno -c /u00/oracle/orabase/network/admin_kerberos/cache oracle/$servn
After saving the above as get_kerb_ticket you can use this script in crontab as follows:
#get kerberos tickets 0 4,10,16,22 * * * /u00/oracle/orabase/local/bin/get_kerb_ticket > /dev/null 2>&1
If the kerberos ticket expires every 8 huours for example you might schedule the script to run every 6 hours like in my example above.
Configuring Kerberos with AES 256 bit encryption
SPN:
First we have to make sure that the SPN of the Oracle Instance host is able to use AES 256 encryption. This can be enabled by setting the following Active Directory attribute of the SPN: “This Account supports Kerberos AES 256 bit encryption”. This setting can be found on the Account tab of the Active Directory user as shown on the picture in the preauthentication section.
Furthermore, we need to make sure that we are using the right SPN for the Keytab file. For this we need to check the /etc/hosts file:
::1 localhost
127.0.0.1 localhost loghost
10.144.215.11 dbhost01 dbhost01.testdomain.com
Username Logon name should always be the second coulmn of the etc/hosts file, so in our case: oracle/dbhost01
The SPN mapped to user should be: oracle/dbhost01@TESTDOMAIN.COM
Make sure that the domain name is written with capitals.
Keytab File:
The third important issue to note here is the creation of the keytab file. When using the ktpass command we need to specify that it should use only one type of encryption. This can be done by using the -crypto parameter as shown in the example below:
c:\Windows\system32>ktpass -princ oracle/dbhost@TESTDOMAIN.COM -mapuser testdomain-com\dbhost -pass secretpassword -ptype KRB5_NT_PRINCIPAL -crypto AES256-SHA1 -out c:\temp\dbhost.keytab Targeting domain controller: dcont01.testdomain.com Succesfully mapped oracle/dbhost to dbhost. Password successfully set! Key created. Output keytab to c:\temp\dbhost.keytab: Keytab version: 0x502 keysize 72 oracle/dbhost@TESTDOMAIN.COM ptype 1 <KRB5_NT_PRINCIPAL> vno 5 etype 0x12 <AES256-SHA1> keylength 32 <key in hex format is displayed here>
To verify that the Oracle instance is using AES 256 we can check the server side trace files. In case AES256 is used you can see the following:
[15-JAN-2016 16:07:09:507] nauk5y2_kt_get_entry: Searching for keytype=18 ,kvno=13;Current keytype=18,kvno=12 [15-JAN-2016 16:07:09:507] nauk5fq_free_principal: entry [15-JAN-2016 16:07:09:507] nauk5fq_free_principal: exit [15-JAN-2016 16:07:09:507] nauk5y2_kt_get_entry: Searching for keytype=18 ,kvno=13;Current keytype=18,kvno=13
Please note the following part in the trace: “Searching for keytype=18” This means that Oracle is using AES 256 encryption. In case you see something else it means that the Oracle Kerberos stack is not using AES 256. For a full list of keytype codes you can see the Oracle documentation: https://docs.oracle.com/cd/B28359_01/network.111/b28530/asokerb.htm#i1006531
You can also verify the Keytab file by using the ktutil command:
$ ktutil ktutil: read_kt /u00/oracle/orabase/network/admin_kerberos/dbhost.keytab ktutil: list -e slot KVNO Principal ---- ---- --------------------------------------------------------------------- 1 7 oracle/dbhost.testdomain.com@TESTDOMAIN.COM (AES-256 CTS mode with 96-bit SHA-1 HMAC)
As you can see above the only encryption allowed by this keytab file is the AES 256 encryption.
Have you ever seen this error on the windows client side?
(7148) [07-JUL-2017 09:54:32:110] nauztk5ainit: Configuration name is “/u01/app/kerberos/krb5.conf”
(7148) [07-JUL-2017 09:54:32:110] nauztk5ainit: Credential Cache Pathname is “OSMSFT://”
(7148) [07-JUL-2017 09:54:32:110] nauztk5ainit: Error code for Client Initialization “4”
(7148) [07-JUL-2017 09:54:32:110] nauztk5ainit: ZTK Context after initialization “00BCCEB0”
(7148) [07-JUL-2017 09:54:32:110] nauztk5ainit: ztk_client_init_context() failed.
(7148) [07-JUL-2017 09:54:32:110] nauztk5ainit: Internal Kerberos error for client init “Can’t open/find Kerberos configuration file”
It seems to be looking for the kerberos conf file on the windows client using the conf parameter in the sqlnet.ora file on the DB server. The windows client has the conf parameter set pointing to a different local directory.