Wednesday, November 6, 2024

Troubleshooting Oracle One-way TLS Connection Errors


While working through several iterations of one-way Transport Layer Security (TLS) connection testing between Oracle Database client and server, I captured the troubleshooting techniques and possible reasons and remediation suggestions for the errors that I encountered.  This blog post catalogs those errors for my reference. 

Error: ORA-28759: failure to open file
Possible Reasons:
1. The client wallet exist does not exist. Copy client wallet from dbca to local client wallet directory (/u01/app/oracle/tls). See section 2.2.
2, The WALLET_LOCATION path in tnsnames.ora does not match the actual client wallet location. For example, following two paths should match:

Actual client wallet location: /u01/app/oracle/tls

The tnsnames.ora definition for HRDB_SSL:

HRDB_SSL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCPS)(HOST = hrdb.example.com)(PORT = 2484))
    (SECURITY=(WALLET_LOCATION=/u01/app/oracle/tls))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = hrdb)
    )
  )


Error: ORA-12154: Cannot connect to database. Cannot find alias %s in %s.
Reason: Either service is not specified in tnsnames.ora or the wrong service name was used in sqlplus command.


Error: ORA-12547: TNS:lost contact
Reason: The PROTOCOL of tnsnames.ora definition is TCP instead of TCPS for SSL/TLS connection or vice versa for non-SSL/TLS connection.  For example, the PROTOCOL of the following tnsnames.ora should be TCPS rather than TCP:


HRDB_SSL =

  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = hrdb.example.com)(PORT = 2484))
    (SECURITY=(WALLET_LOCATION=/u01/app/oracle/tls))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = hrdb)
    )
  )



Error: ORA-28864: SSL connection closed gracefully
Possible Reasons:
1. The PROTOCOL definition of the database server's listener.ora is set to TCP rather than TCPS. For example, changing the blue TCP to TCPS and restarting the Oracle database listener should resolve this error.

cat $ORACLE_HOME/network/admin/listener.ora
SSL_CLIENT_AUTHENTICATION = FALSE

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = hrdb.example.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = hrdb.example.com)(PORT = 2484))
      (SECURITY=(WALLET_LOCATION=/u01/app/oracle/19c/wallet_root/tls))
    )
  )

ADR_BASE_LISTENER = /u01/app/oracle/19c


Error: Segmentation fault (core dumped)
Possible Reasons:
1. The TCPS address has the wrong port. Change to the correct port (2484).


cat $ORACLE_HOME/network/admin/listener.ora

SSL_CLIENT_AUTHENTICATION = FALSE

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = hrdb.example.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
      (ADDRESS = (PROTOCOL = TCPS)(HOST = hrdb.example.com)(PORT = 9999))
      (SECURITY=(WALLET_LOCATION=/u01/app/oracle/19c/wallet_root/tls))
    )
  )

ADR_BASE_LISTENER = /u01/app/oracle/19c


2. The TCPS address has the wrong address. Change to the correct port.

cat $ORACLE_HOME/network/admin/listener.ora
SSL_CLIENT_AUTHENTICATION = FALSE

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = hrdb.example.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
      (ADDRESS = (PROTOCOL = TCPS)(HOST = wronghost.example.com)(PORT = 2484))
      (SECURITY=(WALLET_LOCATION=/u01/app/oracle/19c/wallet_root/tls))
    )
  )

ADR_BASE_LISTENER = /u01/app/oracle/19c



Error: ORA-28860: Fatal SSL error

Possible Reasons:
1. SSL_CLIENT_AUTHENTICATION setting in the database client or server sqlnet.ora may be set to TRUE instead of FALSE in the Oracle database server listener.ora.   Wireshark/tshark analysis of the secure port (2484) while sqlplus connect is run.  From the following output, we see the failed connection:

sudo yum install -y wireshark

sudo tshark -i any -d "tcp.port==2484,ssl" -V -a duration:20 2> /dev/null | egrep "Cipher Suites \(|Cipher Suite:|^            Version: |SSL Record Layer: Handshake Protocol: Client Hello|Handshake Protocol: Server Hello|Record Layer"| uniq | sed -e "s/(0x.*)//g" -e "s/.*SSL Record Layer: Handshake Protocol: Client Hello/Client requested:/g" -e "s/.*Version:/   Protocol Version:/g" -e "s/.*Cipher Suites /   Cipher Suites Requested:/g" -e "s/.*Handshake Protocol: Server Hello/Server replied with:/g" |egrep -v "Server replied with: Done" &
[1] 91106


sqlplus system/Oracle123@hrdb_ssl

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Nov 6 10:54:52 2024
Version 19.25.0.0.0

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

ERROR:
ORA-28860: Fatal SSL error


Enter user-name: ^C
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
   Protocol Version: TLS 1.2 
   Cipher Suites Requested:(25 suites)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA 
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA 
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA 
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA 
                Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA 
                Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA 
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA 
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA 
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV 
Server replied with:
Server replied with:
   Protocol Version: TLS 1.2 
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
    TLSv1.2 Record Layer: Handshake Protocol: Certificate
    TLSv1.2 Record Layer: Handshake Protocol: Server Key Exchange
    TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
    TLSv1.2 Record Layer: Handshake Protocol: Certificate
    TLSv1.2 Record Layer: Handshake Protocol: Client Key Exchange
    TLSv1.2 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)
    TLSv1 Record Layer: Handshake Protocol: Client Hello
   Protocol Version: TLS 1.2 
   Cipher Suites Requested:(6 suites)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
                Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 
                Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 
    TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
Server replied with:
   Protocol Version: TLS 1.2 
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
    TLSv1.2 Record Layer: Handshake Protocol: Client Key Exchange
    TLSv1.2 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
    TLSv1.2 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
    TLSv1.2 Record Layer: Application Data Protocol: http-over-tls

2. The cryptographic cipher suite does not match in the $ORACLE_HOME/network/admin/sqlnet.ora of the database client and server.  For example, the SSL_CIPHER_SUITES in the following slqnet.ora on both client and server should iniclude TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 but one does not:

cat $ORACLE_HOME/network/admin/sqlnet.ora
SQLNET.AUTHENTICATION_SERVICES = (TCPS,NTS,BEQ)
SSL_CLIENT_AUTHENTICATION = FALSE
SSL_VERSION = 1.2
SSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA)

The wireshark/tshark analysis of this connection reveals that the client and server could not agree on the terms of the cryptographic handshake.

sudo tshark -i any -d "tcp.port==2484,ssl" -V -a duration:5 2> /dev/null | egrep "Cipher Suites \(|Cipher Suite:|^            Version: |SSL Record Layer: Handshake Protocol: Client Hello|Handshake Protocol: Server Hello|Record Layer"| uniq | sed -e "s/(0x.*)//g" -e "s/.*SSL Record Layer: Handshake Protocol: Client Hello/Client requested:/g" -e "s/.*Version:/   Protocol Version:/g" -e "s/.*Cipher Suites /   Cipher Suites Requested:/g" -e "s/.*Handshake Protocol: Server Hello/Server replied with:/g" |egrep -v "Server replied with: Done" &
[1] 92079

sqlplus system/Oracle123@hrdb_ssl

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Nov 6 11:16:21 2024
Version 19.25.0.0.0

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

ERROR:
ORA-28860: Fatal SSL error


Enter user-name: ^C
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
   Protocol Version: TLS 1.2 
   Cipher Suites Requested:(2 suites)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV 
Server replied with:
Server replied with:
   Protocol Version: TLS 1.2 
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
    TLSv1.2 Record Layer: Handshake Protocol: Certificate
    TLSv1.2 Record Layer: Handshake Protocol: Server Key Exchange
    TLSv1.2 Record Layer: Handshake Protocol: Client Key Exchange
    TLSv1.2 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
    TLSv1.2 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
    TLSv1.2 Record Layer: Application Data Protocol: Application Data
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)


3. The SSL_VERSION setting in the sqlnet.ora of the database client and server do not match.  For example, if SSL_VERSION is set to 1 rather than 1.2.  You would get very similar wireshark/tshark result as the previous example.


Error: ORA-12560: Database communication protocol error.
Reason: The SSL_VERSION setting in the sqlnet.ora of the database client and server do not match.  For example, if SSL_VERSION of the client sqlnet.ora is set to 1.3 when the server sqlnet.ora is set to 1.2.



Error: ORA-28865: SSL connection has closed
Reason: If you see this error when connecting securely to a PDB, this means that the wallet has not yet been copied to PDB wallet_root/tls directory. For example:

mkdir -p $WALLET_ROOT/26471EC7099D0530E0635D00000A3BC3/tls
cp $WALLET_ROOT/tls/ewallet.p12 $WALLET_ROOT/26471EC7099D0530E0635D00000A3BC3/tls
cp $WALLET_ROOT/tls/cwallet.sso $WALLET_ROOT/26471EC7099D0530E0635D00000A3BC3/tls

No comments: