OUD Logging Analytics

Comprehensive monitoring is one of the most critical aspects of running any on premises deployment or cloud service. Oracle Unified Directory (OUD) provides a wealth of log and metric data through several log publishers and cn=monitor metric data.  In the case of log data, OUD provides access, errors, audit, and other log data with rich information about the usage of the OUD directory server instances.  Oracle Cloud Infrastructure (OCI) offers two Observability & Management (O&M) scalable cloud services that can consume and make actionable the log and metric data from OUD instances running on premises or in any cloud.  The first of these cloud services, O&M Logging Analytics is covered in this blog post.

O&M Logging Analytics makes OUD log data actionable by consuming the log data, analyzing it, and providing customizable dashboards and alerting functions.  The architecture of this solution is composed of the Oracle management agent that runs on the OUD host and your Oracle cloud tenancy.  The Oracle management agent consumes the log data and publishes it into your Oracle cloud tenancy.  The following diagram illustrates the interactions between the Oracle management agent, the OUD instance, and the Oracle cloud tenancy.  Note that all communications between the Oracle management agent and your Oracle cloud tenancy is one way.  No connections are made from the Oracle cloud tenancy to the OUD host. This means that no in-bound ingress firewall rules are required for this solution. 

Once setup, the Oracle management agent begins publishing log data to your Oracle cloud tenancy, 4 OUD specific dashboards begin visualizing the collected data in near real time.  The first of these dashboards is the OUD Health (LA) dashboard.

This dashboard provides operational load and core insights that the operations team will want to see to track the operational load of the service, use for capacity planning, and use to assess the health of the service and instances at a glance.  You can click on any of the widgets in the dashboard to drill down into the data that is being visualized.  Further, you can make a copy of the widget and customize many aspects including how the data is visualized, expand or contract the scope of the data, and look at the underlying raw log data of any widget.

Note also that by default, the dashboard shows data for all OUD instances as an aggregated view. You can scope the view down to a specific OUD instance through the filter. To do this, click on the filter icon at the top of the page just to the right of the dashboard title, select the compartment, Log Group compartment where the data is stored, and then enter the OUD instance into the Entity field.

Once you've selected an instance, all of the dashboard data is reduced to just this instance's log data and the Log Entry Topology gets populated with the OUD instance.

The second dashboard is the OUD Log Analytics dashboard.  This is a composition of a wide variety of best practice analytical insights about the OUD instance's operational load.  Examples include:

• Operation and log volume for the specified time span
• Protocol distribution and cryptography
• Top users
• Top client IP addresses
• Operation distribution
• LDAP status code distribution
• eTime outliers and volume
• Un-indexed searches
• Error trends

Note that you can drill down into any widget to get more detail or adjust the view to get deeper insights into the data that you are viewing all the way down to viewing the raw log entries.

The third dashboard is the OUD Security dashboard.  This dashboard aims to identify potential security threat vectors such as weak cryptography client requests, a ranked order of top failed authentication attempts by user, anonymous clients, privileged user usage, and possible data exfiltration.

The fourth dashboard is the OUD diagnostics dashboard.  This dashboard is most commonly used to streamline the process of getting to root cause of potential and active issues.

The "OUD Issues over the last 48 hours" widget of the OUD Diagnostics dashboard can enable the operations team to identify and remediate issues before they impact end customers of the directory service.

The OUD Clusters widget identifies patterns, outliers, and potential issues from many millions of lines of log data across the entire service through proprietary machine learning algorithms.  The following view illustrates 3 outliers of the 78 distinct cluster patterns of potentially millions of lines of log data.

This view of the OUD Clusters widget illustrates 16 potential issues from potentially millions of lines of log data that are worth investigating.

Now that we've seen some of the insights to be gleaned from the dashboards and widgets, let's walk through the setup workflow.

Here is the setup workflow to begin monitoring your OUD deployments.

1. Setup our OCI tenancy if you don't already have one setup.  You can sign up for a free trial at https://www.oracle.com/cloud/free/

2. Login to your Oracle Cloud Console at https://cloud.oracle.com/

3. Create a compartment for monitoring and operations or administration or choose an existing compartment.  For this demonstration, we will use ocioperations as the compartment name that will contain the log data and dashboards.

4. Enable Logging Analytics and setup requisite Identity & Access Management (IAM) policy by navigating to the Logging Analytics Overview page (https://cloud.oracle.com/loganalytics/overview) and click on "Start Using Logging Analytics".  This workflow will Enable the Logging Analytics service, ad the requisite IAM policy in the root compartment and create a new log group named Default.
Navigate to OCI Console --> Management Agents --> Downloads and Keys

5. Deploy Management Agent to each OUD Host using the following steps

5a. Create an Agent key

Navigate to OCI Console --> Management Agents --> Downloads and Keys --> Create Key

Enter Key name MonitorOUD, select compartment ocioperations, check "Unlimited", and click "Create".

5b. Download Agent key to a local file named MonitorOUD.rsp

Navigate to OCI Console --> Management Agents --> Downloads and Keys

Right click three dots to far right side of MonitorOUD Key and click on "Download key to file".

5c. Edit downloaded agent key file MonitorOUD.rsp and uncomment the following two lines and save the file:

Service.plugin.logan.download=true

Service.plugin.appmgmt.download=true

5d. Download Management Agent

Click on "Agent for LINUX (X86_64)" ZIP file

5e. Download the latest release of JDK 8 from https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

5f. Setup target directory structure on target LINUX host and add OUD runtime user's group to the management agent user so that the management agent will have permissions to consume the OUD log data.

$ sudo mkdir -p /opt/ods/exporter/sw

$ sudo chown -R mgmt_agent:mgmt_agent /opt/ods/exporter

$ sudo usermod -a -G <oud_users_group> mgmt_agent

Note that if you have restricted the default OUD log permissions to only be accessible by the OUD runtime user, you can use setfacl to grant read permission to the management agent user:

$ sudo setfacl -Rd -m "u:<management_agent_user>:r" "<OUD_logs_dir>" 

 5g. Upload the JDK8, the management agent zip file and the management response file (MonitorOUD.rsp) to each target OUD host and put in /opt/ods/exporter/sw

5h. Extract the JDK and management agent software

$ cd /opt/ods/exporter/sw

$ tar -xf jdk-8u421-linux-x64.tar.gz

$ mv jdk1.8.0_421 jdk

$ unzip -qod mgmt_agent oracle.mgmt_agent.240724.1411.Linux-x86_64.zip

5i. Install management agent

$ cd /opt/ods/exporter/sw/mgmt_agent

$ sudo JAVA_HOME=/opt/ods/exporter/sw/jdk /opt/ods/exporter/sw/mgmt_agent/installer.sh /opt/ods/exporter/sw/MonitorOUD.rsp

6. Create an OUD entity for each OUD instance in the OCI console

Navigate to O&M Logging Analytics Administration page

     --> Menu --> Observability & Management --> Logging Analytics --> Administration

Click on "Create Entry", fill out the form and click "Create Entry"

 

7. Associate the OUD Access, Admin, and Errors logs to all of the OUD Entities.

7a. Navigate to O&M Logging Analytics Administration page

     --> Menu --> Observability & Management --> Logging Analytics --> Administration 

7b. Click on one of the OUD entries

7c. Click on "Add association" or "Add Data"

 

7d. Select all of the OUD entities and then click "Next"

7e. Click on the "Search by Name or Description" box and enter "Oracle Unified Directory" and then check the OUD Access, Admin, and Error logs and then click "Validate and configure log collection"

7f.  Wait for the validation process to complete.  Depending on the number of OUD entities to configure, it may take a few minutes.

At this point, basic Logging Analytics configuration for the OUD entities is complete.  Next, navigate to the Logging Analytics Dashboards and click on each of the OUD dashboards to view the data.  It may take a few minutes before data from all OUD instances shows up. 

Navigate to O&M Logging Analytics Dashboards page

     --> Menu --> Observability & Management --> Logging Analytics --> Dashboards

Out of the box OUD Logging Analytics Dashboards include:

• OUD Health (LA) - Basic health for operations team, management and capacity planning
• OUD Security - Security view of the same data
• OUD Log Analytics - Best practice analytics for identifying outliers and 
• OUD Diagnostics - Dig deeper into the data to streamline root cause analysis

Customizing IAM Policy

Some customers like to setup two unique sets of IAM policies for administrative and operational users.  The following example illustrates how to adjust the Logging Analytics IAM policy for this use case where omadmin and omuser are the two groups for these distinct roles.  There is a second set of the same roles for federated users in case that is how your IAM is setup.

General configuration of the O&M Logging IAM Policy is covered in the following documentation link at https://docs.oracle.com/en-us/iaas/logging-analytics/doc/enable-access-logging-analytics-and-its-resources.html.

Here are the sample custom Logging Analytics IAM Policies for configuring administrative and operational user roles.  Note that if your compartment name is different than ocioperations, you will need to adjust this policy example accordingly.

OCI IAM Policy Groups for Observability & Management:

• OCI IAM Management Agent Policy (omagent_policy) - Setup Mgmt Agent and collect metrics
• OCI IAM Log Purge Policy (ompurge_policy) - Enable setting and executing O&M Storage Purge Policy
• OCI IAM O&M Administrator Policy(omadmin_policy/omfedadmin_policy) - Load and edit log parsers/sources and manage dashboards
• OCI IAM O&M User Policy (omuser_policy/omfeduser_policy) - Enable users to view O&M Dashboards, Log Explorer, and Metrics Explorer

OCI IAM Policy Setup Workflow For O&M Admins and Users

1. Create OCI IAM Static Groups and add users to group

Instructions for local OCI IAM users/groups:

Create and populate O&M group for local OCI IAM users

Navigate to: OCI Console —> Identity & Security —> Groups —> Create Group

Name: omadmin

Description: O&M Local Administrators

—> Add local relevant OCI IAM users to this group

Name: omuser

Description: O&M Local Users

—> Add local relevant OCI IAM users to this group

Instructions for federated OCI IAM users/groups:

Create and populate O&M group for federated OCI IAM users

Navigate to: OCI Console —> Identity & Security —> Federation —> Federated Name

—> Oracle Identity Cloud Service Console —> Groups —> Create Group

Name: omfedadmin

Description: O&M Federated Administrators

—> Add local relevant OCI IAM users to this group

Name: omfeduser

Description: O&M Federated Users

—> Add local relevant OCI IAM users to this group

1. Create an OCI IAM Dynamic Groups

Navigate to: OCI Console —> Identity —> Dynamic Groups —> Create Dynamic Group

Observability & Management Management Agent Dynamic Group

Name: omagent

Matching Rule Policy: Instances that meet the criteria defined by any of the following matching rules:

Matching Rule:

ALL {resource.type='managementagent',resource.compartment.id='<compute_compartment1_ocid>'}

ALL {resource.type='managementagent',resource.compartment.id='<compute_compartment2_ocid>'}

Where compartment_ocid is the compartment containing the compute instances.

Observability & Management Log Purge Dynamic Group

Name: ompurge

Matching Rule Policy: Instances that meet the criteria defined by any of the following matching rules:

Matching Rule:

ALL {resource.type='loganalyticsscheduledtask', resource.compartment.id='<compartment_ocid>'}

or, alternatively, to allow purges on all compartments

ALL {resource.type='loganalyticsscheduledtask'}

1. Create OCI IAM policies for Observability Management

Navigate to: OCI Console —> Identity —> Policies —> Create Policy

Observability & Management Management Agent Policy

Name: omagent_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/management-agents/doc/perform-prerequisites-deploying-management-agents.html

Statements:

ALLOW DYNAMIC-GROUP omagent to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} IN COMPARTMENT ID <om_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO MANAGE management-agents IN COMPARTMENT ID <compute_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO USE METRICS IN COMPARTMENT ID <om_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO MANAGE management-agent-install-keys IN COMPARTMENT ID <compute_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO USE loganalytics-log-group in COMPARTMENT ID <om_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO USE loganalytics-collection-warning in COMPARTMENT ID <om_compartment_ocid>

Observability & Management Management Storage Purge Policy

Name: ompurge_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/logging-analytics/doc/manage-storage.html#GUID-DEAA0C68-9FB9-4441-9DCE-AEF6A358B6D5

Statements:

ALLOW DYNAMIC-GROUP ompurge to read compartments in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_STORAGE_PURGE} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_QUERY_VIEW} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ} in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-features-family in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-resources-family in tenancy

allow service loganalytics to READ loganalytics-features-family in tenancy

Observability & Management Admin Policy for local OCI IAM users

Name: omadmin_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omadmin to USE compartments in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-features-family in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-resources-family in tenancy

ALLOW GROUP omadmin to MANAGE management-dashboard-family in compartment ocioperations

ALLOW GROUP omadmin to MANAGE management-saved-search in compartment ocioperations

ALLOW GROUP omadmin to READ metrics in compartment ocioperations

ALLOW GROUP omadmin to MANAGE loganalytics-resources-family in compartment ocioperations

Observability & Management User Policy for local OCI IAM users

Name: omuser_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omuser to USE compartments in tenancy

ALLOW GROUP omuser to READ loganalytics-features-family in tenancy

ALLOW GROUP omuser to READ management-dashboard-family in compartment ocioperations

ALLOW GROUP omuser to READ management-saved-search in compartment ocioperations

ALLOW GROUP omuser to READ metrics in compartment ocioperations

ALLOW GROUP omuser to READ loganalytics-resources-family in compartment ocioperations

Observability & Management Admin Policy for federated OCI IAM users

Name: omfedadmin_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omfedadmin to USE compartments in tenancy

ALLOW GROUP omfedadmin to MANAGE loganalytics-features-family in tenancy

ALLOW GROUP omfedadmin to MANAGE loganalytics-resources-family in tenancy

ALLOW GROUP omfedadmin to MANAGE management-dashboard-family in compartment ocioperations

ALLOW GROUP omfedadmin to MANAGE management-saved-search in compartment ocioperations

ALLOW GROUP omfedadmin to READ metrics in compartment ocioperations

ALLOW GROUP omfedadmin to MANAGE loganalytics-resources-family in compartment ocioperations

Observability & Management User Policy for federated OCI IAM users

Name: omfeduser_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omfeduser to USE compartments in tenancy

ALLOW GROUP omfeduser to READ loganalytics-features-family in tenancy

ALLOW GROUP omfeduser to READ management-dashboard-family in compartment ocioperations

ALLOW GROUP omfeduser to READ management-saved-search in compartment ocioperations

ALLOW GROUP omfeduser to READ metrics in compartment ocioperations

ALLOW GROUP omfeduser to READ loganalytics-resources-family in compartment ocioperations

Bash 101: Relative Pathing

While preparing for an interview in 1992 for a co-operative education program at IBM to support their new RISC SYSTEM/6000 line of servers and it's version of the UNIX operating system, Advanced Interactive eXecutive (AIX), I was introduced to what would become one of my favorite past times, shell scripting.  At that time, the common shells were the Bourne shell (sh), Korn shell (ksh), and C shell (csh).  Later, Linux would introduce what is now the default shell of many UNIX distributions, the Bourne-Again shell (bash).

Though shell scripting is technically not a programming language, everyone that works in the UNIX domain becomes familiar with shell scripting at some level.  Over my 30+ year career in IT, I've easily contributed more than 10,000,000 lines of shell scripting code to the projects that I've worked on.  The purpose of this blog is to pass along some of the gems that I've collected along the way.

When working on projects that include multiple scripts, one of the biggest problems is how do you write all of the scripts such that they will work properly in any directory path. For example, my current project is composed of 84 different scripts.   The answer, is that you define a common path context for all of the scripts in the header of every script in the project.  My standard for this is the following:

#!/bin/bash
cmd=$(basename $0)
curdir=$(dirname $0)
curdir=$(cd ${curdir}; pwd)

Let's unpack this simple set of lines to understand what's going on.

1. Define the shell to use for this script's runtime environment via the shebang directive.

#!/bin/bash

2. Determine the script name.  The $0 variable is the execution path of the script being run.  For example, the execution path could have been to run the script in the current directory with ./myscript.sh.  Or, it could have been called from a directory above with ../myscript.sh.  Or, perhaps it might have been called using the full path of /opt/ods/poc/myscript.sh.  The basename command trims off the path to leave just the script name.  We assign the output of the basename command to the cmd variable so that it can be used as a common reference (${cmd}) throughout the rest of the script.

cmd=$(basename $0)

3. Determine the base directory based of the script's location.  The dirname command trims off the script name leaving just the path to the script.  We temporarily assign the result of the dirname output to the curdir in the first step to determining the current directory.

curdir=$(dirname $0)

4. Lastly, we change directory to the current value of ${curdir} and then use the present working directory (pwd) command to determine the fully qualified path of the directory that we are in.

curdir=$(cd ${curdir}; pwd)

You may be wondering why we didn't just stop at step 3 because technically, that is correctly the current directory.  The reason is because the execution of the script can be called from a variety of ways that are not the full path name.  For example, if I ran the script with ../../../myscript.sh, the first value of curdir would be  "../../../", which is not the fully qualified path.  However, after step 4, the fully qualified path is righly determined to be /opt/ods/poc.

I hope you find this helpful!

Blessings!

Brad

OUD 12cPS4 Use Case: Oracle Database Name Services

Oracle Unified Directory (OUD) is one directory service product included in the Oracle Directory Services Plus (ODS+) suite that is used for a wide variety of use cases requiring LDAPv3 interoperability.  

One very common use case is Oracle Database name resolution. Oracle name resolution is referred to by several names including "net services", Transparent Network Substrate (TNS), and Oracle Names (ONAMES) depending on the Oracle Database version.

Oracle Database name resolution is to Oracle Databases as Domain Name Services (DNS) is to resolving a fully qualified domain name to an IP address to enable a web browser or ssh client to connect to the associated IP address.  For more information on Oracle Database name resolution (a.k.a. Net Services), see the Oracle Database documentation here: https://docs.oracle.com/en/database/oracle/oracle-database/19/netag/

Basic Workflow

In this use case, Oracle Database clients connect anonymously [default] or via basic authentication to the OUD directory server over either the LDAP [default] or LDAPS protocol.  Once connected to OUD, the database client requests the client connect string for a specific database.  The connect string is returned to the database client and then the database client uses the provided connection string to connect to the Oracle Database.  The following is a sample connect string returned by OUD:

(DESCRIPTION =
  (ADDRESS=(PROTOCOL=TCP)(HOST=node1.example.com)(PORT=1521))
  (CONNECT_DATA=(SERVICE_NAME=node1.example.com))
)

Once the database client has the connect string, it uses the information in the connect string to connect to the Oracle Database.

OUD Setup

The following is the basic workflow for setting up a pair of OUD instances for this use case on the same server:

1. Make the directory where the software will be installed

$ sudo mkdir /opt/oud/bits
$ sudo chown -R opc:opc /opt/oud

2. Download JDK and OUD into /opt/oud/bits

• JDK is available at https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
• Download the latest Linux x64 gz version, which is currently jdk-8u381-linux-x64.tar.gz
• OUD is available from eDelivery at https://edelivery.oracle.com/
• Login and search for Oracle Directory Services Plus 12.2.1.4.0
• Download V983402-01.zip
• Patches
• OPatch: https://updates.oracle.com/download/28186730.html (p28186730_139427_Generic.zip)
• OUD: https://updates.oracle.com/download/35263333.html (p35263333_122140_Generic.zip)

3. Extract the JDK and OUD software and patches

$ tar --directory=/opt/oud -zxf /opt/oud/bits/jdk-8u*-linux-x64.tar.gz
$ mv /opt/oud/jdk1.8.0* /opt/oud/jdk
$ unzip -d /opt/oud/bits/. /opt/ods/poc/bits/V983402-01.zip
$ unzip -d /opt/oud/bits/. /opt/ods/poc/bits/p28186730_139427_Generic.zip
$ unzip -d /opt/oud/bits/. /opt/ods/poc/bits/p35263333_122140_Generic.zip

4. Create input files for OUD installation and Oracle inventory

OUD installation response file (/opt/oud/bits/oud12c-standalone.rsp):

[ENGINE]
Response File Version=1.0.0.0.0
[GENERIC]
DECLINE_AUTO_UPDATES=true
MOS_USERNAME=
MOS_PASSWORD=
AUTO_UPDATES_LOCATION=
SOFTWARE_UPDATES_PROXY_SERVER=
SOFTWARE_UPDATES_PROXY_PORT=
SOFTWARE_UPDATES_PROXY_USER=
SOFTWARE_UPDATES_PROXY_PASSWORD=
ORACLE_HOME=/opt/oud/mw_oud12c
INSTALL_TYPE=Standalone Oracle Unified Directory Server (Managed independently of WebLogic server)

Oracle inventory file (/opt/oud/bits/oraInventory.loc)

inventory_loc=/opt/oud/oraInventory
inst_group=opc

Temporary password file for setup (/opt/oud/bits/.pw)

$ echo 'Oracle123' > /opt/oud/bits/.pw

Java security configuration file (/opt/oud/bits/tns-java.security) that allows anonymous cipher suites, which are required for registering databases with dbca

jdk.disabled.namedCurves = secp112r1, secp112r2, secp128r1, secp128r2, \
    secp160k1, secp160r1, secp160r2, secp192k1, secp192r1, secp224k1, \
    secp224r1, secp256k1, sect113r1, sect113r2, sect131r1, sect131r2, \
    sect163k1, sect163r1, sect163r2, sect193r1, sect193r2, sect233k1, \
    sect233r1, sect239k1, sect283k1, sect283r1, sect409k1, sect409r1, \
    sect571k1, sect571r1, X9.62 c2tnb191v1, X9.62 c2tnb191v2, \
    X9.62 c2tnb191v3, X9.62 c2tnb239v1, X9.62 c2tnb239v2, X9.62 c2tnb239v3, \
    X9.62 c2tnb359v1, X9.62 c2tnb431r1, X9.62 prime192v2, X9.62 prime192v3, \
    X9.62 prime239v1, X9.62 prime239v2, X9.62 prime239v3, brainpoolP256r1, \
    brainpoolP320r1, brainpoolP384r1, brainpoolP512r1

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
    DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, NULL, \
    include jdk.disabled.namedCurves

jdk.tls.legacyAlgorithms= \
    K_NULL, C_NULL, M_NULL, \
    DH_anon, ECDH_anon, \
    RC4_128, RC4_40, DES_CBC, DES40_CBC, \
    3DES_EDE_CBC

5. Install the OUD software

$ export JAVA_HOME=/opt/oud/jdk
$ $JAVA_HOME/bin/java -d64 -jar /opt/oud/bits/fmw_12.2.1.4.0_oud.jar -silent -ignoreSysPrereqs -responseFile /opt/oud/bits/oud12c-standalone.rsp -invPtrLoc /opt/oud/bits/oraInventory.loc

6. Patch OUD to current bundle patch level

Set ORACLE_HOME environment variable

$ export ORACLE_HOME=/opt/oud/mw_oud12c

List patch inventory to see current OUD version and what patches are installed

$ $ORACLE_HOME/OPatch/opatch lsinventory

Install the OPatch patch

$ cd /opt/oud/bits/6880880
$ $JAVA_HOME/bin/java -jar /opt/oud/bits/6880880/opatch_generic.jar -silent oracle_home=$ORACLE_HOME

Install OUD patch responding interactively with y to both questions

cd /opt/oud/bits/35263333
$ $ORACLE_HOME/OPatch/opatch apply

List patch inventory to see compare with previous lsinventory output

$ $ORACLE_HOME/OPatch/opatch lsinventory

7. Set OPENDS_JAVA_ARGS environment variable so that when OUD instances start, they will use our custom tns-java.security configuration file rather than the default configuration file

$ export OPENDS_JAVA_ARGS="-Djava.security.properties=/opt/oud/bits/tns-java.security"

Note: Whenever you restart an OUD instance, you will want to set the OPENDS_JAVA_ARGS environment variable before starting the OUD instance.

8. Setup the first OUD instance

$ /opt/oud/mw_oud12c/oud/oud-setup --cli --integration eus --instancePath /opt/oud/mw_oud12c/oud1/OUD --adminConnectorPort 1444 --ldapPort 1389 --ldapsPort 1636 --httpAdminConnectorPort disabled --httpPort disabled --httpsPort disabled --baseDN dc=world --rootUserDN 'cn=Directory Manager' --rootUserPasswordFile /opt/oud/bits/.pw --addBaseEntry --generateSelfSignedCertificate --hostName $(hostname -f) --noPropertiesFile --no-prompt

9. Setup the second OUD instance

$ /opt/oud/mw_oud12c/oud/oud-setup --cli --integration eus --instancePath /opt/oud/mw_oud12c/oud2/OUD --adminConnectorPort 2444 --ldapPort 2389 --ldapsPort 2636 --httpAdminConnectorPort disabled --httpPort disabled --httpsPort disabled --baseDN dc=world --rootUserDN 'cn=Directory Manager' --rootUserPasswordFile /opt/oud/bits/.pw --addBaseEntry --generateSelfSignedCertificate --hostName $(hostname -f) --noPropertiesFile --no-prompt

10. Enable and initialize replication from OUD1 to OUD2

$ /opt/oud/mw_oud12c/oud2/OUD/bin/dsreplication enable --host1 $(hostname -f) --port1 1444 --bindDN1 'cn=Directory Manager' --bindPasswordFile1 /opt/oud/bits/.pw --replicationPort1 1989 --secureReplication1 --host2 $(hostname -f) --port2 2444 --bindDN2 'cn=Directory Manager' --bindPasswordFile2 /opt/oud/bits/.pw --replicationPort2 2989 --secureReplication2 --baseDN dc=world --adminUID admin --adminPasswordFile /opt/oud/bits/.pw --trustAll --no-prompt

11. Confirm that replication is working properly

$ /opt/oud/mw_oud12c/oud1/OUD/bin/dsreplication status --hostname $(hostname -f) --port 1444 --dataToDisplay compact-view --adminUID admin --adminPasswordFile /opt/oud/bits/.pw --advanced --trustAll --no-prompt

12. Configure OUD1 and OUD2 for TNS

Create /opt/oud/bits/tns.batch configuration file with the following content

set-connection-handler-prop --handler-name "LDAPS Connection Handler" --add ssl-cipher-suite:TLS_DH_anon_WITH_AES_256_GCM_SHA384 --add ssl-cipher-suite:TLS_DH_anon_WITH_AES_128_GCM_SHA256

create-password-policy --type generic --policy-name TNSAdmins --set password-attribute:userpassword --set default-password-storage-scheme:AES --set default-password-storage-scheme:Salted\ SHA-512

set-password-policy-prop --policy-name "Default Password Policy" --add default-password-storage-scheme:"EUS PBKDF2 SHA-512"

set-log-publisher-prop --publisher-name "File-Based Access Logger" --set enabled:true

Apply the eus.batch configuration file to OUD1

$ /opt/oud/mw_oud12c/oud1/OUD/bin/dsconfig -h $(hostname -f) -p 1444 -D 'cn=Directory Manager' -j /opt/oud/bits/.pw --batchFilePath /opt/oud/bits/tns.batch --no-prompt

Apply the eus.batch configuration file to OUD2

$ /opt/oud/mw_oud12c/oud1/OUD/bin/dsconfig -h $(hostname -f) -p 2444 -D 'cn=Directory Manager' -j /opt/oud/bits/.pw --batchFilePath /opt/oud/bits/tns.batch --no-prompt

Set OPENDS_JAVA_ARGS to use custom java.security and then restart OUD instances to apply cryptographic changes

$ export OPENDS_JAVA_ARGS="-Djava.security.properties=/opt/oud/bits/tns-java.security" 

Stop OUD1 and OUD2

$ /opt/oud/mw_oud12c/oud1/OUD/bin/stop-ds
$ /opt/oud/mw_oud12c/oud2/OUD/bin/stop-ds

Start OUD1 and OUD2

$ /opt/oud/mw_oud12c/oud1/OUD/bin/start-ds
$ /opt/oud/mw_oud12c/oud2/OUD/bin/start-ds

13. Create realm configuration to add TNS admin and grant privileges to manage database entries.  Here is a sample realm configuration file in LDIF format

dn: ou=TNSAdmins,cn=OracleContext
changetype: add
objectClass: top
objectClass: organizationalUnit
ou: TNSAdmins

dn: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext
changetype: add
objectClass: top
objectClass: organizationalperson
objectClass: inetorgperson
uid: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext
cn: tnsadmin
sn: TNS
givenName: Admin
userPassword: Oracle123
ds-privilege-name: password-reset
ds-privilege-name: unindexed-search
ds-privilege-name: modify-acl
ds-pwp-password-policy-dn: cn=TNSAdmins,cn=Password Policies,cn=config

dn: cn=Common,cn=Products,cn=OracleContext
changetype: modify
replace: orclSubscriberSearchBase
orclSubscriberSearchBase: dc=world

dn: cn=Common,cn=Products,cn=OracleContext
changetype: modify
replace: orclSubscriberNickNameAttribute
orclSubscriberNickNameAttribute: dc

dn: cn=Common,cn=Products,cn=OracleContext
changetype: modify
replace: orclDefaultSubscriber
orclDefaultSubscriber: dc=world

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonUserSearchBase
orclCommonUserSearchBase: ou=People,dc=world

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonUserCreateBase
orclCommonUserCreateBase: ou=People,dc=world

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonDefaultUserCreateBase
orclCommonDefaultUserCreateBase: ou=People,dc=world

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonGroupCreateBase
orclCommonGroupCreateBase: ou=Groups,dc=world

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonDefaultGroupCreateBase
orclCommonDefaultGroupCreateBase: ou=Groups,dc=world

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonGroupSearchBase
orclCommonGroupSearchBase: ou=Groups,dc=world

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonNicknameAttribute
orclCommonNicknameAttribute: uid

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonKrbPrincipalAttribute
orclCommonKrbPrincipalAttribute: userPrincipalName

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonNamingAttribute
orclCommonNamingAttribute: cn

dn: cn=Common,cn=Products,cn=OracleContext,dc=world
changetype: modify
replace: orclCommonWindowsPrincipalAttribute
orclCommonWindowsPrincipalAttribute: samAccountName

dn: cn=OracleContextAdmins,cn=groups,cn=OracleContext,dc=world
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=OracleDomainAdmins,cn=OracleDefaultDomain,cn=OracleDBSecurity,cn=Products,cn=OracleContext,dc=world
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=PolicyCreators,cn=Policies,cn=LabelSecurity,cn=Products,cn=OracleContext,dc=world
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=OracleDBCreators,cn=OracleContext
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=OracleNetAdmins,cn=OracleContext
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=OracleContextAdmins,cn=Groups,cn=OracleContext
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=OracleUserSecurityAdmins,cn=Groups,cn=OracleContext
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=OracleNetAdmins,cn=OracleContext,dc=world
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

dn: cn=OracleDBCreators,cn=OracleContext,dc=world
changetype: modify
add: uniqueMember
uniqueMember: cn=tnsadmin,ou=TNSAdmins,cn=oracleContext

14. Apply the realm configuration changes with ldapmodify

$ /opt/oud/mw_oud12c/oud/bin/ldapmodify -h $(hostname -f) -Z -X -p 1636 -D 'cn=Directory Manager' -j /opt/oud/bits/.pw -c -f /opt/oud/bits/tns.realm

Configure Database

1. Configure an Oracle database to register it into OUD by updating sqlnet.ora and adding ldap.ora

Sample sqlnet.ora illustrating the insertion of LDAP into the NAMES.DIRECTORY_PATH variable

NAMES.DIRECTORY_PATH= (LDAP,TNSNAMES, EZCONNECT)

Sample ldap.ora

DIRECTORY_SERVERS= (ol8.example.com:1389:1636,ol8.example.com:2389:2636)
DEFAULT_ADMIN_CONTEXT = "dc=world"
DIRECTORY_SERVER_TYPE = OID

2. Register Oracle database

Set Oracle database environment

$ export ORACLE_HOME=/opt/ods/poc/db/19c/dbhome_1
$ export ORACLE_BASE=/opt/ods/poc/db/19c/app/oracle
$ export TNS_ADMIN=/opt/ods/poc/db/19c/dbhome_1/network/admin
$ export ORACLE_SID=ol8

Register CDB database

$ /opt/ods/poc/db/19c/dbhome_1/bin/dbca -silent -configureDatabase -sourceDB $ORACLE_SID -registerWithDirService true -dirServiceUserName cn=tnsadmin,ou=TNSAdmins,cn=oracleContext -dirServicePassword Oracle123 -walletPassword Oracle123

Register PDB database

$ $ORACLE_HOME/bin/dbca -silent -configurePluggableDatabase -pdbName PDB1 -sourceDB $ORACLE_SID -registerWithDirService true -dirServiceUserName cn=tnsadmin,ou=TNSAdmins,cn=oracleContext -dirServicePassword Oracle123 -walletPassword Oracle123

3. Confirm registration with tnslookup

Lookup container database (CDB)

$ $ORACLE_HOME/bin/tnsping $ORACLE_SID

Lookup pluggable database (PDB)

$ $ORACLE_HOME/bin/tnsping PDB1_$ORACLE_SID

List all registered databases

$ /opt/oud/mw_oud12c/oud/bin/ldapsearch -T -h $(hostname -f) -p 1636 -X -Z -D cn=tnsadmin,ou=TNSAdmins,cn=oracleContext -j /opt/oud/bits/.pw -b dc=world -s sub '(|(objectClass=orclDBServer)(objectClass=orclNetService))'

I hope that you found this helpful!

Blessings!

Brad

OUD Feature Highlight: EUS/CMU Password Storage Scheme

Beginning with Oracle Unified Directory (OUD) 12c Patch Set 4, Oracle began adding new features and functionality along with bug fixes with each bundle patch release.  The What's New section of the documentation covers new features as they are added with each bundle patch release.

One such feature enhancement introduced a new password storage scheme that is used by Enterprise User Security (EUS) and Centrally managed Users (CMU) architectures for password based user authentication to the Oracle database. 

This new password scheme is a proprietary blend of multiple rounds of PBKDF2 SHA-512, which is much stronger than the storage scheme used for earlier Oracle database versions (e.g. 10g and 11g).  The full list of password storage schemes offered by OUD 12cPS4 are available here.

With the EUS architecture configuration where OUD is the identity store for authentication, identity management solutions like Oracle Identity Manager, SailPoint and others can simply update the password using normal password update and OUD password policy will automatically generate this password hash for database authentication.

With the EUS or CMU architectures where Active Directory (AD) is the identity store, the individual user's orclCommonAttribute value needs to be updated with this new hash in order for password based authentication to work properly.

The standard method of updating the user's orclCommonAttribute attribute value is through the deployment of the password filter to all AD domain controllers.  When a user updates their password with Ctrl-Alt-Delete feature of Windows, the Oracle's password filter (orapwdfltr.dll) captures the clear text password that was entered by the user, hashes the password, and stores the hashed value into the orclCommonAttribute attribute of the user's AD object.  See Doc ID 2640135.1 for more information on how to obtain and deploy the latest version of this password filter.

There is an alternative approach to populating the user's orclCommonAttribute in AD that has the same end result but does not require the password filter.  You can use the OUD encode-password command to generate the hashed value of the password and then update orclCommonAttribute in the user's AD entry.  This approach could be dove tailed into your provisioning solution as well.  Here is a sample workflow:

1. Install OUD.  

Note: If integrating with Identity Management solution, OUD would most likely need to be installed on the host(s) where the Identity Management solution is running in order to securely handle the password.

2. Use the OUD encode-password command to generated the hash of the user's password.

$ export JAVA_HOME=/opt/ods/poc/sw/jdk1.8.0_381
$ mkdir /opt/ods/poc/mw_oud12c/oud/locks
$ echo Johns-N3w_P4ssw0rd > /opt/ods/poc/cfg/...pw
$ /opt/ods/poc/mw_oud12c/oud/bin/encode-password -s MR-SHA512 -f /opt/ods/poc/cfg/...pw
Encoded Password: "{MR-SHA512}vGhKrQ39OEvg9vSOCJzm1TA/Eues1RN37ra+1rOuf6hfAPTFLd00CIVihRZ279OXNYbIEl2G/bjdaqOKxnuaye6rVgZcbdAjSZ9CTweXihU="

3. Use ldapmodify or equivalent API to update the orclCommonAttribute of the user's entry in AD.

$ /opt/ods/poc/mw_oud12c/oud/bin/ldapmodify -h msad.example.com -p 636 -X -Z -D 'cn=Administrator,cn=Users,dc=example,dc=com' -j /opt/ods/poc/cfg/...pw <<EOF
dn: cn=John Doe,cn=Users,dc=example,dc=com
changeType: modify
replace: orclCommonAttribute
orclCommonAttribute: {MR-SHA512}vGhKrQ39OEvg9vSOCJzm1TA/Eues1RN37ra+1rOuf6hfAPTFLd00CIVihRZ279OXNYbIEl2G/bjdaqOKxnuaye6rVgZcbdAjSZ9CTweXihU=
EOF
Processing MODIFY request for cn=John Doe,cn=Users,dc=example,dc=com
MODIFY operation successful for DN cn=John Doe,cn=Users,dc=example,dc=com

4. Test that the authentication works with the updated password.

$ORACLE_HOME/bin/sqlplus jdoe/Johns-N3w_P4ssw0rd@$ORACLE_SID

I hope that you found this helpful!

Blessings!

Brad

EUS: Troubleshooting ORA-01017

Enterprise User Security (EUS) is one of the architectures available for centralizing authentication, authorization, and user/password lifecycle management or Oracle Database users.  One Oracle database error that customers often encounter as they begin evaluating the EUS architecture is the following:

ORA-01017: invalid username/password; logon denied

This error can be especially frustrating because there are a variety of possible causes.

Here are some common causes and corresponding solutions and troubleshooting techniques:

• The wallet wallet containing the ORACLE.SECURITY.DN and ORACLE.SECURITY.PASSWORD entries does not exist

$ ls -al $ORACLE_BASE/admin/$ORACLE_SID/wallet

• The wallet containing the ORACLE.SECURITY.DN and ORACLE.SECURITY.PASSWORD exists but is empty or has missing or incorrect values including case sensitive passwords.  To troubleshoot, retrieve the values from the wallet with:

$ $ORACLE_HOME/bin/mkstore -wrl $ORACLE_BASE/admin/$ORACLE_SID/wallet -list -viewEntry ORACLE.SECURITY.DN -viewEntry ORACLE.SECURITY.PASSWORD <<EOF
YourWalletPassword
EOF

• If Oracle database was upgraded from earlier version to 18c or newer, the mappings may need to be re-created. See Doc ID 2611300.1

• The EUS configuration (e.g. Sample in /<oud_install/oud/config/EUS/modifyRealm.ldif) has not yet been applied or is mis-configured.  See Doc ID 2118421.1

• The Certificate Authority (CA) certificate chain or OUD self-signed certificate is not loaded into the wallet.  To troubleshoot this issue, confirm the presence of the certificate in the wallet with:

$ORACLE_HOME/bin/orapki wallet display -wallet $ORACLE_BASE/admin/$ORACLE_SID/wallet -pwd YourWalletPassword

• Encoded Kerberos ticket is too large. See Doc ID 1956558.1

• Database start fails with ORA-01017. In this case grid user's group needs to be a member of the  OSRACDBA group.  See Doc ID 2313555.1

• Get ORA-01017 with RAC database.  This can be caused by the having inconsistent wallets on each RAC node or by using the same wallet via NFS share on all three nodes but where auto_login only works for the node on which it was set.

• May have specified the wrong ORACLE_SID environment variable value and the authentication fails because you are attempting to connect to the wrong database.

• If using tnsnames.ora, the connect string may be pointing to the wrong database for which the user or user/password combination are not valid.

When troubleshooting error ORA-01017 from the database perspective, you will want to enable tracing to determine the reason for the authentication failure.

  

Step 1: Enable Oracle database tracing by with:

$ $ORACLE_HOME/bin/sqlplus / as sysdba
SQL> alter system set events '28033 trace name context forever, level 9';

Step 2: Perform authentication attempt that fails with ORA-01017     

$ $ORACLE_HOME/bin/sqlplus -S -L <eus_user_id>/YourUsersPassword@$ORACLE_SID
ERROR: ORA-01017: invalid username/password; logon denied

Step 3: Disable Oracle database tracing with:     

$ $ORACLE_HOME/bin/sqlplus / as sysdba
SQL> alter system set events '28033 trace name context off';

Step 4: Lookup the path of the trace files (in case they aren't in default location):

$ $ORACLE_HOME/bin/sqlplus / as sysdba
SQL> sho param dbug;

Step 5: Review trace file looking for KZLD_ERR messages

  

When troubleshooting error ORA-01017 from the directory service perspective, you will want to review the directory service logs.  In the case of Oracle Unified Directory (OUD), you will want to review the /<oud_instance>/OUD/logs/access or /<oud_instance>/OUD/logs/access.log log file depending on which logger is enabled.  Things to look for include:

• Authentication attempt by <eus_user_id> fails because user does not exist (err=32)
• Authentication attempt by <eus_user_id> fails because the wrong password is used (err=49)
• Connection to the OUD instances fails because of inability to  come to agreement on the LDAPS cryptographic negotiation.  Typically see error "no cipher suites in common". See Doc ID 2397791.1 for OUD 12c and Doc ID 2304757.1 for OUD 11g.  Note that this can happen if you've upgraded the JDK 8 to a version that has deprecated use of anonymous and NULL cipher suites.  In this case, you will need to update the jre/lib/security/java.security of the JDK implementation used by OUD to remove anon from jdk.tls.disabledAlgorithms.  Here is a sample java.security for jdk1.8.0_361:

jdk.disabled.namedCurves = secp112r1, secp112r2, secp128r1, secp128r2, \

    secp160k1, secp160r1, secp160r2, secp192k1, secp192r1, secp224k1, \

    secp224r1, secp256k1, sect113r1, sect113r2, sect131r1, sect131r2, \

    sect163k1, sect163r1, sect163r2, sect193r1, sect193r2, sect233k1, \

    sect233r1, sect239k1, sect283k1, sect283r1, sect409k1, sect409r1, \

    sect571k1, sect571r1, X9.62 c2tnb191v1, X9.62 c2tnb191v2, \

    X9.62 c2tnb191v3, X9.62 c2tnb239v1, X9.62 c2tnb239v2, X9.62 c2tnb239v3, \

    X9.62 c2tnb359v1, X9.62 c2tnb431r1, X9.62 prime192v2, X9.62 prime192v3, \

    X9.62 prime239v1, X9.62 prime239v2, X9.62 prime239v3, brainpoolP256r1, \

    brainpoolP320r1, brainpoolP384r1, brainpoolP512r1

jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \

    DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, NULL, \

    include jdk.disabled.namedCurves

jdk.tls.legacyAlgorithms= \

        K_NULL, C_NULL, M_NULL, \

        DH_anon, ECDH_anon, \

        RC4_128, RC4_40, DES_CBC, DES40_CBC, \

        3DES_EDE_CBC

I hope you find this helpful.

Blessigns!

Brad

OUD Feature Highlight: Connection Details

Beginning with Oracle Unified Directory (OUD) 12c Patch Set 4, Oracle began adding new features and functionality along with bug fixes with each bundle patch release.  The What's New section of the documentation covers new features as they are added with each bundle patch release.

One such feature enhancement introduced connection details to each of the OUD log publishers.  When enabled, this enhancement tags each operation in the log file with the following additional details:

• Authentication Distinguished Name:  bindDN=<user_dn>
• Protocol: protocol=<LDAP | LDAPS | HTTP>
• Client: client=<source_ip>:<source_port>
• Server: server=<destination_ip>:<destination_port>
• Cryptographic Protocol: protocol=<TLSv1.3 | TLSv1.2 | TLSv1.1 | ...>
• Cryptographic Cipher Suite: cipherSuite=<TLS_AES_128_GCM_SHA256 | ...>

These additional details can enable logging analytics tools like Oracle Cloud Infrastructure Logging Analytics to provide deep insights into the use and security posture of the OUD service to help you to strengthen security posture, identify out-of-date clients, identify root cause of cryptographic communication breakdowns, and even identify potential threat actors.  

Here are some of the questions that each of these additional details can enable you to answer:

• Authentication Distinguished Name
• What operations did James perform on the directory server over the past 48 hours?
• What users added new entries over the past 90 days?
• Are any clients connecting anonymously to OUD?

• Protocol
• What clients or users are connecting to OUD via non-encrypted LDAP?
• What clients are connecting via REST/SCIM?

• Client
• What is the volume of load per client IP address?
• From what client IP addresses were write operations performed?
• From what client IP addresses where anonymous authentications performed?

• Server
• Is the distribution of load even across servers in the load balanced pool?
• What OUD instances receive write operations?
• Which OUD instances are processing un-indexed searches or other abusive loads?
• Which OUD instances are receiving non-encrypted LDAP connections?

• Cryptographic Protocol
• Which users are requesting weak cryptographic protocols like SSLv3?
• What is the distribution of cryptographic protocols handled by the OUD service?
• Based on client load, are we to disable weak cryptographic protocols?
• Which clients need to be patched or updated to use strong cryptography?
• Which clients need their trust store updated with the latest certificate authority certificate chain or perhaps need the updated self-signed certificates?

• Cryptographic Cipher Suite
• Which users are using anonymous or weak cipher suites when connecting to OUD?
• What is the distribution of cipher suites being used by clients?
• Based on client load, are we to disable weak cryptographic cipher suites?

Enabling these additional connection details is very straight forward and can be enabled via the dsconfig command line (interactively, non-interactively, and in batch) tool or the web-based administrative console (Oracle Unified Directory Services Manager).  

Here is a sample non-interactive dsconfig command that for enabling connection details to the File-based Access logger:

dsconfig set-log-publisher-prop --publisher-name 'File-Based Access Logger --set log-connection-details:true --hostname $(hostname -f) --port 1444 --bindDN 'cn=Directory Manager' --bindPasswordFile /opt/ods/poc/cfg/...pw --trustAll --no-prompt

Here is the equivalent batch configuration entry: 

set-log-publisher-prop --publisher-name 'File-Based Access Logger --set log-connection-details:true 

Here is a an access log excerpt with connection details disabled:

[16/Oct/2023:15:48:10 +0000] CONNECT conn=42 from=10.20.0.104:59560 to=10.20.0.104:1636 protocol=LDAPS
[16/Oct/2023:15:48:10 +0000] BIND REQ conn=42 op=0 msgID=1 type=SIMPLE dn="uid=admin1,ou=Admins,dc=example,dc=com" version=3
[16/Oct/2023:15:48:10 +0000] BIND RES conn=42 op=0 msgID=1 result=0 authDN="uid=admin1,ou=Admins,dc=example,dc=com" etime=1
[16/Oct/2023:15:48:10 +0000] SEARCH REQ conn=42 op=1 msgID=2 base="dc=example,dc=com" scope=sub filter="(uid=user100001)" attrs="dn"
[16/Oct/2023:15:48:10 +0000] SEARCH RES conn=42 op=1 msgID=2 result=0 nentries=1 etime=14
[16/Oct/2023:15:48:10 +0000] UNBIND REQ conn=42 op=2 msgID=3
[16/Oct/2023:15:48:10 +0000] DISCONNECT conn=42 reason="Client Disconnect"

Here is a an access log excerpt with connection details enabled:

[16/Oct/2023:15:49:41 +0000] CONNECT conn=45 from=10.20.0.104:58190 to=10.20.0.104:1636 protocol=LDAPS
[16/Oct/2023:15:49:41 +0000] CONNECT CONN_DETAILS conn=45 tlsVersion=TLSv1.3 cipherSuite=TLS_AES_128_GCM_SHA256
[16/Oct/2023:15:49:41 +0000] BIND REQ conn=45 op=0 msgID=1 client=10.20.0.104:58190 server=10.20.0.104:1636 protocol=LDAPS type=SIMPLE dn="uid=admin1,ou=Admins,dc=example,dc=com" version=3
[16/Oct/2023:15:49:41 +0000] BIND RES conn=45 op=0 msgID=1 client=10.20.0.104:58190 server=10.20.0.104:1636 protocol=LDAPS bindDN=uid=admin1,ou=Admins,dc=example,dc=com result=0 authDN="uid=admin1,ou=Admins,dc=example,dc=com" etime=0
[16/Oct/2023:15:49:41 +0000] SEARCH REQ conn=45 op=1 msgID=2 client=10.20.0.104:58190 server=10.20.0.104:1636 protocol=LDAPS bindDN=uid=admin1,ou=Admins,dc=example,dc=com base="dc=example,dc=com" scope=sub filter="(uid=user100001)" attrs="dn"
[16/Oct/2023:15:49:41 +0000] SEARCH RES conn=45 op=1 msgID=2 client=10.20.0.104:58190 server=10.20.0.104:1636 protocol=LDAPS bindDN=uid=admin1,ou=Admins,dc=example,dc=com result=0
nentries=1 etime=1
[16/Oct/2023:15:49:41 +0000] UNBIND REQ conn=45 op=2 msgID=3 client=10.20.0.104:58190 server=10.20.0.104:1636 protocol=LDAPS bindDN=uid=admin1,ou=Admins,dc=example,dc=com
[16/Oct/2023:15:49:41 +0000] DISCONNECT conn=45 reason="Client Disconnect"

I hope you find this helpful.

Blessings!

Brad

TIME_WAIT: Mitigating TCP/IP Connection Exhaustion

TIME_WAIT is an incredible part of the TCP/IP stack that enables connections to linger until the client properly closes the connection. However, in some cases, the client does not close the connection properly or efficiently. This can result in TCP connections in the TIME_WAIT state to persist until the operating system purges them. For a busy system, this can result in a denial of service on the host because all available connections get tied up in the TIME_WAIT state.

Having worked with many large-scale and high-performance systems through the years, I've seen this scenario play out many times. Fortunately, each operating system has its own way of optimizing for this scenario to minimize the impact.

To determine if this is a problem on your host, track the number of connections in the TIME_WAIT state. For example, on UNIX/Linux and MacOS systems, you can count these connections with netstat:

$ netstat -an|grep -c TIME_WAIT

45564

Here are TCP tunings per operating system that I have used to mitigate this issue:

RedHat/Oracle Linux 8: /etc/sysctl.conf

net.netfilter.nf_conntrack_tcp_timeout_time_wait=1

net.ipv4.tcp_tw_reuse=1

net.ipv4.tcp_fin_timeout=1

RedHat/Oracle Linux 7: /etc/sysctl.conf

net.netfilter.nf_conntrack_tcp_timeout_time_wait=1

net.ipv4.tcp_tw_reuse=1

net.ipv4.tcp_fin_timeout=1

Solaris 10/11:

ndd -set /dev/tcp tcp_time_wait_interval 30000

Windows:

Reduce TcpTimedWaitDelay from the default of 2 minutes (120 seconds) down to around 20 seconds

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

—> TcpTimedWaitDelay": dword:00000028

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

—> StrictTimeWaitSeqCheck: dword:00000001

Notes for Windows:

 · Changing these values requires a reboot. Plan to do that out of your production hours.

 · TcpTimedWaitDelay is 2 minutes by default, even if the value is not present in the registry.

 · You must set the StrictTimeWaitSeqCheck to 0x1 or the TcpTimedWaitDelay value will have no effect.

While changing this parameter the following important points needs to be considered:

Changing these values requires a reboot. Plan to do that out of your production hours.

TcpTimedWaitDelay is 2 minutes by default, even if the value is not present in the registry.

You must set the StrictTimeWaitSeqCheck to 0x1 or the TcpTimedWaitDelay value will have no effect.

References:

TcpTimedWaitDelay - https://technet.microsoft.com/en-us/library/cc938217.aspx

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc731521(v=ws.10)#BKMK_setdynamicportrange

https://support.microsoft.com/en-in/help/929851/the-default-dynamic-port-range-for-tcp-ip-has-changed-in-windows