OUD Logging Analytics

Comprehensive monitoring is one of the most critical aspects of running any on premises deployment or cloud service. Oracle Unified Directory (OUD) provides a wealth of log and metric data through several log publishers and cn=monitor metric data.  In the case of log data, OUD provides access, errors, audit, and other log data with rich information about the usage of the OUD directory server instances.  Oracle Cloud Infrastructure (OCI) offers two Observability & Management (O&M) scalable cloud services that can consume and make actionable the log and metric data from OUD instances running on premises or in any cloud.  The first of these cloud services, O&M Logging Analytics is covered in this blog post.

O&M Logging Analytics makes OUD log data actionable by consuming the log data, analyzing it, and providing customizable dashboards and alerting functions.  The architecture of this solution is composed of the Oracle management agent that runs on the OUD host and your Oracle cloud tenancy.  The Oracle management agent consumes the log data and publishes it into your Oracle cloud tenancy.  The following diagram illustrates the interactions between the Oracle management agent, the OUD instance, and the Oracle cloud tenancy.  Note that all communications between the Oracle management agent and your Oracle cloud tenancy is one way.  No connections are made from the Oracle cloud tenancy to the OUD host. This means that no in-bound ingress firewall rules are required for this solution. 

Once setup, the Oracle management agent begins publishing log data to your Oracle cloud tenancy, 4 OUD specific dashboards begin visualizing the collected data in near real time.  The first of these dashboards is the OUD Health (LA) dashboard.

This dashboard provides operational load and core insights that the operations team will want to see to track the operational load of the service, use for capacity planning, and use to assess the health of the service and instances at a glance.  You can click on any of the widgets in the dashboard to drill down into the data that is being visualized.  Further, you can make a copy of the widget and customize many aspects including how the data is visualized, expand or contract the scope of the data, and look at the underlying raw log data of any widget.

Note also that by default, the dashboard shows data for all OUD instances as an aggregated view. You can scope the view down to a specific OUD instance through the filter. To do this, click on the filter icon at the top of the page just to the right of the dashboard title, select the compartment, Log Group compartment where the data is stored, and then enter the OUD instance into the Entity field.

Once you've selected an instance, all of the dashboard data is reduced to just this instance's log data and the Log Entry Topology gets populated with the OUD instance.

The second dashboard is the OUD Log Analytics dashboard.  This is a composition of a wide variety of best practice analytical insights about the OUD instance's operational load.  Examples include:

• Operation and log volume for the specified time span
• Protocol distribution and cryptography
• Top users
• Top client IP addresses
• Operation distribution
• LDAP status code distribution
• eTime outliers and volume
• Un-indexed searches
• Error trends

Note that you can drill down into any widget to get more detail or adjust the view to get deeper insights into the data that you are viewing all the way down to viewing the raw log entries.

The third dashboard is the OUD Security dashboard.  This dashboard aims to identify potential security threat vectors such as weak cryptography client requests, a ranked order of top failed authentication attempts by user, anonymous clients, privileged user usage, and possible data exfiltration.

The fourth dashboard is the OUD diagnostics dashboard.  This dashboard is most commonly used to streamline the process of getting to root cause of potential and active issues.

The "OUD Issues over the last 48 hours" widget of the OUD Diagnostics dashboard can enable the operations team to identify and remediate issues before they impact end customers of the directory service.

The OUD Clusters widget identifies patterns, outliers, and potential issues from many millions of lines of log data across the entire service through proprietary machine learning algorithms.  The following view illustrates 3 outliers of the 78 distinct cluster patterns of potentially millions of lines of log data.

This view of the OUD Clusters widget illustrates 16 potential issues from potentially millions of lines of log data that are worth investigating.

Now that we've seen some of the insights to be gleaned from the dashboards and widgets, let's walk through the setup workflow.

Here is the setup workflow to begin monitoring your OUD deployments.

1. Setup our OCI tenancy if you don't already have one setup.  You can sign up for a free trial at https://www.oracle.com/cloud/free/

2. Login to your Oracle Cloud Console at https://cloud.oracle.com/

3. Create a compartment for monitoring and operations or administration or choose an existing compartment.  For this demonstration, we will use ocioperations as the compartment name that will contain the log data and dashboards.

4. Enable Logging Analytics and setup requisite Identity & Access Management (IAM) policy by navigating to the Logging Analytics Overview page (https://cloud.oracle.com/loganalytics/overview) and click on "Start Using Logging Analytics".  This workflow will Enable the Logging Analytics service, ad the requisite IAM policy in the root compartment and create a new log group named Default.
Navigate to OCI Console --> Management Agents --> Downloads and Keys

5. Deploy Management Agent to each OUD Host using the following steps

5a. Create an Agent key

Navigate to OCI Console --> Management Agents --> Downloads and Keys --> Create Key

Enter Key name MonitorOUD, select compartment ocioperations, check "Unlimited", and click "Create".

5b. Download Agent key to a local file named MonitorOUD.rsp

Navigate to OCI Console --> Management Agents --> Downloads and Keys

Right click three dots to far right side of MonitorOUD Key and click on "Download key to file".

5c. Edit downloaded agent key file MonitorOUD.rsp and uncomment the following two lines and save the file:

Service.plugin.logan.download=true

Service.plugin.appmgmt.download=true

5d. Download Management Agent

Click on "Agent for LINUX (X86_64)" ZIP file

5e. Download the latest release of JDK 8 from https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

5f. Setup target directory structure on target LINUX host and add OUD runtime user's group to the management agent user so that the management agent will have permissions to consume the OUD log data.

$ sudo mkdir -p /opt/ods/exporter/sw

$ sudo chown -R mgmt_agent:mgmt_agent /opt/ods/exporter

$ sudo usermod -a -G <oud_users_group> mgmt_agent

Note that if you have restricted the default OUD log permissions to only be accessible by the OUD runtime user, you can use setfacl to grant read permission to the management agent user:

$ sudo setfacl -Rd -m "u:<management_agent_user>:r" "<OUD_logs_dir>" 

 5g. Upload the JDK8, the management agent zip file and the management response file (MonitorOUD.rsp) to each target OUD host and put in /opt/ods/exporter/sw

5h. Extract the JDK and management agent software

$ cd /opt/ods/exporter/sw

$ tar -xf jdk-8u421-linux-x64.tar.gz

$ mv jdk1.8.0_421 jdk

$ unzip -qod mgmt_agent oracle.mgmt_agent.240724.1411.Linux-x86_64.zip

5i. Install management agent

$ cd /opt/ods/exporter/sw/mgmt_agent

$ sudo JAVA_HOME=/opt/ods/exporter/sw/jdk /opt/ods/exporter/sw/mgmt_agent/installer.sh /opt/ods/exporter/sw/MonitorOUD.rsp

6. Create an OUD entity for each OUD instance in the OCI console

Navigate to O&M Logging Analytics Administration page

     --> Menu --> Observability & Management --> Logging Analytics --> Administration

Click on "Create Entry", fill out the form and click "Create Entry"

 

7. Associate the OUD Access, Admin, and Errors logs to all of the OUD Entities.

7a. Navigate to O&M Logging Analytics Administration page

     --> Menu --> Observability & Management --> Logging Analytics --> Administration 

7b. Click on one of the OUD entries

7c. Click on "Add association" or "Add Data"

 

7d. Select all of the OUD entities and then click "Next"

7e. Click on the "Search by Name or Description" box and enter "Oracle Unified Directory" and then check the OUD Access, Admin, and Error logs and then click "Validate and configure log collection"

7f.  Wait for the validation process to complete.  Depending on the number of OUD entities to configure, it may take a few minutes.

At this point, basic Logging Analytics configuration for the OUD entities is complete.  Next, navigate to the Logging Analytics Dashboards and click on each of the OUD dashboards to view the data.  It may take a few minutes before data from all OUD instances shows up. 

Navigate to O&M Logging Analytics Dashboards page

     --> Menu --> Observability & Management --> Logging Analytics --> Dashboards

Out of the box OUD Logging Analytics Dashboards include:

• OUD Health (LA) - Basic health for operations team, management and capacity planning
• OUD Security - Security view of the same data
• OUD Log Analytics - Best practice analytics for identifying outliers and 
• OUD Diagnostics - Dig deeper into the data to streamline root cause analysis

Customizing IAM Policy

Some customers like to setup two unique sets of IAM policies for administrative and operational users.  The following example illustrates how to adjust the Logging Analytics IAM policy for this use case where omadmin and omuser are the two groups for these distinct roles.  There is a second set of the same roles for federated users in case that is how your IAM is setup.

General configuration of the O&M Logging IAM Policy is covered in the following documentation link at https://docs.oracle.com/en-us/iaas/logging-analytics/doc/enable-access-logging-analytics-and-its-resources.html.

Here are the sample custom Logging Analytics IAM Policies for configuring administrative and operational user roles.  Note that if your compartment name is different than ocioperations, you will need to adjust this policy example accordingly.

OCI IAM Policy Groups for Observability & Management:

• OCI IAM Management Agent Policy (omagent_policy) - Setup Mgmt Agent and collect metrics
• OCI IAM Log Purge Policy (ompurge_policy) - Enable setting and executing O&M Storage Purge Policy
• OCI IAM O&M Administrator Policy(omadmin_policy/omfedadmin_policy) - Load and edit log parsers/sources and manage dashboards
• OCI IAM O&M User Policy (omuser_policy/omfeduser_policy) - Enable users to view O&M Dashboards, Log Explorer, and Metrics Explorer

OCI IAM Policy Setup Workflow For O&M Admins and Users

1. Create OCI IAM Static Groups and add users to group

Instructions for local OCI IAM users/groups:

Create and populate O&M group for local OCI IAM users

Navigate to: OCI Console —> Identity & Security —> Groups —> Create Group

Name: omadmin

Description: O&M Local Administrators

—> Add local relevant OCI IAM users to this group

Name: omuser

Description: O&M Local Users

—> Add local relevant OCI IAM users to this group

Instructions for federated OCI IAM users/groups:

Create and populate O&M group for federated OCI IAM users

Navigate to: OCI Console —> Identity & Security —> Federation —> Federated Name

—> Oracle Identity Cloud Service Console —> Groups —> Create Group

Name: omfedadmin

Description: O&M Federated Administrators

—> Add local relevant OCI IAM users to this group

Name: omfeduser

Description: O&M Federated Users

—> Add local relevant OCI IAM users to this group

1. Create an OCI IAM Dynamic Groups

Navigate to: OCI Console —> Identity —> Dynamic Groups —> Create Dynamic Group

Observability & Management Management Agent Dynamic Group

Name: omagent

Matching Rule Policy: Instances that meet the criteria defined by any of the following matching rules:

Matching Rule:

ALL {resource.type='managementagent',resource.compartment.id='<compute_compartment1_ocid>'}

ALL {resource.type='managementagent',resource.compartment.id='<compute_compartment2_ocid>'}

Where compartment_ocid is the compartment containing the compute instances.

Observability & Management Log Purge Dynamic Group

Name: ompurge

Matching Rule Policy: Instances that meet the criteria defined by any of the following matching rules:

Matching Rule:

ALL {resource.type='loganalyticsscheduledtask', resource.compartment.id='<compartment_ocid>'}

or, alternatively, to allow purges on all compartments

ALL {resource.type='loganalyticsscheduledtask'}

1. Create OCI IAM policies for Observability Management

Navigate to: OCI Console —> Identity —> Policies —> Create Policy

Observability & Management Management Agent Policy

Name: omagent_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/management-agents/doc/perform-prerequisites-deploying-management-agents.html

Statements:

ALLOW DYNAMIC-GROUP omagent to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} IN COMPARTMENT ID <om_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO MANAGE management-agents IN COMPARTMENT ID <compute_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO USE METRICS IN COMPARTMENT ID <om_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO MANAGE management-agent-install-keys IN COMPARTMENT ID <compute_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO USE loganalytics-log-group in COMPARTMENT ID <om_compartment_ocid>

ALLOW DYNAMIC-GROUP omagent TO USE loganalytics-collection-warning in COMPARTMENT ID <om_compartment_ocid>

Observability & Management Management Storage Purge Policy

Name: ompurge_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/logging-analytics/doc/manage-storage.html#GUID-DEAA0C68-9FB9-4441-9DCE-AEF6A358B6D5

Statements:

ALLOW DYNAMIC-GROUP ompurge to read compartments in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_STORAGE_PURGE} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_QUERY_VIEW} in tenancy

ALLOW DYNAMIC-GROUP ompurge to {LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ} in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-features-family in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-resources-family in tenancy

allow service loganalytics to READ loganalytics-features-family in tenancy

Observability & Management Admin Policy for local OCI IAM users

Name: omadmin_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omadmin to USE compartments in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-features-family in tenancy

ALLOW GROUP omadmin to MANAGE loganalytics-resources-family in tenancy

ALLOW GROUP omadmin to MANAGE management-dashboard-family in compartment ocioperations

ALLOW GROUP omadmin to MANAGE management-saved-search in compartment ocioperations

ALLOW GROUP omadmin to READ metrics in compartment ocioperations

ALLOW GROUP omadmin to MANAGE loganalytics-resources-family in compartment ocioperations

Observability & Management User Policy for local OCI IAM users

Name: omuser_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omuser to USE compartments in tenancy

ALLOW GROUP omuser to READ loganalytics-features-family in tenancy

ALLOW GROUP omuser to READ management-dashboard-family in compartment ocioperations

ALLOW GROUP omuser to READ management-saved-search in compartment ocioperations

ALLOW GROUP omuser to READ metrics in compartment ocioperations

ALLOW GROUP omuser to READ loganalytics-resources-family in compartment ocioperations

Observability & Management Admin Policy for federated OCI IAM users

Name: omfedadmin_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omfedadmin to USE compartments in tenancy

ALLOW GROUP omfedadmin to MANAGE loganalytics-features-family in tenancy

ALLOW GROUP omfedadmin to MANAGE loganalytics-resources-family in tenancy

ALLOW GROUP omfedadmin to MANAGE management-dashboard-family in compartment ocioperations

ALLOW GROUP omfedadmin to MANAGE management-saved-search in compartment ocioperations

ALLOW GROUP omfedadmin to READ metrics in compartment ocioperations

ALLOW GROUP omfedadmin to MANAGE loganalytics-resources-family in compartment ocioperations

Observability & Management User Policy for federated OCI IAM users

Name: omfeduser_policy

Doc References:

• https://docs.oracle.com/en-us/iaas/Content/doc/get-started-management-dashboard.html
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/managementdashboardpolicyreference.htm
• https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/monitoringpolicyreference.htm

Note: Make sure that the log filter of dashboards and Log Explorer for compartment matches the compartment specified.

Statements:

ALLOW GROUP omfeduser to USE compartments in tenancy

ALLOW GROUP omfeduser to READ loganalytics-features-family in tenancy

ALLOW GROUP omfeduser to READ management-dashboard-family in compartment ocioperations

ALLOW GROUP omfeduser to READ management-saved-search in compartment ocioperations

ALLOW GROUP omfeduser to READ metrics in compartment ocioperations

ALLOW GROUP omfeduser to READ loganalytics-resources-family in compartment ocioperations