OCI Logging service

Intro
Logs

Audit logs
Service logs
Custom logs

OCI compute
On-prem host
Logs retention and search

Intro

Oracle adopts Cloud Native Computing Foundation standards (CNCF is like a home for many open source projects):

Fluentd is open-source log collector, it collects events from various data sources and writes them to various locations (files, database, cloud service).
Cloudevents is a specification for describing event data in common formats, supported by different systems, tools, cloud services, etc.

Logs

There are three types:

Audit logs

Read-only logs from the Audit service. It automatically records calls to OCI services. Observability and management - Logging - Audit

Service logs

Service logs can be enabled for supported OCI services. Storage - Object Storage - Bucket (select a bucket) - Logs - Enable logs (for read and write events).

Custom logs

OCI compute

An OCI compute with supported OS (OL7-8, Win, not Solaris) comes with installed unified-monitoring-agent (RPM description: Oracle Unified Agent custom fluentd distribution).

Service: unified-monitoring-agent
Configuration: /etc/unified-monitoring-agent/conf.d/fluentd_config/fluentd.conf

Policy

Identity and Security - Policies - Create Policy, syntax "allow dynamic-group <dg-name> to use log-content in compartment <compartment-name>"

Log group

A log group is a container for organizing logs. Logs must belong to some group. Example:

Log group "compartment-name"-computes (logging destination for all computes in this compartment)
There are multiple logs in this log group (var_log, apache, some-script, some-app)

Agent (fluentd)

A log has one or more agent (fluend) configuration (created via OCI console):

config name, description
dynamic group
log inputs (file path: /var/log/*)
log destination (log group and log name)

On a host, the unified-monitoring-agent service reads agent configuration from OCI, and downloads it to the host.

Example of etc/unified-monitoring-agent/conf.d/fluentd_config/fluentd.conf
<source>
@type tail
tag 676961.varlog
path /var/log/*
pos_file /etc/unifiedmonitoringagent/pos/676961-varlog.pos
path_key tailed_path
<parse>
@type syslog
parser_type regexp
</parse>
</source>

<match 676961.**>
@type oci_logging
log_object_id ocid1.log.oc1.iad.amxxxxxxx # OCI ID for a log
<buffer tag>
@type file
retry_timeout 3h
path /opt/unifiedmonitoringagent/run/buffer/676961
disable_chunk_backup true
chunk_limit_size 5MB
flush_interval 180s
total_limit_size 1GB
overflow_action throw_exception
retry_type exponential_backoff
</buffer>
</match>

On-prem host

Policy

Syntax: "allow group <user-group-name> to use log-content in compartment <compartment-name>"

Log group

A log group is a container for organizing logs. Logs must belong to some group. Example:

Log group "on-prem" (logging destination for on-prem hosts, OL7-8)
There are multiple logs in this log group (var_log, apache, some-script, some-app)

Agent (fluentd)

A log has one or more agent (fluend) configuration (created via OCI console):

config name, description
user group
log inputs (file path: /var/log/*)
log destination (log group and log name)

Agent installation

Install "Oracle Unified Agent custom fluentd distribution" RPM. Look for something like unified-monitoring-agent-2.5.8-0.el7.x86_64.rpm and unified-monitoring-agent-2.5.8-0.el8.x86_64.rpm
Configure OCI profiles to be used by agent.

The file /etc/unified-monitoring-agent/.oci/config must have two identical profiles, DEFAULT and UNIFIED_MONITORING_AGENT.

[DEFAULT]
#log_requests=True
user=ocid1.user.oc1..aaaa..45q  # OCI ID for user, who's member of user group
fingerprint=....
key_file=...
tenancy=ocid1.tenancy.oc1... #tenancy oci id
region=your-region #ex. us-ashburn-1 
[UNIFIED_MONITORING_AGENT]
#log_requests=True
user=ocid1.user.oc1..aaa..n45q # OCI ID for user, who's member of user group
fingerprint=...
key_file=..
tenancy=ocid1.tenancy.oc1... # tenancy oci id
region=your-region #ex. us-ashburn-1

Agent services

unified-monitoring-agent
unified-monitoring-agent_config_downloader

To verify that on-prem host agent can download fluentd config:
$ journalctl -u unified-monitoring-agent_config_downloader

To verify log batching on on-prem host:
$ tail -f /var/log/unified-monitoring-agent/unified-monitoring-agent.log

Logs retention and search

Logs retention is up to 6 months, but logs search is allowed only for last 14 days.

Search example, show/count number of logs.

#!/bin/bash
time_start="2021-09-13T22:30Z"
time_end="2021-09-16T09:30Z"

readonly compartment="ocid1.compartment.oc1....""
readonly log_group="ocid1.loggroup.oc1.iad...."
readonly log="ocid1.log.oc1.iad.a...."

# Count log number
echo Total logs from ${time_start} to ${time_end} is:

oci logging-search search-logs \
--time-start ${time_start} --time-end ${time_end} \
--search-query \
'search "ocid1.compartment.oc1.../ocid1.loggroup.oc1.iad.../ocid1.log.oc1.iad..." | count' \
| jq '.data.results[0].data.count'

exit 0

Search example, print logs after parse json (date, host, message)

#!/bin/bash
# Search all logs in specified time period
time_start="2021-09-15T22:30Z"
time_end="2021-09-16T05:30Z"

readonly compartment="ocid1.compartment.oc1....""
readonly log_group="ocid1.loggroup.oc1.iad...."
readonly log="ocid1.log.oc1.iad.a...."

oci logging-search search-logs \
--time-start ${time_start} --time-end ${time_end} \
--search-query \
'search "ocid1.compartment.oc1.../ocid1.loggroup.oc1.iad.../ocid1.log.oc1.iad..."' \
| jq -j '.data.results[].data.logContent| .time, " ",.data.host, " ",.data.message,"\n"'

# some other jq examples:
#| jq -j '.data.results[].data.logContent.data| " ",.host,": ",.message,"\n"'
#| jq '.data.results[] | .data.logContent.data | .host,.message'
# jq -j '.data.results[].data.logContent.data| " ",.host,": ",.message,"\n"'
exit 0