Back to the main page

Rebuild the VM running FreeIPA (IdM) server

It's shame that certmonger service didn't renew internal certs of IPA v4.4 server, hence now I need to 'rebuild' one IPA server (it's VM on stand alone Dom0, it's OVSv3) and sync from another good one. I need to do that as daily cronjob.
Because of this, I am not able to add new IPA server to all-master relationship or promote IPA client into master.
Here is more aboout certs ... FreeIPA (IdM) comes with internal CA (run by Dogtag Certificate System ) and its own certificates, used for variety of tasks, by variety of IPA internal components.
Cert nickname NSS database location Note
caSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias
  • Main CA signing certificate
  • Issuer: "CN=Certificate Authority,O=DOMAIN.COM"
  • Subject: "CN=Certificate Authority,O=DOMAIN.COM"
  • valid for 20 years after installation
subsystemCert cert-pki-ca /etc/pki/pki-tomcat/alias
  • Used by Dogtag CA to authenticate to LDAP (389DS) and other subsystems
  • Issuer: "CN=Certificate Authority,O=DOMAIN.COM"
  • Subject: "CN=CA Subsystem,O=DOMAIN.COM"
  • needs renewal every 2 years
auditSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias
  • Used to sign audit logs
  • Issuer: "CN=Certificate Authority,O=DOMAIN.COM"
  • Subject: "CN=CA Audit,O=DOMAIN.COM"
  • needs renewal every 2 years
ocspSigningCert cert-pki-ca /etc/pki/pki-tomcat/alias
  • Used to sign OCSP (Online Cert State Protocol) responses
  • Issuer: "CN=Certificate Authority,O=DOMAIN.COM"
  • Subject: "CN=OCSP Subsystem,O=DOMAIN.COM"
  • needs renewal every 2 years
Server-Cert cert-pki-ca /etc/pki/pki-tomcat/alias
  • TLS server cert for Dogtag website and HTTP APIs
  • Issuer: "CN=Certificate Authority,O=DOMAIN.COM"
  • Subject: "CN=ldap04.domain.com,O=DOMAIN.COM"
  • needs renewal every 2 years
ipaCert /etc/httpd/alias
  • RA (Registration Authority) agent cert used by IPA framework to auth and control (do privileged work) Dogtag
  • Issuer: "CN=Certificate Authority,O=DOMAIN.COM"
  • Subject: "CN=IPA RA,O=DOMAIN.COM"
  • needs renewal every 2 years
DOMAIN.COM IPA CA /etc/httpd/alias
  • IPA CA cert, trust anchor
  • Issuer: "CN=Certificate Authority,O=DOMAIN.COM"
  • Subject: "CN=Certificate Authority,O=DOMAIN.COM"
  • valid 20 yrs, this is == caSigningCert cert-pki-ca
"Server-Cert", this is cert for Apache & LDAP servers /etc/httpd/alias
  • Issuer: "CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US"
  • Subject: "CN=ldap04.domain.com,OU=Company,L=Castro Valley,ST=California,C=US"
  • valid 2 years

Logging files are:
The tool is done for Python 2.4

#!/bin/env python # -------------- # Revive ldap04 # -------------- import os import logging, logging.handlers from time import gmtime, strftime, sleep import datetime import socket import sys import subprocess import paramiko import getpass import pwd # logging destination PROGRAM = os.path.basename(sys.argv[0]) # name of this script LOG_PATH = ("/var/log/" + PROGRAM) # put together "/var/log/" if not os.path.exists(LOG_PATH): # create LOG_PATH if doesn't exists os.makedirs(LOG_PATH) os.chown(LOG_PATH,-1,580) # root(-1 means no change):some-group os.chmod(LOG_PATH,0775) # drwxrwxr-x # --- Define logging LOG_FILE = (LOG_PATH + "/" + datetime.datetime.now().strftime("%m-%d-%Y_%Hh%Mm%Ss")) # create logger (interface that script uses for logging) logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) # create file handler (define log destination) handler = logging.handlers.TimedRotatingFileHandler(LOG_FILE, when='MIDNIGHT', backupCount=50) #handler = logging.handlers.TimedRotatingFileHandler(LOG_FILE, when='MIDNIGHT', backupCount=50, utc=False) # not in python2.4 # create formatter (define layout of logs) formatter = logging.Formatter('%(asctime)s:%(levelname)s: %(message)s') handler.setFormatter(formatter) # add handler to logger logger.addHandler(handler) logger.debug("") logger.debug("START AT : " + strftime("%a, %d %b %Y %H:%M:%S", gmtime())) i_am=getpass.getuser() if i_am != "root": logger.debug("Only Root can run this") sys.exit("Only Root can run this") def schedule_downtime(): """ Scheduel ldap04 downtime on monitoring3 """ nagios_server = "monitoring3.domain.com" rootkey = "/root/.ssh/id_rsa.pub" client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=nagios_server, username=i_am, key_filename=rootkey) try: logger.debug("Schedule downtime for ldap04") stdin, stdout, stderr = client.exec_command('/usr/local/sbin/downtime_ldap04.sh') if stdout.channel.recv_exit_status() != 0: logger.debug("Cannot schedule downtime for ldap04") sys.exit("Cannot schedule downtime for ldap04") except paramiko.ssh_exception.SSHException: logger.debug("SSHException: {0}") sys.exit("SSHException: {0}") except paramiko.ssh_exception.ChannelException: logger.debug("paramiko.ssh_exception.ChannelException: {0}") sys.exit("paramiko.ssh_exception.ChannelException: {0}") except paramiko.ssh_exception.NoValidConnectionsError: logger.debug("NoValidConnectionsError: {0}") sys.exit("NoValidConnectionsError: {0}") except paramiko.ssh_exception.PartialAuthentication: logger.debug("PartialAuthentication: {0}") sys.exit("PartialAuthentication: {0}") close() def stop_vm(): """ Halt ldap04 """ #return True if subprocess.call(['/usr/sbin/xm', 'destroy', 'ldap04']) == 0 else False # not in python2.4 logger.debug("Stopping ldap04") try: subprocess.call(['/usr/sbin/xm', 'destroy', 'ldap04']) #subprocess.call(['/usr/sbin/xm', 'destroy', 'ldap04'], shell=True) except: logger.debug("Cannot stop ldap04") sys.exit("Cannot stop ldap04") def del_old_image(): """ Delete fault disk image """ logger.debug("Deleting faulty image /OVS/ldap04/ldap04.img") try: subprocess.call(["/bin/rm -f /OVS/ldap04/ldap04.img"], shell=True) except: logger.debug("Cannot delete faulty image /OVS/ldap04/ldap04.img") sys.exit("Cannot delete faulty image /OVS/ldap04/ldap04.img") def copy_good_image(): """ Copy good disk image of ldap04 """ logger.debug("Copying good image /OVS/ldap04-Feb-27-2019-GOOD-VM/ldap04.img to /OVS/ldap04/") try: subprocess.call(["/bin/cp /OVS/ldap04-Feb-27-2019-GOOD-VM/ldap04.img /OVS/ldap04/ldap04.img"], shell=True) except: logger.debug("Cannot copy good disk image /OVS/ldap04-Feb-27-2019-GOOD-VM/ldap04.img") sys.exit("Cannot copy good disk image /OVS/ldap04-Feb-27-2019-GOOD-VM/ldap04.img") def start_vm(): """ Start ldap04 """ logger.debug("Starting ldap04") try: subprocess.call(['/usr/sbin/xm', 'create', '/etc/xen/ldap04']) except: logger.debug("Cannot start ldap04") sys.exit("Cannot start ldap04") def autofs_check(): """ Check autofs service on ldap04 """ pass def replica_reinit(): """ Re-init replica from ldap03 """ logger.debug("Re-init replica on ldap04 from ldap03") faultipa = "ldap04.domain.com" rootkey = "/root/.ssh/id_rsa.pub" client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=faultipa, username=i_am, key_filename=rootkey) try: stdin, stdout, stderr = client.exec_command('/sbin/ipa-replica-manage re-initialize -p `cat /root/.replicapasswd` --from ldap03.domain.com') #stdin, stdout, stderr = client.exec_command('/sbin/ipa-replica-manage --verbose re-initialize -p `cat /root/.replicapasswd` --from ldap03.domain.com') if stdout.channel.recv_exit_status() != 0: logger.debug("Cannot re-init replica on ldap04") sys.exit("Cannot re-init replica on ldap04") except paramiko.ssh_exception.SSHException: logger.debug("SSHException: {0}") sys.exit("SSHException: {0}") except paramiko.ssh_exception.ChannelException: logger.debug("paramiko.ssh_exception.ChannelException: {0}") sys.exit("paramiko.ssh_exception.ChannelException: {0}") except paramiko.ssh_exception.NoValidConnectionsError: logger.debug("NoValidConnectionsError: {0}") sys.exit("NoValidConnectionsError: {0}") except paramiko.ssh_exception.PartialAuthentication: logger.debug("PartialAuthentication: {0}") sys.exit("PartialAuthentication: {0}") close() def nagios_check_cert(): """ Nagios check certs daily, so we'll force checks now """ nagios_server = "monitoring3.domain.com" rootkey = "/root/.ssh/id_rsa.pub" client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=nagios_server, username=i_am, key_filename=rootkey) try: logger.debug("Check http/ldap certs on ldap04") stdin, stdout, stderr = client.exec_command('/usr/local/sbin/check_certs_ldap04.sh') if stdout.channel.recv_exit_status() != 0: logger.debug("Cannot check http/ldap certs on ldap04") sys.exit("Cannot check http/ldap certs on ldap04") except paramiko.ssh_exception.SSHException: logger.debug("SSHException: {0}") sys.exit("SSHException: {0}") except paramiko.ssh_exception.ChannelException: logger.debug("paramiko.ssh_exception.ChannelException: {0}") sys.exit("paramiko.ssh_exception.ChannelException: {0}") except paramiko.ssh_exception.NoValidConnectionsError: logger.debug("NoValidConnectionsError: {0}") sys.exit("NoValidConnectionsError: {0}") except paramiko.ssh_exception.PartialAuthentication: logger.debug("PartialAuthentication: {0}") sys.exit("PartialAuthentication: {0}") close() if __name__ == '__main__': schedule_downtime() stop_vm() sleep(60) # wait for 1 min del_old_image() sleep(60) copy_good_image() sleep(10) start_vm() print("Waiting 3 min for VM to be online") logger.debug("Waiting 3 min for VM to be online") sleep(180) autofs_check() # not for now replica_reinit() sleep(15) nagios_check_cert() logger.debug("FINISH AT : " + strftime("%a, %d %b %Y %H:%M:%S", gmtime()))

And tool's log is:

2019-04-22 14:17:01,673:DEBUG: START AT : Mon, 22 Apr 2019 21:17:01 2019-04-22 14:17:02,101:DEBUG: Schedule downtime for ldap04 2019-04-22 14:17:02,306:DEBUG: Stopping ldap04 2019-04-22 14:18:04,692:DEBUG: Deleting faulty image /OVS/ldap04/ldap04.img 2019-04-22 14:18:14,711:DEBUG: Copying good image /OVS/ldap04-Feb-27-2019-GOOD-VM/ldap04.img to /OVS/ldap04/ 2019-04-22 14:18:24,735:DEBUG: Starting ldap04 2019-04-22 14:18:26,139:DEBUG: Waiting 3 min for VM to be online 2019-04-22 14:21:26,240:DEBUG: Re-init replica on ldap04 from ldap03 2019-04-22 14:21:53,847:DEBUG: Check http/ldap certs on ldap04 2019-04-22 14:21:54,006:DEBUG: FINISH AT : Mon, 22 Apr 2019 21:21:54

The Nagios server has this script to check certs, it's called by Python function nagios_check_cert().

#!/bin/env bash # force cert checks # https://assets.nagios.com/downloads/nagioscore/docs/externalcmds/cmdinfo.php?command_id=129 readonly PROGNAME=`/bin/basename $0` readonly loggerinfo="logger -t ${PROGNAME} Info:" readonly loggerwarning="logger -t ${PROGNAME} Warning:" readonly loggerproblem="logger -t ${PROGNAME} Problem:" err() { echo "\n ERROR: $* \n" ${loggerproblem} "$*" exit 1 } now=`date +%s` commandfile='/var/spool/nagios/cmd/nagios.cmd' host_name=ldap04.domain.com /bin/printf "[%lu] SCHEDULE_FORCED_SVC_CHECK;${host_name};cert_http_expiration;${now}\n" $now > $commandfile || \ err "Can't check cert_http_expiration service on ${host_name}" ${loggerinfo} "Checked cert_http_expiration service on ${host_name}" /bin/printf "[%lu] SCHEDULE_FORCED_SVC_CHECK;${host_name};cert_ldap_expiration;${now}\n" $now > $commandfile || \ err "Can't check cert_ldap_expiration service on ${host_name}" ${loggerinfo} "Checked cert_ldap_expiration service on ${host_name}" exit 0

The Nagios server has this script to schedule downtime, it's called by Python function schedule_downtime()

#!/bin/env bash # Schedule downtime # https://assets.nagios.com/downloads/nagioscore/docs/externalcmds/index.php?category_id=6&version_id=9&btnSubmit= # SCHEDULE_HOST_DOWNTIME;;;;;;;; # SCHEDULE_HOST_SVC_DOWNTIME;;;;;;;; readonly PROGNAME=`/bin/basename $0` readonly loggerinfo="logger -t ${PROGNAME} Info:" readonly loggerwarning="logger -t ${PROGNAME} Warning:" readonly loggerproblem="logger -t ${PROGNAME} Problem:" err() { echo "\n ERROR: $* \n" ${loggerproblem} "$*" exit 1 } # 2h downtime = 7200 sec now=`date +%s` in2h=`(echo "scale=2; ${now}+7200" | bc -l)` commandfile='/var/spool/nagios/cmd/nagios.cmd' host_name=ldap04.domain.com /bin/printf "[%lu] SCHEDULE_HOST_DOWNTIME;${host_name};${now};${in2h};0;0;7200;Zarko;Revive ca-ldap04\n" $now > $commandfile || \ err "Can't schedule downtime for ${host_name}" ${loggerinfo} "Scheduled downtime for ${host_name}" /bin/printf "[%lu] SCHEDULE_HOST_SVC_DOWNTIME;${host_name};${now};${in2h};0;0;7200;Zarko;Revive ca-ldap04\n" $now > $commandfile || \ err "Can't schedule downtime for services on ${host_name}" ${loggerinfo} "Scheduled downtime for services on ${host_name}" exit 0



Back to the main page