Passed EX342 Linux Troubleshooting

I recently passed my 4th exam out of my 5 RHLS allocation (with my 5th one due to be EX362 IdM).

I found it pretty tough to be honest, tougher than my recent exams. It’s different to the other Red Hat Exams as it’s all about troubleshooting and finding issues and fixing them. In that regards it’s similar to EX415 – and there is overlap in the scope (selinux, aide, audit etc).

I would definitely classify this as one of the toughest exams I have done – there were a couple of tasks that I had not done before (or even knew how to do).  I was pretty pleased with the overall score.

Onto the next one ….

 

EX342 Objective 7.2 – Identify and fix LDAP and Kerberos identity management issues – How to set up LDAP and Kerberos Authentication

Finally I am beginning to understand LDAP + Kerberos. In this guide I am going to configure a server to use Kerberos and then LDAP.

I am not going to explain what LDAP/kerberos is in detail – that’s for a future post.

 

but first some housekeeping …

Hit like if you like this post and would like to see more like it, or follow to be kept informed of any new posts

 

LDAP + Kerberos Explained in a nutshell:

I wish there was a guide that explained it as succinctly as I am about to.

A system that uses local files only for users stores information primarily in two places. It stores User Information in /etc/passwd and stores Authentication information (the actual password) in /etc/shadow

You need both User Information and Authentication pieces in order to login.

Kerberos is used for Authentication Only – that is to say it replicates /etc/shadow like functionality.

LDAP is used in this example for User Information (although technically you can use LDAP for Authentication information too) – that is to say it replicates /etc/passwd like functionality.

 

Prep:

Before we start we need to take stock of what we have. I have a Centos 7.7 server which uses local users only.

[root@ks9 ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

 

I am going to configure it to use a server called idm1.example.com – so I am going to add a local hosts entry. You can have DNS or whatever method is appropriate

[root@ks9 ~]# echo 192.168.1.70 idm1.example.com >> /etc/hosts

 

And finally we are going to confirm that the users admin and test01 do not exist locally

[root@ks9 ~]# getent passwd test01
[root@ks9 ~]# getent passwd admin
# no output for either command

 

 

Set up Kerberos

As I mentioned before we are going set up Kerberos to obtain the Authentication Information – but I need User Information to be stored locally. So upon a login the /etc/passwd will be consulted for half the information and Kerberos for the other half

 

So let’s create a local user called admin – note that admin doesn’t have a password

[root@ks9 ~]# useradd admin
[root@ks9 ~]# getent passwd admin
admin:x:1000:1000::/home/admin:/bin/bash

 

If I try and log in – it doesn’t work

[root@ks9 ~]# ssh admin@ks9.ykw.home
admin@ks9.ykw.home's password:
Permission denied, please try again.
admin@ks9.ykw.home's password:

 

Now we are going to install the kerberos package and configure it

[root@ks9 ~]# yum install -y pam_krb5

 

And now we are going to configure it using authconfig-tui (because I am lazy)

[root@ks9 ~]# authconfig-tui

ldap-kerb1

 

And let’s try logging in again

[root@ks9 ~]# ssh admin@ks9.ykw.home
admin@ks9.ykw.home's password:
Last failed login: Wed Jan 8 13:50:46 GMT 2020 from ks9.ykw.home on ssh:notty
There was 1 failed login attempt since the last successful login.
[admin@ks9 ~]$ id
uid=1000(admin) gid=1000(admin) groups=1000(admin) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[admin@ks9 ~]$ exit
logout
Connection to ks9.ykw.home closed.

 

Boom! easy as pie

To summarise we used a local user and used Kerberos instead of /etc/shadow for the Authentication element. We can’t log in using test01 as the User Information doesn’t exist locally.

 

Set up LDAP (for User Information)

So now we are going to complete the process by setting up LDAP

 

Let’s install packages

[root@ks9 ~]# yum install -y nss-pam-ldapd openldap-clients
# the openldap-clients package has the ldapsearch utility

 

So once again into authconfig-tui to set it up (The Kerberos information is left as-is so I didn’t put the image up again)

[root@ks9 ~]# authconfig-tui

ldap-kerb2

Another point to mention is that technically I could use TLS – but I would have to set up the CA Certificate – so I am doing this without TLS. It is “less” important anyway as no passwords will be transferred over the network (as Kerberos is handling the passwords)

There is one final step – which for some reason authconfig-tui didn’t do for me.

[root@ks9 ~]# authconfig --updateall --enableldap --enableldapauth

 

So now let’s test it out by doing an ldapsearch

[root@ks9 ~]# ldapsearch -x
...
# test04, users, accounts, example.com
dn: uid=test04,cn=users,cn=accounts,dc=example,dc=com
ipaNTSecurityIdentifier: S-1-5-21-306935617-297195054-2692361084-1004
displayName: user user
uid: test04
objectClass: top
objectClass: person
...

 

lots of data coming back which is good – let’s see if the data can actually be used for User Information

[root@ks9 ~]# getent passwd test01
test01:*:503500500:503500500:test user:/home/test01:/bin/sh

 

As we can see – it now considers test01 a valid user. So let’s try logging in using it.

[root@ks9 ~]# ssh test01@ks9.ykw.home
test01@ks9.ykw.home's password:
Last failed login: Wed Jan 8 13:58:54 GMT 2020 from ks9.ykw.home on ssh:notty
There were 2 failed login attempts since the last successful login.
Could not chdir to home directory /home/test01: No such file or directory
-sh-4.2$ exit

 

Boom! we have been able to use LDAP + Kerberos to login. Now to fix that ugly no such directory message. So exit back to root and enable the service to automatically create home directories

[root@ks9 ~]# authconfig --enablemkhomedir --update

 

Let’s try logging in again

[root@ks9 ~]# ssh test01@ks9.ykw.home
test01@ks9.ykw.home's password:
Creating directory '/home/test01'.
Last login: Wed Jan 8 14:15:06 2020 from ks9.ykw.home
-sh-4.2$ pwd
/home/test01
-sh-4.2$ ls
-sh-4.2$ date > file1
-sh-4.2$ cat file1
Wed Jan 8 14:17:02 GMT 2020

 

BOOM! You know LDAP + Kerberos.

 

BONUS – Set up TLS LDAP Connection

We are going to make a small change to enable TLS lookups

At the moment if we try this – we get the following error:

[root@ks9 ~]# ldapsearch -x -Z
ldap_start_tls: Connect error (-11)
additional info: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (self signed certificate in certificate chain)
ldap_result: Can’t contact LDAP server (-1)

 

We need to get the TLS cert and run a command to process it

[root@ks9 ~]# scp idm1.example.com:/etc/ipa/ca.crt /etc/openldap/cacerts/
Password:
ca.crt 100% 1289 498.3KB/s 00:00
[root@ks9 ~]# cacertdir_rehash /etc/openldap/cacerts/
[root@ks9 ~]# ls /etc/openldap/cacerts/
45e037a3.0 ca.crt

 

So now we have a TLS cert and it’s in the right place. Let’s use my favourite tool authconfig-tui

[root@ks9 ~]# authconfig-tui

ldap-kerb3

The above is the only screen we changed

Let’s test it using ldapsearch

[root@ks9 ~]# ldapsearch -x -Z cn
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> (default) with scope subtree
# filter: (objectclass=*)
# requesting: cn
#

# users, compat, example.com
dn: cn=users,cn=compat,dc=example,dc=com
cn: users

 

So now we are going to test it by logging in but before we do –  just to prove it – we are going to log on using a user we haven’t used thus far (test03)

 

and the login itself worked fine:

[root@ks9 ~]# ssh test03@ks9.ykw.home
test03@ks9.ykw.home's password:
Creating directory '/home/test03'.
-sh-4.2$

 

 

 

EX342 My Predictions for the Exam

I am coming close to taking my EX342 exam – so it’s time for my predictions for things that will be on the exam.

Bare in mind that I have not taken the exam, nor do I have any inside knowledge. This is my opinion of what they would cover if I was creating the exam.

 

but first some housekeeping …

Hit like if you like this post and would like to see more like it, or follow to be kept informed of any new posts

 

Overview

There are a list of objectives, and there is finite length an exam could be. As an exam writer, they would be looking to test as much as they can in the most efficient ways possible – that is to say they will want to have maximum coverage.

In my opinion this can only mean that certain things on the objectives will not be on the exam as you can’t fit all the tasks into the 3-4 hours of the exam.

Things are most likely to appear if they are:

  1. More important
  2. Easy to grade
  3. Requirements it has on the environment e.g. does it need a separate server

 

So with that in mind …

 

What I expect will be on exam:

  • A broken system(s) – maybe damaged yum or network info
  • Forgotten root password
  • Broken/Damanged grub file (but not MBR as that’s more tricky to provision the environment for)
  • Basic Audit Rule (simple keyword watch)
  • Basic SELINUX (fcontext, restorecon, or boolean style)
  • Kernel Module parameter (set parameter for when module loads)
  • ISCSI misconfiguration (authentication and inititor name)
  • firewall (add service and ports)
  • yum versionlock (install/add/remove)
  • fix damaged package (i.e. a package with damaged files)
  • fix permission issue in a package
  • RPM DB repair
  • Basic PAM (i.e. the config file is damaged or missing elements)
  • Basic LDAP/Kerberos setup (nothing with certificates)
  • Basic network config (set IP, DNS, gateway)
  • tcpdump example (probably reading in a file and looking for info)
  • pmval (against a existing file)
  • ldd (missing libraries)
  • lost luks header
  • 1-2 disk related issues

Maybes:

  • sosreport
  • systemtap
  • lvm reconfiguration
  • rsyslog using different facilities
  • kdump use different message types, pages. location, compression, or size

 

What I don’t expect to be on the exam:

  • PXE Boot / Rescue a Linux System (as this would require a PXE server)
  • Advanced SELINUX issues (transition rules etc)
  • Partition table corrupt (MBR) (as this may require a PXE boot)
  • Persistant Journal (just don’t see the value)
  • memtest (just don’t see the value)
  • rasdaemon (just don’t see the value)
  • cockpit (just don’t see the value)
  • ISCSI setup on server
  • ISCSI authentication
  • ipv6 rich rules (I don’t think ipv6 is that important in the wild)
  • Network ipv6 (I don’t think ipv6 is that important in the wild)
  • Different targets e.g. Emergency or Rescue
  • GUI related work
  • AIDE – it’s already covered in EX415 and RHCSA
  • Advanced PAM e.g. what EX415 covers
  • Network Routing / Static Routes (too niche)

 

So what do you think?

 

 

 

EX342 Objective 3.4 – Identify and fix iSCSI issues – iSCSI conflict with LV

A bit of a weird one here, I found that iscsi doesn’t play very nice with LVM – find out more

but first some housekeeping …

Hit like if you like this post and would like to see more like it, or follow to be kept informed of any new posts

 

Overview / The Plan

 

I want to configure my server ks9 to be a iSCSI Target Server, and ks6 to be a client/initiator. ks9 doesn’t currently use LVM at all.

[root@ks9 ~]# pvs
-bash: pvs: command not found
[root@ks9 ~]# lvs
-bash: lvs: command not found

I am going to slice my /dev/sdb disk, and create a /dev/sdb1 partition that is 1GB in size

 

Install iSCSI Server / Target

# install binaries
yum install targetd targetcli -y

# enable service
systemctl enable target --now

# configure firewall
firewall-cmd --add-port=3260/tcp --permanent
firewall-cmd --reload

 

Now let’s configure iSCSI

For this we will use targetcli

targetcli

 

confirm everything is blank

/> ls
o- / ....................................................................................... [...]
o- backstores ............................................................................ [...]
| o- block ................................................................ [Storage Objects: 0]
| o- fileio ............................................................... [Storage Objects: 0]
| o- pscsi ................................................................ [Storage Objects: 0]
| o- ramdisk .............................................................. [Storage Objects: 0]
o- iscsi .......................................................................... [Targets: 0]
o- loopback ....................................................................... [Targets: 0]

 

let’s use /dev/sdb1 and configure

# create block device
backstores/block create dev=/dev/sdb1 name=block0


# change into iscsi directory
iscsi

# create iqn
create # this created iqn.2003-01.org.linux-iscsi.ks9.x8664:sn.8f67e8bfac3e

# let's go into the iqn - and configure
cd iqn.2003-01.org.linux-iscsi.ks9.x8664:sn.8f67e8bfac3e/tpg1/

# create an acl
acls/ create wwn=iqn.2019-11.com.redhat:ks6

# create a lun
luns/ create /backstores/block/block0 lun=lun0

# print final config
cd /
ls

my final configuration is

ex342-iscsi-vg-1

 

Save and quit by typing

exit

 

Now let’s go to the client and configure

# install iscsi binaries
yum install -y iscsi-initiator-utils

# update the auto generated name with the one we picked earlier (at the ACL stage)
echo InitiatorName=iqn.2019-11.com.redhat:ks6 > /etc/iscsi/initiatorname.iscsi

# start and enable service
systemctl enable iscsi --now

 

use iscsi to add the network disk

ok, so now discovery time

# iscsiadm -m discovery -t st -p ks9
192.168.1.69:3260,1 iqn.2003-01.org.linux-iscsi.ks9.x8664:sn.8f67e8bfac3e

 

we can see one item returned – so let’s ‘login’ and the disks should become available

# iscsiadm -m node -p ks9 -T iqn.2003-01.org.linux-iscsi.ks9.x8664:sn.8f67e8bfac3e -l
Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.ks9.x8664:sn.8f67e8bfac3e, portal: 192.168.1.69,3260] (multiple)
Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.ks9.x8664:sn.8f67e8bfac3e, portal: 192.168.1.69,3260] successful.

 

We now have a new device /dev/sdb (confusing as it’s the same name as the original target but this is purely coincidental)

# lsscsi
[0:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[3:0:0:0]    disk    LIO-ORG  block0           4.0   /dev/sdb

 

Let’s create a vg and LV, and filesystem

# create pv
pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.

# create vg
vgcreate iscsivg /dev/sdb
Volume group "iscsivg" successfully created

# create lv
lvcreate -L250M -n lv1 iscsivg
Rounding up size to full physical extent 252.00 MiB
Logical volume "lv1" created.

# create and mount filesystem
mkdir /iscsi
mkfs.xfs /dev/iscsivg/lv1
mount /dev/iscsivg/lv1 /iscsi
# copy some content to it
cp /etc/host* /iscsi/

# and now umount it
umount /iscsi

# let's configure fstab so it's automounted
echo "/dev/iscsivg/lv1 /iscsi xfs _netdev 0 0" >> /etc/fstab
mount -a
ls /iscsi/
host.conf  hostname  hosts  hosts.allow  hosts.deny

 

So all is good, the iscsi disk is visible – so let’s reboot ks9 and 10 seconds later reboot ks6

[root@ks9 ~]# reboot
[root@ks6 ~]# reboot

 

What should happen is ks9 reboots, and iscsi services are started, then ks6 boots and mounts the iscsi

# df /iscsi/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/iscsivg-lv1 254628 13144 241484 6% /iscsi

 

Now the fun part

All is good, but now we are going to install lvm2 on ks9 – lvm2 would have been installed anyway had ks9 been using lvm for it’s disks – but before we do, let’s see what ks9 thinks is happening

# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
fd0
sda
├─sda1 xfs 8ae62c35-7792-4025-aa25-af6e0f4578a9 /boot
├─sda2 swap 1af3b5de-c46c-4400-87bd-c41113cb3485 [SWAP]
└─sda3 xfs 346e8fae-6a99-4360-a54d-0c38e977a5f4 /
sdb
└─sdb1 LVM2_member qcMD41-yAbA-OlFu-d1u6-BBBF-xQTh-dZGTEB
sr0

So it’s detected that sdb1 now seems to have a LVM2 Member stamp on it – but it doesn’t know what to do with it – so ignores it

yum install -y lvm2

 

So let’s reboot ks9 and 10 seconds later reboot ks6

[root@ks9 ~]# reboot
[root@ks6 ~]# reboot

 

 

Now we will log in and what do we find?

[root@ks9 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
iscsivg 1 1 0 wz--n- 992.00m 740.00m
[root@ks9 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv1 iscsivg -wi-a----- 252.00m

So ks9 has now taken control of the VG / LV and taken it as it’s own.

 

It’s a sad state of affairs on ks6

[root@ks6 ~]# ls /iscsi/
[root@ks6 ~]# lsblk --scsi
NAME HCTL TYPE VENDOR MODEL REV TRAN
sda 0:0:0:0 disk VMware Virtual disk 1.0
sr0 2:0:0:0 rom NECVMWar VMware IDE CDR10 1.00 ata

 

How do we fix this mess?

 

We are going to remove the vg (safely) and then we are going to stop it being detected

on ks9 (master)

# deactivate lv
lvchange -an /dev/iscsivg/lv1
# deactivate vg
vgchange -an iscsivg
# export vg
vgexport iscsivg

Now edit lvm.conf and add a reject filter

vi /etc/lvm/lvm.conf

and added two lines (lines 144, and 156 respectively)  to effectively ignore /dev/sdb1

filter = [ "r|/dev/sdb1|" ]
global_filter = [ "r|/dev/sdb1|" ]

 

reboot ks9 and ks6

reboot

 

and now to test ….

[root@ks6 ~]# df /iscsi/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/iscsivg-lv1 254628 13144 241484 6% /iscsi
[root@ks6 ~]# find /iscsi/
/iscsi/
/iscsi/host.conf
/iscsi/hostname
/iscsi/hosts
/iscsi/hosts.allow
/iscsi/hosts.deny

 

Whoop! Whoop!

I am sure there must be a better way to do this – but the above is a method I found using trial and error – although I am still getting a warning on ks9

 

 

 

EX342 Objective 4.3 – Identify and restore changed files

I am really enjoying breaking stuff and trying to see if I can fix it. Here’s a fun one.

I am going to chmod chmod

but first some housekeeping …

Hit like if you like this post and would like to see more like it, or follow to be kept informed of any new posts

 

# ll /usr/bin/chmod
-rwxr-xr-x. 1 root root 58592 Aug 20 07:25 /usr/bin/chmod
# chmod 644 /usr/bin/chmod
# ll /usr/bin/chmod
-rw-r--r--. 1 root root 58592 Aug 20 07:25 /usr/bin/chmod
# chmod 755 /usr/bin/chmod
-bash: /usr/bin/chmod: Permission denied

So there we have it. We have in a way broken chmod.

Let’s find out what package provides the chmod function and verify file permissions

# rpm -qf /usr/bin/chmod
coreutils-8.22-24.el7.x86_64

# rpm -V coreutils-8.22-24.el7.x86_64
.M....... /usr/bin/chmod

 

So now we fix it, the generally wisdom is that you can fix it in the following ways:

  1. Use rpm to restore permissions
  2. Reinstall package using yum
  3. Off-piste methods

 

Option 1 – Use rpm to restore permissions

This the easiest method, except it doesn’t work because, yep you guessed it – it uses chmod under the covers to do the work

# rpm --setperms coreutils
...
sh: line 351: /usr/bin/chmod: Permission denied
sh: line 352: /usr/bin/chmod: Permission denied
sh: line 353: /usr/bin/chmod: Permission denied


# ll /usr/bin/chmod
-rw-r--r--. 1 root root 58592 Aug 20 07:25 /usr/bin/chmod

 

 

Option 2 – Reinstall package using yum

This is where we use yum to reinstall the package

yum reinstall coreutils -y

# ll /usr/bin/chmod
-rwxr-xr-x. 1 root root 58592 Aug 20 07:25 /usr/bin/chmod

So that’s worked – but let’s be even more devious – let’s break yum and then break chmod and retry the yum fix

[root@ks6 rpm]# ll /usr/bin/yum /usr/bin/chmod
-rwxr-xr-x. 1 root root 58592 Aug 20 07:25 /usr/bin/chmod
-rwxr-xr-x. 1 root root 801 Apr 13 2018 /usr/bin/yum
[root@ks6 rpm]# chmod -x /usr/bin/chmod /usr/bin/yum
[root@ks6 rpm]# ll /usr/bin/yum /usr/bin/chmod
-rw-r--r--. 1 root root 58592 Aug 20 07:25 /usr/bin/chmod
-rw-r--r--. 1 root root 801 Apr 13 2018 /usr/bin/yum

 

So now when we try to fix it – we get an yum error:

# yum reinstall coreutils -y
-bash: /usr/bin/yum: Permission denied

[root@ks6 rpm]# rpm -qf /usr/bin/yum
yum-3.4.3-158.el7.centos.noarch
[root@ks6 rpm]# rpm -V yum
.M.......    /usr/bin/yum
[root@ks6 rpm]# rpm --setperms yum
sh: line 1: /usr/bin/chmod: Permission denied
sh: line 2: /usr/bin/chmod: Permission denied

So that’s not good – now what? We now need to fix yum AND chmod

 

Option 3 – Off-piste methods

Let’s try ansible

# ansible localhost -m file -a "path=/usr/bin/chmod mode=755"
[WARNING]: Could not match supplied host pattern, ignoring: all

[WARNING]: provided hosts list is empty, only localhost is available

localhost | FAILED! => {
"msg": "Failed to set execute bit on remote files (rc: 126, err: /bin/sh: /usr/bin/chmod: Permission denied\n)"

It seems Ansible uses chmod too

 

Let’s try setacl

Let’s use setfacl to add a temporary execute ACL to yum (band-aid)

# setfacl -m u::rwx /usr/bin/yum
# getfacl /usr/bin/yum
getfacl: Removing leading '/' from absolute path names
# file: usr/bin/yum
# owner: root
# group: root
user::rwx
group::r--
other::r--

# ll /usr/bin/yum
-rwxr--r--. 1 root root 801 Apr 13 2018 /usr/bin/yum

 

Now let’s use yum to fix chmod

# yum reinstall coreutils -y

 

Now that chmod is fixed, we should be able to use rpm to fix yum

# rpm --setperms yum


# ll /usr/bin/yum
-rwxr-xr-x. 1 root root 801 Apr 13 2018 /usr/bin/yum

 

Let’s test yum

# yum info coreutils

 

All goodness. There are other advanced methods that can be broken out

 

And finally,

another tool that does the job is install In the below example, we copy a broken chmod to chmod.bak and then install it back using install and set permissions.

# chmod 644 /usr/bin/chmod
# cp /usr/bin/chmod /usr/bin/chmod.bak
# install -m 0755 /usr/bin/chmod.bak /usr/bin/chmod
# ll /usr/bin/chmod
-rwxr-xr-x. 1 root root 58592 Nov 15 19:13 /usr/bin/chmod

 

Update 1:

I received a great tip from a reader (hat-tip ixs) – for the following command which can be used

/lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod

Booking Red Hat Individual Exams

Red Hat Individual (or Kiosk) exams are the way to go. Red Hat have a frequent habit of cancelling exams at short notice. I recall once they cancelled my exam (due in first week of Jan) it on Christmas Eve and when I emailed them I had received an automated response stating that they are shutdown for the Christmas period and contact them on Jan X (which was the day after my exam was scheduled).

I would say there is an approx 75% chance of Red Hat cancelling exams here in the UK, in my experience. My EX280 was cancelled 2 times before I went down the Kiosk route.

The reason why so many get cancelled is due to the fact that they tack them onto the end of Courses, if not enough people turn up for the course, then they cancel both the course and the exam.

The downside of the Individual Exam is that you are effectively locked away in a room by yourself, there is no physical person walking round like a normal classroom environment. I would prefer a physical classroom exam but more and more use Kiosk Exams.

There are only 3 locations in the UK. Two in Central London and one in Manchester.

Another downside is the availability. I have looking for 2 weeks – checking everyday for a slot in December and the earliest date is 6 weeks away. Granted there is Xmas and New Year – but the average lead time is normally 3-5 weeks.

 

EX342 Objective 1.1 – Understand and employ general methods for troubleshooting – Persistent Journal

So here we are going to be one of those “implied-knowledge” tasks. We are going to set the System Journal (journalctl) to be persistent. This allows the logs from previous boots to be available. By default journalctl logs only the current logs. So let’s do it.

 

but first some housekeeping …

Hit like if you like this post and would like to see more like it, or follow to be kept informed of any new posts

 

 

List Boots

Verify that journalctl isn’t already using persistent journalling. Stating the obvious here but if it’s not broken then why fix it? In my scenario below only a single boot set-of-logs is available

# journalctl --list-boots
0 329084a4e5ea4b9fbd6d074580eeb17b Mon 2019-11-18 13:12:54 GMT—Tue 2019-11-19 14:00:27 GMT

 

Create file structure to host the journal

# linuxbuff tip man journalctl in case you get the path
# linuxbuff tip grep journal /etc/group if you forget the group name

mkdir /var/log/journal
chmod 2755 /var/log/journal
chgrp systemd-journal /var/log/journal
kilall -USR1 systemd-journald # or below if killall not available
# get the pid
ps -ef | grep journal
kill -USR1 501

 

Verify journal is being written to disk

# find /var/log/journal/
/var/log/journal/
/var/log/journal/37edb47b18df88d02e1079f3f5e8b66a
/var/log/journal/37edb47b18df88d02e1079f3f5e8b66a/system.journal

We can see in the above that files are being created under /var/log/journal – time to reboot and check

# reboot

# journalctl --list-boots
-1 329084a4e5ea4b9fbd6d074580eeb17b Mon 2019-11-18 13:12:54 GMT—Tue 2019-11-19 14:02:42 GMT
0 d1743e618e2f4d8082fe4e69758cdef2 Tue 2019-11-19 14:03:16 GMT—Tue 2019-11-19 14:04:07 GMT

 

We can now see that 0 is the current set of journal logs and -1 is the previous, and -2 when its available

 

List previous boot’s logs

journalctl -b-1

 

Print current boot’s logs

journalctl -b0

 

 

EX342 Objective 8 – Gather information to aid third party investigation of issues – sosreport

Right let’s get into sosreport. sosreport report is a utility that collects useful system information to aid remote troubleshooting. It’s a relatively simple task, quick to learn and not many options …. So let’s have a play ….

 

but first some housekeeping …

Hit like if you like this post and would like to see more like it, or follow to be kept informed of any new posts

 

Install sos package which contains sosreport binary

# yum install -y sos

As it’s a standalone binary, no services need to be started

 

 

List plugins

We are going to list all the plugins that are available and look for plugin abrt

# sosreport -l 
# sosreport -l | grep -B3 abrt

The following plugins are currently enabled:

abrt Automatic Bug Reporting Tool
--

Option 'timeout' available to all plugins - time in seconds to allow plugin to run, use 0 for no timeout

abrt.detailed off collect detailed info for every report

 

So we can see that the plugin abrt is enabled – so if you run sosreport without any arguments it will be included.

 

Disable a plugin permanently

If you want to ensure a plugin is never run (let’s say for Security reasons or other) – then you configure it within sosreport’s configuration file

vi /etc/sos.conf

[plugins]
#disable = rpm, selinux, dovecot
disable = abrt

verify plugin is disabled

# sosreport -l | grep -B3 abrt

The following plugins are currently disabled:

abrt skipped Automatic Bug Reporting Tool

 

The configuration does have some other useful parameters and worth familiarising yourself.

 

Enable a plugin permanently

Simple one – take it out of the disable line

 

Run a single plugin (with options)

All other plugins are temporarily ignored

sosreport -o abrt -k abrt.detailed=on

Enable (a disabled) plugins and run report (with options)

sosreport -e abrt -k abrt.detailed=on

 

Run a sos report with option

sosreport -k abrt.detailed=on

 

Exclude plugin during a run

# sosreport -n abrt

 

 

 

Which Red Hat CoE are the easiest or hardest

Maybe you have decided you want to go move on from your RHCSA/RHCE and want to take some ‘Certificate of Expertise’ level exams. Red Hat offer a range of exams, and from this range you can select 5 exams and become a Red Hat Certified Architect. A title that only lasts 3 years (from the date of your first CoE exam). Take a 6th exam and you are RHCA Level 2, take a 7th and you are RHCA Level 3 and so on.

but first some housekeeping …

Hit like if you like this post and would like to see more like it, or follow to be kept informed of any new posts

 

You can find a list of RHCA exams here: https://www.redhat.com/en/services/certification/rhca

rhca-coe-hardest0

Once upon a time, How hard Red Hat Exams are were loosely based on their numbering system:
– A 200-level exam would be at a System Administrator Level
– A 300-level exam would be at a RHCE Level
– A 400-level exam would be at a Advanced Level (RHCE+)

This has changed in recent years but can sometimes still provide a rough guide. I have tried to search a few different sites trying to gather other people’s opinions on how hard they found exams and try and collate them into a list. Obviously how hard someone finds an exam is very subjective but if 10 people say an exam is hard; then the exam is probably going to be hard.

Please note that blog post is based on other people’s comments rather than my own experiences.

hardest-banner

 

Based on what I have found, people fall into 3 main camps:

1. Path of Least Resistance i.e. Easiest exams

These are people that will cherry pick the easiest exams to obtain the RHCA

2. Linux Guru – Toughest Exams

These are people that will cherry pick the hardest exams to try and prove their worth

3. Interest/Work-focused

These are people who mix and match between easy and hard exams, and often link them to ones they do at work or are already familiar with.
I would urge you to consider which of the above you are – are you trying to obtain a title OR are you trying to obtaining skills? I wasn’t able to find out about every exam – but below are the trends I found:

Exams consistently rated as easy:

These are the exams that are almost universally considered Easy, I either found no one (or very few) people claiming they were hard and lots that consider them easy.

– EX318 Virtualization
– EX210 OpenStack
– EX436 Clustering

Exams consistently rated as hard:

These are the exams that are almost universally considered Hard, I either found no one (or very few) people claiming they were easy and lots that consider them hard.
– EX333 Network Services – No Longer available
– EX442 Performance Tuning – Considered the hardest exam still available
– EX342 Troubleshooting

Exams inconsistently rated:

And finally, these are the exams that I found some people claiming were really easy and others claiming them as really hard. However I will mention that more people considered them hard than easy.

– EX280 OpenShift
– EX405 Puppet
– EX407 Ansible

Word of warning:

What I would say in conclusion, is that none of these exams are “gimme’s” that is to say if you do not prepare and walk into an exam based on you being “The main man(or woman) at work” or “I am so good and experienced that I don’t need to study” – you are almost certainly going to fail.

Networking 101 – The basics written for everyone

Networking can be a tough topic to get your head around – below is a scenario that might help you. Below I describe the basics of computer networking.

 

“Let’s say you work at a Desk in an Office. Your desk location probably isn’t known to most people, if they want to send you a letter, they might address it to your building, where the receptionist then directs it to you. Now your office block might have different departments, but the receptionist knows you and where you sit and can ensure the letter gets to you.”

 

So let’s break this down:

“Let’s say you work at a Desk in an Office (your desk is your private IP address). Your desk location probably isn’t known to most people (because it’s your private IP address), if they want to send you a letter (a message/packet), they might address it to your building (your public ip address), where the receptionist (router) then directs it to you. Now your office might have different departments (VLANs), but the receptionist knows you and where you sit and can ensure the letter gets to you.”

The above (simplistic) explanation can be expanded with other analogies e.g. a security guard (firewall) to ensure nothing nasty get’s in or out. But you get the general gist.