Updates to our upgrade process

Originally posted on Jan 24, 2018.

This winter like many other institutions we applied the latest CU to enable the crocodoc to box transition.

The upgrades worked well as we were able to update and adjust the implementation plans we had spent a lot of time on last year.

Changes undone by new CUs.

Cumulative Updates use the full installation procedure and will usually undo changes to configuration files that have been made previously to resolve known issues.  This means that when applying a CU or performing a new upgrade that one should verify whether the installer has undone changes and redo them. This became immediately clear to us this winter.  This blog post covers what we did to make the experience less painful.

Last summer we had already started documenting the fixes we had made separately so that they were not lost and forgotten in old implementation plans, so we had a nice list of fixes to check.  Now when applying a new CU we have to check whether those fixes have been undone.  Here are the changes we made to our procedures to deal with this.

1. Analyse whether changes have been made to key files.

Before the upgrade we copy all the files to which we make changes to a separate directory outside of Blackboard.

mkdir /home/bbuser/original_files
cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/original_files/
cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/original_files/
cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/original_files/
cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/original_files/
cp /usr/local/blackboard/config/service-config.properties /home/bbuser/original_files/
cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/original_files/ 

Then after the upgrade we make copy those files again to a different directory.

mkdir /home/bbuser/post_upgrade_files
cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/service-config.properties /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/post_upgrade_files/ 

We then use diff to analyse whether the files have changed.

diff /home/bbuser/original_files/bb-config.properties /home/bbuser/post_upgrade_files/bb-config.properties

diff /home/bbuser/original_files/bb-config.properties.template /home/bbuser/post_upgrade_files/bb-config.properties.template

diff /home/bbuser/original_files/http-cacheable-extensions.txt /home/bbuser/post_upgrade_files/http-cacheable-extensions.txt

diff /home/bbuser/original_files/nautilus-config.properties /home/bbuser/post_upgrade_files/nautilus-config.properties

diff /home/bbuser/original_files/Search.properties /home/bbuser/post_upgrade_files/Search.properties

diff /home/bbuser/original_files/Search.properties.bb /home/bbuser/post_upgrade_files/Search.properties.bb

diff /home/bbuser/original_files/server.xml.bb /home/bbuser/post_upgrade_files/server.xml.bb

diff /home/bbuser/original_files/service-config.properties /home/bbuser/post_upgrade_files/service-config.properties

diff /home/bbuser/original_files/wrapper.conf /home/bbuser/post_upgrade_files/wrapper.conf 

diff /home/bbuser/original_files/wrapper.conf.bb /home/bbuser/post_upgrade_files/wrapper.conf.bb 

diff /home/bbuser/original_files/xythos-indexing-filter.txt /home/bbuser/post_upgrade_files/xythos-indexing-filter.txt

This tells us not only if our fixes have been undone by the installer, but also whether additional changes have been made to the file concerned.  If no other changes have been made we can copy the old file back rather than edit it within the Blackboard directory, saving typing time.

2. Produce verification scripts to verify whether the changes we have made have been done successfully.

Previously, we made individual test to verify whether a change had been correctly made.  For example:

grep 'eot\|ttf\|woff' /usr/local/blackboard/config/internal/http-cacheable-extensions.txt | wc -l  
>> Result must equal 3.

The number of these grew during the project and so to save time and make it simpler we developed a pair of scripts that would perform the checks and report back in red or green text whether the verifications had passed or failed and what the result of the test was.  We broke these down into two separate scripts, one checking the bb-config.properties file where most of our changes are, and the other checking changes to any other files.

The scripts also check the hostname to be sure whether settings that should be only set on the tasks node are set on the tasks node, e.g. the “backend.processor” setting.

Here are example screenshots showing the output of the scripts.

And here is the code for one of the scripts as an example.

#!/bin/bash

# Checking script for Blackboard 2016 Q4 CU6 Upgrade - bb.config.properties checks





# Set colours

red=$'\e[1;31m'

grn=$'\e[1;32m'

end=$'\e[0m'





# Set variables

# Hostname

hostname=$(hostname)

# Details for SCORM fix

ScormFixinbbcptemplate=$(cat /usr/local/blackboard/config/bb-config.properties.template | grep "ancestors=self'")

ScormFixinbbcp=$(cat /usr/local/blackboard/config/bb-config.properties | grep "ancestors=self'")

# Details for tomcat settings

bbconfigjvmoptionsextratomcat=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.jvm.options.extra.tomcat")

bbconfigjvmoptionsgc=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.jvm.options.gc")

# DMARC setting

bbconfigemaildmarc=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.email.use.dmarc.from.override=")

# Backend processor setting (note different on tasks/web nodes)

backendproc=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.server.backend.processor")

# HTTP compression setting

httpcomp=$(cat /usr/local/blackboard/config/bb-config.properties |grep bbconfig.appserver.http.compression)

# Clear the screen

clear

# Report results

echo "bb.config.properties checks on $hostname"

echo ""

echo ""

echo "Test 1 - Scorm Fix in bbcp-template"

echo ""

if [ "$ScormFixinbbcptemplate" = "bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $ScormFixinbbcptemplate"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $ScormFixinbbcptemplate"

fi

echo ""

echo ""

echo ""

echo "Test 2 - Scorm Fix in bbcp"

echo ""

if [ "$ScormFixinbbcp" = "bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $ScormFixinbbcp"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $ScormFixinbbcp"

fi

echo ""

echo ""

echo""

echo "Test 3 - bbconfig.jvm.options.extra.tomcat"

echo ""

if [ "$bbconfigjvmoptionsextratomcat" = "bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $bbconfigjvmoptionsextratomcat"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $bbconfigjvmoptionsextratomcat"

fi

echo ""

echo ""

echo ""

echo "Test 4 - bbconfig.jvm.options.gc"

echo ""

if [ "$bbconfigjvmoptionsgc" = "bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $bbconfigjvmoptionsgc"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $bbconfigjvmoptionsgc"

fi

echo ""

echo ""

echo ""

echo "Test 5 - bbconfig.email.use.dmarc.from.override"

echo ""

if [ "$bbconfigemaildmarc" = "bbconfig.email.use.dmarc.from.override=false" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $bbconfigemaildmarc"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $bbconfigemaildmarc"

fi

echo ""

echo ""

echo ""

# Note that a different result is expected depending on whether we are on the tasks node or a web node

echo "Test 6 - backend processor setting"

echo ""

if [[ "$backendproc" = "bbconfig.server.backend.processor=true" && "$hostname" = "removed" ]] || \

[[ "$backendproc" = "bbconfig.server.backend.processor=false" && "$hostname" != "removed" ]]; then

   printf "%s\n" "${grn}TEST PASSED${end}: Hostname is $hostname and value returned is $backendproc"

else

   printf "%s\n" "${red}TEST FAILED${end}: Hostname is $hostname and value returned is $backendproc"

fi

echo ""

echo ""

echo ""

echo "Test 7 - http.compression"

echo ""

if [ "$httpcomp" = "bbconfig.appserver.http.compression=off" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $httpcomp"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $httpcomp"

fi

echo ""

echo ""

Final thoughts

Although it is really annoying to have to spend time to reimplement config changes and fixes just to install a CU we are slowly developing our practice to streamline the process as much as we can.  I would be very interested to hear from other Blackboard admins about how they deal with this aspect of upgrades.

Addendum

On the community site Ashley Fisher wrote an interesting reply.  In case it is not transferred to the new community site I'm pasting it below.

I haven't done anything with checking the bb-config.properties yet, thanks for the inspiration.

I did write a 'post-installation' script which I run straight after a new CU/SP, this reapplies some fixes for issues that have given me some grief.

Environment Spec:

RHEL6, Oracle RAC, Bb Learn v3100.0.6

The fixes applied:

     Tomcat Bug #60035:

          https://blackboard.secure.force.com/btbb_articleview?id=kA339000000CfL9 

     ActiveMQ Does not Start on Oracle RAC Environments:

          https://blackboard.secure.force.com/btbb_articleview?id=kA339000000Cej0

     Logspam While Attempting to Lock File:

          https://blackboard.secure.force.com/btbb_articleview?id=kA339000000CfI5

    Xythos Cross-Server Communication Fails:

          https://blackboard.secure.force.com/btbb_articleview?id=kA370000000H0Yq

#!/bin/bash

     (HEADER/DESCRIPTION REMOVED)

# Remove Temporary Install Dir

        if [ -d /app/tmpupgrade/ ]

                then echo "       Check Result: Temporary installer directory exists, removing."

                rm -R /app/tmpupgrade/

        else

                echo "       Check Result: Temporary installer directory did not exist."

        fi

cp ../backup/$(hostname)/tnsnames.ora* /usr/local/blackboard/apps/oracle-client/network/admin/

cp ../backup/$(hostname)/tnsnames.ora* /usr/local/blackboard/config/oracle/network/admin/

cp ../backup/$(hostname)/tnsnames.ora* /usr/local/blackboard/config/oracle-client/

# Create Rollback Directories

        echo "INFO:  Checking and creating backup location."

        if [ -d ../backup ]

                then echo "       Check Result: Backup folder exists."

        else

                echo "       Check Result: Backup folder does not exist, creating."

                mkdir ../backup/

        fi

        if [ -d ../backup/$(hostname) ]

                then echo "       Check Result: Host folder exists."

        else

                echo "       Check Result: Host folder does not exist, creating."

                mkdir ../backup/$(hostname)

        fi

# ActiveMQ

        ## Take Backup

        echo "INFO:  ActiveMQ Broker configuration."

        if [ ! -f ../backup/$(hostname)/wrapper.conf.bb ]

                then

                        echo "       Backup of wrapper.conf.bb does not exist. Backing up."

                        cp /usr/local/blackboard/apps/activemq-broker/conf/wrapper.conf.bb ../backup/$(hostname)/

                else

                        echo "       Backup of wrapper.conf.bb already exists."

        fi

        if [ ! -f ../backup/$(hostname)/message-queue-service-config.xml.bb ]

                then

                        echo "       Backup of message-queue-service-config.xml.bb does not exist. Backing up."

                        cp /usr/local/blackboard/config/message-queue-service-config.xml.bb ../backup/$(hostname)/ 

                else

                        echo "       Backup of message-queue-service-config.xml.bb already exists."

        fi

        ## Move Files

        if [ $(diff ./repository/wrapper.conf.bb /usr/local/blackboard/apps/activemq-broker/conf/wrapper.conf.bb | wc -l) -gt "0" ]

                then

                        echo "       New file and old file are different, replacing with updated wrapper.conf file."

                        cp ./repository/wrapper.conf.bb /usr/local/blackboard/apps/activemq-broker/conf/wrapper.conf.bb

                else

                        echo "       wrapper.conf.bb already fixed."

        fi

        if [ $(diff ./repository/message-queue-service-config.xml.bb /usr/local/blackboard/config/message-queue-service-config.xml.bb | wc -l) -gt "0" ]

                then

                        echo "      New file and old file are different, replacing with updated message-queue-service-config file."

                        cp ./repository/message-queue-service-config.xml.bb /usr/local/blackboard/config/message-queue-service-config.xml.bb

                else

                        echo "       message-queue-service-config.xml.bb already fixed."

        fi

        ## Check

        chkAMQLib=$(cat /usr/local/blackboard/apps/activemq-broker/conf/wrapper.conf.bb | grep 'oracle-client/lib64' | wc -l)

        if [ "$chkAMQLib" -eq "1" ]

                then

                        echo "       ActiveMQ Wrapper Configuration Updated Successfully."

                else

                        echo "ERROR: ActiveMQ Wrapper Configuration Not Updated!."

        fi

# Java File Pref. Lock

        echo "INFO:  Checking for Java systemPrefs lock directory."

        if [ ! -d /home/bbuser/.java/.systemPrefs ]

                then

                        echo "       systemPrefs lock directory does not exist, creating."

                        mkdir /home/bbuser/.java/.systemPrefs

                        chown bbuser:bbgroup -R /home/bbuser/.java/.systemPrefs

                else

                        echo "       systemPrefs lock directory already exists."

        fi

        chown -R bbuser /etc/.java/.systemPrefs

# Cross-Server Security Fix

        echo "INFO:  Cross-Server security fix check."

        if [ $(diff /usr/local/blackboard/config/tomcat/conf/server.xml.bb ./repository/server.xml.bb | wc -l) -gt "0" ]

                then

                        echo "       Cross-Server security fix has not been applied."

                        if [ ! -f ../backup/$(hostname)/server.xml.bb ]

                                then

                                        echo "       Backup does not exist. Taking Backup."

                                        cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb ../backup/$(hostname)/

                        fi

                        echo "       Applying change for cross-server security."

                        cp ./repository/server.xml.bb /usr/local/blackboard/config/tomcat/conf/server.xml.bb

                else

                        echo "       Cross-Server security fix has previously been applied."

        fi

# Tomcat Bug 60035

        echo "INFO:  Tomcat Bug #60035 Check."

        if [ $(diff ./repository/tomcat-util.jar /usr/local/blackboard/apps/tomcat/lib/tomcat-util.jar | wc -l) -gt "0" ]

                then

                        echo "       Tomcat fix has not been applied. Taking Backup."

                        cp /usr/local/blackboard/apps/tomcat/lib/tomcat-util.jar ../backup/$(hostname)/

                        cp /usr/local/blackboard/apps/tomcat/lib/tomcat-coyote.jar ../backup/$(hostname)/

                        echo "       Applying Tomcat Fix."

                        cp ./repository/tomcat-util.jar /usr/local/blackboard/apps/tomcat/lib/tomcat-util.jar

                        cp ./repository/tomcat-coyote.jar /usr/local/blackboard/apps/tomcat/lib/tomcat-coyote.jar

                else

                        echo "       Tomcat fix has been previously applied."

        fi

echo "INFO:  Cascading bbuser permissions over the /usr/local/blackboard folder."

chown -R bbuser:bbgroup /usr/local/blackboard

echo "       Permissions successfully applied to /usr/local/blackboard folder."

echo ""

echo "------------------------------------------------------------------------------------------------------------------------"

echo "   WARN:  You now need to run PushConfigUpdates.sh to apply these changes. This will restart Blackboard on this host.   "

echo "------------------------------------------------------------------------------------------------------------------------"

echo "Would you like to PushConfigUpdates now? (yes/NO):                                                                "

read pushConfig

if [ $pushConfig == "yes" ]

        then

                echo "INFO:  PushConfigUpdates.sh is now running for you."

                /usr/local/blackboard/tools/admin/PushConfigUpdates.sh

        else

                echo "INFO:  You have selected to manually run PushConfigUpdates."

fi

echo ""