Our final upgrade implementation plan (from 2015 Q4 CU1 to 2016 Q4 CU3)

Originally posted on Jun 22, 2017

Here is our final implementation plan. I will write up a further post highlighting parts that may be particularly useful.

Note that some additional fixes are listed here: A successful upgrade

Live Upgrade from 2015 Q4 CU1 to 2016 Q4 CU3 plan

This plan is for the Blackboard LIVE Upgrade from Bb 9.1 2015 Q4 CU1 to 9.1 2016 Q4 CU3.

Server Address

Blackboard preproduction is made up of four applications nodes, a tasks node, an Oracle 12C database, and a NFS server.

tasks.node.ip.removed – tasks node

web1.node.ip.removed – web node

web2.node.ip.removed – web node

web3.node.ip.removed – web node

web4.node.ip.removed – web node

oracle.node.ip.removed – Oracle database

NFS.node.ip.removed – NFS server

Note

Where "on each node" is stated, this refers to each node on which the Blackboard application runs, i.e. the tasks node and the web nodes.

Summary of stages

Stage 1 covers work that can be carried out during the working week prior to the weekend upgrade. This work concerns preparing files in advance. This stage will be dealt with as a separate RFC.
Stage 2 covers taking snapshot backups and starting the live upgrade window. This marks the beginning of the main RFC.
Stage 3 is Oracle patching.
For the new installer to work correctly certain passwords have to be set to be the same value. Stage 4 entails setting the passwords up correctly and ensuring that Blackboard is working with the new passwords before upgrading.
Stage 5 covers upgrading Java and upgrading Blackboard to the new version.
Having successfully upgraded Blackboard we must make various configuration changes. Stage 6 covers changes due to Java 1.8 and fixes to known issues.
Stage 7 encompasses changes made by BB_ADMIN_ROLE team via the GUI and testing.
In Stage 8 we check that Blackboard restarts ok.
Stage 9 involves tidying up after successful testing.
Stage 10 is when we make the service available again to end-users.

Stage 1 - Prior Preparation

Stage 1 covers work that can be carried out during the working week prior to the weekend upgrade. This work concerns preparing files in advance.

BB_ADMIN_ROLE and DBA: confirm and agree password to be used.
DBA: Prepare files for Oracle patching.
LINUX_ROLE: Prepare installation files on each node.

>> as bbuser
>> on each node
cd /usr/local
chown -R bbuser\: bbinstaller
1. Download the installer to /usr/local/bbinstaller on each node.
  1. Go to https://support.blackboard.com and log in.
  2. Go to page https://behind.blackboard.com/downloads/details.aspx?d=1766
  3. Click on Learn 9.1 Q4 2016 Cumulative Update 3 Installer.
  4. Click on I agree to the terms and conditions above.
  5. Click Get File.
  6. Download will start.
  7. Note: Installer has already been downloaded to:
  8. "J:\iSolutions\Service Delivery\eLearning\Blackboard\upgrade_201718\2016-Q4\Installer\2016-Q4-CU3" on 28/4/17.
2. Unzip the installer to /usr/local/bbinstaller/
3. Set files under /usr/local/bbinstaller to be owned by bbuser:bbuser
APP_MNGR_ROLE: Copy /usr/local/blackboard/config/license/blackboard-license.xml to /home/bbuser/ on each node

>> as bbuser
>> on each node
cp /usr/local/blackboard/config/license/blackboard-license.xml /home/bbuser/
BB_ADMIN_ROLE: on each node create file at /usr/local/bbinstaller/installer.properties with following draft contents

antargs.default.vi.db.password=[ ]
antargs.default.users.administrator.password=xxxx
antargs.default.users.integration.password=xxxx
antargs.default.users.rootadmin.password=xxxx
bbconfig.file.license=/home/bbuser/blackboard-license.xml
bbconfig.java.home=/usr/lib/jvm/java-1.8.0-oracle.x86_64
APP_MNGR_ROLE: Backup files under /usr/local/blackboard/docs/login/* to /home/bbuser/login.zip on each node.

>> on each node
cd /home/bbuser
zip -r login.zip /usr/local/blackboard/docs/login/*
>> set permissions
chown bbuser:bbuser login.zip
APP_MNGR_ROLE: Back up the data_integration directory on the tasks node only (task node).

>> On tasks node (task node)
cd /home/bbuser
zip -r data_integration.zip /usr/local/blackboard/data_integration/*
>> Set permissions
chown bbuser:bbuser data_integration.zip
APP_MNGR_ROLE: Back up the snapshot* files in /blackboard/apps/snapshot/data/ directory on the tasks node only.

>> On tasks node (task node)
cd /home/bbuser
zip -r snapshotfiles.zip /usr/local/blackboard/apps/snapshot/data/snapshot.*
>> set permissions
chown bbuser:bbuser snapshotfiles.zip

Stage 2 - LIVE Upgrade Preparation

Stage 2 covers taking snapshot backups and starting the live upgrade window

LINUX_ROLE: Remove Blackboard from NLB Pool, thus preventing user access.

Locate the node objects for the front end servers and 'force offline'
APP_MNGR_ROLE: Stop Blackboard on each node, web nodes (web1 – web4) first, then tasks node (task node).

>> On each node, web first, then tasks.
>> web nodes may be stopped concurrently.

service blackboard stop
LINUX_ROLE: Take "Cold" VMware snapshot #1 on each node' (including db and NFS server).
APP_MNGR_ROLE: Remove the data_integration_log_b directory, to prevent log spam.

>> as bbuser

cd /data/bb/blackboard/content/vi/bb_bb60/lucene/
rm -rf data_integration_log_b/
LINUX_ROLE: Set files to be owned by bbuser:bbuser on each node.

>> On each node
find /data/bb/blackboard/ -maxdepth 2 -mindepth 2 -type d -not -name content | xargs -I {} chown -R bbuser\: {}
LINUX_ROLE: Raise ulimit on each node.

>> On each node
>> Login as root

- vim /etc/security/limits.conf
- add following lines:
* soft nofile 5000
* hard nofile 65536

>> Pay attention that # does not appear in front of either line

- save
- logout
- login

>> as root check:
- ulimit -n
- ulimit -Hn

[root@~]# ulimit -n
5000
[root@~]# ulimit -Hn
65536

>> switch to bbuser and check is the same result:
- su bbuser
- ulimit -n
- ulimit -Hn

[bbuser@]$ ulimit -n
5000
[bbuser@]$ ulimit -Hn
65536

Stage 3 - Oracle Patching

To reduce overall downtime we will apply the latest Oracle patchset to Blackboard
This section took about 25 Minutes during the Dev 1 upgrade.
This section took about 18 Minutes during the Dev 2 upgrade.

DBA: Apply April 2017 Oracle patch as described at [[internal link]]

Stage 4 - Setting the passwords up

For the new installer to work correctly certain passwords have to be set to be the same value. Stage 4 entails setting the passwords up correctly and ensuring that Blackboard is working with the new passwords before upgrading.
This stage took 34 Minutes in the Dev 1 Upgrade
This stage took 1 Hour and 4 Minutes in the Preprod Upgrade

All: Verify that Blackboard is stopped on all nodes.
APP_MNGR_ROLE: On each node edit the bb-config.properties file and ensure that relevant passwords are set to be the same value that has been agreed in Stage 1 above:

>> on each node
>> as bbuser

nano /usr/local/blackboard/config/bb-config.properties

>> ensure

bbconfig.database.server.systemuserpassword=PSWD_REMOVED
bbconfig.database.admin.password=PSWD_REMOVED
antargs.default.vi.db.password=PSWD_REMOVED
antargs.default.vi.stats.db.password=PSWD_REMOVED
antargs.default.vi.report.user.password=PSWD_REMOVED
bbconfig.cs.db.cms-user.pass=PSWD_REMOVED
bbconfig.cs.db.systemuser.pass=PSWD_REMOVED

>> are all set to the same password agreed in Stage 1 Step 1.
LINUX_ROLE: Verification step of above on each node:

>> as bbuser

cat /usr/local/blackboard/config/bb-config.properties | grep PSWD_REMOVED |wc -l

>> Result should equal 7.
DBA: Create duplicate back up table in first two boxes of step 3 below and in Oracle set the passwords to be the same value (PSWD_REMOVED) for the following accounts:
1. Preprod and Live
  1. “system”
  2. “sys”
  3. bb_bb60
  4. cms
  5. bbadmin
  6. bb_bb60_stats
  7. cms_doc
DBA: update passwords via SQL:

select * from cms.xy_file_systems;
_____________________________________________
Create Table cms.xy_file_systems_bak
As
Select * from cms.xy_file_systems;
_____________________________________________
UPDATE cms.xy_file_systems
SET DB_PASSWORD = 'PSWD_REMOVED'
WHERE NAME = 1;
COMMIT;
_____________________________________________
SELECT * FROM bbadmin.bb_instance;
_____________________________________________
Create Table bbadmin.bb_instance_bak
As
Select * from bbadmin.bb_instance;
_____________________________________________
UPDATE bbadmin.bb_instance
SET DB_PASS = 'PSWD_REMOVED', STAT_DB_PASS = 'PSWD_REMOVED'
WHERE PK1 = 1;
COMMIT;
DBA: Check that no database accounts are locked.

select username,
account_status
from dba_users
where lock_date is not null;
APP_MNGR_ROLE: As bbuser, run pushconfigupdates.sh to ensure all new password settings in bb-config.properties have been propagated. Blackboard will start once completed. The push should be done:

>> As bbuser
>> On each node
>> One node at a time, tasks node first, followed by web #nodes
cd /usr/local/blackboard/tools/admin/
./PushConfigUpdates.sh

>> If Blackboard does not start then

service blackboard start
1. on each node,
2. one node at a time starting with the tasks node task node
3. after checking we can log in via GUI
4. run pushconfig on web node web1
5. after checking we can log in via GUI
6. run pushconfig on next web node web2
7. repeat until all nodes have been pushed (web3 and web4)
BB_ADMIN_ROLE: Login to Blackboard on each node as they start and quickly test that it is working ok (login, add something to a course, log out.)
ALL: Note: We now have passwords set as required in Blackboard.
LINUX_ROLE: Take COLD VMware snapshot #2 on all nodes (including db and NFS).
APP_MNGR_ROLE: Stop Blackboard on each node (web nodes first (web1-web4), then tasks node (task node)) in preparation for upgrade. The web nodes can be stopped concurrently.

>> on each node
>> web nodes first, then tasks node
service blackboard stop
ALL: verify Blackboard unavailable.

BB_ADMIN_ROLE: Check via GUI
APP_MNGR_ROLE/LINUX_ROLE: service blackboard status (on each node)

Stage 5 - The upgrade

Stage 5 covers upgrading Java and upgrading Blackboard to the new version.
This section took about 18 Minutes during the Dev 1 upgrade.
This section took about 13 Minutes during the Dev 2 upgrade.
This section took about 43 Minutes during the Preprod upgrade.

Java Upgrade

LINUX_ROLE Install Java 8 JDK on each node.

>> On each node
>> As root, or sudo:
yum install java-1.8.0-oracle-devel.x86_64
Is this ok [y/N]: y
>> To check the latest version use:
yum list available | grep 'java-1.8'
LINUX_ROLE: Check java locations and alternates

>> On each node
alternatives --display java
>> We should see Java 1.8 locations e.g.
>> Current `best' version is /usr/lib/jvm/jre-1.8.0-oracle.x86_64/bin/java

First full upgrade – tasks node tasks.node.ip.removed

NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
APP_MNGR_ROLE: using shell on tasks node tasks.node.ip.removed change user to bbuser

adm su - bbuser
APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
APP_MNGR_ROLE: Run upgrade [as bbuser]

./installer.sh -c /usr/local/bbinstaller/installer.properties | tee -a bb-installer-log.txt 2>&1
BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.

tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

First "app server only" upgrade – web node web1.node.ip.removed

NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
APP_MNGR_ROLE: using on web node web1.node.ip.removed change user to bbuser

adm su - bbuser
APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]

./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.

tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Second "app server only" upgrade – web node web2.node.ip.removed

NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
APP_MNGR_ROLE: using on web node web2.node.ip.removed change user to bbuser

adm su - bbuser
APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]

./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.

tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Third "app server only" upgrade – web node web3.node.ip.removed

NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
APP_MNGR_ROLE: using on web node web3.node.ip.removed change user to bbuser

adm su - bbuser
APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]

./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.

tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Forth "app server only" upgrade – web node web4.node.ip.removed

NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
APP_MNGR_ROLE: using on web node web4.node.ip.removed change user to bbuser

adm su - bbuser
APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]

./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.

tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Stage 6 - Applying Configuration Changes

Having successfully upgraded Blackboard we must make various configuration changes. These cover changes due to Java 1.8 and fixes to known issues.
This section took about 44 Minutes during the Dev 1 upgrade.
This section took about 40 Minutes during the Dev 2 upgrade.
This section took about 1 Hour 30 Minutes during the Preprod upgrade.

LINUX_ROLE: On each node apply first part of log spam fix:

>> on each node
mkdir -p /home/bbuser/.java/.userPrefs
mkdir /home/bbuser/.java/.systemPrefs
chown -R bbuser:bbuser /home/bbuser/.java/
chmod 755 -Rv /home/bbuser/.java/
APP_MNGR_ROLE: On each node restore branding in /home/bbuser/login.zip to /usr/local/blackboard/docs/login/

>> on each node
>> as bbuser
cd /home/bbuser/;mkdir login-extracted;unzip login.zip -d login-extracted;cd login-extracted/usr/local/blackboard/docs/login;cp -n * /usr/local/blackboard/docs/login

>> ignore message "cp: omitting directory `WEB-INF'"
APP_MNGR_ROLE: on tasks node, restore data_integration directory.

>>as bbuser
>>on tasks node task node
>>verify whether
>> /usr/local/blackboard/data_integration/ folder has been wiped.
>>If not ask BB_ADMIN_ROLE to double check contents
>>If BB_ADMIN_ROLE agree it is ok, then skip the following.
cd /home/bbuser/;mkdir data_integration-extracted;unzip data_integration.zip -d data_integration-extracted;cd data_integration-extracted/usr/local/blackboard/data_integration/;cp -n /home/bbuser/data_integration-extracted/* /usr/local/blackboard/data_integration
APP_MNGR_ROLE: Restore the snapshot.* files on the tasks node (task node).

>>as bbuser
>>on tasks node task node
cd /home/bbuser/;mkdir snapshotfiles-extracted;unzip snapshotfiles.zip -d snapshotfiles-extracted;cd snapshotfiles-extracted/usr/local/blackboard/apps/snapshot/data/;cp -n snapshot.* /usr/local/blackboard/apps/snapshot/data/
APP_MNGR_ROLE: on tasks node, restore PHOTOS directory.

>>as bbuser
>>on tasks node task node
>>verify whether
>> /usr/local/blackboard/content/PHOTOS/ folder has been wiped.
>>If not ask BB_ADMIN_ROLE to double check contents
>>If BB_ADMIN_ROLE agree it is ok, then skip the following.
cd /home/bbuser/
mkdir PHOTOS-extracted
unzip PHOTOS.zip -d PHOTOS-extracted
cd /usr/local/blackboard/content/PHOTOS/
>>if data_integration directory has been removed, create a new one
mkdir /usr/local/blackboard/content/PHOTOS/
cp -n /home/bbuser/PHOTOS-extracted/* /usr/local/blackboard/content/PHOTOS/
APP_MNGR_ROLE: In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FS, make following change to /usr/local/blackboard/apps/tomcat/conf/wrapper.conf on each node:

>> as bbuser
>> On each node.
>> Backing up file.
>> Back up original wrapper.conf to /home/bbuser
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/wrapper.conf.bb.$(date +"%Y-%m-%d")
>> As bbuser
>> On each node
cd /usr/local/blackboard/config/tomcat/conf/
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf
>> find the line
wrapper.shutdown.timeout=300
>> and change it to
wrapper.shutdown.timeout=30
>> find the line
wrapper.jvm_exit.timeout=15
>> and change it to
wrapper.jvm_exit.timeout=5
LINUX_ROLE: Verification of above on each node

>>On each node
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf |grep wrapper.shutdown.timeout ; cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf |grep wrapper.jvm_exit.timeout
>> result should be
wrapper.shutdown.timeout=30
wrapper.jvm_exit.timeout=5
APP_MNGR_ROLE: In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FS, make the following change to /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb on each node:

>> as bbuser
>> On each node.
>> Backing up file.
>> Back up original wrapper.conf.bb to /home/bbuser
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/wrapper.conf.bb.$(date +"%Y-%m-%d")
>> As bbuser
>> On each node
cd /usr/local/blackboard/config/tomcat/conf/
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb
>> find the line
wrapper.shutdown.timeout=300
>> and change it to
wrapper.shutdown.timeout=30
>> find the line
wrapper.jvm_exit.timeout=15
>> and change it to
wrapper.jvm_exit.timeout=5
LINUX_ROLE: Verification of above on each node

>>On each node
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb |grep wrapper.shutdown.timeout ; cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb |grep wrapper.jvm_exit.timeout
>> result should be
wrapper.shutdown.timeout=30
wrapper.jvm_exit.timeout=5
APP_MNGR_ROLE: On each node update the http-cacheable-extensions.txt to include reference to resolve issues with Equation editor.

>> on each node
>> as bbuser
cd /usr/local/blackboard/config/internal/
nano http-cacheable-extensions.txt
>> If not already present, add the following to the list in their alphabetical place:
eot
ttf
woff
>> and save it.
LINUX_ROLE: Verification of above on each node

>> As bbuser
>> On each node
grep 'eot\|ttf\|woff' /usr/local/blackboard/config/internal/http-cacheable-extensions.txt | wc -l
>> Result must equal 3.
APP_MNGR_ROLE: As bbuser edit bb-config.properties.template on each node and make the following change.

>> as bbuser
>> On each node.
>> Backing up file.
>> Back up original bb-config.properties.template to /home/bbuser
cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/bb-config.properties.template.$(date +"%Y-%m-%d")
>> as bbuser
>> on each node
nano /usr/local/blackboard/config/bb-config.properties.template
>> Update click jacking setting to resolve issue where
>> SCORM learning objects suffer disconnections
>> Blackboard case #02455370
>> Change
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self
>> to
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

>>Note that this is the last line in the file
LINUX_ROLE: Verification of above on each node.

>> on each node
cat /usr/local/blackboard/config/bb-config.properties.template | grep "ancestors=self'"
>> Result should be bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
APP_MNGR_ROLE: As bbuser edit the bb-config.properties file on each node and make the following five changes.

>> Backing up file.
>> On each node back up original bb-config.properties to /home/bbuser
cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/bb-config.properties.$(date +"%Y-%m-%d")
>> as bbuser
>> on each node
nano /usr/local/blackboard/config/bb-config.properties
_______________
>> Change 1
>> Change bbconfig.max.stacksize.tomcat=
>> from 400k to 512k i.e.
bbconfig.max.stacksize.tomcat=512k
_______________
>> Change 2
>> Change bbconfig.jvm.options.extra.tomcat=
>> So that whole section is replaced with
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/
>> N.B. Be careful of line breaks
>> N.B. copy from SysDoc, print out will not show full text
_______________
>> Change 3
>> Change bbconfig.jvm.options.gc=
>> So that whole section is replaced with
bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
>> N.B. Be careful of line breaks
>> N.B. copy from SysDoc, print out will not show full text
_______________
>> Change 4
>> Check dmarc setting, ensure that:
bbconfig.email.use.dmarc.from.override=false
_______________
>> Change 5
>> Update click jacking setting to resolve issue where
>> SCORM learning objects suffer disconnections
>> Blackboard case #02455370
>> Change
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self
>> to
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

>>Note that this is the last line the file.
LINUX_ROLE: Verification, in either batch or individually...
LINUX_ROLE: Batch verification steps that above five changes have been made successfully, perform on each node.

>> On each node
echo "Change 1";cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.max.stacksize.tomcat";echo "Change 2";cat /usr/local/blackboard/config/bb-config.properties |grep "Djava";echo "Change 3";cat /usr/local/blackboard/config/bb-config.properties |grep UseG1GC;echo "Change 4";cat /usr/local/blackboard/config/bb-config.properties | grep "dmarc.from.override";echo "Change 5";cat /usr/local/blackboard/config/bb-config.properties | grep "ancestors=self'"

>> Should result with:
Change 1
bbconfig.max.stacksize.tomcat=512k
Change 2
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/
Change 3
bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
Change 4
bbconfig.email.use.dmarc.from.override=false
Change 5
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
LINUX_ROLE: Individual verification steps that above five changes have been made successfully on each node.[Expand]
APP_MNGR_ROLE: Apply xythos cross server communication fix on each node.

>> On each node
>> Backup file
cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/server.xml.bb.$(date +"%Y-%m-%d")
>> On each node
>> Edit /usr/local/blackboard/config/tomcat/conf/server.xml.bb
nano /usr/local/blackboard/config/tomcat/conf/server.xml.bb
>> find the stanza (near end of file around lines 203 - 207) defining the xythos admin connector and add secure="true" i.e.
>> Currently it is:
<Connector port="@bbconfig.xythosremoteadmin.xythos.servergroup.portnumber@"
protocol="org.apache.coyote.http11.Http11NioProtocol"
executor="crossServerExecutor"
backlog="@@bbconfig.xythosremoteadmin.connector.backlog@@"
enableLookups="false"
keepAliveTimeout="@@bbconfig.appserver.http.keepalive.timeout@@"
maxKeepAliveRequests="@@bbconfig.appserver.http.keepalive.requests@@"
maxPostSize="@@bbconfig.appserver.max.post.size@@"
maxParameterCount="@@bbconfig.appserver.max.parameter.count@@"
soTimeout="@@bbconfig.appserver.http.so.timeout@@"
URIEncoding="UTF-8"/>
>> Change to:
<Connector port="@bbconfig.xythosremoteadmin.xythos.servergroup.portnumber@"
protocol="org.apache.coyote.http11.Http11NioProtocol"
executor="crossServerExecutor"
backlog="@@bbconfig.xythosremoteadmin.connector.backlog@@"
enableLookups="false"
keepAliveTimeout="@@bbconfig.appserver.http.keepalive.timeout@@"
maxKeepAliveRequests="@@bbconfig.appserver.http.keepalive.requests@@"
maxPostSize="@@bbconfig.appserver.max.post.size@@"
maxParameterCount="@@bbconfig.appserver.max.parameter.count@@"
soTimeout="@@bbconfig.appserver.http.so.timeout@@"
URIEncoding="UTF-8"
secure="true"/> <----- Part to add
LINUX_ROLE: Verification step of above on each node.

>> on each node
cat /usr/local/blackboard/config/tomcat/conf/server.xml.bb |grep secure
secure="true"
secure="true"
secure="true"/>
APP_MNGR_ROLE and DBA: Apply workaround to issue LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh by adding lines to /usr/local/blackboard/config/internal/xythos-indexing-filter.txt detailed here: MicrosoftContentParser failure - java.lang.Exception
APP_MNGR_ROLE: Edit the /usr/local/blackboard/config/service-config.properties on all nodes EXCEPT tasks node to ensure that tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.

As bbuser
on all nodes except tasks node (web1,web2,web3,web4)
Edit the file /usr/local/blackboard/config/service-config.properties

nano /usr/local/blackboard/config/service-config.properties

and alter the line
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2
to read
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0

>> and change logging level to fatal on all nodes but tasks node

blackboard.service.log.param.logdef.services.verbosity=fatal
LINUX_ROLE: Verification of above step:

>>On web nodes (web1,web2,web3,web4)
cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.queuedtaskmanager.param.num-concurrent-tasks ;cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.log.param.logdef.services.verbosity

>>should return
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0
blackboard.service.log.param.logdef.services.verbosity=fatal

On tasks node (task node)
cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.queuedtaskmanager.param.num-concurrent-tasks ;cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.log.param.logdef.services.verbosity

>>should return
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2
blackboard.service.log.param.logdef.services.verbosity=warning
APP_MNGR_ROLE: As bbuser, run pushconfigupdates.sh on each node, tasks node (task node) first, and having checked we can login via the GUI, continue with web nodes (web1-web4) to ensure all new settings in bb-config.properties have been propagated. Note that we are exporting the output to a log file for investigation if required. Blackboard will start once completed.

>> As bbuser
>> Tasks node first
cd /usr/local/blackboard/tools/admin/
./PushConfigUpdates.sh | tee -a /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d") 2>&1

>> On big tv
tail -f /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d")
>>during the process

>>check BB_ADMIN_ROLE can login through GUI
>>then web nodes, one at a time after checking BB_ADMIN_ROLE can log into each one
cd /usr/local/blackboard/tools/admin/
./PushConfigUpdates.sh | tee -a /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d") 2>&1

>> On big tv
tail -f /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d")
>>during the process
ALL: In the Dev 1 upgrade this failed due to the setting for bbconfig.database.stats.maxpoolsize being reset by pushconfigupdates.sh to bbconfig.database.stats.maxpoolsize=@bbconfig.database.stats.maxpoolsize@ We reset this back to We reset this back to the value shown in our backup and then pushconfig succeeded.
ALL: In the Dev 2 upgrade this failed due to the setting for bbconfig.max.heapsize.tomcat being reset by pushconfigupdates.sh to bbconfig.max.heapsize.tomcat=@bbconfig.max.heapsize.tomcat@ We reset this back to the value shown in our backup and then pushconfig succeeded.
BB_ADMIN_ROLE: Restore login page by

>> As bbuser
>> On tasks node task node
cd /usr/local/blackboard/tools/admin/
./AuthenticationOneTimeLogin.sh -u administrator
>> Login with link and pass provided
Then navigate to System Admin, Brands and Themes, Customise Login page, Replace Customised Page
Upload latest login page held in J:\iSolutions\Service Delivery\eLearning\Blackboard\Configuration\front_page
>> If required update gateway page
/usr/local/blackboard/content/vi/bb_bb60/branding/
nano gateway.bb
Check you can log in
APP_MNGR_ROLE: Restart ActiveMQ on tasks node to ensure that it is running correctly before we begin testing.

>> as bbuser
>> on tasks node
cd /usr/local/blackboard/tools/admin
./ServiceController.sh services.activemqbroker.restart
LINUX_ROLE: Take COLD VMware snapshot #3 (including db and NFS server).

Stage 7 – GUI configuration changes and UAT

This stage encompasses changes made by BB_ADMIN_ROLE team via the GUI and testing.

BB_ADMIN_ROLE: Stop Blackboard on all nodes except for tasks node.

>>On all nodes, except tasks node.
service blackboard stop
BB_ADMIN_ROLE: Uninstall building blocks.
BB_ADMIN_ROLE: Make unavailable the following unused building blocks to save Java RAM usage:

>>as bbuser
cd /usr/local/blackboard/tools/admin
./B2Manager.sh -s UNAVAILABLE bbgs-nbc-content-integration
./B2Manager.sh -s UNAVAILABLE bb-word-cleanup
./B2Manager.sh -s UNAVAILABLE bb-auth-provider-cas
./B2Manager.sh -s UNAVAILABLE bb-auth-provider-shibboleth
./B2Manager.sh -s UNAVAILABLE bb-data-integration-ims-xml
./B2Manager.sh -s UNAVAILABLE bb-data-integration-lis
./B2Manager.sh -s UNAVAILABLE bb-data-integration-lis-final
./B2Manager.sh -s UNAVAILABLE bb-data-integration-ss-xml
BB_ADMIN_ROLE: Make inactive the following unused building blocks to save Java RAM usage:

>>as bbuser
cd /usr/local/blackboard/tools/admin
./B2Manager.sh -s INACTIVE bb-cx-ext-angel-plgnhndl
./B2Manager.sh -s INACTIVE bb-cx-ext-ce4-plgnhndl
./B2Manager.sh -s INACTIVE bb-cx-ext-vista-plgnhndl
./B2Manager.sh -s INACTIVE bb-ce4-ext-plgnhndl
./B2Manager.sh -s INACTIVE bb-vista-ext-plgnhndl
./B2Manager.sh -s INACTIVE bb-cookie-disclosure
./B2Manager.sh -s INACTIVE bbgs-nbc-content-integration
./B2Manager.sh -s INACTIVE bb-word-cleanup
./B2Manager.sh -s INACTIVE bb-auth-provider-cas
./B2Manager.sh -s INACTIVE bb-auth-provider-shibboleth
./B2Manager.sh -s INACTIVE bb-data-integration-ims-xml
./B2Manager.sh -s INACTIVE bb-data-integration-lis
./B2Manager.sh -s INACTIVE bb-data-integration-lis-final
./B2Manager.sh -s INACTIVE bb-data-integration-ss-xml
BB_ADMIN_ROLE: On tasks node upload the following Building Blocks via GUI.

[removed]
BB_ADMIN_ROLE: Check Panopto settings:

>>The following needs to be unticked
Blackboard course copy also copies Panopto permissions
>>Note this is last setting on the page.

>>These settings should already be set
User Opt In – Yes
Email Instructors - No
Redirect to Blackboard Default Login – No
Grant TA Creator Access – Yes
Allow TAs to Create Panopto Course Tool Links – No
Grant TA Provisioning Access – Yes
Provisioning by administrators only – No
Panopto link on course menu when provisioned – No
Panopto Link Text - Recorded Sessions
Instructors may create folders - No
Allow all courses to be reset – No
Verbose logging – No
Assign custom role mappings – [BLANK]
BB_ADMIN_ROLE: Check that Rubrics Building Block is available.
BB_ADMIN_ROLE: Restart Blackboard on Tasks node.

service blackboard restart
BB_ADMIN_ROLE: Update the "partner cloud" building block via GUI
BB_ADMIN_ROLE: Restart Blackboard on Tasks node.

service blackboard restart
BB_ADMIN_ROLE: Update the "Blackboard Mobile Web Services" building block via GUI
BB_ADMIN_ROLE: Restart Blackboard on Tasks node.

service blackboard restart
BB_ADMIN_ROLE: Check Blackboard Mobile Web Services Building Block > Server Registration Settings:
1. Environment: Production
2. Authentication Type: Native
BB_ADMIN_ROLE: Update the "Dropbox Integration" building block via GUI
BB_ADMIN_ROLE: Make Blackboard Collaborate and Collaborate Ultra Building Blocks inactive.
BB_ADMIN_ROLE: Make Dropbox Integration Building Block Unavailable.
BB_ADMIN_ROLE: Restart Blackboard on Tasks node.

service blackboard restart
BB_ADMIN_ROLE: Check you can log in to Tasks node via GUI.
BB_ADMIN_ROLE: Start Blackboard on web nodes, one node at a time. Check you can log in before starting the next node.

>>as bbuser
>>on web nodes, one node at a time. Check you can log in before starting the next node.
service blackboard start
BB_ADMIN_ROLE:Setup Qwickly
1. Building Blocks > Qwickly > Settings
2. Set ID as [removed]
3. Set this is a Production System
4. Tick:
  1. Course Availability
  2. Post announcement
  3. Send Email/
  4. Don't tick email instructors
  5. Untick Needs Grading
  6. Tick Teaching Assistant
5. Submit Settings
BB_ADMIN_ROLE:Setup Qwickly link in Tool Panel.
1. System Admin > Tabs and Modules > Tool Panel
2. Check Enterprise Surveys, Goals, Goal Performance, Observer Dashboard, Academic Materials, User directory, Address book are all unavailable.
3. Add Tool
4. Name: "Instructor Quick Tools"
5. Tool: Qwickly
BB_ADMIN_ROLE: Check Spell checking settings:
1. SysAdmin Panel
2. Spell Check Dictionaries
3. Check UK English set as default
4. Make US English unavailable
5. Make Spanish unavailable
BB_ADMIN_ROLE: Enable new language pack
1. SysAdmin Panel
2. Language Packs
3. Make Norweigian available
BB_ADMIN_ROLE: Update Tool Settings:
1. SysAdmin Panel
2. Tools ->
3. Dropbox Intergration -> Set all off
4. Goal -> Set all off
5. Open Photo Roster
  1. Course Tool > ON > UNLOCK > New and existing
6. Pearson Custom Integration
  1. Set all to Off > Unlock > New and existing
7. Pearson's MyLab & Mastering
  1. Set all to Off > Unlock > New and existing
8. Qwickly:
  1. Qwickly Tools Course Tool - Off
  2. Qwickly Attendance - Off and Lock
  3. Qwickly Cloud Assignment - Off and Lock
  4. Qwickly Cloud Link VTBE Mashup - Off and Lock
  5. Qwickly Take Attendance - Off and Lock
  6. Qwickly Attendance - Off and Lock
  7. Qwickly Attendance - Off and Lock
  8. Qwickly Tools (System Tool) - On
BB_ADMIN_ROLE: Disable Open Photo Roster on the following large courses by
1. enter course
2. click Control Panel -> Customisation -> Tool Availability
3. find Open Photo Roster
4. untick box
5. submit
6. Follow this link for list of courses on which tool should be disabled:
BB_ADMIN_ROLE: Enable "forgot your password?" link.
1. Sys Admin
2. Gateway Options
3. Request Forgotten Password > Enable
4. URL for Forgotten Password: https://subscribe.soton.ac.uk/
BB_ADMIN_ROLE: Restart ActiveMQ on tasks node to ensure that it is running correctly before we begin testing.

>> as bbuser
>> on tasks node
cd /usr/local/blackboard/tools/admin
./ServiceController.sh services.activemqbroker.restart
BB_ADMIN_ROLE: UAT Blackboard via GUI on all nodes.

Stage 8 – Restart Test

In this stage we check that Blackboard restarts ok.

LINUX_ROLE: Perform vApp stop of Blackboard nodes in line with standard method used for patching.
LINUX_ROLE: Perform vApp start of Blackboard nodes in line with standard method used for patching.
APP_MNGR_ROLE: Once Blackoard available again, restart ActiveMQ on tasks node to ensure that it is running correctly.

>> as bbuser
>> on tasks node
cd /usr/local/blackboard/tools/admin
./ServiceController.sh services.activemqbroker.restart
BB_ADMIN_ROLE: Check you can log into web nodes.
BB_ADMIN_ROLE: Make a non-immediate announcement on each node, to test later that ActiveMQ is working. Emails should appear within an hour.
BB_ADMIN_ROLE: On each node, add a new item to a test course and check it appears ok.

Stage 9 – Wash-up

This stage involves tidying up after successful testing.

DBA: Reset DB passwords for sys, system; remove duplicate table back up from passwords section above.
APP_MNGR_ROLE: Remove backups of zip files and uncompressed directories.

>> As bbuser
On each node
rm /home/bbuser/login.zip
rm -rf /home/bbuser/login-extracted
>> On tasks node
rm /home/bbuser/snapshotfiles.zip
rm /home/bbuser/data_integration.zip
rm -rf /home/bbuser/data_integration-extracted
rm -rf /home/bbuser/snapshotfiles-extracted
APP_MNGR_ROLE: Remove copied licence file.

>> As bbuser
rm /home/bbuser/blackboard-license.xml
APP_MNGR_ROLE: Remove installer directory.

>> As root
rm -rf /usr/local/bbinstaller
BB_ADMIN_ROLE: Remove password from this documentation.

Stage 10 – GO-LIVE

LINUX_ROLE: Following confirmation by BB_ADMIN_ROLE, make Blackboard live via NLB.
BB_ADMIN_ROLE: Announce service availability via twitter @uos_BB_ADMIN_ROLE and blackboard-users@lists mailing list.
LINUX_ROLE: Assuming all successful, and with agreement of BB_ADMIN_ROLE, delete snapshots.

Summary of VMWare Snapshots

Note all snapshots are "cold" and include all nodes, db, and NFS.

Snapshot #1 – No changes have taken place. Snapshot included all nodes including db and NFS.

Snapshot #2 – Preliminary pre-upgrade work complete, passwords set correctly, Blackboard ready for upgrade.

Snapshot #3 – Blackboard upgraded but GUI based config not yet started.

Search This Blog

VLE Guru