Upgrading our first dev server to 9.1 Q4 2016

Originally posted on Feb 8, 2017

As most will know, since 9.1 Q2 2016 Blackboard uses a new installer.  We had tried this in our upgrade last year but were not able to get the new installer to work.

So this time I reached out to Andrew Hulme at Blackboard to see if we could get some extra support during our upgrade.  Andy brought in his colleague Cherif Abbes and we all joined a Skype session where we could ask questions if we hit against any issues whilst doing the upgrade.  Special thanks to Andy and Cherif for their help!

The upgrade went well after a few bumps and starts.  Here are notes on what we did, and what we found, and what problems we still need to resolve.

Note that we have two development servers, both are single instances with Blackboard and Oracle running on the same box.  They have very little data on them and are used for testing purposes.  Our preproduction environment matches our live environment and has much more data.

What we had already knew (the known knowns)

From the mailing lists, the community here and the guide at https://en-us.help.blackboard.com/Learn/Administrator/Hosting/Install_or_Upgrade_Learn/Upgrade_Learn/Perform_the_Upgrade we already knew:

1. The installer properties file has to point at a licence key held outside of the Blackboard directory

cp /usr/local/blackboard/config/license/blackboard-license.xml /home/bbuser/

and in installer properties:

bbconfig.file.license=/home/bbuser/blackboard-license.xml

2. The blackboard files under /usr/local/blackboard have to be owned by bbuser

To resolve:

cd /usr/local/blackboard find . ! -user bbuser -o ! -group bbuser | xargs -I {} chown bbuser\: {}

3. The installer usually wipes out our branding, so we need to backup our branding files before running the installer:

cd /home/bbuser

zip -r login.zip /usr/local/blackboard/docs/login/*

and after the upgrade to restore...

cd /home/bbuser/

mkdir login-extracted

unzip login.zip -d login-extracted

cd login-extracted/usr/local/blackboard/docs/login

cp -n * /usr/local/blackboard/docs/login

4 The minimum required values for the installer properties file are at https://en-us.help.blackboard.com/Learn/Administrator/Hosting/Install_or_Upgrade_Learn/Install_Learn/Preparing_Installation_Properties#Upgrade

5) Since we are upgrading from a version before 2016 Q2 we have to upgrade to Java 1.8.  Here's how we did it in our environment:

As root, or sudo:

yum install java-1.8.0-oracle-devel.x86_64

Is this ok [y/N]: y

To check the latest version use:

yum list available | grep 'java-1.8'

Then Check java locations and alternates

alternatives --display java

We should see Java 1.8 locations e.g.

Current `best' version is /usr/lib/jvm/jre-1.8.0-oracle.x86_64/bin/java

6. We have to put the Java location in the installer properties file.

bbconfig.java.home=/usr/lib/jvm/java-1.8.0-oracle.x86_64/

7. We had to set JAVA_HOME and we found out that we have to run the upgrade in the same shell instance as we had set JAVA_HOME in.

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/

echo $JAVA_HOME

What we learned through doing our first upgrade (the unknown unknowns that became known)

1. Passwords matter. Cherif gave us this gem which has really helped us to understand:

"antargs.default.vi.db.password=should match the password in bb-config.properties for antargs.default.vi.db.password

This password should be the one for the account under bbconfig.database.identifier=account_name  in Oracle

and should also match the password in bb-config.properties for bbconfig.database.server.systemuserpassword"

2. There are two key log files to check whilst the upgrade is running:

/usr/local/bbinstaller/bb-installer-log.txt

[note] your location may vary, depending on where you place the installer, just make sure not to put it in your /blackboard directory

/usr/local/blackboard/logs/update-tools/update-tool-log.txt

3. We found ulimit errors in the upgrade log. So we will follow Blackboard's recommendation to raise the ulimit in our next dev server upgrade.

4. Although the issue Repeated Data Integration Lucene Indexing Log Messages appears to be resolved in Q4 2016, if you have the issue before the upgrade it will continue.  So check /usr/local/blackboard/logs/bb-services-log and remove the write lock files if necessary before upgrading.

5. We had no login page when we started Blackboard. We had to use the “one time login script" to get in:

cd /usr/local/blackboard/tools/admin/

./AuthenticationOneTimeLogin.sh -u administrator

We were then able to upload our custom login page and login as normal.

6. The new ActiveMQ JDBC persistence seems awesome and you can restart it without downtime:

cd /usr/local/blackboard/tools/admin

./ServiceController.sh services.activemqbroker.restart

Having found out this information we were well placed to start planning our next dev server upgrade.