Upgrading two dev environments to 2018 Q4

Originally posted on Feb 15, 2019.

In the last fortnight we have upgraded two development environments to 2018q4

One server was previously on 2018 Q2 CU1, the other was on 2017 Q4 CU3.

Here are some notes on our experiences.

Dev 1: 2018 Q2 CU1

The upgrade went smoothly.  However after the upgrade we found that all user avatars had been wiped.  It turns out that there is a new task in bb-tasks.xml which moves avatars from /usr/local/blackboard/content/vi/bb_bb60[or BBLEARN1]/avatar/ to /usr/local/blackboard/content/storage/ 

We have a ticket open about this.  The issue did not happen on "Dev 2" so we look forward to see what happens with preprod next week.

Dev 2: 2017 Q4 CU3.

The upgrade failed with a ORA-00001 error.  This was caused by the installer expecting to use a PK1 in the ENTITLEMENT table that had already been used due us running a Blackboard provided fix to this issue:   Missing Navigation Items on the System Admin Panel We had to remove that row and then the upgrade ran ok.  It even added the missing ENTITLEMENT back in itself.

Other findings:

bbconfig.jvm.options.codecache.reserved=

bbconfig.jvm.options.codecache.initial=

in bb-config.properties has been changed to

bbconfig.jvm.options.initial.code.cache.size=

bbconfig.jvm.options.reserved.code.cache.size=

We found out from Blackboard that the recommended settings are:

bbconfig.jvm.options.initial.code.cache.size=256m

bbconfig.jvm.options.reserved.code.cache.size=384m

We also noticed that ‘txt’ has been added to the list of /usr/local/blackboard/config/internal/http-cacheable-extensions.txt  - we apply a fix to this file to resolve an issue with the Math Editor not working in IE (have to add ttf to the list) so even though IE is not supported in 2018 Q4 we will continue to apply the fix, and have updated out plan to account for the new entry.

There are lots of new entries in bb-config.properties.

Most useful to us were:

bbconfig.cloud.lms.region=

For this we were told to use the relevant region from this site: Regions and Availability Zones - Amazon Relational Database Service

bbconfig.xythos.support.email=

This should stop those log spams about the email not being configured!

Passwords are not encyrpted in the bb-config.properties file which is cool.

We also applied a fix to this issue:

Article #: 000050183

Article Title: Course Copies Stuck

Article Summary: Course Copies become stuck with increasing frequency.

Relevant Version(s):

Learn 9.1 Q4 2018 (3500.0.0)

Learn 9.1 Q2 2019

SaaS

Which has since been removed.  The fix was to comment out the bb connect stanza in bb-tasks.xml and bb-tasks.xml.bb.  One of my colleagues has written a script to comment out xml stanzas to help us to do that quickly.  I can post it if anyone is interest.

For reference here are all the other new entries I found - I may have missed some:

bbconfig.password.encryption.enabled=true

## Max Threshold for single cache key  ##

bbconfig.cache.token.threshold.single.count=500

## Max Threshold for total cache keys  ##

bbconfig.cache.token.threshold.total.count=10000

## When running behind a load balancer, this indicates the number

## of seconds that a session will be pinned to a single node

## before rebalancing is allowed.

bbconfig.frontend.sticky.session.timeout=900

##  This is used to identify whether we add "X-Blackboard-Product" ##

##  "X-Blackboard-Appserver" and "Xythos-Wfs-Version" in http      ##

##  response headers. Please take note there is potential of       ##

##  custom client issues if set to false                           ##

bbconfig.appserver.server.tokens.enabled=true

##  External IP that other cluster nodes should use to connect to  ##

##  this node. If unset, defaults to the IP associated with        ##

##  bbconfig.appserver.machinename.                                ##

bbconfig.appserver.external.address=

## The Cache-Control response header value for static resources.   ##

bbconfig.appserver.http.staticresources.cachecontrol=max-age=300,must-revalidate

####################### xythos support email ########################

##   This address will receive critical error notification         ##

##   emails from the system                                        ##

#####################################################################

bbconfig.xythos.support.email=

#####################################################################

## This config applies to all connection pools except Xythos/DW.   ##

## JDBC intercepters allow optimizations and debugging.            ##

#####################################################################

bbconfig.database.jdbc.interceptors=blackboard.db.impl.SchemaCacheInterceptor

## Amount of time before a hung JVM is killed. If not set,         ##

## falls back to wrapper default of 60 seconds.                    ##

## If this value is set too low, heapdumps won't be captured since ##

## JVM is killed before dump completes. If the value is set too    ##

## high an unresponsive JVM will hang for longer before it is      ##

## restarted and is dependent on load balancer to remove from pool ##

bbconfig.jvm.ping.timeout.seconds=60

## This property sets the "too small to care about" size (in bytes). If a file is small, xythos doesn't need to check the

## db/blobs for duplicates. The higher this number the less unnecessary thread use and locking, but the more potential

## for duplicate blobs in the db

bbconfig.cs.min.duplicate.check.size=5

## The pipe-separated list of the paths of the files, that won't be indexed within the full-text index (e.g. Lucene).

## The paths can contain '*'-wildcards to match a set of files.

## E.g.: to exclude the entire /internal content as well as every entry that contains the "do,not,index" in its name:

##   bbconfig.cs.fulltext.excluded=/internal/*|*do,not,index*

bbconfig.cs.fulltext.excluded=/internal/*

bbconfig.cloud.lms.region=

#####################################################################

##        Xythos Signed URL config                                 ##

## This feature allows content to be delivered from cloud storage  ##

## directly via signed URLs to offload repeat requests for the     ##

## same files, especially range requests.                          ##

#####################################################################

bbconfig.appserver.xythos.s3.signed.urls.enabled=true

bbconfig.appserver.xythos.s3.signed.urls.expiration.min=180

bbconfig.appserver.xythos.s3.signed.urls.extra.mimetype.exclusions=

# number of concurrent connections that the system can make to S3 to access the content of the files. Default value is 200.

bbconfig.appserver.xythos.s3.conn.pool.size=200

# time in milliseconds after which WFS deactivates an S3 client and creates a new one. Default value is 15 minutes (15*60*1000)

bbconfig.appserver.xythos.s3.client.reload.ms=900000

#####################################################################

##        Xythos Signed URL config                                 ##

## This feature allows content to be delivered from cloud storage  ##

## directly via signed URLs to offload repeat requests for the     ##

## same files, especially range requests.                          ##

#####################################################################

bbconfig.appserver.xythos.s3.signed.urls.enabled=true

bbconfig.appserver.xythos.s3.signed.urls.expiration.min=180

bbconfig.appserver.xythos.s3.signed.urls.extra.mimetype.exclusions=

#####################################################################

## This forces the Xythos logging on startup to this value and     ##

## and mirrors the logging that we see on the Xythos logging page: ##

## /webapps/cmsadmin/bb_logs_parameters.jsp                        ##

##                                                                 ##

## Options not specified will be disabled.                         ##

##                                                                 ##

## Changes can still be made via above page, but will be reset on  ##

## when any node is restarted or new node started.                 ##

##                                                                 ##

## If left blank, the value will not be modified.                  ##

##

## Recommended settings are:

## Xythos.Log.Info

##

## The full list of options is:

## Xythos.Log.LockTrace,Xythos.Log.SubscriptionsTrace

## Xythos.Log.SqlReqTrace,Xythos.Log.PropertyTrace,

## Xythos.Log.WebDavTrace,Xythos.Log.EventTrace,

## Xythos.Log.AdminTrace,Xythos.Log.CloudStorageTrace,

## Xythos.Log.MiscTrace,Xythos.Log.SearchTrace,

## Xythos.Log.JdbcConnPoolTrace,Xythos.Log.FileSysEntryTrace,

## Xythos.Log.XythosUserTrace,Xythos.Log.CustomTrace,

## Xythos.Log.XythosSessionTrace,Xythos.Log.ExternStoreTrace,

##

#####################################################################

bbconfig.appserver.xythos.logging.options=Xythos.Log.Info

#####################################################################

##                         bb-connect config                       ##

#####################################################################

bbconfig.connect.base.url=https://www.blackboardconnected.com

#####################################################################

##  Controls thread count during content parsing.                  ##

#####################################################################

bbconfig.appserver.xythos.parser.threads=1

#********************************************************************

# Wrapper Invocation Properties

#********************************************************************

bbconfig.wrapper.max_failed_invocations=100

#********************************************************************

# Attendance Service Properties

#********************************************************************

bbconfig.attendance.service.update.task.time.interval=900000

#********************************************************************

# Exclude Download Request Resources

# The value would be extension of file name, eg: .doc, .jpg or .txt

# The "." is required here

#********************************************************************

bbconfig.cloud.metric.request.exclude.resources=.jpg,.gif,.txt,.ico,.png,.doc,.zip,.docx,.pdf

#####################################################################

##                    cloud storage integration                    ##

#####################################################################

bbconfig.clouddrive.onedrive.clientid=000000004013A71B

bbconfig.clouddrive.onedrive.redirecturl=ultra/features/base/cloudstorages/onedrive-callback.html

bbconfig.clouddrive.kloudless.urls.public=

bbconfig.clouddrive.kloudless.clientid=

bbconfig.clouddrive.kloudless.disabled=false

#####################################################################

##                    Data Warehouse Properties                     #

## The following bbconfig.database.dw properties are supported in   #

## SaaS instances only and should not be set by clients.            #

#####################################################################

bbconfig.database.dw.user=

bbconfig.database.dw.password=

bbconfig.database.dw.dbname=

bbconfig.database.dw.account=

bbconfig.database.dw.ddl.account=

bbconfig.database.dw.domain=snowflakecomputing.com

bbconfig.database.dw.enabled=false

bbconfig.database.dw.initpoolsize=1

bbconfig.database.dw.minpoolsize=1

bbconfig.database.dw.maxpoolsize=5

bbconfig.database.dw.timeout=300

#####################################################################

## The following boolean properties control ETL processes that can  #

## be disabled if the Data Warehouse (the .dw properties above) is  #

## used.                                                            #

#####################################################################

bbconfig.etl.enabled=true

bbconfig.update.stats.asr.tables=true

Next week we will upgrade our preprod environment which is much more complex.

Special thanks to Andrew Hulme and Cherif Abbes for their help!