Highlights of fixes from our upgrade implementation plan

Originally posted on Jun 23, 2017

Our implementation plan is quite long and hard to skim over.  Here I've tried to highlight some of the interesting fixes that we have included that may be of particular interest to others.
Note that some additional fixes are listed here: A successful upgrade

File ownership

Blackboard files should be owned by bbuser, otherwise upgrades will fail.
>> On each node 
find /data/bb/blackboard/ -maxdepth 1 -mindepth 1 -type d -not -name content | xargs -I {} chown -R bbuser\: {}

Ulimit

The upgrade produced errors due to ulimit being too low. Therefore we should raise the ulimit as follows:
>> On each node 
>> Login as root  
-     vim /etc/security/limits.conf 
-     add following lines:
 *               soft    nofile          5000
 *               hard    nofile          65536
  >> Pay attention that # does not appear in front of either line
 -     save
 -     logout 
 -     login  
>> as root check: 
 -     ulimit -n 
 -     ulimit -Hn  
[root@~]# ulimit -n 5000 
[root@~]# ulimit -Hn 65536  
>> switch to bbuser and check is the same result:
 -     su bbuser
 -     ulimit -n
 -     ulimit -Hn
[bbuser@]$ ulimit -n 
5000 
[bbuser@]$ ulimit -Hn 
65536

Java Log Spam fix

To prevent log spam in the stdout-stderr log perform the following:
 >> on each node 
mkdir -p /home/bbuser/.java/.userPrefs 
mkdir /home/bbuser/.java/.systemPrefs 
chown -R bbuser:bbuser /home/bbuser/.java/ 
chmod 755 -Rv /home/bbuser/.java/  
 >> on each node 
 >> within /usr/local/blackboard/config/bb-config.properties 
 >> add to bbconfig.jvm.options.extra.tomcat= 
-Djava.util.prefs.systemRoot=/home/bbuser/.java/

Speed up Blackboard start and stop

In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FSmake following change to /usr/local/blackboard/apps/tomcat/conf/wrapper.conf on each node:
>> As bbuser 
>> On each node 
cd /usr/local/blackboard/config/tomcat/conf/ 
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf 
>> find the line 
wrapper.shutdown.timeout=300 
>> and change it to 
wrapper.shutdown.timeout=30 
>> find the line 
wrapper.jvm_exit.timeout=15 
>> and change it to 
wrapper.jvm_exit.timeout=5

And make the following change to /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb on each node:
>> As bbuser 
>> On each node 
cd /usr/local/blackboard/config/tomcat/conf/ 
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb 
>> find the line 
wrapper.shutdown.timeout=300 
>> and change it to 
wrapper.shutdown.timeout=30 
>> find the line 
wrapper.jvm_exit.timeout=15 
>> and change it to 
wrapper.jvm_exit.timeout=5

Resolve Math Editor not loading issue

On each node update the http-cacheable-extensions.txt to include reference to resolve issues with Equation editor.
>> on each node 
>> as bbuser 
cd /usr/local/blackboard/config/internal/ 
nano http-cacheable-extensions.txt 
>> If not already present, add the following to the list in their alphabetical place: 
eot 
ttf 
woff 
>> and save it.

SCORM connection loss fix

As recommended by Blackboard support in case #02455370 apply a fix in two files to prevent SCORM learning objects losing connection to Blackboard.
As bbuser edit bb-config.properties.template on each node and make the following change.
>> as bbuser 
>> on each node 
nano /usr/local/blackboard/config/bb-config.properties.template 
>> Change 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self 
>> to 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk  
>>Note that this is the last line in the file
As bbuser edit the bb-config.properties on each node
>> as bbuser 
>> on each node nano /usr/local/blackboard/config/bb-config.properties 
>> Change 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self 
>> to 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

Summary of all changes made to bb-config.properties

The bb-config.properties file is one of the primary locations for Blackboard settings. With the 2016 Q4 upgrade we made the following adjustments. Note that this includes the Java log spam and SCORM connection loss fix already detailed above.
As bbuser edit the bb-config.properties file on each node and make the following five changes.
>> as bbuser 
>> on each node 
nano /usr/local/blackboard/config/bb-config.properties 
_______________ 
>> Change 1 
>> Change bbconfig.max.stacksize.tomcat= 
>> from 400k to 512k i.e. 
bbconfig.max.stacksize.tomcat=512k 
_______________ 
>> Change 2 
>> Change bbconfig.jvm.options.extra.tomcat= 
>> So that whole section is replaced with 
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions 
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime 
-XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/ 
_______________ 
>> Change 3 
>> Change bbconfig.jvm.options.gc= >> So that whole section is replaced with 
bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication 
_______________ 
>> Change 4 
>> Check dmarc setting, ensure that: 
bbconfig.email.use.dmarc.from.override=false 
_______________ 
>> Change 5 
>> Update click jacking setting to resolve issue where 
>> SCORM learning objects suffer disconnections 
>> Blackboard case #02455370 
>> Change 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self 
>> to 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk  
>>Note that this is the last line the file.

Workaround to issue LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh

LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh is an issue that should be resolved in 2017 Q2, so this workaround should not be needed in future upgrades.
add lines to /usr/local/blackboard/config/internal/xythos-indexing-filter.txt
>> On each node 
>> Edit /usr/local/blackboard/config/internal/xythos-indexing-filter.txt 
nano /usr/local/blackboard/config/internal/xythos-indexing-filter.txt 
>>Ensure the four lines below are included in the file: 
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet  <---this line may already be in the file 
application/vnd.openxmlformats-officedocument.wordprocessingml.document 
application/vnd.openxmlformats-officedocument.presentationml.presentation 
application/vnd.openxmlformats-officedocument.presentationml.slideshow

Adjust logging and task management

Our Blackboard environment has four web nodes and a tasks node. We will set the tasks node to run tasks and the web nodes to not run tasks. We will set the logging on the web nodes to show fatal errors only, whilst the tasks node will log warnings as well (default setting).
N.B. To set a node to run tasks it is also necessary to set
bbconfig.frontend.fullhostname=true
in the bb-config.properties file on that node.  This setting is honoured by the installer, but the following additional necessary setting is wiped by the installer.
Edit the /usr/local/blackboard/config/service-config.properties on all nodes EXCEPT tasks node to ensure that tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.
As bbuser on all nodes except tasks node 
Edit the file /usr/local/blackboard/config/service-config.properties  
nano /usr/local/blackboard/config/service-config.properties  
and alter the line 
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2 
to read 
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0  
>> and change logging level to fatal on all nodes but tasks node  
blackboard.service.log.param.logdef.services.verbosity=fatal

Comments

Popular posts from this blog

My updated theme for Blackboard 3800.13