Highlights of fixes from our upgrade implementation plan

-
Originally posted on Jun 23, 2017

Our implementation plan is quite long and hard to skim over.  Here I've tried to highlight some of the interesting fixes that we have included that may be of particular interest to others.
Note that some additional fixes are listed here: A successful upgrade

File ownership

Blackboard files should be owned by bbuser, otherwise upgrades will fail.
>> On each node 
find /data/bb/blackboard/ -maxdepth 1 -mindepth 1 -type d -not -name content | xargs -I {} chown -R bbuser\: {}

Ulimit

The upgrade produced errors due to ulimit being too low. Therefore we should raise the ulimit as follows:
>> On each node 
>> Login as root  
-     vim /etc/security/limits.conf 
-     add following lines:
 *               soft    nofile          5000
 *               hard    nofile          65536
  >> Pay attention that # does not appear in front of either line
 -     save
 -     logout 
 -     login  
>> as root check: 
 -     ulimit -n 
 -     ulimit -Hn  
[root@~]# ulimit -n 5000 
[root@~]# ulimit -Hn 65536  
>> switch to bbuser and check is the same result:
 -     su bbuser
 -     ulimit -n
 -     ulimit -Hn
[bbuser@]$ ulimit -n 
5000 
[bbuser@]$ ulimit -Hn 
65536

Java Log Spam fix

To prevent log spam in the stdout-stderr log perform the following:
 >> on each node 
mkdir -p /home/bbuser/.java/.userPrefs 
mkdir /home/bbuser/.java/.systemPrefs 
chown -R bbuser:bbuser /home/bbuser/.java/ 
chmod 755 -Rv /home/bbuser/.java/  
 >> on each node 
 >> within /usr/local/blackboard/config/bb-config.properties 
 >> add to bbconfig.jvm.options.extra.tomcat= 
-Djava.util.prefs.systemRoot=/home/bbuser/.java/

Speed up Blackboard start and stop

In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FSmake following change to /usr/local/blackboard/apps/tomcat/conf/wrapper.conf on each node:
>> As bbuser 
>> On each node 
cd /usr/local/blackboard/config/tomcat/conf/ 
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf 
>> find the line 
wrapper.shutdown.timeout=300 
>> and change it to 
wrapper.shutdown.timeout=30 
>> find the line 
wrapper.jvm_exit.timeout=15 
>> and change it to 
wrapper.jvm_exit.timeout=5

And make the following change to /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb on each node:
>> As bbuser 
>> On each node 
cd /usr/local/blackboard/config/tomcat/conf/ 
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb 
>> find the line 
wrapper.shutdown.timeout=300 
>> and change it to 
wrapper.shutdown.timeout=30 
>> find the line 
wrapper.jvm_exit.timeout=15 
>> and change it to 
wrapper.jvm_exit.timeout=5

Resolve Math Editor not loading issue

On each node update the http-cacheable-extensions.txt to include reference to resolve issues with Equation editor.
>> on each node 
>> as bbuser 
cd /usr/local/blackboard/config/internal/ 
nano http-cacheable-extensions.txt 
>> If not already present, add the following to the list in their alphabetical place: 
eot 
ttf 
woff 
>> and save it.

SCORM connection loss fix

As recommended by Blackboard support in case #02455370 apply a fix in two files to prevent SCORM learning objects losing connection to Blackboard.
As bbuser edit bb-config.properties.template on each node and make the following change.
>> as bbuser 
>> on each node 
nano /usr/local/blackboard/config/bb-config.properties.template 
>> Change 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self 
>> to 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk  
>>Note that this is the last line in the file
As bbuser edit the bb-config.properties on each node
>> as bbuser 
>> on each node nano /usr/local/blackboard/config/bb-config.properties 
>> Change 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self 
>> to 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

Summary of all changes made to bb-config.properties

The bb-config.properties file is one of the primary locations for Blackboard settings. With the 2016 Q4 upgrade we made the following adjustments. Note that this includes the Java log spam and SCORM connection loss fix already detailed above.
As bbuser edit the bb-config.properties file on each node and make the following five changes.
>> as bbuser 
>> on each node 
nano /usr/local/blackboard/config/bb-config.properties 
_______________ 
>> Change 1 
>> Change bbconfig.max.stacksize.tomcat= 
>> from 400k to 512k i.e. 
bbconfig.max.stacksize.tomcat=512k 
_______________ 
>> Change 2 
>> Change bbconfig.jvm.options.extra.tomcat= 
>> So that whole section is replaced with 
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions 
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime 
-XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/ 
_______________ 
>> Change 3 
>> Change bbconfig.jvm.options.gc= >> So that whole section is replaced with 
bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication 
_______________ 
>> Change 4 
>> Check dmarc setting, ensure that: 
bbconfig.email.use.dmarc.from.override=false 
_______________ 
>> Change 5 
>> Update click jacking setting to resolve issue where 
>> SCORM learning objects suffer disconnections 
>> Blackboard case #02455370 
>> Change 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self 
>> to 
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>] 
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk  
>>Note that this is the last line the file.

Workaround to issue LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh

LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh is an issue that should be resolved in 2017 Q2, so this workaround should not be needed in future upgrades.
add lines to /usr/local/blackboard/config/internal/xythos-indexing-filter.txt
>> On each node 
>> Edit /usr/local/blackboard/config/internal/xythos-indexing-filter.txt 
nano /usr/local/blackboard/config/internal/xythos-indexing-filter.txt 
>>Ensure the four lines below are included in the file: 
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet  <---this line may already be in the file 
application/vnd.openxmlformats-officedocument.wordprocessingml.document 
application/vnd.openxmlformats-officedocument.presentationml.presentation 
application/vnd.openxmlformats-officedocument.presentationml.slideshow

Adjust logging and task management

Our Blackboard environment has four web nodes and a tasks node. We will set the tasks node to run tasks and the web nodes to not run tasks. We will set the logging on the web nodes to show fatal errors only, whilst the tasks node will log warnings as well (default setting).
N.B. To set a node to run tasks it is also necessary to set
bbconfig.frontend.fullhostname=true
in the bb-config.properties file on that node.  This setting is honoured by the installer, but the following additional necessary setting is wiped by the installer.
Edit the /usr/local/blackboard/config/service-config.properties on all nodes EXCEPT tasks node to ensure that tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.
As bbuser on all nodes except tasks node 
Edit the file /usr/local/blackboard/config/service-config.properties  
nano /usr/local/blackboard/config/service-config.properties  
and alter the line 
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2 
to read 
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0  
>> and change logging level to fatal on all nodes but tasks node  
blackboard.service.log.param.logdef.services.verbosity=fatal

Comments

Post a Comment

Popular posts from this blog

Should you use Significant Figures or Decimal Places when setting Answer Set options for Calculated Formula questions in Blackboard?

-
Image
Blackboard tests are a hot topic at the moment. I recently recently received a question from Dr Gabriel Weymouth : "I'm positive this is clearly documented somewhere, but I can't find it. How do you make BB apply the correct error margins for calculated formula problems. The logical way of doing it (setting a, say, 2% margin) doesn't work:"
Read more

My updated theme for Blackboard 3800.13

-
Image
As we move to SaaS soon I had to update our theme work to be compatible with the latest SaaS version. So this means I have a nice freshly compiled version to share.  The previously shared version had been lost in the transfer to the new Blackboard Community Site. This explains why I now back up all new posts to both this blog as well as posting on the community site.  A lack of time means that right now I cannot share screenshots of how the new additions to the theme look, but I am providing the source code below so you can experiment with copying and pasting them into Stylus easily, without having to unzip the theme. You can get a good idea of how it all looks by referring to my post: Let me show you my Blackboard . Watch this video for a guide on how to try the theme out without having to install it: Note that the improvement to Review Status required me to apply a separate CSS just for instructors. In Blackboard the review status button has a double purpose when maki...
Read more