Our final upgrade implementation plan (from 2015 Q4 CU1 to 2016 Q4 CU3)

Originally posted on Jun 22, 2017

Here is our final implementation plan.  I will write up a further post highlighting parts that may be particularly useful.

Note that some additional fixes are listed here: A successful upgrade

Live Upgrade from 2015 Q4 CU1 to 2016 Q4 CU3 plan

This plan is for the Blackboard LIVE Upgrade from Bb 9.1 2015 Q4 CU1 to 9.1 2016 Q4 CU3.

Server Address

Blackboard preproduction is made up of four applications nodes, a tasks node, an Oracle 12C database, and a NFS server.
tasks.node.ip.removed – tasks node
web1.node.ip.removed – web node
web2.node.ip.removed – web node
web3.node.ip.removed – web node
web4.node.ip.removed – web node
oracle.node.ip.removed – Oracle database
NFS.node.ip.removed – NFS server

Note

Where "on each node" is stated, this refers to each node on which the Blackboard application runs, i.e. the tasks node and the web nodes.

Summary of stages

  1. Stage 1 covers work that can be carried out during the working week prior to the weekend upgrade. This work concerns preparing files in advance. This stage will be dealt with as a separate RFC.
  2. Stage 2 covers taking snapshot backups and starting the live upgrade window. This marks the beginning of the main RFC.
  3. Stage 3 is Oracle patching.
  4. For the new installer to work correctly certain passwords have to be set to be the same value. Stage 4 entails setting the passwords up correctly and ensuring that Blackboard is working with the new passwords before upgrading.
  5. Stage 5 covers upgrading Java and upgrading Blackboard to the new version.
  6. Having successfully upgraded Blackboard we must make various configuration changes. Stage 6 covers changes due to Java 1.8 and fixes to known issues.
  7. Stage 7 encompasses changes made by BB_ADMIN_ROLE team via the GUI and testing.
  8. In Stage 8 we check that Blackboard restarts ok.
  9. Stage 9 involves tidying up after successful testing.
  10. Stage 10 is when we make the service available again to end-users.

Stage 1 - Prior Preparation

Stage 1 covers work that can be carried out during the working week prior to the weekend upgrade. This work concerns preparing files in advance.
  1. BB_ADMIN_ROLE and DBA: confirm and agree password to be used.
  2. DBA: Prepare files for Oracle patching.
  3. LINUX_ROLE: Prepare installation files on each node.
    >> as bbuser
    >> on each node
    cd /usr/local
    chown -R bbuser\: bbinstaller
    1. Download the installer to /usr/local/bbinstaller on each node.
      1. Go to https://support.blackboard.com and log in.
      2. Go to page https://behind.blackboard.com/downloads/details.aspx?d=1766
      3. Click on Learn 9.1 Q4 2016 Cumulative Update 3 Installer.
      4. Click on I agree to the terms and conditions above.
      5. Click Get File.
      6. Download will start.
      7. Note: Installer has already been downloaded to:
      8. "J:\iSolutions\Service Delivery\eLearning\Blackboard\upgrade_201718\2016-Q4\Installer\2016-Q4-CU3" on 28/4/17.
    2. Unzip the installer to /usr/local/bbinstaller/
    3. Set files under /usr/local/bbinstaller to be owned by bbuser:bbuser
  4. APP_MNGR_ROLE: Copy /usr/local/blackboard/config/license/blackboard-license.xml to /home/bbuser/ on each node
    >> as bbuser
    >> on each node
    cp /usr/local/blackboard/config/license/blackboard-license.xml /home/bbuser/
  5. BB_ADMIN_ROLE: on each node create file at /usr/local/bbinstaller/installer.properties with following draft contents
    antargs.default.vi.db.password=[ ]
    antargs.default.users.administrator.password=xxxx
    antargs.default.users.integration.password=xxxx
    antargs.default.users.rootadmin.password=xxxx
    bbconfig.file.license=/home/bbuser/blackboard-license.xml
    bbconfig.java.home=/usr/lib/jvm/java-1.8.0-oracle.x86_64
  6. APP_MNGR_ROLE: Backup files under /usr/local/blackboard/docs/login/* to /home/bbuser/login.zip on each node.
    >> on each node
    cd /home/bbuser
    zip -r login.zip /usr/local/blackboard/docs/login/*
    >> set permissions
    chown bbuser:bbuser login.zip
  7. APP_MNGR_ROLE: Back up the data_integration directory on the tasks node only (task node).
    >> On tasks node (task node)
    cd /home/bbuser
    zip -r data_integration.zip /usr/local/blackboard/data_integration/*
    >> Set permissions
    chown bbuser:bbuser data_integration.zip
  8. APP_MNGR_ROLE: Back up the snapshot* files in /blackboard/apps/snapshot/data/ directory on the tasks node only.
    >> On tasks node (task node)
    cd /home/bbuser
    zip -r snapshotfiles.zip /usr/local/blackboard/apps/snapshot/data/snapshot.*
    >> set permissions
    chown bbuser:bbuser snapshotfiles.zip

Stage 2 - LIVE Upgrade Preparation

Stage 2 covers taking snapshot backups and starting the live upgrade window
  1. LINUX_ROLE: Remove Blackboard from NLB Pool, thus preventing user access.
    Locate the node objects for the front end servers and 'force offline'
  2. APP_MNGR_ROLE: Stop Blackboard on each node, web nodes (web1 – web4) first, then tasks node (task node).
    >> On each node, web first, then tasks.
    >> web nodes may be stopped concurrently.

    service blackboard stop
  3. LINUX_ROLE: Take "Cold" VMware snapshot #1 on each node' (including db and NFS server).
  4. APP_MNGR_ROLE: Remove the data_integration_log_b directory, to prevent log spam.
    >> as bbuser

    cd /data/bb/blackboard/content/vi/bb_bb60/lucene/
    rm -rf data_integration_log_b/
  5. LINUX_ROLE: Set files to be owned by bbuser:bbuser on each node.
    >> On each node
    find /data/bb/blackboard/ -maxdepth 2 -mindepth 2 -type d -not -name content | xargs -I {} chown -R bbuser\: {}
  6. LINUX_ROLE: Raise ulimit on each node.
    >> On each node
    >> Login as root

    - vim /etc/security/limits.conf
    - add following lines:
    * soft nofile 5000
    * hard nofile 65536

    >> Pay attention that # does not appear in front of either line

    - save
    - logout
    - login

    >> as root check:
    - ulimit -n
    - ulimit -Hn

    [root@~]# ulimit -n
    5000
    [root@~]# ulimit -Hn
    65536

    >> switch to bbuser and check is the same result:
    - su bbuser
    - ulimit -n
    - ulimit -Hn

    [bbuser@]$ ulimit -n
    5000
    [bbuser@]$ ulimit -Hn
    65536

Stage 3 - Oracle Patching

To reduce overall downtime we will apply the latest Oracle patchset to Blackboard
This section took about 25 Minutes during the Dev 1 upgrade.
This section took about 18 Minutes during the Dev 2 upgrade.
  1. DBA: Apply April 2017 Oracle patch as described at [[internal link]]

Stage 4 - Setting the passwords up

For the new installer to work correctly certain passwords have to be set to be the same value. Stage 4 entails setting the passwords up correctly and ensuring that Blackboard is working with the new passwords before upgrading.
This stage took 34 Minutes in the Dev 1 Upgrade
This stage took 1 Hour and 4 Minutes in the Preprod Upgrade
  1. All: Verify that Blackboard is stopped on all nodes.
  2. APP_MNGR_ROLE: On each node edit the bb-config.properties file and ensure that relevant passwords are set to be the same value that has been agreed in Stage 1 above:
    >> on each node
    >> as bbuser

    nano /usr/local/blackboard/config/bb-config.properties

    >> ensure

    bbconfig.database.server.systemuserpassword=PSWD_REMOVED
    bbconfig.database.admin.password=PSWD_REMOVED
    antargs.default.vi.db.password=PSWD_REMOVED
    antargs.default.vi.stats.db.password=PSWD_REMOVED
    antargs.default.vi.report.user.password=PSWD_REMOVED
    bbconfig.cs.db.cms-user.pass=PSWD_REMOVED
    bbconfig.cs.db.systemuser.pass=PSWD_REMOVED

    >> are all set to the same password agreed in Stage 1 Step 1.
  3. LINUX_ROLE: Verification step of above on each node:
    >> as bbuser

    cat /usr/local/blackboard/config/bb-config.properties | grep PSWD_REMOVED |wc -l

    >> Result should equal 7.
  4. DBA: Create duplicate back up table in first two boxes of step 3 below and in Oracle set the passwords to be the same value (PSWD_REMOVED) for the following accounts:
    1. Preprod and Live
      1. “system”
      2. “sys”
      3. bb_bb60
      4. cms
      5. bbadmin
      6. bb_bb60_stats
      7. cms_doc
  5. DBA: update passwords via SQL:
    select * from cms.xy_file_systems;
    _____________________________________________
    Create Table cms.xy_file_systems_bak
    As
      Select * from cms.xy_file_systems;
    _____________________________________________
    UPDATE cms.xy_file_systems
    SET DB_PASSWORD = 'PSWD_REMOVED'
    WHERE NAME = 1;
    COMMIT;
    _____________________________________________
    SELECT * FROM bbadmin.bb_instance;
    _____________________________________________
    Create Table bbadmin.bb_instance_bak
    As
      Select * from bbadmin.bb_instance;
    _____________________________________________
    UPDATE bbadmin.bb_instance
    SET DB_PASS = 'PSWD_REMOVED', STAT_DB_PASS = 'PSWD_REMOVED'
    WHERE PK1 = 1;
    COMMIT;
  6. DBA: Check that no database accounts are locked.
    select username,
      account_status
      from dba_users
    where lock_date is not null;
  7. APP_MNGR_ROLE: As bbuser, run pushconfigupdates.sh to ensure all new password settings in bb-config.properties have been propagated. Blackboard will start once completed. The push should be done:
    >> As bbuser
    >> On each node
    >> One node at a time, tasks node first, followed by web #nodes
    cd /usr/local/blackboard/tools/admin/
    ./PushConfigUpdates.sh

    >> If Blackboard does not start then

    service blackboard start
    1. on each node,
    2. one node at a time starting with the tasks node task node
    3. after checking we can log in via GUI
    4. run pushconfig on web node web1
    5. after checking we can log in via GUI
    6. run pushconfig on next web node web2
    7. repeat until all nodes have been pushed (web3 and web4)
  8. BB_ADMIN_ROLE: Login to Blackboard on each node as they start and quickly test that it is working ok (login, add something to a course, log out.)
  9. ALL: Note: We now have passwords set as required in Blackboard.
  10. LINUX_ROLE: Take COLD VMware snapshot #2 on all nodes (including db and NFS).
  11. APP_MNGR_ROLE: Stop Blackboard on each node (web nodes first (web1-web4), then tasks node (task node)) in preparation for upgrade. The web nodes can be stopped concurrently.
    >> on each node
    >> web nodes first, then tasks node
    service blackboard stop
  12. ALL: verify Blackboard unavailable.
    BB_ADMIN_ROLE: Check via GUI
    APP_MNGR_ROLE/LINUX_ROLE: service blackboard status (on each node)

Stage 5 - The upgrade

Stage 5 covers upgrading Java and upgrading Blackboard to the new version.
This section took about 18 Minutes during the Dev 1 upgrade.
This section took about 13 Minutes during the Dev 2 upgrade.
This section took about 43 Minutes during the Preprod upgrade.

Java Upgrade

  1. LINUX_ROLE Install Java 8 JDK on each node.
    >> On each node
    >> As root, or sudo:
    yum install java-1.8.0-oracle-devel.x86_64
    Is this ok [y/N]: y
    >> To check the latest version use:
    yum list available | grep 'java-1.8'

  2. LINUX_ROLE: Check java locations and alternates
    >> On each node
    alternatives --display java
    >> We should see Java 1.8 locations e.g.
    >> Current `best' version is /usr/lib/jvm/jre-1.8.0-oracle.x86_64/bin/java

First full upgrade – tasks node tasks.node.ip.removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APP_MNGR_ROLE: using shell on tasks node tasks.node.ip.removed change user to bbuser
    adm su - bbuser
  3. APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
  4. APP_MNGR_ROLE: Run upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties | tee -a bb-installer-log.txt 2>&1
  5. BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

First "app server only" upgrade – web node web1.node.ip.removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APP_MNGR_ROLE: using on web node web1.node.ip.removed change user to bbuser
    adm su - bbuser
  3. APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
  4. APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
  5. BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Second "app server only" upgrade – web node web2.node.ip.removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APP_MNGR_ROLE: using on web node web2.node.ip.removed change user to bbuser
    adm su - bbuser
  3. APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
  4. APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
  5. BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Third "app server only" upgrade – web node web3.node.ip.removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APP_MNGR_ROLE: using on web node web3.node.ip.removed change user to bbuser
    adm su - bbuser
  3. APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
  4. APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
  5. BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Forth "app server only" upgrade – web node web4.node.ip.removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APP_MNGR_ROLE: using on web node web4.node.ip.removed change user to bbuser
    adm su - bbuser
  3. APP_MNGR_ROLE: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; confirm JDK location in installation properties file is correct and matches above JDK location.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ; echo $JAVA_HOME ; cd /usr/local/bbinstaller ; sleep 5 ; less installer.properties
  4. APP_MNGR_ROLE: Run app-server only upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
  5. BB_ADMIN_ROLE/LINUX_ROLE/or APP_MNGR_ROLE: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt |more
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt |more

Stage 6 - Applying Configuration Changes

Having successfully upgraded Blackboard we must make various configuration changes. These cover changes due to Java 1.8 and fixes to known issues.
This section took about 44 Minutes during the Dev 1 upgrade.
This section took about 40 Minutes during the Dev 2 upgrade.
This section took about 1 Hour 30 Minutes during the Preprod upgrade.
  1. LINUX_ROLE: On each node apply first part of log spam fix:
    >> on each node
    mkdir -p /home/bbuser/.java/.userPrefs
    mkdir /home/bbuser/.java/.systemPrefs
    chown -R bbuser:bbuser /home/bbuser/.java/
    chmod 755 -Rv /home/bbuser/.java/
  2. APP_MNGR_ROLE: On each node restore branding in /home/bbuser/login.zip to /usr/local/blackboard/docs/login/
    >> on each node
    >> as bbuser
    cd /home/bbuser/;mkdir login-extracted;unzip login.zip -d login-extracted;cd login-extracted/usr/local/blackboard/docs/login;cp -n * /usr/local/blackboard/docs/login

    >> ignore message "cp: omitting directory `WEB-INF'"
  3. APP_MNGR_ROLE: on tasks node, restore data_integration directory.
    >>as bbuser
    >>on tasks node task node
    >>verify whether
    >> /usr/local/blackboard/data_integration/ folder has been wiped. 
    >>If not ask BB_ADMIN_ROLE to double check contents
    >>If BB_ADMIN_ROLE agree it is ok, then skip the following.
    cd /home/bbuser/;mkdir data_integration-extracted;unzip data_integration.zip -d data_integration-extracted;cd data_integration-extracted/usr/local/blackboard/data_integration/;cp -n /home/bbuser/data_integration-extracted/* /usr/local/blackboard/data_integration
  4. APP_MNGR_ROLE: Restore the snapshot.* files on the tasks node (task node).
    >>as bbuser
    >>on tasks node task node
    cd /home/bbuser/;mkdir snapshotfiles-extracted;unzip snapshotfiles.zip -d snapshotfiles-extracted;cd snapshotfiles-extracted/usr/local/blackboard/apps/snapshot/data/;cp -n snapshot.* /usr/local/blackboard/apps/snapshot/data/
  5. APP_MNGR_ROLE: on tasks node, restore PHOTOS directory.
    >>as bbuser
    >>on tasks node task node
    >>verify whether
    >> /usr/local/blackboard/content/PHOTOS/ folder has been wiped. 
    >>If not ask BB_ADMIN_ROLE to double check contents
    >>If BB_ADMIN_ROLE agree it is ok, then skip the following.
    cd /home/bbuser/
    mkdir PHOTOS-extracted
    unzip PHOTOS.zip -d PHOTOS-extracted
    cd /usr/local/blackboard/content/PHOTOS/
    >>if data_integration directory has been removed, create a new one
    mkdir /usr/local/blackboard/content/PHOTOS/
    cp -n /home/bbuser/PHOTOS-extracted/* /usr/local/blackboard/content/PHOTOS/
  6. APP_MNGR_ROLE: In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FS, make following change to /usr/local/blackboard/apps/tomcat/conf/wrapper.conf on each node:
    >> as bbuser
    >> On each node.
    >> Backing up file.
    >> Back up original wrapper.conf to /home/bbuser
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/wrapper.conf.bb.$(date +"%Y-%m-%d")
    >> As bbuser
    >> On each node
    cd /usr/local/blackboard/config/tomcat/conf/
    vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf
    >> find the line
    wrapper.shutdown.timeout=300
    >> and change it to
    wrapper.shutdown.timeout=30
    >> find the line
    wrapper.jvm_exit.timeout=15
    >> and change it to
    wrapper.jvm_exit.timeout=5
  7. LINUX_ROLE: Verification of above on each node
    >>On each node
    cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf |grep wrapper.shutdown.timeout ; cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf |grep wrapper.jvm_exit.timeout
    >> result should be
    wrapper.shutdown.timeout=30
    wrapper.jvm_exit.timeout=5
  8. APP_MNGR_ROLE: In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FS, make the following change to /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb on each node:
    >> as bbuser
    >> On each node.
    >> Backing up file.
    >> Back up original wrapper.conf.bb to /home/bbuser
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/wrapper.conf.bb.$(date +"%Y-%m-%d")
    >> As bbuser
    >> On each node
    cd /usr/local/blackboard/config/tomcat/conf/
    vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb
    >> find the line
    wrapper.shutdown.timeout=300
    >> and change it to
    wrapper.shutdown.timeout=30
    >> find the line
    wrapper.jvm_exit.timeout=15
    >> and change it to
    wrapper.jvm_exit.timeout=5
  9. LINUX_ROLE: Verification of above on each node
    >>On each node
    cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb |grep wrapper.shutdown.timeout ; cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb |grep wrapper.jvm_exit.timeout
    >> result should be
    wrapper.shutdown.timeout=30
    wrapper.jvm_exit.timeout=5
  10. APP_MNGR_ROLE: On each node update the http-cacheable-extensions.txt to include reference to resolve issues with Equation editor.
    >> on each node
    >> as bbuser
    cd /usr/local/blackboard/config/internal/
    nano http-cacheable-extensions.txt
    >> If not already present, add the following to the list in their alphabetical place:
    eot
    ttf
    woff
    >> and save it.
  11. LINUX_ROLE: Verification of above on each node

    >> As bbuser
    >> On each node
    grep 'eot\|ttf\|woff' /usr/local/blackboard/config/internal/http-cacheable-extensions.txt | wc -l
    >> Result must equal 3.
  12. APP_MNGR_ROLE: As bbuser edit bb-config.properties.template on each node and make the following change.
    >> as bbuser
    >> On each node.
    >> Backing up file.
    >> Back up original bb-config.properties.template to /home/bbuser
    cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/bb-config.properties.template.$(date +"%Y-%m-%d")
    >> as bbuser
    >> on each node
    nano /usr/local/blackboard/config/bb-config.properties.template
    >> Update click jacking setting to resolve issue where
    >> SCORM learning objects suffer disconnections
    >> Blackboard case #02455370
    >> Change
    #bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
    bbconfig.headers.CSP.frame-ancestors=self
    >> to
    #bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
    bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

    >>Note that this is the last line in the file
  13. LINUX_ROLE: Verification of above on each node.
    >> on each node
    cat /usr/local/blackboard/config/bb-config.properties.template | grep "ancestors=self'"
    >> Result should be bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
  14. APP_MNGR_ROLE: As bbuser edit the bb-config.properties file on each node and make the following five changes.
    >> Backing up file.
    >> On each node back up original bb-config.properties to /home/bbuser
    cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/bb-config.properties.$(date +"%Y-%m-%d")
    >> as bbuser
    >> on each node
    nano /usr/local/blackboard/config/bb-config.properties
    _______________
    >> Change 1
    >> Change bbconfig.max.stacksize.tomcat=
    >> from 400k to 512k i.e.
    bbconfig.max.stacksize.tomcat=512k
    _______________
    >> Change 2
    >> Change bbconfig.jvm.options.extra.tomcat=
    >> So that whole section is replaced with
    bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/
    >> N.B. Be careful of line breaks
    >> N.B. copy from SysDoc, print out will not show full text
    _______________
    >> Change 3
    >> Change bbconfig.jvm.options.gc=
    >> So that whole section is replaced with
    bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
    >> N.B. Be careful of line breaks
    >> N.B. copy from SysDoc, print out will not show full text
    _______________
    >> Change 4
    >> Check dmarc setting, ensure that:
    bbconfig.email.use.dmarc.from.override=false
    _______________
    >> Change 5
    >> Update click jacking setting to resolve issue where
    >> SCORM learning objects suffer disconnections
    >> Blackboard case #02455370
    >> Change
    #bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
    bbconfig.headers.CSP.frame-ancestors=self
    >> to
    #bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
    bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

    >>Note that this is the last line the file.
  15. LINUX_ROLE: Verification, in either batch or individually...
  16. LINUX_ROLE: Batch verification steps that above five changes have been made successfully, perform on each node.
    >> On each node
    echo "Change 1";cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.max.stacksize.tomcat";echo "Change 2";cat /usr/local/blackboard/config/bb-config.properties |grep "Djava";echo "Change 3";cat /usr/local/blackboard/config/bb-config.properties |grep UseG1GC;echo "Change 4";cat /usr/local/blackboard/config/bb-config.properties | grep "dmarc.from.override";echo "Change 5";cat /usr/local/blackboard/config/bb-config.properties | grep "ancestors=self'"

    >> Should result with:
    Change 1
    bbconfig.max.stacksize.tomcat=512k
    Change 2
    bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/
    Change 3
    bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
    Change 4
    bbconfig.email.use.dmarc.from.override=false
    Change 5
    bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
  17. LINUX_ROLE: Individual verification steps that above five changes have been made successfully on each node.[Expand]
  18. APP_MNGR_ROLE: Apply xythos cross server communication fix on each node.
    >> On each node
    >> Backup file
    cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/server.xml.bb.$(date +"%Y-%m-%d")
    >> On each node
    >> Edit /usr/local/blackboard/config/tomcat/conf/server.xml.bb
    nano /usr/local/blackboard/config/tomcat/conf/server.xml.bb
    >> find the stanza (near end of file around lines 203 - 207) defining the xythos admin connector and add secure="true" i.e.
    >> Currently it is:
      <Connector port="@bbconfig.xythosremoteadmin.xythos.servergroup.portnumber@"
      protocol="org.apache.coyote.http11.Http11NioProtocol"
      executor="crossServerExecutor"
      backlog="@@bbconfig.xythosremoteadmin.connector.backlog@@"
      enableLookups="false"
      keepAliveTimeout="@@bbconfig.appserver.http.keepalive.timeout@@"
      maxKeepAliveRequests="@@bbconfig.appserver.http.keepalive.requests@@"
      maxPostSize="@@bbconfig.appserver.max.post.size@@"
      maxParameterCount="@@bbconfig.appserver.max.parameter.count@@"
      soTimeout="@@bbconfig.appserver.http.so.timeout@@"
      URIEncoding="UTF-8"/>
    >> Change to:
      <Connector port="@bbconfig.xythosremoteadmin.xythos.servergroup.portnumber@"
      protocol="org.apache.coyote.http11.Http11NioProtocol"
      executor="crossServerExecutor"
      backlog="@@bbconfig.xythosremoteadmin.connector.backlog@@"
      enableLookups="false"
      keepAliveTimeout="@@bbconfig.appserver.http.keepalive.timeout@@"
      maxKeepAliveRequests="@@bbconfig.appserver.http.keepalive.requests@@"
      maxPostSize="@@bbconfig.appserver.max.post.size@@"
      maxParameterCount="@@bbconfig.appserver.max.parameter.count@@"
      soTimeout="@@bbconfig.appserver.http.so.timeout@@"
      URIEncoding="UTF-8"
      secure="true"/> <----- Part to add
  19. LINUX_ROLE: Verification step of above on each node.
    >> on each node
    cat /usr/local/blackboard/config/tomcat/conf/server.xml.bb |grep secure
      secure="true"
      secure="true"
      secure="true"/>
  20. APP_MNGR_ROLE and DBA: Apply workaround to issue LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh by adding lines to /usr/local/blackboard/config/internal/xythos-indexing-filter.txt detailed here: MicrosoftContentParser failure - java.lang.Exception
  21. APP_MNGR_ROLE: Edit the /usr/local/blackboard/config/service-config.properties on all nodes EXCEPT tasks node to ensure that tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.
    As bbuser
    on all nodes except tasks node (web1,web2,web3,web4)
    Edit the file /usr/local/blackboard/config/service-config.properties

    nano /usr/local/blackboard/config/service-config.properties

    and alter the line
    blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2
    to read
    blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0

    >> and change logging level to fatal on all nodes but tasks node

    blackboard.service.log.param.logdef.services.verbosity=fatal
  22. LINUX_ROLE: Verification of above step:
    >>On web nodes (web1,web2,web3,web4)
    cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.queuedtaskmanager.param.num-concurrent-tasks ;cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.log.param.logdef.services.verbosity

    >>should return
    blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0
    blackboard.service.log.param.logdef.services.verbosity=fatal

    On tasks node (task node)
    cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.queuedtaskmanager.param.num-concurrent-tasks ;cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.log.param.logdef.services.verbosity

    >>should return
    blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2
    blackboard.service.log.param.logdef.services.verbosity=warning
  23. APP_MNGR_ROLE: As bbuser, run pushconfigupdates.sh on each node, tasks node (task node) first, and having checked we can login via the GUI, continue with web nodes (web1-web4) to ensure all new settings in bb-config.properties have been propagated. Note that we are exporting the output to a log file for investigation if required. Blackboard will start once completed.
    >> As bbuser
    >> Tasks node first
    cd /usr/local/blackboard/tools/admin/
    ./PushConfigUpdates.sh | tee -a /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d") 2>&1

    >> On big tv
    tail -f /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d") 
    >>during the process

    >>check BB_ADMIN_ROLE can login through GUI
    >>then web nodes, one at a time after checking BB_ADMIN_ROLE can log into each one
    cd /usr/local/blackboard/tools/admin/
    ./PushConfigUpdates.sh | tee -a /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d") 2>&1

    >> On big tv
    tail -f /home/bbuser/pushconfig-stage5.txt.$(date +"%Y-%m-%d") 
    >>during the process
  24. ALL: In the Dev 1 upgrade this failed due to the setting for bbconfig.database.stats.maxpoolsize being reset by pushconfigupdates.sh to bbconfig.database.stats.maxpoolsize=@bbconfig.database.stats.maxpoolsize@ We reset this back to We reset this back to the value shown in our backup and then pushconfig succeeded.
  25. ALL: In the Dev 2 upgrade this failed due to the setting for bbconfig.max.heapsize.tomcat being reset by pushconfigupdates.sh to bbconfig.max.heapsize.tomcat=@bbconfig.max.heapsize.tomcat@ We reset this back to the value shown in our backup and then pushconfig succeeded.
  26. BB_ADMIN_ROLE: Restore login page by
    >> As bbuser
    >> On tasks node task node
    cd /usr/local/blackboard/tools/admin/
    ./AuthenticationOneTimeLogin.sh -u administrator
    >> Login with link and pass provided
    Then navigate to System Admin, Brands and Themes, Customise Login page, Replace Customised Page
    Upload latest login page held in J:\iSolutions\Service Delivery\eLearning\Blackboard\Configuration\front_page
    >> If required update gateway page
    /usr/local/blackboard/content/vi/bb_bb60/branding/
    nano gateway.bb
    Check you can log in
  27. APP_MNGR_ROLE: Restart ActiveMQ on tasks node to ensure that it is running correctly before we begin testing.
    >> as bbuser
    >> on tasks node
    cd /usr/local/blackboard/tools/admin
    ./ServiceController.sh services.activemqbroker.restart
  28. LINUX_ROLE: Take COLD VMware snapshot #3 (including db and NFS server).

Stage 7 – GUI configuration changes and UAT

This stage encompasses changes made by BB_ADMIN_ROLE team via the GUI and testing.

  1. BB_ADMIN_ROLE: Stop Blackboard on all nodes except for tasks node.
    >>On all nodes, except tasks node.
    service blackboard stop
  2. BB_ADMIN_ROLE: Uninstall  building blocks.
  3. BB_ADMIN_ROLE: Make unavailable the following unused building blocks to save Java RAM usage:
    >>as bbuser 
    cd /usr/local/blackboard/tools/admin 
    ./B2Manager.sh -s UNAVAILABLE bbgs-nbc-content-integration
    ./B2Manager.sh -s UNAVAILABLE bb-word-cleanup
    ./B2Manager.sh -s UNAVAILABLE bb-auth-provider-cas
    ./B2Manager.sh -s UNAVAILABLE bb-auth-provider-shibboleth
    ./B2Manager.sh -s UNAVAILABLE bb-data-integration-ims-xml
    ./B2Manager.sh -s UNAVAILABLE bb-data-integration-lis
    ./B2Manager.sh -s UNAVAILABLE bb-data-integration-lis-final
    ./B2Manager.sh -s UNAVAILABLE bb-data-integration-ss-xml
  4. BB_ADMIN_ROLE: Make inactive the following unused building blocks to save Java RAM usage:
    >>as bbuser 
    cd /usr/local/blackboard/tools/admin 
    ./B2Manager.sh -s INACTIVE bb-cx-ext-angel-plgnhndl
    ./B2Manager.sh -s INACTIVE bb-cx-ext-ce4-plgnhndl
    ./B2Manager.sh -s INACTIVE bb-cx-ext-vista-plgnhndl
    ./B2Manager.sh -s INACTIVE bb-ce4-ext-plgnhndl
    ./B2Manager.sh -s INACTIVE bb-vista-ext-plgnhndl
    ./B2Manager.sh -s INACTIVE bb-cookie-disclosure
    ./B2Manager.sh -s INACTIVE bbgs-nbc-content-integration
    ./B2Manager.sh -s INACTIVE bb-word-cleanup
    ./B2Manager.sh -s INACTIVE bb-auth-provider-cas
    ./B2Manager.sh -s INACTIVE bb-auth-provider-shibboleth
    ./B2Manager.sh -s INACTIVE bb-data-integration-ims-xml
    ./B2Manager.sh -s INACTIVE bb-data-integration-lis
    ./B2Manager.sh -s INACTIVE bb-data-integration-lis-final
    ./B2Manager.sh -s INACTIVE bb-data-integration-ss-xml
  5. BB_ADMIN_ROLE: On tasks node upload the following Building Blocks via GUI.
    [removed]
  6. BB_ADMIN_ROLE: Check Panopto settings:
    >>The following needs to be unticked
    Blackboard course copy also copies Panopto permissions
    >>Note this is last setting on the page.

    >>These settings should already be set
    User Opt In – Yes
    Email Instructors - No
    Redirect to Blackboard Default Login – No
    Grant TA Creator Access – Yes
    Allow TAs to Create Panopto Course Tool Links – No
    Grant TA Provisioning Access – Yes
    Provisioning by administrators only – No
    Panopto link on course menu when provisioned – No
    Panopto Link Text - Recorded Sessions
    Instructors may create folders - No
    Allow all courses to be reset – No
    Verbose logging – No
    Assign custom role mappings – [BLANK]
  7. BB_ADMIN_ROLE: Check that Rubrics Building Block is available.
  8. BB_ADMIN_ROLE: Restart Blackboard on Tasks node.
    service blackboard restart
  9. BB_ADMIN_ROLE: Update the "partner cloud" building block via GUI
  10. BB_ADMIN_ROLE: Restart Blackboard on Tasks node.
    service blackboard restart
  11. BB_ADMIN_ROLE: Update the "Blackboard Mobile Web Services" building block via GUI
  12. BB_ADMIN_ROLE: Restart Blackboard on Tasks node.
    service blackboard restart
  13. BB_ADMIN_ROLE: Check Blackboard Mobile Web Services Building Block > Server Registration Settings:
    1. Environment: Production
    2. Authentication Type: Native
  14. BB_ADMIN_ROLE: Update the "Dropbox Integration" building block via GUI
  15. BB_ADMIN_ROLE: Make Blackboard Collaborate and Collaborate Ultra Building Blocks inactive.
  16. BB_ADMIN_ROLE: Make Dropbox Integration Building Block Unavailable.
  17. BB_ADMIN_ROLE: Restart Blackboard on Tasks node.
    service blackboard restart
  18. BB_ADMIN_ROLE: Check you can log in to Tasks node via GUI.
  19. BB_ADMIN_ROLE: Start Blackboard on web nodes, one node at a time. Check you can log in before starting the next node.
    >>as bbuser
    >>on web nodes, one node at a time. Check you can log in before starting the next node.
    service blackboard start
  20. BB_ADMIN_ROLE:Setup Qwickly
    1. Building Blocks > Qwickly > Settings
    2. Set ID as [removed]
    3. Set this is a Production System
    4. Tick:
      1. Course Availability
      2. Post announcement
      3. Send Email/
      4. Don't tick email instructors
      5. Untick Needs Grading
      6. Tick Teaching Assistant
    5. Submit Settings
  21. BB_ADMIN_ROLE:Setup Qwickly link in Tool Panel.
    1. System Admin > Tabs and Modules > Tool Panel
    2. Check Enterprise Surveys, Goals, Goal Performance, Observer Dashboard, Academic Materials, User directory, Address book are all unavailable.
    3. Add Tool
    4. Name: "Instructor Quick Tools"
    5. Tool: Qwickly
  22. BB_ADMIN_ROLE: Check Spell checking settings:
    1. SysAdmin Panel
    2. Spell Check Dictionaries
    3. Check UK English set as default
    4. Make US English unavailable
    5. Make Spanish unavailable
  23. BB_ADMIN_ROLE: Enable new language pack
    1. SysAdmin Panel
    2. Language Packs
    3. Make Norweigian available
  24. BB_ADMIN_ROLE: Update Tool Settings:
    1. SysAdmin Panel
    2. Tools ->
    3. Dropbox Intergration -> Set all off
    4. Goal -> Set all off
    5. Open Photo Roster
      1. Course Tool > ON > UNLOCK > New and existing
    6. Pearson Custom Integration
      1. Set all to Off > Unlock > New and existing
    7. Pearson's MyLab & Mastering
      1. Set all to Off > Unlock > New and existing
    8. Qwickly:
      1. Qwickly Tools Course Tool - Off
      2. Qwickly Attendance - Off and Lock
      3. Qwickly Cloud Assignment - Off and Lock
      4. Qwickly Cloud Link VTBE Mashup - Off and Lock
      5. Qwickly Take Attendance - Off and Lock
      6. Qwickly Attendance - Off and Lock
      7. Qwickly Attendance - Off and Lock
      8. Qwickly Tools (System Tool) - On
  25. BB_ADMIN_ROLE: Disable Open Photo Roster on the following large courses by
    1. enter course
    2. click Control Panel -> Customisation -> Tool Availability
    3. find Open Photo Roster
    4. untick box
    5. submit
    6. Follow this link for list of courses on which tool should be disabled:
  26. BB_ADMIN_ROLE: Enable "forgot your password?" link.
    1. Sys Admin
    2. Gateway Options
    3. Request Forgotten Password > Enable
    4. URL for Forgotten Password: https://subscribe.soton.ac.uk/
  27. BB_ADMIN_ROLE: Restart ActiveMQ on tasks node to ensure that it is running correctly before we begin testing.
    >> as bbuser
    >> on tasks node
    cd /usr/local/blackboard/tools/admin
    ./ServiceController.sh services.activemqbroker.restart
  28. BB_ADMIN_ROLE: UAT Blackboard via GUI on all nodes.

Stage 8 – Restart Test

In this stage we check that Blackboard restarts ok.
  1. LINUX_ROLE: Perform vApp stop of Blackboard nodes in line with standard method used for patching.
  2. LINUX_ROLE: Perform vApp start of Blackboard nodes in line with standard method used for patching.
  3. APP_MNGR_ROLE: Once Blackoard available again, restart ActiveMQ on tasks node to ensure that it is running correctly.
    >> as bbuser
    >> on tasks node
    cd /usr/local/blackboard/tools/admin
    ./ServiceController.sh services.activemqbroker.restart
  4. BB_ADMIN_ROLE: Check you can log into web nodes.
  5. BB_ADMIN_ROLE: Make a non-immediate announcement on each node, to test later that ActiveMQ is working. Emails should appear within an hour.
  6. BB_ADMIN_ROLE: On each node, add a new item to a test course and check it appears ok.

Stage 9 – Wash-up

This stage involves tidying up after successful testing.
  1. DBA: Reset DB passwords for sys, system; remove duplicate table back up from passwords section above.
  2. APP_MNGR_ROLE: Remove backups of zip files and uncompressed directories.
    >> As bbuser
    On each node
    rm /home/bbuser/login.zip
    rm -rf /home/bbuser/login-extracted
    >> On tasks node
    rm /home/bbuser/snapshotfiles.zip
    rm /home/bbuser/data_integration.zip
    rm -rf /home/bbuser/data_integration-extracted
    rm -rf /home/bbuser/snapshotfiles-extracted
  3. APP_MNGR_ROLE: Remove copied licence file.
    >> As bbuser
    rm /home/bbuser/blackboard-license.xml
  4. APP_MNGR_ROLE: Remove installer directory.
    >> As root
    rm -rf /usr/local/bbinstaller
  5. BB_ADMIN_ROLE: Remove password from this documentation.

Stage 10 – GO-LIVE

  1. LINUX_ROLE: Following confirmation by BB_ADMIN_ROLE, make Blackboard live via NLB.
  2. BB_ADMIN_ROLE: Announce service availability via twitter @uos_BB_ADMIN_ROLE and blackboard-users@lists mailing list.
  3. LINUX_ROLE: Assuming all successful, and with agreement of BB_ADMIN_ROLE, delete snapshots.

Summary of VMWare Snapshots

Note all snapshots are "cold" and include all nodes, db, and NFS.
Snapshot #1 – No changes have taken place. Snapshot included all nodes including db and NFS.
Snapshot #2 – Preliminary pre-upgrade work complete, passwords set correctly, Blackboard ready for upgrade.
Snapshot #3 – Blackboard upgraded but GUI based config not yet started.

Comments

Popular posts from this blog

My updated theme for Blackboard 3800.13