UpgradeWNstoSL5.5

Upgrade of Workernodes in CB6 to SL5.5

• Currently all nodes in CB6 are running SL5.3, the goal is to upgrade them to SL5.5
• The testing node is node16-5; first step is to out it offline so it can drain.

pbsnodes -o node16-5.wn.iihe.ac.be

• In the template for the os version the entry for node16-5 is upgraded to sl550-x86_64

/CBv6/cfg/sites/iihe-production/site/os_version_db.tpl

• This file is then compiled in the CDB and commited
• To make the changes do a runcheck on ccq3

ssh ccq3
cd /opt/CB6/svncheck/
./runcheck

• And do the changes

aii-shellfe --remove node16-5.wn.iihe.ac.be #remove the current configuration for this machine
aii-shellfe --configure node16-5.wn.iihe.ac.be #create the new kickstart file for this machine
aii-shellfe --install node16-5.wn.iihe.ac.be#Update of DHCP and PXE for this machine to be installed at next boot

• To check the status simply issue something like this

for line in $(cat wn_list); do aii-shellfe --status $line; done

REMARK: it is better to do the remove before the runcheck to not clash the xml file with the existing configuration

• Reboot the machine

ssh node16-5.wn
reboot -n

• Once the machine is back you will see in the spma log file the following error message

/var/log/smpa.log
error code: cpio: open failed - Too many open files

• To fix this you have to reboot the machine after moving the kickstart log file

mv ks-post-install.log ks-post-install.log.old

• Once the quattor installation finished successfully you still have to reboot a last time

Re-installation campaign

• The re-installation campaign is executed in the following steps
• Adapt the file os_version_db.tpl in the quattor database, compile, commit and do a runcheck
• First of all, all nodes need to reconfigured with the aii-shellfe command. In the following location you find a little script to do this:

[root@ccq3] /root/joris/nodeReinstallation.sh

• The script takes as an argument a file listing all the workernodes to be removed. The file was generated with the a script found here:

[root@ccq] /root/stephane/generate_list_of_online_nodes_CB6.sh

• Once the script is executed on ccq3 it is time to move to ccq to perform the reboot of all the nodes. To do this Stephane has written a nice script. More details about this script can be found here.
• The script has been adapted by Joris to allow the re-installation of a workernode. Technically a few more reboot cycles have been added with the appropriate handles to check if the installation was done in a correct way. The script can be found here and is executed with the same options as the original script

cd /root/stephane/UserCode/T2B_IIHE/reboot_wns
./reboot_wns.pl --init wnfile_list_sl55campaign
./reboot_wns.pl --start

• In this directory there are some small scripts to test various things, eg.
  • ./print_wn_job_number.pl wn_list: shows the number of jobs running on this node
  • ./clear_after_test.pl: removes all lists of workernodes

List of workernodes that are under update procedure

workernode	status	done
node15-1.wn.iihe.ac.be		y + on
node15-2.wn.iihe.ac.be		y + on
node15-3.wn.iihe.ac.be		y + on
node15-4.wn.iihe.ac.be	not in production
node15-5.wn.iihe.ac.be	not in production
node15-6.wn.iihe.ac.be	not in production
node15-7.wn.iihe.ac.be	not in production
node15-8.wn.iihe.ac.be		y + on
node16-1.wn.iihe.ac.be		y + on
node16-2.wn.iihe.ac.be	reserved	y
node16-3.wn.iihe.ac.be		y + on
node16-4.wn.iihe.ac.be		y + on
node16-5.wn.iihe.ac.be		y + on
node16-6.wn.iihe.ac.be	removed
node16-7.wn.iihe.ac.be		y + on
node16-8.wn.iihe.ac.be	reserved
node16-9.wn.iihe.ac.be		y + on
node16-10.wn.iihe.ac.be	reserved	y
node17-1.wn.iihe.ac.be	reserved	y
node17-2.wn.iihe.ac.be	n/a
node17-3.wn.iihe.ac.be		y + on
node17-4.wn.iihe.ac.be		y + on
node17-5.wn.iihe.ac.be	reserved	y
node17-6.wn.iihe.ac.be		y + on
node17-7.wn.iihe.ac.be		y + on
node17-8.wn.iihe.ac.be		y + on
node17-9.wn.iihe.ac.be		y + on
node17-10.wn.iihe.ac.be		y + on
node17-11.wn.iihe.ac.be		y + on
node17-12.wn.iihe.ac.be		y + on
node17-13.wn.iihe.ac.be		y + on
node17-14.wn.iihe.ac.be	down
node18-1.wn.iihe.ac.be		y + on
node18-2.wn.iihe.ac.be		y + on
node18-3.wn.iihe.ac.be		y + on
node18-4.wn.iihe.ac.be		y + on
node18-5.wn.iihe.ac.be		y + on
node18-6.wn.iihe.ac.be		y + on
node18-7.wn.iihe.ac.be		y + on
node18-8.wn.iihe.ac.be		y + on
node19-1.wn.iihe.ac.be		y + on
node19-2.wn.iihe.ac.be		y + on
node19-3.wn.iihe.ac.be		y + on
node19-4.wn.iihe.ac.be		y + on
node19-5.wn.iihe.ac.be		y + on
node19-6.wn.iihe.ac.be		y + on
node19-7.wn.iihe.ac.be		y + on
node19-8.wn.iihe.ac.be		y + on
node19-9.wn.iihe.ac.be		y + on
node19-10.wn.iihe.ac.be		y + on
node19-11.wn.iihe.ac.be		y + on
node19-12.wn.iihe.ac.be		y + on
node19-13.wn.iihe.ac.be		y + on
node19-14.wn.iihe.ac.be		y + on
node19-15.wn.iihe.ac.be		y + on
node19-16.wn.iihe.ac.be		y + on
node19-17.wn.iihe.ac.be		y + on
node19-18.wn.iihe.ac.be		y + on
node19-19.wn.iihe.ac.be		y + on
node19-20.wn.iihe.ac.be		y + on
node19-21.wn.iihe.ac.be		y + on
node19-22.wn.iihe.ac.be		y + on
node19-23.wn.iihe.ac.be		y + on
node19-24.wn.iihe.ac.be		y + on
node19-25.wn.iihe.ac.be		y + on
node19-26.wn.iihe.ac.be		y + on
node19-27.wn.iihe.ac.be		y + on
node19-28.wn.iihe.ac.be		y + on
node19-29.wn.iihe.ac.be		y + on
node19-30.wn.iihe.ac.be		y + on
node19-31.wn.iihe.ac.be		y + on
node19-32.wn.iihe.ac.be		y + on
node20-1.wn.iihe.ac.be		y + on
node20-2.wn.iihe.ac.be		y + on
node20-3.wn.iihe.ac.be		y + on
node20-4.wn.iihe.ac.be		y + on
node20-5.wn.iihe.ac.be		y + on
node20-6.wn.iihe.ac.be		y + on
node20-7.wn.iihe.ac.be		y + on
node20-8.wn.iihe.ac.be		y + on
node20-9.wn.iihe.ac.be		y + on
node20-10.wn.iihe.ac.be		y + on
node20-11.wn.iihe.ac.be		y + on
node20-12.wn.iihe.ac.be		y + on
node20-13.wn.iihe.ac.be		y + on
node20-14.wn.iihe.ac.be		y + on
node20-15.wn.iihe.ac.be		y + on
node20-16.wn.iihe.ac.be		y + on
node20-17.wn.iihe.ac.be		y + on
node20-18.wn.iihe.ac.be		y + on
node20-19.wn.iihe.ac.be		y + on
node20-20.wn.iihe.ac.be		y + on
node20-21.wn.iihe.ac.be		y + on
node20-22.wn.iihe.ac.be		y + on
node20-23.wn.iihe.ac.be	need to be included in maui
node20-24.wn.iihe.ac.be	need to be included in maui

Template:TracNotice