Over the last couple of days Ewan and I have moved five 9TB disk servers, five twin worker nodes, a couple of head nodes and two UPS's. We had help moving the rack, and then we were able to reassemble it all. The site was back up and passing SAM tests in time for us to come out of scheduled maintenance at 1700.
The current setup means we have three ce's, the original SL3 based ce is driving most of the newer SL4 based WNs. The two new SL4 based ce's send jobs to a new torque server, and onto two subclusters, one for the 32 bit hardware (Dell 2.8GHz xeons), and the other the Intel Clovertown quads. We will migrate the workers off the old ce onto the new ones over the next few days, before decommissioning the original SL3 based ce.
The migration of data from our old se head node, is complete. There were three files that were listed in the database but did not exist on the physical storage. I used the scripts from :
which matched the three files names that were left in my dpm-drain logs.
Two of the files could be removed with rfrm but one refused to appear in the normal dpns-ls listing so was not removed.
We decided to ignore this one and remove the files system from the pool with the command.
dpm-rmfs --server t2se01.physics.ox.ac.uk --fs /storage
The new DPM head node will be setup and then the mysql database dumped and restored on to it shortly.
Meanwhile we are awaiting the backplanes in our storage servers to be swapped out to avoid the burnout issue we have suffered on one of them.
SouthGrid technical meeting will be held tomorrow at Birmingham.