Second phase of Oxford's move

Over the last couple of days Ewan and I have moved five 9TB disk servers, five twin worker nodes, a couple of head nodes and two UPS's. We had help moving the rack, and then we were able to reassemble it all. The site was back up and passing SAM tests in time for us to come out of scheduled maintenance at 1700.
The current setup means we have three ce's, the original SL3 based ce is driving most of the newer SL4 based WNs. The two new SL4 based ce's send jobs to a new torque server, and onto two subclusters, one for the 32 bit hardware (Dell 2.8GHz xeons), and the other the Intel Clovertown quads. We will migrate the workers off the old ce onto the new ones over the next few days, before decommissioning the original SL3 based ce.

The migration of data from our old se head node, is complete. There were three files that were listed in the database but did not exist on the physical storage. I used the scripts from :
https://twiki.cern.ch/twiki//bin/view/LCG/CheckDpmConsistency
which matched the three files names that were left in my dpm-drain logs.

Two of the files could be removed with rfrm but one refused to appear in the normal dpns-ls listing so was not removed.
We decided to ignore this one and remove the files system from the pool with the command.
dpm-rmfs --server t2se01.physics.ox.ac.uk --fs /storage

The new DPM head node will be setup and then the mysql database dumped and restored on to it shortly.

Meanwhile we are awaiting the backplanes in our storage servers to be swapped out to avoid the burnout issue we have suffered on one of them.

SouthGrid technical meeting will be held tomorrow at Birmingham.

Oxford DPM progressing Slowly

Removing zero length files from the DPM storage pool with the rfrm command has helped the dpm-drain command to start progressing again.
The command still fails after 10-20 files and the transfer speed in very low, but at least we are making progress.

SouthGrid Update

Having installed half of the Oxford cluster at Begbroke last Tuesday. The Air conditioning failed during the night, a valve on the Chillers failed cutting off the water supply to the Chillers which in turn switched themselves off. The room rapidly heated up to >40 degrees. After investigation and repairs the AC went back on and all has been well so far. More automated warning systems are required.

Cambridge have set up space tokens for both ATLASDATADISK and ATLASMCDISK. They have also started upgrading to SL4 (64bit) worker nodes.

Britol completed upgrading the Worker nodes to SL4 on Monday. They had some problems caused by se linux, preventing normal loging but all now appears well.

Oxford Update

Last week Ewan and I started the DIY move to Begbroke. We moved 40 1 u servers over two mornings. One of the (now empty) Dell racks was moved on Wednesday afternoon.
The worker nodes were reinstalled in that rack on Thursday, we had one psu failure out of 27 nodes. These nodes will be installed with sl4 shortly.

This week we have emptied one of the Viglen racks and moved the servers yesterday.
We hope to move the rack this afternoon and get the worker nodes back on line asap as we are at half capacity currently.

On Firday 28th March we had one of our new 9TB file servers burn out its backplane. This is very similar to the problems RAL have been seeing. The backplane was swapped out and the server is back on line now.
We are in talks with the supplier.

Oxford Update

Oxford adds another space token

With some advice from Graeme I have added another space token at Oxford
Commands recorded for posterity.

dpm-reservespace --gspace 3T --lifetime Inf --group atlas/Role=production --token_desc ATLASMCDISK
dpns-mkdir /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-chgrp atlas/Role=production /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-chmod 775 /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-setacl -m d:g:atlas/Role=production:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-setacl -m g:atlas/Role=production:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk

We started our DIY move today, two trips in the lab van with 8 1U servers each time.
Tomorrow we plan to move more old WN's and then an empty rack later in the week.

SouthGrid Update

Bristol:
The first stage of the HPC cluster is running LCG jobs, and is being correctly accounted for.

The HPC WNs have AMD 2218 cores, 2.6GHz; these are said to be
1.745 KSi2K each core.
Currently gridpp can run max 32 jobs on this small stage 1 HPC cluster;

Cambridge:
Santanu is continuing to work with LHCb to solve all the problems running their code at Cambridge.
The WNs will be upgraded to SL4 within the next few weeks.

Birmingham:
When over 100 of the 120 Babar cluster died after a power shutdown at the end of January, it was deemed not worth restoring the cluster. Two twin 1 u servers have been bought to replace this which will provide 32 cores and 78.4KSI2K.
The old escience cluster is being setup as an SL4 ce and WN farm in as a template for the way they will drive the new University 'Blue Bear' HPC cluster. This cluster is made up of 31 dual 3GHz xeons.
The main grid cluster (aka the atlas cluster) has been expanded to 60 cores.

The PPS is not being maintained at the moment.

RALPPD:
Chirs has got space tokens working at RALPP ( updated to dCach1.8-12p6 - from p4 - and also rebooted everything after putting in the srmSpaceManager enabled config files).
The new hardware has been installed:
8 boxes, 16 nodes, 32 CPUs so 128 cores.

CPUs are "E5410 @ 2.33GHz" not sure of the kSI2k rating yet.


JET:
Running stable. WNs were updated to SL4 earlier this year.

Oxford:
Quotes to move the kit to Begbroke seem too high so we are going to adopt a DIY approach.
Draining t2se01 is taking for ever. The dpm-drain command terminates sometimes after only 20mins (~6GB data transfered). We did however have a good run on the night of the 26th which lasted over 10 hours.
Oddly the error log files are often the same size although not totally consistent.

-rw-r--r--    1 root     root        22519 Feb 22 14:32 dpm-drain-errorlog-se01-1
-rw-r--r--    1 root     root        22519 Feb 22 15:56 dpm-drain-errorlog-se01-2
-rw-r--r--    1 root     root        22564 Feb 22 17:05 dpm-drain-errorlog-se01-3
-rw-r--r--    1 root     root        22519 Feb 22 18:20 dpm-drain-errorlog-se01-4
-rw-r--r--    1 root     root        22519 Feb 25 12:35 dpm-drain-errorlog-se01-5
-rw-r--r--    1 root     root        22519 Feb 25 13:04 dpm-drain-errorlog-se01-6
-rw-r--r--    1 root     root        22519 Feb 25 16:53 dpm-drain-errorlog-se01-7
-rw-r--r--    1 root     root        22519 Mar 26 13:51 dpm-drain-errorlog-se01-8
-rw-r--r--    1 root     root        22519 Mar 26 14:17 dpm-drain-errorlog-se01-9
-rw-r--r--    1 root     root      1193287 Mar 27 00:56 dpm-drain-errorlog-se01-10
-rw-r--r--    1 root     root       567836 Mar 27 13:59 dpm-drain-errorlog-se01-11
-rw-r--r--    1 root     root        25241 Mar 27 14:57 dpm-drain-errorlog-se01-12
-rw-r--r--    1 root     root        22598 Mar 27 15:37 dpm-drain-errorlog-se01-13
-rw-r--r--    1 root     root        22598 Mar 27 16:22 dpm-drain-errorlog-se01-14
-rw-r--r--    1 root     root        22598 Mar 27 17:11 dpm-drain-errorlog-se01-15
-rw-r--r--    1 root     root        22598 Mar 27 21:55 dpm-drain-errorlog-se01-16
-rw-r--r--    1 root     root        22598 Mar 27 23:15 dpm-drain-errorlog-se01-17

Oxford Update

The Original 74 cpu SL3 Dell workernodes have been taken down in preparation for reinstallation as SL4 worker nodes.
We will maintain a separate ce to drive these but intend to separate out the torque server.
The new ce will be an SL4 ce.
The ce and torque server are likely to be virtual machines running under VMware.


The move of the 4 racks up to Begbroke is still uncertain.
We are awaiting quotes from companies to move the equipment for us. The DIY price of just paying for a truck and driver has also been looked into but issues of insurance and warantee's may prevent use of this option.

The steps up to the new computer room have been made smaller to allow installation of a scissor lift, to lift racks to the false floor height. The date for the lift to be installed is still unclear.

The grand opening on the 15th April is all too close.

SouthGrid Technical Meeting at JET

The SouthGrid Technical Board met at JET.
All sites are moving towards SL4. The recent updates will be applied shortly.
The SouthGrid vo has been setup and a central LFC is being provided at RAL Tier 1 for it.
The outstanding tickets were looked at and all found to be solved. A problem was found, tickets still open in footprints were closed in GGUS, so the link may not be working properly.
Birmingham is setting up some ex escience nodes to be the interface to the new HPC cluster. This will bring back some of the spec int power lost due to hardware failures after recent electrical work.

Oxford Update

Plans to move the Oxford gridpp cluster up to Begbroke are being formulated.
The first part of the plan is to ensure that only these nodes are using the subnet in question. We did some tidying up over the last week or so, before having the subnet rerouted to both Physics and Begbroke. This change was made this morning at 8:50, and mostly went smoothly.
Our ui needs to be moved back on to the physics subnet to allow NFS mounting of home directories to work.
A new rack, PDU and network switch has been ordered to allow us to move a few test nodes up to Begbroke in advance of the main move.
We aim to complete the move late Jan/ early Feb.

The disk on our installation server which holds ganglia data and central syslog data failed today. We will restore from backups.
t2wn05 has a failed hard disk which may have been acting as a black hole over the weekend.

Working with ZEUS and LHCb VO's to improve usage of our cluster uncovered some configuration problems.

1. Not all the nodes had the latest DESY VOMS server certs applied (stopped zeus working)
2. sgm ROLES were not mapped correctly for LHCB.

Finally the APEL problems seem to be behind us.

1. Configuration seemed to have changed at the last running of yaim before Christmas which stopped any records getting published
2. Installing the latest Development Apel rpms fixed the problem of not seeing the newer spec value for our new ce.

Scheduled Power outage at Birmingham causes problems

The scheduled power outage at Birmingham on Saturday 8th December caused 19 Babar SL4 systems to fail. 4 bad disks appeared on the SL3 cluster. The age of this equipment is a cause for concern.

There has been some concern expressed at small sites such at Bristol that the number of Atlas tests submitted by Steve Lloyds tests can over whelm their sites.

Oxford Gridpp Site becomes an NGS Affiliate

Not to be out done by Scotgrid, I should also point out that Oxford became an NGS affiliate at the same meeting (Dec 6th). See https://www.ngs.ac.uk/guide/affiliates/oxford-gridpp/

Oxford have added support for vo.southgrid.ac.uk, gridpp and supernemo.vo.eu-egee.org

Birmingham HV Network Upgrade

High Voltage Network Upgrade, over this weekend, means several systems will be off over the weekend.
It is hoped to keep the core service nodes up and running, but the number of worker nodes will be limited.

ALICE VO Box was not accessable to the users for a day, no problems were found by Yves.
Now reported as OK.

SouthGrid Update

Bristol:
Had some problems with LHCb users
EDFA-JET:
Upgraded WN's to SL4
Birmingham:
Disk failed on the se raid 5 disk array.
Oxford:
Upgraded the SL3 cluster to update 37. Some problems with the se, the DPM pool nodes had not had the latest lcg-vomscerts rpm applied. Secondly the site-info.def file on some of the nodes had an old entry for the ops vo which meant the gridmap file was not being created correctly.
This was changed to include:

VO_OPS_VOMS_SERVERS="'vomss://lcg-voms.cern.ch:8443/voms/ops?/ops/'
                     'vomss://voms.cern.ch:8443/voms/ops?/ops/'"
VO_OPS_VOMSES="'ops lcg-voms.cern.ch 15009 /DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch ops'
               'ops voms.cern.ch     15009 /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch     ops'"

The addition of voms.cern.ch being the important bit, (lcg-voms.cern.ch) was the old entry.

RALPPD:
The BDII failed on Monday 3rd. A reboot fixed this.

So now that Oxford is uptodate we can go ahead and add support for some new VOs,
SouthGrid, gridpp and supernemo.

Random rm failures at Oxford

Random SAM test failures for rm, and later complaints from ATLAS were traced to one of the DPM pool nodes not having had the latest VOMS certs applied.

dCache Tuning

I've been having a few issues since the start of the CMS CSA07 data challenge with SAM test failures with what seem to be mostly timeouts against my dCache Storage Element so I've been looking at improving my setup.

One suggestion was to set up separate queues in dCache for local access (dcap, gsidcap and xrootd) and remote access (GridFTP).

In general this is supposed to help when local farm jobs are reading slowly from lots of files and blocking the queues preventing the short GridFTP jobs from starting. Which is not the current case on my Storage Element, but it might also help by limiting the number of concurrent GridFTP transfers, which are very resource hungry without limiting the local access which is not.

It was a very easy change to do requiring only changed to the /opt/d-cache/config/dCacheSetup file, not the indevidual batch files (on all the servers of course, though). I uncommented and set the following variables:

poolIoQueue=dcapQ,gftpQ
gsidcapIoQueue=dcapQ
dcapIoQueue=dcapQ
gsiftpIoQueue=gftpQ
remoteGsiftpIoQueue=gftpQ


The first variable sets up the two queues (the first queue is also the default on if no queue is specified).

Then the rest of the settings specify which queue the different doors use.

Unfortunately, the queue lengths are set per pool in the pool setup file so I had to edit a file for each pool on all the disk servers to change:

mover set max active NNNN

to:

mover set max active -queue=dcapQ 1000
mover set max active -queue=gftpQ 3

After the changes to the config files I then had to restart all the services to pick up the new config. I also took the opportunity to enable readonly xrootd access to the SE but adding:

XROOTD=yes

to /opt/d-cache/etc/node_config on all the nodes

and setting:

xrootdIsReadOnly=true

in the dCacheSetup file.

After the restart the new queues showed up in the queue info pages and the xrootd doors on all the nodes showed up on the Cell Services page.

I was also able to read files out from the xrootd door using standard babat tools (and was correctly blocked from writing data).

SL4 Worker Node Migration at RALPP

Since I've now finished the migration my worker nodes to SL4 I thought I should describe the method used.

The basic decision was to try to keep running an SL3 service in parallel with the initial test SL4 service and then gradually migrate nodes to the new service once it was production quality. I already had split my Torque/Maui services off onto a separate node and wanted to keep that setup with the SL4 service but did not want to (a) duplicate the torque server or (b) create another 24 queues for all the VOs. To get round this I decided to:

• Install a new "SL4" CE pointing to the production PBS node, this needed a different site info.def file with it named as the CE_HOST and the GlueOperatingSystem settings set for SL4 obviously
  
• Create node properties on the SL3 and SL4 nodes to let the batch system route jobs based on OS
• Hack the lcgpbs jobmanagers on the two CEs to apply requirements on the node properties as it submits the job

Running multiple CEs all pointing to the same torque server is fairly simple to do, there is a "BATCH_SERVER" setting in YAIM (3.1 and later, TORQUE_SERVER before that) that you just point at your the torque/maui server and that configures the CE to submit it jobs via that machine. Then there are a couple of other things you have to take care of:

1. The gridmapdir has to be shared between all the CEs. Otherwise there is a possibility that either the same DN will be mapped to multiple pool accounts or worse that different DNs will be mapped to the same pool account by the different CEs.
2. The worker nodes need to have the ssh host keys for all the CEs to be able to get the job data back but YAIM will only set one up. The fix is to edit the NODES line in "/opt/edg/etc/edg-pbs-knownhosts.conf" to add all the CEs and your torque server
3. If the CEs are submitting the same worker nodes you might also want to mount the VO tag are across all the CEs so that VOs don't have to publish the same tags to all the CEs

Node properties are very easy either just edit the torque nodes file to add them or use "qmgr -c "set node $node properties += SL4". I also added "test" and "prod" properties to all the nodes but more of that below.

Finally I needed to change the job manager to require the properties to direct jobs going to the different CEs to different classes of workers based on the above properties. The lcgpbs jobmanager already writes a node requirement in into the job script it submits to torque and so it is easy to rewrite this to add node properties as well. If you look in "/opt/globus/setup/globus/lcgpbs.in" you'll see three places where it writes "#PBS -l nodes=" to set the requirement on the number of CPUs and you need to add :SL4 (or :SL3) to the end of the write.

After doing that, installing some SL4 worker nodes was very simple, about the only necessary change to the site-info.def file was to change the "GLOBUS_TCP_PORT_RANGE" to be space separated rather than comma separated.

With the above hacks in place I was able to leave my old CE happily submitting jobs to the SL3 nodes while I was testing the SL4 worker nodes then gradually move the worker nodes over to Sl4. Before Moving the final worker nodes over I modified the batch system information provider to report the queues as "Draining" whatever their real status. Once all the worker nodes were migrated to SL4 I could just remove the changes to the lcgpbs jobmanager changes and both CEs became equivalent.

Oxford's Tier 2 Upgrade is joining the grid.

The 22 new worker nodes are starting to come on line now.
They are running SL4 32bit mode for now. They will provide an additional 431 K Spec Int 2000.

A second ce t2ce03.physics.ox.ac.uk has been setup to serve the SL4 WN's. We had some trouble with the BDII being on the original ce so have split that function off onto a new node (Well actually a VM ).

The upgrade also includes 4 heads nodes with dual PSUs, and mirrored systems disks, which can be used for service functions or as worker nodes. All the head nodes and disk servers are protected by UPS.

The 11 storage servers (9TB usable each) will be brought on line over the next week.

The two new (Viglen supplied) racks are on the right hand side, with the older Dell kit on the left.

Oxford Local Computer Room Goes Live

The local computer room was completed last Friday. All power is ready under each of the 21 rack positions. Each rack position has 4 CAT6 cables connected to the networking rack which can be seen. Other things completed were; ceiling lights, painting, smoke detection system, and door fitting.

On Monday 10th two existing compute racks were installed and two empty racks for the cluster upgrade arrived. A rack full of worker nodes for the existing grid cluster can be seen and is up and running.

Today the servers arrived from Viglen and installation has commenced.

Oxford Computer Room Update

Progress on wiring for Power and Networking is scheduled to be completed this week.

The Oxford Grid Cluster upgrade has been ordered, and should be delivered in early September, to be installed here.