SouthGrid

Installing and Deploying a Cluster Publisher

2011-09-06T13:53:00.004+01:00

As part of the battle to replace out LCG-CEs with CreamCEs I realised that the reason one of our new CreamCEs was not getting many jobs was because it was not publishing a cluster/subcluster into the BDII (despite having a /var/lib/bdii/gip/static-file-Cluster.ldif file) and so I guess wasn't matching any resources.

Since, I eventually wanted to go to a stand alone Cluster Publisher I thought it would be easiest to push ahead and install that rather than try to install one one the CreamCE and remove it later.

So with a shiny new VM in hand and certificate I plunged onwards.

First step was to define the cluster variables in site-info.def (or in this case a specific node file):


cat /opt/glite/yaim/etc/nodes/heplnv146.pp.rl.ac.uk
CE_HOST_heplnx206_pp_rl_ac_uk_CE_TYPE=cream
CE_HOST_heplnx206_pp_rl_ac_uk_CE_InfoJobManager=pbs
CE_HOST_heplnx206_pp_rl_ac_uk_QUEUES="grid"
CE_HOST_heplnx207_pp_rl_ac_uk_CE_TYPE=cream
CE_HOST_heplnx207_pp_rl_ac_uk_CE_InfoJobManager=pbs
CE_HOST_heplnx207_pp_rl_ac_uk_QUEUES="grid"
CLUSTER_HOST=heplnv146.pp.rl.ac.uk
CLUSTERS=GRID
CLUSTER_GRID_CLUSTER_UniqueID=grid.pp.rl.ac.uk
CLUSTER_GRID_CLUSTER_Name=grid.pp.rl.ac.uk
CLUSTER_GRID_SITE_UniqueID=UKI-SOUTHGRID-RALPP
CLUSTER_GRID_CE_HOSTS="heplnx206.pp.rl.ac.uk heplnx207.pp.rl.ac.uk"
CLUSTER_GRID_SUBCLUSTERS="GRID"
SUBCLUSTER_GRID_SUBCLUSTER_UniqueID=grid.pp.rl.ac.uk
SUBCLUSTER_GRID_HOST_ApplicationSoftwareRunTimeEnvironment="
        LCG-2
        LCG-2_1_0
        LCG-2_1_1
        LCG-2_2_0
        LCG-2_3_0
        LCG-2_3_1
        LCG-2_4_0
        LCG-2_5_0
        LCG-2_6_0
        LCG-2_7_0
        GLITE-3_0_0
        RALPP
        SOUTHHGRID
        GRIDPP
        R-GMA
"
SUBCLUSTER_GRID_HOST_ArchitectureSMPSize=4
SUBCLUSTER_GRID_HOST_ArchitecturePlatformType=x86_64
SUBCLUSTER_GRID_HOST_BenchmarkSF00=0
SUBCLUSTER_GRID_HOST_BenchmarkSI00=2390
SUBCLUSTER_GRID_HOST_MainMemoryRAMSize=2000
SUBCLUSTER_GRID_HOST_MainMemoryVirtualSize=2000
SUBCLUSTER_GRID_HOST_NetworkAdapterInboundIP=FALSE
SUBCLUSTER_GRID_HOST_NetworkAdapterOutboundIP=TRUE
SUBCLUSTER_GRID_HOST_OperatingSystemName=ScientificSL
SUBCLUSTER_GRID_HOST_OperatingSystemRelease=5.4
SUBCLUSTER_GRID_HOST_OperatingSystemVersion=Boron
SUBCLUSTER_GRID_HOST_ProcessorClockSpeed=2300
SUBCLUSTER_GRID_HOST_ProcessorModel=Xeon
SUBCLUSTER_GRID_HOST_ProcessorOtherDescription='Cores=3.7656,Benchmark=9.56-HEP-SPEC06'
SUBCLUSTER_GRID_HOST_ProcessorVendor=Intel
SUBCLUSTER_GRID_SUBCLUSTER_Name=grid.pp.rl.ac.uk
SUBCLUSTER_GRID_SUBCLUSTER_PhysicalCPUs=546
SUBCLUSTER_GRID_SUBCLUSTER_LogicalCPUs=2056
SUBCLUSTER_GRID_SUBCLUSTER_WNTmpDir=/scratch

Then it was a simple case of installing the rpms and running YAIM:


yum install emi-cluster
/opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-CLUSTER

At that point we seemed to have a working system, the BDII was running and queriable, I count connect to the gridftp server and it had set up expriment and cluster directories in /opt/edg/var/info/ and /opt/glite/var/info/.

Fine, next step was to rsync the contents of those directories from the torque server that then exports them to the CEs - well actually to /export/gridtags and /export/glitetags and symlink the previous locations to those. cfengine had already set the node up as a nfs server for me so exporting the new areas and updating the CEs to mount it from there was a matter of moments.

A quick check of the resource BDII looked fine so it was a simple matter to add the new source into the site bdii and tweak the static-file-CE.ldif file on the CreamCE to assign it to the new cluster.

One thing remained, when testing the gridftp server with uberftp* I'd noticed that I was not mapped to my usual pool account - not surprising as I had not mounted the site gridmapdir so it was using its local one. However, reasoning that the gridftp server was the same rpm as the one on the CreamCE that was using Argus for authentication and mapping I had a poke around on the CreamCE and in YAIM and tried installing the argus-gsi-pep-callout rpm and coping over /etc/grid-security/gsi-authz.conf and /etc/grid-security/gsi-pep-callout.conf from the CreamCE.

Another quick test with uberftp and yes, I am mapped to my normal pool account so it appears I have a Cluster Publisher with Argus integration working. That means the one things at the site not using Argus are the gLite CreamCE which will be replaced soon by another EMI one and dCache which will get banning from Argus when I update to the next Golden Release.

*uberftp heplnv146.pp.rl.ac.uk "ls /etc"

EMI CREAM

2011-07-22T10:11:00.002+01:00

We have installed emi creamce at Oxford. It was quite straight forward and apparently everything was setup by yaim properly except that emi cream uses normal /etc/, /usr/ directories instead of /opt/glite. It uses just one repository for all packages, no more separate TORQUE_* repositories.
Jobs were running perfectly and all test jobs completed successfully. But it was only getting lhcbpilot jobs and after looking more closely it was the classic "GlueCEStateWaitingJobs: 444444" problem.

Drilling through many layer of wrapper it comes to this issue
/sbin/runuser -s /bin/sh ldap -c "diagnose -g --host=t2ce02.physics.ox.ac.uk"
ERROR: 'diagnose' failed
ERROR: user 'ldap' is not authorized to execute command 'diagnose'

I think this is the less documented part of emi creamce. In glite, slapd and bdii-update process was run by edguser but with emi it is run by ldap user.
Edited maui.cfg file
ADMIN3 edginfo rgma edguser ldap

It solved the problem as I was using our site wide maui.cfg file instead of default created by yaim. Just a heads-up if you are planning to install emi creamce

SAM to MyEGEE to finally MyEGI

2011-03-11T18:36:00.000+01:00

I have updated to latest release of wlcg nagios to gridppnagios. It is a major release in the sense that it stopped configuring MyEGEE for portal and replaced it with MyEGI. MyEGEE would be there until I drop the myegee DB from gridppnagios machine but don't trust it anymore. I got two complain about MyEGEE within few hours of updating it so I can say that people are looking at it.
The other main change is that now Nagios Configuration Generator(NCG) is using Aggregated Topology Provider(ATP) instead of SAMDB to configure nagios. ATP is part of the ROC/NGI nagios package which aggregate information from GOCDB, Top BDII and VO feed etc and it is single authoritative information source with topology information. But it is the central ATP(http://grid-monitoring.cern.ch/atp) which is being used by all ROC/NGI's for topology configuration for the sake of uniformity and probably reliability . Old SAM infrastructure can now retire in peace.
So MyEGI, It is a kind of all in one (https://gridppnagios.physics.ox.ac.uk/myegi).
It has Gridmap, metric status, history and so on. Aesthetically MyEGEE was better but MyEGI has more functionality and if you are still not convince then check the comparison of SAM, MyEGEE and MyEGI here (https://tomtools.cern.ch/confluence/display/SAM/MyEGI+vs+MyEGEE+vs+SAM+Portal ).
MyEGI have very good search options and also has advanced filter so you can optimize your search and add URL to your bookmark for instance status of your site.
I just discovered two bugs and the irritating things is that it is showing advance date on history bar. So if you want to see the status at 11 March, check for 12 March !
A bug has been opened and hopefully it will be fixed soon
https://tomtools.cern.ch/jira/browse/SAM-1325
https://tomtools.cern.ch/jira/browse/SAM-1326

Going through the Argus Valley

2011-02-28T15:17:00.002+01:00

Being an early adopter site for Argus, Oxford got one of the first MUPJ from ATLAS using glexec through Argus and it failed! although we were passing ops glexec tests for long.
Our understanding of Argus was that it must have a policy which authorize pilots to switch to a normal user, so I had a policy like this to authorize pilot for glexec

resource "http://authz-interop.org/xacml/resource/resource-type/wn" {
obligation "http://glite.org/xacml/obligation/local-environment-map" {
}

action "http://glite.org/xacml/action/execute" {
rule permit { pfqan="/ops/Role=pilot" }
rule permit { pfqan="/atlas/Role=pilot" }
rule permit { pfqan="/cms/Role=pilot" }
}
}

After discussion with Argus experts on mailing list, it turned out that when pilot framework ask glexec to switch user from pilot to the effective user, LCMAP PEP plugin send the proxy of effective user to ARGUS server for authorization and mapping. So Argus must have policy which authorize effective user also. I have changed policy to look like that

rule permit {pfqan = "/atlas/Role=pilot" }
rule permit {pfqan = "/atlas/Role=lcgadmin" }
rule permit {pfqan = "/atlas/Role=production" }
rule permit {pfqan = "/atlas/" }

It solved the problem. Doesn't it look like that every atlas user is allowed to switch identity through glexec ? As for as Argus is concerned, yes. But glexec configuration is defined at WN and only groups which are whitelisted at /opt/glite/etc/glexec.conf are allowed to use glexec, any other user trying glexec will be shot down at WN itself. By default only pilot users are whitelisted at WN.
So in nutshell, policies at Argus should resemble that of the CE.

glite-APEL Node

2011-01-07T14:47:00.000+01:00

On Thursday 9th December we brought the new glite-APEL box on line.

The VM hosted by t2delltest, had already been installed and Kashif had installed the cert.

We ran apel on all the ce's and t2torque02 and then one last time on t2mon02.

Then reconfigured t2ce02 to point at the new apel box and ran apel on it. We saw new records created on the box. (After sorting some permissions issues, need to rerun yaim with each ce (and t2torque02) set in the site-info.def file. Each run did the magic to allow that node to write to the db. (FQDN's should be used).
We then changed the reference to t2mon02 to t2apel01 in the site-info.def file on pplxconfig and it propagated round the other nodes.
The first run that night failed due to a java out of memory error.
I tweaked the config file /opt/glite/etc/glite-apel-publisher/publisher-config-yaim.xml
to

150000
from the original 300000

All apel logfiles on all ce's , t2torque02 and t2apel01 now appear to be good.
Cristina can see records appearing at RAL.

The old mysql database from t2mon02 has been backed up in /data/sysadmin (pplxfs2)

Tracing a Grid Job (A recap)

2010-09-09T16:14:00.001+01:00

Just in case we should forget how to trace a grid job I record some steps below.

For example you discover via a CMS SAM page you are failing some test (could equally be any other SAM page such as LHCb) , you click on the detailed out put and see a reference to the job id:
on t2ce05 contains the string: sOFavxScVKU-GbSYaCmx-A
on t2ce05
grep sOFavxScVKU-GbSYaCmx-A /opt/edg/var/gatekeeper/grid-jobmap_20100906
reveals the batch system job id: lrmsID=2998805.t2torque02.physics.ox.ac.uk
on the batch server t2torque02 in our case, either:
tracejob 2998805

or

grep 2998805 /var/spool/pbs/server_logs/20100909

The tracejob option is easier!

This will let you know which worker node ran the job. You can then have a look at it to check for full disks, memory faults etc or segfaults in the log files......

Now in reverse

A job is misbehaving on your node and you need to see who is running it.
The special case here is that its an ATLAS pilot job, this does not have a normal grid job id.

Get the PID from top, use
pstree -H pid
to highlight the processes parents.
(Use pstree -A -H pid if on an putty window on Windows)

This reveals which pbs job it is
eg 3020508.t2torque02.physics.ox.ac.uk

The job can be traced on the panda monitor, using the search facility on the LH toolbar.
This gives the job details including the users name. A GGUS ticket could then be raised against ATLAS asking for the user to be informed.

APEL on ngsce-test

2010-09-01T13:14:00.001+01:00

APEL was failing on ngsce-test with the following error.

java.io.FileNotFoundException: /var/spool/pbs/server_priv/accounting/20090522 (Too many open files)

The solution was to type:
ulimit -n 10240

I've added this to the /opt/glite/bin/apel-pbs-log-parser script.

A fix is in test, so a new version of APEL will fix it.
see GGUS ticket
https://gus.fzk.de/ws/ticket_info.php?ticket=60674

Argus Server at Oxford

2010-08-27T11:54:00.002+01:00

We finally managed to install Argus server at Oxford with messy workaround. Installation and configuration was reasonably ok, and once policy structure was clear then writing and loading policy was also easy. Details are here http://www.gridpp.ac.uk/wiki/Oxford.

The main issue was host certificate issued by UK CA which contains an "emailAddress" and supposedly this is depreciated year(s) ago and most developers assume that there is no "emailAddress" in host certificate. Although still it is a bug in Argus and hopefully would be resolved in next release.
So the workaround
By default pap-admin command uses host certificate in /etc/grid-security/ if started from root but since there is a problem with host certificate so I copied my personal certificate proxy from UI and started pap-admin using that proxy. Then added ACE
pap-admin ace
"/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=t2argus02.physics.ox.ac.uk/OID.1.2.840.113549.1.9.1=lcg_manager@physics.ox.ac.uk" ALL
This workaround was suggested by Andrea Ceccanti

The only issue is that if you want to restart pap service then first remove ACE using remove-ace command, restart pap and then add ACE again.

Oxford's blanking panels

2010-06-23T15:03:00.005+01:00

Having just read Stuart's ScotGrid blog post about cooling in the top of racks I thought I'd let you know about the panels we use.

We have been specifying that all empty racks slots should be filled by blanking panels since our 2007 purchase. The they used to use metal blanking panels.

These days they tend to supply the 1U APC plastic clip in panels, as can be seen in the RH rack in the photo.
These cost £25-£30 per pack of 10 but we managed to get a bulk (200) purchase in 2008 which worked out at about £1.69 each.

http://www.apc.com/resource/include/techspec_index.cfm?base_sku=AR8136BLK200

Jobs with analysis role

2010-05-18T12:44:00.003+01:00

It started with a ticket from dzero about job failure at creamce at oxford. On investigation it was found that these jobs were coming with /dzero/users/Role=analysis/Capability=NULL and expectantly lcmaps failing with this error "no entry found for /dzero/users/Role=NULL/Capability=NULL ".
But the jobs from the same user were running on lcg-CE so on further investigation it turn out that lcmaps-voms plugins were failing on lcg-CE too but as per lcmaps policy it runs lcmaps-poolacount plugin after voms plugin failure and lcmaps-poolaccount uses individual DN mapping from grid-mapfile. So lcg-CE was mapping correctly to dzero pool account but through wrong procedure.
creamce don't use edg-mkgridmap file for creating grid-mapfile so no individual mapping is defined in grid-mapfile.
Solution was quite easy and we have to just define MAP_WILDCARDS=yes in vo.d/dzero and rerunning yaim created a slightly different grid-mapfile and groupmapfile with wild-cards.

dzero/Role=lcgadmin/Capability=NULL" dzerosgm
"/dzero/Role=lcgadmin" dzerosgm
"/dzero/Role=production/Capability=NULL" dzeroprd
"/dzero/Role=production" dzeroprd
"/dzero/*/Role=*" .dzero
"/dzero/*" .dzero
"/dzero/Role=NULL/Capability=NULL" .dzero
"/dzero" .dzero

So any job coming with different Role would be mapped to normal pool account.
The issue was discussed in this ticket https://savannah.cern.ch/bugs/index.php?26990

A week of upgrades for the RAL Tier 2 - Part 1 -The Network

2009-11-07T12:25:00.004+01:00

Well it has been a long week at the RAL Tier 2. We've finally had our much postponed downtime to update our dCache installation (delayed once when one of the disk servers got a corrupt filesystem, then to avoid a CMS analysis test and finally to avoid an Atlas analysis test). The delays, however, did mean we could also include the long planned network upgrade in the downtime - this was probably a good thing.

So we had quite a programme of work for a five day downtime:

Replace the PNFS namespace in dCache with Chimera
Update dCache from 1.9.1 to the "Golden Release" 1.9.5
Install a new network switch and set up a 10Gb/s link between the two halves of our farm

Indeed, heading into work on Friday with neither dCache nor the network working, I thought I would be extending the downtime into the next week but by lunchtime things had improved and we were able to come out of the downtime on time at 5pm - although despite a full suit of "OK" SAM tests GridView still has us down until nearly eight o'clock.

Taking the last of the upgrades first: before last week we had the two halves of our farm in two different rooms. Each half of the farm has it's own Nortel 55XX network stack. Most of the storage is in the room known as Lab 8 in the R1 office building with a 10Gb/s connection to site Router A, whilst most of the compute nodes are in the Atlas lower machine room, A5Lower, with a 2x1Gb/s connection to Site Router A. That 2x1Gb/s connection between the storage and compute nodes was our main bottleneck - it would regularly run at over 99% capacity for days during Atlas Hammercloud tests.

The Plan was to install a Nortel 5650 switch into the stack in A5Lower then set up a direct 10Gb/s fibre link from there to Lab 8 - cutting out the 2x1GB/s link and Router A. That sounded fairly trivial and when I went down with Networking on Thursday afternoon to set it up I expected to be back in a hour to carry on struggling with our, at that time, broken dCache.

Due to cabling issues we had to re-order the switches in the stack and I also had to swap out a 5510 I had borrowed from the Tier 1 and replace it with a new one. So we broke up the current stack and tried to stack the 5650 with one of the 5510s. According to everything we had read they should have see each other, the 5650 should have downloaded an updated version of the firmware and software to the older 5510 and then the should have joined together as a single switch. But ours did not talk to each other.

Well possibly the version of the software on the 5510s was too old, so we went to each switch in turn, set it up with an IP address downloaded a new version on the firmware and software and restarted it.

By the end of Thursday - we were more-or-less back where we had started - we had a stack of 5510s (still without the 5650) .

On Friday morning Nick found a setting on the 5650 to allow "hybid stack mode" and suddenly everything worked.

We soon had all the correct VLANs set up and the two halves of our network were talking over the new fast link.

Backing up MySQL databases

2009-10-20T09:33:00.003+01:00

Oxford have installed a simple script to backup the DPM mysql db once a day at 6am.
The script was loosely based on Glasgow's example here .

In order to restrict the file names produced to just 7, I've opted to use the current day rather than date.

[root@t2se01 ~]# cat /root/mysql-dump-pdg.pl
#!/usr/bin/perl
#
# Loosely based on the Glasgow script but simplified.
#
# Select the current day only as we want to have just seven unique file names which will be overwritten
# thus reducing the total backup size.

@weekDays = qw(Sunday Monday Tuesday Wednesday Thursday Friday Saturday);
($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
$theTime = "$weekDays[$dayOfWeek]";
#print $theTime;

$backup_dir="/var/lib/mysqldumps";
$mysql_user="root";
$mysql_pw_file="/root/mysql-pw";
$keep_days=7;

# Read mysql password
open(PW, $mysql_pw_file) || die "Failed to open password file $mysql_pw_file: $!\n";
$mysql_pw=;
chomp $mysql_pw;
close PW;

# Dump the db now
chdir $backup_dir || die "Failed to change to backup directory $backup_dir: $!\n";

system "/usr/bin/mysqldump --user=$mysql_user --password=$mysql_pw --opt --all-databases | gzip -c > mysql-dump-$theTime.sql.gz";
die "Mysql failed died with exit code $?\n" if $? != 0;

This is run by /etc/cron.d/mysql-dump
PATH=/sbin:/bin:/usr/sbin:/usr/bin
0 6 * * * root /root/mysql-dump-pdg.pl

So far it seems to work in testing!

Oxford Grid now SL5

2009-10-19T10:35:00.002+01:00

All but one worker node on the Oxford Grid site has been reinstalled running SL5.
Currently these are served by one ce, t2ce05, but more will be added shortly to offer resilience.

Quarterly Report DPM script

2009-10-14T10:49:00.004+01:00

Each quarter we needs to report on disk usage at our sites.
This can be tricky but the following script will help at DPM sites:

#!/bin/bash

DAY=`date +%F`
echo $DAY
for zz in `dpns-ls /dpm/physics.ox.ac.uk/home/`;do
dpns-du -z -s /dpm/physics.ox.ac.uk/home/$zz>>Oxford-SE-Usage-$DAY;
done

You will need to modify it appropriately for your site.
Extra added 20.10.09
This makes use of the dpns-du command in the gridpp-dpm toolkit available from :
http://www.sysadmin.hep.ac.uk/rpms/fabric-management/RPMS.storage/

Details of the other commands are on the wiki

120 new cores for EFDA-JET

2009-03-05T10:47:00.004+01:00

30 new Sunfire 2200 m2 servers have been incorporated into the EFDA-JET site. Each has dual processor dual core Opteron 2218 processors, so that increases the number of Worker Nodes cores by 120 up to 254. Each node has 8GB RAM.

CMS at Oxford

2009-02-26T13:49:00.002+01:00

Oxford was failing a ce CMS SAM test with a warning, probably due to some permissions problems in the se.
Following commands illuminated things:

This extract from /var/log/dpm/log
02/26 10:49:30 3869,24 dpm_srv_proc_put: processing request c75ce541-b2cd-4bdc-bf8f-c86ecb0be6ed from /C=UK/O=eScience/OU=CLRC/L=RAL/CN=chris cms brew
02/26 10:49:30 3869,24 dpm_srv_proc_put: calling Cns_stat
02/26 10:49:30 3869,24 dpm_srv_proc_put: calling Cns_creatx
02/26 10:49:30 3869,24 dpm_srv_proc_put: srm://t2se01.physics.ox.ac.uk:8446/srm/managerv2?SFN=/dpm/physics.ox.ac.uk/home/cms/store/user/test/oneEvt.root: DPM_FAILED (Permission denied)
02/26 10:49:30 3869,24 dpm_srv_proc_put: returns 0, status=DPM_FAILED (Permission denied)

Shows the test file creation failing

[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/
drwxrwxr-x 1 24135 1399 0 Jan 13 18:55 PhEDEx_Debug
drwxrwxr-x 2 24135 3490 0 Oct 13 12:15 PhEDEx_LoadTest07
drwxrwxr-x 0 24135 1399 0 Feb 26 12:20 brew
drwxrwxr-x 2 24135 1399 0 Jan 27 15:16 mc
drwxrwxr-x 2 24351 3422 0 Feb 06 18:57 unmerged
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 user
[root@t2se01 dpm]# dpns-listgrpmap |grep 1399
1399 cms
[root@t2se01 dpm]# dpns-listgrpmap |grep 3406
3406 cms/Role=lcgadmin
[root@t2se01 dpm]# dpns-getacl /dpm/physics.ox.ac.uk/home/cms/store/
# file: /dpm/physics.ox.ac.uk/home/cms/store/
# owner: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=chris cms brew
# group: cms/Role=cmst1admin
user::rwx
group::rwx #effective:rwx
group:cms/Role=lcgadmin:rwx #effective:rwx
group:cms/Role=production:rwx #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:cms/Role=lcgadmin:rwx
default:group:cms/Role=production:rwx
default:mask::rwx
default:other::r-x
[root@t2se01 dpm]# dpns-getacl /dpm/physics.ox.ac.uk/home/cms/store/brew
# file: /dpm/physics.ox.ac.uk/home/cms/store/brew
# owner: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=chris cms brew
# group: cms
user::rwx
group::rwx #effective:rwx
group:cms/Role=lcgadmin:rwx #effective:rwx
group:cms/Role=production:rwx #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:cms/Role=lcgadmin:rwx
default:group:cms/Role=production:rwx
default:mask::rwx
default:other::r-x
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/
drwxrwxr-x 1 24135 1399 0 Jan 13 18:55 PhEDEx_Debug
drwxrwxr-x 2 24135 3490 0 Oct 13 12:15 PhEDEx_LoadTest07
drwxrwxr-x 0 24135 1399 0 Feb 26 12:20 brew
drwxrwxr-x 2 24135 1399 0 Jan 27 15:16 mc
drwxrwxr-x 2 24351 3422 0 Feb 06 18:57 unmerged
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 user
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 test
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user/test
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 SAM-t2se01.physics.ox.ac.uk
[root@t2se01 dpm]# dpns-chgrp 1399 /dpm/physics.ox.ac.uk/home/cms/store/user
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 test
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/
drwxrwxr-x 1 24135 1399 0 Jan 13 18:55 PhEDEx_Debug
drwxrwxr-x 2 24135 3490 0 Oct 13 12:15 PhEDEx_LoadTest07
drwxrwxr-x 1 24135 1399 0 Feb 26 12:33 brew
drwxrwxr-x 2 24135 1399 0 Jan 27 15:16 mc
drwxrwxr-x 2 24351 3422 0 Feb 06 18:57 unmerged
drwxrwxr-x 1 24352 1399 0 Jan 21 18:21 user
[root@t2se01 dpm]# dpns-chgrp 1399 /dpm/physics.ox.ac.uk/home/cms/store/user/test
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/brew
-rw-rw-r-- 1 24135 1399 4788418 Feb 26 12:34 oneEvt.root
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user/test
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 SAM-t2se01.physics.ox.ac.uk
-rw-rw-r-- 1 24135 1399 4788418 Feb 26 12:36 oneEvt.root

Automount problems on torque server

2008-12-19T10:50:00.003+01:00

We've been having a few problems with our torque server failing to automout disks randomly.

Most of the time the mounts succeeded but occasionally they would fail with just:


Dec 19 08:05:06 heplnx201 kernel: RPC: error 5 connecting to server nfsserver
Dec 19 08:05:06 heplnx201 automount[23438]: >> mount: nfsserver:/opt/ppd/mount: can't read superblock
Dec 19 08:05:06 heplnx201 automount[23438]: mount(nfs): nfs: mount failure nfsserver:/opt/ppd/mount on /net/mount
Dec 19 08:05:06 heplnx201 automount[23438]: failed to mount /net/mount
Dec 19 08:05:07 heplnx201 kernel: RPC: Can't bind to reserved port (98).
Dec 19 08:05:07 heplnx201 kernel: RPC: can't bind to reserved port.

With the wonders of Google I was able to find out that error 98 is address in use and that what is going on is that the client is unable to find a free port in it's port range to initiate the connection to the server.

The culprit seems to be torque, which when I checked with a netstat -a was using very single port from 600 to 1023, which quite neatly overlaid the nfs client port range of 600-1023.

Here Google failed me and I was unable to find anyway to limit the port range used by torque.

So for now I've taken the quick option of extending the nfs client port range down to port 300 with:

echo 300 > /proc/sys/sunrpc/min_resvport

I think I'd like to move the nfs client port range out of the priveledged port range altogether. I think this should be possible, the RFC says that it SHOULD use a port below 1023 but MAY use a higher port, but I'd like to test it a bit before I configure a major server like that.

static-file-Cluster.ldif edit required post yaim at Oxford

2008-12-19T10:41:00.002+01:00

Every time we run yaim at Oxford we have to fix the number of cpu's in our cluster by hand.
on t2ce02:
diff static-file-Cluster.ldif-fixed /opt/glite/etc/gip/ldif/static-file-Cluster.ldif
64c64
< GlueSubClusterPhysicalCPUs: 384
---
> GlueSubClusterPhysicalCPUs: 2
[root@t2ce02 ~]# cp static-file-Cluster.ldif-fixed /opt/glite/etc/gip/ldif/static-file-Cluster.ldif

On t2ce04:
Physical cpu's needs to be 74. After the change the ldap query shows:
ldapsearch -x -H ldap://t2bdii01.physics.ox.ac.uk:2170 -b Mds-vo-name=UKI-SOUTHGRID-OX-HEP,o=grid|grep -i physicalcpu
GlueSubClusterPhysicalCPUs: 74
GlueSubClusterPhysicalCPUs: 384

EFDA-JET Service nodes upgraded to glite 3.1

2008-12-16T16:30:00.008+01:00

We upgraded our service nodes to Scientific Linux 4.7 and glite-3.1. The worker nodes had been upgraded earlier. The problems/issues we had while upgrading to Scientific Linux 4.7 are listed below:

Storage Engine

While installing the SE glite middleware (glite-SE_dpm_mysql), there was
a missing dependency issue for the perl-SOAP-Lite package.

Error: Missing Dependency: perl-SOAP-Lite >= 0.67 is needed by package
gridview-wsclient-common

Doing a

# yum install perl-SOAP-Lite

only installs perl-SOAP-Lite-0.65, which is lower than the version needed.

The perl-SOAP-Lite rpm was downloaded from a different repository. We
initially downloaded the perl-SOAP-Lite-0.67.el4 but this one failed to install as it needed MQSeries and other packages to be installed. We finally downloaded perl-SOAP-Lite-0.67-1.1.fc1.rf.noarch.rpm and it installed without any problems.

When the node was configured by yaim, the following error was obtained

sed: can't read /opt/bdii/etc/schemas: No such file or directory

The file /opt/bdii/etc/schemas was missing. The fix is to copy the schemas.example file to schemas

# cp -i /opt/bdii/doc/schemas.example /opt/bdii/etc/schemas

First SAM test failed. lcg-lr was missing, we needed to install lcg_util.
This installed a new version of lcg_util that was on the other nodes. lcg_util
was then updated on all the nodes.

Compute Element (& site BDII)

We run the compute element service and the site BDII service on the same node.

While installing the glite-BDII packages, we obtained the following dependency errors.

Error: Missing Dependency: glite-info-provider-ldap = 1.1.0-1 is needed by package glite-BDII
Error: Missing Dependency: glue-schema = 1.3.0-3 is needed by package glite-BDII
Error: Missing Dependency: bdii = 3.9.1-5 is needed by package glite-BDII

Using yum to install the missing packages, installs these packages at a higher level and still causes the installation of glite-BDII packages to fail, as it needs these packages at the versions listed above. These packages were instead installed by hand. A GGUS ticket (Ticket-ID: 42456), which suggested that this problem is fixed in the latest release (update 34).

As with the SE install above, we had the same problem with the schemas file, missing. The above fix was repeated here.

When running yaim, we had the following errors,

grep: a: No such file or directory
grep: VO: No such file or directory
grep: or: No such file or directory
grep: a: No such file or directory
grep: VOMS: No such file or directory
grep: FQAN: No such file or directory
grep: as: No such file or directory
grep: an: No such file or directory
grep: argument: No such file or directory
qmgr: Syntax error - cannot locate attribute
set queue lhcb acl_groups += /opt/glite/yaim/bin/yaim: supply a VO or a VOMS FQAN as an argument

To fix it we edited the file /opt/glite/yaim/functions/utils/users_getvogroup and commented out

#echo "$0: supply a VO or a VOMS FQAN as an argument"

On Gstat web monitoring page, it was being reported that the SE service was missing ('SE missing in Gstat service'). To fix this problem, we edited the file /opt/bdii/etc/bdii-update.conf and add the following line for our SE.

SE ldap://grid001.jet.efda.org:2170/mds-vo-name=resource,o=grid

Mon Box

When running yaim, we had the following errors

Problem starting rgma-servicetool

Starting rgma-servicetool: [FAILED]
For more details check /var/log/glite/rgma-servicetool.log
Stopping rgma-gin: [ OK ]
Starting rgma-gin: [FAILED]

Fixed by defining a new java by adding the following to the site-info.def

HOSTNAME=`hostname`
if [ "$HOSTNAME" == "$MON_HOST" ] ; then
JAVA_LOCATION="/usr/lib/jvm/jre-1.5.0-sun"
else
JAVA_LOCATION="/usr/java/j2sdk1.4.2_12"
fi

We had the same 'schemas' file missing problem here as well.

Networking

EFDA-JET has a slightly unusually set up as we are restricted to a small number of external IP addresses. All nodes are on the same LAN with private IP addresses, whilst the service nodes also have external addresses. In the hosts files on the service nodes, all service nodes are referenced by their external addresses, whilst on the worker nodes, the service nodes are referenced by their private addresses.

This worked well for glite 3.0, but not for glite 3.1, where we saw clients on the worker nodes trying to contact the service nodes via their external addresses. It looks like glite 3.1 iservices are passing IP addresses for clients to be call back on at a later time. The complete solution was to run iptables on the worker nodes and NAT translate outgoing connections for external addresses of the service nodes to their corresponding internal addresses. This was done by adding the following to /etc/rc.local on the worker nodes.

/sbin/service iptables start
/sbin/iptables -A OUTPUT -t nat -d <CE-ext-addr> -j DNAT \
--to-destination <CE-int-addr>
/sbin/iptables -A OUTPUT -t nat -d <SE-ext-addr> -j DNAT \
--to-destination <SE-int-addr>

dCache Update

2008-12-04T17:22:00.003+01:00

We updated dCache this morning to 1.9.0. Now that sounds like a major jump but reading the release notes it is only a minor step up from the 18.0.15pX series of releases.

The upgrade itself was trivial, just installing the new dcache-server rpm and running install.sh across all the nodes.

We also took the opportunity to update the version of Postgresql on the head node from 8.3.1 to 8.3.5 using rpms from pgsqlrpms.org. I'm hoping that I will now be able to use their prebuilt slony-1 rpm to set up master slave mirroring of the databases from the dCache head node to a live mirror node.

Finally we updated the SL version of all the dCache nodes to SL4.6 from a mix of SL4.4, SL4.5 and SL4.6. We're now using the SL-Contrib xfs kernel modules on all nodes and the Araca drivers complied into the 2.6.9-78 series of kernels on all nodes with Areca raid cards rather than our own builds and have had no issues.

Fix ACls on ATLASLOCALGROUPDISK at Oxford

2008-10-15T15:45:00.002+01:00

Today I ran Graeme's script to fix the acls on the ATLASLOCALGROUPDISK space token.
Should have done this a few weeks ago but ..
There is nothing currntly stored here yet.

[root@t2se01 ~]# ./atlas-uk-local-dpm-token-fix.sh
Debug: t2se01.physics.ox.ac.uk - physics.ox.ac.uk - atlaslocalgroupdisk
Fixing permissions on /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk...
Searching /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk...

dpns-ls /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
shows nothing

dpns-getacl /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
# file: /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
# owner: root
# group: atlas/uk
user::rwx
group::rwx #effective:rwx
group:atlas/Role=lcgadmin:rwx #effective:rwx
group:atlas/Role=production:rwx #effective:rwx
group:atlas/uk:rwx #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:atlas/Role=lcgadmin:rwx
default:group:atlas/Role=production:rwx
default:group:atlas/uk:rwx
default:mask::rwx
default:other::r-x

SouthGrid update

2008-10-09T12:33:00.002+01:00

The Birmingham site has suffered some reliability problems caused by Site Networking problems.
Physics are working with central IS to resolve these issues.

Bristol has been having problems with their SE. Despite work over the weekend to fsck all the partitions by hand, the array is still causing problems.

Oxford has recently been having a strange Maui problem that means only about 60-70% of the available cores get allocated jobs. Manually 'qrun' ing the jobs causes them to run ok.
Then more recently the maui process actually started crashing. Investigations are ongoing although things seem a bit better just now.

RalPPD have installed the latest purchase of WNs into production adding another 160 job slots worth 270 kSI2k bringing us up to 1025kSI2k Total. Further disk servers have arrived but will take a month to commission.

Upgrade at RALPP

2008-08-28T12:32:00.004+01:00

We had a downtime this morning to:

Upgrade the kernel on the Grid Service and dCache pool nodes
Install the latest gLite updates on the service nodes
Upgrade dCache to the most recent patch version

Upgrading gLite and the kernel on the service nodes seems to have gone smoothly (still waiting for the SAM Admin jobs I submitted to get beyond "Waiting").

However, I had a bit more fun upgrading the kernel on the dCache Pool nodes. This is supposed to be much easier now the Areca drivers are in the stock SL4 kernel and the xfs kernel modules are in the SL Contrib yum repository so we don't have to build our own rpms like we have in the past, and indeed both these parts worked fine. But five of the nodes with 3ware cards did not appear again after I (remotely) shutdown -r now'd them. Of course these are the nodes in the Atlas Center so I had to walk across to try to find out what the problem was. The all seemed to have hung up at the end of the shutdown "Unmounting the filesystems". All came back cleanly after I hit the reset buttons.

The second problem (which had me worried for a time ) was with one of the Areca nodes. I was checking them to see if the XFS kernel modules had installed correctly and that the raid partition was mounted and on this node it wasn't, but the kernel modules had installed correctly. Looking a bit harder I found that the whole device seemed to be missing. Connecting to the RAID Card web interface I find that instead of two RAID sets (system and data) it has the two system disks in a RAIDo pair and 22 free disks (cue heart palpitations). Looking (in a rather panicked fashion) through the admin interface options I find "Rescue RAID set" and give it a go. After a reboot I connected to the web interface again and now I see both RAID sets. Phew! It's too early to start the celebrations though becuase when I log in the partition isn't mounted and when I try by hand it complains that the Logical Volume isn't there. Uh oh, cue much googling and reading of man pages.

pvscan sees the physical volume, vgscan sees the volume group and lvscan sees the logical volume but it's "NOT Available". I tried vgscan --mknodes, no that didn't work. I finally got it working with:

vgchange --available y --verbose raid

Then I could mount the partition and all the data appeared to be there.

After all that the upgrade of dCache was very simple. Just a case of adding the latest rpm to my yum repository, running yum update dcache-server then /opt/d-cache/install/install.sh. The latter complained of some depreciated config file options I'll have to look at but dCache came up
smoothly.

I'd qsig -s STOP'd all the jobs whist doing the obviously and here's an interesting ploy of the network traffic into the Worker Nodes over the last day.

As you can see once I restarted the jobs they more or less picked up without missing a beat. And yes, they are reading data at 600 MB/sec and the dCache is quite happily serving to them at that rate.

More spacetokens at Oxford

2008-08-27T10:02:00.001+01:00

Expanding the spacetokens at Oxford showed that the dpm-updatespace command has to have integer values so for 4.5T use 4500G

/opt/lcg/bin/dpm-updatespace --token_desc ATLASMCDISK --gspace 4500G --lifetime Inf

I used Graemes script to setup the ATLASGROUPDISK permissions after the reservespace command:

/opt/lcg/bin/dpm-reservespace --gspace 2T --lifetime Inf --group atlas/Role=production --token_desc ATLASGROUPDISK

Graeme's script:

[root@t2se01 ~]# more atlas-group-disk-dpm.sh
#!/bin/bash

DOMAIN=$(hostname -d)

dpns-mkdir /dpm/$DOMAIN/home/atlas/atlasgroupdisk/
dpns-chgrp atlas/Role=production /dpm/$DOMAIN/home/atlas/atlasgroupdisk/
dpns-setacl -m d:g:atlas/Role=production:7,d:m:7 /dpm/$DOMAIN/home/atlas/atlasgroupdisk/

for physgrp in exotics higgs susy beauty sm; do
dpns-entergrpmap --group atlas/phys-$physgrp/Role=production
dpns-mkdir /dpm/$DOMAIN/home/atlas/atlasgroupdisk/phys-$physgrp
dpns-chgrp atlas/phys-$physgrp/Role=production /dpm/$DOMAIN/home/atlas/atlasgroupdisk/phy
s-$physgrp
dpns-setacl -m d:g:atlas/phys-$physgrp/Role=production:7,d:m:7 /dpm/$DOMAIN/home/atlas/at
lasgroupdisk/phys-$physgrp
done

ATLASDATADISK space was increased to 15TB
dpm-updatespace --token_desc ATLASDATADISK --gspace 15T --lifetime Inf

ATLASLOCALGROUPDISK was created and setup:
/opt/lcg/bin/dpm-reservespace --gspace 1T --lifetime Inf --group atlas --token_desc ATLASLOCALGROUPDISK

dpns-mkdir /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk

dpns-chgrp atlas/uk /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
dpns-setacl -m d:g:atlas/uk:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
dpns-setacl -m g:atlas/uk:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk

Brief Bristol Update

2008-08-20T16:42:00.005+01:00

Brief Bristol update: new hardware to replace HPC CE received &
being built. New hardware for StoRM SE & a gridftp nodes received,
Dr Wakelin building them.
Our 50TB of new storage should be ready in September.

New hardware to replace MON received, being built. Will replace small
cluster WN this fall (possibly increase number) & possibly also
its CE & DPM SE.

Both clusters mostly stable, except for occasional gpfs timeouts on
HPC & recent intermittent problems with SCSI resets on DPM SE.

Delays due to Yves, Jon & Winnie very busy with other very high prio.

Setting up the Atlas Space Tokens on dCache

2008-08-18T12:16:00.005+01:00

Well the request from Atlas to have space tokens set up is quite complicated but here's my first attempt at setting them up for dCache:

The want to have different permissions on different space tokens. I think the only way to do that is to create different LinkGroups to associate with the space tokens. Here is the section from my LinkGroupAuorization.conf file for Atlas now:

LinkGroup atlas-link-group
/atlas/Role=production

LinkGroup atlas-group-link-group
/atlas/Role=production
/atlas/phys-exotics/Role=production
/atlas/phys-higgs/Role=production
/atlas/phys-susy/Role=production
/atlas/phys-beuty/Role=production
/atlas/phys-sm/Role=production

LinkGroup atlas-user-link-group
/atlas/Role=*

LinkGroup atlas-localgroup-link-group
/atlas/uk/Role=*

However it appears a Link can only be associated with one LinkGroup so we also have to create a Link for each of these. Luckily it appears that a PoolGroup can be associated with multiple links so we don't have to split up the Atlas space (phew).

So I created a bunch of Links and LinkGroups in the PoolManager like this:

psu create link atlas-localgroup-link world-net atlas
psu set link atlas-localgroup-link -readpref=20 -writepref=20 -cachepref=20 -p2ppref=-1
psu add link atlas-localgroup-link atlas-pgroup
psu add link atlas-localgroup-link atlas
psu create linkGroup atlas-localgroup-link-group
psu set linkGroup custodialAllowed atlas-localgroup-link-group false
psu set linkGroup replicaAllowed atlas-localgroup-link-group true
psu set linkGroup nearlineAllowed atlas-localgroup-link-group false
psu set linkGroup outputAllowed atlas-localgroup-link-group false
psu set linkGroup onlineAllowed atlas-localgroup-link-group true
psu addto linkGroup atlas-localgroup-link-group atlas-localgroup-link

Obviously repeated for each of the other extra LinkGroups

Then it's just a case of creating the space tokens in the SrmSpaceManager:

reserve -vog=/atlas -vor=NULL -acclat=ONLINE -retpol=REPLICA -desc=ATLASUSERDISK -lg=atlas-user-link-group 2500000000000 "-1"
reserve -vog=/atlas/uk -vor=NULL -acclat=ONLINE -retpol=REPLICA -desc=ATLASLOCALGROUPDISK -lg=atlas-localgroup-link-group 9000000000000 "-1"
reserve -vog=/atlas -vor=production -acclat=ONLINE -retpol=REPLICA -desc=ATLASGROUPDISK -lg=atlas-group-link-group 3000000000000 "-1"

I'm not sure the last one will work as expected I don't know how the -vog=/atlas will map with the multiple VOMS groups in the LinkGroupAuthorization.conf file. But I've no idea how to specify multiple VOMS groups there.

OK, that should get us the Space tokens, but Atlas are also requesting specific permissions on directories and that's completely orthogonal to the space tokens. All I've got to play with there are the normal UNIX users and groups.

So I start off by creating 6 extra groups and amking it the primary group for a single pool account (which is also in the main atlas group) , I also add the atlasprd account to the physics group groups since they want that to have write access to the group areas. Here's the relevant bit from /etc/groups, you can work out the changes to /etc/passwd yourselves.

atlas:x:24259:atlas002,atlas003,atlas004,atlas005,atlas006,atlas007
atl-exo:x:24358:atlasprd,atlas002
atl-higg:x:24359:atlasprd,atlas003
atl-susy:x:24360:atlasprd,atlas004
atl-b:x:24361:atlasprd,atlas005
atl-sm:x:24362:atlasprd,atlas006
atl-uk:x:24365:atlas007

Now I've got the users and groups set up I can create the directories:

mkdir /pnfs/pp.rl.ac.uk/data/atlas/atlaslocalgroupdisk
chown atlas007:atl-uk /pnfs/pp.rl.ac.uk/data/atlas/atlaslocalgroupdisk
chmod 755 /pnfs/pp.rl.ac.uk/data/atlas/atlaslocalgroupdisk

[root@heplnx204 etc]# ls -l /pnfs/pp.rl.ac.uk/data/atlas/atlasgroupdisk/
total 3
drwxrwxr-x  1 atlas005 atl-b    512 Aug 18 13:21 phys-beauty
drwxrwxr-x  1 atlas002 atl-exo  512 Aug 18 13:21 phys-exotics
drwxrwxr-x  1 atlas003 atl-higg 512 Aug 18 13:21 phys-higgs
drwxrwxr-x  1 atlas006 atl-sm   512 Aug 18 13:21 phys-sm
drwxrwxr-x  1 atlas004 atl-susy 512 Aug 18 13:21 phys-susy

But no I have to make sure dCache maps the right voms credentials to the correct account:
First of in /etc/grid-security/storage-authzdb

authorize atlas001 read-write 37101 24259 / / /
authorize atlas002 read-write 37102 24358 / / /
authorize atlas003 read-write 37103 24359 / / /
authorize atlas004 read-write 37104 24360 / / /
authorize atlas005 read-write 37105 24361 / / /
authorize atlas006 read-write 37106 24362 / / /
authorize atlas007 read-write 37107 24365 / / /
authorize atlasprd read-write 51000 24259 / / /

and in /etc/grid-security/grid-vorolemap

# Added role /alice/Role=production
"*" "/alice/Role=production" aliceprd

# Added role /atlas
"*" "/atlas" atlas001
"*" "/atlas/phys-exotics" atlas002
"*" "/atlas/phys-higgs" atlas003
"*" "/atlas/phys-susy" atlas004
"*" "/atlas/phys-beauty" atlas005
"*" "/atlas/phys-sm" atlas006
"*" "/atlas/uk" atlas007

# Added role /atlas/Role=lcgadmin
"*" "/atlas/Role=lcgadmin" atlas001

This has not been fully tested yet, in particular it's not clear the the ATLASGROUPDISK space token will handle the way I expect.

Oh, and doing this has once again made me realise that I don't really understand what Units and Links are and do in dCache, so I'm offering a beer to anyone who can explain this to me.

Update on 28/08/08

It looks like this doesn't work fully, dCache doesn't support secondary groups so the atlasprd user who is in group atlas cannot write to the /pnfs/pp.rl.ac.uk/data/atlas/atlasgroupdisk/* areas even though it has secondary group membership of the groups which do have write access. I'm now waiting for feedback from atlas to know how they want the permissions configured in view of this.

Adding multiple clusters to get different memory limit queues

2008-07-25T09:59:00.000+01:00

I've been thinking about doing this for ages. The aim is to have different queues with different memory limits to better direct jobs with higher memory requirements to nodes with more memory.

The current method of doing this is to set up up a separate queue for each level with the default memory requirement set. Then in the information system publish separate clusters and subclusters for each of the queues.

So I first created grid500, grid1000 and grid2000 queues with no memory limits, configured them normal using yaim so they would accept jobs from all my supported VOs and checked that job submission to them worked as expected.

I then edited the static-file-Cluster.ldif file on the CE to add extra clusters and subclusters for each of the queues and set the memory for each of the clusters to the memory for the queue. So for example for the grid500 queue I created a 500.pp.rl.ac.uk cluster like so:


dn: GlueClusterUniqueID=500.pp.rl.ac.uk,mds-vo-name=resource,o=grid
objectClass: GlueClusterTop
objectClass: GlueCluster
objectClass: GlueInformationService
objectClass: GlueKey
objectClass: GlueSchemaVersion
GlueClusterName: 500.pp.rl.ac.uk
GlueClusterService: heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid500
GlueClusterUniqueID: 500.pp.rl.ac.uk
GlueForeignKey: GlueSiteUniqueID=UKI-SOUTHGRID-RALPP
GlueForeignKey: GlueCEUniqueID=heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid500
GlueInformationServiceURL: ldap://heplnx206.pp.rl.ac.uk:2170/mds-vo-name=resource,o=grid
GlueSchemaVersionMajor: 1
GlueSchemaVersionMinor: 3

dn: GlueSubClusterUniqueID=500.pp.rl.ac.uk, GlueClusterUniqueID=500.pp.rl.ac.uk,mds-vo-name=resource,o=grid
objectClass: GlueClusterTop
objectClass: GlueSubCluster
objectClass: GlueHostApplicationSoftware
objectClass: GlueHostArchitecture
objectClass: GlueHostBenchmark
objectClass: GlueHostMainMemory
objectClass: GlueHostNetworkAdapter
objectClass: GlueHostOperatingSystem
objectClass: GlueHostProcessor
objectClass: GlueInformationService
objectClass: GlueKey
objectClass: GlueSchemaVersion
GlueChunkKey: GlueClusterUniqueID=heplnx206.pp.rl.ac.uk
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_1_0
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_1_1
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_2_0
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_3_0
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_3_1
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_4_0
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_5_0
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_6_0
GlueHostApplicationSoftwareRunTimeEnvironment: LCG-2_7_0
GlueHostApplicationSoftwareRunTimeEnvironment: GLITE-3_0_0
GlueHostApplicationSoftwareRunTimeEnvironment: RALPP
GlueHostApplicationSoftwareRunTimeEnvironment: SOUTHHGRID
GlueHostApplicationSoftwareRunTimeEnvironment: GRIDPP
GlueHostApplicationSoftwareRunTimeEnvironment: R-GMA
GlueHostArchitectureSMPSize: 2
GlueHostArchitecturePlatformType: i586
GlueHostBenchmarkSF00: 0
GlueHostBenchmarkSI00: 1000
GlueHostMainMemoryRAMSize: 500
GlueHostMainMemoryVirtualSize: 1000
GlueHostNetworkAdapterInboundIP: FALSE
GlueHostNetworkAdapterOutboundIP: TRUE
GlueHostOperatingSystemName: ScientificSL
GlueHostOperatingSystemRelease: 4.4
GlueHostOperatingSystemVersion: Beryllium
GlueHostProcessorClockSpeed: 2800
GlueHostProcessorModel: PIV
GlueHostProcessorVendor: intel
lueSubClusterName: 500.pp.rl.ac.uk
GlueSubClusterUniqueID: 500.pp.rl.ac.uk
GlueSubClusterPhysicalCPUs: 0
GlueSubClusterLogicalCPUs: 0
GlueSubClusterTmpDir: /tmp
GlueSubClusterWNTmpDir: /tmp
GlueInformationServiceURL: ldap://heplnx206.pp.rl.ac.uk:2170/mds-vo-name=resource,o=grid
GlueSchemaVersionMajor: 1
GlueSchemaVersionMinor: 3

Note that there has to be a blank line in the file after the end of the subcluster definition or else the gip script that adds the VO tags doesn't.

I also had to edit the CE/Queue entries in static-file-CE.ldif to change these two entries for each of the grid500, grid1000 and grid2000 queues


GlueCEHostingCluster: 500.pp.rl.ac.uk
GlueForeignKey: GlueClusterUniqueID=500.pp.rl.ac.uk

These clusters seem to have appeared correctly on the gStat pages.

So now when I edg-job-list-match a jdl with the following requirements I get:

Requirements = ( RegExp("heplnx206\.pp\.rl\.ac\.uk.*", other.GlueCEUniqueID) && other.GlueHostMainMemoryRAMSize >= 500);


***************************************************************************
COMPUTING ELEMENT IDs LIST
The following CE(s) matching your job requirements have been found:

*CEId*
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-dteam
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-short
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid1000
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid2000
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid500
***************************************************************************

Requirements = ( RegExp("heplnx206\.pp\.rl\.ac\.uk.*", other.GlueCEUniqueID) && other.GlueHostMainMemoryRAMSize >= 1000);


***************************************************************************
COMPUTING ELEMENT IDs LIST
The following CE(s) matching your job requirements have been found:

*CEId*
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-dteam
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-short
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid1000
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid2000
***************************************************************************

Requirements = ( RegExp("heplnx206\.pp\.rl\.ac\.uk.*", other.GlueCEUniqueID) && other.GlueHostMainMemoryRAMSize >= 1500);


***************************************************************************
COMPUTING ELEMENT IDs LIST
The following CE(s) matching your job requirements have been found:

*CEId*
heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-grid2000
***************************************************************************

Requirements = ( RegExp("heplnx206\.pp\.rl\.ac\.uk.*", other.GlueCEUniqueID) && other.GlueHostMainMemoryRAMSize >= 2001);


===================== edg-job-list-match failure ======================
No Computing Element matching your job requirements has been found!
======================================================================

Which looks very much like what I want to do.

Then to let Torque/Maui know about the memory requirements for each of these new queues I set a default memory requirement for each with something like:

qmgr -c "set queue grid1000 resources_default.mem = 1000mb"

(I think this is a non-enforcing requirement so jobs will not be killed for going over it. To do that I think you need to set "resources_max.mem".)

Now I just need to do the same configuration on heplnx207, phase out the "per VO" queues and persuade users to put the memory requirements of their jobs in their JDL files

Oxford adds the ATLAS proddisk space token

2008-07-11T14:22:00.002+01:00

Using the same procedure as before I added the new SPACE TOKEN for ATLAS:

export DPNS_HOST=t2se01.physics.ox.ac.uk
/opt/lcg/bin/dpm-reservespace --gspace 2T --lifetime Inf --group atlas/Role=production --token_desc ATLASPRODDISK
/opt/lcg/bin/dpns-mkdir /dpm/physics.ox.ac.uk/home/atlas/atlasproddisk
/opt/lcg/bin/dpns-chgrp atlas/Role=production /dpm/physics.ox.ac.uk/home/atlas/atlasproddisk
dpns-chmod 775 /dpm/physics.ox.ac.uk/home/atlas/atlasproddisk
dpns-setacl -m d:g:atlas/Role=production:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlasproddisk
dpns-setacl -m g:atlas/Role=production:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlasproddisk

SouthGrid Update

2008-07-02T11:37:00.003+01:00

Just to update that the rebuild of the Bristol
DPM SE to SL4 on 19th June by Yves & Winnie was a smooth success.

Oxford had a strange error on its site BDII (still running SL3) after a re run of yaim it stopped advertising the site web address and several other entries so gstat had a warning. The following were missing.:
dn: GlueSiteUniqueID=UKI-SOUTHGRID-OX-HEP,mds-vo-name=UKI-SOUTHGRID-OX-HEP,o=g
objectClass: GlueSite
GlueSiteUniqueID: UKI-SOUTHGRID-OX-HEP
GlueSiteName: UKI-SOUTHGRID-OX-HEP
GlueSiteDescription: LCG Site
GlueSiteUserSupportContact: mailto:lcg_manager@physics.ox.ac.uk
GlueSiteSysAdminContact: mailto:lcg_manager@physics.ox.ac.uk
GlueSiteSecurityContact: mailto:lcg_manager@physics.ox.ac.uk
GlueSiteLocation: Oxford, UK
GlueSiteLatitude: 51.7595
GlueSiteLongitude: -1.2595
GlueSiteWeb: http://www.physics.ox.ac.uk
GlueSiteSponsor: none
GlueSiteOtherInfo: TIER-2
GlueSiteOtherInfo: rl.ac.uk

It transpired that the file /opt/bdii/etc/bdii-update.conf
had the entry GIP pointing to

> GIP file:///opt/glite/libexec/glite-info-wrapper

which did not exist, changing it to

< GIP file:///opt/lcg/libexec/lcg-info-wrapper

and restarting bdii has fixed the missing web address etc problem.

We were about to setup an SL4 based site bdii anyway so Ewan did so and curiously got the same errors, even though the glite 3.1 files do all now exist in the /opt/glite directories.
Investigations continue but for now we are sticking with the working glite 3.0 based site bdii.

PS A thread on LCG-ROLLOUT mentions the same errors:
http://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind0806&L=lcg-rollout&T=0&F=&S=&X=3AEC8F06E68018C3DE&Y=p.gronbech1%40physics.ox.ac.uk&P=4737

SouthGrid update

2008-06-17T13:45:00.002+01:00

Yves will be helping Bristol upgrade the se to SL4 on Thursday.
They had problems with transtec raid array, specifically the battery backed up cache. New parts have now arrived.

The MESC cluster at Birmingham is working well now, the four LHC VOs are supported, but it is not full yet. (Both Atlas and LHCb are aware of it)
We may have to advertise it to the VOs.

Yves hopes to be able to start configuring Blue Bear (The new HPC cluster) after the grand opening next week. There will also be an meeting between NGS people and Birmingham to see how they can work better together, probably by NGS enabling Blue Bear.

64 bit tarball was used at Bristol and (32 bit on MESC). Some extra i386 rpms are required.

JET have had problems, failing SAM tests but OK for real jobs. Will reinstall the ce.

Stop Press: Now working since the reinstall.

Oxford have not published accounting since the introduction of two SL4 based ce's. We will be working on fixing this so we have the data for the Quaterly report.

Second phase of Oxford's move

2008-05-21T11:27:00.002+01:00

Over the last couple of days Ewan and I have moved five 9TB disk servers, five twin worker nodes, a couple of head nodes and two UPS's. We had help moving the rack, and then we were able to reassemble it all. The site was back up and passing SAM tests in time for us to come out of scheduled maintenance at 1700.
The current setup means we have three ce's, the original SL3 based ce is driving most of the newer SL4 based WNs. The two new SL4 based ce's send jobs to a new torque server, and onto two subclusters, one for the 32 bit hardware (Dell 2.8GHz xeons), and the other the Intel Clovertown quads. We will migrate the workers off the old ce onto the new ones over the next few days, before decommissioning the original SL3 based ce.

The migration of data from our old se head node, is complete. There were three files that were listed in the database but did not exist on the physical storage. I used the scripts from :
https://twiki.cern.ch/twiki//bin/view/LCG/CheckDpmConsistency
which matched the three files names that were left in my dpm-drain logs.

Two of the files could be removed with rfrm but one refused to appear in the normal dpns-ls listing so was not removed.
We decided to ignore this one and remove the files system from the pool with the command.
dpm-rmfs --server t2se01.physics.ox.ac.uk --fs /storage

The new DPM head node will be setup and then the mysql database dumped and restored on to it shortly.

Meanwhile we are awaiting the backplanes in our storage servers to be swapped out to avoid the burnout issue we have suffered on one of them.

SouthGrid technical meeting will be held tomorrow at Birmingham.

Oxford DPM progressing Slowly

2008-04-28T14:47:00.003+01:00

Removing zero length files from the DPM storage pool with the rfrm command has helped the dpm-drain command to start progressing again.
The command still fails after 10-20 files and the transfer speed in very low, but at least we are making progress.

SouthGrid Update

2008-04-14T16:33:00.003+01:00

Having installed half of the Oxford cluster at Begbroke last Tuesday. The Air conditioning failed during the night, a valve on the Chillers failed cutting off the water supply to the Chillers which in turn switched themselves off. The room rapidly heated up to >40 degrees. After investigation and repairs the AC went back on and all has been well so far. More automated warning systems are required.

Cambridge have set up space tokens for both ATLASDATADISK and ATLASMCDISK. They have also started upgrading to SL4 (64bit) worker nodes.

Britol completed upgrading the Worker nodes to SL4 on Monday. They had some problems caused by se linux, preventing normal loging but all now appears well.

Oxford Update

2008-04-08T11:11:00.004+01:00

Last week Ewan and I started the DIY move to Begbroke. We moved 40 1 u servers over two mornings. One of the (now empty) Dell racks was moved on Wednesday afternoon.
The worker nodes were reinstalled in that rack on Thursday, we had one psu failure out of 27 nodes. These nodes will be installed with sl4 shortly.

This week we have emptied one of the Viglen racks and moved the servers yesterday.
We hope to move the rack this afternoon and get the worker nodes back on line asap as we are at half capacity currently.

On Firday 28th March we had one of our new 9TB file servers burn out its backplane. This is very similar to the problems RAL have been seeing. The backplane was swapped out and the server is back on line now.
We are in talks with the supplier.

Oxford Update

2008-04-08T11:11:00.001+01:00

Oxford adds another space token

2008-03-31T16:36:00.004+01:00

With some advice from Graeme I have added another space token at Oxford
Commands recorded for posterity.


dpm-reservespace --gspace 3T --lifetime Inf --group atlas/Role=production --token_desc ATLASMCDISK
dpns-mkdir /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-chgrp atlas/Role=production /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-chmod 775 /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-setacl -m d:g:atlas/Role=production:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk
dpns-setacl -m g:atlas/Role=production:7,m:7 /dpm/physics.ox.ac.uk/home/atlas/atlasmcdisk

We started our DIY move today, two trips in the lab van with 8 1U servers each time.
Tomorrow we plan to move more old WN's and then an empty rack later in the week.

SouthGrid Update

2008-03-28T14:40:00.003+01:00

Bristol:
The first stage of the HPC cluster is running LCG jobs, and is being correctly accounted for.

The HPC WNs have AMD 2218 cores, 2.6GHz; these are said to be
1.745 KSi2K each core.
Currently gridpp can run max 32 jobs on this small stage 1 HPC cluster;

Cambridge:
Santanu is continuing to work with LHCb to solve all the problems running their code at Cambridge.
The WNs will be upgraded to SL4 within the next few weeks.

Birmingham:
When over 100 of the 120 Babar cluster died after a power shutdown at the end of January, it was deemed not worth restoring the cluster. Two twin 1 u servers have been bought to replace this which will provide 32 cores and 78.4KSI2K.
The old escience cluster is being setup as an SL4 ce and WN farm in as a template for the way they will drive the new University 'Blue Bear' HPC cluster. This cluster is made up of 31 dual 3GHz xeons.
The main grid cluster (aka the atlas cluster) has been expanded to 60 cores.

The PPS is not being maintained at the moment.

RALPPD:
Chirs has got space tokens working at RALPP ( updated to dCach1.8-12p6 - from p4 - and also rebooted everything after putting in the srmSpaceManager enabled config files).
The new hardware has been installed:
8 boxes, 16 nodes, 32 CPUs so 128 cores.

CPUs are "E5410 @ 2.33GHz" not sure of the kSI2k rating yet.

JET:
Running stable. WNs were updated to SL4 earlier this year.

Oxford:
Quotes to move the kit to Begbroke seem too high so we are going to adopt a DIY approach.
Draining t2se01 is taking for ever. The dpm-drain command terminates sometimes after only 20mins (~6GB data transfered). We did however have a good run on the night of the 26th which lasted over 10 hours.
Oddly the error log files are often the same size although not totally consistent.


-rw-r--r--    1 root     root        22519 Feb 22 14:32 dpm-drain-errorlog-se01-1
-rw-r--r--    1 root     root        22519 Feb 22 15:56 dpm-drain-errorlog-se01-2
-rw-r--r--    1 root     root        22564 Feb 22 17:05 dpm-drain-errorlog-se01-3
-rw-r--r--    1 root     root        22519 Feb 22 18:20 dpm-drain-errorlog-se01-4
-rw-r--r--    1 root     root        22519 Feb 25 12:35 dpm-drain-errorlog-se01-5
-rw-r--r--    1 root     root        22519 Feb 25 13:04 dpm-drain-errorlog-se01-6
-rw-r--r--    1 root     root        22519 Feb 25 16:53 dpm-drain-errorlog-se01-7
-rw-r--r--    1 root     root        22519 Mar 26 13:51 dpm-drain-errorlog-se01-8
-rw-r--r--    1 root     root        22519 Mar 26 14:17 dpm-drain-errorlog-se01-9
-rw-r--r--    1 root     root      1193287 Mar 27 00:56 dpm-drain-errorlog-se01-10
-rw-r--r--    1 root     root       567836 Mar 27 13:59 dpm-drain-errorlog-se01-11
-rw-r--r--    1 root     root        25241 Mar 27 14:57 dpm-drain-errorlog-se01-12
-rw-r--r--    1 root     root        22598 Mar 27 15:37 dpm-drain-errorlog-se01-13
-rw-r--r--    1 root     root        22598 Mar 27 16:22 dpm-drain-errorlog-se01-14
-rw-r--r--    1 root     root        22598 Mar 27 17:11 dpm-drain-errorlog-se01-15
-rw-r--r--    1 root     root        22598 Mar 27 21:55 dpm-drain-errorlog-se01-16
-rw-r--r--    1 root     root        22598 Mar 27 23:15 dpm-drain-errorlog-se01-17

Oxford Update

2008-03-25T12:02:00.002+01:00

The Original 74 cpu SL3 Dell workernodes have been taken down in preparation for reinstallation as SL4 worker nodes.
We will maintain a separate ce to drive these but intend to separate out the torque server.
The new ce will be an SL4 ce.
The ce and torque server are likely to be virtual machines running under VMware.

The move of the 4 racks up to Begbroke is still uncertain.
We are awaiting quotes from companies to move the equipment for us. The DIY price of just paying for a truck and driver has also been looked into but issues of insurance and warantee's may prevent use of this option.

The steps up to the new computer room have been made smaller to allow installation of a scissor lift, to lift racks to the false floor height. The date for the lift to be installed is still unclear.

The grand opening on the 15th April is all too close.

SouthGrid Technical Meeting at JET

2008-02-05T11:57:00.000+01:00

The SouthGrid Technical Board met at JET.
All sites are moving towards SL4. The recent updates will be applied shortly.
The SouthGrid vo has been setup and a central LFC is being provided at RAL Tier 1 for it.
The outstanding tickets were looked at and all found to be solved. A problem was found, tickets still open in footprints were closed in GGUS, so the link may not be working properly.
Birmingham is setting up some ex escience nodes to be the interface to the new HPC cluster. This will bring back some of the spec int power lost due to hardware failures after recent electrical work.

Oxford Update

2008-01-22T13:11:00.000+01:00

Plans to move the Oxford gridpp cluster up to Begbroke are being formulated.
The first part of the plan is to ensure that only these nodes are using the subnet in question. We did some tidying up over the last week or so, before having the subnet rerouted to both Physics and Begbroke. This change was made this morning at 8:50, and mostly went smoothly.
Our ui needs to be moved back on to the physics subnet to allow NFS mounting of home directories to work.
A new rack, PDU and network switch has been ordered to allow us to move a few test nodes up to Begbroke in advance of the main move.
We aim to complete the move late Jan/ early Feb.

The disk on our installation server which holds ganglia data and central syslog data failed today. We will restore from backups.
t2wn05 has a failed hard disk which may have been acting as a black hole over the weekend.

Working with ZEUS and LHCb VO's to improve usage of our cluster uncovered some configuration problems.

Not all the nodes had the latest DESY VOMS server certs applied (stopped zeus working)
sgm ROLES were not mapped correctly for LHCB.

Finally the APEL problems seem to be behind us.

Configuration seemed to have changed at the last running of yaim before Christmas which stopped any records getting published
Installing the latest Development Apel rpms fixed the problem of not seeing the newer spec value for our new ce.

Scheduled Power outage at Birmingham causes problems

2008-01-04T15:47:00.000+01:00

The scheduled power outage at Birmingham on Saturday 8th December caused 19 Babar SL4 systems to fail. 4 bad disks appeared on the SL3 cluster. The age of this equipment is a cause for concern.

There has been some concern expressed at small sites such at Bristol that the number of Atlas tests submitted by Steve Lloyds tests can over whelm their sites.

Oxford Gridpp Site becomes an NGS Affiliate

2007-12-11T10:48:00.000+01:00

Not to be out done by Scotgrid, I should also point out that Oxford became an NGS affiliate at the same meeting (Dec 6th). See https://www.ngs.ac.uk/guide/affiliates/oxford-gridpp/

Oxford have added support for vo.southgrid.ac.uk, gridpp and supernemo.vo.eu-egee.org

Birmingham HV Network Upgrade

2007-12-07T17:52:00.000+01:00

High Voltage Network Upgrade, over this weekend, means several systems will be off over the weekend.
It is hoped to keep the core service nodes up and running, but the number of worker nodes will be limited.

ALICE VO Box was not accessable to the users for a day, no problems were found by Yves.
Now reported as OK.

SouthGrid Update

2007-12-07T10:49:00.000+01:00

Bristol:
Had some problems with LHCb users
EDFA-JET:
Upgraded WN's to SL4
Birmingham:
Disk failed on the se raid 5 disk array.
Oxford:
Upgraded the SL3 cluster to update 37. Some problems with the se, the DPM pool nodes had not had the latest lcg-vomscerts rpm applied. Secondly the site-info.def file on some of the nodes had an old entry for the ops vo which meant the gridmap file was not being created correctly.
This was changed to include:


VO_OPS_VOMS_SERVERS="'vomss://lcg-voms.cern.ch:8443/voms/ops?/ops/'
                     'vomss://voms.cern.ch:8443/voms/ops?/ops/'"
VO_OPS_VOMSES="'ops lcg-voms.cern.ch 15009 /DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch ops'
               'ops voms.cern.ch     15009 /DC=ch/DC=cern/OU=computers/CN=voms.cern.ch     ops'"

The addition of voms.cern.ch being the important bit, (lcg-voms.cern.ch) was the old entry.

RALPPD:
The BDII failed on Monday 3rd. A reboot fixed this.

So now that Oxford is uptodate we can go ahead and add support for some new VOs,
SouthGrid, gridpp and supernemo.

Random rm failures at Oxford

2007-12-05T23:41:00.001+01:00

Random SAM test failures for rm, and later complaints from ATLAS were traced to one of the DPM pool nodes not having had the latest VOMS certs applied.

dCache Tuning

2007-10-22T12:26:00.000+01:00

I've been having a few issues since the start of the CMS CSA07 data challenge with SAM test failures with what seem to be mostly timeouts against my dCache Storage Element so I've been looking at improving my setup.

One suggestion was to set up separate queues in dCache for local access (dcap, gsidcap and xrootd) and remote access (GridFTP).

In general this is supposed to help when local farm jobs are reading slowly from lots of files and blocking the queues preventing the short GridFTP jobs from starting. Which is not the current case on my Storage Element, but it might also help by limiting the number of concurrent GridFTP transfers, which are very resource hungry without limiting the local access which is not.

It was a very easy change to do requiring only changed to the /opt/d-cache/config/dCacheSetup file, not the indevidual batch files (on all the servers of course, though). I uncommented and set the following variables:

poolIoQueue=dcapQ,gftpQ
gsidcapIoQueue=dcapQ
dcapIoQueue=dcapQ
gsiftpIoQueue=gftpQ
remoteGsiftpIoQueue=gftpQ

The first variable sets up the two queues (the first queue is also the default on if no queue is specified).

Then the rest of the settings specify which queue the different doors use.

Unfortunately, the queue lengths are set per pool in the pool setup file so I had to edit a file for each pool on all the disk servers to change:

mover set max active NNNN

to:

mover set max active -queue=dcapQ 1000
mover set max active -queue=gftpQ 3

After the changes to the config files I then had to restart all the services to pick up the new config. I also took the opportunity to enable readonly xrootd access to the SE but adding:

XROOTD=yes

to /opt/d-cache/etc/node_config on all the nodes

and setting:

xrootdIsReadOnly=true

in the dCacheSetup file.

After the restart the new queues showed up in the queue info pages and the xrootd doors on all the nodes showed up on the Cell Services page.

I was also able to read files out from the xrootd door using standard babat tools (and was correctly blocked from writing data).

SL4 Worker Node Migration at RALPP

2007-10-10T12:41:00.000+01:00

Since I've now finished the migration my worker nodes to SL4 I thought I should describe the method used.

The basic decision was to try to keep running an SL3 service in parallel with the initial test SL4 service and then gradually migrate nodes to the new service once it was production quality. I already had split my Torque/Maui services off onto a separate node and wanted to keep that setup with the SL4 service but did not want to (a) duplicate the torque server or (b) create another 24 queues for all the VOs. To get round this I decided to:

Install a new "SL4" CE pointing to the production PBS node, this needed a different site info.def file with it named as the CE_HOST and the GlueOperatingSystem settings set for SL4 obviously
Create node properties on the SL3 and SL4 nodes to let the batch system route jobs based on OS
Hack the lcgpbs jobmanagers on the two CEs to apply requirements on the node properties as it submits the job

Running multiple CEs all pointing to the same torque server is fairly simple to do, there is a "BATCH_SERVER" setting in YAIM (3.1 and later, TORQUE_SERVER before that) that you just point at your the torque/maui server and that configures the CE to submit it jobs via that machine. Then there are a couple of other things you have to take care of:

The gridmapdir has to be shared between all the CEs. Otherwise there is a possibility that either the same DN will be mapped to multiple pool accounts or worse that different DNs will be mapped to the same pool account by the different CEs.
The worker nodes need to have the ssh host keys for all the CEs to be able to get the job data back but YAIM will only set one up. The fix is to edit the NODES line in "/opt/edg/etc/edg-pbs-knownhosts.conf" to add all the CEs and your torque server
If the CEs are submitting the same worker nodes you might also want to mount the VO tag are across all the CEs so that VOs don't have to publish the same tags to all the CEs

Node properties are very easy either just edit the torque nodes file to add them or use "qmgr -c "set node $node properties += SL4". I also added "test" and "prod" properties to all the nodes but more of that below.

Finally I needed to change the job manager to require the properties to direct jobs going to the different CEs to different classes of workers based on the above properties. The lcgpbs jobmanager already writes a node requirement in into the job script it submits to torque and so it is easy to rewrite this to add node properties as well. If you look in "/opt/globus/setup/globus/lcgpbs.in" you'll see three places where it writes "#PBS -l nodes=" to set the requirement on the number of CPUs and you need to add :SL4 (or :SL3) to the end of the write.

After doing that, installing some SL4 worker nodes was very simple, about the only necessary change to the site-info.def file was to change the "GLOBUS_TCP_PORT_RANGE" to be space separated rather than comma separated.

With the above hacks in place I was able to leave my old CE happily submitting jobs to the SL3 nodes while I was testing the SL4 worker nodes then gradually move the worker nodes over to Sl4. Before Moving the final worker nodes over I modified the batch system information provider to report the queues as "Draining" whatever their real status. Once all the worker nodes were migrated to SL4 I could just remove the changes to the lcgpbs jobmanager changes and both CEs became equivalent.

Oxford's Tier 2 Upgrade is joining the grid.

2007-09-24T10:55:00.000+01:00

The 22 new worker nodes are starting to come on line now.
They are running SL4 32bit mode for now. They will provide an additional 431 K Spec Int 2000.

A second ce t2ce03.physics.ox.ac.uk has been setup to serve the SL4 WN's. We had some trouble with the BDII being on the original ce so have split that function off onto a new node (Well actually a VM ).

The upgrade also includes 4 heads nodes with dual PSUs, and mirrored systems disks, which can be used for service functions or as worker nodes. All the head nodes and disk servers are protected by UPS.

The 11 storage servers (9TB usable each) will be brought on line over the next week.

The two new (Viglen supplied) racks are on the right hand side, with the older Dell kit on the left.

Oxford Local Computer Room Goes Live

2007-09-14T17:26:00.000+01:00

The local computer room was completed last Friday. All power is ready under each of the 21 rack positions. Each rack position has 4 CAT6 cables connected to the networking rack which can be seen. Other things completed were; ceiling lights, painting, smoke detection system, and door fitting.

On Monday 10th two existing compute racks were installed and two empty racks for the cluster upgrade arrived. A rack full of worker nodes for the existing grid cluster can be seen and is up and running.

Today the servers arrived from Viglen and installation has commenced.

Oxford Computer Room Update

2007-08-21T13:44:00.001+01:00

Progress on wiring for Power and Networking is scheduled to be completed this week.

The Oxford Grid Cluster upgrade has been ordered, and should be delivered in early September, to be installed here.

SL4 progress

2007-07-18T10:44:00.001+01:00

Cambridge has converted its DPM server to 64bit SL4. Plan to start migrating WNs next week.

Birmingham
Very easy to deploy, 32bit SL4, yum and yaim used. Using existing second ce to direct jobs to the WNs. Have passed tests from OPS. Babar farm switched off due to Air Conditioning problems.

RALPPD
dcache servers are running SL4
30 WN cpus now running 32bit SL4 there is a new ce to direct jobs to these. This will be advertised from next Monday (23rd July)

SouthGrid shared calendar setup in google to help coordinate holidays and meetings.

Oxford site ce swamped by Biomed jobs

2007-07-17T12:49:00.000+01:00

At the end of last week the Oxford ce was swamped by hundreds of biomed jobs, the que was disabled, and the ce rebooted, but manual killing and tidying up was required before the ce stabilised.

Oxford DWB Computer room update

2007-07-17T12:40:00.001+01:00

The floor is complete

External power boards ready, and live.

Walls painted, smoke detections systems installed (red pipes) and the ceiling is being installed this week.

Under floor electrical wiring and network cabling should start tomorrow.

Oxford local Computer room update

2007-06-22T15:45:00.000+01:00

Work is progressing on the new local room, which is just as well as there are delays on the Begbroke room, which will not be ready till late summer/early autumn.
The floor has been sealed with Vinyl.

Electrical switching has been connected up.

And the false floor is being installed.

Southgrid Update

2007-06-22T15:41:00.001+01:00

Bristol.
Plans under way to make use of the new HPC cluster. Meetings started to work out a strategy and solve technical problems.

Cambridge
DPM upgrade was a nightmare, with help from Grieg and Yves, Santanu has now got the se up upgraded to DPM 1.6.4

Birmingham
Problems publishing APEL data are under investigation

Oxford
Support for ngs.ac.uk enabled, tests by Steven Young, from NGS at Oxford are starting. Pete attended the NGS User forum and training event held in the OERC building in Oxford.

Rapid progress on Oxford's local computer room

2007-06-12T09:35:00.001+01:00

This was the space allocated on level 1 just after the old offices had been cleared out on April 11th.

Since then the walls have been dry lined, the AC units and pipe work are in place.

Heavy electrical work is ongoing and the floor is being prepared.

Also the forth wall has been built.

We are hopeful that the room will be complete by the end of July.
The floor will be sealed this week prior to the false floor being installed. Electrical cabling will then commence.

Nagios Monitoring

2007-05-25T15:35:00.000+01:00

Nagios is being setup at Oxford. So far all nodes are tested using ssh to check that they are up and running.
NRPE is being installed to allow check on disk space to be carried out.
Further instructions can be found in the talk by Chris Brew at HEPSYSMAN
http://hepwww.rl.ac.uk/sysman/may2007/agenda.html
or at the System management wiki
http://www.sysadmin.hep.ac.uk/wiki/Nagios

SouthGrid Dashboard

2007-05-25T15:32:00.000+01:00

SouthGrid dashboard setup a la ScotGrid and North Grid.
See http://www.gridpp.ac.uk/wiki/Southgrid-Dashboard

Multiple failures at Oxford explained

2007-04-30T15:06:00.000+01:00

Oxford ran out of disk space on its DPM SE. This caused the rm SAM test to fail. This was due to ATLAS taking up all the available disk space on our SE. We managed to clear some space from dteam and this allowed us to start passing the tests again. The bigger problem remains , that as there is currently no quota mechanism in DPM, we can not prevent this happening again. We only have two (1.6TB) pools and both are assigned to all VO's. It is not possible to allocate a pool exclusively to ops, or to keep ATLAS on their own without completely removing all data and re designing the pools. This is a non starter.
When more disk space is added consideration will be given to allocate dedicated pools for some VOs.

Oxford then started failing other tests, this was caused by multiple worker nodes having either full /home or / partitions. This highlights the necessity of monitoring disk usage with Nagios.

Oxford tries out MonAMI

2007-03-27T13:07:00.001+01:00

During the gridpp collaboration meeting I was persuaded to give MonAMI a go.
Installing the rpm from the sourceforge web site was easy enough.
http://monami.sourceforge.net/
Also see the link from the gridpp wiki http://www.gridpp.ac.uk/wiki/MonAMI

As I already use ganglia the idea was that I'd run some checks on disk space and DPM and send the output to ganglia. The first thing we noticed was that in order for some of the features to work you need to be running at least v3 of ganglia. I was still running v2.5, a quick upgrade of the gmond rpms and a new gmond.conf was required.
You do also require mysql. (For the DPM plugin - more later)

The main configuration file is /etc/monami.conf, but this can read further files in /etc/monami.d, so we set about making a basic file to monitor the root file system.

[filesystem]
name=root-fs
location=/

[sample]
interval=1m
read = root-fs.blocks.free
write = ganglia

[snapshot]
name=simple-snapshot
filename=/tmp/monami-simple-snapshot

[ganglia]
multicast_ip_address = 239.2.11.95
multicast_port = 8656

more coming soon....

CAMONT jobs successfuly running at Oxford

2007-03-08T13:07:00.000+01:00

The CAMONT VO has now been working correctly at Oxford since Friday 2nd March.
Karl Harrison of Cambridge has been running jobs from Cambrdige.

In another Cambridge collaboration, LHCB software has been installed on a Windows server 2003 test node at Oxford by Ying Ying Li from Cambridge. They are testing the use of Windows for LHCb analysis code, and having tetsed at Cambridge were looking to prove it could work at other sites. Ideally they would like some more test nodes and 0.5 TB of disk space. This may be harder to find.

Cambridge ran the Atlas DPM ACL fix on Monday 5th when I (PDG) visited Santanu. Now all SouthGrid sites have run the required fix.

I took the opportunity to measure the power consumption of the new Dell 1950's (Intel 5150 cpus). Idle power consumtion is about 200W rising to 285 under load (4 cpu intensive jobs).

Oxford and the ATLAS DPM ACL fix.

2007-03-01T10:57:00.000+01:00

I tried to run the ATLAS patch program yesterday to fix the ACL's on the DPM server at Oxford.
This update has be provided as a binary from ATLAS that has to be run as root on the se. This was potentially dangerous and many sites had delayed running this, and objected to the fact that we don't really know what it is doing. Anyway the pragmatic approach seemed to be that most other sites had run it now so I would.
The configuration file has to be edited to match the local sites config.
I perfomed a normal file backup using the HFS software Tivolis Storage Manager.
dsmc incr
Then dumped the mysql data base.
mysqldump --user=root --password=****** --opt --all-databases | gzip -c > mysql-dump-280207.sql.gz
As our main DPM server was currently set readonly (To cope with the DPM bug of not sharing across pools properly) we decided to set it back to read/write for the update.
dpm-modifyfs --server t2se01.physics.ox.ac.uk --fs /storage --st 0
Then run the update program (refered to as a script in some docs):
./UpdateACLForMySQL
Unfortuneatly I had used the wrong password in the config file so it failed,
this is where a strange feature of the update program was discovered.
After it runs it removes several entries from the config file , the password and the gid entry, so after several attempts the correct config file was used and the update appears to have been successful.
dpns-getacl /dpm/physics.ox.ac.uk/home/atlas/dq2

Shows the acls.
# file: /dpm/physics.ox.ac.uk/home/atlas/dq2
# owner: atlas002
# group: atlas
user::rwx
group::rwx #effective:rwx
group:atlas/Role=production:rwx #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:atlas/Role=production:rwx
default:mask::rwx
default:other::r-x

I reset the main DPM server back to read only:
dpm-modifyfs --server t2se01.physics.ox.ac.uk --fs /storage --st RDONLY

The process was not simple or clear and I hope not to have to do more for other VO's...

Birmingham suffering from multiple hardware failures

2007-03-01T10:54:00.001+01:00

The Babar cluster at Birmingham which is made up of older kit salvaged from QMUL and Bristol plus the original Birmingham cluster, is suffering from h/w problems.
7 worker node disks have died, some systems have kernel panics, and the globus MDS service is playing up. Yves is working hard to fix things but maybe we are just getting to the end of the useful life of much of this kit?

glite UI update fix

2007-02-27T15:22:00.001+01:00

To resolve the missing dependancy on the ui for the last two updates of glite
in particular the glite-ui-config rpm required python-fpconst.
You can get this rpm from cern see:

http://linuxsoft.cern.ch/repository//python-fpconst.html

Use
wget http://linuxsoft.cern.ch/cern/SLC30X/i386/SL/RPMS/python-fpconst-0.6.0-3.noarch.rpm

to add this to your local repository.
Then yum -y update will work once again.

Another Workernode Hard drive failure at Oxford

2007-02-26T15:04:00.000+01:00

t2wn37 Hard drive failed over the weekend. Dell will replace.
This follows on from t2wn04 last week, and the PSU in t2lfc01 a few weeks before. t2lfc01 was one of the gridpp supplied nodes from Streamline. Replacement of the PSU took several weeks.

Fusion jobs successfully running at Oxford

2007-02-20T17:52:00.000+01:00

FUSION jobs have now run succesfully at Oxford.

Birmingham Network reconfiguration

2007-02-20T13:37:00.000+01:00

Yves reports:
Our site has been down since yesterday morning 10am until today 1000am due to network problem which IS have linked to a faulty link with a campus switch.
....
IS have temporarily disabled the physics link to the library switch, one of our two links to the network, and this has fixed the connectivity problem from the outside world to our grid box.

They will re-instate the link (for resilience) when they've got to the bottom of the problem (faulty fibre, or whatever).
-->

So, it'be interesting to see the gridmon result in the current configuration while waiting for IS to understand the problem.

This may be the cause of the 33% UDP Packet loss we have been seeing to/from birmingham.

FUSION VO Problems at Oxford

2007-02-20T13:34:00.000+01:00

At Oxford we had reports from Fusion of problems:
"We have checked that FUSION jobs fail at your site with the error "37 the provided RSL 'queue' parameter is invalid". This is because "fusion" is missing at the end of the file /opt/globus/share/globus_gram_job_manager/lcgpbs.rvf in your CE ("fusion" should be included in the list of Values of the attribute "queue"). We also noticed that the FUSION VOMS server certificate ([1]) is not installed at /etc/grid-security/vomsdir/ in your CE."

I down loaded the cert from :
http://swevo.ific.uv.es/vo/files/swevo.ific.uv.es-oct2006.pem

and ran
/opt/glite/yaim/scripts/run_function /root/yaim-conf/site-info.def config_globus
which made the 4 VO's I recently added appear in the files lcgpbs.rvf and pbs.rvf in
/opt/globus/share/globus_gram_job_manager/ .
I can only assume that we had had errors when we ran yaim the first time as the 4 new
VO's had not appeared the first time.

Problems with DNS style VO names

2007-02-19T14:22:00.000+01:00

We have now discovered that adding the camont VO is not straight forward due to the new DNS style VO name.
The current yaim cannot handle the long format for VO names. The new yaim 3.1 which is not yet released should help but has not yet been tested.
Yves has had a look at it and it is very different from the current version.
"Hello all,

I got hold of the new version of yaim and there are some non-trivial differences with the production version. I think it would be ill advised to try the new version in production. I think we could enable this new vo style by configuring gip by hand and then perform the correct queue to group mapping for pbs/condor. But instead of all sites doing this (plus potential RB complications?), couldn't we revert to the current vo style (if running jobs is urgent), I do not understand while the new vo style should be implemented on production sites when it is still awaiting certification and has not even been tried on the pre-production service?

Thanks,

Yves
"

New VO's added at Oxford

2007-02-15T12:15:00.000+01:00

Support for Minos, Fusion, Geant 4 and camont were added yesterday at Oxford.

The new CA rpms were also installed so now we should be green again.

Latest glite update problem on UI

2007-02-12T17:50:00.000+01:00

Got the below error on my UI when I tried to update to the latest rpms.
This has already been reported as a GGUS ticket no:
https://gus.fzk.de/ws/ticket_info.php?ticket=18358

gronbech@ppslgen:/var/local> ssh root@t2ui02 'yum -y update;pakiti'
Gathering header information file(s) from server(s)
Server: Oxford LCG Extras
Server: gLite packages
Server: gLite updated packages
Server: gLite updated packages
Server: LCG CA packages
Server: SL 3 errata
Server: SL 3 main
Finding updated packages
Downloading needed headers
Resolving dependencies
.....Unable to satisfy dependencies
Package SOAPpy needs python-fpconst >= 0.6.0, this is not available.

Steve Lloyds ATLAS Test jobs

2007-02-12T14:46:00.000+01:00

Work was carried out at Oxford to find out why the Atlas test jobs were not working.
It seemed there were some old references to pool accounts of the format atlas0100 and upwards which should have been atlas100 upwards. Once all references to these were removed the jobs started working. The problem effected both the ce and the DPM server.

PDG requested that 12.0.5 be installed at Oxford via the web page:
https://atlas-install.roma1.infn.it/atlas_install/protected/rai.php
but wonders if he should have been using
https://atlas-install.roma1.infn.it/atlas_install/

The installation was complete by Friday 9th Feb.
Results for Oxford were all fine until the problems over the weekend.
http://hepwww.ph.qmul.ac.uk/~lloyd/atlas/atest.php

The problems at Bristol are caused by the worker having very small
home disk partitions. The atlas software can not be loaded as their is insufficent space to expand the tar file.

Oxford instabilities over the weekend

2007-02-12T14:41:00.000+01:00

The Oxford site had trouble over the weekend due to the system disk on the ce getting full.
This was mainly due to large number of old log files. These have been migrated off to part of the software directory for now.
The dpm server was also in a bad state and services had to be restarted.

Meanwhile PDG is on the process of adding support for some new VO's; namely:
MINOS, FUSION, GEANT4 and CAMONT

While also on TPM duty this week.

RALPPD to get another upgrade

2007-02-12T14:37:00.000+01:00

Chris Brew announced on Friday 9th Feb:
RALPPD have been awarded another chunck of money to be spent by March 31st.2007
This will allow them to purchase one rack of CPU's and one rack of Disks.
The CPU's will be equivalent to 275KSI2K bringing the total to about 600KSI2K, and the new disks will be 78TB,s bringing the total to 158TB,s.
This includes the 50TB's currently on loan to the T1 will be returned shortly.
The hardware will be identical to the recent T1 purchase.

Cambridge New Systems Arrive

2007-02-12T14:19:00.000+01:00

Santanu announced on 19.1.07:

Just to let you know that all the new machines have arrived; just waiting for the rack to be delivered and the Dell engineer (that's actually the part of the contact) to come and switch it on.

When done, it's gonna give LCG/gLite another 128 CPUs and if our experiment with CamGrid and Condor succeeds, it will top up another ~500 CUPs. Now we can mount /experiment-software and LCG middleware area onto any CamGrid machine with any root permissions, WNs outbound connection is also sorted out. Now need to think about the stupid "WN pool account"

Intel call it as "Woodcrest". All the nodes are dual-core dual CPU, so 4 CPUs under the same roof
Dell Model : PE1950
Processor : Xeon 5150Ghz/4MB 1333FSB
Memory : 8*1GB dual rank DIMMs

2006-11-30T17:27:00.001+01:00

School of System Engineering
The University of Reading

Are now setting up their cluster, Emails between Jeremy Coles, Pete Gronbech and Yves have been passing giving advice on setup.

2006-11-30T17:16:00.000+01:00

Oxford instability of the ce over the last few days, first the bdii crashed and was restarted.
Second the globus_mds service crashed and was restarted. IN the end I decided to reboot the node today.
New jobs seem to be arriving now and we are passing SAM tests again.

2006-11-21T12:54:00.000+01:00

13.11.06
Ralppd : upgraded to torque 2 and dCache 1.7 .

Lawrie at Birmingham complained about the amount of spam to "lcg" style mailing lists? At least half my spam comes from the following lists:

proj-atlas-geant4-emb@cern.ch
project-lcg-vo-sites@cern.ch
project-lcg-vo-atlas-sites@cern.ch
project-lcg-security-csirts@cern.ch

maybe one or two more. One answer would be to change these mailing list names and then don't advertise them on a web page! Or make them closed lists.

I have created a new page as a header for the shared private technical documents.
See the link at the bottom of
https://www.gridpp.ac.uk/southgrid/TechnicalBoard.html

It now includes phone numbers (off the GOCDB) for each site.

Some questions arise,
Should we have a separate security email list for the RAL site security for major problems not just LCG VO type probs?

Should Cambridge have an lcg-security email list that includes the site one. My worry is that LCG security challenges may not be seen by Santanu is he is not directly on the cert@cam list.

2006-10-26T13:51:00.000+01:00

A torque security flaw was made public last week, most southgrid sites shutdown queues on Friday night.
Then patched up over the next 3 days.
Cambridge was not affected as they use Condor.

2006-10-26T13:50:00.000+01:00

Sysman meeting at Cambridge followed by the SouthGrid Technical meeting, shared support was discussed and generally agreed, but needs to be formalised and put in place at Birmingham. RALPPD were not there to discuss.
Pete had another meeting with the Oxford Campus Network guys, who have been doing some tests and managed to get better through put by tweeking the kernel. It is now thought that the change that happened on August 15th may have increased the latency between sites and this may be the cause of the reduced bandwidth.
Further tests will be done.

2006-10-13T16:10:00.001+01:00

SouthGrid has been busy taking part in the Service Challenge Throughput tests.
Oxford carried out tests on September 26th and 27th. The first Oxford to RALpp went ok although slowly, the second RALPP to Oxford was extremly disapointing. Practically zero bandwidth.

Further iperf tests have been carried out between servers both within Oxford and SouthGrid. At the moment the main cause of the problem seems to be the installation of a new Campus Firewall on August 15th. See: the Gridmon web site.

Work with David Wallom of Oxford Grid has continued in order to enable the NGS VO on Oxford's cluster.

Another worker node PSU failed two weeks ago and was replaced under warrenty by Dell.

Security Update

2006-09-06T09:45:00.000+01:00

Oxford UI updated other nodes proceding.
Bristol, Cambridge and Birmingham have confirmed they too have updated.

Oxford updated

2006-09-05T13:33:00.000+01:00

Over the weekend I updated the rpms on Oxfords gird nodes. On Monday I re ran yaim to make the changes take effect. Testing was hammpered by the SFT page being very slow and the submission page not working well. On Tuesday I discovered Oxford had been failing the SFT's due to pbs not working properly, because there was no longer a nodes, file. This was traced to a typo in the site-info.def file. yaim was re run on which seemed to cure things.

Yves has had trouble getting >250Mb/s from Birmingham, he thinks this is due to dpm problems rather than raw bandwidth issues as iperf tests give much better results.

We are now testing Oxford instead of Birmingham.

Friday pm

2006-09-01T16:46:00.000+01:00

Bristol - Birmingham tests due to start today.
Winnie had some problems diagnosing the error messages from the transfer tests, Yves thinks the documentation is OK if you are an expert but it could be improved.

Yves has been benchmarking new systems, Intel Woodcrest vs AMD Opteron, Ethernet vs Infiniband, to provide data for Birminghams future escience cluster purchase. Some results will be available later.

update

2006-09-01T16:32:00.000+01:00

Four sites out of 5 are now running glite 3.0.2
Just Oxford to go which is being upgraded today.
rm failures at Oxford were cured by re running yaim on the se's. May be gridftp had gone mad?

EDFA-JET is now fully operational and running SFT's succesfully.

Yves continuing to test DPM-DPM throughput and has been tuning the kernel and tcp ip parameters to optimise performance.

Yves also carried out CASTOR DPM tests last Sunday, http://www.gridpp.ac.uk/wiki/RAL_Tier1_CASTOR_SRM_tests_T1toT2

Condor Problems at Cambridge

2006-08-25T16:15:00.002+01:00

The glite CE requires a version of Condor which is a developmet fork, and not the production release.
Santanu expects very few production sites to ever consider using a development release and yet LCG has a dependacy on it.
The second number in the release version is even for production and odd for development.
The numbers in question are; lcg requires 6.7.10-1 but Cambridge says 6.6.x-x is more likely at a production site or may be the next release which will be 6.8.x-x.

First Post

2006-08-25T12:54:00.000+01:00

Visited Yves at Birmingham on Monday 21st . Discussed the throughput tests he has been carrying out between Bristol and Ral and Bham. Carried out some tests between Oxford and Bham.

Whole building power testing at Oxford on Wednesday 23rd. Set queues to disabled on Monday to force quese to drain. I had previously marked all the nodes offline which meant Oxford failed some SFT's, just disabling the VO queues is a better way to do it. All systems came back OK on Thursday morning.

Yves has been helping Culham get the EDFA-JET site up and running via email. They are very nearly there.