EMI CREAM

We have installed emi creamce at Oxford. It was quite straight forward and apparently everything was setup by yaim properly except that emi cream uses normal /etc/, /usr/ directories instead of /opt/glite. It uses just one repository for all packages, no more separate TORQUE_* repositories.
Jobs were running perfectly and all test jobs completed successfully. But it was only getting lhcbpilot jobs and after looking more closely it was the classic "GlueCEStateWaitingJobs: 444444" problem.

Drilling through many layer of wrapper it comes to this issue
/sbin/runuser -s /bin/sh ldap -c "diagnose -g --host=t2ce02.physics.ox.ac.uk"
ERROR: 'diagnose' failed
ERROR: user 'ldap' is not authorized to execute command 'diagnose'

I think this is the less documented part of emi creamce. In glite, slapd and bdii-update process was run by edguser but with emi it is run by ldap user.
Edited maui.cfg file
ADMIN3 edginfo rgma edguser ldap

It solved the problem as I was using our site wide maui.cfg file instead of default created by yaim. Just a heads-up if you are planning to install emi creamce

SAM to MyEGEE to finally MyEGI

I have updated to latest release of wlcg nagios to gridppnagios. It is a major release in the sense that it stopped configuring MyEGEE for portal and replaced it with MyEGI. MyEGEE would be there until I drop the myegee DB from gridppnagios machine but don't trust it anymore. I got two complain about MyEGEE within few hours of updating it so I can say that people are looking at it.
The other main change is that now Nagios Configuration Generator(NCG) is using Aggregated Topology Provider(ATP) instead of SAMDB to configure nagios. ATP is part of the ROC/NGI nagios package which aggregate information from GOCDB, Top BDII and VO feed etc and it is single authoritative information source with topology information. But it is the central ATP(http://grid-monitoring.cern.ch/atp) which is being used by all ROC/NGI's for topology configuration for the sake of uniformity and probably reliability . Old SAM infrastructure can now retire in peace.
So MyEGI, It is a kind of all in one (https://gridppnagios.physics.ox.ac.uk/myegi).
It has Gridmap, metric status, history and so on. Aesthetically MyEGEE was better but MyEGI has more functionality and if you are still not convince then check the comparison of SAM, MyEGEE and MyEGI here (https://tomtools.cern.ch/confluence/display/SAM/MyEGI+vs+MyEGEE+vs+SAM+Portal ).
MyEGI have very good search options and also has advanced filter so you can optimize your search and add URL to your bookmark for instance status of your site.
I just discovered two bugs and the irritating things is that it is showing advance date on history bar. So if you want to see the status at 11 March, check for 12 March !
A bug has been opened and hopefully it will be fixed soon
https://tomtools.cern.ch/jira/browse/SAM-1325
https://tomtools.cern.ch/jira/browse/SAM-1326

Going through the Argus Valley

Being an early adopter site for Argus, Oxford got one of the first MUPJ from ATLAS using glexec through Argus and it failed! although we were passing ops glexec tests for long.
Our understanding of Argus was that it must have a policy which authorize pilots to switch to a normal user, so I had a policy like this to authorize pilot for glexec

resource "http://authz-interop.org/xacml/resource/resource-type/wn" {
obligation "http://glite.org/xacml/obligation/local-environment-map" {
}

action "http://glite.org/xacml/action/execute" {
rule permit { pfqan="/ops/Role=pilot" }
rule permit { pfqan="/atlas/Role=pilot" }
rule permit { pfqan="/cms/Role=pilot" }
}
}


After discussion with Argus experts on mailing list, it turned out that when pilot framework ask glexec to switch user from pilot to the effective user, LCMAP PEP plugin send the proxy of effective user to ARGUS server for authorization and mapping. So Argus must have policy which authorize effective user also. I have changed policy to look like that

rule permit {pfqan = "/atlas/Role=pilot" }
rule permit {pfqan = "/atlas/Role=lcgadmin" }
rule permit {pfqan = "/atlas/Role=production" }
rule permit {pfqan = "/atlas/" }

It solved the problem. Doesn't it look like that every atlas user is allowed to switch identity through glexec ? As for as Argus is concerned, yes. But glexec configuration is defined at WN and only groups which are whitelisted at /opt/glite/etc/glexec.conf are allowed to use glexec, any other user trying glexec will be shot down at WN itself. By default only pilot users are whitelisted at WN.
So in nutshell, policies at Argus should resemble that of the CE.

glite-APEL Node

On Thursday 9th December we brought the new glite-APEL box on line.

The VM hosted by t2delltest, had already been installed and Kashif had installed the cert.

We ran apel on all the ce's and t2torque02 and then one last time on t2mon02.

Then reconfigured t2ce02 to point at the new apel box and ran apel on it. We saw new records created on the box. (After sorting some permissions issues, need to rerun yaim with each ce (and t2torque02) set in the site-info.def file. Each run did the magic to allow that node to write to the db. (FQDN's should be used).
We then changed the reference to t2mon02 to t2apel01 in the site-info.def file on pplxconfig and it propagated round the other nodes.
The first run that night failed due to a java out of memory error.
I tweaked the config file /opt/glite/etc/glite-apel-publisher/publisher-config-yaim.xml
to

150000
from the original 300000

All apel logfiles on all ce's , t2torque02 and t2apel01 now appear to be good.
Cristina can see records appearing at RAL.

The old mysql database from t2mon02 has been backed up in /data/sysadmin (pplxfs2)

Tracing a Grid Job (A recap)

Just in case we should forget how to trace a grid job I record some steps below.

For example you discover via a CMS SAM page you are failing some test (could equally be any other SAM page such as LHCb) , you click on the detailed out put and see a reference to the job id:
on t2ce05 contains the string: sOFavxScVKU-GbSYaCmx-A
on t2ce05
grep sOFavxScVKU-GbSYaCmx-A /opt/edg/var/gatekeeper/grid-jobmap_20100906
reveals the batch system job id: lrmsID=2998805.t2torque02.physics.ox.ac.uk
on the batch server t2torque02 in our case, either:
tracejob 2998805

or

grep 2998805 /var/spool/pbs/server_logs/20100909

The tracejob option is easier!

This will let you know which worker node ran the job. You can then have a look at it to check for full disks, memory faults etc or segfaults in the log files......

Now in reverse

A job is misbehaving on your node and you need to see who is running it.
The special case here is that its an ATLAS pilot job, this does not have a normal grid job id.

Get the PID from top, use
pstree -H pid
to highlight the processes parents.
(Use pstree -A -H pid if on an putty window on Windows)

This reveals which pbs job it is
eg 3020508.t2torque02.physics.ox.ac.uk

The job can be traced on the panda monitor, using the search facility on the LH toolbar.
This gives the job details including the users name. A GGUS ticket could then be raised against ATLAS asking for the user to be informed.

APEL on ngsce-test

APEL was failing on ngsce-test with the following error.

java.io.FileNotFoundException: /var/spool/pbs/server_priv/accounting/20090522 (Too many open files)

The solution was to type:
ulimit -n 10240

I've added this to the /opt/glite/bin/apel-pbs-log-parser script.

A fix is in test, so a new version of APEL will fix it.
see GGUS ticket
https://gus.fzk.de/ws/ticket_info.php?ticket=60674

Argus Server at Oxford

We finally managed to install Argus server at Oxford with messy workaround. Installation and configuration was reasonably ok, and once policy structure was clear then writing and loading policy was also easy. Details are here http://www.gridpp.ac.uk/wiki/Oxford.

The main issue was host certificate issued by UK CA which contains an "emailAddress" and supposedly this is depreciated year(s) ago and most developers assume that there is no "emailAddress" in host certificate. Although still it is a bug in Argus and hopefully would be resolved in next release.
So the workaround
By default pap-admin command uses host certificate in /etc/grid-security/ if started from root but since there is a problem with host certificate so I copied my personal certificate proxy from UI and started pap-admin using that proxy. Then added ACE
pap-admin ace
"/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=t2argus02.physics.ox.ac.uk/OID.1.2.840.113549.1.9.1=lcg_manager@physics.ox.ac.uk" ALL
This workaround was suggested by Andrea Ceccanti

The only issue is that if you want to restart pap service then first remove ACE using remove-ace command, restart pap and then add ACE again.

Oxford's blanking panels

Having just read Stuart's ScotGrid blog post about cooling in the top of racks I thought I'd let you know about the panels we use.

We have been specifying that all empty racks slots should be filled by blanking panels since our 2007 purchase. The they used to use metal blanking panels.

These days they tend to supply the 1U APC plastic clip in panels, as can be seen in the RH rack in the photo.
These cost £25-£30 per pack of 10 but we managed to get a bulk (200) purchase in 2008 which worked out at about £1.69 each.

http://www.apc.com/resource/include/techspec_index.cfm?base_sku=AR8136BLK200

Jobs with analysis role

It started with a ticket from dzero about job failure at creamce at oxford. On investigation it was found that these jobs were coming with /dzero/users/Role=analysis/Capability=NULL and expectantly lcmaps failing with this error "no entry found for /dzero/users/Role=NULL/Capability=NULL ".
But the jobs from the same user were running on lcg-CE so on further investigation it turn out that lcmaps-voms plugins were failing on lcg-CE too but as per lcmaps policy it runs lcmaps-poolacount plugin after voms plugin failure and lcmaps-poolaccount uses individual DN mapping from grid-mapfile. So lcg-CE was mapping correctly to dzero pool account but through wrong procedure.
creamce don't use edg-mkgridmap file for creating grid-mapfile so no individual mapping is defined in grid-mapfile.
Solution was quite easy and we have to just define MAP_WILDCARDS=yes in vo.d/dzero and rerunning yaim created a slightly different grid-mapfile and groupmapfile with wild-cards.

dzero/Role=lcgadmin/Capability=NULL" dzerosgm
"/dzero/Role=lcgadmin" dzerosgm
"/dzero/Role=production/Capability=NULL" dzeroprd
"/dzero/Role=production" dzeroprd
"/dzero/*/Role=*" .dzero
"/dzero/*" .dzero
"/dzero/Role=NULL/Capability=NULL" .dzero
"/dzero" .dzero

So any job coming with different Role would be mapped to normal pool account.
The issue was discussed in this ticket https://savannah.cern.ch/bugs/index.php?26990

A week of upgrades for the RAL Tier 2 - Part 1 -The Network

Well it has been a long week at the RAL Tier 2. We've finally had our much postponed downtime to update our dCache installation (delayed once when one of the disk servers got a corrupt filesystem, then to avoid a CMS analysis test and finally to avoid an Atlas analysis test). The delays, however, did mean we could also include the long planned network upgrade in the downtime - this was probably a good thing.

So we had quite a programme of work for a five day downtime:

1. Replace the PNFS namespace in dCache with Chimera
2. Update dCache from 1.9.1 to the "Golden Release" 1.9.5
3. Install a new network switch and set up a 10Gb/s link between the two halves of our farm

Indeed, heading into work on Friday with neither dCache nor the network working, I thought I would be extending the downtime into the next week but by lunchtime things had improved and we were able to come out of the downtime on time at 5pm - although despite a full suit of "OK" SAM tests GridView still has us down until nearly eight o'clock.

Taking the last of the upgrades first: before last week we had the two halves of our farm in two different rooms. Each half of the farm has it's own Nortel 55XX network stack. Most of the storage is in the room known as Lab 8 in the R1 office building with a 10Gb/s connection to site Router A, whilst most of the compute nodes are in the Atlas lower machine room, A5Lower, with a 2x1Gb/s connection to Site Router A. That 2x1Gb/s connection between the storage and compute nodes was our main bottleneck - it would regularly run at over 99% capacity for days during Atlas Hammercloud tests.

The Plan was to install a Nortel 5650 switch into the stack in A5Lower then set up a direct 10Gb/s fibre link from there to Lab 8 - cutting out the 2x1GB/s link and Router A. That sounded fairly trivial and when I went down with Networking on Thursday afternoon to set it up I expected to be back in a hour to carry on struggling with our, at that time, broken dCache.

Due to cabling issues we had to re-order the switches in the stack and I also had to swap out a 5510 I had borrowed from the Tier 1 and replace it with a new one. So we broke up the current stack and tried to stack the 5650 with one of the 5510s. According to everything we had read they should have see each other, the 5650 should have downloaded an updated version of the firmware and software to the older 5510 and then the should have joined together as a single switch. But ours did not talk to each other.

Well possibly the version of the software on the 5510s was too old, so we went to each switch in turn, set it up with an IP address downloaded a new version on the firmware and software and restarted it.

By the end of Thursday - we were more-or-less back where we had started - we had a stack of 5510s (still without the 5650) .

On Friday morning Nick found a setting on the 5650 to allow "hybid stack mode" and suddenly everything worked.

We soon had all the correct VLANs set up and the two halves of our network were talking over the new fast link.

Backing up MySQL databases

Oxford have installed a simple script to backup the DPM mysql db once a day at 6am.
The script was loosely based on Glasgow's example here .

In order to restrict the file names produced to just 7, I've opted to use the current day rather than date.

[root@t2se01 ~]# cat /root/mysql-dump-pdg.pl
#!/usr/bin/perl
#
# Loosely based on the Glasgow script but simplified.
#
# Select the current day only as we want to have just seven unique file names which will be overwritten
# thus reducing the total backup size.


@weekDays = qw(Sunday Monday Tuesday Wednesday Thursday Friday Saturday);
($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
$theTime = "$weekDays[$dayOfWeek]";
#print $theTime;

$backup_dir="/var/lib/mysqldumps";
$mysql_user="root";
$mysql_pw_file="/root/mysql-pw";
$keep_days=7;


# Read mysql password
open(PW, $mysql_pw_file) || die "Failed to open password file $mysql_pw_file: $!\n";
$mysql_pw=;
chomp $mysql_pw;
close PW;

# Dump the db now
chdir $backup_dir || die "Failed to change to backup directory $backup_dir: $!\n";

system "/usr/bin/mysqldump --user=$mysql_user --password=$mysql_pw --opt --all-databases | gzip -c > mysql-dump-$theTime.sql.gz";
die "Mysql failed died with exit code $?\n" if $? != 0;

This is run by /etc/cron.d/mysql-dump
PATH=/sbin:/bin:/usr/sbin:/usr/bin
0 6 * * * root /root/mysql-dump-pdg.pl

So far it seems to work in testing!

Oxford Grid now SL5

All but one worker node on the Oxford Grid site has been reinstalled running SL5.
Currently these are served by one ce, t2ce05, but more will be added shortly to offer resilience.

Quarterly Report DPM script

Each quarter we needs to report on disk usage at our sites.
This can be tricky but the following script will help at DPM sites:

#!/bin/bash

DAY=`date +%F`
echo $DAY
for zz in `dpns-ls /dpm/physics.ox.ac.uk/home/`;do
dpns-du -z -s /dpm/physics.ox.ac.uk/home/$zz>>Oxford-SE-Usage-$DAY;
done


You will need to modify it appropriately for your site.
Extra added 20.10.09
This makes use of the dpns-du command in the gridpp-dpm toolkit available from :
http://www.sysadmin.hep.ac.uk/rpms/fabric-management/RPMS.storage/

Details of the other commands are on the wiki

120 new cores for EFDA-JET

30 new Sunfire 2200 m2 servers have been incorporated into the EFDA-JET site. Each has dual processor dual core Opteron 2218 processors, so that increases the number of Worker Nodes cores by 120 up to 254. Each node has 8GB RAM.

CMS at Oxford

Oxford was failing a ce CMS SAM test with a warning, probably due to some permissions problems in the se.
Following commands illuminated things:

This extract from /var/log/dpm/log
02/26 10:49:30 3869,24 dpm_srv_proc_put: processing request c75ce541-b2cd-4bdc-bf8f-c86ecb0be6ed from /C=UK/O=eScience/OU=CLRC/L=RAL/CN=chris cms brew
02/26 10:49:30 3869,24 dpm_srv_proc_put: calling Cns_stat
02/26 10:49:30 3869,24 dpm_srv_proc_put: calling Cns_creatx
02/26 10:49:30 3869,24 dpm_srv_proc_put: srm://t2se01.physics.ox.ac.uk:8446/srm/managerv2?SFN=/dpm/physics.ox.ac.uk/home/cms/store/user/test/oneEvt.root: DPM_FAILED (Permission denied)
02/26 10:49:30 3869,24 dpm_srv_proc_put: returns 0, status=DPM_FAILED (Permission denied)

Shows the test file creation failing

[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/
drwxrwxr-x 1 24135 1399 0 Jan 13 18:55 PhEDEx_Debug
drwxrwxr-x 2 24135 3490 0 Oct 13 12:15 PhEDEx_LoadTest07
drwxrwxr-x 0 24135 1399 0 Feb 26 12:20 brew
drwxrwxr-x 2 24135 1399 0 Jan 27 15:16 mc
drwxrwxr-x 2 24351 3422 0 Feb 06 18:57 unmerged
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 user
[root@t2se01 dpm]# dpns-listgrpmap |grep 1399
1399 cms
[root@t2se01 dpm]# dpns-listgrpmap |grep 3406
3406 cms/Role=lcgadmin
[root@t2se01 dpm]# dpns-getacl /dpm/physics.ox.ac.uk/home/cms/store/
# file: /dpm/physics.ox.ac.uk/home/cms/store/
# owner: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=chris cms brew
# group: cms/Role=cmst1admin
user::rwx
group::rwx #effective:rwx
group:cms/Role=lcgadmin:rwx #effective:rwx
group:cms/Role=production:rwx #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:cms/Role=lcgadmin:rwx
default:group:cms/Role=production:rwx
default:mask::rwx
default:other::r-x
[root@t2se01 dpm]# dpns-getacl /dpm/physics.ox.ac.uk/home/cms/store/brew
# file: /dpm/physics.ox.ac.uk/home/cms/store/brew
# owner: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=chris cms brew
# group: cms
user::rwx
group::rwx #effective:rwx
group:cms/Role=lcgadmin:rwx #effective:rwx
group:cms/Role=production:rwx #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:cms/Role=lcgadmin:rwx
default:group:cms/Role=production:rwx
default:mask::rwx
default:other::r-x
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/
drwxrwxr-x 1 24135 1399 0 Jan 13 18:55 PhEDEx_Debug
drwxrwxr-x 2 24135 3490 0 Oct 13 12:15 PhEDEx_LoadTest07
drwxrwxr-x 0 24135 1399 0 Feb 26 12:20 brew
drwxrwxr-x 2 24135 1399 0 Jan 27 15:16 mc
drwxrwxr-x 2 24351 3422 0 Feb 06 18:57 unmerged
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 user
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 test
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user/test
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 SAM-t2se01.physics.ox.ac.uk
[root@t2se01 dpm]# dpns-chgrp 1399 /dpm/physics.ox.ac.uk/home/cms/store/user
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 test
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/
drwxrwxr-x 1 24135 1399 0 Jan 13 18:55 PhEDEx_Debug
drwxrwxr-x 2 24135 3490 0 Oct 13 12:15 PhEDEx_LoadTest07
drwxrwxr-x 1 24135 1399 0 Feb 26 12:33 brew
drwxrwxr-x 2 24135 1399 0 Jan 27 15:16 mc
drwxrwxr-x 2 24351 3422 0 Feb 06 18:57 unmerged
drwxrwxr-x 1 24352 1399 0 Jan 21 18:21 user
[root@t2se01 dpm]# dpns-chgrp 1399 /dpm/physics.ox.ac.uk/home/cms/store/user/test
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/brew
-rw-rw-r-- 1 24135 1399 4788418 Feb 26 12:34 oneEvt.root
[root@t2se01 dpm]# dpns-ls -l /dpm/physics.ox.ac.uk/home/cms/store/user/test
drwxrwxr-x 1 24352 3406 0 Jan 21 18:21 SAM-t2se01.physics.ox.ac.uk
-rw-rw-r-- 1 24135 1399 4788418 Feb 26 12:36 oneEvt.root

Automount problems on torque server

We've been having a few problems with our torque server failing to automout disks randomly.

Most of the time the mounts succeeded but occasionally they would fail with just:

Dec 19 08:05:06 heplnx201 kernel: RPC: error 5 connecting to server nfsserver
Dec 19 08:05:06 heplnx201 automount[23438]: >> mount: nfsserver:/opt/ppd/mount: can't read superblock
Dec 19 08:05:06 heplnx201 automount[23438]: mount(nfs): nfs: mount failure nfsserver:/opt/ppd/mount on /net/mount
Dec 19 08:05:06 heplnx201 automount[23438]: failed to mount /net/mount
Dec 19 08:05:07 heplnx201 kernel: RPC: Can't bind to reserved port (98).
Dec 19 08:05:07 heplnx201 kernel: RPC: can't bind to reserved port.

With the wonders of Google I was able to find out that error 98 is address in use and that what is going on is that the client is unable to find a free port in it's port range to initiate the connection to the server.

The culprit seems to be torque, which when I checked with a netstat -a was using very single port from 600 to 1023, which quite neatly overlaid the nfs client port range of 600-1023.

Here Google failed me and I was unable to find anyway to limit the port range used by torque.

So for now I've taken the quick option of extending the nfs client port range down to port 300 with:

echo 300 > /proc/sys/sunrpc/min_resvport

I think I'd like to move the nfs client port range out of the priveledged port range altogether. I think this should be possible, the RFC says that it SHOULD use a port below 1023 but MAY use a higher port, but I'd like to test it a bit before I configure a major server like that.

static-file-Cluster.ldif edit required post yaim at Oxford

Every time we run yaim at Oxford we have to fix the number of cpu's in our cluster by hand.
on t2ce02:
diff static-file-Cluster.ldif-fixed /opt/glite/etc/gip/ldif/static-file-Cluster.ldif
64c64
< GlueSubClusterPhysicalCPUs: 384
---
> GlueSubClusterPhysicalCPUs: 2
[root@t2ce02 ~]# cp static-file-Cluster.ldif-fixed /opt/glite/etc/gip/ldif/static-file-Cluster.ldif

On t2ce04:
Physical cpu's needs to be 74. After the change the ldap query shows:
ldapsearch -x -H ldap://t2bdii01.physics.ox.ac.uk:2170 -b Mds-vo-name=UKI-SOUTHGRID-OX-HEP,o=grid|grep -i physicalcpu
GlueSubClusterPhysicalCPUs: 74
GlueSubClusterPhysicalCPUs: 384

EFDA-JET Service nodes upgraded to glite 3.1

We upgraded our service nodes to Scientific Linux 4.7 and glite-3.1. The worker nodes had been upgraded earlier. The problems/issues we had while upgrading to Scientific Linux 4.7 are listed below:

Storage Engine

While installing the SE glite middleware (glite-SE_dpm_mysql), there was
a missing dependency issue for the perl-SOAP-Lite package.

Error: Missing Dependency: perl-SOAP-Lite >= 0.67 is needed by package
gridview-wsclient-common

Doing a

# yum install perl-SOAP-Lite

only installs perl-SOAP-Lite-0.65, which is lower than the version needed.

The perl-SOAP-Lite rpm was downloaded from a different repository. We
initially downloaded the perl-SOAP-Lite-0.67.el4 but this one failed to install as it needed MQSeries and other packages to be installed. We finally downloaded perl-SOAP-Lite-0.67-1.1.fc1.rf.noarch.rpm and it installed without any problems.

When the node was configured by yaim, the following error was obtained

sed: can't read /opt/bdii/etc/schemas: No such file or directory

The file /opt/bdii/etc/schemas was missing. The fix is to copy the schemas.example file to schemas

# cp -i /opt/bdii/doc/schemas.example /opt/bdii/etc/schemas

First SAM test failed. lcg-lr was missing, we needed to install lcg_util.
This installed a new version of lcg_util that was on the other nodes. lcg_util
was then updated on all the nodes.

Compute Element (& site BDII)

We run the compute element service and the site BDII service on the same node.

While installing the glite-BDII packages, we obtained the following dependency errors.

Error: Missing Dependency: glite-info-provider-ldap = 1.1.0-1 is needed by package glite-BDII
Error: Missing Dependency: glue-schema = 1.3.0-3 is needed by package glite-BDII
Error: Missing Dependency: bdii = 3.9.1-5 is needed by package glite-BDII

Using yum to install the missing packages, installs these packages at a higher level and still causes the installation of glite-BDII packages to fail, as it needs these packages at the versions listed above. These packages were instead installed by hand. A GGUS ticket (Ticket-ID: 42456), which suggested that this problem is fixed in the latest release (update 34).

As with the SE install above, we had the same problem with the schemas file, missing. The above fix was repeated here.

When running yaim, we had the following errors,

grep: a: No such file or directory
grep: VO: No such file or directory
grep: or: No such file or directory
grep: a: No such file or directory
grep: VOMS: No such file or directory
grep: FQAN: No such file or directory
grep: as: No such file or directory
grep: an: No such file or directory
grep: argument: No such file or directory
qmgr: Syntax error - cannot locate attribute
set queue lhcb acl_groups += /opt/glite/yaim/bin/yaim: supply a VO or a VOMS FQAN as an argument

To fix it we edited the file /opt/glite/yaim/functions/utils/users_getvogroup and commented out

#echo "$0: supply a VO or a VOMS FQAN as an argument"

On Gstat web monitoring page, it was being reported that the SE service was missing ('SE missing in Gstat service'). To fix this problem, we edited the file /opt/bdii/etc/bdii-update.conf and add the following line for our SE.

SE ldap://grid001.jet.efda.org:2170/mds-vo-name=resource,o=grid

Mon Box

When running yaim, we had the following errors


Problem starting rgma-servicetool

Starting rgma-servicetool: [FAILED]
For more details check /var/log/glite/rgma-servicetool.log
Stopping rgma-gin: [ OK ]
Starting rgma-gin: [FAILED]

Fixed by defining a new java by adding the following to the site-info.def

HOSTNAME=`hostname`
if [ "$HOSTNAME" == "$MON_HOST" ] ; then
JAVA_LOCATION="/usr/lib/jvm/jre-1.5.0-sun"
else
JAVA_LOCATION="/usr/java/j2sdk1.4.2_12"
fi

We had the same 'schemas' file missing problem here as well.

Networking

EFDA-JET has a slightly unusually set up as we are restricted to a small number of external IP addresses. All nodes are on the same LAN with private IP addresses, whilst the service nodes also have external addresses. In the hosts files on the service nodes, all service nodes are referenced by their external addresses, whilst on the worker nodes, the service nodes are referenced by their private addresses.

This worked well for glite 3.0, but not for glite 3.1, where we saw clients on the worker nodes trying to contact the service nodes via their external addresses. It looks like glite 3.1 iservices are passing IP addresses for clients to be call back on at a later time. The complete solution was to run iptables on the worker nodes and NAT translate outgoing connections for external addresses of the service nodes to their corresponding internal addresses. This was done by adding the following to /etc/rc.local on the worker nodes.

/sbin/service iptables start
/sbin/iptables -A OUTPUT -t nat -d <CE-ext-addr> -j DNAT \
--to-destination <CE-int-addr>
/sbin/iptables -A OUTPUT -t nat -d <SE-ext-addr> -j DNAT \
--to-destination <SE-int-addr>

dCache Update

We updated dCache this morning to 1.9.0. Now that sounds like a major jump but reading the release notes it is only a minor step up from the 18.0.15pX series of releases.

The upgrade itself was trivial, just installing the new dcache-server rpm and running install.sh across all the nodes.

We also took the opportunity to update the version of Postgresql on the head node from 8.3.1 to 8.3.5 using rpms from pgsqlrpms.org. I'm hoping that I will now be able to use their prebuilt slony-1 rpm to set up master slave mirroring of the databases from the dCache head node to a live mirror node.

Finally we updated the SL version of all the dCache nodes to SL4.6 from a mix of SL4.4, SL4.5 and SL4.6. We're now using the SL-Contrib xfs kernel modules on all nodes and the Araca drivers complied into the 2.6.9-78 series of kernels on all nodes with Areca raid cards rather than our own builds and have had no issues.

Fix ACls on ATLASLOCALGROUPDISK at Oxford

Today I ran Graeme's script to fix the acls on the ATLASLOCALGROUPDISK space token.
Should have done this a few weeks ago but ..
There is nothing currntly stored here yet.

[root@t2se01 ~]# ./atlas-uk-local-dpm-token-fix.sh
Debug: t2se01.physics.ox.ac.uk - physics.ox.ac.uk - atlaslocalgroupdisk
Fixing permissions on /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk...
Searching /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk...

dpns-ls /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
shows nothing

dpns-getacl /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
# file: /dpm/physics.ox.ac.uk/home/atlas/atlaslocalgroupdisk
# owner: root
# group: atlas/uk
user::rwx
group::rwx #effective:rwx
group:atlas/Role=lcgadmin:rwx #effective:rwx
group:atlas/Role=production:rwx #effective:rwx
group:atlas/uk:rwx #effective:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:atlas/Role=lcgadmin:rwx
default:group:atlas/Role=production:rwx
default:group:atlas/uk:rwx
default:mask::rwx
default:other::r-x