Simple CVMFS puppet Module

Oxford was one of the first site to test CVMFS and also to use cern CVMFS module. Initially installation of CVMFS was not well documented  so cern cvmfs puppet module was very helpful in installing and configuring cvmfs.
Installation became easy and more clear with the newer version of cvmfs. One of my ops action was to install gridpp multi vo cvmfs repo with cern cvmfs puppet module. We realized that it is easy to write a trimmed down version of cern  cvmfs module rather than use cern cvmfs module directly. The result is cvmfs_simple module which is available on GitHub.

'include cvmfs_simple' will set up LHC repos and gridpp repo

Only mandatory parameter is

cvmfs_simple::config::cvmfs_http_proxy : 'squid-server'

It is also possible to add local cvmfs repository. Extra repos can be configured by passing values from hiera

cvmfs_simple::extra::repo: ['gridpp', 'oxford']

Oxford is using a local cvmfs repo to distribute software for local users. oxford.pp can be used as template for setting new local cvmfs repo.

cvmfs_simple doesn't support all use cases and it expects that everyone is using hiera ;) . Please feel free to change it for your use case. 

Nagios Monitoring for Non LHC VO’s

A brief description of monitoring framework before coming to the actual topic of Non LHC VO's monitoring.

Service Availability Monitoring (SAM) is a framework for monitoring grid sites remotely. It consists of many components to perform various functions. It can be broadly divided into

‘What to Monitor’ or Topology Aggregation:  Collection of service endpoints and metadata from different sources like GOCDB, BDII, VOMS etc. Custom topological source (VO Feeds) can also be used.

Profile Management:  Mapping of services to the test to be performed.  This service is provided by POEM ( Profile Management) database.  It provides a web based interface to group various metrics into profiles.

Monitoring: Nagios is used as monitoring engine. It is automatically configured based on the information provided by Topology Aggregator and POEM.

SAM software was developed under EGEE project at CERN and now maintained by EGI.

It is mandatory for grid sites to pass ops VO functional test to be part of WLCG. Every NGI maintains a Regional SAM Nagios and result from regional SAM Nagios also goes to central MyEGI which is used for Reliability/Availability calculation.   

UK Regional Nagios is maintained at Oxford

https://gridppnagios.physics.ox.ac.uk/nagios/

and a backup instance at Lancaster

https://gridppnagios.lancs.ac.uk/nagios/

VO-Nagios

There was no centralize monitoring of Non LHC VO’s for long time and it contributed to bad user experience as it was difficult to find whether a site is broken or problem at the user end.  It was decided to host a multi VO Nagios at Oxford as we had experience with WLCG Nagios.

https://vo-nagios.physics.ox.ac.uk/nagios/

It is currently monitoring five VO’s

gridpp

t2k

snoplus.snolab.ca

pheno

vo.soutgrid.ac.uk

Sites can look for tests associated with only their site

https://vo-nagios.physics.ox.ac.uk/nagios/cgi-bin/status.cgi?hostgroup=site-UKI-SOUTHGRID-OX-HEP&style=detail

VO managers may be interested to see tests associated with a particular VO only

https://vo-nagios.physics.ox.ac.uk/nagios/cgi-bin/status.cgi?servicegroup=VO_pheno&style=detail

We are using VO-feed mechanism to aggregate site metadata and endpoint information. Every VO has a vo-feed available on a web server.  Currently we are maintaining this VO-feed 

http://www-pnp.physics.ox.ac.uk/~mohammad/snoplus.snolab.ca.feed.xml

http://www-pnp.physics.ox.ac.uk/~mohammad/t2k.org.feed.xml

http://www-pnp.physics.ox.ac.uk/~mohammad/vo.southgrid.ac.uk.feed.xml

http://www-pnp.physics.ox.ac.uk/~mohammad/pheno.feed.xml

http://www-pnp.physics.ox.ac.uk/~mohammad/gridpp.feed.xml

VO feed provides list of services to be monitored. I am generating this VO-feed through a script

https://github.com/kashif74/vo-feed

Jobs are submitted using a proxy generated from a Robot Certificate assigned to Kashif Mohammad. These jobs are like normal grid user jobs and test things like GCC version and CA version. Jobs are submitted every eight hour and this is a configurable option.  We are monitoring CREAMCE, ARC-CE and SE only. Services like BDII, WMS etc. are already monitored by Regional Nagios so there was no need for the duplication.  

For more information, these links can be consulted

https://wiki.egi.eu/wiki/SAM

https://tomtools.cern.ch/confluence/display/SAMDOC/SAM+Overview.html

https://tomtools.cern.ch/confluence/display/SAMDOC/SAM+Public+Site.html

Configuring ARC CE and Condor with puppet

ARC CE and condor using puppet

We have started testing Condor and ARC CE with the intention of moving away from Torque.  Almost one third of cluster has been moved to condor and we are quite satisfied with Condor as a batch system.  Condor setup was fairly easy but configuring ARC CE was bit challenging.  I believe that new version of ARC CE has fixed most of the issue I faced.  Andrew Lahiff was of great help in troubleshooting our  problems .Our setup consists of

1           CE :  Configured as ARC CE and  Condor submit host and runs Condor SCHEDD process

2              Central manager :  Condor Server and  runs Condor COLLECTOR and NEGOTIATOR process

3              WN’s :  Runs Condor  STARTD process, also installed emi-wn and glexec metapackages.

CE , Central Manager and condor part of WN’s  were completely  configured  with puppet.  I have to run yaim on WN’s t configure emi-wn and glexec.

I used puppet modules from https://github.com/HEP-puppet which were initially written by Luke Kreczko from Bristol.  We are using Hiera to pass parameters but most puppet modules works without Hiera as well.  I am not intending to go into details of condor or ARC CE but rather use of puppet modules to install and configure Condor and ARC CE.

Condor :

 https://github.com/HEP-Puppet

It was a pleasing experience to configure condor with puppet.

     Git clone https://github.com/HEP-Puppet/htcondor.git to module directory on puppet server

     include htcondor

on CE, Central Manager and WN’s and then Hiera tells that which service has to be configured on a particular machine.

# Condor

htcondor::managers:

- 't2condor01.physics.ox.ac.uk'

htcondor::computing_elements:

- 't2arc01.physics.ox.ac.uk'

htcondor::worker_nodes:

- 't2wn*.physics.ox.ac.uk'

htcondor::uid_domain: 'physics.ox.ac.uk'

htcondor::collector_name: 'SOUTHGRID_OX'

htcondor::pool_password: 'puppet:///site_files/grid/condor_pool_password'

This configures a basic condor cluster.  There is no user account at this stage so a test user account can be created on all three machines and basic condor jobs can be tested.  Htcondor manual is here

http://research.cs.wisc.edu/htcondor/manual/

Setting up user accounts :

https://github.com/HEP-Puppet/puppet-grid_pool_accounts

I  used this module to create user accounts only  for central manager and ce.  Since I have to run yaim on WN’s to setup emi-wn and glexec so  created user account on WN through yaim.

This puppet module can parse a glite type users.conf to create users account or range of  id’s can be passed to the module.

Setting up voms server :

https://github.com/HEP-Puppet/puppet-voms

It is used to set voms client on central-manager and ce.  One way to use this module is to pass name of each VO separately as described in the readme file of the module.

     Git clone   https://github.com/HEP-Puppet/puppet-voms.git

     Class { ‘voms::atlas’}

I  have used small wrapper class to pass all VO’s as array to wrapper class

     Git clone  https://github.com/oxford-physics/puppet-setup-voms-client.git

     include include setup_grid_accounts

Then pass name of the VO’s through Hiera setup_grid_accounts::vo_list:

    - 'alice'

    - 'atlas'

    - 'cdf'

    - 'cms'

    - 'dteam'

    - 'dzero'

ARC CE :

 https://github.com/HEP-Puppet/arc_ce

include arc_ce and on CE and then pass configuration parameters from Hiera. It has a very long list of configurable parameters and most of the default values works ok.  Since most of values are passed through Hiera so arc Hiera file is quite long, I am giving few of the examples

   arc_ce::infosys_registration:

  clustertouk1:

    targethostname: 'index1.gridpp.rl.ac.uk'

    targetport: '2135'

    targetsuffix: 'Mds-Vo-Name=UK,o=grid'

    regperiod: '120'

  arc_ce::queues:

     gridAMD:

       default_memory: '2048'

       cluster_cpudistribution:

         - '1cpu:4'

      cluster_description:

          OSFamily: 'linux'

          OSName: 'ScientificSL'

          OSVersion: '6.5'

          OSVersionName: 'Carbon'

          CPUVendor: 'GenuineIntel'

          CPUClockSpeed: '2334'

          CPUModel: 'xeon'

          NodeMemory: '2048'

          totalcpus: '168'

This almost sets up condor cluster with arc ce. There are few bits in arc and puppet modules which are there as a workaround for things which have already been fixed upstream. It needs some testing and clean up.

WN's needs some small runtime env setting specific to ARC. When jobs arrive at WN's it looks into /etc/arc/runtime/ directory for ENV settings 

 Our's runtime tree is like this.
├── APPS
│   └── HEP
│       └── ATLAS-SITE-LCG
└── ENV
    ├── GLITE
    └── PROXY

It can be just empty files. SAM-Nagios doesn't submit jobs if ARC CE is not publishing GLITE env.

I may have missed few things so please feel free to point it out.

 

 

Configuring CVMFS for smaller VOs

We have just configured cvmfs for t2k, hone, mice and ilc after sitting on the request for long time. The main reason for delay was the assumption that we need to change cvmfs puppet module to accommodate non lhc VOs.   It turns out to be quite straight forward with  little effort.
We are using cern cvmfs module and there was an update a month ago so it is better to keep it updated.
  https://github.com/cvmfs/puppet-cvmfs

 Using hiera to pass parameters to module, our hiera bit for cvmfs

   t2k.gridpp.ac.uk:
      cvmfs_server_url: 'http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/@org@.gridpp.ac.uk;http://cvmfs01.nikhef.nl/cvmfs/@org@.gridpp.ac.uk'
    mice.gridpp.ac.uk:
      cvmfs_server_url: 'http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/@org@.gridpp.ac.uk;http://cvmfs01.nikhef.nl/cvmfs/@org@.gridpp.ac.uk'
    hone.gridpp.ac.uk:
      cvmfs_server_url: 'http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/@org@.gridpp.ac.uk;http://cvmfs01.nikhef.nl/cvmfs/@org@.gridpp.ac.uk'
    ilc.desy.de:
      cvmfs_server_url: 'http://grid-cvmfs-one.desy.de:8000/cvmfs/@fqrn@;http://cvmfs-stratum-one.cern.ch:8000/cvmfs/@fqrn@;http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/@fqrn@'

One important bit is the name of cvmfs repository e.g  t2k.gridpp.ac.uk instead of t2k.org

Other slight hitch is public key distribution of various cvmfs repositories.  Installation of cvmfs also fetch cvmfs-keys-*.noarch rpm which put all the keys for cern based repository into /etc/cvmfs/keys/.

I have to copy publich key for gridpp.ac.uk and desy.de to /etc/cvmfs/keys. It can be fetched from  repository
wget http://grid.desy.de/etc/cvmfs/keys/desy.de.pub -O desy.de.pub
or copied from
https://www.gridpp.ac.uk/wiki/RALnonLHCCVMFS

we  distributed the keys through puppet but outside cvmfs module.
It would be great if some one can convince cern to include public keys of other repositories into cvmfs-keys-* rpm. I am sure that there is not going to be many cvmfs stratum 0s.

Last part of the configuration is to change SW_DIR in site-info.def or vo.d directory
 SW_DIR="/cvmfs/t2k.gridpp.ac.uk"

WNs requires re-yaim  to configure SW_DIR in /etc/profile.d/grid-env.sh.  You can also edit grid-env.sh file manually and distribute it through your favourite configuration management system.

A dramatic effect on Atlas jobs when xrootd dies

This week for the first time at our site the xrootd server process on our DPM SE died.

The ganglia plot shows a dramatic falloff in load.
As all the jobs started to fail to access the data. The number of jobs running in the batch systems remained high so pbswebmon did not alert us although Kashif had noticed the jobs were very inefficient on Tuesday evening. Which in hind sight was the give away that something was amiss. We recieved a ticket from Atlas and Ewan restarted the daemon and all recovered.

Should we Hyperthread

Following the recent discussion on hypertheading on the TB-Support mail list and having several sets of nodes that have hyperthreading and three or four gigabytes of memory per core we decided to run some tests to see if opening some job slots on the virtual cores would increase our throughput.

The first test was to benchmark the nodes with hyperthreading enabled out to the full number of cores. We have three sets of nodes with hyperthreading capabilities with E5520, X5650 and E5645 CPUs. For each type we ran one to n instances of the HEPSPEC benchmarking tests when n is the number of real plus the number of virtual cores.

Figure (1): Total Node HEPSPEC by number of concurrent tests

These are shown in Figure (1) and clearly show the nearly linear rise as the test run on the real CPUs then flattening of as more of the virtual cores run becoming almost completely flat or even dropping again as all the virtual cores are used. However it does show a clear increase in the total HEPSPEC rating of the node when using half of the virtual cores. That should mean that there will be a real gain in output by enabling jobs on these virtual cores, as long as real work scales like the HEPSPEC tests and we don't run into network, memory of disk I/O bottlenecks.

Armed with this information we decided to check the real world performance of the nodes with E5520 and X5650 CPUs with jobs running on half the virtual CPUs.

To do this we took half of each set of nodes and increased the number of job slots by 50% (8 to 12 for the E5520s and 12 to 18 for the X5650s, the E5645 nodes are still in test after delivery and not yet ready for production) and reduced the pbs_mom cpumult and wallmult parameters to reflect the lower per core HEPSPEC rating once we start using the virtual cores and returned them to running production jobs.

They have now been running real jobs for seven days and we have enough statistics to start comparing the nodes running jobs on virtual cores with those not doing so.

Figure (2): Average Job Efficiency for nodes using and not using virtual cores

Figure (2) shows the average job efficiency (total CPU time divided by total wall time) for jobs run on nodes with and without the virtual cores in use. There is no sign of a drop in efficiency when running on the virtual cores so it would appear that at 12 or 18 jobs per node we are not yet hitting network, memory or disk I/O bottlenecks.

Figure (3): Average Job Efficiency for different VOs and Roles on the different classes of nodes

Figure (3) shows the average job efficiency for different VOs and roles on the different classes of nodes, and shows no sign of a systematic drop in efficiency when running jobs of the virtual cores (only the prdlhcb group shows signs of such an effect and the statistics are somewhat lower for that).

Figure(4):HEPSPEC06 Scaled CPU Hours per Node per Day

Figure (5): HEPSPEC06 Scaled CPU Hours per Node per Day

Figures (4) and (5) HEPSPEC06 scaled CPU hours accumulated per day per core or node (total unscaled CPU time of all jobs on that class multiplied by the HS06 rating divided by the number of days the test ran and the number of cores or nodes in that class) shows as hoped that although the individual cores accumulate HEPSPEC06 scaled CPU hours faster when not running on virtual cores that is more than offset by the increase in the number of slots per node.

Figure (6): Average number of Jobs per Core per Day

Figure (7): Average number of Jobs per Node per Day

Finally Figures (6) and (7) Jobs per day per node or core (Total number of jobs run on each class of node decided by the length of the test and the number of nodes or cores in that class) shows a similar effect - the individual cores manage to do more work when no jobs are running on the virtual cores but the increase in the number of slots more than make up for it.

In conclusion it appears that running jobs on half of the virtual cores for nodes that are hyperthreading capable gives a 30-40% increase in the total "installed capacity" provided by those nodes without any apparent decrease in the efficiency of jobs running on those nodes.

We will continue the test for another week but unless the numbers change drastically we will be changing our policy and running jobs on half of the virtual cores on hyperthreading capable nodes.

Chris and Rob.

HEPSEPC06 on AMD Interlagos 6276

I have been running HEPSPEC06 on a recent Dell 815 with the new AMD 16 core Interlagos processors.

The only valid HEPSPEC06 result (for current GridPP use) is the SL5 (64 bit OS) but 32 bit compiler result but for interest we ran also with 64 bit compiler switches.

Then we installed SL6 and re-ran both 32 and 64 bit compiler options.

The results are on the GridPP wiki, but the most notable thing is the performance boost you can get going from 32bit on SL5 to 64 bit on SL6.

The boost is nearly 25% which could mean a lot to the experiments and the sites productivity if they can be persuaded to migrate sooner rather than later.

https://www.gridpp.ac.uk/wiki/HEPSPEC06#UKI-SOUTHGRID-OX

Installing and Deploying a Cluster Publisher

As part of the battle to replace out LCG-CEs with CreamCEs I realised that the reason one of our new CreamCEs was not getting many jobs was because it was not publishing a cluster/subcluster into the BDII (despite having a /var/lib/bdii/gip/static-file-Cluster.ldif file) and so I guess wasn't matching any resources.

Since, I eventually wanted to go to a stand alone Cluster Publisher I thought it would be easiest to push ahead and install that rather than try to install one one the CreamCE and remove it later.

So with a shiny new VM in hand and certificate I plunged onwards.

First step was to define the cluster variables in site-info.def (or in this case a specific node file):

cat /opt/glite/yaim/etc/nodes/heplnv146.pp.rl.ac.uk
CE_HOST_heplnx206_pp_rl_ac_uk_CE_TYPE=cream
CE_HOST_heplnx206_pp_rl_ac_uk_CE_InfoJobManager=pbs
CE_HOST_heplnx206_pp_rl_ac_uk_QUEUES="grid"
CE_HOST_heplnx207_pp_rl_ac_uk_CE_TYPE=cream
CE_HOST_heplnx207_pp_rl_ac_uk_CE_InfoJobManager=pbs
CE_HOST_heplnx207_pp_rl_ac_uk_QUEUES="grid"
CLUSTER_HOST=heplnv146.pp.rl.ac.uk
CLUSTERS=GRID
CLUSTER_GRID_CLUSTER_UniqueID=grid.pp.rl.ac.uk
CLUSTER_GRID_CLUSTER_Name=grid.pp.rl.ac.uk
CLUSTER_GRID_SITE_UniqueID=UKI-SOUTHGRID-RALPP
CLUSTER_GRID_CE_HOSTS="heplnx206.pp.rl.ac.uk heplnx207.pp.rl.ac.uk"
CLUSTER_GRID_SUBCLUSTERS="GRID"
SUBCLUSTER_GRID_SUBCLUSTER_UniqueID=grid.pp.rl.ac.uk
SUBCLUSTER_GRID_HOST_ApplicationSoftwareRunTimeEnvironment="
        LCG-2
        LCG-2_1_0
        LCG-2_1_1
        LCG-2_2_0
        LCG-2_3_0
        LCG-2_3_1
        LCG-2_4_0
        LCG-2_5_0
        LCG-2_6_0
        LCG-2_7_0
        GLITE-3_0_0
        RALPP
        SOUTHHGRID
        GRIDPP
        R-GMA
"
SUBCLUSTER_GRID_HOST_ArchitectureSMPSize=4
SUBCLUSTER_GRID_HOST_ArchitecturePlatformType=x86_64
SUBCLUSTER_GRID_HOST_BenchmarkSF00=0
SUBCLUSTER_GRID_HOST_BenchmarkSI00=2390
SUBCLUSTER_GRID_HOST_MainMemoryRAMSize=2000
SUBCLUSTER_GRID_HOST_MainMemoryVirtualSize=2000
SUBCLUSTER_GRID_HOST_NetworkAdapterInboundIP=FALSE
SUBCLUSTER_GRID_HOST_NetworkAdapterOutboundIP=TRUE
SUBCLUSTER_GRID_HOST_OperatingSystemName=ScientificSL
SUBCLUSTER_GRID_HOST_OperatingSystemRelease=5.4
SUBCLUSTER_GRID_HOST_OperatingSystemVersion=Boron
SUBCLUSTER_GRID_HOST_ProcessorClockSpeed=2300
SUBCLUSTER_GRID_HOST_ProcessorModel=Xeon
SUBCLUSTER_GRID_HOST_ProcessorOtherDescription='Cores=3.7656,Benchmark=9.56-HEP-SPEC06'
SUBCLUSTER_GRID_HOST_ProcessorVendor=Intel
SUBCLUSTER_GRID_SUBCLUSTER_Name=grid.pp.rl.ac.uk
SUBCLUSTER_GRID_SUBCLUSTER_PhysicalCPUs=546
SUBCLUSTER_GRID_SUBCLUSTER_LogicalCPUs=2056
SUBCLUSTER_GRID_SUBCLUSTER_WNTmpDir=/scratch

Then it was a simple case of installing the rpms and running YAIM:

yum install emi-cluster
/opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-CLUSTER

At that point we seemed to have a working system, the BDII was running and queriable, I count connect to the gridftp server and it had set up expriment and cluster directories in /opt/edg/var/info/ and /opt/glite/var/info/.

Fine, next step was to rsync the contents of those directories from the torque server that then exports them to the CEs - well actually to /export/gridtags and /export/glitetags and symlink the previous locations to those. cfengine had already set the node up as a nfs server for me so exporting the new areas and updating the CEs to mount it from there was a matter of moments.

A quick check of the resource BDII looked fine so it was a simple matter to add the new source into the site bdii and tweak the static-file-CE.ldif file on the CreamCE to assign it to the new cluster.

One thing remained, when testing the gridftp server with uberftp* I'd noticed that I was not mapped to my usual pool account - not surprising as I had not mounted the site gridmapdir so it was using its local one. However, reasoning that the gridftp server was the same rpm as the one on the CreamCE that was using Argus for authentication and mapping I had a poke around on the CreamCE and in YAIM and tried installing the argus-gsi-pep-callout rpm and coping over /etc/grid-security/gsi-authz.conf and /etc/grid-security/gsi-pep-callout.conf from the CreamCE.

Another quick test with uberftp and yes, I am mapped to my normal pool account so it appears I have a Cluster Publisher with Argus integration working. That means the one things at the site not using Argus are the gLite CreamCE which will be replaced soon by another EMI one and dCache which will get banning from Argus when I update to the next Golden Release.

*uberftp heplnv146.pp.rl.ac.uk "ls /etc"

EMI CREAM

We have installed emi creamce at Oxford. It was quite straight forward and apparently everything was setup by yaim properly except that emi cream uses normal /etc/, /usr/ directories instead of /opt/glite. It uses just one repository for all packages, no more separate TORQUE_* repositories.
Jobs were running perfectly and all test jobs completed successfully. But it was only getting lhcbpilot jobs and after looking more closely it was the classic "GlueCEStateWaitingJobs: 444444" problem.

Drilling through many layer of wrapper it comes to this issue
/sbin/runuser -s /bin/sh ldap -c "diagnose -g --host=t2ce02.physics.ox.ac.uk"
ERROR: 'diagnose' failed
ERROR: user 'ldap' is not authorized to execute command 'diagnose'

I think this is the less documented part of emi creamce. In glite, slapd and bdii-update process was run by edguser but with emi it is run by ldap user.
Edited maui.cfg file
ADMIN3 edginfo rgma edguser ldap

It solved the problem as I was using our site wide maui.cfg file instead of default created by yaim. Just a heads-up if you are planning to install emi creamce

SAM to MyEGEE to finally MyEGI

I have updated to latest release of wlcg nagios to gridppnagios. It is a major release in the sense that it stopped configuring MyEGEE for portal and replaced it with MyEGI. MyEGEE would be there until I drop the myegee DB from gridppnagios machine but don't trust it anymore. I got two complain about MyEGEE within few hours of updating it so I can say that people are looking at it.
The other main change is that now Nagios Configuration Generator(NCG) is using Aggregated Topology Provider(ATP) instead of SAMDB to configure nagios. ATP is part of the ROC/NGI nagios package which aggregate information from GOCDB, Top BDII and VO feed etc and it is single authoritative information source with topology information. But it is the central ATP(http://grid-monitoring.cern.ch/atp) which is being used by all ROC/NGI's for topology configuration for the sake of uniformity and probably reliability . Old SAM infrastructure can now retire in peace.
So MyEGI, It is a kind of all in one (https://gridppnagios.physics.ox.ac.uk/myegi).
It has Gridmap, metric status, history and so on. Aesthetically MyEGEE was better but MyEGI has more functionality and if you are still not convince then check the comparison of SAM, MyEGEE and MyEGI here (https://tomtools.cern.ch/confluence/display/SAM/MyEGI+vs+MyEGEE+vs+SAM+Portal ).
MyEGI have very good search options and also has advanced filter so you can optimize your search and add URL to your bookmark for instance status of your site.
I just discovered two bugs and the irritating things is that it is showing advance date on history bar. So if you want to see the status at 11 March, check for 12 March !
A bug has been opened and hopefully it will be fixed soon
https://tomtools.cern.ch/jira/browse/SAM-1325
https://tomtools.cern.ch/jira/browse/SAM-1326

Going through the Argus Valley

Being an early adopter site for Argus, Oxford got one of the first MUPJ from ATLAS using glexec through Argus and it failed! although we were passing ops glexec tests for long.
Our understanding of Argus was that it must have a policy which authorize pilots to switch to a normal user, so I had a policy like this to authorize pilot for glexec

resource "http://authz-interop.org/xacml/resource/resource-type/wn" {
obligation "http://glite.org/xacml/obligation/local-environment-map" {
}

action "http://glite.org/xacml/action/execute" {
rule permit { pfqan="/ops/Role=pilot" }
rule permit { pfqan="/atlas/Role=pilot" }
rule permit { pfqan="/cms/Role=pilot" }
}
}


After discussion with Argus experts on mailing list, it turned out that when pilot framework ask glexec to switch user from pilot to the effective user, LCMAP PEP plugin send the proxy of effective user to ARGUS server for authorization and mapping. So Argus must have policy which authorize effective user also. I have changed policy to look like that

rule permit {pfqan = "/atlas/Role=pilot" }
rule permit {pfqan = "/atlas/Role=lcgadmin" }
rule permit {pfqan = "/atlas/Role=production" }
rule permit {pfqan = "/atlas/" }

It solved the problem. Doesn't it look like that every atlas user is allowed to switch identity through glexec ? As for as Argus is concerned, yes. But glexec configuration is defined at WN and only groups which are whitelisted at /opt/glite/etc/glexec.conf are allowed to use glexec, any other user trying glexec will be shot down at WN itself. By default only pilot users are whitelisted at WN.
So in nutshell, policies at Argus should resemble that of the CE.

glite-APEL Node

On Thursday 9th December we brought the new glite-APEL box on line.

The VM hosted by t2delltest, had already been installed and Kashif had installed the cert.

We ran apel on all the ce's and t2torque02 and then one last time on t2mon02.

Then reconfigured t2ce02 to point at the new apel box and ran apel on it. We saw new records created on the box. (After sorting some permissions issues, need to rerun yaim with each ce (and t2torque02) set in the site-info.def file. Each run did the magic to allow that node to write to the db. (FQDN's should be used).
We then changed the reference to t2mon02 to t2apel01 in the site-info.def file on pplxconfig and it propagated round the other nodes.
The first run that night failed due to a java out of memory error.
I tweaked the config file /opt/glite/etc/glite-apel-publisher/publisher-config-yaim.xml
to

150000
from the original 300000

All apel logfiles on all ce's , t2torque02 and t2apel01 now appear to be good.
Cristina can see records appearing at RAL.

The old mysql database from t2mon02 has been backed up in /data/sysadmin (pplxfs2)

Tracing a Grid Job (A recap)

Just in case we should forget how to trace a grid job I record some steps below.

For example you discover via a CMS SAM page you are failing some test (could equally be any other SAM page such as LHCb) , you click on the detailed out put and see a reference to the job id:
on t2ce05 contains the string: sOFavxScVKU-GbSYaCmx-A
on t2ce05
grep sOFavxScVKU-GbSYaCmx-A /opt/edg/var/gatekeeper/grid-jobmap_20100906
reveals the batch system job id: lrmsID=2998805.t2torque02.physics.ox.ac.uk
on the batch server t2torque02 in our case, either:
tracejob 2998805

or

grep 2998805 /var/spool/pbs/server_logs/20100909

The tracejob option is easier!

This will let you know which worker node ran the job. You can then have a look at it to check for full disks, memory faults etc or segfaults in the log files......

Now in reverse

A job is misbehaving on your node and you need to see who is running it.
The special case here is that its an ATLAS pilot job, this does not have a normal grid job id.

Get the PID from top, use
pstree -H pid
to highlight the processes parents.
(Use pstree -A -H pid if on an putty window on Windows)

This reveals which pbs job it is
eg 3020508.t2torque02.physics.ox.ac.uk

The job can be traced on the panda monitor, using the search facility on the LH toolbar.
This gives the job details including the users name. A GGUS ticket could then be raised against ATLAS asking for the user to be informed.

APEL on ngsce-test

APEL was failing on ngsce-test with the following error.

java.io.FileNotFoundException: /var/spool/pbs/server_priv/accounting/20090522 (Too many open files)

The solution was to type:
ulimit -n 10240

I've added this to the /opt/glite/bin/apel-pbs-log-parser script.

A fix is in test, so a new version of APEL will fix it.
see GGUS ticket
https://gus.fzk.de/ws/ticket_info.php?ticket=60674

Argus Server at Oxford

We finally managed to install Argus server at Oxford with messy workaround. Installation and configuration was reasonably ok, and once policy structure was clear then writing and loading policy was also easy. Details are here http://www.gridpp.ac.uk/wiki/Oxford.

The main issue was host certificate issued by UK CA which contains an "emailAddress" and supposedly this is depreciated year(s) ago and most developers assume that there is no "emailAddress" in host certificate. Although still it is a bug in Argus and hopefully would be resolved in next release.
So the workaround
By default pap-admin command uses host certificate in /etc/grid-security/ if started from root but since there is a problem with host certificate so I copied my personal certificate proxy from UI and started pap-admin using that proxy. Then added ACE
pap-admin ace
"/C=UK/O=eScience/OU=Oxford/L=OeSC/CN=t2argus02.physics.ox.ac.uk/OID.1.2.840.113549.1.9.1=lcg_manager@physics.ox.ac.uk" ALL
This workaround was suggested by Andrea Ceccanti

The only issue is that if you want to restart pap service then first remove ACE using remove-ace command, restart pap and then add ACE again.

Oxford's blanking panels

Having just read Stuart's ScotGrid blog post about cooling in the top of racks I thought I'd let you know about the panels we use.

We have been specifying that all empty racks slots should be filled by blanking panels since our 2007 purchase. The they used to use metal blanking panels.

These days they tend to supply the 1U APC plastic clip in panels, as can be seen in the RH rack in the photo.
These cost £25-£30 per pack of 10 but we managed to get a bulk (200) purchase in 2008 which worked out at about £1.69 each.

http://www.apc.com/resource/include/techspec_index.cfm?base_sku=AR8136BLK200

Jobs with analysis role

It started with a ticket from dzero about job failure at creamce at oxford. On investigation it was found that these jobs were coming with /dzero/users/Role=analysis/Capability=NULL and expectantly lcmaps failing with this error "no entry found for /dzero/users/Role=NULL/Capability=NULL ".
But the jobs from the same user were running on lcg-CE so on further investigation it turn out that lcmaps-voms plugins were failing on lcg-CE too but as per lcmaps policy it runs lcmaps-poolacount plugin after voms plugin failure and lcmaps-poolaccount uses individual DN mapping from grid-mapfile. So lcg-CE was mapping correctly to dzero pool account but through wrong procedure.
creamce don't use edg-mkgridmap file for creating grid-mapfile so no individual mapping is defined in grid-mapfile.
Solution was quite easy and we have to just define MAP_WILDCARDS=yes in vo.d/dzero and rerunning yaim created a slightly different grid-mapfile and groupmapfile with wild-cards.

dzero/Role=lcgadmin/Capability=NULL" dzerosgm
"/dzero/Role=lcgadmin" dzerosgm
"/dzero/Role=production/Capability=NULL" dzeroprd
"/dzero/Role=production" dzeroprd
"/dzero/*/Role=*" .dzero
"/dzero/*" .dzero
"/dzero/Role=NULL/Capability=NULL" .dzero
"/dzero" .dzero

So any job coming with different Role would be mapped to normal pool account.
The issue was discussed in this ticket https://savannah.cern.ch/bugs/index.php?26990

A week of upgrades for the RAL Tier 2 - Part 1 -The Network

Well it has been a long week at the RAL Tier 2. We've finally had our much postponed downtime to update our dCache installation (delayed once when one of the disk servers got a corrupt filesystem, then to avoid a CMS analysis test and finally to avoid an Atlas analysis test). The delays, however, did mean we could also include the long planned network upgrade in the downtime - this was probably a good thing.

So we had quite a programme of work for a five day downtime:

1. Replace the PNFS namespace in dCache with Chimera
2. Update dCache from 1.9.1 to the "Golden Release" 1.9.5
3. Install a new network switch and set up a 10Gb/s link between the two halves of our farm

Indeed, heading into work on Friday with neither dCache nor the network working, I thought I would be extending the downtime into the next week but by lunchtime things had improved and we were able to come out of the downtime on time at 5pm - although despite a full suit of "OK" SAM tests GridView still has us down until nearly eight o'clock.

Taking the last of the upgrades first: before last week we had the two halves of our farm in two different rooms. Each half of the farm has it's own Nortel 55XX network stack. Most of the storage is in the room known as Lab 8 in the R1 office building with a 10Gb/s connection to site Router A, whilst most of the compute nodes are in the Atlas lower machine room, A5Lower, with a 2x1Gb/s connection to Site Router A. That 2x1Gb/s connection between the storage and compute nodes was our main bottleneck - it would regularly run at over 99% capacity for days during Atlas Hammercloud tests.

The Plan was to install a Nortel 5650 switch into the stack in A5Lower then set up a direct 10Gb/s fibre link from there to Lab 8 - cutting out the 2x1GB/s link and Router A. That sounded fairly trivial and when I went down with Networking on Thursday afternoon to set it up I expected to be back in a hour to carry on struggling with our, at that time, broken dCache.

Due to cabling issues we had to re-order the switches in the stack and I also had to swap out a 5510 I had borrowed from the Tier 1 and replace it with a new one. So we broke up the current stack and tried to stack the 5650 with one of the 5510s. According to everything we had read they should have see each other, the 5650 should have downloaded an updated version of the firmware and software to the older 5510 and then the should have joined together as a single switch. But ours did not talk to each other.

Well possibly the version of the software on the 5510s was too old, so we went to each switch in turn, set it up with an IP address downloaded a new version on the firmware and software and restarted it.

By the end of Thursday - we were more-or-less back where we had started - we had a stack of 5510s (still without the 5650) .

On Friday morning Nick found a setting on the 5650 to allow "hybid stack mode" and suddenly everything worked.

We soon had all the correct VLANs set up and the two halves of our network were talking over the new fast link.

Backing up MySQL databases

Oxford have installed a simple script to backup the DPM mysql db once a day at 6am.
The script was loosely based on Glasgow's example here .

In order to restrict the file names produced to just 7, I've opted to use the current day rather than date.

[root@t2se01 ~]# cat /root/mysql-dump-pdg.pl
#!/usr/bin/perl
#
# Loosely based on the Glasgow script but simplified.
#
# Select the current day only as we want to have just seven unique file names which will be overwritten
# thus reducing the total backup size.


@weekDays = qw(Sunday Monday Tuesday Wednesday Thursday Friday Saturday);
($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
$theTime = "$weekDays[$dayOfWeek]";
#print $theTime;

$backup_dir="/var/lib/mysqldumps";
$mysql_user="root";
$mysql_pw_file="/root/mysql-pw";
$keep_days=7;


# Read mysql password
open(PW, $mysql_pw_file) || die "Failed to open password file $mysql_pw_file: $!\n";
$mysql_pw=;
chomp $mysql_pw;
close PW;

# Dump the db now
chdir $backup_dir || die "Failed to change to backup directory $backup_dir: $!\n";

system "/usr/bin/mysqldump --user=$mysql_user --password=$mysql_pw --opt --all-databases | gzip -c > mysql-dump-$theTime.sql.gz";
die "Mysql failed died with exit code $?\n" if $? != 0;

This is run by /etc/cron.d/mysql-dump
PATH=/sbin:/bin:/usr/sbin:/usr/bin
0 6 * * * root /root/mysql-dump-pdg.pl

So far it seems to work in testing!

Oxford Grid now SL5

All but one worker node on the Oxford Grid site has been reinstalled running SL5.
Currently these are served by one ce, t2ce05, but more will be added shortly to offer resilience.