Manage VOBox Proxy

For the Impatient!

In all of the following, commands prompted by VO-Box> are to be issued once logged on the VO-Box itself, while LCG-UI> means a generic gLite/EMI User Interface:

• Register a fresh proxy on the VO-Box:

  LCG-UI> voms-proxy-init --voms alice:/alice/Role=lcgadmin
  
  LCG-UI> export GT_PROXY_MODE=rfc         # note: will only work with a WLCG VOBOX
  
  LCG-UI> myproxy-init -s myproxy.cern.ch -d -n -t 48 -c 3000
  
  LCG-UI> gsissh -p 1975 your-VOBOX
  
  VO-Box> vobox-proxy register -t 48
  

• Define the correct proxy in your environment if you need to submit test jobs or get job outputs manually (replace the dots with the long file name of your registered proxy):

  VO-Box> export X509_USER_PROXY=/var/lib/vobox/alice/proxy_repository/.....
  

• The JAliEn services must themselves define the correct proxy to be used, e.g. as follows:

  VO-Box> cat ~/.alien/config/CE.env
  # CE environment bootstrap
  
  export X509_USER_PROXY=$(
  ls -t /var/lib/vobox/alice/proxy_repository/*lcgadmin | sed q
  )
  VO-Box> cat ~/.alien/config/ml.env
  # MonaLisa environment bootstrap
  
  export X509_USER_PROXY=$(
  ls -t /var/lib/vobox/alice/proxy_repository/*lcgadmin | sed q
  )
  export MYPROXY_SERVER=myproxy.cern.ch
  

• Restart the JAliEn services to ensure the new proxy is used:

  VO-Box> /cvmfs/alice.cern.ch/scripts/vobox/jalien-vobox.sh restart
  

Involved Proxies

The VO-Box uses several proxies for different tasks. Apart from the proxy used by the VO-Box administrator to log in on the machine, there are two more resident on the machine and one remotely stored in a server. The machinery is better understood by first describing sequentially the operations, with details to follow.

1. The user generates a proxy on some LCG UI, which he will use to gsissh to the VO-Box
2. The user stores a long-lived (e.g. one month) myProxy on myproxy.cern.ch.
3. From the VO-Box, the user registers the login proxy to the VO-Box Proxy Renewal Service. The proxy thus generated will be called, in the following, the user proxy. Since gLite 3.1, this needs do be a VOMS-extended proxy (i.e. a proxy that carries extra VO-specific information).
4. The VO-Box Proxy Renewal Service keeps the user proxy alive by periodically getting a new one from the MyProxy server. To authenticate to the latter, it uses its copy of the machine proxy.
5. The JAliEn CE running on the VO-Box uses the user proxy to submit jobs to the site CE(s). Just before submitting a bunch of job agents, the JAliEn CE will itself check lifetime of the user proxy and try to restart the proxy renewal daemon if the duration is significantly less than 48h.

Summary

The proxy certificates involved in the management of an LCG VO-Box for ALICE are as follows:

• the login proxy, which is used by the manager to login (via gsissh) to the VO-Box
• the myProxy registered on the MyProxy server (myproxy.cern.ch)
• the machine proxy, which is used by the VO-Box to authenticate to the myproxy server
• the user proxy (a VOMS proxy), which is used by JAliEn to submit jobs to the site CE(s).

Proxies can be examined in a few ways:

• By using Globus tools (possibly wrapped in JAliEn commands). If you don't specify the proxy file name, the $X509_USER_PROXY environment variable will be used. By default (i.e. if neither is specified) proxies are stored in /tmp/x509up_uXXXX, where XXXX is the local numeric userid of the user owning the proxy.

  LCG-UI> grid-proxy-info [-f ]
  

• By directly using openssl tools (man openssl for more detailed help, probably more than you will ever want to know), e.g.:

  VO-Box> openssl x509 -noout -text -in .....
  

• However, in order to show also the VOMS extensions, you'll need a different command:

  LCG-UI> voms-proxy-info -all [-f ]
  

The login proxy

This is a plain user proxy that the VO-Box administrator uses to log in on the machine, via:

LCG-UI> voms-proxy-init --voms alice:/alice/Role=lcgadmin
LCG-UI> gsissh -p 1975 your-VOBOX

On the VO-Box, upon login the $X509_USER_PROXY variable will point to it, i.e. to a file in the /tmp directory called something like /tmp/x509up_p17069.fileuEDDS2.1. Please note that this proxy is not the one used to start services or submit jobs, nor it is in any way automatically managed. If this proxy expires, the JAliEn services should take no exception (they shouldn't even notice).

You will need $X509_USER_PROXY to point to a valid user proxy in order to make the JAliEn services work, as documented above.

The myProxy

The myProxy is a special long-lived proxy that resides on a remote server (in our instance, myproxy.cern.ch) and is used to obtain shorter-lived delegated proxies by a service. For details, please check the relevant section ("Advanced proxy management") in the LCG User Guide and possibly the original MyProxy literature.

To generate a MyProxy to be subsequently used by the VO-Box, from an LCG UI on which the user's certificate and key are installed issue the following command:

LCG-UI>  export GT_PROXY_MODE=rfc     # note: will only work with a WLCG VOBOX

LCG-UI> myproxy-init -s myproxy.cern.ch -d -n -t 48 -c 3000

The command lines options are important, and the meaning is as follows:

Option	Description
-d	Use the user's certificate subject as username
-s myproxy.cern.ch	Use this particular MyProxy server. This is the one all VO-Boxes are registered to.
-n	Allow retrieval of a proxy without a password
-c 3000	Lifetime, in hours, of the MyProxy stored in the server. This value (4 months) is a suggestion: the actual value can be as large as the remaining number of hours in the lifetime of your certificate. The proxy renewal daemon running on the VO-Box can warn you (by email) some time before the expiration date (see below).
-t 48	The maximum lifetime (in hours) of derived proxies

Note

The default value for the -t option (12 hours) is too short for our application, so it is important not to forget this option. One fishy issue with this is that if you forget the option and the system tries to obtain a longer proxy, no error message will be issued and the derived proxy will just be of the maximum allowable length. There is a check in the JAliEn code, so one way to diagnose this problem is to check the CE log file ~/ALICE/alien-logs/CE.log.0 and look for something like this:

Dec 31 21:30:23  info   Proxy timeleft is 43188 (threshold is 165600)

It should rather look like this instead:

Dec 31 21:30:23  info   Proxy timeleft is 172090 (threshold is 165600)

This is done by checking the actual remaining lifetime of the received proxy, so please disregard differences of a few seconds. To fix the problem, generate again a fresh MyProxy with a longer value for -t. Unfortunately, there is apparently no way for querying the MyProxy and get the value that was used for -t, except trying to request a very long proxy and see what comes back.

The MyProxy server can be queried (to obtain e.g. the lifetime of the MyProxy) by issuing:

LCG-UI> myproxy-info -s myproxy.cern.ch -d

The user proxy and the Proxy Renewal Service

This is the most important proxy, since it is the one used by JAliEn to start the services and to submit jobs to the site CE(s). It is generated by registering the login proxy to a database, which is managed by the VO-Box Proxy Renewal Service (PRS) which will take care of renewing it:

VO-Box> vobox-proxy register -t 48

There's a couple more options to vobox-proxy other that can be useful:

Option	Description
--proxy-safe 3600	This tells the PRS to warn you 3600 seconds (one hour) before the user proxy expires. Since the PRS renews it more often, you should never get such a message; if you get one, it probably means the VOBOX has some problem.
--myproxy-safe 864000	Tells the PRS to warn you 10 days before the long-lived proxy stored on the server expires. If you get such a message, you are supposed to generate a fresh one by myproxy-init.
--email your-address	The email address for the alert messages. Please note that site firewall rules in most cases will prevent mail messages from the VO-Box to be sent.

Upon such registration, in /var/lib/vobox/alice/proxy_repository a delegated proxy will be put with a file name that matches the DN of the user. This proxy is periodically renewed by the PRS, each time obtaining a fresh proxy with the requested duration, by default 12h. This being too short for most of ALICE jobs, please use the -t 48 arguments of the vobox-proxy register command, to allow the PRS to handle all renewals.

The PRS database can be queried with the following command (see vobox-proxy --help for more options):

VO-Box> vobox-proxy query -dn all

The script that starts/stops the Proxy Renewal Service is /etc/init.d/alice-box-proxyrenewal. This should be already in the init.d services list, so you should not need to do anything.

When you log in on the VO-Box, the $X509_USER_PROXY points to your login proxy. Please define in ~/.alien/config/{CE,ml}.env the correct proxy to be used before (re)starting JAliEn services, as documented above:

VO-Box> /cvmfs/alice.cern.ch/scripts/vobox/jalien-vobox.sh restart

The machine proxy

This (last!) proxy is used by the Proxy Renewal Service running on the VO-Box to authenticate with the MyProxy server. This is kept in the /var/lib/vobox/alice directory:

[alicesgm@alibox2 ~]$ grid-proxy-info -f /var/lib/vobox/alice/renewal-proxy.pem


subject  : /C=IT/O=INFN/OU=Host/L=Torino/CN=alibox2.to.infn.it/CN=441535218
issuer   : /C=IT/O=INFN/OU=Host/L=Torino/CN=alibox2.to.infn.it
identity : /C=IT/O=INFN/OU=Host/L=Torino/CN=alibox2.to.infn.it
type     : RFC 3820 compliant impersonation proxy
strength : 1024 bits
path     : /var/lib/vobox/alice/renewal-proxy.pem
timeleft : 21:47:13

As you can see, it's not a user proxy, but it's periodically generated from the host certificate /etc/grid-security/hostcert.pem (which is, of course, root-owned) by a cron job that calls the following command:

~$ /etc/init.d/alice-box-proxyrenewal proxy

Summary

The /etc/init.d/alice-box-proxyrenewal script has two different functions:

• It is called in the /etc/rc.d sequence with [start|stop] options. In this mode, it will start (or stop) the proxy renewal service and go through the services (if any) in /var/lib/vobox/alice/[start|stop] and start/stop them.
• It is then called by a cron job (check /etc/cron.d/alice-box-proxyrenewal) with the argument proxy. In this mode, it renews the machine proxy /var/lib/vobox/alice/renewal-proxy.pem.

Since the alicesgm user has no root privileges on the VO-Box, if this proxy expires you need to ask the site manager (or whoever manages the VO-Box) to check what's wrong and generate a new one.