Skip to content

Migration to JAliEn

The following guide outlines key points for end users to consider when migrating to the new JAliEn.

New interactive shells

There are now two new interactive clients available to the user:

  • The alien.py Python shell is part of the xjalienfs alidist package and comes with additional command line utilities such as alien_ls, alien_find and alien_cp. Any shell command can be run as a standalone one shot command, for instance alien.py ls -l /alice. More details on running alien.py-based commands can be found here.

  • The jalien Java shell comes with an authentication and caching background agent called JBox. When you start the shell, JBox runs in the background if it doesn’t run already. Other clients, such as alien.py can connect to the local JBox instance as well. The JBox process will automatically go away if the Grid connection is not active for an extended period.

Client authentication

All clients use authentication technology that is built on the tried and trusted AliEn authentication mechanism with password protected key and certificate files. You can continue using your current AliEn credentials and in fact all components expect to find them in ~/.globus/user{cert,key}.pem as per usual. The only change is that running alien-token-init is no longer required.

Example usage: alien.py

The shell will request the password for userkey.pem and create a token, or will use previously generated token.
See Authentication section for more information about JAliEn tokens.

The following example demonstrates commands for a simple file transfer using the alien.py client:

user@lxplus ~> /cvmfs/alice.cern.ch/bin/alienv enter AliPhysics/vAN-20191214_JALIEN-1
[AliPhysics/vAN-20191214_JALIEN-1] ~ > alien.py
Welcome to the ALICE GRID
support mail: adrian.sevcenco@cern.ch

AliEn[user]:/alice/cern.ch/user/u/user/ >ls -l minimal_example
-rwxr-xr-x     user   user           72 Dec 20 15:27    minimal.jdl
-rwxr-xr-x     user   user           95 Dec 20 15:09    minimal.jdl_20191220_151131
-rwxr-xr-x     user   user           65 Dec 20 15:11    minimal.jdl_20191220_152735
-rwxr-xr-x     user   user           31 Dec 20 15:09    payload.sh

AliEn[user]:/alice/cern.ch/user/u/user/ >cp minimal_example/minimal.jdl file:///tmp/local_file.jdl
jobID: 1/1 >>> Start
jobID: 1/1 >>> STATUS: OK ; SPEED = 14.24 bytes/s ; MESSAGE: [SUCCESS]

Changes in file locations

To ensure backward compatibility, make sure to pass the prefix for both source and destination of a cp operation:

  • For the old aliensh client remote files were prefixed with alien://
  • JAliEn alien.py now prefixes for local files with file://

AliPhysics and JAliEn

The switch to the new JAliEn services should mostly be transparent for the end user.

There is a separate AliPhysics build that comes with the new JAliEn features instead of the legacy AliEn. The user should select the appropriate JAliEn build using alienv entercommand.

New builds of AliPhysics

The AliPhysics with JAliEn builds deployed to CVMFS are clearly distinguishable by the suffix _JALIEN in their names.

For example to get a list of available packages: 

~$ /cvmfs/alice.cern.ch/bin/alienv q | grep -i aliphysics | grep -i jalien
...
VO_ALICE@AliPhysics::vAN-20201114_JALIEN-1
VO_ALICE@AliPhysics::vAN-20201115_JALIEN-1

The JAliEn plugin is loaded automatically and from the ROOT shell there is no change in the interface with respect to the old one. The commands discussed further work for both JAliEn-ROOT (TJAlien) and AliEn-ROOT-Legacy (TAlien).

Maintain Compatibility

In order to preserve compatibility don’t use TJAlien and TAlien classes explicitly.
Usage of gSystem->Exec(“alien_ls *”) style of commands is also highly discouraged.

The API calls should be used instead. Any Grid command can be used with this interface and the results are returned as TGridResult objects, also avoiding any subsequent parsing steps of command outputs. The following macro example works for both JAliEn and AliEn:

TGrid::Connect(“alien://”)
gGrid->Ls(“ls /alice/data”)
TGridResult *res = gGrid->Command(“find /some/query pattern”)

ROOT5 vs ROOT6

The AliPhysics JAliEn daily builds are available with ROOT6 only. The AliPhysics with JAliEn plugin are published in CVMFS with a _JALIEN suffix, as mentioned above. All O2 builds with Grid features enabled come with JAliEn only. The ROOT5 builds are compiled with AliEn-ROOT-Legacy only.

The end goal is that with any AliPhysics version, TGrid::Connect() will work as usual and you will have alien_ls and the rest of the shell commands available in a similar way. Please note that the two shells (alien.py vs aliensh) have different syntax for certain commands and that alien-token-init is required for the AliEn legacy only.

Why now / who is affected

The AliEn-ROOT-Legacy requires some very old dependencies. Some of them are no longer available, for instance OpenSSL 1.0.x on macOS. Moving to JAliEn lets us upgrade those dependencies (the most recent OpenSSL versions, XRootD upgraded from 3.x to 4). This resolves the issue with running the analysis software on macOS and brings all the benefits coming with up to date software. The new components are also easier to develop and maintain so you can expect faster bug fixes and more features. On top of that, the switch to JAliEn is a prerequisite to sustain the increased workload during the Run 3 activities.

Release schedule

Here’s the general rule of thumb if you are not sure if you are connected to the Grid using AliEn or JAliEn:

  • ROOT 5 builds always come with AliEn-ROOT-Legacy
  • Local ROOT 6 builds on your laptop will from now on come with JAliEn-ROOT
  • Any O2 build with Grid features enabled comes with JAliEn-ROOT
  • AliPhysics with ROOT 6 deployed to CVMFS and used by the trains continues to run with AliEn-ROOT-Legacy until next AliRoot release (most likely this will be v5-09-54)

How to get legacy aliensh back

The CVMFS releases are still deployed with the legacy aliensh. Before the alidist update only builds with --defaults jalien included the JAliEn features. After the alidist update there is currently no easy way to switch back to AliPhysics + ROOT6 + AliEn-ROOT-Legacy.

However, if you need the shell only, you can still try building the AliEn-ROOT-Legacy package with: aliBuild build AliEn-ROOT-Legacy. Keep in mind that this doesn’t work on macOS any more.

ChangeLog

  • Interactive shells: jalien (Java) and alien.py (Python). Both can be used as libraries to create a custom Grid tool, with a starting point in Python here.
  • Token: no more alien-token-init, it’s handled automatically in the background
  • Improved communication protocol from end client to the API servers. WebSockets can now be easily used in other applications to interface to the Grid
  • JAliEn central servers are easier to scale horizontally
  • ROOT Grid plugin changes should be completely transparent to the end user, it’s just a different shared library that is loaded (libJAliEnROOT.so instead of libAliEnROOTLegacy.so).
  • The alidist build recipes (AliPhysics and O2)
  • XRootD version updated from 3.x branch to 4.11.1
  • ROOT 5 is not supported anymore. Please switch to ROOT 6.