Skip to content

Files operations

This chapter will refer to operations with/on actual files (not with/on metadata)
This implies the downloading and/or uploading of files

Inspect LFNs

  • cat/less/more LFN
    • the LFN will be downloaded to a temporary file and passed to cat/less/more

Edit LFNs

The LFN will be downloaded to a temporary file that will be passed to an editor. If the md5 sum is different, the initial lfn will be renamed to a backup name, and the current edited file will be uploaded as the same LFN. N.B. for now, the number of replicas and their locations will not be kept. Specify the specs when editing, these will be applied on uploaded file.

  • vi/vim/nano/mcedit LFN@SPECS
    • @SPECS is optional, it will specify the specifications of uploaded file
  • edit/sensible-editor LFN@SPECS
    • this require the existence of $EDITOR env var that will be used as editing application

Transfer files

Conventions

  • local path specification require the prefix file: (or the old one of file://)
  • remote path does not require any prefix (but it can have the alien: or alien:// prefix)
  • source argument: if ended by / the content of source will be copied, otherwise the source will be copied.
alien.py cp -h
Command format is of the form of (with the strict order of arguments):
cp <options> src dst
or
cp <options> -input input_file
where src|dst are local files if prefixed with file:// or file: or grid files otherwise
and -input argument is a file with >src dst< pairs
after each src,dst can be added comma separated specifiers in the form of: @disk:N,SE1,SE2,!SE3
where disk selects the number of replicas and the following specifiers add (or remove) storage endpoints from the received list
options are the following :
-h : print help
-f : replace destination file (if destination is local it will be replaced only if integrity check fails)
-cksum : check hash sum of the file; for downloads the central catalogue md5 will be verified
-S <aditional streams> : uses num additional parallel streams to do the transfer. (max = 15)
-chunks <nr chunks> : number of chunks that should be requested in parallel
-chunksz <bytes> : chunk size (bytes)
-T <nr_copy_jobs> : number of parralel copy jobs from a set (for recursive copy); defaults to 8 for downloads
-timeout <seconds> : the job will fail if did not finish in this nr of seconds
-retry <times> : retry N times the copy process if failed
-ratethreshold <bytes/s> : fail the job if the speed is lower than specified bytes/s
-noxrdzip: circumvent the XRootD mechanism of zip member copy and download the archive and locally extract the intended member.
N.B.!!! for recursive copy (all files) the same archive will be downloaded for each member.
If there are problems with native XRootD zip mechanism, download only the zip archive and locally extract the contents

for the recursive copy of directories the following options (of the find command) can be used:
-glob <globbing pattern> : this is the usual AliEn globbing format; N.B. this is NOT a REGEX!!! defaults to all "*"
-select <pattern> : select only these files to be copied; N.B. this is a REGEX applied to full path!!!
-name <pattern> : select only these files to be copied; N.B. this is a REGEX applied to a directory or file name!!!
-name <verb>_string : where verb = begin|contain|ends|ext and string is the text selection criteria.
verbs are aditive : -name begin_myf_contain_run1_ends_bla_ext_root
N.B. the text to be filtered cannont have underline <_> within!!!
-parent <parent depth> : in destination use this <parent depth> to add to destination ; defaults to 0
-a : copy also the hidden files .* (for recursive copy)
-j <queue_id> : select only the files created by the job with <queue_id>  (for recursive copy)
-l <count> : copy only <count> nr of files (for recursive copy)
-o <offset> : skip first <offset> files found in the src directory (for recursive copy)

-parent N

will keep in the destination N components; for replication of alien paths use -parent 99 This will create the same paths on local machine as in AliEn; (local /alice should be accesible)

-glob pattern_string

standard (or legacy) selection rules. N.B.!! selections works only on valid path names

-select | -name regex_pattern

regex (PCRE) based selections.
DO NOT FORGET TO ESCAPE THE SELECTION RULES!
Usually this is a sore point in usage: the interpreter will preprocess the command line before passing the control to the executable