This page describes how to handle data stored on grid storage.
voms-proxy-init --voms MYEXPERIMENT
Where MYEXPERIMENT is one of 'cms, icecube, beapps'
GFAL is a wrapper around the latest grid commands. Learning to use it means that whatever middleware requires to be used in the future, you don't need to learn new commands (like srm, lcg, etc)
If you want more information on the options that can be used, please use man gfal-command !
Here are all the commands that can be used:
- gfal-ls: get information on a file
- gfal-mkdir: remove a directory
- gfal-rm: removes a file. To remove an entire directory, use -r
- gfal-copy: copy files.
There are 2 types of file url:
- Distant files: their url is of the type srm://<name_of_server>:<port>/some/path, eg for IIHE:
- Local files: their url is of the type file://path_of_the_file, eg for IIHE:
- To get a list of all distant urls for all the Storage Elements, one can do:
lcg-infosites --vo cms se
- To list the contents of a directory /pnfs/iihe/cms :
- To create a directory:
- copy file from local disk to remote server
gfal-copy file:///user/$USER/MyFile.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/
- To copy a file from remote server to our Storage Element:
gfal-copy srm://srm-eoscms.cern.ch:8443/srm/v2/server?SFN=/eos/cms/store/group/comm_trigger/L1TrackTrigger/BE5D_620_SLHC6/singleMu/NoPU/reDIGI_SLHC6-TrackTrigger_muon_pgun-0499.root srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/odevroed/eosTransfer.root
- To delete a file on remote server
- To remove a directory and its entire content on remote server ?!? not working for now ?):
gfal-rm -r srm://maite.iihe.ac.be:8443/pnfs/iihe/cms/store/user/$USER/NewDir
Bulk file transfers
There is an elegant way to run gfal-copy through several files. This is done using the --from-file option.
gfal-copy -f --from-file files.txt file://$PWD
where files.txt is a file where every line is a source in srm url syntax.
Make some tests with one line in datafile and make sure the srm url is OK for both source and destination before running over several files.
Copy directories from and to pnfs within the IIHE
A script to copy full directories to and from pnfs exists on the slc6 UI's:
copyDirectoryPnfs.py Move all files in a directory to or from pnfs This script assumes that you copy within the IIHE The script does not do recursive copying Make sure you have a valid proxy, made with voms-proxy-init --voms cms:/cms/becms Mandatory options: --in= : directory to copy from --out= : directory to copy to Both directories need to be complete (i.e. including the /pnfs or /user part example: copyDirectoryPnfs.py --out=/user/odevroed/newfile --in=/pnfs/iihe/cms/store/user/odevroed/newdir Optional: -h, --help : print this help message
Direct dcache access to files is only possible if the software supports it. PNFS (the directory structure seen under /pnfs/) is NOT a real filesystem: it is an immutable filesystem, and is mounted read-only.
- Therefore, here is a list of the commands that work:
- And replacement commands:
- dccp: to copy files from /pnfs to local disk. Example:
- To open files using root, use eg
When reading out the rootfiles is rather slow or it doesn't work at all and nothing is wrong with the root file (e.g. in an interactive analysis on beo or msa) you can increase your dCache readahead buffer. Don't make the buffer larger than 50MB! To enlarge the buffer set this in you environment :
setenv DCACHE_RAHEAD 1 setenv DCACHE_RA_BUFFER 50000000
export DCACHE_RAHEAD=true export DCACHE_RA_BUFFER=50000000
See the dChache fanpage for further reading.