You must login to use this action: subscribe.

Clear message

Reading and Writing data from the grid

This assumes you have access to the grid (certificate, ui and everything). If not, rather check the explanation page first.

Reading Data

There are (at least) two ways to discover the already stored files:

  1. via the lcg-ls command, which can take different types of protocols and doors etc. An example would be:

    lcg-ls  srm://dcache-se-desy.desy.de/pnfs/desy.de/ilc/tpc/
  2. Easier to use is the logical file catalog. You need to initialise a shell (choose your flavour) variable with the right one:
    export LFC_HOST=`lcg-infosites --vo ilc lfc`

And then use the type of command that follows below.
Top level in the lfc is:

lfc-ls -l /grid/ilc/tpc

Four years are present (always with the subdirectory t24/):

lfc-ls -l /grid/ilc/tpc/2008/t24
lfc-ls -l /grid/ilc/tpc/2009/t24
lfc-ls -l /grid/ilc/tpc/2010/t24
lfc-ls -l /grid/ilc/tpc/2011/t24

2008

lfc-ls -l /grid/ilc/tpc/2008/t24/micromegas

2009

lfc-ls /grid/ilc/tpc/2009/t24/micromegas/
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/june
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/august
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/november
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/lcio/december
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/june
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/august
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/november
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/misc/december
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/june
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/august
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/november
lfc-ls /grid/ilc/tpc/2009/t24/micromegas/rawdata/december

lfc-ls  /grid/ilc/tpc/2009/t24/bgem/
lfc-ls  /grid/ilc/tpc/2009/t24/bgem/lcio/sept **EMPTY**
lfc-ls  /grid/ilc/tpc/2009/t24/bgem/misc/sept **EMPTY**
lfc-ls  /grid/ilc/tpc/2009/t24/bgem/rawdata/sept

lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/
lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/lcio
lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/lcio/july **EMPTY**
lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/mapping
lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/mapping/july **EMPTY**
lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/rawdata
lfc-ls -l /grid/ilc/tpc/2009/t24/j-gem/rawdata/july

2010:

lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/
lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/lcio/
lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/lcio/sept **EMPTY**
lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/rawdata/
lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/rawdata/sept
lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/misc/
lfc-ls -l /grid/ilc/tpc/2010/t24/jgem/misc/sept **EMPTY**

2011:

lfc-ls -l /grid/ilc/tpc/2011/t24/desygem
lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/lcio
lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/lcio/june
lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/rawdata
lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/rawdata/june
lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/misc
lfc-ls -l /grid/ilc/tpc/2011/t24/desygem/misc/june

Copying files to your local computer

Simply use the lcg-copy command once you know the filename. An example that won't work is (the file doesn't exist, replace to your liking):

  lcg-cp -v --vo ilc lfn:/grid/ilc/tpc/2008/tb-desy/testrun1/file1 file:$PWD/file1

Writing data to the grid storage

Please be aware that you know what you are doing! Please test at a safe location before testing with the real paths!
Writing privileges need and additional pre-requisite: you need a valid grid certificate with the sufficient rights to write to the right storage. To my understanding you need to have the role /ilc/tpc/production; check with the VOMS server (with an imported, valid certificate in your browser) at: https://grid-voms.desy.de:8443/voms/ilc
If you don't have it yet, but would like to write data simply request the sufficient rights at the same place. Unfortunately this is a two-step procedure, since you first need to be a member of /ilc/tpc before you can request the additional /production role. Usually the admins are very quick if your name and affiliation are known. If you are impatient, you could try to send a friendly mail to speed things up.

Initialisation is similar to the usual setup, but roles are acquired only when the requested:

  1. Obtain proxy with write rights:
      voms-proxy-init -debug -voms ilc:/ilc/tpc/Role=production
  2. (optional: see if it worked; especially with the role)
      voms-proxy-info -debug -all
  3. To make life easier, use the logical file catalog (see also above about reading the data from the grid).
      export LFC_HOST=`lcg-infosites --vo ilc lfc` 
  4. If your directory doesn't yet exist in the logical file catalog, you have to create it. According to the current scheme, this would read like:
      lfc-mkdir -p /grid/ilc/tpc/$YEAR/t24/$MODULETYPE/$FILETYPE/$MONTH
    The current scheme, as evolved over time is as follows:

    YEAR

    [2008|2009|2010|2011|etc.]

    MODULETYPE

    [jgem|bgem|micromegas|desygem|...]

    FILETYPE

    [lcio|rawdata|misc|...]

    MONTH

    [january|february|march|april|may|june|july|etc.]

  5. Suppose that the file you want to write to grid storage is called foobar.slcio (in your local directory), then the right command is lcg-cr (copy and register) -- which takes three arguments and at least one option. An example would be (on a single line, the slashes indicate line breaks):

      lcg-cr --verbose --vo ilc -l lfn:/grid/ilc/tpc/2014/t24/test/foobar.slcio -d srm://dcache-se-desy.desy.de/pnfs/desy.de/ilc/tpc/2014/t24/test/foobar.slcio file:$PWD/foobar.slcio
    In a more formal way:
         lcg-cr --verbose --vo ilc -l lfn:${LFN} -d srm://dcache-se-desy.desy.de/${DEST} file:${SOURCE}
    The options are:
    • --verbose creates verbose output

    • --vo sets the VO explicitly

    • -l specifies the logical filename ${LFN}

    • -d specifies the destination ${DEST} (in this case via the dcache door at DESY)

    • file: specifies the proctocol (none in this case, meaning POSIX file access) and the filename ${SOURCE} (including the path!)

If the copy command works you'll see a file transfer progress bar and get a summary in the end. Please check the file catalogue if the file is really there (see file reading) and possibly also check a file out.

Good luck!

MarlinTPC: HowToReadAndWriteDataFromGrid (last edited 2012-08-30 17:37:20 by ChristophRosemann)