ISIS Instrument Data Cache 2023

Brief

We are still working towards disconnecting the Archive from the Workspaces and replacing it with the Instrument Data Cache. This is so we can make the service adhere to the ISIS Data Policy. This is just a change to the Workspaces and will not impact the Archive in any other way. It will still be accessible from Workspaces, though it will require you to input your Federal Credentials.

The Instrument Data Cache can be accessed from all workspaces from the applications menu under the Data section. Each experiment has its own directory named after the RB Number and contains the RAW and NeXus files generated by the instrument. Data assigned an RB Number that is not in the Scheduler will be put in a directory called “unknown” under the respective instrument.

We currently store ~3 years of instrument data, approximately 800TB.

If you need to access older data, you can download it from DataGateway.

Introduction

The ISIS Instrument Data Cache is an alternative to the ISIS Archive on IDAaaS. The ISIS Data Cache is a store of instrument data located on SCD’s Ceph storage cluster that contains data copied directly from instrument PCs as it is produced.

We have also copied the last 3 years of data from the Archive onto the Data Cache so that you will have access to past data.

Why?

Data in the cache is access-restricted so that only members of the relevant experiment group and instrument scientists will have access to the data, whilst nobody else will be able to access the data.

Data Cache access is read-only. This brings us into conformity with the ISIS data policy, which specifies an embargo period of 3 years on experiment data.

How frequently is the data updated?

We are running a service called the Cache Updater which will check periodically (ideally at least every 10 seconds) to see if there are any new files to be copied from instrument PCs.

New data from instrument PCs should appear in the Data Cache in under 5 minutes depending on the file size.

What data are captured?

All run files should be captured, for RB pre-fixed experiments (listed below) will be in RB folders, others will be in a folder called Unknown.

How to access the Data Cache?

Please access the shortcut under Data on the Applications menu called “Instrument Data Cache” at your workspaces.

This will take you to the data cache via the file browser GUI. For command line access, there is a symlink at /data/cache which points to the cache directory.

Where to find the data?

The data cache has the following directory structure: INSTRUMENT/YEAR/RB NUMBER

For example, to find instrument data from the 2022 WISH experiment RB2200001-1, you would go to the directory WISH/2022/RB2200001-1 in the data cache.

All members of the RB2200001 experiment and all WISH instrument scientists should have access to the data in this directory.

There will also be an “unknown” folder under each year, e.g. WISH/2022/unknown. All run files that could not be matched to an RB number in the Experiment Scheduler will be placed in a directory in the unknown folder.

For example, if WISH instrument data that cannot be matched to an RB is copied on the 22nd March 2022, it will be copied to the folder WISH/2022/unknown/2022-03-22.

We get the RB number from the .nxs (nexus) file for the run, so please make sure that correct RB number are input for their experiment so that it can be matched to an experiment record on the Scheduler and thus be placed in the correct folder.

Monitoring and Alerting

We have monitoring and alerting system set up so that we should be alerted if the Cache Updater service goes down. Alerts will be sent and responded to both during the day and out-ofhours for this service so that disruption to instrument data access is minimized.

Support

Get in touch if you have any queries: supportanalysis@stfc.ac.uk.