Hi Pascal and MPB,

 

I hope you had a good Christmas break. As an estimate I would say that the “Nexus file handling consolidation” would take 2 releases to complete (at least), so I would say its timescale matches more with it being an Epic rather than BAU.

 

Ok, just to keep track of the benchmarking we want to do:

- Loading large files

- Loading a large number of small files

- Loading partial files

 

The key stakeholders I have written down at the moment are:
- Pascal

- Dirk Honecker

- A NR representative

 

Who would be a good representative for NR? Do we want any other key stakeholders?

 

Once we have agreed on all key stakeholders, the next thing will be for me to ask these stakeholders for data that will allow me to benchmark for the cases listed above.

 

Thanks,

Rob

 

From: Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk>
Sent: Wednesday, December 20, 2023 10:51 AM
To: Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk>
Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>; mpb@mantidproject.org
Subject: Re: [Mpb] Re: Nexus loading epic - stakeholders needed

 

Hi Rob,

 

Thanks for the reply. It indeed sounds like a bit of a mess we have put ourselves in.

I am just not clear whether this warrants an EPIC or if this should be a BAU item but maybe I still misunderstand the distinction.

Another test I can think of but didnt specifically write is the speed to load partial files.

ON WISH for instance, loading just the monitors (ie just 5 spectra out of 200000) still takes a long time and this is something you might want to do across many many files to check stability.

Re. Benchmarking, at the very least, we should compare with scipp (I have not used it myself but I have heard it is faster). There are also many libraries that are found in the X-ray world (some can be downloaded from SOLEIL synchrotron website if I recall correctly) that are designed to access lots of data very quickly. I think it could also be worth having a chat with Tristan as he has done some work on this, including recently, for the disordered materials data analysis software. 

 

Hope this helps,

 

P

 

Sent from my iPad



On 20 Dec 2023, at 09:55, Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk> wrote:



Hi Pascal,

 

The enabler epic is called “Nexus file handling consolidation”. After speaking with Martyn, its become clear that we currently access nexus files in three different ways across the Mantid code base:

 

1.   Using the HDF5 API directly

2.   Using the NeXus API directly

3.   Using our own internal wrapper around the NeXus API

 

This spread of mechanisms presents several challenges for developers including increased maintenance, a requirement for more domain knowledge, and repeated bugs in near duplicate code across each implementation. Furthermore, our own internal wrapper around the NeXus API causes performance bottlenecks, and it is also believed that the NeXus API is less performant than the HDF5 API.

 

Given that all modern files are HDF5 (with the possible exception of the HDF4 files created by Muon?), we wish to consolidate these mechanisms so that we only use the HDF5 API and a more performant internal wrapper around the HDF5 API. This will make it easier to identify other performance bottlenecks in the loading process, and might also provide a performance improvement in and of itself.

 

So in short, this enabler epic is related to the “Nexus load optimization” epic. It should make it easier for developers to identify bottlenecks, and may also provide a performance improvement.

 

What are the available routines that we want to benchmark against?

 

Thanks,

Rob

 

From: Pascal Manuel - STFC UKRI <pascal.manuel@stfc.ac.uk>
Sent: Tuesday, December 19, 2023 5:26 PM
To: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>
Cc: King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>; mpb@mantidproject.org
Subject: [Mpb] Re: Nexus loading epic - stakeholders needed

 

Hi Sarah,

 

I think the project is simple enough; improve loading speed so I am not sure we should really absolutely every group represented there but I am happy to take the hit for diffraction. The test will be in speed improvement and it should be done with loading a certain number of big files and also loading a big number of small files to be representative of everyone’s need.

There are a few available routines that seem to do a quicker job at the moment that we should benchmark against. 

What is the enabler project?

 

Best 

 

P

 

Sent from my iPhone




On 19 Dec 2023, at 16:55, Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk> wrote:



Hi Steve,

 

That’s great thanks. Should we have someone from Reflectometry too or is Dirk ok to represent them too?

 

Thanks

 

Sarah

 

From: Stephen King - STFC UKRI <stephen.king@stfc.ac.uk>
Sent: 19 December 2023 16:45
To: mpb@mantidproject.org
Subject: [Mpb] Re: Nexus loading epic - stakeholders needed

 

Hi Sarah,

 

Dirk Honecker has volunteered for SANS.

 

Thanks,
Steve

 

From: Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk>
Sent: 18 December 2023 16:22
To: mpb@mantidproject.org
Subject: [Mpb] Nexus loading epic - stakeholders needed

 

Dear all,

 

We are progressing with drafting two epics to improve Nexus loading (main project and an enabler project) with the hope of putting them up for review at the next meeting in February. For the main epic we need key stakeholders from the different science groups. At the moment Rob has listed board members. However I  wondered if there were any others he could approach instead? I’m more than happy if you all feel you would be appropriate stakeholders but wanted to make sure before we finish drafting this one.

 

For reference this is the main epic that I’m needing stakeholders for - https://isisneutronmuon.atlassian.net/jira/software/c/projects/SS/boards/5?selectedIssue=SS-51

 

For info – both of the epics are a bit more fully formed than when Pas looked last week. Unfortunately Rob’s initial attempts to upload the information did not work! All resolved now though.

 

Please could you let me know of any alternative/additional stakeholders by 12th January?

 

Thanks

 

Sarah

 

Sarah Foxley

Mantid Team Leader and Mantid Project Manager

Science and Technology Facilities Council

Phone – 01235 446938

sarah.foxley@stfc.ukri.org

She/Her/Hers

<image001.png>

 

_______________________________________________
Mantid Programme Board mailing list -- mpb@mantidproject.org
To unsubscribe send an email to mpb-leave@mantidproject.org