Hi Steve,
Thank you, I’ll put you down as a key stakeholder in that case. It sounds like the benchmarking should include the loading of event data as well as histogram data. Will one of the stakeholders (Pascal,
Steve or Dirk) be able to provide me with event data to use for benchmarking?
Thanks,
Rob
From: Cottrell, Stephen (STFC,RAL,ISIS) <stephen.cottrell@stfc.ac.uk>
Sent: Tuesday, January 2, 2024 11:13 AM
To: Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk>
Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>; mpb@mantidproject.org; Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk>
Subject: RE: [Mpb] Re: Nexus loading epic - stakeholders needed
Hi,
Muons currently have the problem that loading a large number of small files (as required by the ALC interface) is very time consuming, so good to see this on the list for benchmarking. I would be
a stakeholder for this.
In the future the muon community will be dealing with large datasets gathered in event mode (rather than the current datasets containing histograms), filtering events (e.g. by time) to create histograms.
Hard to be specific at present as there’s a lot about this still in design at present. However, I guess neutron instruments are already doing this (?), so perhaps this is already covered?
Steve
From: Robert Applin - STFC UKRI <robert.applin@stfc.ac.uk>
Sent: Tuesday, January 2, 2024 9:32 AM
To: Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk>
Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>;
mpb@mantidproject.org
Subject: [Mpb] Re: Nexus loading epic - stakeholders needed
Hi Pascal and MPB,
I hope you had a good Christmas break. As an estimate I would say that the “Nexus file handling consolidation” would take 2 releases to complete (at least), so I would say its timescale matches more
with it being an Epic rather than BAU.
Ok, just to keep track of the benchmarking we want to do:
- Loading large files
- Loading a large number of small files
- Loading partial files
The key stakeholders I have written down at the moment are:
- Pascal
- Dirk Honecker
- A NR representative
Who would be a good representative for NR? Do we want any other key stakeholders?
Once we have agreed on all key stakeholders, the next thing will be for me to ask these stakeholders for data that will allow me to benchmark for the cases listed above.
Thanks,
Rob
From: Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk>
Sent: Wednesday, December 20, 2023 10:51 AM
To: Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk>
Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>;
mpb@mantidproject.org
Subject: Re: [Mpb] Re: Nexus loading epic - stakeholders needed
Hi Rob,
Thanks for the reply. It indeed sounds like a bit of a mess we have put ourselves in.
I am just not clear whether this warrants an EPIC or if this should be a BAU item but maybe I still misunderstand the distinction.
Another test I can think of but didnt specifically write is the speed to load partial files.
ON WISH for instance, loading just the monitors (ie just 5 spectra out of 200000) still takes a long time and this is something you might want to do across many many files to check stability.
Re. Benchmarking, at the very least, we should compare with scipp (I have not used it myself but I have heard it is faster). There are also many libraries that are found in the X-ray world (some can be downloaded from SOLEIL synchrotron
website if I recall correctly) that are designed to access lots of data very quickly. I think it could also be worth having a chat with Tristan as he has done some work on this, including recently, for the disordered materials data analysis software.
Hope this helps,
P
Sent from my iPad
On 20 Dec 2023, at 09:55, Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk> wrote: