No. I would advise an NR representative too, or at least let them decide. I'll ask them. Steve From: Foxley, Sarah (STFC,RAL,ISIS) &lt;sarah.foxley(a)stfc.ac.uk&gt; Sent: 19 December 2023 16:55 To: King, Stephen (STFC,RAL,ISIS) &lt;stephen.king(a)stfc.ac.uk&gt;uk>; mpb(a)mantidproject.org Subject: RE: Nexus loading epic - stakeholders needed Hi Steve, That's great thanks. Should we have someone from Reflectometry too or is Dirk ok to represent them too? Thanks Sarah From: Stephen King - STFC UKRI <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>> Sent: 19 December 2023 16:45 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, Dirk Honecker has volunteered for SANS. Thanks, Steve From: Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Sent: 18 December 2023 16:22 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Nexus loading epic - stakeholders needed Dear all, We are progressing with drafting two epics to improve Nexus loading (main project and an enabler project) with the hope of putting them up for review at the next meeting in February. For the main epic we need key stakeholders from the different science groups. At the moment Rob has listed board members. However I wondered if there were any others he could approach instead? I'm more than happy if you all feel you would be appropriate stakeholders but wanted to make sure before we finish drafting this one. For reference this is the main epic that I'm needing stakeholders for - https://isisneutronmuon.atlassian.net/jira/software/c/projects/SS/boards/5?… For info - both of the epics are a bit more fully formed than when Pas looked last week. Unfortunately Rob's initial attempts to upload the information did not work! All resolved now though. Please could you let me know of any alternative/additional stakeholders by 12th January? Thanks Sarah Sarah Foxley Mantid Team Leader and Mantid Project Manager Science and Technology Facilities Council Phone - 01235 446938 sarah.foxley@stfc.ukri.org<mailto:sarah.foxley@stfc.ukri.org> She/Her/Hers [stfc_logo]

Hi Pascal, The enabler epic is called “Nexus file handling consolidation”. After speaking with Martyn, its become clear that we currently access nexus files in three different ways across the Mantid code base: * Using the HDF5 API directly * Using the NeXus API directly * Using our own internal wrapper around the NeXus API This spread of mechanisms presents several challenges for developers including increased maintenance, a requirement for more domain knowledge, and repeated bugs in near duplicate code across each implementation. Furthermore, our own internal wrapper around the NeXus API causes performance bottlenecks, and it is also believed that the NeXus API is less performant than the HDF5 API. Given that all modern files are HDF5 (with the possible exception of the HDF4 files created by Muon?), we wish to consolidate these mechanisms so that we only use the HDF5 API and a more performant internal wrapper around the HDF5 API. This will make it easier to identify other performance bottlenecks in the loading process, and might also provide a performance improvement in and of itself. So in short, this enabler epic is related to the “Nexus load optimization” epic. It should make it easier for developers to identify bottlenecks, and may also provide a performance improvement. What are the available routines that we want to benchmark against? Thanks, Rob From: Pascal Manuel - STFC UKRI &lt;pascal.manuel(a)stfc.ac.uk&gt; Sent: Tuesday, December 19, 2023 5:26 PM To: Foxley, Sarah (STFC,RAL,ISIS) &lt;sarah.foxley(a)stfc.ac.uk&gt; Cc: King, Stephen (STFC,RAL,ISIS) &lt;stephen.king(a)stfc.ac.uk&gt;uk>; mpb(a)mantidproject.org Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, I think the project is simple enough; improve loading speed so I am not sure we should really absolutely every group represented there but I am happy to take the hit for diffraction. The test will be in speed improvement and it should be done with loading a certain number of big files and also loading a big number of small files to be representative of everyone’s need. There are a few available routines that seem to do a quicker job at the moment that we should benchmark against. What is the enabler project? Best P Sent from my iPhone On 19 Dec 2023, at 16:55, Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> wrote:  Hi Steve, That’s great thanks. Should we have someone from Reflectometry too or is Dirk ok to represent them too? Thanks Sarah From: Stephen King - STFC UKRI <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>> Sent: 19 December 2023 16:45 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, Dirk Honecker has volunteered for SANS. Thanks, Steve From: Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Sent: 18 December 2023 16:22 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Nexus loading epic - stakeholders needed Dear all, We are progressing with drafting two epics to improve Nexus loading (main project and an enabler project) with the hope of putting them up for review at the next meeting in February. For the main epic we need key stakeholders from the different science groups. At the moment Rob has listed board members. However I wondered if there were any others he could approach instead? I’m more than happy if you all feel you would be appropriate stakeholders but wanted to make sure before we finish drafting this one. For reference this is the main epic that I’m needing stakeholders for - https://isisneutronmuon.atlassian.net/jira/software/c/projects/SS/boards/5?… For info – both of the epics are a bit more fully formed than when Pas looked last week. Unfortunately Rob’s initial attempts to upload the information did not work! All resolved now though. Please could you let me know of any alternative/additional stakeholders by 12th January? Thanks Sarah Sarah Foxley Mantid Team Leader and Mantid Project Manager Science and Technology Facilities Council Phone – 01235 446938 sarah.foxley@stfc.ukri.org<mailto:sarah.foxley@stfc.ukri.org> She/Her/Hers <image001.png> _______________________________________________ Mantid Programme Board mailing list -- mpb@mantidproject.org<mailto:mpb@mantidproject.org> To unsubscribe send an email to mpb-leave@mantidproject.org<mailto:mpb-leave@mantidproject.org>

Hi Pascal and MPB, I hope you had a good Christmas break. As an estimate I would say that the “Nexus file handling consolidation” would take 2 releases to complete (at least), so I would say its timescale matches more with it being an Epic rather than BAU. Ok, just to keep track of the benchmarking we want to do: - Loading large files - Loading a large number of small files - Loading partial files The key stakeholders I have written down at the moment are: - Pascal - Dirk Honecker - A NR representative Who would be a good representative for NR? Do we want any other key stakeholders? Once we have agreed on all key stakeholders, the next thing will be for me to ask these stakeholders for data that will allow me to benchmark for the cases listed above. Thanks, Rob From: Manuel, Pascal (STFC,RAL,ISIS) &lt;pascal.manuel(a)stfc.ac.uk&gt; Sent: Wednesday, December 20, 2023 10:51 AM To: Applin, Robert (STFC,RAL,ISIS) &lt;robert.applin(a)stfc.ac.uk&gt; Cc: Foxley, Sarah (STFC,RAL,ISIS) &lt;sarah.foxley(a)stfc.ac.uk&gt;uk>; King, Stephen (STFC,RAL,ISIS) &lt;stephen.king(a)stfc.ac.uk&gt;uk>; mpb(a)mantidproject.org Subject: Re: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Rob, Thanks for the reply. It indeed sounds like a bit of a mess we have put ourselves in. I am just not clear whether this warrants an EPIC or if this should be a BAU item but maybe I still misunderstand the distinction. Another test I can think of but didnt specifically write is the speed to load partial files. ON WISH for instance, loading just the monitors (ie just 5 spectra out of 200000) still takes a long time and this is something you might want to do across many many files to check stability. Re. Benchmarking, at the very least, we should compare with scipp (I have not used it myself but I have heard it is faster). There are also many libraries that are found in the X-ray world (some can be downloaded from SOLEIL synchrotron website if I recall correctly) that are designed to access lots of data very quickly. I think it could also be worth having a chat with Tristan as he has done some work on this, including recently, for the disordered materials data analysis software. Hope this helps, P Sent from my iPad On 20 Dec 2023, at 09:55, Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk<mailto:robert.applin@stfc.ac.uk>> wrote:  Hi Pascal, The enabler epic is called “Nexus file handling consolidation”. After speaking with Martyn, its become clear that we currently access nexus files in three different ways across the Mantid code base: 1. Using the HDF5 API directly 2. Using the NeXus API directly 3. Using our own internal wrapper around the NeXus API This spread of mechanisms presents several challenges for developers including increased maintenance, a requirement for more domain knowledge, and repeated bugs in near duplicate code across each implementation. Furthermore, our own internal wrapper around the NeXus API causes performance bottlenecks, and it is also believed that the NeXus API is less performant than the HDF5 API. Given that all modern files are HDF5 (with the possible exception of the HDF4 files created by Muon?), we wish to consolidate these mechanisms so that we only use the HDF5 API and a more performant internal wrapper around the HDF5 API. This will make it easier to identify other performance bottlenecks in the loading process, and might also provide a performance improvement in and of itself. So in short, this enabler epic is related to the “Nexus load optimization” epic. It should make it easier for developers to identify bottlenecks, and may also provide a performance improvement. What are the available routines that we want to benchmark against? Thanks, Rob From: Pascal Manuel - STFC UKRI <pascal.manuel@stfc.ac.uk<mailto:pascal.manuel@stfc.ac.uk>> Sent: Tuesday, December 19, 2023 5:26 PM To: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Cc: King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>>; mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, I think the project is simple enough; improve loading speed so I am not sure we should really absolutely every group represented there but I am happy to take the hit for diffraction. The test will be in speed improvement and it should be done with loading a certain number of big files and also loading a big number of small files to be representative of everyone’s need. There are a few available routines that seem to do a quicker job at the moment that we should benchmark against. What is the enabler project? Best P Sent from my iPhone On 19 Dec 2023, at 16:55, Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> wrote:  Hi Steve, That’s great thanks. Should we have someone from Reflectometry too or is Dirk ok to represent them too? Thanks Sarah From: Stephen King - STFC UKRI <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>> Sent: 19 December 2023 16:45 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, Dirk Honecker has volunteered for SANS. Thanks, Steve From: Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Sent: 18 December 2023 16:22 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Nexus loading epic - stakeholders needed Dear all, We are progressing with drafting two epics to improve Nexus loading (main project and an enabler project) with the hope of putting them up for review at the next meeting in February. For the main epic we need key stakeholders from the different science groups. At the moment Rob has listed board members. However I wondered if there were any others he could approach instead? I’m more than happy if you all feel you would be appropriate stakeholders but wanted to make sure before we finish drafting this one. For reference this is the main epic that I’m needing stakeholders for - https://isisneutronmuon.atlassian.net/jira/software/c/projects/SS/boards/5?… For info – both of the epics are a bit more fully formed than when Pas looked last week. Unfortunately Rob’s initial attempts to upload the information did not work! All resolved now though. Please could you let me know of any alternative/additional stakeholders by 12th January? Thanks Sarah Sarah Foxley Mantid Team Leader and Mantid Project Manager Science and Technology Facilities Council Phone – 01235 446938 sarah.foxley@stfc.ukri.org<mailto:sarah.foxley@stfc.ukri.org> She/Her/Hers <image001.png> _______________________________________________ Mantid Programme Board mailing list -- mpb@mantidproject.org<mailto:mpb@mantidproject.org> To unsubscribe send an email to mpb-leave@mantidproject.org<mailto:mpb-leave@mantidproject.org>

Hi Steve, Yes, several beamlines filter their event collected data by time or by values from blocks (temperature, etc..). I would say this could also do with a speed up (but maybe there has been one, it is not an option I use very often). I can provide an example from WISH for Rob to benchmark. I seem to remember that loading this type of files in mantid was quite dependent on the data size (which can end up being very big for WISH if the sample was a good scatterer and the collection time was >1h). Best, P Sent from my iPad On 2 Jan 2024, at 12:12, Cottrell, Stephen (STFC,RAL,ISIS) &lt;stephen.cottrell(a)stfc.ac.uk&gt; wrote:  Hi, Muons currently have the problem that loading a large number of small files (as required by the ALC interface) is very time consuming, so good to see this on the list for benchmarking. I would be a stakeholder for this. In the future the muon community will be dealing with large datasets gathered in event mode (rather than the current datasets containing histograms), filtering events (e.g. by time) to create histograms. Hard to be specific at present as there’s a lot about this still in design at present. However, I guess neutron instruments are already doing this (?), so perhaps this is already covered? Steve From: Robert Applin - STFC UKRI &lt;robert.applin(a)stfc.ac.uk&gt; Sent: Tuesday, January 2, 2024 9:32 AM To: Manuel, Pascal (STFC,RAL,ISIS) &lt;pascal.manuel(a)stfc.ac.uk&gt; Cc: Foxley, Sarah (STFC,RAL,ISIS) &lt;sarah.foxley(a)stfc.ac.uk&gt;uk>; King, Stephen (STFC,RAL,ISIS) &lt;stephen.king(a)stfc.ac.uk&gt;uk>; mpb(a)mantidproject.org Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Pascal and MPB, I hope you had a good Christmas break. As an estimate I would say that the “Nexus file handling consolidation” would take 2 releases to complete (at least), so I would say its timescale matches more with it being an Epic rather than BAU. Ok, just to keep track of the benchmarking we want to do: - Loading large files - Loading a large number of small files - Loading partial files The key stakeholders I have written down at the moment are: - Pascal - Dirk Honecker - A NR representative Who would be a good representative for NR? Do we want any other key stakeholders? Once we have agreed on all key stakeholders, the next thing will be for me to ask these stakeholders for data that will allow me to benchmark for the cases listed above. Thanks, Rob From: Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk<mailto:pascal.manuel@stfc.ac.uk>> Sent: Wednesday, December 20, 2023 10:51 AM To: Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk<mailto:robert.applin@stfc.ac.uk>> Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>>; mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: Re: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Rob, Thanks for the reply. It indeed sounds like a bit of a mess we have put ourselves in. I am just not clear whether this warrants an EPIC or if this should be a BAU item but maybe I still misunderstand the distinction. Another test I can think of but didnt specifically write is the speed to load partial files. ON WISH for instance, loading just the monitors (ie just 5 spectra out of 200000) still takes a long time and this is something you might want to do across many many files to check stability. Re. Benchmarking, at the very least, we should compare with scipp (I have not used it myself but I have heard it is faster). There are also many libraries that are found in the X-ray world (some can be downloaded from SOLEIL synchrotron website if I recall correctly) that are designed to access lots of data very quickly. I think it could also be worth having a chat with Tristan as he has done some work on this, including recently, for the disordered materials data analysis software. Hope this helps, P Sent from my iPad On 20 Dec 2023, at 09:55, Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk<mailto:robert.applin@stfc.ac.uk>> wrote:  Hi Pascal, The enabler epic is called “Nexus file handling consolidation”. After speaking with Martyn, its become clear that we currently access nexus files in three different ways across the Mantid code base: 1. Using the HDF5 API directly 2. Using the NeXus API directly 3. Using our own internal wrapper around the NeXus API This spread of mechanisms presents several challenges for developers including increased maintenance, a requirement for more domain knowledge, and repeated bugs in near duplicate code across each implementation. Furthermore, our own internal wrapper around the NeXus API causes performance bottlenecks, and it is also believed that the NeXus API is less performant than the HDF5 API. Given that all modern files are HDF5 (with the possible exception of the HDF4 files created by Muon?), we wish to consolidate these mechanisms so that we only use the HDF5 API and a more performant internal wrapper around the HDF5 API. This will make it easier to identify other performance bottlenecks in the loading process, and might also provide a performance improvement in and of itself. So in short, this enabler epic is related to the “Nexus load optimization” epic. It should make it easier for developers to identify bottlenecks, and may also provide a performance improvement. What are the available routines that we want to benchmark against? Thanks, Rob From: Pascal Manuel - STFC UKRI <pascal.manuel@stfc.ac.uk<mailto:pascal.manuel@stfc.ac.uk>> Sent: Tuesday, December 19, 2023 5:26 PM To: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Cc: King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>>; mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, I think the project is simple enough; improve loading speed so I am not sure we should really absolutely every group represented there but I am happy to take the hit for diffraction. The test will be in speed improvement and it should be done with loading a certain number of big files and also loading a big number of small files to be representative of everyone’s need. There are a few available routines that seem to do a quicker job at the moment that we should benchmark against. What is the enabler project? Best P Sent from my iPhone On 19 Dec 2023, at 16:55, Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> wrote:  Hi Steve, That’s great thanks. Should we have someone from Reflectometry too or is Dirk ok to represent them too? Thanks Sarah From: Stephen King - STFC UKRI <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>> Sent: 19 December 2023 16:45 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, Dirk Honecker has volunteered for SANS. Thanks, Steve From: Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Sent: 18 December 2023 16:22 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Nexus loading epic - stakeholders needed Dear all, We are progressing with drafting two epics to improve Nexus loading (main project and an enabler project) with the hope of putting them up for review at the next meeting in February. For the main epic we need key stakeholders from the different science groups. At the moment Rob has listed board members. However I wondered if there were any others he could approach instead? I’m more than happy if you all feel you would be appropriate stakeholders but wanted to make sure before we finish drafting this one. For reference this is the main epic that I’m needing stakeholders for - https://isisneutronmuon.atlassian.net/jira/software/c/projects/SS/boards/5?… For info – both of the epics are a bit more fully formed than when Pas looked last week. Unfortunately Rob’s initial attempts to upload the information did not work! All resolved now though. Please could you let me know of any alternative/additional stakeholders by 12th January? Thanks Sarah Sarah Foxley Mantid Team Leader and Mantid Project Manager Science and Technology Facilities Council Phone – 01235 446938 sarah.foxley@stfc.ukri.org<mailto:sarah.foxley@stfc.ukri.org> She/Her/Hers <image001.png> _______________________________________________ Mantid Programme Board mailing list -- mpb@mantidproject.org<mailto:mpb@mantidproject.org> To unsubscribe send an email to mpb-leave@mantidproject.org<mailto:mpb-leave@mantidproject.org>

Hi All, Max Skoda, cc:d here, has volunteered from NR. I will let Max speak for himself, but he would be keen for some consideration of the structure/content of our Nexus files. Steve From: Applin, Robert (STFC,RAL,ISIS) &lt;robert.applin(a)stfc.ac.uk&gt; Sent: 02 January 2024 14:10 To: Cottrell, Stephen (STFC,RAL,ISIS) &lt;stephen.cottrell(a)stfc.ac.uk&gt; Cc: Foxley, Sarah (STFC,RAL,ISIS) &lt;sarah.foxley(a)stfc.ac.uk&gt;uk>; King, Stephen (STFC,RAL,ISIS) &lt;stephen.king(a)stfc.ac.uk&gt;uk>; mpb(a)mantidproject.org; Manuel, Pascal (STFC,RAL,ISIS) &lt;pascal.manuel(a)stfc.ac.uk&gt; Subject: RE: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Steve, Thank you, I’ll put you down as a key stakeholder in that case. It sounds like the benchmarking should include the loading of event data as well as histogram data. Will one of the stakeholders (Pascal, Steve or Dirk) be able to provide me with event data to use for benchmarking? Thanks, Rob From: Cottrell, Stephen (STFC,RAL,ISIS) <stephen.cottrell@stfc.ac.uk<mailto:stephen.cottrell@stfc.ac.uk>> Sent: Tuesday, January 2, 2024 11:13 AM To: Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk<mailto:robert.applin@stfc.ac.uk>> Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>>; mpb@mantidproject.org<mailto:mpb@mantidproject.org>; Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk<mailto:pascal.manuel@stfc.ac.uk>> Subject: RE: [Mpb] Re: Nexus loading epic - stakeholders needed Hi, Muons currently have the problem that loading a large number of small files (as required by the ALC interface) is very time consuming, so good to see this on the list for benchmarking. I would be a stakeholder for this. In the future the muon community will be dealing with large datasets gathered in event mode (rather than the current datasets containing histograms), filtering events (e.g. by time) to create histograms. Hard to be specific at present as there’s a lot about this still in design at present. However, I guess neutron instruments are already doing this (?), so perhaps this is already covered? Steve From: Robert Applin - STFC UKRI <robert.applin@stfc.ac.uk<mailto:robert.applin@stfc.ac.uk>> Sent: Tuesday, January 2, 2024 9:32 AM To: Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk<mailto:pascal.manuel@stfc.ac.uk>> Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>>; mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Pascal and MPB, I hope you had a good Christmas break. As an estimate I would say that the “Nexus file handling consolidation” would take 2 releases to complete (at least), so I would say its timescale matches more with it being an Epic rather than BAU. Ok, just to keep track of the benchmarking we want to do: - Loading large files - Loading a large number of small files - Loading partial files The key stakeholders I have written down at the moment are: - Pascal - Dirk Honecker - A NR representative Who would be a good representative for NR? Do we want any other key stakeholders? Once we have agreed on all key stakeholders, the next thing will be for me to ask these stakeholders for data that will allow me to benchmark for the cases listed above. Thanks, Rob From: Manuel, Pascal (STFC,RAL,ISIS) <pascal.manuel@stfc.ac.uk<mailto:pascal.manuel@stfc.ac.uk>> Sent: Wednesday, December 20, 2023 10:51 AM To: Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk<mailto:robert.applin@stfc.ac.uk>> Cc: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>>; King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>>; mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: Re: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Rob, Thanks for the reply. It indeed sounds like a bit of a mess we have put ourselves in. I am just not clear whether this warrants an EPIC or if this should be a BAU item but maybe I still misunderstand the distinction. Another test I can think of but didnt specifically write is the speed to load partial files. ON WISH for instance, loading just the monitors (ie just 5 spectra out of 200000) still takes a long time and this is something you might want to do across many many files to check stability. Re. Benchmarking, at the very least, we should compare with scipp (I have not used it myself but I have heard it is faster). There are also many libraries that are found in the X-ray world (some can be downloaded from SOLEIL synchrotron website if I recall correctly) that are designed to access lots of data very quickly. I think it could also be worth having a chat with Tristan as he has done some work on this, including recently, for the disordered materials data analysis software. Hope this helps, P Sent from my iPad On 20 Dec 2023, at 09:55, Applin, Robert (STFC,RAL,ISIS) <robert.applin@stfc.ac.uk<mailto:robert.applin@stfc.ac.uk>> wrote:  Hi Pascal, The enabler epic is called “Nexus file handling consolidation”. After speaking with Martyn, its become clear that we currently access nexus files in three different ways across the Mantid code base: 1. Using the HDF5 API directly 2. Using the NeXus API directly 3. Using our own internal wrapper around the NeXus API This spread of mechanisms presents several challenges for developers including increased maintenance, a requirement for more domain knowledge, and repeated bugs in near duplicate code across each implementation. Furthermore, our own internal wrapper around the NeXus API causes performance bottlenecks, and it is also believed that the NeXus API is less performant than the HDF5 API. Given that all modern files are HDF5 (with the possible exception of the HDF4 files created by Muon?), we wish to consolidate these mechanisms so that we only use the HDF5 API and a more performant internal wrapper around the HDF5 API. This will make it easier to identify other performance bottlenecks in the loading process, and might also provide a performance improvement in and of itself. So in short, this enabler epic is related to the “Nexus load optimization” epic. It should make it easier for developers to identify bottlenecks, and may also provide a performance improvement. What are the available routines that we want to benchmark against? Thanks, Rob From: Pascal Manuel - STFC UKRI <pascal.manuel@stfc.ac.uk<mailto:pascal.manuel@stfc.ac.uk>> Sent: Tuesday, December 19, 2023 5:26 PM To: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Cc: King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>>; mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, I think the project is simple enough; improve loading speed so I am not sure we should really absolutely every group represented there but I am happy to take the hit for diffraction. The test will be in speed improvement and it should be done with loading a certain number of big files and also loading a big number of small files to be representative of everyone’s need. There are a few available routines that seem to do a quicker job at the moment that we should benchmark against. What is the enabler project? Best P Sent from my iPhone On 19 Dec 2023, at 16:55, Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> wrote:  Hi Steve, That’s great thanks. Should we have someone from Reflectometry too or is Dirk ok to represent them too? Thanks Sarah From: Stephen King - STFC UKRI <stephen.king@stfc.ac.uk<mailto:stephen.king@stfc.ac.uk>> Sent: 19 December 2023 16:45 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Re: Nexus loading epic - stakeholders needed Hi Sarah, Dirk Honecker has volunteered for SANS. Thanks, Steve From: Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk<mailto:sarah.foxley@stfc.ac.uk>> Sent: 18 December 2023 16:22 To: mpb@mantidproject.org<mailto:mpb@mantidproject.org> Subject: [Mpb] Nexus loading epic - stakeholders needed Dear all, We are progressing with drafting two epics to improve Nexus loading (main project and an enabler project) with the hope of putting them up for review at the next meeting in February. For the main epic we need key stakeholders from the different science groups. At the moment Rob has listed board members. However I wondered if there were any others he could approach instead? I’m more than happy if you all feel you would be appropriate stakeholders but wanted to make sure before we finish drafting this one. For reference this is the main epic that I’m needing stakeholders for - https://isisneutronmuon.atlassian.net/jira/software/c/projects/SS/boards/5?… For info – both of the epics are a bit more fully formed than when Pas looked last week. Unfortunately Rob’s initial attempts to upload the information did not work! All resolved now though. Please could you let me know of any alternative/additional stakeholders by 12th January? Thanks Sarah Sarah Foxley Mantid Team Leader and Mantid Project Manager Science and Technology Facilities Council Phone – 01235 446938 sarah.foxley@stfc.ukri.org<mailto:sarah.foxley@stfc.ukri.org> She/Her/Hers <image001.png> _______________________________________________ Mantid Programme Board mailing list -- mpb@mantidproject.org<mailto:mpb@mantidproject.org> To unsubscribe send an email to mpb-leave@mantidproject.org<mailto:mpb-leave@mantidproject.org>