Hi Steve,
We’ve had a look at this further and another option suggested by a colleague at ORNL. Both options would probably need us to make changes to the underlying database as well as to Mantid which would take longer.
The solution we’re going to try is making e-mail address mandatory information when filling in an error report. Whilst we recognise this may well cut down on the number of reports we receive it will increase
our ability to respond to the ones we do get.
We will need to update our privacy policy to reflect the change and bring in procedures for scrubbing the data (e-mail address & name information) after a specified amount of time.
We are confident we could implement this before Christmas. However it will only be available in Nightly versions after it gets merged in and v6.9 (due Feb 2024) onwards. This will be the case for any solution we implement. Therefore we still will receive error
reports we can’t action and so talking to instrument scientists and users about making their error reports useful (particularly for those using older versions of the software) is still needed.
all the best
Sarah
From: King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>
Sent: 27 November 2023 17:08
To: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>
Cc: mpb@mantidproject.org
Subject: RE: Mantid Error Reports - low figures
Rob Dalgliesh & I were just talking about this and he made what sounds like a sensible, easy, interim solution: what if the error reporter generated a unique reference number which was fed into the stack trace
but which people could also copy in an email to mantid-help if they sent one. Then you could tie up a bug report with the stack trace?
Steve
From: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>
Sent: 27 November 2023 15:10
To: King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>
Cc: mpb@mantidproject.org
Subject: RE: Mantid Error Reports - low figures
Hi Steve,
I can’t tell you how many were duplicates like this. It’s really very difficult from our end to know if it is a duplicate or not. Two different users could produce the same stack trace in two different ways.
If all we have is the stack trace we just don’t know if it is a duplicate or not. As such we don’t treat them as duplicates unless we have proof.
What I would say though is that even without adding details, just adding a name or e-mail address means we know how often someone is experiencing the same issue (assuming it’s the same stack trace each time). There is a world of difference between an issue
that is happening once in a blue moon vs one that is happening regularly. If nothing else it flags up to us that we have a user that we really need to resolve their issues. If we have contact information that we can try to reach out.
All the best
Sarah
From: King, Stephen (STFC,RAL,ISIS) <stephen.king@stfc.ac.uk>
Sent: 27 November 2023 14:30
To: Foxley, Sarah (STFC,RAL,ISIS) <sarah.foxley@stfc.ac.uk>
Cc: mpb@mantidproject.org
Subject: RE: Mantid Error Reports - low figures
Hi Sarah,
I just flagged this with the SANS Group.
Diego has made a fair comment: if there is an operation he does that invokes the Reporter than he says the first time it happens he tries to give a good report. But the 2nd, 3rd, 4th,
etc, time he doesn’t see the point.
So of all the reports you analysed, can you tell how many were ‘duplicates’ like this?
Steve
From: Sarah Foxley - STFC UKRI <sarah.foxley@stfc.ac.uk>
Sent: 27 November 2023 13:48
To: mpb@mantidproject.org
Subject: [Mpb] Mantid Error Reports - low figures
Dear all,
As part of the work to complete the Error Reporter Epic for our next prioritisation meeting error reports over a 4 week period from mid-September to mid-October were looked at. Of all the reports submitted
only 3% were actionable – i.e. only 3% had contact information to be able to request further information to try and resolve them and/or provided sufficient information to enable us to reproduce the error.
Clearly this highlights how important it is to overhaul the error reporting system. However, even if the epic comes up at the next prioritisation meeting in February (as we currently plan) we are unlikely to be able to start work until
the summer. Even then we might not get a solution in place before the end of 2024.
As a result I wonder if it would be possible for you to highlight this issue with your networks? The hope being we can improve on this figure a little before the new Error Reporter is in place. If we are only able to respond to around 3%
of error reports then it is no wonder we have gained a reputation for not doing anything. We have a much higher response rate for e-mails through to
mantid-help@mantidproject.org so I would recommend people use this rather than relying on the error reporter.
I have also highlighted this research to our partners on the Mantid Technical Working Group so that they can highlight the issue with their networks at the other facilities.
Many thanks
Sarah
Sarah Foxley
Mantid Team Leader and Mantid Project Manager
Science and Technology Facilities Council
Phone – 01235 446938
She/Her/Hers
