[OpenSPIM] asynchronous writing of stacks bug?

[LUKE ADAM STUYVENBERG]

Hi Alexis,

> I did some test and have good news: the new SPIMAcquisition.jar solves the
> issue. The slices are now reliably written in the right order. I also have
> the feeling it is a bit faster than the 'original/old' one and the metadata a
> properly written.
Excellent!

> -test2: same thing, but with the new SPIMAcquisition.jar. Everything is fine
> (slices in the right order & metadata present). I observed the async monitor:
> it climbed to 27/240 (writing) then slowly decreased to 0/240 (Idle). Overall
> it felt faster.
My async monitor still rarely exceeds 1 before writing it out, though I've
noticed that sometimes it snowballs (i.e. if it doesn't get one written quickly
enough, they start to pile up until a break in the acquisition). Still, 27/240
gives lots of room, and that limit is pretty conservative. It's good to know
things are working properly now, and the associated speed boost is also a big
plus.

> I'll keep on using this version until the next update.
If I don't get the update uploaded today, it will almost certainly be Monday. At
this point my biggest obstacle is deciding where to put the checkboxes to make
it show the monitor. ;-)

I should warn you in advance, though I'll try to make it clear when I announce
the update: the next update *will* obsolete the changes you made to the Cobalt
adapter. The reason is because it will be designed to work with the Cobalt
adapter in MM's nightly build -- I'm afraid you will need to revert your
changes.

Thanks again for experimenting, and for bringing this to light!

Luke

On Fri, 23 Aug 2013 11:34:18 +0200 Alexis Maizel
<Alexis.Maizel at cos.uni-heidelberg.de> wrote

> Hi Luke,
> 
> I did some test and have good news: the new SPIMAcquisition.jar solves the
> issue. The slices are now reliably written in the right order. I also have
> the feeling it is a bit faster than the 'original/old' one and the metadata a
> properly written.
> 
> You can get the stacks and logs of my tests here:
> http://dl.dropbox.com/u/484859/2013-08-23.zip
> 
> I did 3 acquisitions:
> - test 1: A  stack of 40 slices (1024*1024)  with the original 
> SPIMAcquisition.jar; the slices are mixed up. You will also notice in the
> log.txt file the message "Warning: asynchronous writer may have been
> cancelled before completing. (0)" I had that everytime the metadata are not
> written. I did not hit "abort" at any time; after acquisition I simply waited
> for the file to finish writing to disk (a couple of minutes)  and opened it
> in Fiji (then the message appeared in the log).
> 
> -test2: same thing, but with the new SPIMAcquisition.jar. Everything is fine
> (slices in the right order & metadata present). I observed the async monitor:
> it climbed to 27/240 (writing) then slowly decreased to 0/240 (Idle). Overall
> it felt faster.
> 
> -test3: time lapse (5 time steps) of  a 40-slices stack; one stack every
> 120sec. Absolutely no problems. The async monitor showed the same behaviour
> as for test 2.
> 
> I'll keep on using this version until the next update.
> 
> Thanks a lot for your help.
> 
> With my best regards,
> 
> Alexis
> 
> 
> >> - First, it does some timing over the acquisition, outputting the results
> >> to the log; although it isn't very detailed, it gives a breakdown of how
time
> >> is used while acquiring (for example, if the async isn't appropriately
> >> asynchronous, the acquisition time will be roughly the same as though it
were
> >> off).
> >> - Second, and more importantly for you, the async writer has been
> >> completely revamped. It now queues the begin/end stack events as well as
new
> >> slices, so these operations shouldn't *ever* get out of order. It also puts
> >> up a monitor showing the state of its queue, which will let us know if it's
> >> filling up or not.
> >> If you're willing to play guinea pig again, please try replacing
> >> mmplugins/SPIMAcquisition.jar with the version from:
> >> https://dl.dropboxusercontent.com/u/57890359/SPIMAcquisition.jar
> >> Anyone is free to try this JAR out, but I haven't put it on our Fiji
> >> update site for a few reasons -- the new features are only conditionally
> >> useful, and can't (yet) be disabled. That said, if the new async fixes your
> >> output problems, I'll polish these changes and incorporate them.
> >>
> >>
> >> Thanks,
> >> Luke
> >>
> >>