[OpenSPIM] asynchronous writing of stacks bug?

[Luke Stuyvenberg]

Hi Alexis,

Here's the email I sent, sans the rest of the conversation thread (which may have been the cause of the rejection).


Thanks,
Luke


P.S. The source code for the linked plugin is available, but split up into two branches on github -- async-fix and profiling -- in case anyone is looking for it. At the moment, neither merges cleanly into the main branch, since I've just merged the new device manager code yesterday.

On 08/21/13, "Luke Stuyvenberg"  wrote:
> Hi Alexis,
> 
> > some more tests and hopefully some progresses. I scanned a 82Mb stack (1024*1024*41 @ 16bits)
> > 
> > - RAM usage by Fiji and stack writing to disk: 
> > ~20Mb used at start (out of 1Gb allocated) 
> > as the scan progresses in ramps up to ~110Mb and then slowly decreases as the size of the file on disk increases; finally the garbage collector resets to 20Mb
> > it takes up to 2 minutes to write the 84Mb and at least 1 minute more to write the remaining 2Mb (the metadata, I presume...)
> Okay; although the writing times are not at all ideal (my 400MB stacks were written synchronously and in their entirety in a little over a minute and a half), the RAM usage is more or less as expected.
> 
> 
> > If you open that stack & metadata, you will see that the planes are not written in the right order. Look at the sequence (toward the end of the file above):
> 
> I have some mixed feelings on this; this is the definitive proof I was looking for, but I'm genuinely at a loss as to how this may have come about. My best guess is that the synchronization is too loose, and somehow the acquisition ("main") thread is writing slices (because it feels it needs to) before the writing thread.
> 
> 
> > I have the feeling that my problems are somehow linked to Bio-Format	not behaving properly. 
> 
> I've tried the new Bio-Formats daily builds update site through the Fiji updater, and haven't noticed any problems from it; perhaps try enabling that update site and downloading the very latest versions?
> 
> 
> Finally, I have most of the promised plugin updates. There are two main additions:
> - First, it does some timing over the acquisition, outputting the results to the log; although it isn't very detailed, it gives a breakdown of how time is used while acquiring (for example, if the async isn't appropriately asynchronous, the acquisition time will be roughly the same as though it were off).
> - Second, and more importantly for you, the async writer has been completely revamped. It now queues the begin/end stack events as well as new slices, so these operations shouldn't *ever* get out of order. It also puts up a monitor showing the state of its queue, which will let us know if it's filling up or not.
> If you're willing to play guinea pig again, please try replacing mmplugins/SPIMAcquisition.jar with the version from: https://dl.dropboxusercontent.com/u/57890359/SPIMAcquisition.jar
> Anyone is free to try this JAR out, but I haven't put it on our Fiji update site for a few reasons -- the new features are only conditionally useful, and can't (yet) be disabled. That said, if the new async fixes your output problems, I'll polish these changes and incorporate them.
> 
> 
> Thanks,
> Luke
> 
>