[OpenSPIM] OpenSPIM 64-bit [0000129]

[Johannes Schindelin]

Hi Nikul,

On Wed, 5 Feb 2014, Nikul wrote:

> The write speed to the RAID array through openspim is really bad at this
> point(2-3 fps). I suspect this is due to the use of the BioFormats
> writer.

And you would be correct. We identified quite a few bottlenecks in
BioFormats (actually on the reader side, but some of them extend to the
writer, too) and the improvements are hopefully coming with a future
BioFormats version.

In the concrete case of writing data to a RAID as fast as possible, I
would suggest writing *raw* data, though. You can always stitch back the
metadata. The only metadata you have to store at the same time as the
frames you are writing out are the metadata that *do* change with the
frame, i.e. z position and time. All the other metadata can be written out
at a much, much slower pace, some of them actually need to be written only
once per experiment.

That should save a *ton* of time.

Ciao,
Johannes

P.S.: It might make sense to experiment with a low-impact compression
scheme such as LZO (e.g. via https://github.com/Karmasphere/lzo-java)
because it can reduce a large chunk of the redundancies at the cost of CPU
but at the benefit of the network or processor bus. If you find time to
experiment with this, I would be very much interested in hearing how you
fared!