[OpenSPIM] OpenSPIM 64-bit [0000129]

[Johannes Schindelin]

Hi Nikul,

On Wed, 5 Feb 2014, Nikul wrote:

> > P.S.: It might make sense to experiment with a low-impact compression
> > scheme such as LZO (e.g. via https://github.com/Karmasphere/lzo-java)
> > because it can reduce a large chunk of the redundancies at the cost of
> > CPU but at the benefit of the network or processor bus. If you find
> > time to experiment with this, I would be very much interested in
> > hearing how you fared!
> 
> We might look at some sort of compression later, but for now I am going
> to concentrate on trying to maximize throughput.

To be clear, my suggestion of LZO was trying to maximize throughput. The
bottleneck is most likely the storage -- even if you would use SSD or a
super-parallel RAID, the data transport between RAM and the storage device
is very, very slow compared to the transport between the RAM and the CPU.

That is why I suggested giving LZO a try: it might very well turn out that
throughput is *higher* because your CPU is unlikely to be highly impacted
by the use of it (I expect it still to be used less than 50% maximum even
if Micro-Manager is recording *and* LZO is used to compress the data
stream) and at the same time, the bottleneck -- the data transport to the
storage device -- is used to the same extent. Only that it would now
transport between 20-50% more data (depending on your exact data, of
course, but should be in that ball park given that fluorescence images
have a histogram very favorable to even simplistic compression).

Ciao,
Johannes