Fusion

= Fusion =

Registered multi-view OpenSPIM data need to be fused into a single output image in order to achieve complete coverage of a large specimen. Fusion means here combining information from different views in areas where the views overlap. Several strategies to do so exist, many are published (ADD REFS LATER). We will focus here on the two fusion methods implemented in Fiji - the content based multiview fusion and multiview deconvolution.

''Note that fused data are different, not necessarily better compared to raw SPIM data. Both fusion algorithms described here potentially deteriorate the quality of the data in some respects while improving other aspects. We will discuss the fusion artefacts in the respective sections. However it should be said that sometimes it is beneficial to NOT fuse the data at all and perform analysis on the raw registered image stacks (for example segmentation of cells in the individual views and reconciliation of the results in the segmentation domain). In the section on of this tutorial on browsing we will describe how to view raw registered multi-view OpenSPIM views.

= Content Based Fusion =

The content based multi-view fusion evaluates local information entropy in the areas where several views overlap and combines the views by enhancing the low entropy information from the view containing useful data while suppressing the high entropy noise from the blurred data in other views. The principles of the method are discussed in depth here and the parameters of the Fiji plugin implementing the method are described here. As before we enhance these technical description with tutorial style walk through using the sample OpenSPIM data.

Content based multi-view fusion requires significant computational resources. The input raw data stacks are large and when they are transformed to the position where they overlap, the bounding box of the output volume can become several fold larger. To process on such large volumes can take significant amount of time and it may fail due to insufficient memory even on the largest computer systems (we did experience out of memory exceptions on a system with 128GB of RAM!). Therefore we will proceed sequentially, minimising the memory footprint.


 * first we will fuse four times down-sampled data with all computationally demanding options turned off - see First approximate run.
 * next we will crop the output volume to include only the specimen - see Cropping.
 * finally we will fuse the cropped volume with all options turned on - see Final run.

First approximate run
The purpose of this run is to get to the output fused data as quickly as possible in order to evaluate them and decide on how to continue.

Run
As before, let us annotate the output the fusion plugin sends to the Log window. dir: /home/tomancak/Desktop/OpenSPIM_for_website/tiffs/registration spim_TL05_Angle0.tif.registration Z-stretching = 9.30232558139535

The plugin identifies the registration files and reads in the z-scaling.

0: -1 channel 0 takes it from channel 0 tp -1 Version 0.55 (Thu Jun 06 00:11:09 CEST 2013): Starting Bead Extraction Read 1046 beads for spim_TL05_Angle0.tif (id = 0) Read 1172 beads for spim_TL05_Angle1.tif (id = 1) Read 1009 beads for spim_TL05_Angle2.tif (id = 2) Read 1039 beads for spim_TL05_Angle3.tif (id = 3) Read 1212 beads for spim_TL05_Angle4.tif (id = 4) (Thu Jun 06 00:11:09 CEST 2013): Finished Bead Extraction (Thu Jun 06 00:11:09 CEST 2013): Starting Registration (Thu Jun 06 00:11:09 CEST 2013): Finished Registration

It then repeats the registration steps. We will recall that the actual registration took very short time, once the beads are segmented reading them in and repeating the optimization is simpler then programming a specific function that would load the matrices from the files (apparently - Stephan?).

(Thu Jun 06 00:11:09 CEST 2013): Starting Fusion Dimension of final output image: From : (0.0, 0.0, -485.1395) to (1535.1057, 1127.0483, 984.4027) Size: (1535.1057, 1127.0483, 1469.5422) needs 9699 MB of RAM Scaled size(4): (384, 282, 367) needs 152 MB of RAM

Here we see the benefits of downsampling, the full resolution output image would need almost 10GB of RAM and thats only the final output excluding all intermediate steps - the actual memory footprint is several fold higher. One can easily run out of memory with this data. However scaled we are down to much more reasonable 152MB for the output image.

Location of pixel (0,0,0) in global coordinates is: (0.0, 0.0, -485.1395) (Thu Jun 06 00:11:09 CEST 2013): Reserving memory for fused image. Loading source images (Channel 0). (Thu Jun 06 00:11:17 CEST 2013): Computing output image (Channel 0). (Thu Jun 06 00:11:19 CEST 2013): Closing all input images (Channel 0). (Thu Jun 06 00:11:19 CEST 2013): Done computing output image (Channel 0). (Thu Jun 06 00:11:19 CEST 2013): Displaying image (Channel 0). (Thu Jun 06 00:11:20 CEST 2013): Finished Fusion Finished processing.

Now here is the action, the plugin loads the images, fuses them without doing any entropy evaluation, closes the input images and displays the output. Since we are working with 4 times down-sampled data it all takes only seconds.

A new window will pop-up. This is the Output that we will discuss in the next section.

Cropping
In the next step of the fusion pipeline we will crop the output image as much as possible. Why? You may have noticed that the bounding box of the fused image is actually rather large (4 x 384x282x367). This is because we have rotated the input image stacks in 3d and the resulting cubic volume has to include them all. Saving output images in this form would increase the storage requirements by orders of magnitude. To reduce the storage footprint we will use this initial run of fusion to define a volume that fits the specimen most efficiently. It will also reduce memory requirements allowing us to run the fusion with full resolution images.

''Note that in reality it makes sense to define the crop volume only after the time-lapse registration, because that will be the final output of the pipeline and the crop area has to be defined relative to that registration. However to preserve linearity we will describe cropping here.

Final run
= Deconvolution =