METHOD AND APPARATUS

Abstract

A method of reconstructing an electron microscopy image of size [Mx N] pixels of a first sample, the method implemented by a computer comprising a processor and a memory, the method comprising: providing a set of pre-learned dictionaries, including a first pre-learned dictionary including a set of p.sub.1 atoms; acquiring a sparse set of S acquired sub-images, including a first sub-image of size [ab] pixels wherein a, b[2, min {M,N}], of the first sample; and reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and the set of pre-learned dictionaries.

Claims

1. A method of reconstructing an electron microscopy image of size [MN] pixels of a first sample, the method implemented by a computer comprising a processor and a memory, the method comprising: providing a set of pre-learned dictionaries, including a first pre-learned dictionary including a set of p.sub.1 atoms; acquiring a sparse set of S acquired sub-images, including a first sub-image of size [ab] pixels wherein a, b[2, min{M, N}], of the first sample; and reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and the set of pre-learned dictionaries.

2. The method according to claim 1, wherein the set of pre-learned dictionaries includes a second pre-learned dictionary comprising a set of p.sub.2 atoms; and wherein reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and the set of pre-learned dictionaries comprises reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and a combination of the first pre-learned dictionary and the second pre-learned dictionary.

3. The method according to claim 2, wherein the combination of the first pre-learned dictionary and the second pre-learned dictionary comprises and/or is a pair-wise combination of the first pre-learned dictionary and the second pre-learned dictionary.

4. The method according to claim 1, comprising training the first pre-learned dictionary using the using the sparse set of S sub-images of the first sample.

5. The method according to claim 4, wherein reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and the set of pre-learned dictionaries comprises reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and the first pre-learned dictionary trained using the using the sparse set of S sub-images of the first sample.

6. The method according to claim 1, comprising selecting a subset of $p_1^{}$ atoms from the set of p.sub.1 atoms included in the first pre-learned dictionary; and wherein reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and the set of pre-learned dictionaries comprises reconstructing the electron microscopy image of the first sample using the sparse set of S sub-images of the first sample and the selected subset of $p_1^{}$ atoms included in the first pre-learned dictionary.

7. The method according to claim 6, wherein selecting the subset of $p_1^{}$ atoms from the set of p.sub.1 atoms included in the first pre-learned dictionary comprises selecting the subset of $p_1^{}$ atoms from the set of p.sub.1 atoms included in the first pre-learned dictionary based on residual energies of the respective atoms of the set of p.sub.1 atoms.

8. The method according to claim 1, wherein providing the first pre-learned dictionary including the set of p.sub.1 atoms comprises training a dictionary using a fully sampled acquired electron microscopy image of a second sample and/or a fully sampled simulated electron microscopy image of the second sample.

9. The method according to claim 8, wherein the first sample and the second sample are mutually different.

10. The method according to claim 1, comprising transforming one or more atoms of the set of p.sub.1 atoms included in the first pre-learned dictionary.

11. A method of controlling an electron microscope, the method implemented, at least in part, by a computer comprising a processor and a memory, the method comprising: obtaining parameters of the electron microscopy; acquiring a first sparse set of S acquired sub-images of a first sample comprising controlling the electron microscope using the obtained parameters of the electron microscopy; reconstructing a first electron microscopy image of size [MN] pixels of the first sample according to claim 1 using the first sparse set of S sub-images of the first sample; comparing the first electron microscopy image against thresholds of one or more target properties of the first electron microscopy image; adapting the parameters of the electron microscopy based on a result of the comparing; and acquiring a second sparse set of S acquired sub-images of the first sample comprising controlling the electron microscope using the adapted parameters of the electron microscopy.

12. The method according to claim 11, comprising: reconstructing a second electron microscopy image of size [MN] pixels of the first sample using the second sparse set of S sub-images of the first sample.

13. (canceled)

14. An electron microscope including a computer comprising a processor and a memory configured to implement a method according to claim 1.

15. (canceled)

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0143] For a better understanding of the invention, and to show how exemplary embodiments of the same may be brought into effect, reference will be made, by way of example only, to the accompanying diagrammatic Figures, in which:

[0144] FIG. 1 shows K-SVD [1616] dictionaries (generated using MATLAB) for all four input images, showing the varying features learned from each image. PSNR values shown correspond to a fully sampled (100%) reconstruction of the source image using the given dictionary.

[0145] FIG. 2 shows reconstructions of the images Barbara and Atomic using their two corresponding dictionaries. Highlighted images along the diagonal show images inpainted with their own dictionaries. PSNR values are measured against the reference image (left).

[0146] FIG. 3 shows an illustration of the formation of a [1616] patch (centered atom from the image Atomic) inpainted with (non-circular) dictionary atoms from the image Barbara showing the cumulative combination of dictionary elements. Sparse coding was performed with a custom OMP implementation in python.

[0147] FIG. 4 shows K-SVD [1616] dictionaries (generated using MATLAB) for all magnifications, showing the varying features learned from each image. PSNR values shown correspond to a fully sampled (100%) reconstruction of the source image using the given dictionary.

[0148] FIG. 5 shows 25% (cropped) reconstructions of heat-treated Ceria using 3M and 12M magnifications, showing a reduced reconstruction quality when using inappropriately scaled dictionary elements. Highlighted images along the diagonal show images inpainted with their own dictionaries. PSNR values are measured against the reference image (left).

[0149] FIG. 6 shows a grid of 25% reconstructions of the 4 input images (top) using dictionaries generated from the references (left). Highlighted images along the diagonal show images inpainted with their own dictionaries. PSNR values are measured against the reference image (left).

[0150] FIG. 7 is a table of PSNR values for the transfer experiment. Colour bars are scaled according to input images (columns).

[0151] FIG. 8 is a table of SSIM values for the transfer experiment. Colour bars are scaled according to input images (columns).

[0152] FIG. 9 shows a grid of 25% reconstructions of all images at different magnifications using dictionaries generated from the references (left). Highlighted images along the diagonal show images inpainted with their own dictionaries. PSNR values are measured against the reference image (left).

[0153] FIG. 10 is a table of PSNR values for the magnification experiment. Colour bars are scaled according to input images (columns).

[0154] FIG. 11 is a Table of SSIM values for the magnification experiment. Colour bars are scaled according to input images (columns).

[0155] FIG. 12A schematically depicts traditional reconstruction (single instance); FIG. 12B schematically depicts an example of dual-instance configuration; FIG. 12C schematically depicts frame-by-frame reconstruction; and FIG. 12D schematically depicts data visualisation.

[0156] FIG. 13A schematically depicts a new method for real-time dictionary transfer according to an exemplary embodiment, using separate training and reconstruction instances of BPFA (each with their own set of parameters and coefficients) operating on the same transient dictionary D.sub.t; and FIG. 13B shows definitions of parameters for the method of FIG. 13A.

[0157] FIG. 14 shows image reconstruction quality results (against sampling %) for an atomic-resolution image using a static 2D DCT Dictionary, a pre-trained dictionary at 100% sampling (i.e. using dictionary transfer from the reference) and a subsampled dictionary trained at each of the sampling ratios.

[0158] FIG. 15 shows image reconstruction quality results (against sampling %) for a natural (test) image using a static 2D DCT Dictionary, a pre-trained dictionary at 100% sampling (i.e. using dictionary transfer from the reference) and a subsampled dictionary trained at each of the sampling ratios.

[0159] FIG. 16 shows generic dictionary transfer results for a set of various images (some natural, some STEM), showing the quality of reconstructions of each of the images with each corresponding BPFA dictionary.

[0160] FIG. 17 shows generic dictionary transfer results for a series of STEM images of High Resolution Ceria at various magnifications, showing the quality of reconstructions of each of the images with each corresponding BPFA dictionary.

[0161] FIG. 18 shows (Both) Plot of PSNR (dB) quality metric against time-to-solution(s) for a 25% subsampled image of high-resolution Ceria (8M) comparing the use of a dictionary pre-trained on the target image at 100% sampling (Dictionary Transfer) against learning a dictionary directly from the subsampled frame (Subsampled Training). Each data-point represents a batch ( of the dataset) averaged over 10 independent trials. As shown in FIG. 18, dictionary transfer (from 100% reference) is capable of producing an improved reconstruction for multiple epochs, due to the time required for the dictionary to converge when training directly from subsampled data. In a real-time reconstruction scenario (where each frame may only be reconstructed for 1-2 epochs) this provides a significant advantage, essentially allowing for an equivalent reconstruction in a fraction of the time.

[0162] FIG. 19 shows: (Top, left) Input sub-sampled data with 2% sparse acquisition. Inpainting using sparse dictionary learning and reconstruction are shown in top right. (Bottom, left) Simulated image of the same sample. Inpainting of input using the dictionary learnt from the simulated data, shown in bottom right. The result shows a superior reconstruction by using a dictionary from a simulated image.

[0163] FIG. 20 shows: (Left) Input sub-sampled experimental data inpainted using BPFA trained on the sparse sampled input, (right) same sample as left but this time it is inpainted using a dictionary from simulated data. The results show a super-resolved image where the higher frequencies in the power-spectra (overlaid) are transferred from the simulated data.

DETAILED DESCRIPTION OF THE DRAWINGS

[0164] While the main goal of this research is to investigate the application of these methods to subsampled acquisition, particularly for materials which may not currently be imaged, the general cross-compatibility of representative dictionaries applies to those trained on fully-sampled data, and sub-sampled data alike. For simplicity, and the ease of reproducing our results, the following experiments use dictionaries trained on fully-sampled images. All reconstructions in this paper therefore used K-SVD (performed in MATLAB using ksvdbox13) to generate a representative dictionary from a fully sampled image, which were then artificially subsampled (randomly, to 25%) and reconstructed using OMP (via a custom OpenMP, CPU parallelised C.sup.++ implementation). For consistency, all dictionaries were the same size (128 elements) and were trained with the same parameters.

Experiment 1: Dictionary Transfer

[0165] The concept of dictionary transfer is simple; learning a dictionary from one image (whether that be fully sampled or not) and using it to inpaint another subsampled image. One may initially suspect that if the two images are very different, this process would lead to a very poor reconstruction. However, due to the nature of a sparse linear combination of dictionary elements, widely varying signals may be represented with different combinations (and weights) of only a few atoms. The benefit of this is clear, if a pre-learned dictionary is capable of generating a reasonable quality reconstruction of any given (potential) STEM image, a fully sampled live preview of the sample may be reconstructed in minimal time (either with a significantly reduced dictionary learning step for untrained materials, or taking only the time required for the sparse-coding step for materials the dictionary is known to be appropriate for), allowing for the spatial position, magnification and more to be fine-tuned by the microscope operator before a final scan is taken. To test the feasibility of dictionary transfer, we perform an experiment on four different images, the first of which is an example of a natural image (photograph) for comparison, and the following three of which are HAADF STEM images:

TABLE-US-00002 Barbara A publicly available test image. Ceria A high resolution image of heat-treated ceria (acquired with a JEOL 2100F Cs-corrected STEM) Atomic An atomic resolution image selected from the Warwick STEM dataset Nanoparticle A high-resolution image of a nanoparticle (with atomic resolution) selected from the Warwick STEM dataset

[0166] FIG. 1 shows the K-SVD generated dictionaries with elements of size [1616] for each of the different images, containing very different elements in order to reconstruct the corresponding image features; such as the dictionary generated from the Atomic labelled image, which is composed of circles (atoms) in varying positions within the patch, and that of Barbara containing the characteristic gradients, stripes and checkerboard patterns common in the test image.

[0167] As can be seen above, whilst the natural image Barbara produced a wide range of dictionary elements with varying patterns, the STEM images all produced significantly less varied dictionaries, with many atoms appearing similar. This suggests that whilst the dictionary size of 128 atoms chosen to for the experiment was appropriate for Barbara, it is seemingly too large for the three STEM images, which appear to have over-fitted dictionaries; Each of the STEM may be reconstructed to a similar quality with many fewer dictionary elements than are present, due to the reduced variety of signal patterns contained within STEM images. Clearly, if this is the case for a wide range of STEM images, the requirement for fewer atoms may be exploited in the quest for real-time reconstructions, as the time-to-solution for many of the aforementioned sparse-coding (and dictionary learning) algorithms significantly depend on the dictionary size. Additionally, if the types of signal patterns present in most STEM images require fewer dictionary elements to be accurately reconstructed, the plausibility of a master dictionary able to inpaint STEM images in most scenarios without becoming prohibitively large increases.

[0168] In FIG. 2, we show the subsampled reconstructions of the images Barbara and Atomic using both generated dictionaries. As expected, the best images arise on the diagonal i.e. where the image was inpainted using the atoms trained on its corresponding (fully-sampled) reference image. In the off-diagonal cases, while the reconstruction is successful (i.e. low-frequency/large-scale details are recovered), fine details are lost, such as the stripes in Barbara's clothing and the tablecloth, despite the reasonably high PSNR metric when compared with the diagonal case. In the case of using the Atomic dictionary to reconstruct Barbara, significant noise is also incurred, with artefacts appearing as circular spots visible in the reconstruction. Despite this, however, the results are encouraging, especially in the case of using the dictionary elements learned from Barbara to reconstruct the Atomic image. As shown in FIG. 3, the elements of the Barbara dictionary contain no circular atoms but are able to combine together to form a surprisingly accurate approximation of a given reference patch of the Atomic image. This suggests that a master dictionary with optimised elements (trained to maximally represent a large database of STEM images) may be able to quickly reconstruct a much wider range of signal patterns than one may initially suspect, from simple lines such as atomic planes, to much more complex shapes.

[0169] As illustrated in FIG. 3, a linear combination of weights and seemingly inappropriate dictionary elements from the image Barbara cumulatively form a reconstructed patch which iteratively approximates the true reference patch, shown on the right. For the full results of this experiment, including all images reconstructed with each dictionary along with the PSNR and SSIM quality metrics, see FIGS. 6 to 11 and the corresponding description.

Experiment 2: Magnification/Feature Size

[0170] For STEM images, as well as considering the different features of various potential materials one may wish to represent with a given dictionary, the microscope adds many other factors which significantly change the visual appearance of the sample, such as the chosen detector (ABF/HAADF), focus conditions and magnification. Any successful master dictionary used to inpaint the live output of a microscope must also be able to account for these microscope conditions, the most demonstrable of which is magnification, as even the same sample of material may appear vastly different at high and low magnification. Intuitively, the size of the features (in any given [1616] patch) for each image is different; for low magnification such as 3M, clusters of nanoparticles form a rough topography with fine details only in the form of atomic planes (represented in the dictionary with elements containing alternating horizontal stripes), whereas in the 12M image, fine details are drastically different as atomic columns are clearly resolved, forming many unique shapes on a much smaller scale.

[0171] In this experiment, we test the cross-compatibility of dictionaries generated for the same sample of high-resolution heat-treated Ceria at 7 different magnifications from 3M to 12M acquired with a JEOL 2100F Cs-corrected STEM.

[0172] FIG. 4 shows the K-SVD generated dictionaries with elements of size [1616] for each image of the sample at different magnifications, clearly showing the changes in learned features (and feature sizes) as the magnification is varied between 3M and 12M. For small changes in magnification, such as 12M to 10M or 3M to 4M, the set of features in the corresponding learned dictionaries appear very similar, but with a different scaling or feature-size. For example, in the case of moving from 3M to 4M, both dictionaries contain mostly horizontally aligned stripes, yet the vertical scaling of the lines for the 4M dictionary is larger, with often only one dominant line (as compared with 2 for the 3M dictionary) appearing in each dictionary element. This is consistent with expectations, as (in reference to a [1616] patch of the given image) the atomic planes appear larger (span more pixels) at a higher magnification.

[0173] With a significant increase in magnification, previously unresolved atomic columns become visible, leading to many much finer and more unique features in the image. FIG. 5 shows reconstructions of a cropped region for both the 3M and 12M images, inpainted with their own dictionary and that at the opposing end of the magnification scale.

[0174] As shown in FIG. 5, in both off-diagonal cases, while low-frequency information such as topology is recovered, most fine details are lost in reconstruction. For the 12M image, the atomic columns appear blurred, and for the 3M image, noise is again incurred, and the atomic planes are mostly absent. For the full results of this experiment, including images at all magnifications reconstructed with each dictionary along with the PSNR and SSIM quality metrics, see FIGS. 6 to 11 and the corresponding description.

Discussion

[0175] For each reconstruction performed in this paper, we have stated the PSNR (and, in FIGS. 6 to 11 and the corresponding description, the SSIM) quality metric as measured against the fully-sampled reference image used to generate the dictionary. In the first experiment (Dictionary Transfer), we investigated the performance of dictionary transfer across four different images, 3 of which were from an electron microscope. As shown in FIGS. 7 and 8, for both PSNR and SSIM, the highest quality reconstruction was achieved using the dictionary learned from the corresponding fully-sampled image (along the highlighted diagonal). In extreme cases, such as attempting to reconstruct the image Atomic using the dictionary learned from the image Ceria, the very low metric value (such as 10.99 dB) is indicative of the poor reconstruction quality using the inappropriate dictionary elements. However, as shown in FIG. 2, in the case of reconstructing Atomic with the elements learned from Barbara, while a very (visually) similar image is recovered, the PSNR value is much (6 dB) lower, suggesting a much worse reconstruction than is observed. While in this case, SSIM appeared to perform better as a predictor of reconstruction quality, it too was found to be limited as an overall metric for STEM image reconstruction analysis, especially for the results of the second experiment (Magnification/Feature Size) where (other than outliers) the difference in SSIM between the worst and the best reconstruction is often very marginal, despite the fact that in many cases, significant parts of the useful information (for STEM analysis) is completely lost. Additionally, these metrics are only able to be calculated as we have access to the fully-sampled reference images, something that is not possible in a true real-time CS-STEM setup. For this reason, much of the discussion in this paper is qualitative (i.e. looking at the reconstructed images and determining how much critical information such as atomic plane spacing is retained) and not quantitative, suggesting that a novel quality metric (perhaps specific to STEM images) may be required in order to more accurately quantity these effects and/or enable reference-less calculation, though this is outside the scope of this paper.

[0176] The results of the first experiment (Dictionary Transfer) are very encouraging, appearing to suggest that if the training data (i.e. the patches of the reference image) is varied enough, such as in the case of the image Barbara, then (as illustrated in FIG. 3) the dictionary elements are able to combine to represent features which were completely absent from the source image (e.g. atomic circles). In fact, Barbara, which produced the most varied of dictionary elements (see FIG. 1) was able to reconstruct a reasonable image for all four test images (see FIG. 6) retaining much of the useful information in all cases. One observation from the results of experiment Dictionary Transfer is the importance of feature-size, and the complexity of the elements in the learned dictionary. For dictionaries with very specific elements, such as the dictionary (of only circular signals) learned from the Atomic image, or dictionaries with many extremely similar elements such as the dictionary for the Ceria image, the performance of the dictionary when transferred to another image is limited, these two dictionaries performed the worst (in general) for all images other than their corresponding reference. As shown in FIG. 2 and FIG. 6, the use of an inappropriate dictionary, especially with elements trained to represent feature-sizes smaller than are present in the (subsampled) input image often result in noisy reconstructions with artefacts of the limited ability of the dictionary elements to represent a given patch. This is most obvious in the reconstructions of Barbara using the Atomic dictionary, resulting in a very poor reconstruction (for all fine details) and a patchwork of spots (dictionary artefacts) over the image, as well as the reconstruction of Barbara with the Ceria dictionary, which resulted in a very blurry image. However, this poor performance in dictionary transfer appears to be due to the limited range of dictionary elements required to reconstruct the types of signal patterns contained within the corresponding images; As previously discussed, this reduced complexity in reconstruction of STEM images compared with natural images may be exploited as to produce a maximally representative (whilst still remining reasonably sized) dictionary able to reconstruct a wide range of images in STEM applications, further increasing the feasibility of building a master dictionary for STEM research.

[0177] The second experiment (Magnification/Feature Size) investigated the cross-compatibility of STEM dictionaries of the same material at seven different magnifications, as an example of how live microscope conditions affect reconstruction quality. As previously discussed, FIG. 4 shows how the feature-size (and content) of the learned dictionary elements changes as the magnification of the source image is varied. FIGS. 9 to 11 show the full results and quality metrics, once again aligning with expectations, showing that (in general), dictionaries learned at the appropriate magnification produce better reconstructions. It should be noted, however, that the diagonal is much less pronounced in these results than for the first experiment, and the differences between the best and worst reconstructions is much smaller, suggesting these dictionaries are at least more cross-compatible than dictionaries from entirely different images. The results of the second experiment, while once again encouraging, suggest that any pre-learned (master) dictionary trained for real-time CS-STEM application must not only be able to represent a wide range of materials with vastly different features, but must also account for the different feature sizes required at different magnifications, even for the same material. As magnification is known at the point of acquisition, this could take the form of multiple, separate, pre-trained dictionaries for different feature sizes, instantly switched between according to the current magnification, or perhaps a method of adaptive dictionary element selection (pruning) and/or (re-)scaling in order to match the required feature size on-the-fly, potentially driven by a neural network.

[0178] The two experiments in this paper investigated the cross-compatibility of sparsifying dictionaries in general (Dictionary Transfer) and the effects of microscope magnification on the transferral of a dictionary between different images of the same sample (Magnification/Feature Size). In both cases, the results were encouraging, but highlighted aspects that would need to be considered for any true real-time CS-STEM setup. Independent of the method chosen for learning a sparsifying basis and/or reconstructing the image (K-SVD, BPFA, neural networks etc.), our results suggest that the training data required for such a master dictionary (or network) must be sufficiently extensive (resulting in a wide range of dictionary elements) in order to produce complex features, and must contain elements with the appropriate feature size, particularly in the case of STEM images which may be obtained at any magnification. However, in order to implement a master dictionary in this way, there is a lot of work yet be done. Firstly, there are other microscope conditions mentioned, but not investigated here; such as the chosen detector (ABF/HAADF) and focus condition, which also dramatically alter the appearance of the acquired image, and thus must be accounted for (i.e. representable) by the elements of the (or an alternate) master dictionary. Once each of these effects has been determined, and a strategy devised to account for each, the next step will be the formation/training of the master dictionary. At this point, a method must be determined for producing a dictionary which is representative of the widest range of STEM images whilst remaining as small (number of elements) as possible as time-to-solution often scales linearly with dictionary size. For this, we envision an extension of current dictionary learning methods, operating on a large dataset of STEM images, iteratively converging on elements which are maximally representative of image patches across the entire dataset; this algorithm would likely require a constraint to limit the visual similarity of any two elements within the dictionary, perhaps calculated via their mutual coherence, or perhaps SSIM. Further to this, our results suggest that current image quality metrics (PSNR and SSIM), are inadequate as a predictor of image quality for STEM reconstructions, often giving high values for (objectively) poor reconstructions and vice-versa. Even without these limitations, these metrics are only possible when one has access to a fully-sampled reference image, something which is not possible when performing CS-STEM on beam-sensitive materials using only subsampled acquisition. Therefore, future work on CS-STEM may require the development of an alternate reconstruction quality metric, placing more weight the preservation of fine details (such as atomic planes) rather than the low-frequency topographical features such general contrast.

[0179] When developing a master dictionary, one may also want a metric to quantify the ability of a given dictionary to represent a wide range of STEM images, likely taking into account the variety and orthogonality of the dictionary elements present, but also may benefit from an extension of the previously proposed reconstruction metric across the dataset. In order to keep the time-to-solution for live sparse-pursuit based reconstructions as small as possible, one may wish to implement an efficient dictionary pruning strategy on a per-patch basis, providing each sparse-coding step with a reduced set of dictionary elements most appropriate to the (subsampled) input patch. Previous work has already demonstrated the ability of dictionary learning methods to separate different feature-types into multiple dictionaries, referred in the literature as Cartoon and Texture features (representing the low-frequency signals and high-frequency fine details of an image respectively). This method may be leveraged to provide a single Cartoon dictionary for all images, whilst swapping between the most appropriate Texture dictionary to represent fine details for a given material, magnification or focus condition, all whilst further reducing the overall time-to-solution.

[0180] Any process of generating a master dictionary which is maximally representative of a wide range of STEM images will require the gathering of a large dataset of training data. For currently imageable materials (in STEM), this will likely be in the form of fully-sampled data, and thus, the dictionary learning algorithm may take many forms. If this process were to be successful, it appears likely from our results that this pre-trained dictionary alone may be appropriate enough to enable real-time reconstructions of these (known to be well represented) materials without the need for any further training, skipping the dictionary learning step entirely, and thus significantly reducing time-to-solution. For other materials, it may also be the case that the pre-trained dictionary (especially via the combination of many different elements, as illustrated in FIG. 3) is sufficient for the inpainting of previously unseen signals in images of beam-sensitive materials; however it is likely that in this case, some further training of the dictionary will be required in order to account for any completely novel features contained within images outside of the available training data. Even in this case however, the existence of a master dictionary will be able to reduce the time-to-solution for such reconstructions, as it may be used as the initialiser for a shortened dictionary learning process, effectively providing a hot start for the algorithm, resulting in faster reconstructions even for blind inpainting methods. Furthermore, when training such a dictionary, the specific use-case may be leveraged to produce even more efficient dictionaries for a given application. As discussed previously, in addition to the material/image(s) used in training, different microscope conditions (such as magnification) also impact the cross-compatibility of a given dictionary; this suggests that better quality reconstructions may be obtained by training dictionaries more specifically for the use-case, such as a dictionary trained to accurately reconstruct images of a silicon microprocessor at a specific magnification. In this case, after each final acquisition is taken, and subsequently reconstructed, a short dictionary learning step may be performed to further improve the given dictionary, resulting in a continuously improving master dictionary for specific applications.

[0181] Finally, as is the case for many fields involving image processing, as the speed and accuracy of neural networks continues to improve, and consumer hardware becomes more and more efficient at machine learning, we must consider the application of a neural network in place of traditional compressive sensing methods. Neural networks have already successfully been applied to image inpainting tasks, such as image denoising auto-encoders, the removal of text from an image and the use of generative adversarial networks (GANs) to inpaint large regions of an image. Similar to the concept of a master dictionary, a neural network trained on a sufficiently large dataset, able to inpaint an image (or patch) with an arbitrary mask may produce similar, if not superior results in a shorter amount of time when compared to traditional sparse-coding methods.

CONCLUSION

[0182] The aim of this paper was to perform an initial investigation into the feasibility of dictionary transfer in general and a pre-trained master dictionary for live CS-STEM. We performed two experiments to investigate how well a dictionary learned from one image is able to reconstruct another, completely different signal and the effects of varying magnification for reconstructing images of the same sample with dictionaries learned at different magnifications. Our results show the (at times unintuitive) ability of a sufficiently complex dictionary to represent completely novel features (i.e. features that were not present in the image used in training). This suggests that if an appropriately generic and representative master dictionary was formed, dictionary transfer provides a mechanism to significantly reduce the computation required to fit a dictionary, or in some cases to skip the learning stage altogether when reconstructing images of different samples, or different frames of the same sample at different microscope conditions. While much work is yet to be done, these results show that significantly reducing the time-to-solution for reconstructing subsampled STEM images via the formation of a maximally representative dictionary is possible, bringing us a step closer to a true application of real-time CS-STEM.

Real-Time Dictionary Transfer

[0183] FIG. 13A schematically depicts a new method for real-time dictionary transfer according to an exemplary embodiment, using separate training and reconstruction instances of BPFA (each with their own set of parameters and coefficients) operating on the same transient dictionary D.sub.t; and FIG. 13B shows definitions of parameters for the method of FIG. 13A.

[0184] Extending the concepts described above, a new method of achieving (now real-time) dictionary transfer is briefly as follows: [0185] An initial dictionary D.sub.0 is initialised (e.g. generated randomly or imported from a previous result) with parameters governing the dimensionality of the dictionary, and Bayesian priors denoting the probability of element selection, noise variance e.t.c. [0186] Two instances of BPFA are initialised, each with a different input source. The training instance Io takes the image we wish to transfer from as its input, while the reconstruction instance I.sub.1 takes the image we wish to transfer to as its input. Both share a connection to the same transient dictionary D.sub.t. [0187] The training and reconstruction iterations are performed sequentially/in parallel (in any order). In each iteration, the transient dictionary pixels (and parameters) are updated by the training loop, resulting in the dictionary learning the features of the transfer source Y.sub.0. Simultaneously, the statistical parameters (e.g. ) of the dictionary are updated by the reconstruction loop, allowing the dictionary to converge to a solution containing the features of Y.sub.0, and the selection probability (i.e. BPFA priors/parameters) governed by the target Y.sub.1. [0188] At any given step, a visualisation may be produced, representing the projection/reconstruction of the subsampled target as a product of the dictionary learned from the transfer source, and the appropriate coefficients determined by the reconstruction instance, thereby allowing for real-time dictionary transfer.

[0189] In more detail, the sixth aspect provides a method of real-time dictionary transfer, the method implemented by a computer comprising a processor and a memory, the method comprising: initialising an initial dictionary Do, for example wherein the initial dictionary is generated randomly or imported such as from a previous result, with parameters governing dimensionality and/or sparsity of the dictionary and/or Bayesian priors, for example denoting respective probabilities of element selection, noise variance, etc.; [0190] initialising a first instance of BPFA with a transfer source Y.sub.0 (i.e. a first input source such as a training image), wherein the first instance of BPFA comprises and/or is a training instance I.sub.0 of BPFA, for example wherein the training instance/, of BPFA takes the transfer source Y.sub.0 to be transferred from as an input thereof, having a first connection, for example a shared connection, to a transient dictionary D.sub.t derived from the initial dictionary D.sub.0; [0191] initialising a second instance of BPFA with a target source Y.sub.1 (i.e. a second input source such as a subsampled image to be inpainted), wherein the transfer source Y.sub.0 and the target source Y.sub.1 are mutually different, wherein the second instance of BPFA comprises and/or is a reconstruction instance I.sub.1 of BPFA, for example wherein the reconstruction instance I.sub.1 of BPFA takes an image to be transferred to as an input thereof, having a second connection, for example a shared connection, to the transient dictionary D.sub.t (i.e. the same transient dictionary D.sub.t); [0192] performing one or more training iterations and reconstruction iterations (also known as loops), for example serially (i.e. sequentially, consecutively, alternately) and/or in parallel (i.e. simultaneously, concurrently), using the first instance of BPFA and the second instance of BPFA, respectively, comprising, for each respective training iteration and reconstruction iteration: [0193] updating, by the first instance of BPFA (i.e. the training instance I.sub.0 of BPFA) during the training iteration (for example, during each training iteration), pixels of the transient dictionary D.sub.t (and optionally, statistical parameters thereof, for example , .sub., .sub.d, .sub.), whereby the transient dictionary D.sub.t learns the features of the transfer source Y.sub.0; and updating, by the second instance of BPFA (i.e. the reconstruction instance I.sub.1 of BPFA) during the reconstruction iteration (for example, during each reconstruction iteration), statistical parameters, for example , .sub., .sub.d, .sub.) of the transient dictionary Dt, whereby the transient dictionary D.sub.t converges, for example after a plurality of iterations, thereby containing features of the transfer source Y.sub.0 and/or selection probabilities (such as BPFA priors/parameters) governed by the target source Y.sub.1; and [0194] producing, using the transient dictionary D.sub.t, a visualisation , representing a reconstruction of the target source Y.sub.1 (i.e. reconstructing a visualisation of the target source Y.sub.1 using the transient dictionary D.sub.t).

[0195] In this way, the method provides real-time dictionary transfer, for example for real-time CS-STEM as described herein. The visualisation , representing the reconstruction of the target source Y.sub.1, is a product of the transient dictionary D.sub.t, learned from the transfer source Y.sub.0, and the appropriate coefficients determined by the reconstruction instance I.sub.1 of BPFA, thereby providing real-time dictionary transfer.

[0196] The method according to the sixth aspect may include any step and/or feature as described with respect to the first aspect and/or the second aspect.

[0197] During initialization, the following parameters are set and are not altered by either instance of BPFA: [0198] K: Size of the dictionary/No. of elements [governs dimensionality] [0199] B: Shape of the dictionary elements/patches [governs dimensionality] [0200] .sub.a, .sub.b: (shown as a, b in FIG. 13B): Hyper-parameters [governs element sparsity]

[0201] The following parameters are related to the statistical properties of the BPFA algorithm and are updated by BOTH instances (i.e. the reconstruction instance continues to fit the statistical parameters to the subsampled input, despite not affecting the dictionary elements themselves): [0202] also shown as .sub.k: Bayesian prior, essentially a measure of the global probability of selection for a given dictionary element. [0203] .sub.: Precision of the weights/coefficients [shown in in FIG. 13B] [0204] .sub.: Precision of the noise [shown as .sub.n in FIG. 13B] [0205] .sub.d: Precision of the dictionary elements [not shown in in FIG. 13B]

[0206] The initial dictionary is just used to initialise a transient dictionary D.sub.t, it could be totally randomly generated, be some form of transform dictionary e.g. DCT, in both cases, parameters are initialised to default values. Additionally, it could be loaded back in from previous experiments (along with its parameters).

[0207] In the case of reconstructing a single image (where the pixel values of the image are constant), then a dictionary will converge to a final solution and the changes to the dictionary upon each update will tend to zero, at this point we may stop training and consider it a final dictionary. However. this form of using a transient dictionary, as showed in the flow chart if FIG. 13A, is really designed for a time-variable input feed, i.e. a subsampled target such as the target source Y.sub.1, which is constantly changing, such as the output feed of an electron microscope, or a webcam. In this case, as the features of the image are continuously changing, the dictionary will continue to learn features and evolve to account for new information indefinitely, until we stop the process of viewing the reconstruction in real-time.

GLOSSARY OF TERMS

[0208] ABF: Annular Bright-Field. A method of imaging samples in STEM using bright-field detectors in which an image formed using a ring-shaped detector by low-angled forward scattered electrons, not including the most central part of the transmitted beam. [0209] CS-STEM: Compressive-Sensing (driven) Scanning Transmission Electron Microscopy. The acquisition and subsequent reconstruction of a full image of a given sample using only subsampled measurements. [0210] DCT: Discrete Cosine Transform. A transform of a signal or image from the spatial domain to the frequency domain using sinusoidal functions (in this context, discretised into a set of dictionary elements) [0211] HAADF: High-Angle Annular Dark-Field. A method of imaging samples in STEM by collecting scattered electrons with an annular dark-field detector lying outside of the path of the transmitted electron beam. [0212] K-SVD: Dictionary Learning algorithm performing generalised K-Means clustering via Singular Value Decomposition (alternating between a sparse coding step and updating individual dictionary atoms to better fit the data). [0213] I.sub.0 Norm: The count of the total number of non-zero elements of a given vector.

[00002]${.Math.e.Math.}_0=\underset{i}{.Math.}\left({.Math.e_i.Math.}^0\right)$ [0214] I.sub.1 Norm: The sum of the magnitudes of all elements in a given vector.

[00003]${.Math.e.Math.}_1=\underset{i}{.Math.}\left({.Math.e_i.Math.}^1\right)$ [0215] MOD: Method of Optimal Directions. One of the first sparse dictionary learning algorithms developed in, alternating between a sparse coding step and updating the dictionary via matrix pseudo-inverse calculations [0216] OMP: Orthogonal Matching Pursuit. [0217] PSNR: Peak-Signal-to-Noise-Ratio. An image quality metric measuring the ratio between the maximal power of a signal and the power of corrupting noise, as measured against a reference image. [0218] SSIM: Structural Similarity Index. An image quality metric measuring the visible structural similarity (between two images). [0219] BPFA instance (I.sub.X): An independent instance of the BPFA algorithm, parameters, data storage, and solution (coefficients). Each instance operates on a single input signal (Y) and dictionary (D) and attempts to find a sparse representation and/or learn an appropriate dictionary from the given input signal. Multiple instances may share the same input and/or dictionary. [0220] Arbitrary Unwrapping: Efficient generation/extraction of an arbitrary (e.g. programmatically indexed) batch of the input signal y from the full input Y. [0221] Arbitrary Wrapping: Efficient recombination of the full {circumflex over (X)} from an arbitrary set of coefficients () and the corresponding dictionary (D) [potentially forming an incomplete image, ] [0222] Traditional (BPFA) Reconstruction: Almost all previous implantations of BPFA dictionary learning and image reconstruction are designed in the following way: one (i.e. a single) instance of the BPFA algorithm is initialised, a series of dictionary learning steps/batches are performed (See FIGS. 12A and 12B). [0223] Frame Reconstruction ({circumflex over (X)}): The complete reconstruction of a given frame using a static/unchanging dictionary. (i.e. a projection of {circumflex over (X)}=D after a complete epoch of sparse-coding since the frame was received) (See FIG. 12C). (x): [0224] n.b. As the dictionary is unchanged throughout the epoch of sparse coding, all of the sparse representations of patches (coefficients of ) rely on the same dictionary (D), thus more accurately representing the target frame. [0225] Transient Dictionary (D.sub.t): A dictionary of elements which is subject to constant change of both the pixel values and parameters between iterations of the BPFA algorithm i.e. the specific dictionary seen by each batch/iteration for a given BPFA instance exists only during that batch/iteration, as the dictionary elements may/will be updated by a separate instance at regular intervals throughout each epoch of sparse coding. [0226] Visualization (): Any projection of =D.sub.t, where D.sub.t is a transient dictionary, or an inter-epoch projection of {circumflex over (X)}=D (for any dictionary), forming an incomplete reconstruction (See FIG. 12D). [0227] n.b. This means that during the course of an epoch reconstructing/visualising with a transient dictionary, rows of a calculated in previous batches may have been determined using a dictionary with different pixel values in the chosen dictionary elements. This results in a slight divergence from the original solution for each previously encoded overlapping patchthis could be a positive or negative change with respect to the overall reconstruction quality.

[0228] Although a preferred embodiment has been shown and described, it will be appreciated by those skilled in the art that various changes and modifications might be made without departing from the scope of the invention, as defined in the appended claims and as described above.

[0229] At least some of the example embodiments described herein may be constructed, partially or wholly, using dedicated special-purpose hardware. Terms such as component, module or unit used herein may include, but are not limited to, a hardware device, such as circuitry in the form of discrete or integrated components, a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks or provides the associated functionality. In some embodiments, the described elements may be configured to reside on a tangible, persistent, addressable storage medium and may be configured to execute on one or more processors. These functional elements may in some embodiments include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination. In particular, the features of any one example embodiment may be combined with features of any other embodiment, as appropriate, except where such combinations are mutually exclusive. Throughout this specification, the term comprising or comprises means including the component(s) specified but not to the exclusion of the presence of others.

[0230] Attention is directed to all papers and documents which are filed concurrently with or previous to this specification in connection with this application and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference.

[0231] All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive.

[0232] Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

[0233] The invention is not restricted to the details of the foregoing embodiment(s). The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.