- Improved Interactive Medical Image Analysis Through Haptic Display
methods
Filip Malmberg, Erik Vidholm, Ingela Nyström, Ewert Bengtsson, Stefan Seipel
Partners: Lennart Thurfjell, GE Healthcare, Uppsala/London, UK; Gunnar Jansson, Dept. of Psychology, UU;
Funding: Swedish Research Council; TN-faculty, UU
Period: 0301-
Abstract: Modern medical imaging techniques provide 3D images of increasing complexity. Better ways of exploring these images for diagnostic and treatment planning purposes are needed. Combined stereoscopic and haptic display of the images form a powerful platform for such image analysis.
In order to work with specific patient cases, it is necessary to be able to work directly with the medical image volume and to generate the relevant 3D structures as they are needed for the visualization. Most work so far on haptic display use predefined object surface models. In this Project, we are creating the tools necessary for effective interactive exploration of complex medical image volumes for diagnostic or treatment planning purposes through combined use of haptic and 3D stereoscopic display techniques. The developed methods are tested on real medical application data.
Our current applications are interactive liver segmentation from CT images, see Project 36, hardware assisted visualization of breast MR images, see Project 28, and interactive segmentation of back muscles in MR images, see Project 29. A software package for interactive visualization and segmentation developed within this Project has been released under an open-source license. The package is available for download at http://www.cb.uu.se/research/haptics. In 2009, this Project was presented at the International Conference on Pattern Recognition and Information Processing (PRIP 2009), where Nyström was invited speaker.
- Analysis of Dynamic Breast MRI
Ewert Bengtsson, Ingela Nyström
Partners: Stuart Crozier, Andrew Mehnert, ITEE Dept., University of Queensland, Brisbane, Australia
Funding: TN-faculty, UU; The Australian Research Council
Period: 0503-
Abstract: The pattern of change of signal intensity over time in contrast enhanced magnetic resonance (MR) images of the breast is a useful indicator of malignancy. The methods used for assessing and visualizing this in current clinical practice are tedious; it is difficult to visualize and evaluate 4D (3D volumes over time) data effectively. During his sabbatical at the University of Queensland in 2004-2005, Bengtsson joined a new Project in this area which developed into a collaborative Project which is continuing. A basic concept for the visualization is to convert the time course for each voxel into a colour coded representation where intensity and saturation represents the uptake of contrast and hue represents the washout which is different for normal and malignant tissue. We thus obtain a colour coded volume image. A program where this is visualized with hardware accelerated maximum intensity Projection (MIP) in the hue-saturation-value (HSV) colour space has been developed. Currently the resulting visualizations are being evaluated on a larger material collected in Brisbane.
- Interactive Segmentation of Back Muscles in MR Images
Filip Malmberg, Ewert Bengtsson, Ingela Nyström
Partners: Andrew Mehnert and Craig Engstrom, ITEE Dept., University of Queensland, Brisbane, Australia
Funding: TN-faculty, UU; The Australian Research Council
Period: 0803-
Abstract: In cricket fast bowlers, substantial volume asymmetries of the quadratus lumborum (QL) muscle have been associated with a significantly increased risk of developing lumbar spine stress fractures. In this Project, we aim to measure such asymmetries by segmenting the QL muscle in MR images. At the University of Queensland, an automatic segmentation method has been developed for this purpose. The method uses a shape atlas, built from a large number of reference segmentations, to segment a specific muscle. Currently, we are investigating if interactive segmentation methods can be used to accelerate the creation of reference segmentations. By accelerating this process, we make it easier to adapt the atlas-based automatic segmentation method to other interesting muscles. A 3D visualization of a segmented QL muscle is shown in Figure 9
Figure 9:
Surface rendering of a segmented Quadratus Lumborum muscle.
|
|
- Orbit Segmentation for Craniomaxillofacial Surgery Planning
Filip Malmberg, Ewert Bengtsson, Ingela Nyström
Partners: Jan Michael Hirsch and Elias Messo, Dept. of Surgical Sciences, UU Hospital
Funding: TN-faculty, UU. NovaMedTech
Period: 0912-
Abstract: A central problem in cranio-maxillofacial (CMF) surgery is to restore the normal anatomy of the skeleton after defects, i.e., malformations, tumors and trauma to the face. This is particularly difficult when a fracture causes vital anatomical structures such as the bone segments to be displaced significantly from their proper position, when bone segments are missing, or when a bone segment is located in such a position that any attempt to restore it into its original position poses considerable risk for causing further damage to vital anatomical structures such as the eye or the central nervous system. There is ample evidence that careful pre-operative planning can significantly improve the precision and predictability and reduce morbidity of the craniofacial reconstruction. In addition, time in the operating room can be reduced.
An important component in surgery planning is to be able to accurately measure the extent of certain anatomical structures. Of particular interest in CMF surgery are the shape and volume of the orbits (eye sockets) comparing the left side with the right side. These properties can be measured in CT images of the skull, but this requires accurate segmentation of the orbits. Today, segmentation is usually performed by manual tracing of the orbit in a large number of slices of the CT image. This task is very time-consuming, and sensitive to operator errors. Semi-automatic segmentation methods could reduce the required operator time significantly.
In this Project, we are developing a prototype of a semi-automatic system for segmenting the orbit in CT images. Segmentations obtained with an early version of this system are shown in Figure 10.
Figure 10:
(Left) A segmentation of the orbit in a 2D slice from a CT volume. (Right) A preliminary 3D segmentation of the orbit in a CT volume.
|
|
- Efficient Algorithms for Computer Graphics
Ewert Bengtsson, Anders Hast
Partner: Tony Barrera, Uppsala
Funding: TN-faculty, UU
Period: 9911-
Abstract: Computer graphics is increasingly being used to create realistic images of 3D objects for applications in entertainment, (animated films, games), commerce (showing 3D images of products on the web), industrial design and medicine. For the images to look realistic high quality shading and surface texture and topology rendering is necessary. A proper understanding of the mathematics behind the algorithms can make a big difference in rendering quality as well as speed. We are in this Project re-examining some of the established algorithms and are finding new mathematical ways of simplifying the expressions and increasing the implementation speeds without sacrificing image quality. We have also invented a number of completely new algorithms. The Project is carried out in close collaboration with Tony Barrera, an autodidact mathematician. It has been running since 1999 and resulted in more than 25 international publications and a PhD thesis.
- Feeling is Believing
Ingrid Carlbom, Ewert Bengtsson, Robin Strand
Partners: Stefan Johansson (Teknovest AB), Roland Johansson, Dept, of
Neurophysiology, Umeå University; PiezoMotors AB.
Funding: Knowledge Foundation (KK Stiftelsen)
Period: 090501-
Abstract: Preliminary study of a tactile fingertip actuator array. We are building an interaction device that will form an experimental prototype for a tactile fingertip actuator array that generates a texture. See Project 33, Whole Hand Haptics with True 3D Displays.
- Whole Hand Haptics with True 3D Displays
Ingrid Carlbom, Ewert Bengtsson, Robin Strand, Martin Ericsson, Filip Malmberg, Ingela Nyström, Stefan Seipel
Partners: Jonny Gustafsson and Lars Mattson, Industrial Metrology and Optics Group, KTH; Jan-Michaél Hirsch, Dept. of Surgical Sciences, Oral & Maxillofacial Surgery, at UU and Consultant at Dept. of Plastic- and Maxillofacial Surgery, UU Hospital; Gunnar Jansson, Dept. of Psychology, UU; Stefan Johansson (Teknovest AB), Roland Johansson, Dept, of Neurophysiology, Umeå University; Håkan Lanshammar, Dept. of Information Technology, UU; PiezoMotors AB, Imagination Studios; SenseGraphics AB.
Funding: Knowledge Foundation (KK Stiftelsen)
Period: 090810-
Abstract: We are building an augmented reality system which will allow users to touch and manipulate high contrast, high resolution, three-dimensional (3D) virtual objects suspended in space using a glove that gives realistic whole hand haptic feedback both in terms of force feedback to the finger joints and tactile feedback to the fingertips. Three-dimensional visualization and haptics are very active research areas and there are numerous commercial products and research papers discussing various aspects of these topics. However, none of these systems provide anywhere near the functionality that we are aiming for in terms of haptic fidelity and of unaided 3D visual perception for multiple users. Our driving application is cranio-maxillofacial surgery, giving a surgeon the ability to plan complex procedures, moving bone and implants into their desired positions and then stretching the soft tissue to accommodate the structural changes. However, the system is not limited to this application but will significantly enhance other applications in, for example, computer-aided design, training simulators for object assembly, and 3D games. During 2009 we acquired the robot arm from which we will attach the haptic glove. We have also acquired and tested the Projectors for the display system. Finally we completed the designs of the first generation of the holographic display and as well as the first prototypes of the touch pad and one joint for the glove (see Figure 11).
Figure 11:
Drawings of the first prototypes of the touch pad and one joint for the glove.
|
|
- ProViz - Interactive Visualization of 3D Protein Images
Lennart Svensson, Stina Svensson, Ingela Nyström, Ewert Bengtsson, Ida-Maria Sintorn
Partners: Dept. of Cell and Molecular Biology, Karolinska Institute; SenseGraphics AB
Funding: The Visualization Program by Knowledge Foundation; Vaardal Foundation; Foundation for Strategic Research; VINNOVA; Invest in Sweden Agency
Period: 0807-
Abstract: The traditional methods for solving the structure of proteins are X-ray crystallography and NMR spectroscopy. An alternative approach, Molecular Electron Tomography (MET), has more recently gained interest within the field of structural biology as it enables studies of individual structures in tissue at nanometer scale, whereas the old methods only allowed studies in solution. Another important fact is that MET enables studies of the dynamics of proteins. As MET results in images of low resolution (lower than, e.g., X-ray crystallography) and low signal-to-noise ratio a new kind of software for postprocessing is required, which allows for proper visualization and analysis, as well as data fusion with measurements from X-ray crystallography and NMR. The target of the ProViz Project is to create these software tools. Our approach is to use stereo visualization and haptic rendering of the images in order to facilitate a good understanding of the data as well as interactivity in semi-automatic methods. During 2009, our focus has been on methods for automatical parameter generation for transfer functions in the volume rendering of MET data-sets. One example of a MET data set rendered through these methods can be seen in Figure 12.
In October 2009, the ProViz Project was represented on a journey to Tokyo, Japan, organized by Invest in Sweden Agency and Knowledge Foundation through the Visualization program. One part of the journey was a Visualization Seminar with oral presentations and an exhibition. The seminar was held at the Swedish Embassy and had around seventy participants from Japanese Universities and companies. During the exhibition, we visualized MET data-sets, preprocessed by methods developed in the Project to facilitate the interpretation of the data, in stereo for the first time.
Figure 12:
The Tobacco Mosaic Virus visualized according to an automatic visualization scheme, with a transfer function that consists of Gaussians and piecewise-linear primitives.
|
|
- Improved Methods for Interactive Seeded Segmentation
Filip Malmberg, Ingela Nyström, Ewert Bengtsson
Funding: TN-faculty, UU
Period: 0901-
Abstract: Image segmentation, the process of identifying and separating relevant objects and structures in an image, is a fundamental problem in image analysis. Accurate segmentation of objects of interest is often required before further processing and analysis can be performed. Despite years of active research, fully automatic segmentation of arbitrary images remains an unsolved problem.
Seeded segmentation methods attempt to solve the segmentation problem in the presence of prior knowledge in the form of a partial segmentation. Given an image where a small subset of the image elements (called seed-points) have been assigned correct segmentation labels (e.g., object or background), an automatic algorithm completes the labeling for all image elements. The seed-points may be provided either by some automatic pre-processing algorithm, or by a human user in an interactive setting.
This Project concerns the development of general tools for interactive seeded segmentation. In particular, we are studying methods based on the Image Foresting Transform (IFT). In an article published at the 13th International Workshop on Combinatorial Image Analysis (IWCIA'09), we presented the sub-pixel IFT, which allows region boundaries to be located with sub-pixel precision. An illustration of a segmentation obtained with the sub-pixel IFT is shown in Figure 13.
Figure 13:
Lateral ventricles of a human brain, segmented from an MR volume image using 20 single-voxel seed-points. A polygonal surface was extracted from the segmented volume using the Marching Cubes algorithm, which takes sub-pixel information into account. Both segmentations were produced using the same seed-points. (Left) Surface extracted from crisp IFT segmentation. (Right) Surface extracted from sub-pixel IFT segmentation.
|
|
- Interactive Organ Segmentation from Abdominal CT Images
Filip Malmberg, Ingela Nyström, Ewert Bengtsson
Partners: Sven Nilsson, Milan Golubovic, Dept. of Oncology, Radiology, and Clinical Immunology, UU Hospital
Funding: Swedish Research Council; TN-faculty, UU
Period: 0501-
Abstract: The manual step in semi-automatic segmentation of medical volume images typically involves initialization procedures, such as placement of seed-points or positioning of surface models inside the object to be segmented. The initialization is then used as input to an automatic segmentation algorithm. We investigate how such initialization tasks can be facilitated by using haptic feedback.
In this Project, we develop interactive methods for segmenting the organs from abdominal CT scans. For example, liver segmentation is of importance in hepatic surgery planning, where it is a first step in the process of finding vessels and tumours, and the classification of liver segments. Liver segmentation may also be useful for monitoring patients with liver metastases, where disease progress is correlated to enlargement of the liver.
We have developed a fully 3D liver segmentation method where high accuracy and precision is efficiently obtained via haptic interaction in a 3D user interface. Our method makes it possible to avoid time-consuming manual delineation, which otherwise is a common option prior to, e.g., hepatic surgery planning. Currently, we are incorporating and adapting region-based segmentation methods, such as the image foresting transform (IFT), into our software. Figure 14 illustrates interactive organ segmentation with IFT.
Figure 14:
(Top) A user segments organs in a CT image by placing seed-points representing organ and background, respectively. The segmentation is updated dynamically in real-time. (Bottom left) A user working with the haptic display. (Bottom right) Final segmentation of a number of organs.
|
|
- CerviScan
Ewert Bengtsson, Patrik Malm, Hyun-Ju Choi, Carolina Wählby, Bo Nordin
Funding: Vinnova
Period: 0801-
Abstract: Cervical cancer is killing a quarter million women every year. Screening based on so called PAP-smears have proven very effective to reduce cancer mortality but require much work of well trained cytotechnologist. For 50 years research to automate the screening has been in progress, Bengtsson was very active in this field 1973-1993. Since about 10 years, commercial automated systems have been in operation but unfortunately those systems have many limitations.
In India there is no effective screening program in operation and around 70,000 women die from the disease each year. Now an effort to develop a screening system adapted to Indian situation has started at the research institute CDAC in Thiruvananthapuram, Kerala in cooperation with the Regional Cancer Centre there. Based on our earlier experience in the field CBA has been invited to collaborate with this Project and we have received support from Vinnova and VR for this. We will in particular study whether the 3D chromatin texture of the cervical cells can be utilized as a robust feature for detecting (pre-) cancerous lesions.
Initial work on the Project started at CBA during 2008 but the main Project was kicked off at a meeting in India in early December 2009.
As always the image segmentation is a crucial part of solving the problem and a paper on a new approach to segmentation of cervical cells was presented at the International Symposium on Visual Computing in November 2009. One problem when developing new algorithms is to evaluate the performance and to compare the effects on different parameter settings. As a new approach to that problem a generator for synthesized PAP-smear images was created and described in a paper which was submitted to ISBI 2010.
Dr Hyun-Ju Choi from Korea has studied 3D nuclear texture for other applications and she has received a scholarship to be a visiting researcher at CBA November 2008 to November 2009, see Project 39. Her stay has been extended until May 2010 to make it possible to relate her results to this Project.
- Trabecular Bone Characterization from 3D Micro-CT Images
Cris Luengo
Partner: Manuel Herrera Lara, University of Alicante, Spain
Funding: S-faculty, SLU
Period: 0908-
Abstract: Osteoporosis is a common disease among ageing adults that
can be disabling when severe. In Europe, osteoporotic fractures are
responsible for a higher disease burden (in terms of disability and
excess mortality) than any common cancer except lung cancer. Women
over 45 spend, on average, more days in hospital due to osteoporosis
than due to many other diseases, including diabetes, breast cancer
and myocardial infarction. In the year 2000, there were an estimated
3.79 million osteoporotic fractures in Europe, one fourth of which were
hip fractures. Hip fractures are associated with serious disability
and estimated mortality rates of up to 24% in the first year after
fracture. The incidence rate has increased significantly since then,
and is expected to continue increasing due to our increased longevity.
Improved understanding of osteoporosis will lead to better screening,
diagnosis, treatment and prevention of the disease. As a contribution
to this understanding, the proposed Project aims at developing better
methods to quantify structural properties of trabecular bone from
micro-CT images. Osteoporosis affects primarily the trabecular bone,
which undergoes structural changes. These changes are the main
cause of bone fragility. Being able to characterise the trabecular
structure is therefore of great importance when studying osteoporosis.
This Project will initially provide tools for increased scientific
understanding of the disease, but these same tools will likely soon
also be applicable to clinical diagnosis, as the in vivo imaging
techniques mature.
During this year, some preliminary data was obtained, and a Project
proposal has been submitted to several funding agencies.
- 3D Texture Measures for Cell Image Analysis
Hyun-Ju Choi, Ewert Bengtsson
Funding: Korea Research Foundation and Swedish Research Council, VR
Period: 0811-1005
Abstract: Hyun-Ju Choi is a visiting researcher at CBA 2008-2010. During this time she is developing 3D texture analysis methods. The standard 2D texture analysis methods, GLCM and GLRLM, have been extended and implemented for 3D volume data. New feature sets based on fractal analysis and wavelet transform for describing quantitatively nuclear chromatin changes in 3 dimensions have been developed. The measures will during 2010 be used with real image stacks from PAP-smears to detect malignancy related changes. During 2009 earlier results from 3D texture analysis were published.
