ImageCLEF - Experimental Evaluation in Visual Information Retrieval

Foreword

Preface

Acknowledgements

Contents

List of Contributors

Introduction

Seven Years of Image Retrieval Evaluation

Introduction

Evaluation of IR Systems

IR Test Collections

Cross--Language Evaluation Forum (CLEF)

ImageCLEF

Aim and Objectives

Tasks and Participants

Data sets

Contributions

Organisational Challenges

Conclusions

References

Data Sets Created in ImageCLEF

Introduction

Collection Creation

Requirements and Specification

Collection Overview

Image Collections for Photographic Retrieval

The St. Andrews Collection of Historic Photographs

The IAPR TC--12 Database

The Belga News Agency Photographic Collection

Image Collections for Medical Retrieval

The ImageCLEFmed Teaching Files

The RSNA Database

Automatic Image Annotation and Object Recognition

The IRMA Database

The LookThatUp (LTU) Data set

The PASCAL Object Recognition Database

The MIR Flickr Image Data Set

Image Collections in Other Tasks

The INEX MM Wikipedia Collection

The KTH--IDOL2 Database

Conclusions

References

Creating Realistic Topics for Image Retrieval Evaluation

Introduction

User Models and Information Sources

Machine--Oriented Evaluation

User Models

Information Sources for Topic Creation

Concrete Examples for Generated Visual Topics in Several Domains

Photographic Retrieval

Medical Retrieval

The Influence of Topics on the Results of Evaluation

Classifying Topics Into Categories

Links Between Topics and the Relevance Judgments

What Can Be Evaluated and What Can Not?

Conclusions

References

Relevance Judgments for Image Retrieval Evaluation

Introduction

Overview of Relevance Judgments in Information Retrieval

Test Collections

Relevance Judgments

Relevance Judging for the ImageCLEF Medical Retrieval Task

Topics and Collection

Judges

Relevance Judgment Systems and the Process of Judging

Conclusions and Future Work

References

Performance Measures Used in Image Information Retrieval

Evaluation Measures Used in ImageCLEF

Measures for Retrieval

Measuring at Fixed Recall

Measuring at Fixed Rank

Measures for Diversity

Collating Two Measures Into One

Miscellaneous Measures

Considering Multiple Measures

Measures for Image Annotation and Concept Detection

Use of Measures in ImageCLEF

Conclusions

References

Fusion Techniques for Combining Textual and Visual Information Retrieval

Introduction

Information Fusion and Orthogonality

Methods

Results

Early Fusion Approaches

Late Fusion Approaches

Inter--media Feedback with Query Expansion

Other Approaches

Overview of the Methods from 2004--2009

Justification for the Approaches and Generally Known Problems

Conclusions

References

Track Reports

Interactive Image Retrieval

Interactive Studies in Information Retrieval

iCLEF Experiments on Interactive Image Retrieval

iCLEF Image Retrieval Experiments: The Latin Square Phase

iCLEF Experiments with Flickr

The Target Collection: Flickr

Annotations

The Task

Experiments

Task Space, Technology and Research Questions

Use Cases for Interactive Image Retrieval

Challenges: Technology and Interaction

References

Photographic Image Retrieval

Introduction

Ad hoc Retrieval of Historic Photographs: ImageCLEF 2003--2005

Test Collection and Distribution

Query Topics

Relevance Judgments and Performance Measures

Results and Analysis

Ad hoc Retrieval of Generic Photographs: ImageCLEFphoto 2006-2007

Test Collection and Distribution

Query Topics

Relevance Judgments and Performance Measures

Results and Analysis

Visual Sub--task

Ad hoc Retrieval and Result Diversity: ImageCLEFphoto 2008--2009

Test Collection and Distribution

Query Topics

Relevance Judgments and Performance Measures

Results and Analysis

Conclusion and Future Prospects

References

The Wikipedia Image Retrieval Task

Introduction

Task Overview

Evaluation Objectives

Wikipedia Image Collection

Additional Resources

Topics

Relevance Assessments

Evaluation

Participants

Approaches

Results

Discussion

Best Practices

Open Issues

Conclusions and the Future of the Task

References

The Robot Vision Task

Introduction

The Robot Vision Task at ImageCLEF 2009: Objectives and Overview

The Robot Vision Task 2009

Robot Vision 2009: The Database

Robot Vision 2009: Performance Evaluation

Robot Vision 2009: Approaches and Results

Moving Forward: Robot Vision in 2010

The Robot Vision Task at ICPR2010

The Robot Vision Task at ImageCLEF2010

Conclusions

References

Object and Concept Recognition for Image Retrieval

Introduction

History of the ImageCLEF Object and Concept Recognition Tasks

2006: Object Annotation Task

2007: Object Retrieval Task

2008: Visual Concept Detection Task

2009: Visual Concept Detection Task

Approaches to Object Recognition

Descriptors

Feature Post--processing and Codebook Generation

Classifier

Post--Processing

Results

2006: Object Annotation Task

2007: Object Retrieval Task

2008: Visual Concept Detection Task

2009: Visual Concept Detection Task

Evolution of Concept Detection Performance

Discussion

Combinations with the Photo Retrieval Task

Conclusion

References

The Medical Image Classification Task

Introduction

History of ImageCLEF Medical Annotation

The Aim of the Challenge

The Database

Error Evaluation

Approaches to Medical Image Annotation

Image Representation

Classification Methods

Hierarchy

Unbalanced Class Distribution

Results

Conclusion

References

The Medical Image Retrieval Task

Introduction

Participation in the Medical Retrieval Task

Development of Databases and Tasks over the Years

2004

2005--2007

2008--2009

Evolution of Techniques Used by the Participants

Visual Retrieval

Textual Retrieval

Combining Visual and Textual Retrieval

Case--Based Retrieval Topics

Results

Visual Retrieval

Textual Retrieval

Mixed Retrieval

Relevance Feedback and Manual Query Reformulation

Main Lessons Learned

Conclusions

References

Participant reports

Expansion and Re--ranking Approaches for Multimodal Image Retrieval using Text--based Methods

Introduction

Integrated Retrieval Model

Handling Multi--modality in the Vector Space Model

Document and Query Expansion

Re--ranking

Level 1: Narrowing-down and Re-indexing

Level 2: Cover Coefficient Based Re--ranking

Results

Conclusions

References

Revisiting Sub--topic Retrieval in the ImageCLEF 2009 Photo Retrieval Task

Introduction

Background and Related Work

Sub--topic Retrieval

The Probability Ranking Principle

Beyond Independent Relevance

Document Clustering and Inter--Cluster Document Selection

Re--examining Document Clustering Techniques

Clustering for Sub--topic Retrieval

Empirical Study

Results

Conclusions

References

Knowledge Integration using Textual Information for Improving ImageCLEF Collections

Introduction

System Description

Photo Retrieval System

Medical Retrieval System

Photo Task

Using Several IR and a Voting System

Filtering

Clustering

The Medical Task

Metadata Selection using Information Gain

Expanding with Ontologies

Fusion of Visual and Textual Lists

Conclusion and Further Work

References

Leveraging Image, Text and Cross--media Similarities for Diversity--focused Multimedia Retrieval

Introduction

Content--Based Image Retrieval

Fisher Vector Representation of Images

Image Retrieval at ImageCLEF Photo

Text Representation and Retrieval

Language Models

Text Enrichment at ImageCLEF Photo

Text--Image Information Fusion

Cross--Media Similarities

Cross--Media Retrieval at ImageCLEF Photo

Diversity--focused Multimedia Retrieval

Re--ranking Top--Listed Documents to Promote Diversity

Diversity--focused Retrieval at ImageCLEF Photo

Conclusion

References

University of Amsterdam at the Visual Concept Detection and Annotation Tasks

Introduction

Concept Detection Pipeline

Point Sampling Strategy

Color Descriptor Extraction

Bag--of--Words model

Machine Learning

Experiments

Spatial Pyramid Levels

Point Sampling Strategies and Color Descriptors

Combinations of Sampling Strategies and Descriptors

Discussion

ImageCLEF 2009

Evaluation Per Image

Conclusion

ImageCLEF@ICPR 2010

Conclusion

References

Intermedia Conceptual Indexing

Introduction

Conceptual Indexing

Concept Usage and Definition in IR

Concept Mapping to Text

Mapping Steps

IR Models Using Concepts

Experiments using the ImageCLEF Collection

Image Indexing using a Visual Ontology

Image Indexing Based on VisMed Terms

FlexiTile Matching

Medical Image Retrieval Using VisMed Terms

Spatial Visual Queries

Multimedia and Intermedia Indexing

Conclusions

References

Conceptual Indexing Contribution to ImageCLEF Medical Retrieval Tasks

Introduction

Semantic Indexing Using Ontologies

Conceptual Indexing

Language Models for Concepts

Concept Detection

Concept Evaluation Using ImageCLEFmed 2005--07

From Concepts to Graphs

A Language Model for Graphs

Graph Detection

Graph Results on ImageCLEFmed 2005--07

Mixing Concept Sources

Query Fusion

Document Model Fusion

Joint Decomposition

Results on ImageCLEFmed 2005--07

Adding Pseudo--Feedback

Pseudo--Relevance Feedback Model

Results

Conclusions

References

Improving Early Precision in the ImageCLEF Medical Retrieval Task

Introduction

What is Early Precision?

Why Improve Early Precision?

ImageCLEF

Our System

User Interface

Image Database

Query Parsing and Indexing

Improving Precision

Modality Filtration

Using Modality Information for Retrieval

Using Interactive Retrieval

Conclusions

References

Lung Nodule Detection

Introduction

Lung Cancer --- Clinical Motivation

Computer--Aided Detection of Lung Nodules

Ground Truth for Lesions

Review of Existing Techniques

Gray--Level Threshold

Template Matching

Spherical Enhancing Filters

Description of Siemens LungCAD System

Lung Segmentation

Candidate Generation

Feature Extraction

Classification

Multiple Instance Learning

Exploiting Domain Knowledge in Data--Driven Training--Gated Classifiers

Ground Truth Creation: Learning from Multiple Experts

ImageCLEF Challenge

Materials and Methods

Results

Discussion and Conclusions

Clinical Impact

Future Extensions of CAD

References

Medical Image Classification at Tel Aviv and Bar Ilan Universities

Introduction

Visual Words in Medical Archives

The Proposed TAU--BIU Classification System Based on a Dictionary of Visual--Words

Patch Extraction

Feature Space Description

Quantization

From an Input Image to a Representative Histogram

Classification

Experiments and Results

Sensitivity Analysis

Optimizing the Classifier

Classification Results

Discussion

References

Idiap on Medical Image Classification

Introduction

Multiple Cues for Image Annotation

High--Level Integration

Mid--Level Integration

Low--Level Integration

Exploiting the Hierarchical Structure of Data: Confidence Based Opinion Fusion

Facing the Class Imbalance Problem: Virtual Examples

Experiments

Features

Classifier

Experimental Set--up and Results

Conclusions

References

External views

Press Association Images --- Image Retrieval Challenges

Press Association Images --- A Brief History

The Press Association

Images at the Press Association

User Search Behaviour

Types of Users

Types of Search

Challenges

Semantic Web for Multimedia Applications

Introduction to the Semantic Web

Success Stories and Research Areas

The Semantic Web Project at Press Association Images

Utilizing Semantic Web Technologies for Improving User Experience in Image Browsing

PA Data set: Linking to the Linked Data Cloud

Information Extraction and Semantic Annotation

Conclusions and Future Work

References

Image Retrieval in a Commercial Setting

Introduction

Evaluating Large Scale Image Search Systems

Query Logs and Click Data

Background Information on Image Search

Multilayer Perceptron

Click Data

Data Representation

Textual Features

Visual Features

Evaluation and Results

Analysis of Features

Discussion of Results

Looking Ahead

References

An Overview of Evaluation Campaigns in Multimedia Retrieval

Introduction

ImageCLEF in Multimedia IR (MIR)

INEX XML Multimedia Track

MIREX

GeoCLEF

TRECVid

VideOlympics

PASCAL Visual Object Classes (VOC) Challenge

MediaEval and VideoCLEF

Past Benchmarking Evaluation Campaigns

Comparison with ImageCLEF

Utility of Evaluation Conferences

Impact and Evolution of Metrics

Conclusions

References

Glossary

Index