End-to-end scene text recognition pipelines are commonly based in a two-stage approach, first applying a text localization algorithm to the input image and then recognizing the text present in the cropped bounding boxes provided by the detector.

In this project we study the use of single shot CNN architectures for scene text understanding tasks. For example, a Fully Convolutional Network that, given a scene image, is able to predict at the same time bounding boxes and a compact text representation of the words within them; or a CNN architecture for text spotting that is able to directly output readings without any explicit text localization step, and can be trained in an end-to-end manner.

Topic modeling frameworks, such as the Latent Dirichlet Allocation (LDA) algorithm, are statistical models for discovering the the latent topics that occur in a corpus of textual documents. This way, each individual text document can be represented as a probability distribution over the set of discovered topics, and thus can be projected to a point in a topic space.

Our research puts forward the idea of embedding images into text topic spaces by mining a large scale collection of multi-modal (text and image) documents. To do so we first learn a topic model on the text corpus of a dataset composed by pairs of correlated texts and images. Then, we train a deep CNN model to predict text representations (topic-probabilities) directly from the image pixels. In other words the learned topic model teaches the CNN to predict the semantic context of images.

This deep-embedding framework can be used to perform different tasks, such as self-supervised learning of visual features, multi-modal image retrieval, or even to generate contextualized lexicons for scene text recognition.

Text Proposals have emerged as a class-dependent version of object proposals -- efficient approaches to reduce the search space of possible text object locations and extents in an image. Combined with strong word classifiers, text proposals currently yield top state-of-the-art results in end-to-end scene text recognition. In this paper we propose an improvement over the original Text Proposals algorithm (Gomez and Karatzas 2015), combining it with Fully Convolutional Networks to improve the ranking of proposals. Results on the ICDAR RRC and the Coco-text datasets show superior performance over the current state-of-the-art.

This work focuses on the problem of script identification in scene text images. Facing this problem with state of the art CNN classifiers is not straightforward, as they fail to address a key characteristic of scene text instances: their extremely variable aspect ratio. Instead of resizing input images to a fixed aspect ratio as in the typical use of holistic CNN classifiers, we propose here a patch-based classification framework in order to preserve discriminative parts of the image that are characteristic of its class.

We describe a novel method based on the use of ensembles of conjoined networks to jointly learn discriminative stroke-parts representations and their relative importance in a patch-based classification scheme. Our experiments with this learning procedure demonstrate state-of-the-art results in two public script identification datasets.

Optical Character Recognition (OCR) is nowadays considered a solved problem when a clean binarized and well formatted input image, with text in a standard font and language, is provided. On the contrary, the automated localization, extraction and recognition of "scene text" in uncontrolled environments is still an open Computer Vision problem. At the core of the problem lies the extensive variability of scene text in terms of its location, rotation, physical appearance and design.

Scene text extraction methodologies have been traditionally based in classification of individual regions or patches, using a priori knowledge for a given script or language. Human perception of text, on the other hand, is based on perceptual organisation through which text emerges as a perceptually significant group of atomic objects. Therefore humans are able to detect text even in languages and scripts never seen before. My research revolves around these ideas and poses the text extraction problem as the detection of meaningful groups of regions. I'm working in a text detection method built around a perceptual organisation framework that exploits collaboration of proximity and similarity laws to create text-group hypotheses.