dberga - Tumblr blog

[dberga]

Publication in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

https://ieeexplore.ieee.org/document/10356628

A New Framework for Evaluating Image Quality Including Deep Learning Task Performances as A Proxy

iquaflow is a framework that provides a set of tools to assess image quality. The user can add custom metrics that can be easily integrated and a set of unsupervised methods is offered by default. Furthermore, iquaflow measures quality by using the performance of AI models trained on the images as a proxy. This also helps to easily make studies of performance degradation of several modifications of the original dataset, for instance, with images reconstructed after different levels of lossy compression; satellite images would be a use case example, since they are commonly compressed before downloading to the ground. In this situation, the optimization problem involves finding images that, while being compressed to their smallest possible file size, still maintain sufficient quality to meet the required performance of the deep learning algorithms. Thus, a study with iquaflow is suitable for such case. All this development is wrapped in Mlflow : an interactive tool used to visualize and summarize the results. This document describes different use cases and provides links to their respective repositories. To ease the creation of new studies, we include a cookiecutter repository. The source code, issue tracker and aforementioned repositories are all hosted on GitHub.

https://github.com/satellogic/iquaflow

[dberga]

My mods for Kenshi!!!

I have developed new mods for Kenshi (using the object-oriented tools from the Forgotten Construction Set) and integrating Ogre's meshes and textures with Blender. Check mods in Nexus and Steam: https://next.nexusmods.com/profile/dber43/mods?gameId=736 https://steamcommunity.com/id/dber/myworkshopfiles/?appid=233860

[dberga]

Publication in Remote Sensing 15(9)

https://www.mdpi.com/2072-4292/15/9/2451 QMRNet: Quality Metric Regression for EO Image Quality Assessment and Super-Resolution

The latest advances in super-resolution have been tested with general-purpose images such as faces, landscapes and objects, but mainly unused for the task of super-resolving earth observation images. In this research paper, we benchmark state-of-the-art SR algorithms for distinct EO datasets using both full-reference and no-reference image quality assessment metrics. We also propose a novel Quality Metric Regression Network (QMRNet) that is able to predict the quality (as a no-reference metric) by training on any property of the image (e.g., its resolution, its distortions, etc.) and also able to optimize SR algorithms for a specific metric objective. This work is part of the implementation of the framework IQUAFLOW, which has been developed for the evaluation of image quality and the detection and classification of objects as well as image compression in EO use cases. We integrated our experimentation and tested our QMRNet algorithm on predicting features such as blur, sharpness, snr, rer and ground sampling distance and obtained validation medRs below 1.0 (out of N = 50) and recall rates above 95%. The overall benchmark shows promising results for LIIF, CAR and MSRN and also the potential use of QMRNet as a loss for optimizing SR predictions. Due to its simplicity, QMRNet could also be used for other use cases and image domains, as its architecture and data processing is fully scalable. -IQUAFLOW https://github.com/satellogic/iquaflow -IQUAFLOW-Modifiers: https://github.com/satellogic/iquaflow/tree/main/iquaflow/datasets - QMRNet https://github.com/satellogic/iquaflow/tree/main/iquaflow/quality_ metrics

Benchmark SiSR https://github.com/dberga/iquaflow-qmr-sisr Benchmark EO datasets https://github.com/dberga/iquaflow-qmr-eo Benchmark QMRLoss https://github.com/dberga/iquaflow-qmr-loss

[dberga]

L'entre is a band created by Guiu Vallvé and Eugeni Puigdomenech with the first album Estímul (autoedited): 9 songs that use digital and analogic elements. Some image processing effects have been created with Style transfer AI software.

See some effects in XXA and Fluxe

[dberga]

IQUAFLOW: An image quality framework   iquaflow is an image quality framework that aims at providing a set of tools to assess image quality. One of the main contributions of this framework is that it allows to measure quality by using the performance of AI models trained on the images as a proxy. The framework includes ready-to-use metrics such as SNR, MTF, FWHM or RER. It also includes modifiers to alter images (noise, blur, jpeg compression, quantization, etc). In both cases, metrics and modifiers, it is easy to implement new ones. Adittionaly, we include dataset preparation and sanity check tools and all the necessary tools to carry new experiments. Repo: https://github.com/satellogic/iquaflow Other use cases: https://github.com/dberga/iquaflow-qmr-sisr https://github.com/dberga/iquaflow-qmr-eo https://github.com/dberga/iquaflow-qmr-loss IQUAFLOW paper: https://arxiv.org/abs/2210.13269 QMRNet paper: https://arxiv.org/abs/2210.06618

[dberga]

My Arma 3 Steam Mods (download/subscribe here):  https://steamcommunity.com/id/dber/myworkshopfiles/?appid=107410 https://github.com/dberga/arma3_missions

---MASSIVE ARMA BATTLE ROYALE--- (Vanilla content-only)

100 players, all vs all. AI Survival NPCs is implemented for single and multiplayer

1. Get weapons and ammo from building loot (black squares). 2. Find a Vehicle (blue spawn). 3. Make sure to be inside the (green zone). 4. Be the last one standing

 ---Invade & Control--- (Vanilla content-only) - [NATO vs CSAT vs AAF]

-All AI and sites Support and Invade distinct regions following the Sector tactic, even AI-playable units. -Available recruits (from spawnpoint barracks or from any squad that its leader rankid is lower than yours). -Fully scripted helicopter transports (to assigned task), helicopter attacks (to each sector) and defenses (loiter). -Fully scripted Sector generation and side respawns after capture. -Fully scripted main missions (kill officer from other team, destroy control tower, destroy base of each side, download intel, etc.) -Fully scripted markers for HQ teams and units. -Fully scripted respawn points with random vehicles, trucks, tanks, helis and planes for all sides. -Fully scripted medics healing in each group. -Fully scripted ticket system updated by scoreside.

[dberga]

Publication in Neural Computation 34 (2)

Lateral connections in the primary visual cortex (V1) have long been hypothesized to be responsible of several visual processing mechanisms such as brightness induction, chromatic induction, visual discomfort and bottom-up visual attention (also named saliency). Many computational models have been developed to independently predict these and other visual processes, but no computational model has been able to reproduce all of them simultaneously. In this work we show that a biologically plausible computational model of lateral interactions of V1 is able to simultaneously predict saliency and all the aforementioned visual processes. Our model's (NSWAM) architecture is based on Pennachio's neurodynamic model of lateral connections of V1. It is defined as a network of firing rate neurons, sensitive to visual features such as brightness, color, orientation and scale. We tested NSWAM saliency predictions using images from several eye tracking datasets. We show that accuracy of predictions, using shuffled metrics, obtained by our architecture is similar to other state-of-the-art computational methods, particularly with synthetic images (CAT2000-Pattern & SID4VAM) which mainly contain low level features. Moreover, we outperform other biologically-inspired saliency models that are specifically designed to exclusively reproduce saliency. Hence, we show that our biologically plausible model of lateral connections can simultaneously explain different visual proceses present in V1 (without applying any type of training or optimization and keeping the same parametrization for all the visual processes). This can be useful for the definition of a unified architecture of the primary visual cortex.

Link to paper: https://direct.mit.edu/neco/article-abstract/34/2/378/108538/A-Neurodynamic-Model-of-Saliency-Prediction-in-V1?redirectedFrom=fulltext Preprint: https://arxiv.org/abs/1811.06308 Code: https://github.com/dberga/NSWAM

[dberga]

Publication in Pattern Recognition Letters 150 “Saliency for free: Saliency prediction as a side-effect of object recognition”

Accepted publication in PRL 150. Link to Paper: https://www.sciencedirect.com/science/article/abs/pii/S0167865521001987

Preprint: https://arxiv.org/pdf/2107.09628.pdf

[dberga]

Accepted paper in VISAPP 16(4) “Hallucinating Saliency Maps for Fine-Grained Image Classification for Limited Data Domains”

Accepted paper in “Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4 VISAPP: VISAPP” Paper: https://arxiv.org/pdf/2007.12562.pdf

[dberga]

Joined the Multimedia Technology Unit (MTU) from Eurecat

https://multimedia-eurecat.github.io/ https://eurecat.org/

Started my advanced research position at Eurecat, collaborating with public and private projects. Very happy to join the MTU image team!

[dberga]

Publication in Neurocomputing 417 “Modeling Bottom-Up and Top-Down Attention with a Neurodynamic Model of V1”

Title: Modeling Bottom-Up and Top-Down Attention with a Neurodynamic Model of V1

Abstract: Previous studies suggested that lateral interactions of V1 cells are responsible, among other visual effects, of bottom-up visual attention (alternatively named visual salience or saliency). Our objective is to mimic these connections with a neurodynamic network of firing-rate neurons in order to predict visual attention. Early visual subcortical processes (i.e. retinal and thalamic) are functionally simulated. An implementation of the cortical magnification function is included to define the retinotopical projections towards V1, processing neuronal activity for each distinct view during scene observation. Novel computational definitions of top-down inhibition (in terms of inhibition of return, oculomotor and selection mechanisms), are also proposed to predict attention in Free-Viewing and Visual Search tasks. Results show that our model outpeforms other biologically inspired models of saliency prediction while predicting visual saccade sequences with the same model. We also show how temporal and spatial characteristics of saccade amplitude and inhibition of return can improve prediction of saccades, as well as how distinct search strategies (in terms of feature-selective or category-specific inhibition) can predict attention at distinct image contexts.

Link: https://www.sciencedirect.com/science/article/abs/pii/S0925231220311553

DOI:   10.1016/j.neucom.2020.07.047

Preprints (draft version): https://arxiv.org/abs/1904.02741 https://www.biorxiv.org/content/10.1101/590174

Code and Data: https://github.com/dberga/NSWAM

[dberga]

Accepted paper in CL-ICML

Title: Disentanglement of Color and Shape Representations for Continual Learning David Berga, Marc Masana, Joost Van de Weijer https://sites.google.com/view/cl-icml/schedule-and-accepted-papers?authuser=0 Abstract: We hypothesize that disentangled feature representations suffer less from catastrophic forgetting. As a case study we perform explicit disentanglement of color and shape, by adjusting the network architecture. We tested classification accuracy and forgetting in a task-incremental setting with Oxford-102 Flowers dataset. We combine our method with Elastic Weight Consolidation, Learning without Forgetting, Synaptic Intelligence and Memory Aware Synapses, and show that feature disentanglement positively impacts continual learning performance. Links to paper: https://drive.google.com/file/d/1YMmNU1IrnD_SRMK1dC6LWqbzIZFVh1pq/view https://arxiv.org/abs/2007.06356

Link to conference presentation: https://sites.google.com/view/cl-icml/accepted-papers https://www.youtube.com/watch?v=NZLIm0uPKEY

[dberga]

Accepted in CVPR 2020

Title: MineGAN: effective knowledge transfer from GANs to target domains with few images Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, Joost van de Weijer

Abstract: One of the attractive characteristics of deep neural networks is their ability to transfer knowledge obtained in one domain to other related domains. As a result, high-quality networks can be trained in domains with relatively little training data. This property has been extensively studied for discriminative networks but has received significantly less attention for generative models. Given the often enormous effort required to train GANs, both computationally as well as in the dataset collection, the re-use of pretrained GANs is a desirable objective. We propose a novel knowledge transfer method for generative models based on mining the knowledge that is most beneficial to a specific target domain, either from a single or multiple pretrained GANs. This is done using a miner network that identifies which part of the generative distribution of each pretrained GAN outputs samples closest to the target domain. Mining effectively steers GAN sampling towards suitable regions of the latent space, which facilitates the posterior finetuning and avoids pathologies of other methods such as mode collapse and lack of flexibility. We perform experiments on several complex datasets using various GAN architectures (BigGAN, Progressive GAN) and show that the proposed method, called MineGAN, effectively transfers knowledge to domains with few target images, outperforming existing methods. In addition, MineGAN can successfully transfer knowledge from multiple pretrained GANs. Link to paper: https://openaccess.thecvf.com/content_CVPR_2020/html/Wang_MineGAN_Effective_Knowledge_Transfer_From_GANs_to_Target_Domains_With_CVPR_2020_paper.html https://arxiv.org/pdf/1912.05270.pdf

Code and Data: https://github.com/yaxingwang/MineGAN  https://github.com/dberga/MineGAN

[dberga]

Published and Presented my work in ICCV 2019

Title: SID4VAM: A Benchmark Dataset with Synthetic Images for Visual Attention Modeling David Berga, Xosé Ramón Fdez-Vidal, Xavier Otazu, Xosé M. Pardo

Abstract: A benchmark of saliency models performance with a synthetic image dataset is provided. Model performance is evaluated through saliency metrics as well as the influence of model inspiration and consistency with human psychophysics. SID4VAM is composed of 230 synthetic images, with known salient regions. Images were generated with 15 distinct types of low-level features (e.g. orientation, brightness, color, size...) with a target-distractor pop-out type of synthetic patterns. We have used Free-Viewing and Visual Search task instructions and 7 feature contrasts for each feature category. Our study reveals that state-of-the-art Deep Learning saliency models do not perform well with synthetic pattern images, instead, models with Spectral/Fourier inspiration outperform others in saliency metrics and are more consistent with human psychophysical experimentation. This study proposes a new way to evaluate saliency models in the forthcoming literature, accounting for synthetic images with uniquely low-level feature contexts, distinct from previous eye tracking image datasets. Link: http://openaccess.thecvf.com/content_ICCV_2019/html/Berga_SID4VAM_A_Benchmark_Dataset_With_Synthetic_Images_for_Visual_Attention_ICCV_2019_paper.html PDF: http://openaccess.thecvf.com/content_ICCV_2019/papers/Berga_SID4VAM_A_Benchmark_Dataset_With_Synthetic_Images_for_Visual_Attention_ICCV_2019_paper.pdf

Code and Data: http://www.cvc.uab.es/neurobit/?page_id=53

[dberga]

Presented my work in a Poster in the ACMCV 2019 Meeting

http://acmcv.cat/

Program: http://158.109.8.52/?page_id=56

[dberga]

Joined the LAMP Group from the Computer Vision Center

http://www.cvc.uab.es/LAMP/ http://www.cvc.uab.es/

I started a researcher position (postdoctoral) under the supervision of Joost van de Weijer, focusing on modeling in Deep Learning, specifically with GANs and Lifelong Learning algorithms and possible links with brain mechanisms in relation to memory consolidation.

[dberga]

Presented my work in a Poster in the ECVP 2019

https://kuleuvencongres.be/ecvp2019/Home Program: https://www.conftool.pro/ecvp2019/index.php?page=browseSessions&path=adminSessions https://kuleuvencongres.be/ecvp2019/documents/ecvp2019-abstract-book-09082019.pdf Poster PDF: https://f1000research.com/posters/8-1554