GEOINT 2023

The availability of satellite imagery has grown enormously in recent years and the data volume is so large it is difficult to locate objects or activities of interest on a large scale. Deep learning can be used to solve this problem by training deep neural networks to analyze the imagery and perform broad area searches for specific spatial, temporal, and spectral signatures. Such networks are typically trained from a collection of annotated examples. However, the logistics of properly training such a network can be tricky, especially in the context of geospatial imagery that comes in a mix of spatial resolutions, spectral bands, revisit rates, etc. To make it easier to train and deploy geospatial search models, Kitware’s team (click here to see our subs) developed an open source software framework called WATCH (Wide Area Terrestrial Change Hypercube). Developed as part of the ongoing IARPA SMART program, WATCH aims to detecting and characterizing heavy construction sites across time using government and commercial imagery (LandSat, Sentinel-2, Maxar, and Planet). However, this software can be applied to other areas. In this training session, we will walk participants through an example of leveraging WATCH to train a broad area search model. We will also explain some of the theory and best practices for training such models and summarize some of the latest research advances in the field.

Remote sensing methods often use neural networks pretrained on large ground-level imagery datasets (e.g. Imagenet). Unfortunately, features trained from ground-level imagery show reduced discrimination on remote sensing images. It is also prohibitively costly to manually label huge datasets for remote sensing applications. In this training session, we will examine self-supervised techniques for remote sensing. Self-supervision allows automated remote sensing approaches to leverage large amounts of unlabeled data and improve accuracy across many tasks. Self-supervised techniques do not use labels and can pre-train or fine-tune models to create image embeddings that perform similarly to fully-labeled ImageNet embeddings. The techniques work particularly well for problems with few to no labels. Our team has developed and used these methods on DARPA Learning with Less Labeling (LwLL), DARPA Science of Artificial Intelligence and Learning for Open-world Novelty (SAIL-ON), and AFRL’s VIGILANT and Bomb Damage Assessment programs. Examples of the state-of-the-art techniques we will cover include contrastive learning techniques (SimCLR and SelfSim), reconstructing a masked image with Mask AutoEncoders (MAE), using GSD for superresolution masked reconstruction (ScaleMAE), and object detection-focused techniques (ReSIM and CutLER). Our focus will be effectiveness in image classification and object detection and we will touch on other applications such as perceptually lossless compression. This tutorial will introduce GEOINT attendees to the latest research on self-supervised learning in remote sensing (EBK T14. Neural Networks/Artificial Intelligence) so they can apply these techniques to improve automated image analysis results while reducing the need for expensive labeling in their projects.

The IARPA SMART program aims to research new methods and build a system to search vast catalogs of satellite images from various sources to find and characterize relevant change events as they evolve over time. Kitware is leading one of the performer teams on this program to address the problems of broad area search to find man-made change, with an initial focus on construction sites. This is a “changing needle in a changing haystack” search effort. Our system called WATCH will categorize detected sites into stages of construction with defined geospatial and temporal bounds and predict end dates for activities that are considered in progress. WATCH has been deployed on AWS infrastructure and the software has been released as open source and is for free community use. This talk will summarize our progress during the first half of the eighteen-month Phase 2 program and share improvements since our early results presented at GEOINT 2022. Notably, our detection scores on validation regions have improved dramatically since last year. We will also highlight our research results, system integration/deployment progress, and open source tools that are now available for community use.

To gather meaningful information from geospatial imagery, GEOINT analysts need to trust its authenticity. This is especially important today as we face an elevated threat of disinformation, a rapidly growing number of commercial satellite providers, and quickly evolving AI techniques for image manipulation and generation. Modern forensics technology that is specialized in overhead imagery and can detect and localize manipulations can provide analysts with the tools they need to establish trust. This presentation will highlight Kitware’s collaborative effort on the DARPA MediFor and SemaFor programs. We provide several image forensic analytics from five academic and industry partners through VIGILANT, our satellite imagery exploitation tool (developed by Kitware with funding from AFRL and NGA) that is available with unlimited rights to the government. The great variability in the pedigree of overhead images (panchromatic, multispectral, SAR, metadata, etc.) calls for a versatile suite of forensic tools that leverage domain knowledge to the fullest extent. During this talk, we will also discuss the latest developments in the image manipulation localization and characterization of image-text inconsistencies from the DARPA SemaFor program.

Performing fine-grained classification of aircraft from satellite imagery is challenging due to subtle inter-class differences and long-tailed training sample distributions. Kitware developed Low-Shot Attribute Value Analysis (LAVA) to address these unique challenges. First, we show that automatic keypoint detection using convolutional neural networks (CNNs) can effectively find the nose, tail, and wingtips of aircraft in satellite imagery. Second, because CNNs are not rotation equivariant, they are trained with data augmentation to be rotation invariant. We hypothesize this invariance restricts the ability of CNNs to learn valuable features for fine-grained classification. Instead of learned invariance, we rotate aircraft into a canonical orientation during training and testing, enabling better feature learning and improving performance. We canonicalize aircraft by determining their heading from the detected key-points placed at the nose, tail, and each wingtip. We synthetically rotate aircraft so their nose is to the upper left corner of each training and test image chip. Third, we show that breaking fine-grained object classification into attribute-based sub-classifications, in the form of predicting wing, engine, fuselage, and tail (WEFT) features, outperforms direct prediction of aircraft type while offering better explanations via WEFT attributes (e.g. 2 engines, swept wings, rounded nose). Detected keypoint-based mensuration, including the handling of variable wing sweep aircraft, is used to filter aircraft types, as aircraft may have similar WEFT features but different measured dimensions. While our results are focused on aircraft, keypoint prediction, mensuration, canonicalization, and attribute-based classification generally apply to remote sensing for fine-grained object classification.

This presentation summarizes work recently completed with NGA funding under the SAFFIRE program in which we characterized the performance of machine learning-based object detection models as a function of sensor design parameters, specifically for satellites and the resulting overhead imagery. As part of the study, Kitware extended its eXplainable Artificial Intelligence ToolKit (XAITK) to measure the performance of visual detectors over a range of sensor parameters. This extension leverages an image simulation tool called pyBSM to re-render imagery consistent with different sensor and/or platform configurations. Applying this set of tools to commercial satellite imagery, we characterized the performance of detectors for aircraft and several ground vehicle classes in the NIIRS 5-7 range. Experiments showed that deep network-based detectors have smooth performance degradations within the tested range of parameters. In a separate experiment, we attempted to validate a previously reported finding that the bit depth of overhead imagery has a significant impact on downstream detection; our experiments refuted this earlier work, as we found no significant effect from bit depth. Lastly, we used an end-to-end deep learning algorithm to select sparse multi-spectral bands from commercial satellite imagery that were co-optimized with downstream detection but found only minimal differences based on the specific bands that were selected. Kitware also performed experiments with classified imagery, using similar methodologies to those applied to unclassified data.