Methodology — Finding the Chinese Balloon in Satellite Images
A satellite collects images of the Earth by taking pictures in different wavelengths of light and stitching them together. The satellite photographs the same location on the ground from slightly different vantage points as it moves at high speeds overhead. An object like the Chinese balloon, floating somewhere between the satellite and the ground, looks like it is hovering over different parts of the Earth in each image.
This phenomenon, where an object looks like it is in different places depending on where you view it from, is called the parallax effect. To see how it works, hold your finger in front of you and close one eye, and then the other eye, while looking at it. Your finger appears to jump from left to right depending on which eye you keep open.
Based on this effect, Synthetaic’s founder, Corey Jaskolski, knew that the balloon would appear as several differently colored blobs in a satellite photo. He drew a sketch of what the balloon might look like and uploaded it to his company’s image analysis platform, RAIC, which stands for rapid automatic image categorization. RAIC helped him to identify objects that resembled his sketch in satellite imagery provided by Planet Labs and the European Space Agency.
A series of thumbnails of satellite images, all showing the Chinese spy balloon over different locations in the world. In most of the images, the balloon is depicted as three dots in red, green and blue close together in a line. In the top left corner is a reproduction of a digital sketch with three blobs in red, green and blue.
found by RAIC
Satellite images found by RAIC
found by RAIC
Sources: Planet Labs; European Space Agency — modified Copernicus Sentinel data via Sentinel Hub.
RAIC found the first satellite image of the balloon within two minutes. Using a tool that models air parcel trajectories, Mr. Jaskolski narrowed down where else in North America it might have flown. RAIC found the balloon six more times.
RAIC takes a novel approach to detecting objects in satellite imagery, Dr. Alemohammad, the satellite imagery expert, said. It uses an algorithm that doesn’t have to be pretrained with lots of reference images for what an object looks like. It can find objects based on a single initial sketch, and users can quickly refine the results to “teach” it to be more accurate. Dr. Alemohammad said that this approach, combined with the availability of almost daily satellite images from Planet Labs, made the balloon’s discovery possible.
It was harder for the model to estimate where and when the balloon might have been in Asia. So Mr. Jaskolski analyzed a massive volume of satellite imagery from the second half of January covering large parts of central and eastern China, North Korea, South Korea and Japan. RAIC found the balloon at five additional locations.
A series of three satellite images in black and white. The first two images show the satellite scene with the balloon in different locations. The third image is a combination of the first two images, with a line, labeled “parallax distance,” drawn between the balloon locations in the first two images.
A monochrome version of the satellite images shows how the balloon appears to be in different locations.
The balloon is photographed at the same location from separate vantage points just moments apart, making it possible to calculate its height. By layering two photographs so that their ground features align — like the river in the images above — Mr. Jaskolski measured the distance between the balloon’s apparent positions, known as the “parallax distance.”
Each satellite captured several images as it passed over the balloon. The Times and Synthetaic used these images to calculate the balloon’s altitude.
An illustration of a satellite moving at high speed over the Earth. The Chinese balloon is shown floating above the Earth underneath the satellite. Three dotted lines connect the satellite and the Earth’s surface at several points along the satellite’s path, and the lines cross at the center of the balloon.
Drawing lines connecting the satellite, the balloon and the ground and overlaying those lines creates “similar triangles” between the lines. For each image, Planet Labs provided the height and speed of the satellite, the time the image was taken and the image resolution. This information, along with the parallax distance, enabled The Times to calculate the balloon’s height using the geometric properties of similar triangles. The Times then added the ground elevation at each location to calculate the balloon’s height above sea level.