Documentation.

Getting started

You’ll need Python 3.11+ and a C++20 compiler. Clone the repository with its ML-SHARP submodule, then build from the repository root.

git clone --recurse-submodules https://github.com/frozein/SpatialPhotos.git
cd SpatialPhotos
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ./ml-sharp rectpack ninja
python -m ddgs_cpu

On Windows, use python instead of python3 and activate with .venv\Scripts\activate.bat from a Visual Studio Developer Command Prompt.

Already cloned the repository? Run git submodule update --init --recursive from its root.

Open the local viewer

Use Node.js 22.12+ (or Node.js 20.19+), then run:

cd viewer
npm install
npm run dev

Open /example/ on the local server and choose or drop your .spatial file. You can also view a file on the homepage.

Generating photos

With the virtual environment active, run predict.py from the repository root with an input image and output path. The ML-SHARP model checkpoint (about 2.8 GB) downloads automatically on first use.

python predict.py input.jpg output.spatial

Pass --checkpoint /path/to/sharp.pt to use a local checkpoint.

Process a directory

A directory input requires a directory output. Each image is saved as name.spatial.

python predict.py ./images ./spatial-photos

Generation options

python predict.py input.jpg output.spatial   --quality 85 --slices 30 --block-size 32

The web component

The spatial-photos package includes the viewer, Three.js, and TypeScript declarations. It works anywhere you can use a web component.

npm install spatial-photos

import 'spatial-photos';

<spatial-photo
  src="/photo.spatial"
  sensitivity="0.075"
  snappiness="5.5"
  style="width: 100%; height: 480px"
></spatial-photo>

On desktop, move your pointer to look around; leaving the photo returns the camera to center. On mobile, tap or drag. Set the size with CSS; without an explicit height, the component uses a 4:3 aspect ratio.

By default, the whole original photo is visible with letterboxing. Set fit="cover" to fill the component by cropping. Outfill is reserved for panning.

Styling

Style the host with CSS. The canvas, status, and progress CSS parts and --spatial-photo-background variable are available for customization.

spatial-photo {
  width: 100%;
  height: 480px;
  border-radius: 12px;
  --spatial-photo-background: #181a17;
}

Loading & events

The load(source) method accepts a URL, File, Blob, ArrayBuffer, or Uint8Array. It resolves to photo info, rejects on errors, and returns null when canceled or queued before attachment.

const photo = document.querySelector('spatial-photo');

photo.addEventListener('load', event => {
  console.log(event.detail);
});

photo.addEventListener('error', event => {
  console.error(event.detail);
});

// file is a File selected from an <input type="file">.
try {
  const info = await photo.load(file);
  console.log(info);
} catch (error) {
  console.error('Could not open photo:', error);
}

The load, error, and progress events carry data in event.detail. Progress contains { loaded, total, progress }; unknown totals are null. Removing the element cancels loading and releases its GPU resources.

The .spatial format

A spatial photo is a limited 3D representation estimated from a single image. The scene is divided into depth layers, called slices. Each slice contains a grid of small image blocks; empty blocks are omitted, and each block corner stores a depth that defines its 3D position.

Block images are packed into two texture atlases: one for RGB color and one for alpha. A .spatial file stores everything in three parts.

1. Header

   68 bytes beginning with SPA\x00. Stores expanded image, original image, and atlas dimensions, slice count, block size, camera focal length, and payload lengths.

2. Geometry

   Each slice has bitmasks for existing grid corners and blocks. Present corners store 32-bit floating-point depths; blocks store a pair of one-byte atlas coordinates measured in blocks. Shared corner depths are stored once per slice.

3. Images

   The color JPEG, followed by the grayscale alpha JPEG. Both use the same quality setting and are compressed with lossy JPEG encoding.

Multi-byte numeric fields are little-endian. See exporter.py for the binary layout and encoding.

Project structure