Paper | Video | Documentation
One library for autonomous driving datasets. 123D converts raw data from Argoverse 2, nuScenes, nuPlan, KITTI-360, PandaSet, and Waymo into a unified Apache Arrow format, and then gives you a single API to read cameras, lidar, HD maps, and labels across all of them.
- 2026-09-08: We released
py123d_garage, a framework for multi-dataset training of driving policies, evaluation with NAVSIM metrics, and simulation interfaces for CARLA and AlpaSim. - 2026-06-03: Added Modalities: Semantics/instances for cameras (WOD-Perc., KITTI-360) and lidar (WOD-Perc., PandaSet, nuScenes) (experimental). Also added Radar (nuScenes, PAI-AV/NCore).
- 2026-06-03: Added Datasets: nuReasoning (experimental).
- 2026-05-08: Our paper 123D: Unifying Multi-Modal Autonomous Driving Data at Scale is out on arXiv.
- Dataset download: Fetch supported datasets from their official sources via the CLI, and optionally convert directly into the unified format.
- Hydra-based conversion CLI: YAML configs to manage your data pipelines.
- Apache Arrow storage: columnar, memory-mapped, zero-copy reads. Fast and memory efficient.
- Multiple sensor codecs: MP4/JPEG/PNG for cameras; LAZ/Draco/Arrow IPC for lidar.
- No sensor duplication: Converted logs reference source camera/lidar files via relative paths, so you don't store sensors twice.
- Unified API: Read cameras, lidar, maps, and labels through a single interface, regardless of the source dataset.
- Built-in visualization: interactive 3D viewer (Viser), and matplotlib plotting.
pip install py123dPer-dataset extras (e.g. py123d[av2], py123d[nuscenes], py123d[waymo]) install the parser dependencies for each dataset on demand. See the Demo below for an example.
Demo using the Argoverse 2 Sensor dataset, which is publicly readable from S3 and requires no cloud authentication.
The av2-sensor-stream config downloads the requested logs/maps into a managed temp directory, converts them into our self-contained Arrow format, and cleans up the source files afterwards. PY123D_DATA_ROOT controls where the converted logs/maps are written. The script below installs the Av2 extra, converts the first 3 validation logs (~250 MB each), and launches the Viser viewer:
# 1. Install
pip install py123d[av2]
export PY123D_DATA_ROOT=/path/to/py123d_data
# 2. Download + Convert
py123d-conversion dataset=av2-sensor-stream \
dataset.parser.splits='[av2-sensor_val]' \
dataset.parser.downloader.num_logs=3
# 3. Launch Viewer
py123d-viser scene_filter=av2-sensorOpen http://localhost:8080 to browse the converted scenes interactively.
| Scale | Sensors [#/Hz] | Annotations [✓/Hz] | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Year | Dur. [h] | Dist. [km] | Logs [#] | Cam. | LiDAR | 3D Box | Tls. | Map | |
| Manual | nuScenes | 2020 | 5.6 | 100.9 | 1,000 | 6 / 12 | 1 / 20 | ✓ / 2 | ✗ | ✓ |
| WOD-Perc. | 2020 | 6.4 | 154.0 | 1,150 | 5 / 10 | 5 / 10 | ✓ / 10 | ✗ | ✓ | |
| AV2-Sens. | 2021 | 4.4 | 87.5 | 1,000 | 9 / 20 | 2 / 10 | ✓ / 10 | ✗ | ✓ | |
| PandaSet | 2021 | 0.2 | 8.3 | 103 | 6 / 10 | 2 / 10 | ✓ / 10 | ✗ | ✗ | |
| KITTI-360 | 2022 | 2.7 | 73.7 | 9 | 4 / 10 | 1 / 10 | ✓ / 10 | ✗ | ✓ | |
| Auto-labeled | WOD-Mot. | 2021 | 574.1 | 10,323.5* | 103,354 | ✗ | ✗ | ✓ / 10 | ✓ / 10 | ✓ |
| nuPlan | 2024 | 1,174.3 | 17,808.6 | 15,910 | 8 / 10† | 5 / 20† | ✓ / 20 | ✓ / 20 | ✓ | |
| – nuPlan-mini | 2024 | 7.2 | 103.0 | 64 | 8 / 10† | 5 / 20† | ✓ / 20 | ✓ / 20 | ✓ | |
| PAI-AV | 2025 | 1,707.0 | 69,265.7 | 307,332 | 7 / 30 | 1 / 10 | ✓ / 10 | ✗ | ✗ | |
| – NCore | 2026 | 6.3 | 167.6 | 1,147 | 7 / 30 | 1 / 10 | ✓ / 10 | ✗ | ✗ | |
| Synth. | CARLA | 2017 | var. | var. | var. | var. | var. | ✓ | ✓ | ✓ |
| – L3AD | 2026 | 7.3 | 138.7 | 789 | 6 / 10 | 2 / 10 | ✓ / 10 | ✓ / 10 | ✓ | |
* Computed only from the non-overlapping 20 s training files. † Released for a 120 h subset; full coverage on mini.
v0.7.0 (2026-09-08)
- New modalities: per-pixel depth camera, a route modality with
SceneAPIroute storage/retrieval, tracking for derived modalities, optional GNSS solution-quality / speed-accuracy fields, and ego-state pose uncertainty (#155, #173, #177, #170, #174). - Viser viewer: reworked camera strip with real camera names, stabilized playback colors and mp4 export, camera exposure/provenance metadata, and a rigid follow-camera with an exact default viewpoint (#163, #164).
- Added optional scene indexing: custom anchor filters, lazy enumeration, and recursive log discovery (#176, #166, #157).
- Added configurable point-cloud compression (Draco/LAZ) and pluggable JPEG/PNG camera decoders (#156, #167).
- Added TruckDrive dataset support, and Griffin map support via bundled CARLA town maps.
No breaking changes to the public API, Arrow schema, or CLI entry points.
v0.6.0 (2026-06-28)
- Added radar: new
Radardatatype and Arrow radar storage, with parsers for nuScenes and PAI-AV, plus Viser visualization (#145). - Added semantic & instance segmentation: camera and lidar segmentation labels with extensible registries, parsers for nuScenes (lidar), WOD-Perception and PandaSet (semantic lidar/camera) and KITTI-360, Waymo instance maps, and Viser rendering of semantic/instance segmentation (#144).
- Added Griffin aerial-ground cooperative dataset support. (#147)(#151).
- Added nuReasoning dataset support: parser, map parser, and HuggingFace downloader (#143).
- Documentation: new
CONTRIBUTING.md, an "adding datasets" guide, per-dataset sensor-timeline plots, and a reorganized API reference (docs/api→docs/reference) (#152). - Fixes: non-strict msgpack map keys in metadata retrieval; Waymo dataset-id string (legacy convention preserved);
__slots__correctness in box-detection wrappers; bounding-box visualization consistency.
No breaking changes to the existing public API, Arrow schema, or CLI entry points. New radar and segmentation modalities follow new naming conventions.
v0.5.0 (2026-05-28)
- New
Polyline2D/PolylineSE2/PolylineSE3.subline()to extract a sub-polyline between two distances along a path. - New
MapAPI.get_layer_graph()returning anetworkx.DiGraphof lane / lane-group predecessor–successor topology. - Added
BoxDetectionsSE3.box_detections_se2for projecting SE3 detections to SE2. - Global runtime settings consolidated into
common.runtime, configured via Hydra, with configurable LRU cache sizes (e.g.max_lru_cached_tables). - Added a minimal PyTorch dataloader example (
examples/01_torch_dataloader.py). - Fixes: nuPlan map predecessor/successor naming for lanes and lane connectors;
np.nanlane speed limits; scene-sampling stride withtarget_iteration_stride; matplotlib observation rendering.
Breaking changes: OccupancyMap2D.contains_vectorized renamed to contains_points_2d (now shape-preserving); dataset_paths moved from py123d.common to py123d.common.runtime.
v0.4.0 (2026-05-19)
- Added L3AD dataset support (CARLA-derived) with a HuggingFace-hosted downloader that fetches pre-converted Arrow logs directly into
$PY123D_DATA_ROOT. - New
SceneAPI.get_modality_between_timestamps()for time-windowed retrieval. - nuPlan route and lidar tokens exposed through the custom-modality interface.
- Visualization: expanded viser color and rendering options, with configurable output resolution and frame rate.
- Parser fixes for PAI-AV and ncore: bounding box alignment and file-derived vehicle dimensions.
No breaking changes to the public API, Arrow schema, or CLI entry points.
v0.3.0 (2026-04-28)
- Refactored dataset download interface, with new download/stream options for nuScenes, PandaSet (HF mirror), AV2-sensor, WOD-perception, WOMD (#126), and nuPlan.
- Added ncore dataset support with parser, downloader, and on-the-fly conversion (#125).
- Waymo Open Motion: remaining WOMD splits, WOMD-specific fields via custom modalities, and skip-logs / skip-map options for storage-constrained runs.
- Map improvements: speed bumps as surface map objects in Waymo (#130);
align_road_edges_to_trafficfor traffic-aligned road edges (#123). - Parser fixes across PandaSet (lidar/ego poses, extrinsics), nuPlan (#128), KITTI-360 labels, nuScenes map path in stream mode, WOD motion streaming, and
pai-av/ncorelabels. - Runtime and packaging: corrected sync-table entries when inferring dynamics; Ray executor compatibility;
google-cloud-storagemoved to the Waymo extra.
Includes all fixes from v0.2.1 and v0.2.2. No breaking changes to the public API, Arrow schema, or CLI entry points.
v0.2.0 (2026-04-14)
-
Transferred repository to KE:SAI.
-
Aligned ego and agent dynamics to a unified global/ego-frame convention, with velocity/acceleration inference in
LogWriterfrom poses forEgoStateandBoxDetectionsSE3(#119, #120). -
Improved OpenDRIVE maps: 3D road-edge lifting, lane-boundary reconstruction, and cleaner map-metadata location handling (#121).
-
Parser and visualization fixes: NuScenes interpolated parser defaults to 10 Hz sync with camera-pose interpolation; PandaSet extrinsic/undistortion fixes; viser fixes (#117); new matplotlib camera-rig and lidar-reprojection utilities.
No breaking changes to the public API, Arrow schema, or CLI entry points.
v0.1.0 (2026-03-22)
- Asynchronous (native-rate) data storage: modalities are now written at their original capture rate, not just at the a frame-wise rate.
- New parser architecture with
BaseLogParser.iter_modalities_asyncfor native-rate iteration alongside the existing synchronized path. - Added NVIDIA Physical AI AV dataset support (experimental).
- Added standalone OpenDRIVE / CARLA map parser.
- Refactored
conversion/module intoparser/with consistent naming across all dataset parsers. - Refactored Viser 3D viewer. Adds more control and dark mode.
- Added
LaneType,IntersectionType,StopZoneTypeto map data structure. - Replaced Waymo heavy dependencies with lightweight protobufs.
- Various fixes to camera-to-global transforms across all datasets.
v0.0.9 (2026-02-09)
- Added Waymo Open Motion Dataset support.
- Replaced gpkg map implementation with Arrow-based format for improved performance.
- Added sensor names and timestamps to camera and Lidar data across all datasets.
- Added ego-to-camera transforms in static metadata.
- Implemented geometry builders for PoseSE2/PoseSE3 from arbitrary rotation/translation representations.
- Added support for loading merged point clouds in API.
- Improved map querying speed and OpenDrive lane connectivity handling.
- Added recommended conversion options to dataset YAML configuration files.
- Fixed PandaSet static extrinsics and KITTI-360 timestamp handling.
- Fixed memory issues when converting large datasets (e.g., nuPlan).
v0.0.8 (2025-11-21)
- Release of package and documentation.
- Demo data for tutorials.
@article{Dauner2026ARXIV,
title={123D: Unifying Multi-Modal Autonomous Driving Data at Scale},
author={Dauner, Daniel and Charraut, Valentin and Berle, Bastian and Li, Tianyu and Nguyen, Long and Wang, Jiabao and Jing, Changhui and Igl, Maximilian and Caesar, Holger and Ivanovic, Boris and Geiger, Andreas and Chitta, Kashyap},
journal={arXiv preprint arXiv:2605.08084},
year={2026}
}123D is released under the Apache License 2.0.