From 385be2eca72eb3833efa4be0a0088b34e764788a Mon Sep 17 00:00:00 2001 From: shuishen <1109946754@qq.com> Date: Mon, 31 Aug 2026 09:04:45 +0800 Subject: [PATCH] feat(pointcloud): complete annotation and result lifecycle workflows --- PROJECT_CONTEXT.md | 63 +++++++++++++++++++++++++++++-- 1 files changed, 58 insertions(+), 5 deletions(-) diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index dc9ff71..503c28e 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -1,6 +1,6 @@ # GeoAI Workbench Current Context -Last updated: 2026-08-25 +Last updated: 2026-08-28 This file is the current project snapshot for new Codex tasks. Keep it concise and replace stale facts instead of appending a conversation diary. @@ -161,7 +161,10 @@ - Start command: `py -3.12 .\scripts\serve_workbench_console.py` - URL: `http://127.0.0.1:6173` (built console); Vite development is `http://127.0.0.1:6174/apps/workbench-console/`. Only `6xxx` ports are accepted. - Scope: independent local experiment workbench for this repository. It has no code, account, or product API link to the two drone-product repositories. The map client directly requests public ArcGIS tiles and, only when configured, TianDiTu tiles. -- Current workflow: `00-change-detection`, `01-object-detection`, `02-semantic-mapping`, `04-spatial-measurement`, `09-anomaly-detection`, and `15-trajectory-analysis` support new local runs, searchable case libraries, and visual result workspaces. Object detection and change detection probe only their fixed CUDA interpreters and select CUDA when available, otherwise retain their fixed CPU fallback. Anomaly detection accepts separate normal-reference and inspection groups through bounded binary uploads, runs asynchronously, and shows references, input/output comparison, heatmaps, candidates and downloads. Change detection shows two-date imagery, a ChangeStar raster overlay, pixel-coordinate GeoJSON, per-feature probability stats and downloads. Its scan workspace accepts new user-uploaded pairs with selectable threshold/minimum-area combinations, asynchronous execution, per-combination overlays/statistics and GeoJSON, without treating scan inference as a formal model run. Spatial measurement shows raster/vector results, per-object metrics and downloads. Change detection and anomaly detection upload large raster/image files as raw binary streams and copy bytes into new run directories before fixed capability scripts run; source files are not rewritten. The server creates a new run ID for every submission, only accepts allowlisted input types and enabled tasks, and calls fixed virtual-environment scripts. +- Current workflow: `00-change-detection`, `01-object-detection`, `02-semantic-mapping`, `04-spatial-measurement`, `09-anomaly-detection`, and `15-trajectory-analysis` support new local runs, searchable case libraries, and visual result workspaces. Object detection, change detection, and point-cloud supervised model work use one fixed-environment `auto`/`cpu`/`cuda` policy: auto selects verified CUDA or records a CPU fallback reason; explicit CUDA fails rather than silently using CPU; explicit CPU never probes the GPU. New metadata records requested/actual device, environment, PyTorch version and fallback evidence. Photo reconstruction, point-cloud geometry/rule classification, file conversion, exports and the console remain CPU; current photo-MVS CUDA tools are not console-selectable. Anomaly detection accepts separate normal-reference and inspection groups through bounded binary uploads, runs asynchronously, and shows references, input/output comparison, heatmaps, candidates and downloads. Change detection shows two-date imagery, a ChangeStar raster overlay, pixel-coordinate GeoJSON, per-feature probability stats and downloads. Its scan workspace accepts new user-uploaded pairs with selectable threshold/minimum-area combinations, asynchronous execution, per-combination overlays/statistics and GeoJSON, without treating scan inference as a formal model run. Spatial measurement shows raster/vector results, per-object metrics and downloads. Change detection and anomaly detection upload large raster/image files as raw binary streams and copy bytes into new run directories before fixed capability scripts run; source files are not rewritten. The server creates a new run ID for every submission, only accepts allowlisted input types and enabled tasks, and calls fixed virtual-environment scripts. +- Result removal: every connected case library exposes a complete-result removal control. It fetches a server-derived plan and requires confirmation before deleting only a discovered console-owned `shared/outputs/<capability>/runs/<run-id>` output plus its fixed raw/processed copies and known dependent artifacts. `baseData`, external inputs, sibling runs, baselines, and validation assets remain protected. Point-cloud trained-model removal is separate and additionally clears only that model's discovered inference/automatic-annotation outputs while retaining human annotation revisions. +- Execution policy: every supported workflow retains a CPU-compatible path. The RTX 3050 is used only for verified PyTorch model/vision training or inference through fixed environments; `auto` records CUDA selection or CPU fallback, explicit `cuda` fails when unavailable, and file I/O, preprocessing, geometry/export, and the console remain CPU unless separately validated. +- Console acceptance and build policy: Codex does not drive a browser for local-console visual acceptance. It asks the user for targeted screenshots and independently runs build, HTTP, artifact, and automated checks. Run `npm run build` only after frontend source or build-time client configuration changes, not after server-only work or new data/results. - File exposure: the static handler permits only console assets, `shared/outputs`, and source images required for result comparison. Upload APIs write new raw and processed run directories but do not expose the rest of the repository. - Styling: `apps/workbench-console/src/styles.css` is organized into design variables, application shell, shared workbench components, capability workspaces, maps, and responsive rules. New capability pages must use a scoped workspace class and shared spacing variables instead of global override patches. @@ -176,20 +179,25 @@ | `07-risk-rule-engine` | Runnable CPU Demo verified | C capability: deterministic spatial zone-intersection and attribute-threshold scoring, not a built-in `geoai-py` function. It consumes CRS-aware GeoJSON inputs and produces an inspectable risk GeoTIFF, preview PNG, per-object scoring GeoJSON/CSV, summary and full run metadata. | | `09-anomaly-detection` | Runnable CPU CLI Demo verified | B capability: interpretable image features plus robust rules and Isolation Forest score visual outliers; `geoai-py` supplies `masks_to_vector`. Results are review candidates, not business-event conclusions. | | `15-trajectory-analysis` | Runnable CPU Demo verified | C capability consuming timestamped tracks: spatial metrics, DBSCAN and explicit behavior rules; not a built-in `geoai-py` function. | -| `05-3d-pointcloud` | Runnable CPU Demo verified; semantic classification baseline added | B capability: pycolmap CPU SfM plus OpenMVS CPU MVS reconstructs dense point clouds, textured meshes and browser-viewable GLB from a coherent photo sequence; Open3D plus `geoai.masks_to_vector` produces DSM/elevated-surface GeoJSON. A separate explainable CPU RGB/local-geometry baseline produces semantic review candidates, not trained-model labels. | +| `05-3d-pointcloud` | Runnable CPU Demo verified; multimodal fusion sample and semantic baseline added | B capability: pycolmap CPU SfM plus OpenMVS CPU MVS reconstructs dense point clouds, textured meshes and browser-viewable GLB from a coherent photo sequence; Open3D plus `geoai.masks_to_vector` produces DSM/elevated-surface GeoJSON. A bounded CPU photo/pose/RGB-LAS preflight and full-photo multi-view feature sample have verified the supplied 550-image ODM export can proceed to reviewed-label and spatial-split preparation, while the separate explainable RGB/local-geometry baseline produces semantic review candidates, not trained-model labels. | | `03`, `06`, `08`, `10` through `14`, `16` through `18` | Directory and initial README only | No verified local Demo yet. Start each one through `$geoai-capability-builder`. | ## Three-dimensional Reconstruction And Point-cloud Snapshot -- Scripts: `capabilities/05-3d-pointcloud/run_pointcloud_understanding.py`, `run_photo_reconstruction.py`, read-only DJI RTK/LRF diagnostic `diagnose_photo_pose_alignment.py`, and `prepare_odx_reconstruction_inputs.py`; fixture generator: `generate_validation_inputs.py`. +- Scripts: `capabilities/05-3d-pointcloud/run_pointcloud_understanding.py`, `run_photo_reconstruction.py`, `prepare_multimodal_pointcloud_dataset.py`, `diagnose_multimodal_projection.py`, `prepare_multiview_point_features.py`, read-only DJI RTK/LRF diagnostic `diagnose_photo_pose_alignment.py`, and `prepare_odx_reconstruction_inputs.py`; fixture generator: `generate_validation_inputs.py`. - Boundary: B. `geoai-py 0.42.0` has no point-cloud/SfM/MVS API in the installed package. pycolmap 4.1.1 performs CPU SIFT matching and incremental sparse SfM. Open3D 0.19.0 performs existing-point-cloud I/O, voxel filtering, RANSAC dominant-ground fitting and alpha-shape mesh construction. `geoai.masks_to_vector` receives the 0/255 elevated-surface raster; Rasterio is used as a documented completeness repair when GeoAI contour output does not cover every retained raster region. - Input/output contract: an existing PLY/PCD/XYZ/LAS/LAZ produces classified/mesh PLY, DSM and height GeoTIFFs, elevated-surface raster/preview, elevated-footprint GeoJSON, CSV and metadata. A coherent JPG/JPEG sequence produces a COLMAP sparse model, coloured PLY, PCA preview PNG, local camera poses CSV, EXIF GPS manifest and metadata with input SHA-256; the verified OpenMVS CPU branch additionally produces dense PLY, mesh PLY, texture PNGs, source GLB, a reliable-texture preview GLB, a complete-geometry neutral GLB, and a point-colour complete-geometry GLB. The reliable preview vertically flips atlas pixels for browser sampling and removes faces whose four interior samples hit black atlas background or OpenMVS's orange `RGB(255,127,39)` no-coverage colour; its JSON report records every kept/rejected count. The geometry preview retains all reconstructed mesh faces without inventing texture; the point-colour preview uses nearest fused RGB values and is not a per-face photo-texture claim. - Environment: `.venvs/05-3d-pointcloud` is a dedicated Python 3.12 environment that exposes the verified package directory from `04-spatial-measurement` through a local `.pth` file and adds Open3D 0.19.0, Laspy 2.7.0, Trimesh 4.12.2, pycolmap 4.1.1 and Matplotlib 3.11.1. It uses CPU only; `pip check` passes. +- Device policy: photo reconstruction has a complete CPU route and defaults to CPU on every computer. GPU is limited to optional dense depth estimation/fusion; image preparation, sparse SfM, meshing, texturing, GLB export, metadata, and the local console remain CPU. PyTorch CUDA works on the RTX 3050 Laptop GPU, but the tested official CUDA OpenMVS binary failed the matched six-photo validation during depth maps with CUDA error 801 and official CUDA COLMAP failed during device/context selection. CUDA MVS is therefore neither selectable in the console nor considered available; CPU is the supported route. The CLI's `--dense-device cuda` trial requires native CUDA/GPU log evidence, records requested/actual device without a silent CPU fallback, and now returns the native OpenMVS log on failure for later server or driver/toolchain verification. - Validation: normal known-geometry PLY (9,600 source points) completed in 4.177 seconds, retained 7,644 downsampled points, 2,901 elevated points, 52 footprint regions and a 15,110-triangle mesh. Difficult sparse/slope/noise PLY completed in 3.831 seconds with 73 fragmented elevated footprints, explicitly demonstrating isolated-high-point false positives. Both PNG previews and GeoJSON outputs were inspected. - Real input: PCL tutorial `table_scene_lms400.pcd` is staged unchanged at `shared/data/raw/05-3d-pointcloud/pcl-table-scene-20260821/`, SHA-256 `E285D415641E0D9DE695B611DB874CC8FE995E8089B77A50D6056D24D8CBCC58`. Its 460,400 input points completed in 4.833 seconds with processing-copy `--ground-up-axis auto`: 41,143 downsampled points, 19,281 elevated points, 193 footprints and 7,090 mesh triangles. Its source-local vertical was not Z, so a 61.1666-degree dominant-plane rotation is recorded. This validates PCD handling and axis alignment, not drone reconstruction accuracy or semantic class accuracy. - Photo validation: the user-provided 18 DJI H30T side-view JPGs were staged unchanged under `shared/data/raw/05-3d-pointcloud/h30t-sideview-20260820-v2/`. The preserved original 18-photo high-resolution baseline has 3,379,097 dense points and 799,632 faces but failed the LRF diagnostic (13.3814 m median 3D error; 11.9794 m horizontal; -4.8131 m signed vertical). The current ODX-aligned local configuration uses COLMAP `OPENCV` intrinsics, RTK/GPS spatial matching (four neighbours), 2,000 px feature extraction, 18,000 features and a 0.05 m position prior. Normal and difficult six-photo validations each registered 6/6 images with 0.7581 px and 0.7596 px mean reprojection error. The full 18-photo sparse model registered 18/18 images, has 23,084 sparse points and 0.8410 px mean reprojection error. Its 2,400 px OpenMVS CPU run completed in 2,390.086 seconds, yielding 3,756,551 dense points and a 799,645-face mesh. The reliable-texture preview retains 293,614 faces (36.72%) after black/orange-atlas rejection; the point-colour and neutral complete-geometry previews retain all 799,645 faces. The LRF diagnostic now reports 0.4038 m median 3D nearest-point distance, 0.3073 m horizontal error and 0.0709 m signed vertical error. These target observations are diagnostic only, not independent surveyed check points. A two-photo low-baseline trial failed cleanly, documenting the need for sufficient baseline. - WebODM reference: `reference_project/WebODM-master` confirms that its task/UI service delegates reconstruction to NodeODX and that the engine supports ODX `geo.txt`, GCPs, and trusted `cameras.json`/fixed-camera-parameter workflows. The local NodeODX submodule is not populated, Docker is absent, and neither NodeODX nor NodeODM is installed, so WebODM was not executed. The new read-only preparation run `shared/outputs/05-3d-pointcloud/odx-input-h30t-sideview-20260822-v2/` validated all 18 DJI images and writes an `EPSG:4326` ODX `geo.txt`, camera-position GeoJSON, source-hash manifest, parameter advice, and compatibility report. It mirrors ODX's DJI orientation normalization and per-photo RTK accuracy inflation; global ODX `gps-accuracy` must remain unset because it would overwrite those per-photo values. The package intentionally does not treat LRF targets as GCPs or fabricate a `cameras.json` calibration. -- Console: `GET/POST /api/3d-pointcloud/runs` remains enabled on the independent local console at `http://127.0.0.1:6188`. The top workflow tabs separate `照片三维重建` from `既有点云理解`: photos accept 3-30 JPG/JPEG files through bounded binary uploads, preserve raw bytes in a new run directory, use an asynchronous fixed-script CPU SfM/MVS job and poll queued/sparse/dense/completed/failed status; the GPS/RTK prior toggle selects the verified spatial-neighbour configuration only for photos with valid position metadata. Existing point-cloud uploads accept at most two allowlisted PLY/PCD/XYZ/LAS/LAZ files and use only the fixed Open3D script to produce raster/vector outputs. The case library discovers `ODX 对齐 CPU 高分重建（H30T 18 图）` from output metadata. Dense CPU-MVS results have reliable-texture, point-colour and complete-geometry modes, a wireframe toggle, local-`+Z` top-down button, filter report, dense PLY and raw mesh downloads. The geometry mode is deliberately neutral and retains every mesh face; point-colour uses fused-point RGB; texture remains evidence-only. The console only needs `npm run build` after Vue/TypeScript source changes, not after user uploads, reconstruction runs or automatic case discovery. +- Multimodal real-data preflight: `baseData/las/Task-of-2026-08-22T084705607Z-georeferenced_model.las` is a 2.58 GB, 75,870,130-point RGB LAS with `EPSG:32650`; its 550 photos, one Brown `cameras.json` calibration and 550 `shots.geojson` records have exact filename correspondence. The preflight at `shared/outputs/05-3d-pointcloud/multimodal-preflight-20260825-550-photos/` transformed every shot WGS84 geometry to `EPSG:32650` and measured 0.0 m maximum XY residual against the reported camera centre. A deterministic 120,000-point CPU LAS sample projected inside all 12 distributed calibrated photo frames (minimum 17,319; median 34,403 points), but its 1,500-point red overlays were too sparse for visual acceptance. The replacement diagnostic at `shared/outputs/05-3d-pointcloud/multimodal-projection-diagnostic-20260826-v2/` uses a one-million-point z-buffer RGB render, photo/point blend, point/edge overlay, DJI XMP gimbal-direction check, and inverse-rotation negative control. It auto-detected 8-bit RGB values stored in 16-bit LAS fields. Current ODM rotation passed: median RGB MAE 30.501 vs 60.612 inverse, 48.5% median improvement over shuffled photo locations, and 0.647-degree median gimbal-axis difference. This admits the next photo-feature/visibility stage but is not sub-pixel calibration, semantic truth, occlusion handling, or Point Transformer V3 accuracy evidence. +- Multi-view fusion sample: `shared/outputs/05-3d-pointcloud/multiview-feature-validation-20260826/` verified a 50,000-point/12-photo CPU route with 96.254% visible points and median four valid views. The full-photo sampled run at `shared/outputs/05-3d-pointcloud/multiview-feature-550-photo-sample-20260826/` then aggregated original XYZ/RGB with eight local photo descriptors (RGB, hue sin/cos, saturation, gradient, texture) over 550 photos after two-pixel closest-depth z-buffer filtering. It completed in 899.880 seconds CPU: all 50,000 sampled points were visible with at least two/three views, 99.978% had at least ten, median 154 and mean 147.2697 observations, and every feature value was finite. Its NPZ stores per-point feature mean/stddev and count but no semantic label. A new 50,000-point `multiview-annotation-source.ply` preserves original LAS RGB and has exact vertex-to-NPZ ordering, point-count and SHA-256 evidence. The console reveals it as a manual label source only after validating the fixed PLY/NPZ names, checksums and both array lengths; coverage pseudo-colours are excluded. `train_multiview_point_transformer.py` now provides a CPU-compatible 23-feature, two-stage local point-attention baseline with 16 neighbours and spatial XY train/validation/test blocks. It requires at least two classes, 500 reviewed points per class, and every class in every split; it is not official Point Transformer V3/Pointcept, whose `spconv` dependency stack is not installed or validated. A CPU synthetic 1,201-point acceptance case completed in 0.212 s and emitted a model, preview and metrics. The real 50,000-point sample remains unlabelled; field accuracy and PTv3 accuracy are not established. +- Console: `GET/POST /api/3d-pointcloud/runs` remains enabled on the independent local console at `http://127.0.0.1:6188`. The top workflow tabs separate `照片三维重建` from `既有点云理解`: photos accept 3-1000 JPG/JPEG files through bounded binary uploads, preserve raw bytes in a new run directory, use an asynchronous fixed-script CPU SfM/MVS job and poll queued/sparse/dense/completed/failed status; large photo sets can take hours or longer. The GPS/RTK prior toggle selects the verified spatial-neighbour configuration only for photos with valid position metadata. Existing point-cloud uploads accept at most two allowlisted PLY/PCD/XYZ/LAS/LAZ files and use only the fixed Open3D script to produce raster/vector outputs. The case library discovers `ODX 对齐 CPU 高分重建（H30T 18 图）` from output metadata. Dense CPU-MVS results have reliable-texture, point-colour and complete-geometry modes, a wireframe toggle, local-`+Z` top-down button, filter report, dense PLY and raw mesh downloads. The geometry mode is deliberately neutral and retains every mesh face; point-colour uses fused-point RGB; texture remains evidence-only. The console only needs `npm run build` after Vue/TypeScript source changes, not after user uploads, reconstruction runs or automatic case discovery. +- Photo jobs publish actual fixed-script workflow milestones to the console: feature extraction, matching, sparse mapping, RGB preparation, undistortion, dense fusion, mesh, texture and packaging. The displayed percentage is a weighted completed-stage indicator, not a remaining-time estimate, because COLMAP and OpenMVS do not expose reliable live percentages within each native stage. +- The fixed CPU photo-reconstruction timeout is 24 hours for large sequences. It remains a safety ceiling, not a predicted completion time. - Limitations and next decision: this is a dense/textured local model that materially improves the DJI LRF diagnostic, but it is not a survey-grade result or independent check-point validation. pycolmap dense PatchMatch still requires CUDA; the verified dense route is OpenMVS CPU. It makes no DSM, orthomosaic, GIS-coordinate, semantic-class or engineering-measurement claim. Further accuracy work needs DJI/H30T intrinsics plus GNSS antenna-to-camera lever-arm/boresight calibration, or surveyed GCPs and separate check points. A CUDA/cloud ODX/WebODM workflow is a separate option only when it consumes the prepared `geo.txt`, trusted calibration/control, and reports independent checkpoint residuals. Existing-point-cloud ground/elevated is geometric rather than building/tree/equipment classification; its alpha-shape mesh is not watertight or accuracy-validated. OpenMVS and WebODM are AGPL-3.0-only, so commercial use needs a separate license assessment; NodeODX/ODX, DJI imagery, and all source data licenses remain separate. ### Point-cloud semantic classification update @@ -376,6 +384,20 @@ - Truth contract: generated annotation PLYs preserve observed RGB/XYZ. Rule colours are never labels. Separate revisions record source SHA-256, artifact, indices, class codes and class counts below `shared/outputs/05-3d-pointcloud/annotations/`. +- Annotation workspace: users can add one local PLY/PCD/XYZ/LAS/LAZ source from + the console. The bounded binary upload retains immutable raw bytes and a + background CPU job creates a separate annotation preview before source + discovery. The local `annotation-classes.json` taxonomy starts with six + review classes and supports custom Chinese label/English key/RGB entries with + unused LAS-compatible codes. Every annotation revision snapshots its active + class definitions; used custom codes cannot be removed. This is annotation + metadata only, not an automatic semantic-model capability or labelled truth. +- Source lifecycle: the annotation workspace has a server-calculated **Remove + complete data chain** action. It requires confirmation and removes only the + fixed local output/raw/processed source chain and dependent annotations, + training models, and inference outputs. It never accepts a browser path or + deletes `baseData`/external inputs; a derived source without a console raw + copy reports that its external input is retained. A mistaken revision can be deleted after confirmation; the deletion endpoint is constrained to that revision directory and never removes source clouds, semantic outputs or training outputs. @@ -399,11 +421,42 @@ API checks passed. The annotation viewer now uses sRGB-correct anti-aliased circular points and separates browse left-drag rotation from brush/rectangle left-drag labels; right-drag rotates, middle-drag pans and the wheel zooms. + Annotation sources use direct full-point display: the viewer has no adaptive + local-detail layer, and every point in the selected source is available for + browsing and manual annotation. Point indices remain stable for saved revisions + and training. Large sources use correspondingly more browser memory and GPU + resources. + New annotation uploads now use a separate CPU RGB/XYZ preview preparation + path, so their extent does not enter the DSM/raster/vector pipeline. The + output retains every readable finite source point. Textured PLY meshes without readable + vertex RGB are retained as neutral geometry-review sources but cannot train + the current RGB semantic model. + The model workspace now provides automatic annotation of the current local RGB + source without a second upload. It runs a selected completed model + asynchronously, records source/model checksums plus per-point confidence in a + thresholded candidate artifact, and requires an explicit confirmation before + merging candidates into a new revision. Existing human labels take priority on + overlap. A 50,000-point GPU validation completed in 3.740 seconds and retained + 44,736 candidates at a 0.95 threshold; this verifies the route, not model + accuracy on the full source. + A textured-mesh RGB baking route now accepts a ZIP containing exactly one PLY + mesh and its referenced texture images, samples face-centre UV texture colour + into a new RGB/XYZ PLY, and exposes a 50,000-point validation source before a + larger preview is made. Real `Model_0.zip` validation had one + 7,430,655-vertex / 14,820,237-face mesh and 141 texture images; it baked + 50,000 RGB samples in 174.921 CPU seconds, using 140 textures, with 80.17% + non-grey colours. A full direct-display source completed in 275.120 seconds + with all 14,820,237 textured mesh faces represented as RGB face-centre points + (222,303,737-byte PLY). User visual review of UV alignment remains required. Middle-button auto-scroll is suppressed and a rotate-centre tool places the orbit target at a clicked visible point without writing a label. A model-orientation popover provides independent local X/Y/Z display rotation from 0-360 degrees about the model centre; it leaves source coordinates and stored annotation indices unchanged while selection follows the display transform. + Sparse SfM, semantic, supervised-prediction, and mesh preview viewers share + that orientation control, click-to-set orbit centre, top-down, reset, and + middle-button pan without browser auto-scroll. These are display-only + transforms and do not change source or downloadable artifacts. Console/unit/type checks passed. Visual annotation review awaits a user screenshot under the no-browser-inspection rule. - Applied-model workflow: the supervised area now discovers only complete local -- Gitblit v1.9.3