From fbb068ec702338d609c1ca6eddbdb9f182d8f211 Mon Sep 17 00:00:00 2001 From: shuishen <1109946754@qq.com> Date: Mon, 24 Aug 2026 11:37:16 +0800 Subject: [PATCH] feat: extend local GeoAI capability workflows --- capabilities/05-3d-pointcloud/README.md | 423 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 419 insertions(+), 4 deletions(-) diff --git a/capabilities/05-3d-pointcloud/README.md b/capabilities/05-3d-pointcloud/README.md index 9ec374c..176cfa2 100644 --- a/capabilities/05-3d-pointcloud/README.md +++ b/capabilities/05-3d-pointcloud/README.md @@ -1,6 +1,421 @@ -# 3D and Point Cloud +# Three-dimensional reconstruction and point-cloud understanding -- 输入:重叠照片、LAS/LAZ/PLY 点云或现有三维成果。 -- 输出:点云、网格、DSM、分类或构件结果。 -- 首个 Demo:先分析已有 ODM/点云成果,不在本机从零训练重型模型。 +## Boundary +Classification: **B - ecosystem combination**. + +`geoai-py 0.42.0` has no installed API for SfM/MVS reconstruction or point-cloud +classification. This Demo uses pycolmap/COLMAP for CPU sparse SfM and OpenMVS for +CPU dense MVS, meshing and texturing from a coherent photo sequence, plus Open3D +for existing point-cloud geometry. `geoai.masks_to_vector` +turns an elevated-surface label raster into the GeoJSON footprint output. Open3D, +Rasterio, GeoPandas, Laspy, and pycolmap supply the remaining work. This is not a +built-in `geoai-py` 3D reconstruction function. + +## Runnable Demo + +Input: one existing `PLY`, `PCD`, `XYZ`, `LAS`, or `LAZ` point cloud. The first +geometry branch consumes a reconstruction output and creates inspectable raster +and vector outputs. The photo branch accepts a coherent JPG/JPEG sequence and +creates a CPU sparse SfM point cloud. A user-supplied survey, reconstruction +export, or image sequence is copied unchanged into a raw run directory before a +console run is processed. + +```powershell +$py = .\.venvs\05-3d-pointcloud\Scripts\python.exe +& $py .\capabilities\05-3d-pointcloud\run_pointcloud_understanding.py ` + --input .\shared\data\raw\05-3d-pointcloud\validation-20260821\normal_site.ply ` + --output .\shared\outputs\05-3d-pointcloud\validation-normal-20260821 +``` + +Photo sparse reconstruction: + +```powershell +$py = .\.venvs\05-3d-pointcloud\Scripts\python.exe +& $py .\capabilities\05-3d-pointcloud\run_photo_reconstruction.py ` + --input .\shared\data\raw\05-3d-pointcloud\h30t-sideview-20260820-v2 ` + --output .\shared\outputs\05-3d-pointcloud\real-h30t-sideview-20260821-v2 +``` + +CPU dense MVS after a verified sparse model (the original source images remain +unchanged; the output directory must be new): + +```powershell +$py = .\.venvs\05-3d-pointcloud\Scripts\python.exe +& $py .\capabilities\05-3d-pointcloud\run_cpu_dense_reconstruction.py ` + --input .\shared\data\raw\05-3d-pointcloud\h30t-sideview-20260820-v2 ` + --sparse-model .\shared\outputs\05-3d-pointcloud\real-h30t-sideview-20260821-v2\sparse_model\0 ` + --output .\shared\outputs\05-3d-pointcloud\cpu-mvs-full-<new-run-id> ` + --openmvs-bin .\shared\tools\openmvs-2.4.0\vc17\x64\Release ` + --threads 12 --max-resolution 2400 --dense-resolution-level 0 ` + --dense-number-views 8 --dense-number-views-fuse 2 --target-faces 800000 +``` + +WebODM/ODX-compatible RTK input preparation (metadata only; it neither starts +WebODM nor reconstructs a model): + +```powershell +$py = .\.venvs\05-3d-pointcloud\Scripts\python.exe +& $py .\capabilities\05-3d-pointcloud\prepare_odx_reconstruction_inputs.py ` + --input .\shared\data\raw\05-3d-pointcloud\h30t-sideview-20260820-v2 ` + --output .\shared\outputs\05-3d-pointcloud\odx-input-h30t-sideview-<new-run-id> ` + --webodm-reference .\reference_project\WebODM-master +``` + +The generated package contains `geo.txt` in ODX's `EPSG:4326` format, a +camera-position GeoJSON, a source-hash manifest, recommended ODX parameters, +and an explicit compatibility report. It reproduces the referenced ODX DJI +conversion: `FlightYawDegree`, `90 + GimbalPitchDegree`, and +`GimbalRollDegree`, with the reported RTK deviations doubled as ODX does. +Leave ODX's global `--gps-accuracy` unset for this package because it would +overwrite the per-photo horizontal/vertical deviations in `geo.txt`. + +The CPU pipeline performs voxel downsampling, RANSAC dominant-ground fitting, +height-above-ground labelling, alpha-shape mesh reconstruction, rasterization, +and GeoAI footprint vectorization. The output directory must be new or empty. + +Use `--ground-up-axis auto` only when the source-local Z axis is not vertical; +it rotates a processing copy from the dominant RANSAC plane and records that +rotation. The raw point cloud is never changed. + +## Input and output contract + +Existing-point-cloud input yields `*.classified.ply`, an alpha-shape +`*.reconstruction.ply` when the geometry permits it, DSM and height GeoTIFFs, +an elevated-surface GeoTIFF, a PNG raster preview, elevated-footprint GeoJSON, +summary CSV, and fixed `run_metadata.json`. + +Photo input first yields a COLMAP sparse model, coloured `sparse_points.ply`, a +principal-axis PNG preview, `camera_poses_local.csv`, an EXIF GPS input manifest, +and fixed `run_metadata.json`. The verified CPU dense branch additionally yields +`dense.ply`, a triangle `mesh.ply`, OpenMVS texture atlas PNGs, an original +`textured.glb`, a self-contained `textured_preview_filtered.glb` for reliable +texture inspection, and `geometry_preview.glb` for complete-surface inspection. +The reliable-texture preview flips atlas pixels vertically to match browser sampling +while preserving OpenMVS UV coordinates, then removes any triangle whose four +interior samples reach black atlas background or OpenMVS's orange +`RGB(255,127,39)` no-coverage colour. `textured_preview_filter_report.json` +records retained and rejected faces. `geometry_preview.glb` keeps every mesh face +with a neutral material: it does not invent colour or texture and separates +geometric coverage from texture coverage. It records the registered +images, sparse/dense point counts, mesh counts, source hashes and CPU parameters. +When an OpenMVS dense PLY contains fused RGB, `point_color_preview.glb` also +assigns each complete-mesh vertex the RGB of its nearest fused point. This keeps +all mesh faces and uses observed point colours, but it is a nearest-point colour +projection rather than a per-face photo texture. + +Default local-coordinate thresholds: voxel size `0.20`, RANSAC plane distance +`0.15`, elevated height `0.75`. Review them against the source units and density. + +## Coordinates and limitations + +PLY/PCD/XYZ input normally has no declared CRS. Its output is explicitly +`local_point_cloud_coordinates`; GeoTIFF/GeoJSON coordinates are not geographic +coordinates, metres, or survey-grade locations. LAS/LAZ source CRS is not +promoted in this first Demo, so it is also treated as local. + +The generated `validation-*` PLYs are known-geometry test fixtures, not field +data or an accuracy dataset. `normal_site.ply` checks a dense flat site; +`difficult_sparse_site.ply` checks sparse tilted terrain and isolated high +noise. The difficult case is expected to show false elevated cells around noise: +it validates handling and output, not semantic accuracy. + +Photo SfM uses `SIMPLE_RADIAL` camera parameters. Position priors can preserve +the RTK-scale camera trajectory, but neither camera calibration nor exterior +orientation is survey-grade. pycolmap's dense PatchMatch requires CUDA and +remains unavailable, but the separate OpenMVS 2.4.0 Windows CPU binary has been +verified for dense depth fusion, meshing and texturing. This does not create a +DSM, orthomosaic, semantic class, GIS coordinate or survey-grade measurement +claim. + +Real DJI H30T validation: 18 side-view photos from `2026-08-20` completed CPU +sparse SfM in 140.832 seconds. All 18 photos registered into one camera group; +the result has 11,975 coloured sparse points and 1.1992 px mean reprojection +error. A six-photo representative CPU MVS check completed depth fusion, meshing +and texturing in 112 seconds (232,060 dense points, 199,970 faces). The full +18-photo CPU MVS run then completed in 461 seconds: 681,788 dense points and a +399,932-face textured mesh. It uses a 1,600 px dense-image cap and 12 CPU +threads, so it is a verified CPU baseline rather than a maximum-quality GPU run. +A two-photo low-baseline check failed cleanly, documenting the need for sufficient +baseline. Reliable-texture reprocessing of the full 18-photo output reuses the +completed dense point cloud, disables mesh hole closing/smoothing, requests no +texture outlier removal, and filters the browser preview as above. The new raw +mesh has 346,878 faces; 128,772 faces (37.12%) pass the four-sample atlas check. +The remaining faces are deliberately absent from the preview because the photos +do not provide consistent texture there. This avoids orange/black placeholders, +but does not create a watertight model or improve the original coverage. + +High-resolution CPU verification of the same 18 photos completed in 2,341.305 +seconds at 2,400 x 1,774 depth maps, level 0, eight neighbouring views, and a +two-view fusion threshold. It produced 3,379,097 dense points and a 799,632-face +mesh, but only 293,229 faces (36.67%) passed the reliable-texture check. The mesh +itself has one 799,646-face main connected component, so the dominant visible +holes in reliable-texture display are coverage rejections, not deletion of the +geometric surface. The console exposes both modes; this does not prove that the +complete geometry is accurate in vegetation or occluded areas. +The same high-resolution result also has a 799,632-face point-colour preview; +it is useful for inspecting coloured complete geometry, but must not be treated +as an image-texture accuracy claim. + +Metric diagnostic for the RTK/fixed-focal high-resolution result rejects it for +height or measurement use. The read-only +`diagnose_photo_pose_alignment.py` script compares source DJI RTK/LRF XMP with +a sparse model and optional OpenMVS `dense.ply`. For all 18 LRF target points, +the latest dense cloud has a 13.3814 m median nearest-point distance, 11.9794 m +median horizontal error, and -4.8131 m median signed vertical error. The RTK +camera trajectory itself is materially better (0.5978 m median fit error), so +the dense failure is not a viewer/texture issue. A fixed-pose sparse trial using +RTK camera centres plus LRF target directions kept camera centres at 0 m error +but reached 12.0193 px mean reprojection error, compared with 1.2105 px for the +RTK-constrained SfM baseline. It therefore rejects the uncalibrated assumptions +that the GNSS receiver is the optical centre and the LRF target is the RGB +principal point; it must not be used as an MVS input. + +## ODX-aligned CPU result + +The original result above remains a preserved baseline. A new local sparse +model uses an ODX-style configuration: `OPENCV` intrinsics (the local analogue +of ODX brown calibration), RTK/GPS spatial matching with four neighbours, +2,000 px feature extraction, 18,000 features, and a 0.05 m position-prior +loss scale. Both a normal six-photo set and a difficult six-photo set registered +6/6 images, with 0.7581 px and 0.7596 px mean reprojection error respectively. +The complete 18-photo model registered 18/18 images with 23,084 sparse points +and 0.8410 px mean reprojection error. + +Its CPU OpenMVS run at 2,400 px, eight views and two-view fusion completed in +2,390.086 seconds. It has 3,756,551 dense points and a 799,645-face mesh. The +reliable-texture browser GLB retains 293,614 faces (36.72%) after rejecting +black/orange atlas placeholders; the point-colour and neutral complete-geometry +GLBs both retain all 799,645 faces. It is available in the case library as +`ODX 对齐 CPU 高分重建(H30T 18 图)`. + +The read-only DJI RTK/LRF diagnostic reports a 0.4038 m median nearest dense +point distance for all 18 targets, with 0.3073 m horizontal and 0.0709 m signed +vertical median error. This is a material improvement over the preserved old +baseline, but the LRF observations are used here as a diagnostic rather than +independent surveyed check points. The result is suitable for visual comparison +with WebODM and continuing reconstruction evaluation; it is not a survey-grade +DSM, orthomosaic, GIS-coordinate, semantic-class, or engineering measurement +claim. + +## WebODM/ODX reference and acceptance gate + +The local `reference_project/WebODM-master` was inspected to understand the +comparison output. WebODM is the task/UI service; the current reference's +default processing node is `webodm/nodeodx` (NodeODX), which runs the actual +OpenSfM/OpenDroneMap pipeline. It accepts `geo.txt`, GCPs, and a verified +`cameras.json`; it can also lock verified camera parameters during bundle +adjustment. These are substantive inputs, not display settings that can be +copied into the current COLMAP/OpenMVS code. + +The new `prepare_odx_reconstruction_inputs.py` produces a checked input package +from DJI metadata. For the supplied H30T sequence, the verified package at +`shared/outputs/05-3d-pointcloud/odx-input-h30t-sideview-20260822-v2/` contains +18 camera records. Its median per-photo ODX horizontal/vertical observations +are 0.00472 m and 0.01162 m respectively. It does not create a guessed +`cameras.json`, and it excludes LRF target observations from GCP input. + +This workstation has neither Docker nor NodeODX/NodeODM available, so no claim +is made that WebODM or ODX was executed locally. Before sending this photo set +to an ODX worker or cloud service, provide one of: a camera-specific H30T RGB +calibration plus antenna/camera lever-arm and boresight; or at least three +surveyed stable GCPs, each marked in two or more images, with separate check +points. Accept a resulting model only after it reports independent checkpoint +residuals and they meet the intended measurement tolerance. + +Further accuracy improvement requires camera-specific H30T calibration and +antenna-to-camera lever-arm/boresight data, or at least three stable surveyed +3D control points marked in two or more images with separate check points. A +uniform height translation is not a valid replacement for those inputs. + +The sparse PLY contains only `element vertex 11975`; it has no faces or normals. +A viewer that triangulates it as a surface creates false sheets or spikes. The +console renders sparse data strictly as coloured `THREE.Points`; dense MVS cases +offer reliable-texture and complete-geometry modes with orbit, zoom, reset and +optional wireframe. A point-colour mode can additionally project fused point RGB +onto every mesh vertex. Complete geometry uses a neutral material and retains +all mesh faces, while reliable texture retains only faces with image evidence. +The original mesh and atlas files remain downloadable for review. +The CPU mesh still has holes or potentially incorrect surfaces in occluded, +vegetated, low-texture or reflective regions and cannot be assumed equivalent to +a higher-resolution GPU/cloud output without visual review. + +The separate real input `shared/data/raw/05-3d-pointcloud/pcl-table-scene-20260821/table_scene_lms400.pcd` +is the PCL tutorial's `table_scene_lms400` scan, downloaded unchanged from the +PCL data repository (SHA-256 `E285D415641E0D9DE695B611DB874CC8FE995E8089B77A50D6056D24D8CBCC58`). +Its 460,400 source points validate reading a real PCD and the `--ground-up-axis +auto` processing-copy branch. It is a tabletop scan, not an aerial reconstruction +or semantic truth set. + +Ground/elevated is a geometric split, not building, tree, equipment, or hazard +recognition. Slopes, water, vertical dominant surfaces, dense vegetation, or a +weak/no ground return can invalidate the RANSAC assumption. The mesh is an +inspectable alpha-shape approximation and is not guaranteed watertight or +metric-accurate. Obtain registered source data and manually reviewed truth before +operational or commercial use. + +## Semantic point-cloud classification baseline + +The existing-point-cloud route now writes a separate CPU semantic-classification +baseline. It uses RGB plus locally estimated ground/height, density, vertical +span, and horizontal line continuity. It is still **B** work: `geoai-py` is used +only for the existing elevated-surface raster-to-vector branch, not for a native +point-cloud classifier. + +Classes are `ground`, `vegetation`, `building_structure`, `power_line`, +`pole_tower`, and `other_unknown`. The last three names describe review outputs: +the first version is deliberately a transparent rule baseline rather than a +trained semantic model. It exports a colour-coded preview PLY (capped at 400,000 +points for the browser), semantic LAS class codes, class GeoTIFF/PNG, candidate +GeoJSON, class-count CSV, and `run_metadata.json`. + +Real LAS check: the unchanged user input `baseData/las/part_01.las` contains +10,473,127 points, RGB and return attributes, but every source LAS +`classification` value is `0`; it is not semantic truth. Run +`validation-las-part01-semantic-20260822-v11-original-rgb` used 0.20 local-unit voxel sampling +and local-neighbourhood shape features (horizontal linearity for conductors and +vertical linearity for pole/tower members). It completed in 73.586 seconds CPU +with 1,370,984 output points: 356,170 ground, 714,122 vegetation, 1,031 +structure, 3,183 pole/tower candidates, 8,162 power-line candidates, and 288,316 +unknown. The preview retains every line/pole candidate before sampling the other +classes, because uniform preview sampling hid sparse conductors. Candidate +precision and recall remain unvalidated. Do not use these candidates as asset +inventory, clearance, or inspection conclusions until a user reviews the browser +preview and supplies labelled ground truth. + +The generated H30T dense cloud was also checked as +`validation-h30t-dense-semantic-20260822-v2`: 3,756,551 input points became +1,205,115 sampled points in 38.673 CPU seconds. Its dominant plane was too +steep to be accepted as ground, so the output uses only the local low-point +fallback and records `ground_plane_available: false`. It produced 181,436 ground, +663,962 vegetation, 988 structure, 641 pole/tower candidates, zero power-line +candidates and 358,088 unknown. This proves the technical route accepts the +reconstructed PLY; it does not validate class accuracy for this side-view model. + +## Console + +The independent local experiment console exposes `GET/POST +/api/3d-pointcloud/runs`. It accepts at most two PLY/PCD/XYZ/LAS/LAZ files for +new geometry runs, generates a server-side run ID, preserves raw source bytes and +SHA-256, and stores raw, processed, and output artifacts separately. Its case +library also discovers sparse photo-SfM and dense CPU-MVS result directories. +Sparse results display an interactive Three.js point viewer, registration +statistics, poses, input manifest and downloads. Dense results display reliable +texture, point-colour and complete-geometry modes, plus dense PLY, mesh PLY and +texture downloads. The point-colour mode projects fused RGB point values to the +complete mesh; the complete geometry mode retains all mesh faces with a neutral +material. Neither fabricates a photo texture. The +sparse viewer supports orbit, zoom, point-size adjustment and reset, and never triangulates PLY vertices. It +has no connection to a drone product, product account, product API, or product +database. + +The console keeps incompatible inputs in two workflow tabs. `照片三维重建` +accepts 3-30 JPG/JPEG files from one coherent flight/camera sequence through +bounded binary uploads, preserves source bytes and SHA-256 values in a new run +directory, and starts a background CPU sparse-SfM then OpenMVS dense-MVS job. +The page polls `queued`, `sparse_sfm`, `dense_mvs`, `complete`, or `failed` and +automatically adds a completed result to the case library. Enable `GPS/RTK +先验` only when all input photos have valid position metadata; it selects the +verified spatial matching configuration. Disable it for ordinary cameras, which +use exhaustive matching. `既有点云理解` remains the PLY/PCD/XYZ/LAS/LAZ-to- +raster/vector route. Creating a new run or discovering its result does not +rebuild the console; `npm run build` is only needed after changing Vue/TypeScript +frontend source. + +The console separates this capability into four workspaces: photo 3D +reconstruction; point-cloud geometry processing; point-cloud semantic +classification; and annotation, training, and model application. Photo cases +are isolated from point-cloud cases. Geometry and semantic workspaces each have +their own filtered case library; a source run can appear in both only when it +has both output families. The model workspace contains the linked annotation, +training, and portable-model workflow without placing it below every semantic +result. + +```powershell +py -3.12 .\scripts\serve_workbench_console.py --port 6188 +``` + +## Licenses + +- `geoai-py`: MIT. This does not grant rights for data or other dependencies. +- Open3D: MIT; check release and bundled dependency notices before redistribution. +- Laspy: BSD-3-Clause; Rasterio, GeoPandas, and Shapely retain their own licenses. +- No model weights or third-party point-cloud dataset are bundled. Record the + licence of each survey/reconstruction source before commercial use. +- The PCL tutorial scan remains subject to the upstream PCL data repository's + licensing and attribution terms; it is validation input only and is not + cleared here for commercial redistribution. +- WebODM is AGPL-3.0. NodeODX/ODX, OpenSfM/OpenDroneMap, their images, and any + cloud processing service require separate license and data-handling review. + +## Human annotation and supervised CPU training + +The workbench now provides a separate human annotation and supervised training +area. A semantic run writes a 400,000-point `*.semantic-annotation-source.ply` +that preserves observed RGB and coordinates. Rule-coloured previews are never + used as model inputs or ground truth. Brush and rectangle selections are saved + as separate revisions under `shared/outputs/05-3d-pointcloud/annotations/<annotation-id>/`. + The annotation viewer renders observed RGB through an sRGB-correct, anti-aliased + circular point sprite. It separates navigation from annotation: browse mode + uses left-drag rotation, while brush/rectangle mode reserves left-drag for + labels and retains right-drag rotation, middle-drag panning and wheel zoom. + Middle-button auto-scroll is suppressed within the viewer. The rotate-centre + tool sets the camera target to a clicked visible point without creating a label. + The model-orientation control independently rotates the displayed model about + its centre on local X/Y/Z axes from 0 through 360 degrees. It is a viewer-only + transform: source coordinates, semantic outputs and saved point indices remain + unchanged, while point picking and rectangle selection use the transformed view. + A saved revision is never edited in place. If it contains an error, the console + can remove that selected revision after confirmation; it cannot delete source + LAS/PLY data, semantic results, or training outputs. Each revision records + source run/checksum, point indices, class codes and time. + +`train_pointcloud_semantic_model.py` trains a compact PointNet-style shared MLP +from user-confirmed labels only. It runs locally with `--device cpu` and later +uses the same code/data with `--device cuda`. Training requires two or more +classes and at least 500 confirmed points per class, partitions XY blocks into +train/validation/test, and writes `model.pt`, `metrics.json`, a predicted PLY, +per-class precision/recall/F1 and a confusion matrix. Metrics apply only to the +labelled source blocks and are not field-wide accuracy claims. + +CPU validation `validation-las-part01-semantic-20260822-v11-original-rgb` processed unchanged +`baseData/las/part_01.las` in 73.586 seconds and created its 400,000-point RGB +annotation source. Its 3,183 pole/tower and 8,162 power-line results remain rule +candidates until users supply labels and supervised evaluation is completed. + +## Applying a trained model + +The supervised model can be applied to a new coloured PLY, PCD, LAS, or LAZ +point cloud without retraining. This is still B work: PyTorch, Open3D, NumPy, +and Laspy perform inference; `geoai-py` is not a built-in point-cloud semantic +classifier. + +The input must contain readable observed RGB values. XYZ-only inputs fail rather +than inventing colour features. Coordinates remain unchanged. XYZ features are +normalised using the new input's own centre and extent, matching the feature +definition used during training without forcing the old scene coordinates onto a +new site. + +```powershell +.\.venvs\05-3d-pointcloud\Scripts\python.exe ` + .\capabilities\05-3d-pointcloud\apply_pointcloud_semantic_model.py ` + --model .\shared\outputs\05-3d-pointcloud\training-runs\<model-id>\model.pt ` + --input .\path\to\new-rgb-cloud.las ` + --output .\shared\outputs\05-3d-pointcloud\model-inference-runs\<new-run-id> ` + --device cpu +``` + +Each new output directory contains `predicted-semantic-preview.ply` (a +class-aware deterministic browser preview capped at 400,000 points), +`predicted-semantic-classified.las` (LAS classification codes), +`class-counts.csv`, `prediction-summary.json`, and `run_metadata.json` with +model/input SHA-256 values, versions, normalisation, class counts, runtime, and +limitations. No vector asset layer is fabricated from semantic predictions. + +Verified CPU check on 2026-08-24: model +`semantic-model-20260824-023343-571983` applied to the 400,000-point RGB +annotation source in 1.608 seconds through the CLI. The local-console upload and +background-job route was also verified, including PLY and classified-LAS artifact +download. This verifies the workflow, not field accuracy. Its labelled spatial +test F1 is vegetation 0.979, power line 0.815, and pole/tower 0.348; the latter +has substantial false-positive risk and all output remains review candidates. -- Gitblit v1.9.3