Static and Dynamic Point Cloud Coding

Overview

  • 3D Scenes are represented by dynamic point cloud videos for better flexibility and compactness.
  • We design the compression and streaming system that is scalable, efficient, and capable of delivering high-quality renders.
  • For scalability, we adopt the octree structure to represent the point cloud, and represent the 3D scene with scalable point-based latent features.
  • A feed-forward neural network decodes the feature points to efficiently renderable 3D Gaussians.

     

Low Latency Point Cloud Rendering

[arXiv] [Code]

Technical Overview

  • Our proposed P2ENet (based on 3D Convolutions by MinkowskiEngine with extended capability) is a feed-forward neural network that transform a point cloud to 3D Gaussians that support smooth and fast rendering.
  • This is achieved by predicting the coordinate offsets, scales and rotation matrices from the original point cloud.
  • Once trained, the network can render a point cloud with ~300K points in real-time on GPU.

Results

Static Scenes

Surface Normal
Color Image
Methods Mesh Rasterization (Ground Truth) OpenGL Pointersect (Non-Realtime) P2ENet (Ours)

Dynamic Scenes

Bits-to-Photon (B2P): Point Cloud Compression for Direct Rendering

[arXiv]

Technical Overview

  • Joint Optimization: First method to jointly optimize point cloud compression for both rate-distortion performance and rendering quality.
  • 3D Gaussian Representation: Generates a bit stream that can be directly decoded into renderable 3D Gaussians, improving rendering quality and efficiency.
  • Scalable Bit Stream: Supports multiple levels of detail at different bit-rate ranges, allowing flexibility in bandwidth usage.
  • Real-Time Performance: Achieves real-time color decoding and high-quality rendering.

Results
THuman; PSNR THuman; LPIPS 8iVFB; PSNR 8iVFB; LPIPS
B2P (Ours): 0.49 bpp G-PCC + OpenGL: 0.53 bpp

Related Publications

[1] Yueyu Hu, Ran Gong, Qi Sun, and Yao Wang. “Low Latency Point Cloud Rendering with Learned Splatting”, to appear at CVPR Workshop (AIS: Vision, Graphics and AI for Streaming), 2024. [arXiv] [Code] [Workshop]

[2] Yueyu Hu, Ran Gong, Yao Wang. “Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering”, arXiv:2406.05915, 2024. [arXiv]