【代码复现】Any6D: Model-free 6D Pose Estimation of Novel Objects

发表于 分类于 阅读次数： Waline： 本文字数： 522 阅读时长 ≈ 2 分钟
复现Any6D的代码，并进行测试。

安装Any6D环境

  1. 创建环境：

    1
    conda create -n any6d python=3.9

  2. 激活环境

    1
    conda activate any6d

  3. 安装cuda12.1：

    1
    conda install cuda -c nvidia/label/cuda-12.1.0

  4. 安装Eigen3 3.4.0并添加环境变量：

    1
    2
    conda install conda-forge::eigen=3.4.0
    export CMAKE_PREFIX_PATH="$CMAKE_PREFIX_PATH:/eigen/path/under/conda"

    注意，这里的/eigen/path/under/conda应该在类似于/home/tanmx/miniconda3/envs/any6d/include/eigen3的路径下，且：

    1
    2
    $ ls /home/tanmx/miniconda3/envs/any6d/include/eigen3
    Eigen  signature_of_eigen3_matrix_library  unsupported

  5. 修改requirements.txt文件：

    1. 第86行：huggingface-hub==0.24.0 -> huggingface-hub；
    2. 第91行：accelerate -> accelerate==0.31.0。

  6. 安装依赖：

    1
    python -m pip install -r requirements.txt

  7. 安装NVDiffRast：

    1
    python -m pip install --no-cache-dir git+https://github.com/NVlabs/nvdiffrast.git

  8. 安装Kaolin：

    1
    python -m pip install --no-cache-dir kaolin==0.16.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.4.0_cu121.html

  9. 安装PyTorch3D：

    1
    pip install --extra-index-url https://miropsota.github.io/torch_packages_builder pytorch3d==0.7.8+pt2.4.1cu121

  10. 安装Boost：

    1
    sudo apt install build-essential libboost-system-dev libboost-thread-dev libboost-program-options-dev libboost-test-dev

  11. 修改foundationpose/bundlesdf/mycuda/setup.py：

    1. 第35行的include_dirs变量，由：

      1
      2
      3
      4
      include_dirs=[
          "/usr/local/include/eigen3",
          "/usr/include/eigen3",
      ],

      修改为：

      1
      2
      3
      4
      5
      include_dirs=[
          # "/usr/local/include/eigen3",
          # "/usr/include/eigen3",
          '/home/tanmx/miniconda3/envs/any6d/include/eigen3',
      ],

  12. 编译扩展：

    1
    CMAKE_PREFIX_PATH=$CONDA_PREFIX/lib/python3.9/site-packages/pybind11/share/cmake/pybind11 bash foundationpose/build_all_conda.sh

  13. 编译SAM2：

    1
    2
    cd sam2
    pip install -e .

  14. 编译bop_toolkit：

    1
    2
    cd bop_toolkit
    python setup.py install

下载数据集

  1. YCBV物体模型：

    • https://drive.google.com/file/d/1gmcDD-5bkJfcMKLZb3zGgH_HUFbulQWu/view

  2. HO3D评估文件：

    • https://drive.google.com/drive/folders/1Wk-HZDvUExyUrRn7us4WWEbHnnFHgOAX

  3. Any6D Anchor Results：

    • https://huggingface.co/datasets/taeyeop/Any6D/resolve/main/dexycb_reference_view_ours.zip

将上述文件分别组织如下：

1
2
3
4
ho3d/
├── evaluation/         # HO3D evaluation files (e.g., annotations)
├── masks_XMem/         # Segmentation masks generated by XMem
└── YCB_Video_Models/   # 3D models for YCB objects (used in HO3D)

下载预训练模型

  1. foundationpose预训练模型：

    • https://drive.google.com/drive/folders/1DFezOAD0oD1BblsXVxqDsl8fj0qzB82i

  2. sam2预训练模型：

    • https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt

  3. instantmesh预训练模型：

    • https://huggingface.co/TencentARC/InstantMesh/resolve/main/diffusion_pytorch_model.bin
    • https://huggingface.co/TencentARC/InstantMesh/resolve/main/instant_mesh_base.ckpt
    • https://huggingface.co/TencentARC/InstantMesh/resolve/main/instant_mesh_large.ckpt
    • https://huggingface.co/TencentARC/InstantMesh/resolve/main/instant_nerf_base.ckpt
    • https://huggingface.co/TencentARC/InstantMesh/resolve/main/instant_nerf_large.ckpt

将上述文件分别组织如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
foundationpose/
└── weights/
    ├── 2024-01-11-20-02-45/
    └── 2023-10-28-18-33-37/

sam2/
└── checkpoints/
    ├── sam2.1_hiera_large.pt

instantmesh/
└── ckpts/
    ├── diffusion_pytorch_model.bin
    └── instant_mesh_large.ckpt

运行代码

Demo

  1. 不使用instantmesh + sam2

    1
    python run_demo.py --ycb_model_path ./ho3d/YCB_Video_Models/

  2. 使用instantmesh + sam2

    1
    python run_demo.py --ycb_model_path ./ho3d/YCB_Video_Models/ --img_to_3d

在HO3D上运行

见GitHub仓库。