ShapeNet with SUNCG

Front readme image

The focus of this example is the loader.ShapeNetLoader in combination with the SUNCG loader, this is an advanced example, please make sure that you have read:

Usage

Execute in the BlenderProc main directory:

blenderpoc run examples/datasets/shapenet_with_suncg/main.py <PATH_TO_ShapeNetCore.v2> <PATH_TO_SUNCG_HOUSE_JSON> examples/datasets/shapenet_with_suncg/output

  • examples/datasets/shapenet_with_suncg/main.py: path to the python file with pipeline configuration.

  • <PATH_TO_ShapeNetCore.v2>: path to the downloaded shape net core v2 dataset, get it [here](http://www.shapenet.org/)

  • <PATH_TO_SUNCG_HOUSE_JSON>: path to a house.json file from the SUNCG dataset.

  • examples/datasets/shapenet_with_suncg/output: path to the output directory.

Visualization

In the output folder you will find a series of .hdf5 containers. These can be visualized with the script:

blenderproc vis hdf5 examples/datasets/shapenet_with_suncg/output/*.hdf5

Steps

  • At first the SUNCG scene is loaded.

  • The ShapeNetLoader loads all the object paths with the used_synset_id = 02801938, this id stands for the category basket.

  • One of them is now randomly selected and loaded.

  • Then we select that one object and change its location to be above an object with the catgory “bed”.

  • We also add a solidify modifier as a few of the objects in the ShapeNet dataset have only a really thin outer shell, this might lead to bad results in the physics simulation.

  • We enable the rigid body component of the objects which makes them participate in physics simulations.

  • The physics simulation is run to let the ShapeNet object fall down on the bed.

  • We finally sample some cameras around this ShapeNet object, which are located in a HalfSphere above the ShapeNet object.

  • Now we only have to render it and store it in a .hdf5 container

Python file (main.py)

SuncgLoader

label_mapping = bproc.utility.LabelIdMapping.from_csv(bproc.utility.resolve_resource(os.path.join('id_mappings', 'nyu_idset.csv')))
suncg_objs = bproc.loader.load_suncg(args.house, label_mapping=label_mapping)

This loader automatically loads a SUNCG scene/house given the corresponding house.json file. Therefore, all objects specified in the given house.json file are imported and textured. The SuncgLoader also sets the category_id of each object, such that semantic segmentation maps can be rendered in a following step.

ShapeNetLoader

# load selected shapenet object
shapenet_obj = bproc.loader.load_shapenet(args.shape_net, used_synset_id="02801938")

This loads a ShapeNet Object, it only needs the path to the ShapeNetCore.v2 folder, which is saved in data_path. The synset_id = 02801938 is set to the id of a basket, which means a random basket will be loaded.

The position will be in the center of the scene, and we add the custom property cp_physics: True so that the object will fall during the physics simulation. We also add a custom property to make the selection with EntityManipulator in the next step easier.

EntityManipulator

# Sample a point above any bed
sample_point = bproc.sampler.upper_region(
    objects_to_sample_on=bed_objs,
    min_height=0.75,
    use_ray_trace_check=True
)
# move the shapenet object to the sampled position
shapenet_obj.set_location(sample_point)

# adding a modifier we avoid that the objects falls through other objects during the physics simulation
shapenet_obj.add_modifier(name="SOLIDIFY", thickness=0.001)

# enable rigid body component of the objects which makes them participate in physics simulations
shapenet_obj.enable_rigidbody(active=True, mass_factor=2000, collision_margin=0.0001)
for obj in bproc.filter.all_with_type(suncg_objs, bproc.types.MeshObject):
    obj.enable_rigidbody(active=False, mass_factor=2000, collision_margin=0.0001)

With this we change the location of the ShapeNet Object. For that we first select the object, via the "filter", based on these conditions it returns the ShapeNetObject, which we will manipulate next.

We first set the location to be sampled above an entity, which has the category “bed”. We add a solidify modifier and add mass to get a correct physics interaction.

PhysicsPositioning

bproc.object.simulate_physics_and_fix_final_poses(min_simulation_time=0.5, max_simulation_time=4, check_object_interval=0.25)

We then run the physics simulation, for more information about that please see the example/physiscs_positioning/README.md. The high mass factor and the small collision margin guarantee that the object does not move too much.