ShapeNet with Scenenet

Front readme image

The focus of this example is the loader.ShapeNetLoader in combination with the SceneNet loader, this is an advanced example, please make sure that you have read:

  • shapenet: Rendering ShapeNet objects

  • scenenet: Rendering SceneNet scenes with sampled camera poses.

Usage

Execute in the BlenderProc main directory:

blenderproc run examples/datasets/shapenet_with_scenenet/main.py <PATH_TO_SCENE_NET_OBJ_FILE> <PATH_TO_TEXTURE_FOLDER> <PATH_TO_ShapeNetCore.v2> examples/datasets/shapenet_with_scenenet/output

  • examples/datasets/shapenet_with_scenenet/main.py: path to the python file with pipeline configuration.

  • <PATH_TO_SCENE_NET_OBJ_FILE>: path to the used SceneNet .obj file, download via this script

  • <PATH_TO_TEXTURE_FOLDER>: path to the downloaded texture files, you can find them [here](http://tinyurl.com/zpc9ppb)

  • <PATH_TO_ShapeNetCore.v2>: path to the downloaded shape net core v2 dataset, get it [here](http://www.shapenet.org/)

  • examples/datasets/shapenet_with_scenenet/output: path to the output directory.

As this example requires a bed to be present in the scene, it will only work with the 1Bedroom/* SceneNet scenes.

Visualization

In the output folder you will find a series of .hdf5 containers. These can be visualized with the script:

blenderproc vis hdf5 examples/datasets/shapenet_with_scenenet/output/*.hdf5

Steps

  • At first the SceneNet scene is loaded and we add the custom property cp_physics to make sure that the sampled ShapeNet objects, bounds of the SceneNet scene.

  • As explained in the scenenet example, the textures are randomly sampled.

  • The ShapeNetLoader loads all the object paths with the synset_id = 02801938, this id stands for the category basket.

  • One of them is now randomly selected and loaded.

  • Then we select that one object and change its location to be above an object with the catgory_id = 4, which stands for bed.

  • We also add a solidify modifier as a few of the objects in the ShapeNet dataset have only a really thin outer shell, this might lead to bad results in the physics simulation.

  • The physics simulation is run to let the ShapeNet object fall down on the bed.

  • We finally sample some cameras around this ShapeNet object, which are located in a HalfSphere above the ShapeNet object.

  • Now we only have to render it and store the results it in a .hdf5 container

Python file (main.py)

SceneNetLoader

# Load the scenenet room and label its objects with category ids based on the nyu mapping
label_mapping = bproc.utility.LabelIdMapping.from_csv(bproc.utility.resolve_resource(os.path.join('id_mappings', 'nyu_idset.csv')))
room_objs = bproc.loader.load_scenenet(args.scene_net_obj_path, args.scene_texture_path, label_mapping)

This loader automatically loads a SceneNet scene/house given the corresponding .obj file. The textures are randomly sampled from the texture folder, for more information see the scenenet example. The bproc.loader.load_scenenet() also sets the category_id of each object, such that semantic segmentation maps can be rendered in a following step.

ShapeNetLoader

shapenet_obj = bproc.loader.load_shapenet(args.shapenet_path, used_synset_id="02801938")

This loads a ShapeNet Object, it only needs the path to the ShapeNetCore.v2 folder, which is saved in data_path. The used_synset_id = 02801938 is set to the id of a basket, which means a random basket will be loaded.

The position will be in the center of the scene and the object will fall during the physics simulation.

EntityManipulator

# Collect all beds
beds = bproc.filter.by_cp(room_objs, "category_id", label_mapping.id_from_label("bed"))
# Sample the location of the ShapeNet object above a random bed
shapenet_obj.set_location(bproc.sampler.upper_region(beds, min_height=0.3, use_ray_trace_check=True))

# Make sure the ShapeNet object has a minimum thickness (this will increase the stability of the simulator)
shapenet_obj.add_modifier("SOLIDIFY", thickness=0.0025)
# Make the ShapeNet object actively participating in the simulation and increase its mass to stabilize the simulation
shapenet_obj.enable_rigidbody(True, mass_factor=2000, collision_margin=0.00001, collision_shape="MESH")

With this we change the location and the custom properties of the ShapeNet Object. For that we first select the object, via the "filter", based on these conditions it returns the ShapeNetObject, which we will manipulate next.

We first set the location to be sampled above an entity, which has the category “bed”. We add a solidify modifier and add mass to get a correct physics interaction.

PhysicsPositioning

bproc.object.simulate_physics_and_fix_final_poses(
    solver_iters=30,
    substeps_per_frame=40,
    min_simulation_time=0.5,
    max_simulation_time=4,
    check_object_interval=0.25
)

We then run the physics simulation, for more information about that please see the example/physiscs_positioning. The high mass factor and the small collision margin guarantee that the object does not move too much. Important here are the amount of solver_iters and substeps_per_frame as they have to be high, as lot of objects in the ShapeNet dataset consist out of thin small pieces. Without this they might slide into the SceneNet objects.