3D Joint positions from SMPL Model (ref: [19])

[J-M-pixel]

Hi,

I am trying to acquire mesh vertices and 3D joint positions by providing pose and shape parameters (provided as ground truth in SURREAL) to the SMPL model given in ref. [19]. However, the 24 3D joint positions which I get from SMPL model, are different from what is provided as the ground truth 3D joint positions in SURREAL.

Could you please guide me so that I could acquire same 3D joint positions from SMPL model as provided by SURREAL in the ground truth ?

-- Thanks

[gulvarol]

Hi, did you rotate the SMPL model by zrot, i.e. multiply zrot (convert euler to rotation matrix) by pose[0:3] (convert axis-angle to rotation matrix)? The global rotation around the z-axis was saved in zrot variable.

[xiangdonglai]

Hi Gül,

Do you mean that from the output of SMPL (rotation in pose[0:3] is already taken into account) we just need an additional rotation around the z-axis?

I do this but the result seems still inconsistent. Could you please clarify?

Best,
Donglai

[buaacyh]

When I test the gt2d joints of the surreal dataset, I find the problem that left and right joints are exchanged(compare to smpl ground truth). I draw the picture and draw the 23rd joints of surreal which shows the point is left fingers. However, when I find in smpl ground truth, the 23rd joints means right fingers. Is my problem?

[ypflll]

@jameel-malik Also meet this problem. Do u find a solution?

[ypflll]

@gulvarol Hi, Gül. Thank you for sharing your good work.
I also met a problem when I try to project SMPL 3d vertices to 2d. I use the code below:

## Load SMPL model (female model or male model)
m = load_model(model_name)

## for the first frame
m.pose[:] = info['pose'][:,0]
m.betas[:] = info['shape'][:,0]

### matrix get from the matlab code, here I use files in 01_01
intrinsic = np.array([[600,0,160],[0,600,120],[0,0,1]])
extrinsic = np.array([[0,0,-1.,-1.9292990],[0,1.,0,0.98388302],[-1.,0,0,6.7456222]])
M = np.dot(intrinsic, extrinsic)

vertices = np.concatenate((m.r, np.ones((m.r.shape[0],1))), axis=1)
coord_projected = np.dot(M, vertices.T)
coord_projected[0,:] /= coord_projected[2,:]
coord_projected[1,:] /= coord_projected[2,:]

Result (green points are the projected joints3D which seems right):

I tried to multiply a rotation matrix provided by zrot as you mentioned above, but still not right:

theta = info['zrot'][0]   #*180/math.pi
Rot_z = np.array([[math.cos(theta), -math.sin(theta), 0], [math.sin(theta), math.cos(theta), 0], [0, 0, 1]])
m.pose[0:3] = np.dot(Rot_z, m.pose[0:3])

Result:

This really confuses me. What's the righ way to get compatibility between SMPL and SURREAL?

[ypflll]

@gulvarol Still not very clear about this. Can you take a look at this?

[gulvarol]

Sorry for the delay. I have added some sample code to visualize the projection from SMPL vertices and from SMPL 3D joints to 2D here: https://github.com/gulvarol/surreal/tree/master/datageneration/misc/smpl_relations

Indeed, there is a left/right swap that needs to be done. Sorry for inconvenience.

I also added some rotation and translation. Now the projected vertices match the image as below:

However, note the slight mismatch between the 3D joints, which happens especially in male bodies (almost negligible). It almost never happens in female bodies. It's because the joint_regressor [24 x 232] is a subset of the J_regressor [24 x 6890] from the female pkl model.

surreal/datageneration/main_part1.py

Lines 329 to 331 in 6a80037

	def reset_joint_positions(orig_trans, shape, ob, arm_ob, obname, scene, cam_ob, reg_ivs, joint_reg):
	# since the regression is sparse, only the relevant vertex
	# elements (joint_reg) and their indices (reg_ivs) are loaded

[ypflll]

@gulvarol Thank for clearifying this.
I've solved this by refering another repo, which uses the relationship between 2d and 3d joints, solving a linear equation and geting the intrinsic matrix.
A little complex and time costing, but works.

[J-M-pixel]

Hello Gul Varol,
Thanks for the clarification and the additional code to support.
Thanks again !

[ypflll]

@gulvarol Tested this code. Find a bug: for train/run2/132_50/132_50_c0002_info.mat, frame 49, get this:

Eps for rotmat2rotvec(R) is too small. Change 1e-4 to 1e-3 can avoid this. Not sure what is the critical value.

[gulvarol]

Thanks for reporting, I removed the dependency to the rotations.py file and replaced with the functions from transforms3d package instead. Hopefully it's more robust.