Ref frame for extrinsics

@dyfcalid

Hi @dyfcalid , This one is related to the cam_representation question. Why are the extrinsics re computed? What are the original extrinsics with respect to? The following is the piece of code that I am referring to:


                cam_extrinsics = np.array(info_dict['extrinsic'])
                # Re-calculate extrinsic matrix based on ground coordinate
                R_vg = np.array([[0, 1, 0],
                                    [-1, 0, 0],
                                    [0, 0, 1]], dtype=float)
                R_gc = np.array([[1, 0, 0],
                                    [0, 0, 1],
                                    [0, -1, 0]], dtype=float)
                cam_extrinsics[:3, :3] = np.matmul(np.matmul(
                                            np.matmul(np.linalg.inv(R_vg), cam_extrinsics[:3, :3]),
                                                R_vg), R_gc)
                cam_extrinsics[0:2, 3] = 0.0

same questions

Hi , there are three camera coord sys:

Waymo (OpenLane) camera coord sys. x-front, y-left, z-up

normal (aka. standard) camera coord sys widely used. x-right, y-down, z-front

LaneNet (3D-LaneNet) camera coord sys. x-right, y-front, z-up

The transformation matrices in the code are as following:

cam_representation: Waymo to normal
https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L333-L338
R_gc: normal to LaneNet
https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L305-L307
R_vg: LaneNet to Waymo
https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L302-L304
original cam_extrinsics in JSON: Waymo camera to Waymo vehicle
inv(R_vg): Waymo to LaneNet

Note that the 3D lane GT are annotated in the Waymo camera coord sys. With matrices above, the final cam_extrinsics can transform GT to road coord (which is right-down of the camera and actually has the same z height with Waymo vehicle coord rather than on the road, with setting x/y=0 in the last line of code).
https://github.com/OpenPerceptionX/OpenLane/blob/1e8e61ccb456232955cf999b8db2aa4023945d25/eval/LANE_evaluation/lane3d/eval_3D_lane.py#L308-L314

In conclusion, there are some redundancies in the process and one could try to use only R_vg and original cam_extrinsics to see if they could get the same results.

@RicardLee

@RicardLee

LaneNet (3D-LaneNet) camera coord sys. x-right, y-front, z-up

How can we hope to achieve the statement In conclusion, "try to use only R_vg and original cam_extrinsics to see if they could get the same results"? The original extrinsic matrix has an offset that is zeroed when creating the virtual ground.

@presmann

@presmann Hi, the conclusion means that zeroed operation would be applied except for R_vg and cam_extrinsics.

Hi, after reading the above explanations, i'm still confused, sorry for this...

cam_extrinsics[:3, :3] = np.matmul(np.matmul(np.matmul(np.linalg.inv(R_vg), cam_extrinsics[:3, :3]), R_vg), R_gc)
In the formula above, there are 3 matmuls:

np.matmul(np.linalg.inv(R_vg), cam_extrinsics[:3, :3]): this one transforms waymo camera to lanenet, right?
np.matmul(xxx, R_vg): Does this one transform waymo camera to waymo vehicle? Why not just use origin cam extrinsics?
np.matmul(xxx, R_gc): i think i'm already wrong in step 2, so still confused about this step...

One more question, gt_cam_pitch = 0 means simply no pitch? or is this related to the 3 matmuls?

Thanks!

Hi, matrix multiplication should be applied from right to left.
Assume that Point A_1 in waymo camera coord:
1.A_2 = np.matmul( cam_representation, A_1): transform the waymo camera-coord to normal camera-coord.
2.A_3 = np.matmul( R_gc, A_2):transform the normal camera-coord to LaneNet camera-coord.
3.A_4 = np.matmul( R_vg, A_3):transform the LaneNet camera-coord to waymo camera-coord.
4.A_5 = np.matmul( cam_extrinsics[:3, :3], A_4):transform the waymo camera-coord to waymo vehicle-coord.
5.A_6 = np.matmul( np.linalg.inv(R_vg), A_5):transform the waymo vehicle-coord to LaneNet vehicle-coord.
Finally, we get the 3d-lane in LaneNet vehicle-coord (road coord).
As the conclusion says that there are some redundancies in the process.
The cam pitch is not used in the following process and it would be deprecated.

thanks for your excellent work ! i have a few question about output:

Does the output 3d-lane is in LaneNet vehicle-coord (road coord) : x-right,y-front,z-up，and the origin point of z is not at road plane，but as same as Waymo vehicle coord。
If want to output 3d-lane in Waymo vehicle-coord at last predict, just use A_5 to calculate。
waiting for your reply，thanks

@whoakang hi，

Waymo vehicle coord is as following: x-front,y-left,z-up. It's different from LaneNet vehicle-coord (road coord).
the 3d-lane in LaneNet vehicle-coord (road coord) is the last output so that the A_6 is necessary.

thanks for your reply.

you mean the origin point of road coordinate system and Waymo vehicle coord is same?

@whoakang hi, after setting x/y=0 , the origin point of the road coordinate and waymo vehicle is different.

Navigation Menu

Repository navigation

Ref frame for extrinsics #24

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Ref frame for extrinsics #24

Description

Activity

nickle-fang commented on Jul 25, 2022

RicardLee commented on Aug 25, 2022

presmann commented on Sep 12, 2022

RicardLee commented on Sep 21, 2022

notabigfish commented on Oct 17, 2022

RicardLee commented on Oct 17, 2022

raikaland commented on Oct 25, 2022

RicardLee commented on Oct 25, 2022

raikaland commented on Nov 2, 2022

RicardLee commented on Nov 2, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions