Load carrier functionality¶

Introduction¶

The load carrier functionality is contained in an internal load carrier component and can only be used through the software components providing a load carrier functionality.

The load carrier functionality is provided by the ItemPick and BoxPick and SilhouetteMatch modules.

Load Carrier¶

A load carrier (bin) is a container with four walls, a floor and a rectangular rim, which can contain objects.

A load carrier is defined by its outer_dimensions and inner_dimensions. The maximum outer_dimensions are 2.0 meters in every dimension.

Note

Typically, outer and inner dimensions of a load carrier are available in the specifications of the load carrier manufacturer.

The rc_visard can persistently store up to 50 different load carrier models, each one identified by a different id. The configuration of a load carrier model is normally performed offline, during the set up the desired application. This can be done via the REST-API interface or in the rc_visard Web GUI.

Note

The configured load carrier models are persistent even over firmware updates and rollbacks.

Detection of load carriers¶

The load carrier detection algorithm is based on the detection of the load carrier rectangular rim. By default, the rectangular rim_thickness is computed from the outer and inner dimensions. As an alternative, its value can also be explicitly specified by the user.

The origin of a detected load carrier is in the center of the load carrier outer box and its z axis is perpendicular to the load carrier floor. The detection functionality also determines if the detected load carrier is overfilled.

_images/itempick_load_carrier_reference_rim_sidebyside.svg

Fig. 60 Load carrier models and reference frame.

The user can optionally specify a prior for the load carrier pose. The detected load carrier pose is guaranteed to have the minimum rotation with respect to the load carrier prior pose. If no prior is specified, the algorithm searches for a load carrier whose floor is perpendicular to the estimated gravity vector.

Detection of filling level¶

The load carrier functionality contains the detect_filling_level service to compute the filling level of a detected load carrier.

The load carrier is subdivided in a configurable number of cells in a 2D grid. The maximum number of cells is 10x10. For each cell, the following values are reported:

level_in_percent: minimum, maximum and mean cell filling level in percent from the load carrier floor. These values can be larger than 100% if the cell is overfilled.
level_free_in_meters: minimum, maximum and mean cell free level in meters from the load carrier rim. These values can be negative if the cell is overfilled.
cell_size: dimensions of the 2D cell in meters.
cell_position: position of the cell center in meters (either in camera or external frame, see Hand-eye calibration). The z-coordinate is on the level of the load carrier rim.
coverage: represents the proportion of valid pixels in this cell. It varies between 0 and 1 with steps of 0.1. A low coverage indicates that the cell contains several missing data (i.e. only a few points were actually measured in this cell).

These values are also calculated for the whole load carrier itself. If no cell subdivision is specified, only the overall filling level is computed.

Fig. 61 Visualizations of the load carrier filling level: overall filling level without grid (left), 4x3 grid (center), 8x8 grid (right). The load carrier content is shown in a green gradient from white (on the load carrier floor) to dark green. The overfilled regions are visualized in red. Numbers indicate cell IDs.

Interaction with other components¶

Internally, the load carrier functionality depends on, and interacts with other on-board components as listed below.

Note

All changes and configuration updates to these components will affect the performance of the load carrier component.

Stereo camera and Stereo matching¶

The load carrier component makes internally use of the following data:

Rectified images from the Stereo camera component (rc_stereocamera);
Disparity, error, and confidence images from the Stereo matching component (rc_stereomatching).

All processed images are guaranteed to be captured after the component trigger time.

Estimation of gravity vector¶

For each load carrier detection, the component estimates the gravity vector by subscribing to the rc_visard’s IMU data stream.

Note

The gravity vector is estimated from linear acceleration readings from the on-board IMU. For this reason, the load carrier component requires the rc_visard to remain still while the gravity vector is being estimated.

IO and Projector Control¶

In case the rc_visard is used in conjunction with an external random dot projector and the IO and Projector Control component (rc_iocontrol), it is recommended to connect the projector to GPIO Out 1 and set the stereo-camera component’s acquisition mode to SingleFrameOut1 (see Stereo matching parameters, so that on each image acquisition trigger an image with and without projector pattern is acquired.

Alternatively, the output mode for the GPIO output in use should be set to ExposureAlternateActive (see Description of run-time parameters).

In either case, the Auto Exposure Mode exp_auto_mode should be set to AdaptiveOut1 to optimize the exposure of both images (see Stereo camera parameters.

No additional changes are required to use the load carrier component in combination with a random dot projector.

Hand-eye calibration¶

In case the camera has been calibrated to a robot, the loadcarrier component can automatically provide poses in the robot coordinate frame. For the loadcarrier nodes’ Services, the frame of the output poses can be controlled with the pose_frame argument.

Two different pose_frame values can be chosen:

Camera frame (camera). All poses provided by the components are in the camera frame, and no prior knowledge about the pose of the camera in the environment is required. This means that the configured load carriers move with the camera. It is the user’s responsibility to update the configured poses if the camera frame moves (e.g. with a robot-mounted camera).
External frame (external). All poses provided by the components are in the external frame, configured by the user during the hand-eye calibration process. The component relies on the on-board Hand-eye calibration component to retrieve the sensor mounting (static or robot mounted) and the hand-eye transformation. If the mounting is static, no further information is needed. If the sensor is robot-mounted, the robot_pose is required to transform poses to and from the external frame.

Note

If no hand-eye calibration is available, all pose_frame values should be set to camera.

All pose_frame values that are not camera or external are rejected.

Parameters¶

The load carrier functionality is used internally by several other components and their parameters and services are provided through these nodes. They can also be used in the Web GUI on the page of the corresponding module. The user can explore and configure the load carrier component’s run-time parameters, e.g. for development and testing, using the corresponding module page in the Web GUI or the REST-API interface.

Parameter overview¶

This component offers the following run-time parameters:

Table 48 The load carrier component’s parameters¶
Name	Type	Min	Max	Default	Description
`load_carrier_crop_distance`	float64	0.0	0.05	0.005	Safety margin in meters by which the load carrier inner dimensions are reduced to define the region of interest for detection
`load_carrier_model_tolerance`	float64	0.003	0.025	0.008	Indicates how much the estimated load carrier dimensions are allowed to differ from the load carrier model dimensions in meters

Description of run-time parameters¶

Each run-time parameter is represented by a row on the Settings section of the Web GUI’s module page in the subsection Load Carrier Detection Parameters. The name in the Web GUI is given in brackets behind the parameter name and the parameters are listed in the order they appear in the Web GUI:

load_carrier_model_tolerance (Model Tolerance): indicates how much the estimated load carrier dimensions are allowed to differ from the load carrier model dimensions in meters.
load_carrier_crop_distance (Crop Distance): sets the safety margin in meters by which the load carrier’s inner dimensions are reduced to define the region of interest for detection.

Services¶

The user can explore and call the load carrier component’s services, e.g. for development and testing, using the REST-API interface or the rc_visard Web GUI on the page of the module offering the load carrier functionality.

Each service response contains a return_code, which consists of a value plus an optional message. A successful service returns with a return_code value of 0. Negative return_code values indicate that the service failed. Positive return_code values indicate that the service succeeded with additional information. The smaller value is selected in case a service has multiple return_code values, but all messages are appended in the return_code message.

The following table contains a list of common codes:

Table 49 Return codes of the load carrier services¶
Code	Description
0	Success
-1	An invalid argument was provided
-4	Data acquisition took longer than the maximum allowed time of 5.0 seconds
-10	New element could not be added as the maximum storage capacity of load carriers has been exceeded
-302	More than one load carrier provided to the `detect_load_carriers` or `detect_filling_level` services, but only one is supported
10	The maximum storage capacity of load carriers has been reached
11	An existent persistent model was overwritten by the call to `set_load_carrier`
100	The requested load carriers were not detected in the scene
102	The detected load carrier is empty
300	A valid `robot_pose` was provided as argument but it is not required

All software components providing the load carrier functionality offer the following services.

`set_load_carrier`¶

Persistently stores a load carrier on the rc_visard. All configured load carriers are persistent over firmware updates and rollbacks.

The definition for the request arguments with corresponding datatypes is:

{
  "load_carrier": {
    "id": "string",
    "inner_dimensions": {
      "x": "float64",
      "y": "float64",
      "z": "float64"
    },
    "outer_dimensions": {
      "x": "float64",
      "y": "float64",
      "z": "float64"
    },
    "pose": {
      "orientation": {
        "w": "float64",
        "x": "float64",
        "y": "float64",
        "z": "float64"
      },
      "position": {
        "x": "float64",
        "y": "float64",
        "z": "float64"
      }
    },
    "pose_frame": "string",
    "rim_thickness": {
      "x": "float64",
      "y": "float64"
    }
  }
}

Details for the definition of the load_carrier type are given in Detection of load carriers.

The definition for the response with corresponding datatypes is:

{
  "return_code": {
    "message": "string",
    "value": "int16"
  }
}

`get_load_carriers`¶

Returns the configured load carriers with the requested load_carrier_ids. If no load_carrier_ids are provided, all configured load carriers are returned.

The definition for the request arguments with corresponding datatypes is:

{
  "load_carrier_ids": [
    "string"
  ]
}

The definition for the response with corresponding datatypes is:

{
  "load_carriers": [
    {
      "id": "string",
      "inner_dimensions": {
        "x": "float64",
        "y": "float64",
        "z": "float64"
      },
      "outer_dimensions": {
        "x": "float64",
        "y": "float64",
        "z": "float64"
      },
      "pose": {
        "orientation": {
          "w": "float64",
          "x": "float64",
          "y": "float64",
          "z": "float64"
        },
        "position": {
          "x": "float64",
          "y": "float64",
          "z": "float64"
        }
      },
      "pose_frame": "string",
      "rim_thickness": {
        "x": "float64",
        "y": "float64"
      }
    }
  ],
  "return_code": {
    "message": "string",
    "value": "int16"
  }
}

`delete_load_carriers`¶

Deletes the configured load carriers with the requested load_carrier_ids. All load carriers to be deleted must be explicitly stated in load_carrier_ids.

The definition for the request arguments with corresponding datatypes is:
{
  "load_carrier_ids": [
    "string"
  ]
}
The definition for the response with corresponding datatypes is:
{
  "return_code": {
    "message": "string",
    "value": "int16"
  }
}

`detect_load_carriers`¶

Triggers a load carrier detection as described in Detection of load carriers.

Request:

The definition for the request arguments with corresponding datatypes is:
{
  "load_carrier_ids": [
    "string"
  ],
  "pose_frame": "string",
  "region_of_interest_id": "string",
  "robot_pose": {
    "orientation": {
      "w": "float64",
      "x": "float64",
      "y": "float64",
      "z": "float64"
    },
    "position": {
      "x": "float64",
      "y": "float64",
      "z": "float64"
    }
  }
}
Required arguments:

pose_frame: see Hand-eye calibration.

load_carrier_ids: IDs of the load carriers which should be detected.

Potentially required arguments:

robot_pose: see Hand-eye calibration.

Optional arguments:

region_of_interest_id: ID of the region of interest where to search for the load carriers.

Response:

The definition for the response with corresponding datatypes is:

{
  "load_carriers": [
    {
      "id": "string",
      "inner_dimensions": {
        "x": "float64",
        "y": "float64",
        "z": "float64"
      },
      "outer_dimensions": {
        "x": "float64",
        "y": "float64",
        "z": "float64"
      },
      "overfilled": "bool",
      "pose": {
        "orientation": {
          "w": "float64",
          "x": "float64",
          "y": "float64",
          "z": "float64"
        },
        "position": {
          "x": "float64",
          "y": "float64",
          "z": "float64"
        }
      },
      "pose_frame": "string",
      "rim_thickness": {
        "x": "float64",
        "y": "float64"
      }
    }
  ],
  "return_code": {
    "message": "string",
    "value": "int16"
  },
  "timestamp": {
    "nsec": "int32",
    "sec": "int32"
  }
}

load_carriers: list of detected load carriers.

timestamp: timestamp of the image set the detection ran on.

return_code: holds possible warnings or error codes and messages.

`detect_filling_level`¶

Triggers a load carrier filling level detection as described in Detection of filling level.

Request:

The definition for the request arguments with corresponding datatypes is:
{
  "filling_level_cell_count": {
    "x": "uint32",
    "y": "uint32"
  },
  "load_carrier_ids": [
    "string"
  ],
  "pose_frame": "string",
  "region_of_interest_id": "string",
  "robot_pose": {
    "orientation": {
      "w": "float64",
      "x": "float64",
      "y": "float64",
      "z": "float64"
    },
    "position": {
      "x": "float64",
      "y": "float64",
      "z": "float64"
    }
  }
}
Required arguments:

pose_frame: see Hand-eye calibration.

load_carrier_ids: IDs of the load carriers which should be detected.

Potentially required arguments:

robot_pose: see Hand-eye calibration.

Optional arguments:

region_of_interest_id: ID of the region of interest where to search for the load carriers.

filling_level_cell_count: Number of cells in the filling level grid.

Response:

The definition for the response with corresponding datatypes is:

{
  "load_carriers": [
    {
      "cells_filling_levels": [
        {
          "cell_position": {
            "x": "float64",
            "y": "float64",
            "z": "float64"
          },
          "cell_size": {
            "x": "float64",
            "y": "float64"
          },
          "coverage": "float64",
          "level_free_in_meters": {
            "max": "float64",
            "mean": "float64",
            "min": "float64"
          },
          "level_in_percent": {
            "max": "float64",
            "mean": "float64",
            "min": "float64"
          }
        }
      ],
      "filling_level_cell_count": {
        "x": "uint32",
        "y": "uint32"
      },
      "id": "string",
      "inner_dimensions": {
        "x": "float64",
        "y": "float64",
        "z": "float64"
      },
      "outer_dimensions": {
        "x": "float64",
        "y": "float64",
        "z": "float64"
      },
      "overall_filling_level": {
        "cell_position": {
          "x": "float64",
          "y": "float64",
          "z": "float64"
        },
        "cell_size": {
          "x": "float64",
          "y": "float64"
        },
        "coverage": "float64",
        "level_free_in_meters": {
          "max": "float64",
          "mean": "float64",
          "min": "float64"
        },
        "level_in_percent": {
          "max": "float64",
          "mean": "float64",
          "min": "float64"
        }
      },
      "overfilled": "bool",
      "pose": {
        "orientation": {
          "w": "float64",
          "x": "float64",
          "y": "float64",
          "z": "float64"
        },
        "position": {
          "x": "float64",
          "y": "float64",
          "z": "float64"
        }
      },
      "pose_frame": "string",
      "rim_thickness": {
        "x": "float64",
        "y": "float64"
      }
    }
  ],
  "return_code": {
    "message": "string",
    "value": "int16"
  },
  "timestamp": {
    "nsec": "int32",
    "sec": "int32"
  }
}

load_carriers: list of detected load carriers and their filling level.

timestamp: timestamp of the image set the detection ran on.

return_code: holds possible warnings or error codes and messages.

Load carrier functionality¶

Introduction¶

Load Carrier¶

Detection of load carriers¶

Detection of filling level¶

Interaction with other components¶

Stereo camera and Stereo matching¶

Estimation of gravity vector¶

IO and Projector Control¶

Hand-eye calibration¶

Parameters¶

Parameter overview¶

Description of run-time parameters¶

Services¶

set_load_carrier¶

get_load_carriers¶

delete_load_carriers¶

detect_load_carriers¶

detect_filling_level¶

`set_load_carrier`¶

`get_load_carriers`¶

`delete_load_carriers`¶

`detect_load_carriers`¶

`detect_filling_level`¶