realsense component

Driver for the Intel realsense cameras.

Multiple image (multiple out) output ports can be published, organized into individual banks. Each bank contains a number of coherent images, e.g. time synchronized.

The internal architecture implements a graph of named source and sink slots, which can be published individually into images of a bank. Hardware-dependent functions can stream raw data into slots, which can then filtered or otherwise processed by generic services into other slots and eventually published with publish (activity).

Note that the term slot in this documentation always refers to an internal image of the processing graph, while the term bank refers to a published image in a bank of the port image (multiple out).

In addition to hardware-specific services, there are generic services, common to all camera interface providers. These include basic filtering, recording and publishing functions.

Ports

image (multiple out)

Data structure

struct ::or_image::bank image
- struct ::or::time::ts ts
  - long sec
  - long nsec
- sequence< struct ::or_image::frame, 8 > images
  - struct ::or_image::info info
    
    unsigned long seq
    
    optional< struct ::or::sensor::pixmap_roi > roi
    
    unsigned long subsx
    
    unsigned long subsy
    
    unsigned long x
    
    unsigned long y
    
    unsigned long width
    
    unsigned long height
    
    optional< struct ::or_image::distortion > distortion
    
    enum ::or_image::distortion_model model ∈ { R3T2 }
    
    sequence< double, 8 > coeffs
    
    optional< struct ::or_image::intrinsic > intrinsics
    
    double fx
    
    double fy
    
    double cx
    
    double cy
    
    optional< struct ::or_image::extrinsic > extrinsics
    
    struct ::or::t3d::pos T
    
    double x
    
    double y
    
    double z
    
    struct ::or::t3d::att R
    
    double qw
    
    double qx
    
    double qy
    
    double qz
  - struct ::or::sensor::pixmap pixmap
    
    unsigned long seq
    
    unsigned long width
    
    unsigned long height
    
    unsigned long stride
    
    enum ::or::sensor::pixmap::encoding pixfmt ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }
    
    sequence< octet > data

Published image banks.

Images in a bank are considered as a coherent set of individual frames, hence sharing a common ts timestamp information.

Each image in the array contains its own metadata information info and a pixmap. The seq member represents a sequence number and different images read from the same bank index with the same seq number should be considered identical. This is especially useful to check if the calibration data has changed.

info.roi indicates that the image data was extracted from a bigger image. In particular, the intrinsics refer to the original image dimensions.

info.distortion represents the image distortion information and the associated correction model.

info.intrinsics is the intrinsic matrix mapping from 2D coordinates to 3D information, according to the following model where u,v are image pixel coordinates, w a free parameter and X, Y, Z 3D coordinates in the camera frame:

| w.u   | fx  0 cx | | X
| w.v = | 0  fy cy |.| Y
| w     | 0   0  1 | | Z

info.extrinsics is a 3D transformation converting 3D coordinates from the camera frame to another frame common to all images in a bank:

| X'     | X
| Y' = R.| Y + T
| Z'     | Z

pixmap.data contain the actual pixel values, arranged as a row-major 2D array of height×width pixels. The width in bytes of each pixel row is indicated by stride.

Each pixel has the format pixmap.pixfmt, defining the number of channels: MONO is 1 integer grayscale channel, RGB or BGR are 3-color integer channels, possibly with an additional transparency channel A. Z and XYZ are 1 and 3 floating point depth channels repectively. The number of bits encoding each channel value is suffixed. JPEG indicates a compressed RGB color image.

accel (out)

Data structure

struct ::or_pose_estimator::state accel
- struct ::or::time::ts ts
  - long sec
  - long nsec
- boolean intrinsic
- optional< struct ::or::t3d::pos > pos
  - double x
  - double y
  - double z
- optional< struct ::or::t3d::att > att
  - double qw
  - double qx
  - double qy
  - double qz
- optional< struct ::or::t3d::vel > vel
  - double vx
  - double vy
  - double vz
- optional< struct ::or::t3d::avel > avel
  - double wx
  - double wy
  - double wz
- optional< struct ::or::t3d::acc > acc
  - double ax
  - double ay
  - double az
- optional< struct ::or::t3d::aacc > aacc
  - double awx
  - double awy
  - double awz
- optional< struct ::or::t3d::pos_cov > pos_cov
  - double cov[6]
- optional< struct ::or::t3d::att_cov > att_cov
  - double cov[10]
- optional< struct ::or::t3d::att_pos_cov > att_pos_cov
  - double cov[12]
- optional< struct ::or::t3d::vel_cov > vel_cov
  - double cov[6]
- optional< struct ::or::t3d::avel_cov > avel_cov
  - double cov[6]
- optional< struct ::or::t3d::acc_cov > acc_cov
  - double cov[6]
- optional< struct ::or::t3d::aacc_cov > aacc_cov
  - double cov[6]

gyro (out)

Data structure

struct ::or_pose_estimator::state gyro
- struct ::or::time::ts ts
  - long sec
  - long nsec
- boolean intrinsic
- optional< struct ::or::t3d::pos > pos
  - double x
  - double y
  - double z
- optional< struct ::or::t3d::att > att
  - double qw
  - double qx
  - double qy
  - double qz
- optional< struct ::or::t3d::vel > vel
  - double vx
  - double vy
  - double vz
- optional< struct ::or::t3d::avel > avel
  - double wx
  - double wy
  - double wz
- optional< struct ::or::t3d::acc > acc
  - double ax
  - double ay
  - double az
- optional< struct ::or::t3d::aacc > aacc
  - double awx
  - double awy
  - double awz
- optional< struct ::or::t3d::pos_cov > pos_cov
  - double cov[6]
- optional< struct ::or::t3d::att_cov > att_cov
  - double cov[10]
- optional< struct ::or::t3d::att_pos_cov > att_pos_cov
  - double cov[12]
- optional< struct ::or::t3d::vel_cov > vel_cov
  - double cov[6]
- optional< struct ::or::t3d::avel_cov > avel_cov
  - double cov[6]
- optional< struct ::or::t3d::acc_cov > acc_cov
  - double cov[6]
- optional< struct ::or::t3d::aacc_cov > aacc_cov
  - double cov[6]

Services

publish (activity)

Inputs

string<16> slot (default "") Slot name
string<16> bank (default "") Published bank name
octet index (default "0") Published bank index

Throws

exception ::camera::einval
- string<128> what

Context

In task worker0
Updates port image (multiple out)

Start publishing a named slot in a port.

This exports the internal image slot into the bank port at the position index in the bank.

unpublish (activity)

Inputs

string<16> bank (default "") Published bank name
octet index (default "0") Published bank index

Throws

exception ::camera::einval
- string<128> what

Context

In task worker0
Updates port image (multiple out)

Stop publishing a port.

See publish (activity).

jpeg (function)

Inputs

string<16> input (default "") Input slot name
unsigned short quality (default "80") Compression quality [0-100]
string<16> output (default "") Output slot name

Throws

exception ::camera::einval
- string<128> what

Compress a slot using JPEG compression.

This converts the input slot to the output slot output using JPEG compression, with the specified quality.

record_mpeg (function)

Inputs

string<16> input (default "") Input slot name
unsigned short width (default "640") Image width
unsigned short height (default "480") Image height
unsigned short fps (default "25") Image height
string<128> output (default "") Output file (.avi extension)

Throws

exception ::camera::einval
- string<128> what

Record a slot using MPEG compression in an AVI container.

The input slot is first resized to the desired width and height and compressed into the output file. The playback video frame rate is fixed and equal to fps, no matter the input slot rate. The output file name must end with .avi.

Additionnaly, an output slot named after the output file name, with any leading slash (/) stripped, contains the last compressed image, appropriately scaled to width and height.

cancel (function)

Inputs

string<16> slot

Throws

exception ::camera::einval
- string<128> what
exception ::realsense::e_rs
- string<128> what

Cancel all jobs publishing to slot.

The effect of this function is that slot will no longer be updated and the associated processing will be cancelled. This is useful to reconfigure the processing graph with e.g. different parameters or different inputs.

status (function)

Inputs

string<16> slot

Throws

exception ::camera::einval
- string<128> what

Raise any processing errors associated with slot.

If, for some reason, a slot fails to be updated, the error will be available by checking the result of this function. Processing errors typically include wrong pixel format for a filter or wrong image dimensions.

get_streams (function)

Outputs

sequence< enum ::realsense::stream, 16 > streams ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Get available streams.

get_stream_indices (function)

Inputs

enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Outputs

sequence< long, 16 > indices

Get available indices for a given stream type.

get_stream_fps (function)

Inputs

enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Outputs

sequence< long, 16 > fps

Get available streaming rates for a given stream type.

get_video_stream_formats (function)

Inputs

enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Outputs

sequence< enum ::or::sensor::pixmap::encoding, 16 > format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }

Throws

exception ::realsense::einval

Get available video streaming formats for a given stream type.

get_depth_stream_formats (function)

Inputs

enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Outputs

sequence< enum ::or::sensor::pixmap::encoding, 16 > format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }

Throws

exception ::realsense::einval

Get available video streaming formats for a given stream type.

get_video_stream_sizes (function)

Inputs

enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Outputs

sequence< struct ::realsense::video_size, 16 > sizes
- long width
- long height

Throws

exception ::realsense::einval

Get available video streaming sizes for a given stream type.

get_depth_stream_sizes (function)

Inputs

enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Outputs

sequence< struct ::realsense::video_size, 16 > sizes
- long width
- long height

Throws

exception ::realsense::einval

Get available depth streaming sizes for a given stream type.

stream_video (function)

Inputs

enum ::realsense::stream type (default "::realsense::STREAM_COLOR") Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
short index (default "-1") Index for streams of the same type (-1 for any)
short width (default "0") Image width (0 for any)
short height (default "0") Image height (0 for any)
enum ::or::sensor::pixmap::encoding format (default "::or::sensor::pixmap::ANY") Pixel format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }
short fps (default "0") Frames per second (0 for any)
string<16> name (default "color") Slot name

Throws

exception ::realsense::einval
exception ::realsense::e_rs
- string<128> what

Stream video data.

The stream type should be one of STREAM_COLOR or STREAM_INFRARED and index one of the available index returned by get_stream_indices (function), or -1 for a default one. Image is made available in the named output slot.

stream_depth (function)

Inputs

enum ::realsense::stream type (default "::realsense::STREAM_DEPTH") Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
short index (default "-1") Index for streams of the same type (-1 for any)
short width (default "0") Image width (0 for any)
short height (default "0") Image height (0 for any)
enum ::or::sensor::pixmap::encoding format (default "::or::sensor::pixmap::ANY") Pixel format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }
short fps (default "0") Frames per second (0 for any)
string<16> name (default "color") Slot name

Throws

exception ::realsense::einval
exception ::realsense::e_rs
- string<128> what

Stream depth data.

The stream type should be STREAM_DEPTH and index one of the available index returned by get_stream_indices (function), or -1 for a default one. Image is made available in the named output slot.

stream_motion (function)

Inputs

enum ::realsense::stream type (default "::realsense::STREAM_GYRO") Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
short index (default "-1") Index for streams of the same type (-1 for any)
short fps (default "0") Frames per second (0 for any)

Throws

exception ::realsense::einval
exception ::realsense::e_rs
- string<128> what

Stream motion data.

The stream type should be STREAM_GYRO or STREAM_ACCEL and index one of the available index returned by get_stream_indices (function), or -1 for a default one. Data is published the corresponding port: gyro (out) or accel (out).

Tasks

worker0

Context

Free running
Updates port image (multiple out)

Worker task 0 processing and publishing frames.

Worker tasks are independant of each other can do parallel processing of multiple images.

worker1

Context

Free running
Updates port image (multiple out)

Worker task 1 processing and publishing frames.

main

Context

Free running
Updates port accel (out)
Updates port gyro (out)

Project

General

Profile

Image processing software » realsense-genom3

Indexes

realsense component

Ports

image (multiple out)

accel (out)

gyro (out)

Services

publish (activity)

unpublish (activity)

jpeg (function)

record_mpeg (function)

cancel (function)

status (function)

get_streams (function)

get_stream_indices (function)

get_stream_fps (function)

get_video_stream_formats (function)

get_depth_stream_formats (function)

get_video_stream_sizes (function)

get_depth_stream_sizes (function)

stream_video (function)

stream_depth (function)

stream_motion (function)

Tasks

worker0

worker1

main