A Reusable Camera Interface for Rovers Daniel Clouse Jet - PowerPoint PPT Presentation

A Reusable Camera Interface for Rovers Daniel Clouse Jet Propulsion Laboratory California Institute of Technology Pasadena, California Workshop on Software Development and Integration into Robotics 14 April, 2007

Acknowledgements  Coauthors - Issa Nesnas, Clayton Kunz  Sponsor - NASA Mars Technology Program  Ideas - Lorenzo Fluckeiger, Richard Madison, Michael McHenry, Hans Utz  Design Reviewers - I-Hsiang Shu, Jeffrey Edlund, Nik Melchoir, Robert Steele

CLARAty Overview Decision Explore Site Layer Deploy Acquire & Goto Target 3 Instrument Analyze Declarative Activity Acquire Image Goto Target 1 Functional Abstraction Swappable Algorithm or Rover Robot Adaptation Target Tracker Navigator Functional Falcon Pt Cloud Morphin Layer Stereovision Pose Estimator Locomotor JPLV SAPP R8_Model Camera Motor IMU 1394 Cam R8_Motor ISIS Rocky 8 ROAMS Rocky 7 ATRV Jr. http://claraty.jpl.nasa.gov

Application Scenario Show Movie of Single Cycle Instrument Placement http://claraty.jpl.nasa.gov/man/overview/movies/movie_viewer.php?file=FY05/scip_05.mp4&w=720&h=480

Application Scenario (Single Cycle Instrument Placement) Target Hand- off Target Designated Target time = t2 (avoiding an obstacle) time = t1 (a) (b) Camera Use Cases: • Monocular multi-resolution nav imaging for target tracking • Synchronized binocular stereo from navcams for target ranging and fine pointing • Synchronized quad stereo imaging from nav and hazcams for target hand- off • Synchronized binocular stereo from hazcams for obstacle avoidance

History of Camera Interface Revision 1  Single threaded  Did not allow for extension except along hardware axis Revision 2  Image acquisition is task safe  But camera settings may be changed by another thread  Bridge pattern to allow functional extensibility  But numerous classes; hard to adapt; never extended  Defined interface for setting/logging camera parameters  But difficult to use, extend

Revision 2 Class Hierarchy Device_Group Device Device_Impl Camera_Group Camera Camera_Impl Camera is extendible X_Hw_Camera X_Camera_Group Logical Camera Hierarchy Physical Camera Generic Classes Adaptation Classes Hardware Classes

Requirements Common tasks are easy.  Acquire an image  Set/Get camera parameters Full access to hardware. Avoid implementing unused features.  Video Synchronization within changing camera groups.  To support Firewire limits Task safety.  For both acquire and parameter setting

A Bad Solution for Task Safety Image<uint8_t> img1; X_Camera cam1(hw_cam1); cam1.lock(); cam1.set_brightness(0.35); cam1.acquire(img1); cam1.unlock(); Problems.  User must reestablish param settings for every lock.  Assumes user’s will write cooperative code.  Deadlock / Starvation are possible.

Logical/Physical Cameras for Task Safety Physical camera object represents a piece of hardware.  Interface is specific to the camera hardware  Parameter change happens immediately  Acquire uses current hardware parameter settings Logical camera object maintains a single user’s view.  Base class defines a common interface for all logical cameras.  Interface may be extended to support hardware-specific functions.  Parameter values are cached  Acquire atomically sets params in hardware and acquires image. No special user code is required for task safety. Image<uint8_t> img1, img2; X_Hw_Camera hw_cam1(id_unique_to_hw); X_Camera cam1(hw_cam1); cam1.set_brightness(0.35); cam1.acquire(img1); cam1.acquire(img2);

Revision 3 Class Hierarchy

Definition of Logical Camera Class class Camera : public Device { public: virtual bool set_contrast(double gain_percent) = 0; virtual double get_contrast() const = 0; virtual bool set_brightness(double offset_percent) = 0; virtual double get_brightness() const = 0; virutal bool set_exposure(double seconds) = 0; virutal double get_exposure() const = 0; enum IMAGE_FORMAT { MONO8, YUV411, …, RGB8, MONO16, RGB16 }; virtual bool set_format(IMAGE_FORMAT format, int width, int height); virtual IMAGE_FORMAT get_format() const = 0; virtual int get_width() const = 0; virutal int get_height() const = 0; virtual void acquire(Image<uint8_t> & image, Time* timestamp = NULL, Feature_Map* feat_map = NULL) = 0; virtual void acquire(Image<uint16_t> & image, Time* timestamp = NULL, Feature_Map* feat_map = NULL) = 0; };

Synchronized Acquisition: Problem Snap images from multiple cameras at the same instant. For framegrabber cameras, synch signal is on dedicated wire.  The subset of synchable cameras is defined by the hardware.  One framegrabber channel per camera, so bandwidth is not an issue.  Configuration is generally static. For Firewire cameras, synch signal is on the bus.  Any subset of cameras on the same bus may be synced.  Other constraints (bus bandwidth, DMA slots) limit the size of the sync set.  Configuration is dynamic. We need to support both models cleanly.

A Solution: Transient Camera Groups Device_Group specifies a set of devices.  Devices may be added/removed from the set dynamically.  Device_Groups may be created/destroyed dynamically.  Provides for controlling power to devices. Derived Camera_Group specifies set of syncable logical cameras .  A logical camera may belong to more than one Camera_Group.  Acquire synced images by calling Camera_Group::acquire(). X_Camera_Group is derived for each camera implementation.  Implements sync mechanism for particular hardware.  Enforces hardware-specific constraints (DMA channels, bus bandwidth, which framegrabbers are connected, etc).

Definitions of Grouping Classes class Device_Group { public: unsigned int size() const; void append(Device& device); void remove(Device& device) throw(invalid argument); }; class Camera_Group : public Device_Group { public: virtual int acquire(vector<Image<uint8_t>* > & img_vec, vector<Time> * timestamp_vec = NULL, vector<Feature_Map*> * feature_map_vec = NULL); virtual int acquire(vector<Image<uint16_t>* > & img_vec, vector<Time> * timestamp_vec = NULL, vector<Feature_Map*> * feature_map_vec = NULL); };

Camera_Group Code Example X_Hw_Camera hw_cam1(id1), hw_cam2(id2); X_Camera cam1(hw_cam1), cam2(hw_cam2); X_Camera_Group grp(cam1, cam2); cam1.set_brightness(0.35); cam2.set_brightness(0.35); Vector<Image<uint8_t> > images(2); grp.acquire(images);

Implementation of MR1394_Camera Firewire cameras controlled via MindReady driver. Camera class hierarchy presents a simple interface to the user. The apparent simplicity hides some implementation problems:  When possible, maintain cameras in video mode for fast acquisition.  Resource bookkeeping (DMA channels, bus bandwidth) across multiple threads.  Avoid changing camera parameters with each acquire .

Lessons Learned Good interfaces don’t come easy.  Thinking about the design is essential.  But experience gained from implementation and use may lead to improvement.  Don’t be afraid to re-implement. Interface flexibility is not always a win.  Example: Bridge pattern used in revision 2.  Generality adds complexity, difficulty of maintenance. Adding reusability as a requirement makes the implementation of an interface more complicated.  Presentation of false simplicity to the user can make the user’s life easier.  But makes the implementation more difficult.  Increased implementation difficulty is only justifiable if offset by the savings experience by large number of users, applications, etc.

Backup

Different Hardware Architecture Synchronized Synchronized Monocular Monocular stereo camera stereo camera camera camera FireWire Serial Bus / Analog signals Digital signals Video Switcher Backplane or SBC Backplane (VME, PCI) Processor board or Processor board Single board computer (SBC) Image acquisition boards Serial bus / FireWire boards Inertial Digital I/O board Measurement Unit Analog I/O board Wireless ethernet Serial Bus / Digital signals Microprocessors Digital & Analog signals Analog signals Digital I/O Potentio- Analog I/O meters Serial comm Accelero- Science meters Instrument Gyroscopes Actuator / Encoders / Actuator / Encoders / Potentiometers Potentiometers