I'm going to be using this issue as a place to track progress on a large refacto…r of rviz which will hopefully land in the next version of ROS, ROS Melodic or ROS [Melodic Morenia](https://discourse.ros.org/t/ros-lunar-loggerhead-release/1870), as well as be used in ROS 2.
At a high level the goal is to break rviz up into a few parts, separating as much as possible the non-ROS specific parts from the ROS specific parts, so that as much code as possible can be reused in ROS 1 and ROS 2.
This will be sort of an experiment to see how realistic it is to approach ROS 2 porting this way. And so, if anything with the approach is not working out then I may adjust the plan to try and compensate for that.
---
# Status
TODO: I will fill this out more as work materializes.
---
# The Plan
TODO: I'll continue to refine and clean this up as I go, but this is just a dump of my personal notes at the moment.
I'll try to use rviz in reference to ROS 1 and rviz2 in reference to ROS 2.
## Requirements
These requirements are things that would be best to have in ROS 2 to consider rviz2 functional:
- Support all platforms that ROS 2 supports (Linux, macOS, Windows)
- Support plugins (like rviz in ROS 1)
- Display Type
- Tool (button on the tool bar and controls cursor/modality)
- Panel (dockable window in the GUI)
- View Controller
- Support many displays in default display type plugins:
- Markers (all the current marker types)
- tf
- Robot display
- image
- depth image
- PointCloud2
- map
- Grid
- Laser Scan
- Support certain tools:
- move
- initial pose
- goal
- Support certain view controllers
- Orbit
- Panels:
- Displays (add/remove/config)
- Views (select/config/zero)
- Toolbar (is this a panel?)
- Notifications (rviz has poor visibility on most problems, should have a space for it)
- rviz as a widget
- i.e. should be possible/easy to embed rviz2 in another qt app
- rviz as a component (share a process with other nodes)
- save/load configuration files
## Later Requirements
- More Display types:
- Axes
- Effort
- Camera
- Grid Cells
- InteractiveMarker
- Path
- Point
- Pose Array
- Polygon
- Odometry
- Range
- Wrench
- More View Controllers:
- fixed orientation ortho
- fps
- third person follower
- xy orbit
- More tools:
- focus
- measure
- point
- pose
- selection
- More panels:
- time
- selection
## Goals
- Fork rviz repository to share history and contributions
- Try to share code with rviz and rviz2
- break rviz into reusable pieces (same repo where possible), which can be used to build rviz2
- later refactor rviz to use same pieces, as much as possible
- Find a reliable way to automatically test rendering
- Use ament_index for plugins in rviz2 (should be faster RPP)
## Ideas
- Remote control rviz2's display types, settings, view controllers via ros services
- Share code with, or reuse, ign-render (need to talk with @iche033)
- Make rviz plugins nodes rather than custom rviz plugins?
- Create a way to make rviz plugins that can be used in either rviz or rviz2
- my idea for this would involve a new way to write a plugin (different from now in rviz)
- it could work in both because it wouldn't use neither roscpp or rclcpp nor use tf or Ogre directly
- instead it would abstract each of those
- handling/abstracting the message types might be the most difficult part
- the goal would be to allow porting an rviz plugin to make it future proof, or share an impl between both versions
- rosbag remote control panel, i.e. start, stop, set rate, loop, scrubber, etc
- might be better to just have rviz and a tool like this in rqt together
## Design Decisions
### GUI Library
Use Qt 5.
It's the obvious choice for portable windowing in a desktop environment.
It has a large user base, and good documentation.
rviz from ROS 1 uses it so it will maximize reuse of code.
#### Alternative - Web
I looked again at robot web tools and recent advancements in emscripten (c++->llvm->asm->javascript), webgl, and other stuff.
I still think starting with C++ and Qt is more viable.
Qt is still less risky and likely to be much more performant for now.
#### Alternative - Electron
Electron is the node.js + webkit app sdk that atom.io is built on, as well as Visual Studio - Code.
My experience with it is that it can be performance bound, especially if you don't know what you're doing.
It also brings a whole new set of dependencies and tools/languages/libraries for use to learn.
On the other hand, it would potentially open the door for other developers with those skill sets to get involved.
Ultimately it sounds risky, potentially slow, and brings in a lot of new dependencies.
### 3D Engine
Use Ogre, version 1.10.
Ogre is what rviz uses, so it will make porting the easiest.
Ogre is already portable (mac, linux, windows, ios, android, etc...), well licensed (MIT), and familiar.
It is also what Gazebo uses, so we can drawn their developers, experience, and/or code as well.
There are two main Ogre versions right now, 1.x and 2.x, see:
http://www.ogre3d.org/about/what-version-to-choose
2.1 seems cool, but it doesn't yet support macOS (will soon with new Metal API) and it doesn't support older machines and opengles 2 or 3.
In general it is a lot less portable, but it would be cool to see how hard it would be to make rviz2 able to use it optionally to try it out.
1.10 supports the platforms and features we need.
rviz also currently supports it so it will make porting easier.
#### Alternative - OpenGL
Using opengl or opengles or webgl or ... whatever has been proposed in the past, since what rviz does it relatively simple most of the time.
It might also make things a little more performant in a few cases, but Ogre provides us a lot, especially when dealing with meshes and textures and such.
#### Alternative - Unity, Unreal, or some other popular game engine
These are all either free for some cases or semi-open, but most are not under an open license for all cases.
The license is a big part of why Ogre rather than one of these.
Also these engines tend to be a lot more batteries included on project layout and management which makes it more difficult to integrate with Qt and CMake and all the other typical things we use.
#### Alternative - VTK
This is Kitware's visualization toolkit library, which is used by lots of scientific applications to render data.
It is also used for visualization within PCL.
This is a serious contender, as it has efficient and well developed mechanisms to render data.
I'm not familiar with how well it can handle meshes and things like that, which is a place where rviz's needs are different from PCL, for example.
This is a serious alternative, because it is purpose built for data visualization (as opposed to Ogre which is really a game engine), it's portable, and it has a decent license (BSD-like http://www.vtk.org/licensing/).
In the past it has been a dependency pain point, but it has become more stable in recent years.
I could still be swayed to use this rather than Ogre, but given that code sharing with rviz is a priority, and given that Ogre is more likely to be portable to other systems (iOS and Android for example), I have a slight preference for Ogre still.
#### Alternative - ITK
ITK is Kitware's data visualization tool based on VTK, which is similar to rviz in a lot of ways.
I talked with the Kitware people a while back and talked about using ITK directly instead of rviz being a custom app.
My conclusion was that it might work, but some work would need to be done to ITK to meet our use cases, so we'd be subject to the flexibility of ITK.
For one thing, ITK doesn't have a notion of frames, so we'd still have to do all the data transforms before giving them to ITK.
Another detractor for me was that ITK is a swiss army knife and it would be a lot of work (or not possible) to whittle down the interface so that it is as simple and lean to use as rviz is now.
This is quite a different direction to take, but a decent contender nonetheless.
### Mesh Asset Library
Use Assimp.
This is what rviz uses now.
Seems to work.
#### Alternative - Custom parsing code
Gazebo does custom parsing of collada, for example.
Doesn't seem sustainable, assimp has worked so far for rviz.
## Plan
As described above, the plan is to break rviz into parts some of which will be reused in ROS 2 and ROS 1, while others will be duplicated in ROS 1 and ROS 2 because they have communication specific stuff in it.
### Components
These are the distinct components I can foresee right now (will likely change as it becomes clearer):
- render_frame (reusable by rviz)
- takes Ogre objects only
- does not depend on `ros_comm`/`rclcpp` or message types
- basically marries Qt and Ogre, plus hooks up correctly to get input out and data to render in
- `rviz/src/rviz/ogre_helpers/*` and `rviz/src/rviz/render_panel.*`, maybe others in rviz
- plugin loading
- might be replaced with pluginlib port to ros 2
- just finding plugins and loading them
- plugin API's (maybe reusable by rviz)
- just the API for custom displays, views, tools, and panels
- they could be broken up by type, maybe based on dependencies
- views, tools, and panels could probably avoid dependencies on rclcpp and messages
- displays might be tougher to decouple from ros stuff
- display primitives (maybe reusable by rviz)
- things that are not in Ogre's API, but are common between rviz and rviz2
- for example the code to render Axes (input would be pose, output would be Ogre objects)
- other examples which might fit here, image, camera, point cloud, depth image, path, others...
- could avoid dependency on message types with an additional abstraction (might not be worth it)
- would be code in `rviz/src/rviz/ogre_helpers/*`
- default plugins
- implementation of default displays, panels, tools, and view controllers
- would implement plugin API's and be loaded with plugin loading
- would render to render frame, possibly using display primitives or custom Ogre stuff instead
- rviz2
- Qt app that combines render_frame and default plugins to make complete rviz2 app