Migration to one GitHub repo for MoveIt!

Community Groups Manipulation
1

We are planning to consolidate most of the moveit* Github repositories to a unified “moveit” repo for the following advantages:

  • Easier to create pull requests that currently span multiple repos
  • Easier to test pull requests when working on a different fork or branch locally
  • Less duplicate work when maintaining continuous integration, creating distro branches, etc
  • Easier for beginners to find code in one central place

Disadvantages:

  • Will break everyone’s current source build workspace layout
  • Will be effort to migrate current issues, branches, pull requests to the new repo
  • All packages must be released at the same time

Goals:

  • Document the merge process exactly so that developers can replicate locally their feature branches
  • Do some test merges first before switching
  • Merge the git histories of the various repos
  • Merge for all current branches - Indigo, Jade, and Kinetic
  • Merge as many PRs first to reduce future sync issues

Method:

  • For at least phase one of the migration, directly copy in the package name as a subfolder in the new “moveit” repo
  • Do not flatten any subfolders that contain multiple subpackages i.e. “moveit/moveit_ros/planning” could become “moveit/moveit_ros_planning” since this better represents the package name, but this should not be in phase 1
  • Notably we will keep moveit_msgs and moveit_pr2 separate.

Timeline:

  • August 5th, 2016
  • Target ROS-Kinetic and begin releasing debians for Kinetic from this repo

Target Repos:

  • moveit_core
  • moveit_ros
  • moveit_planners
  • moveit_plugins
  • moveit_setup_assistant
  • moveit_commander
  • moveit_ikfast
  • moveit_experimental

Experimental Repo:
https://github.com/ros-planning/moveit

Migration notes:
https://github.com/davetcoleman/moveit_merge

6 Likes
2

+1 for this in general; I think the advantages significantly outweigh the disadvantages. Especially having one place to look for issues, and being able to submit a single PR that spans a bunch of packages.

One small note: there is also a slight disadvantage to having “desktop” and “robot” stuff in the same repo (although moveit_ros already has this issue). If someone clones the repo into a catkin workspace on a robot and wants to build everything, they’ll need to install rviz (for example), which will in turn install a ton of graphical dependencies. This is not good on a robot that may be running ubuntu server. This can be worked around by having deployment scripts that (through selective copy or rsync or symlinking) create a workspace with only the packages you want to build. Just something to think about though.

3

+1 generally a good idea to have far fewer repositories

You might want to consider separating the moveit_ros repo from the ROS-independent components. I also agree with Jon’s comment about keeping on-board components separate from desktop stuff.

Even if you end up with 2 or 3 repos, that’s still a big improvement over the current plethora.

4

This is going to be a ‘me too +1’ post, but the point Jon raises is indeed something to be careful about. catkin_tools can better cope with this, but it would need to be documented clearly, and a blanket rosdep install .. /path/to/dir/containing/moveit/clone step won’t work anymore either.

5

I do see the advantages of having a unified repo for development. On the other hand the migration will create a lot of pain and I’m not sure, that we can target this for a Kinetic release, which should be out as soon as possible.
How often did you observed changes across the different repos in the past? From my impression, they are nicely separated from each other. Of course, introducing new features and API changes in core have to be reflected in downstream packages…
In order to test PRs, I have setup a separate workspace to not conflict with my own development environment and I have separate .rosinstall files to easily switch between branches in all repos simultaneously.

Altogether, I’m undecided whether to merge or not. Pros and cons for both options are balanced in my opinion.

6

Other disadvantages:

I know that the NASA JSC team working on Valkyrie has many packages in separate repositories, but they encourage all issues to be filed in a single repo that also hosts the wiki ( https://github.com/NASA-JSC-Robotics/valkyrie/ ). That helps with issues, but not pull requests.

I’m personally “-1”: I think the disadvantages outweigh the advantages.

7

I’ll abstain from voting, but I also see the pros and cons. I don’t know, but the advantages may outweigh the cost.

However, I didn’t see anyone mention the constraint that bloom can only release multiple packages from the same repository when they share the same version number. So if packages across the different repositories have different version numbers now, then part of the migration will be to jump them all up to a common version that’s newer than all previous versions.

2 Likes
8

That’s what I meant by

But I believe most of MoveIt! already has the same version numbers, because the stuff I’m suggesting we merge is so dependent on each other

9

Oh, ok, sounds good to me :+1:

10

Yeah, it’s always a balancing act. And for every project it’s always different.

On one end everything is released separately and completely independent. On the other end it’s one big repo that gets released monolithically.

I believe that MoveIt is farther toward the many packages model since there are lots of coupled releases. However I do suggest not overcompensating and going to the other extreme of one repo as well. A good discussion on a case by case basis of how closely coupled repos are is probably worth conducting.

Good cases for keeping things independent generally are things like plugins and other things that are more loosely coupled and bring in large extra dependencies. (Note that MoveIt having core plugins means that this rule of thumb is probably not exactly appropriate for this project.)

2 Likes
11

Lots of good points here. I look at MoveIt! similar to PCL, OpenCV, and the navigation stack - all which are centralized under one repo. If companies like Google can maintain their vast amount of software on a single repo, it seems like our modest MoveIt! can also. I understand there are corner cases that might require manually pulling out a package, but I don’t think they warrant the drag it has caused on the agility of the MoveIt! project’s progress.

they’ll need to install rviz (for example), which will in turn install a ton of graphical dependencies. This is not good on a robot that may be running ubuntu server

As you mentioned, simply add CATKIN_IGNORE, delete the folder, or using catkin-tools’ blacklist feature

consider separating the moveit_ros repo from the ROS-independent components

moveit_core was suppose to be ROS agnostic, but it in fact does have ROS dependencies such as ROS Time, ROS Package, ROS Console, and lots of ROS messages. I also believe no one is using it outside of ROS, so why should we waste so much developer time maintaining this separation?

I’m not sure that we can target this for a Kinetic release

We are going to be using Kinetic until 2021, so if we don’t target it we can’t really migrate because of the need for backports, etc. Considering Kinetic is not released yet, and even if it is we can still switch over, I think we should try our best for it.

How often did you observed changes across the different repos in the past?

Many times. Many of the features MoveIt! lacks require deep architecture changes across, at minimum, moveit_core and moveit_ros. For example: “KinematicsBase changes” required these PRs:
https://github.com/ros-planning/moveit_core/pull/231
https://github.com/ros-planning/moveit_ros/pull/581
https://github.com/ros-planning/moveit_ikfast/pull/50

Even something as simple as upgrading to Eigen3 requires me filling out many many PR pages across MoveIt!

git clone takes much longer even if someone just wanted a single package

I believe almost everyone who builds MoveIt! from source already downloads all the separate repos per the Quick Start guide, so it shouldn’t change the download time, right? Of course if our repo gets too big we could delete some of the history.

releasing and building a subset of packages is more of a hassle for meta-packages

good point

12

If companies like Google can maintain their vast amount of software on a single repo, it seems like our modest MoveIt! can also.

Google also employs many people to maintain that infrastructure and to provide tools to make the workflow easy to use. While I agree fewer repositories is easier for moveit’s developers in this situation, I don’t think you’re comparing apples to oranges when you compare moveit in a single repository to all of Google’s code in a single Perforce repo (which is also different from git – it’s not like Google uses a single git repository).

Again, not arguing against the move, just pointing out that Google’s setup is very different from both what moveit has now and what you’re proposing.

Considering Kinetic is not released yet, and even if it is we can still switch over, I think we should try our best for it.

Kinetic was release on May 23rd, 2016, but maybe you mean moveit for Kinetic.

Of course if our repo gets too big we could delete some of the history.

I can’t -1 this enough.

2 Likes
13

Good points on Git vs. Perforce. I did mean MoveIt!'s release of Kinetic. By some of the history, I was referring to old commits that have large file size or unused branches.

14

Thanks @davetcoleman for taking the time to have this discussion - I agree that a lot of good points have been brought up, and I like your summary in that last comment.

CATKIN_IGNORE helps avoid the need for big dependencies when only building a few packages. I was thinking more of the problem of doing “rosdep install --from-path src” but that can also be worked around by installing deps only for the packages/subdirectories that you want to build. To be clear, I think that these issues are far outweighed by the advantages that have been listed of moving to one repo. Especially if tutorials are added/updated to show people useful workflows.

15

Ok, that’s good, so long as you’re removing unused branches and stuff. Removing actual history would not be good. That brings up another thing to watch out for, which is that it would be best to try to preserve the history when merging the repositories. It’s easy to accidentally miss the history when moving the files.

1 Like
17

I instantly favored this idea when I heard it in a ROSCon 2015 video where Dave mentioned it. I’m still +1 now.
As I recently started helping to make releases of MoveIt!, I add some thoughts to the maintainer’s perspective. Not all / none of these may be of significant importance, but hoping these help for some purpose:

Advantages is to reduce the cost of:

  • back/forwardporting between branches; Doing so involves at least: paying attention to any existing portable commits/PRs, cherry-picking commits, opening PRs, waiting for CI result and adding correction if needed, waiting for the reviews, discussion if needed. Spanning over multiple repositories just multiplies the cost. Ideally the original authors of PRs take care of porting, but I doubt we can force that and thus it’s one of maintainers’ responsibility currently (at least at MoveIt!).

  • managing version;

    But I believe most of MoveIt! already has the same version numbers, because the stuff I’m suggesting we merge is so dependent on each other

    Actually there are slight differences. For example, to cope with the difference between Indigo and Jade, I incremented the minor version of Jade to 0.8.x. Version ain’t nothing but a number? No it takes some thoughts, and the more repositories we have the more time needs to be spent among maintainers.

  • sorting changelog; I tend to spend some time to edit changelog upon making a release to group into categories (fix, feature etc.) since bulk of raw commit messages don’t make much sense to me personally. This gets more important I think for packages with vast amount of users. Again the more the repos the more the time it takes, and moreover more mental stress.

  • running prerelease test manually; first going to web and get the necessary script snippet, and run the snippet locally. Again the mroe the repos the more the time, which is non-trivial amount for volunteer maintainers.

    • BTW there’s a package called industrial_ci that has a feature to start the ROS prerelease test from PR (usage).
18

This week’s example: currently @v4hn is adding a much needed feature to MoveIt! but he has to create 4 PR’s to apply it throughout the code base. And if you include adding documentation, 5! This means I get tons of emails for all these PRs, have to merge them in the correct order, and its very difficult to make sure they all work together correctly.

https://github.com/ros-planning/moveit_msgs/pull/21
https://github.com/ros-planning/moveit_core/pull/297
https://github.com/ros-planning/moveit_ros/pull/686
https://github.com/ros-planning/moveit_setup_assistant/pull/115

1 Like
19

I’ve created an automated merge script that keeps each repo’s git history in place. I’ve tested this merge script with Jade and it passes CI except for some previously disabled tests that are now enabled - you can see the experimental merged repo here.

Thanks to @130s and @v4hn for their input on this.

1 Like
20

Sorry if I say anything obviously stupid here… I’m quite new to ROS and MoveIt.

From my end-user point of view, ROS Kinetic is the recommended version since its release on May 23rd 2016 and compatible with Ubuntu’s latest 16.04 LTS. MoveIt is an essential part of the ROS ecosystem, and should therefore become quickly available for new ROS releases.

So, I don’t think that this change should block an initial MoveIt release for Kinetic. Even if the initial one is done from the multiple repos as before, it should not prevent future MoveIt bug fix releases for Kinetic to be done from the unified repo.

But maybe I’m totally wrong and this subject isn’t what’s currently blocking the release.

1 Like
21

@dbolkensteyn That’s a valid comment. Thank you for sharing your thoughts!
I’ll wait for “more-core” maintainers to chime in, but as @gavanderhoorn commented on answers, main reason why MoveIt! hasn’t been released into Kinetic is the dependency. MoveIt! is a huge set of software, so is the number of the depended packages.

Also we’re making fundamental changes that will require far less maintenance effort, one of which as discussed above is to consolidate the distributed repositories into fewer repo. Since ROS Kinetic is “LTS”, meaning once it’s released we won’t be able to make a substantial change for 5 years, we’re rather taking time now to make as much work done as possible. In fact some maintainers (notably @davetcoleman @v4hn @rhaschke) have been really working hard these few weeks for making it happen for Kinetic.

3 Likes