Skip to end of metadata
Go to start of metadata

Migration of the RHQ source code to GitHub.

Abstract

The purpose of this page is to prepare the move of the RHQ sources from git.fedorahosted.org to Github.com

The motivation is to use the existing tooling around GitHub and their infrastructure for easy forking of the project and

There exists already an rhq-project organization on GitHub at https://github.com/rhq-project which will host the repositories. Some repositories already exist there:

  • helpers: the modules/helpers from the fedorahosted git (sans rtfilter)
  • translations: helper project for external translators - may no longer be needed (after merging the .ru changes back to master)
  • samples: examples for various parts of RHQ (cli, rest api, plugins)
  • presentation_material: screenshots etc. that can be used for translations

There are some items to define and check before the move can happen:

  • Define future repository structure
    • rhq-core
    • rhq-plugins
    • helpers
  • Define branches to migrate
  • Define a future commit policy
  • Do a dry-run of the move
  • Add jobs to jenkins to run against the dry run repo on github and validate the results
  • Ensure all core developer have a github account

Define future repository structure

There is a general agreement that plugins should be split out of the main repository to allow for 3rd parties that want to write/update plugins. 

Which other modules should end up in their own repository?

Conclusion from the rhq-devel mailing list is:

  • rhq-core: everything excluding (helpers and plugins), including the platform plugin (which makes core functional in a standalone way) and rtfilter (more an option than a helper)
  • rhq-plugins: all plugins except the platform plugin, which stays in core
  • helpers: the helpers module except for the rtfilter, which stays in core

Dependencies

Core/master will have snapshot version S and it will not be published to the central maven repo.

Core/releaseX will have release version X and releases will be published to the central maven repo.

Plugins/master will have the same snapshot version S as core/master, and will depend on it.  This means that building plugins/master will first require a build of core/master to make available the snapshot dependencies.  But this means that the latest RHQ-team managed plugins can be developed more rapidly, staying in lock-step with core.

Plugins/release will depend on a released version of core.  This means that it's very easy to check out plugins/release and build against published artifacts.  Or, to develop a new plugin against a released version. Contributors will likely work against a plugins/release version.  Only plugin developers wanting to work against core/master features will be required to build core.

Helpers/master will have the same snapshot version S as core/master but will depend on the most recent release version.  Therefore helpers can always be easily built against the central maven repo.  For example, the 4.7-SNAPSHOT module of helpers' will depend on the published 4.6 modules of 'core' such as core/domain or core/plugin-api.

CI (Jenkins)

CI will require more jobs than today.  Jobs that build core, plugins and helpers as needed.  Although, this will not be three times the number of jobs today.

Anticipated Jobs:

  1. rhq-master-core-compile
  2. rhq-master-core
  3. rhq-master-core-upgrade
  4. rhq-master-core-ora
  5. rhq-master-core-ora-upgrade
  6. rhq-master-core-api
  7. rhq-master-core-code-coverage
  8. rhq-master-core-gwt-locales
  9. rhq-master-plugins
  10. rhq-master-plugins-ap
  11. rhq-master-plugins-code-coverage
  12. rhq-master-helpers
  • Same set for current release jobs? (+12)
  • code-coverage jobs? core and plugins?
  • publish jobs?
  • brew jobs?
  • other jobs?

Define branches to migrate

The following creates a text file with all the branches:

$ git branch -r | grep -v "upstream/" | grep -v "origin/release-3.0.1" | sed -e 's/^ *origin\///' >  branch-list.txt

This file can then manually be trimmed down to the branches we want to migrate.

Branches that should not be migrated:

  • Fully merged ones, those can be obtained with $ git branch -a --merged master
  • Personal branches -- developers can port them over to their private GH repo
  • Obsolete branches, those deemed no longer relevant

General strategy

There are two ways for "only getting the branches we want" over to github:

  1. get rid of them on FH, then move
  2. keep them on FH, but don't push them to GH 

We will likely go with option (2) as it allows to keep their history on FH in a r/o sense.  No mater what, a cleanup of FH branches will be performed, as many seem obsolete.

Building from GitHub

For core/master the build stays as it is today - only the SCM url changes

For plugins/master the build stays as it is today - only the SCM url changes

When core is built for a release then plugins will also be built for a release.  Once released  we can then upgrade the pom dependency to the released version and rebuild with that version.

Example

Latest stable version is 4.6, development train is 4.7-SNAPSHOT

  • rhq-core/master is at 4.7-SNAPSHOT as well as its own dependencies on rhq-core modules. It can not have rhq-plugins or rhq-helpers dependencies
  • rhq-plugins/master is at 4.7-SNAPSHOT as well as its dependencies on rhq-core or rhq-plugins modules. It can not have rhq-helpers dependencies
  • rhq-helpers/master is at 4.7-SNAPSHOT and uses 4.6 for rhq-core or rhq-plugins dependencies.
  • uploads at source forge would be rhq-server, rhq-plugins, rhq-helpers

At release time

  1. we build rhq-core to obtain 4.7 of the repo including 4.7 version of modules/core and publish the later to Nexus
  2. we build rhq-plugins to obtain 4.7 of the repo including 4.7 version of modules/plugins and publish the later to Nexus
  3. we switch rhq-helpers to depend on 4.7 stable, rebuild and then publish
  4. core, plugins and helpers are updated to 4.8-SNAPSHOT version

If modules/core change drastically

  • we build rhq-core to create 4.6.1 and publish the modules/core bits to nexus
  • rhq-plugins and rhq-helpers will then depend on 4.6.1 stable version
  • development continues at 4.7-SNAPSHOT versions

Contributions

Contributors can elect to work in master or in a release branch, which should be the most recent release branch unless this is for some reason not possible.  Plugin (or helper) contributors may very likely prefer a release branch as it will prevent then from having to build rhq-core.  Also, they can test against a stable version of core.  And, plugin work will be forward-compatible if it does not utilize deprecated features or API.

Contributions developed against a release branch may or may not be accepted for that release branch.  It may be decided that the work be applied to master only, or, in some cases not at all.  Contributions will be reviewed and recommendations made. Although, senior contributors may be granted full commit rights.

Contributions applied to a release branch will be git cherry-picked to master unless the change is not compatible or relevant to the master branch.

Published Artifacts

The rhq-core artifacts that will be published to the central repo will be only those that are required to support the rhq-plugins and rhq-helpers builds.  This is likely to be limited to the artifacts under rhq/modules/core/.. .  The artifacts will be published for each release version of rhq-core.  They will be published to the jboss.org repo, which then replicates/pushes to central.

No JON product artifacts will be published, only RHQ artifacts.

Branching, Cloning, Committing

Cloning in git is cheap and easy in github.  Github also has support for so called "pull-requests".  There will be no more personal branches in GitHub.  Developers will use private repositories for this. The same may apply for feature branches.

When a personal/feature branch is ready to go into master, the developer opens a pull-request, a gate-keeper will review and then merge to master (TBD process details). The same goes for pull-requests into release branches. Some projects with smaller code bases even have automatic build runs on original-source + pull-request to ensure that the "merge" will be OK.

For the moment we will allow everyone from the core team to directly commit to the new master, but are encouraging the use of pull-requests for the purpose of code review.

You can read about pull requests and how they used in github help: https://help.github.com/articles/using-pull-requests, https://help.github.com/articles/creating-a-pull-request, https://help.github.com/articles/closing-a-pull-request and to some extent http://scottchacon.com/2011/08/31/github-flow.html.

Two notes on pull requests that helped me realize that they are general-purpose code review tool, not just a way of "pulling" code from somewhere:

  • a pull request doesn't need to be merged/rebased - we can create pull requests any time someone wants to discuss/review a piece of code and then close the pull request once we conclude the disussion without actually making any kind of code change in the target branch.
  • pull requests can be created between branches in the same repository or between any branch in a forked repository and any branch in its "source" repository (regardless of the owners of those repos).

<<More Examples/Demos?>>

Pull-requests

Pull-requests must be resolved via fetch and rebase and not via the GitHub merge-pull-request button to keep a linear history.

The "merge button" on a pull request does exactly that - it merges the pull request to the "base" branch. We don't want that though because it makes a mess of the "base" branch history - hence the above note about needing to fetch and rebase the pull request to accept it in the main codebase.

The easiest way to do it is by understanding pull-requests as merely branches in the remote repo. By default, one cannot see them when doing a git fetch or git pull but they can be made available by reconfiguring your local repo:

In your $REPO/.git/config, locate the [remote "<GITHUB_REMOTE>"] section and add the following line to it:

For this to work, the remote repository must not contain any branch called "pull/X" (where X is a number), otherwise you will get errors when trying to fetch/pull from the remote repo. There is no reason though for existence of branches called that name in the remote repository so it should not be an issue.

So for example, for my own fork of the main RHQ repo to be setup as a remote caled "origin" (the default "assumed" name of the remote repo, because I want to do most of the development in my own fork and only merge to mainline repo when everything is ready and code-reviewed), and the main RHQ repo to be setup as a remote called "mainline", I'd do:

Now I'd edit my .git/config file in the local checkout and add fetch = +refs/pull/*/head:refs/remotes/origin/pull/* to the [remote "origin"] section and fetch = +refs/pull/*/head:refs/remotes/mainline/pull/* to the [remote "mainline"] section.

Now that we have the repository set up to see the pull requests as remote branches, we can do the following to accept the pull request "8" that has been made against our mainline repo:

The only thing left then is to close the pull request itself. This can be done through the web UI, or you can go to http://metlos.github.com/hub/, follow the simple instructions and once you have that script you can close the pull request with:

The ability to close a pull request from the commandline is something I added to generally cool tool called hub (https://github.com/defunkt/hub, or my fork https://github.com/metlos/hub) but it is actually not needed for that. The comment and closing of a pull request is done by calling github's REST API and you could write a script of your own for that easily (or even, god forbid, use the web UI for that ) I decided to update that tool because it nicely solves the OAuth2 authentication with github and other "infrastructural" things I couldn't be bothered dealing with.

Note that the above steps with creating a temporary local branch for merging the pull request are only necessary if you are merging a contribution of an outside contributor. When you working on something in your own fork and are ready to have it code reviewed and merged to mainline, the workflow (in a setup like above) is slightly simpler:

  1. create a pull request from your feature branch to mainline's master (or whatever the target branch should be)
  2. have it code-reviewed and assuming the same setup of branches as in the previous example, do:

Migration Dry-Run

Heiko has done that and reported back to rhq-devel. In this dry-run, the plugins were split off into a separate repository.

Results are at

rhq and rhq-plugins on Heiko's github account

Important result: make sure to remove the origin that was cloned from fedorahosted as early as possible so that no pushes to that can happen.

<<Will a more complete dry-run be necessary?>>

Ensure all core developer have a github account

Every core RHQ developer needs to get a github account and list it here:

  • Heiko Rupp: pilhuhn
  • Jirka Kremser: Jiri-Kremser
  • John Sanda: jsanda
  • Jay Shaughnessy: jshaughn
  • John Mazzitelli: jmazzitelli
  • Lukas Krejci: metlos
  • Mike Thomson: mtho11
  • Simeon Pinder: spinder
  • Stefan Negrea: stefannegrea
  • Thomas Segismont: tsegismont
  • Charles (does he need/want one?)
  • Larry (probably needs one?)
  •  

Migration Process ("The Plan")

  1. sync all work to FH 
  2. freeze the git.FH repo and perhaps even revoke write rights for (almost) everyone  (FH admins can switch the entire repo to r/o temporarily to prevent accidental pushs)
  3. sync the whole tree over to GH
    1.  - we may filter out old branches that we don't want to keep
  4. split out the plugins into their own repo (except platform)
  5. split out helpers into their own repo (except rtfilter*)
    1. remove those parts from the main repo
    2. fix the <scm> entries in the pom files
      1. only have <scm> entries in the roots of the repositories
      2. remove the <scm> entries from other poms
  6. have every developer move away the old FH based single repo
  7. have every developer clone the repos from GH – it is still possible to fetch/cherry-pick work from the old local repo and use it in the GH based one, so that developers can still work during the transitioning
    1.  - no usage of submodules
  8. change plugins + other repos (if applicable) to build against the latest published maven artifacts (this is needed to allow 3rd parties to build them without the need for pulling and building the main repo
  9. Change jenkins jobs to build the repos -- making sure that e.g. plugins also only use published artifacts
  10. have every developer only push to GH into the respective repo. As e.g. the plugin repo has no classes from core, there will be no cross-talking

Migration script from the dry-run

Start with a fresh clone of the fedorahosted repository. Don't use the normal repo

## Do not yet remove origin, as it is needed in the next steps

# determine list of branches

# determine list of merged branches - this is probably pretty incomplete wrt feature-branches

# remove merged branches from full list of branches
$ comm -23 ../branch-list-full.txt ../merged-branches > ../branch-list.txt

Branchlist on April 2th, 2013

## make sure a branch origin/release-3.0.1 is filtered as it confuses git (not to confuse with the branch 'release-3.0.1')

# run this script to "nail" the branches":

Now after we have all branches from the old repo, remove
it from the list of remotes and add the new target repo, then push
branches and tags

Plugins

Now that this is done we can pull out the plugins

1) create the plugin repo on github
$ git remote add origin git@github.com:rhq-project/rhq-plugins.git
2) filter down the plugins

Run this script to "nail" the branches":

Now filter down to the plugins

This took another 10-15 mins locally (with a hard disk)

Now continue with step 5 of the git stackoverflow entry (http://stackoverflow.com/a/359759/100957 ) , which is

And then push to GitHub:

Remove modules/plugins/platform from rhq-plugins repo

Now get rid of the plugins in the first ("main") repository – all except modules/plugins/platforms

NOTE:  best is to check out the main tree from github and then do this.
Or make sure to remove origin from the remotes, so that this can not mess up
fedorahosted (as I accidentally did)

Helpers

## Do the dance as above for the helpers

Afterwards

4) in rhq-plugins repository and helpers repo

  • copy over the .mailmap file from the main repo and commit
  • fix the dev-container relative path within the pom.xml
  • copy the.gitignore file from rhq-core repo into the other repos
  • fix plugin build to not try to validate the platform plugin, but add this as dependency

release builds and plugins

ear build

Ear build needs to exclude the plugins except for the platform plugin.

release script

The release script needs to add the plugins from the rhq-plugins repo into the generated rhq-server.zip

Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.