fetch = +refs/pull/*/head:refs/remotes/<GITHUB_REMOTE>/pull/*
The purpose of this page is to prepare the move of the RHQ sources from git.fedorahosted.org to Github.com
The motivation is to use the existing tooling around GitHub and their infrastructure for easy forking of the project and
There exists already an rhq-project organization on GitHub at https://github.com/rhq-project which will host the repositories. Some repositories already exist there:
helpers: the modules/helpers from the fedorahosted git (sans rtfilter)
translations: helper project for external translators - may no longer be needed (after merging the .ru changes back to master)
samples: examples for various parts of RHQ (cli, rest api, plugins)
presentation_material: screenshots etc. that can be used for translations
There are some items to define and check before the move can happen:
Define future repository structure
rhq-core
rhq-plugins
helpers
Define branches to migrate
Define a future commit policy
Do a dry-run of the move
Add jobs to jenkins to run against the dry run repo on github and validate the results
Ensure all core developer have a github account
There is a general agreement that plugins should be split out of the main repository to allow for 3rd parties that want to write/update plugins.
Which other modules should end up in their own repository?
Conclusion from the rhq-devel mailing list is:
rhq-core: everything excluding (helpers and plugins), including the platform plugin (which makes core functional in a standalone way) and rtfilter (more an option than a helper)
rhq-plugins: all plugins except the platform plugin, which stays in core
helpers: the helpers module except for the rtfilter, which stays in core
Core/master will have snapshot version S and it will not be published to the central maven repo.
Core/releaseX will have release version X and releases will be published to the central maven repo.
Plugins/master will have the same snapshot version S as core/master, and will depend on it. This means that building plugins/master will first require a build of core/master to make available the snapshot dependencies. But this means that the latest RHQ-team managed plugins can be developed more rapidly, staying in lock-step with core.
Plugins/release will depend on a released version of core. This means that it's very easy to check out plugins/release and build against published artifacts. Or, to develop a new plugin against a released version. Contributors will likely work against a plugins/release version. Only plugin developers wanting to work against core/master features will be required to build core.
Helpers/master will have the same snapshot version S as core/master but will depend on the most recent release version. Therefore helpers can always be easily built against the central maven repo. For example, the 4.7-SNAPSHOT module of helpers' will depend on the published 4.6 modules of 'core' such as core/domain or core/plugin-api.
CI will require more jobs than today. Jobs that build core, plugins and helpers as needed. Although, this will not be three times the number of jobs today.
Anticipated Jobs:
rhq-master-core-compile
rhq-master-core
rhq-master-core-upgrade
rhq-master-core-ora
rhq-master-core-ora-upgrade
rhq-master-core-api
rhq-master-core-code-coverage
rhq-master-core-gwt-locales
rhq-master-plugins
rhq-master-plugins-ap
rhq-master-plugins-code-coverage
rhq-master-helpers
Same set for current release jobs? (+12)
code-coverage jobs? core and plugins?
publish jobs?
brew jobs?
other jobs?
The following creates a text file with all the branches:
$ git branch -r | grep -v "upstream/" | grep -v "origin/release-3.0.1" | sed -e 's/^ *origin\///' > branch-list.txt
This file can then manually be trimmed down to the branches we want to migrate.
Branches that should not be migrated:
Fully merged ones, those can be obtained with $ git branch -a --merged master
Personal branches -- developers can port them over to their private GH repo
Obsolete branches, those deemed no longer relevant
There are two ways for "only getting the branches we want" over to github:
get rid of them on FH, then move
keep them on FH, but don't push them to GH
We will likely go with option (2) as it allows to keep their history on FH in a r/o sense. No mater what, a cleanup of FH branches will be performed, as many seem obsolete.
For core/master the build stays as it is today - only the SCM url changes
For plugins/master the build stays as it is today - only the SCM url changes
When core is built for a release then plugins will also be built for a release. Once released we can then upgrade the pom dependency to the released version and rebuild with that version.
Latest stable version is 4.6, development train is 4.7-SNAPSHOT
rhq-core/master is at 4.7-SNAPSHOT as well as its own dependencies on rhq-core modules. It can not have rhq-plugins or rhq-helpers dependencies
rhq-plugins/master is at 4.7-SNAPSHOT as well as its dependencies on rhq-core or rhq-plugins modules. It can not have rhq-helpers dependencies
rhq-helpers/master is at 4.7-SNAPSHOT and uses 4.6 for rhq-core or rhq-plugins dependencies.
uploads at source forge would be rhq-server, rhq-plugins, rhq-helpers
At release time
we build rhq-core to obtain 4.7 of the repo including 4.7 version of modules/core and publish the later to Nexus
we build rhq-plugins to obtain 4.7 of the repo including 4.7 version of modules/plugins and publish the later to Nexus
we switch rhq-helpers to depend on 4.7 stable, rebuild and then publish
core, plugins and helpers are updated to 4.8-SNAPSHOT version
If modules/core change drastically
we build rhq-core to create 4.6.1 and publish the modules/core bits to nexus
rhq-plugins and rhq-helpers will then depend on 4.6.1 stable version
development continues at 4.7-SNAPSHOT versions
Contributors can elect to work in master or in a release branch, which should be the most recent release branch unless this is for some reason not possible. Plugin (or helper) contributors may very likely prefer a release branch as it will prevent then from having to build rhq-core. Also, they can test against a stable version of core. And, plugin work will be forward-compatible if it does not utilize deprecated features or API.
Contributions developed against a release branch may or may not be accepted for that release branch. It may be decided that the work be applied to master only, or, in some cases not at all. Contributions will be reviewed and recommendations made. Although, senior contributors may be granted full commit rights.
Contributions applied to a release branch will be git cherry-picked to master unless the change is not compatible or relevant to the master branch.
The rhq-core artifacts that will be published to the central repo will be only those that are required to support the rhq-plugins and rhq-helpers builds. This is likely to be limited to the artifacts under rhq/modules/core/.. . The artifacts will be published for each release version of rhq-core. They will be published to the jboss.org repo, which then replicates/pushes to central.
No JON product artifacts will be published, only RHQ artifacts.
Cloning in git is cheap and easy in github. Github also has support for so called "pull-requests". There will be no more personal branches in GitHub. Developers will use private repositories for this. The same may apply for feature branches.
When a personal/feature branch is ready to go into master, the developer opens a pull-request, a gate-keeper will review and then merge to master (TBD process details). The same goes for pull-requests into release branches. Some projects with smaller code bases even have automatic build runs on original-source + pull-request to ensure that the "merge" will be OK.
For the moment we will allow everyone from the core team to directly commit to the new master, but are encouraging the use of pull-requests for the purpose of code review.
You can read about pull requests and how they used in github help: https://help.github.com/articles/using-pull-requests, https://help.github.com/articles/creating-a-pull-request, https://help.github.com/articles/closing-a-pull-request and to some extent http://scottchacon.com/2011/08/31/github-flow.html.
Two notes on pull requests that helped me realize that they are general-purpose code review tool, not just a way of "pulling" code from somewhere:
a pull request doesn't need to be merged/rebased - we can create pull requests any time someone wants to discuss/review a piece of code and then close the pull request once we conclude the disussion without actually making any kind of code change in the target branch.
pull requests can be created between branches in the same repository or between any branch in a forked repository and any branch in its "source" repository (regardless of the owners of those repos).
<<More Examples/Demos?>>
Pull-requests must be resolved via fetch and rebase and not via the GitHub merge-pull-request button to keep a linear history.
The "merge button" on a pull request does exactly that - it merges the pull request to the "base" branch. We don't want that though because it makes a mess of the "base" branch history - hence the above note about needing to fetch and rebase the pull request to accept it in the main codebase.
The easiest way to do it is by understanding pull-requests as merely branches in the remote repo. By default, one cannot see them when doing a git fetch or git pull but they can be made available by reconfiguring your local repo:
In your $REPO/.git/config, locate the [remote "<GITHUB_REMOTE>"] section and add the following line to it:
fetch = +refs/pull/*/head:refs/remotes/<GITHUB_REMOTE>/pull/*
For this to work, the remote repository must not contain any branch called "pull/X" (where X is a number), otherwise you will get errors when trying to fetch/pull from the remote repo. There is no reason though for existence of branches called that name in the remote repository so it should not be an issue.
So for example, for my own fork of the main RHQ repo to be setup as a remote caled "origin" (the default "assumed" name of the remote repo, because I want to do most of the development in my own fork and only merge to mainline repo when everything is ready and code-reviewed), and the main RHQ repo to be setup as a remote called "mainline", I'd do:
git clone git://github.com/metlos/rhq.git #read-only URL - I don't have to fill in my SSH password when I just fetch/pull cd rhq git remote set-url --push origin git@github.com:metlos/rhq.git #write URL - in this case I will have to supply my SSH password git remote add mainline git://github.com/rhq-project/rhq.git git remote set-url --push mainline git@github.com:rhq-project/rhq.git
Now I'd edit my .git/config file in the local checkout and add fetch = +refs/pull/*/head:refs/remotes/origin/pull/* to the [remote "origin"] section and fetch = +refs/pull/*/head:refs/remotes/mainline/pull/* to the [remote "mainline"] section.
Now that we have the repository set up to see the pull requests as remote branches, we can do the following to accept the pull request "8" that has been made against our mainline repo:
git fetch mainline #just to make sure we have up-to-date picture of the mainline repo git checkout -t -b pulls2merge/8 mainline/pull/8 #this will create a local branch called "pulls2merge/8" tracking the pull request "8" in the mainline repo git rebase mainline/master #or whatever the target branch in the mainline repo should be, i.e. it could be mainlne/release-4.6.1 git checkout mainline-master #this is the name of the local tracking branch of the mainline/master, of course this would be a different branch if we were pulling the request to a different release - omitted the setup of it in this example git merge pulls2merge/8 #this should result in a fast-forward merge and therefore keep the history linear git push mainline HEAD:master #the arguments might not be necessary depending on how you set up the behavior of git push in your repo - e.g. http://stackoverflow.com/questions/948354/git-push-current-branch/948397#948397 git branch -D pulls2merge/8 #delete the local branch, we no longer need it
The only thing left then is to close the pull request itself. This can be done through the web UI, or you can go to http://metlos.github.com/hub/, follow the simple instructions and once you have that script you can close the pull request with:
hub pull-request -c 8 "Merged to mainline."
The ability to close a pull request from the commandline is something I added to generally cool tool called hub (https://github.com/defunkt/hub, or my fork https://github.com/metlos/hub) but it is actually not needed for that. The comment and closing of a pull request is done by calling github's REST API and you could write a script of your own for that easily (or even, god forbid, use the web UI for that ) I decided to update that tool because it nicely solves the OAuth2 authentication with github and other "infrastructural" things I couldn't be bothered dealing with.
Note that the above steps with creating a temporary local branch for merging the pull request are only necessary if you are merging a contribution of an outside contributor. When you working on something in your own fork and are ready to have it code reviewed and merged to mainline, the workflow (in a setup like above) is slightly simpler:
create a pull request from your feature branch to mainline's master (or whatever the target branch should be)
have it code-reviewed and assuming the same setup of branches as in the previous example, do:
git fetch mainline git checkout local-feature-branch git rebase mainline/master git checkout mainline-master git merge local-feature-branch git push mainline HEAD:master hub pull-request -c <PULL_REQUEST_NUMBER> "Pushed to mainline."
Heiko has done that and reported back to rhq-devel. In this dry-run, the plugins were split off into a separate repository.
Results are at
rhq and rhq-plugins on Heiko's github account
Important result: make sure to remove the origin that was cloned from fedorahosted as early as possible so that no pushes to that can happen.
<<Will a more complete dry-run be necessary?>>
Every core RHQ developer needs to get a github account and list it here:
Heiko Rupp: pilhuhn
Jirka Kremser: Jiri-Kremser
John Sanda: jsanda
Jay Shaughnessy: jshaughn
John Mazzitelli: jmazzitelli
Lukas Krejci: metlos
Mike Thomson: mtho11
Simeon Pinder: spinder
Stefan Negrea: stefannegrea
Thomas Segismont: tsegismont
Charles (does he need/want one?)
Larry (probably needs one?)
sync all work to FH
freeze the git.FH repo and perhaps even revoke write rights for (almost) everyone (FH admins can switch the entire repo to r/o temporarily to prevent accidental pushs)
sync the whole tree over to GH
- we may filter out old branches that we don't want to keep
split out the plugins into their own repo (except platform)
split out helpers into their own repo (except rtfilter*)
remove those parts from the main repo
fix the <scm> entries in the pom files
only have <scm> entries in the roots of the repositories
remove the <scm> entries from other poms
have every developer move away the old FH based single repo
have every developer clone the repos from GH – it is still possible to fetch/cherry-pick work from the old local repo and use it in the GH based one, so that developers can still work during the transitioning
- no usage of submodules
change plugins + other repos (if applicable) to build against the latest published maven artifacts (this is needed to allow 3rd parties to build them without the need for pulling and building the main repo
Change jenkins jobs to build the repos -- making sure that e.g. plugins also only use published artifacts
have every developer only push to GH into the respective repo. As e.g. the plugin repo has no classes from core, there will be no cross-talking
Start with a fresh clone of the fedorahosted repository. Don't use the normal repo
$ git clone ssh://git.fedorahosted.org/git/rhq/rhq.git $ cd rhq
## Do not yet remove origin, as it is needed in the next steps
# determine list of branches
$ git branch -r | grep -v "upstream/" | grep -v "origin/release-3.0.1" | sed -e 's/^ *origin\///' | sort > ../branch-list-full.txt
# determine list of merged branches - this is probably pretty incomplete wrt feature-branches
git branch -a --merged master | sed -e 's/^.*origin\///' | grep -v "master" \| sort > ../merged-branches
# remove merged branches from full list of branches
$ comm -23 ../branch-list-full.txt ../merged-branches > ../branch-list.txt
$ vi ../branch-list.txt ### only leave branches in we are interested in moving over
Branchlist on April 2th, 2013
bug/783603 bug/918677 bug/927868 bug/rhq-1 feature/cassandra-backend hotfix/jon2.4.2 hotfix/jon3.0.0 hotfix/jon3.0.1 hotfix/jon3.1.2 master nightly/rhq.jon patch-3.0.0-perf patch/3.0.1 release-3.0.0 release-3.0.0-patch1 release-3.0.0-patch2 release-3.0.0-patch3 release-3.0.0-test-build release-3.0.0.Beta1 release-4.0.0 release-4.0.0-test-build release-4.0.0.B01 release-4.1.0 release-4.2.0 release-4.3.0.GA release-4.4.0 release-4.5.0 release-4.6.0 release/jon2.4.x release/jon3.0.x release/jon3.1.x
# put the branch list into a variable as we will need that a few times now $ BRANCHES=`cat ../branch-list.txt`
## make sure a branch origin/release-3.0.1 is filtered as it confuses git (not to confuse with the branch 'release-3.0.1')
# run this script to "nail" the branches":
for B in $BRANCHES do git branch -t $B origin/$B done
Now after we have all branches from the old repo, remove
it from the list of remotes and add the new target repo, then push
branches and tags
git remote remove origin # remove fedorahosted git remote -v ## must not list fedorahosted anymore git remote add upstream <github url> ## rhq-core # git remote add origin git@github.com:rhq-project/rhq-core.git $ git push --all -v ## push all branches to GitHub $ git push --tags -v ## push all tags to GitHub
Now that this is done we can pull out the plugins
1) create the plugin repo on github
$ git remote add origin git@github.com:rhq-project/rhq-plugins.git
2) filter down the plugins
cd /tmp git clone /tmp/rhq rhq-plugins ## just use what is on disk cd rhq-plugins
Run this script to "nail" the branches":
for B in $BRANCHES do git branch -t $B origin/$B done
Now filter down to the plugins
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter modules/plugins -- --all
This took another 10-15 mins locally (with a hard disk)
Now continue with step 5 of the git stackoverflow entry (http://stackoverflow.com/a/359759/100957 ) , which is
git reset --hard git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d git reflog expire --expire=now --all git gc --aggressive --prune=now
And then push to GitHub:
git remote remove origin git remote -v # must not list fedorahosted and also not github / rhq-core # git remote add upstream <github url> ## git remote add upstream git@github.com:rhq-project/rhq-plugins.git git push --dry-run --all -v upstream # git push --all upstream # git push --tags upstream
Remove modules/plugins/platform from rhq-plugins repo
git clone git@github.com:rhq-project/rhq-plugins.git cd rhq-plugins # nail plugins for B in $BRANCHES ; do git checkout -t -b $B origin/$B ; done # Now remove the platform plugin and modify pom.xml for B in $BRANCHES do git checkout -f $B git rm -r --cached platform cp pom.xml ../pom && grep -v platform ../pom > pom.xml git add pom.xml git commit -m "Platform plugin lives in the rhq-core repository" done
Now get rid of the plugins in the first ("main") repository – all except modules/plugins/platforms
NOTE: best is to check out the main tree from github and then do this.
Or make sure to remove origin from the remotes, so that this can not mess up
fedorahosted (as I accidentally did)
# clone rhq-core git clone git@github.com:rhq-project/rhq-core.git cd rhq-core/modules/plugins # generate plugins list for P in *; do if test '-d $P -a $P != "platform" -a $P != "pom.xml" '; then echo $P; fi; done > ../p-list vi ../p-list ## ensure platform and pom.xml are not in there export PL=`cat ../p-list ` # nail branches if that was a fresh checkout above for B in $BRANCHES ; do git branch -t $B origin/$B; done ## Now do the work $ for B in $BRANCHES; do git checkout -f $B for P in $PL do git rm -r --cached $P cp pom.xml ../pom grep -v "<module>$P" < ../pom > pom.xml done git add pom.xml git commit -m "Plugins now live in rhq-plugins repository" done
## Do the dance as above for the helpers
git clone git@github.com:rhq-project/rhq-core.git rhq-helpers cd rhq-helpers/ for B in $BRANCHES ; do git branch -t $B origin/$B; done git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter modules/helpers/ -- --all git reset --hard git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d git reflog expire --expire=now --all git gc --aggressive --prune=now for B in $BRANCHES; do git checkout -f $B ; git rm -r --cached rtfilter; git rm -r --cached rtfilter-subsystem; cp pom.xml ../pom && grep -v rtfilter ../pom > pom.xml; git add pom.xml; git commit -m "RtFilter is part of the rhq-core repository. Removing it here"; done git remote remove origin git remote -v git remote add origin git@github.com:rhq-project/rhq-helpers.git git push --dry-run --all -v git push --all -v git push --tags -v
4) in rhq-plugins repository and helpers repo
copy over the .mailmap file from the main repo and commit
fix the dev-container relative path within the pom.xml
copy the.gitignore file from rhq-core repo into the other repos
fix plugin build to not try to validate the platform plugin, but add this as dependency
Ear build needs to exclude the plugins except for the platform plugin.
The release script needs to add the plugins from the rhq-plugins repo into the generated rhq-server.zip