Loading Profile...
How do I layout subversion with multi-module builds?
What is the best way to layout subversion repository, with multi-module builds, when the modules are shared by many projects? I've read its bad to use externals.
Perhaps the answer is "do not use multi-module builds". That's what we are doing now. But if that is the answer, what is the motivation for a multi-module build in the first place. Why have modules if they are not shared? And if I do share the module, just as a regular dependency somewhere else, why do I have to choose which "parent" project gets to have the library as a multi-module child project?
Perhaps the answer is "do not use multi-module builds". That's what we are doing now. But if that is the answer, what is the motivation for a multi-module build in the first place. Why have modules if they are not shared? And if I do share the module, just as a regular dependency somewhere else, why do I have to choose which "parent" project gets to have the library as a multi-module child project?
I’m confused
1
person has this question
I have this question, too!
Tell me when someone answers.
The more people who ask this question, the more it gets noticed.
The more people who ask this question, the more it gets noticed.
The best answers from the company
-
The best way I've seen multi-module builds used is for separating extension or client APIs from the rest of the implementation code, and also to enforce certain architectural design elements.
First, for separating extension APIs and client stubs, multi-module builds ensure that your API/stub code is clean and can be separated from the implementation code without worry. It also has the clear benefit of producing _much_ smaller binaries for plugin/client developers, since they have only what they need to get the job done (eg. component interfaces or communication protocols). In effect, these modules are sort of a slave to the main build in that they are identified in terms of the main project - groupId and version, and even artifactId prefix usually. When a new Foo-application comes out, a new Foo-client and Foo-ext-api are released in lock step, and never is Foo-client released without Foo-application...at least, in normal circumstances.
On the other hand, there may be times when you have specific layering within the architecture of your application...layering that you want to remain distinct, in order to prevent creating a gridlock situation where so many nearly-circular dependencies have been created between components in the different layers that you can't adjust to the future needs of the business. I've worked on large, spread-out teams before that had large numbers of consultants and greenhorns on board, and this "seizing up" effect caused constant problems. Finally, we wound up splitting our application build according to layer - data, service, view, and so on - and assigning strict dependencies between them. This way, if a developer violated the architecture by introducing a bidirectional dependency between layers, the build would fail. This was incredibly hard to do in Ant (our build tool at the time; this was before Maven), but it was worthwhile.
As for your situation, where you have projects that are libraries used in multiple projects, it seems like you have a much more complex release cycle for these libraries. In other words, your utils project may have a new released associated with the release of any of projects {A, B, C, ...}. These releases have a rippling effect on the other projects than the one which caused the utils release, and it sounds like you're handling it fairly well. The use case you have - keeping all trunks up to date with the latest release - isn't actually addressed explicitly within Maven. Currently, my advice would be tthe approach you've implemented; some sort of custom plugin that can update all projects in {A,B,C,...} based on a new release of utils...sort of a reverse-dependency mapping. Nexus can help here by untangling the locations of binaries for things like utils releases, but unfortunately we don't have a standard solution for the rippling update problem yet.
I hope this answers your question.
The company and 1 other person say
this answers the question
-
Instead of requiring the 5 or more projects that rely on these shared projects to integrate them into a multi-module build, I'd recommend starting to use a repository manager to which you would publish internal artifacts. If you did this, then you don't have to approach all of your projects as including every internal dependency in a multi-module project build.
When I learned of externals a few years ago in Subversion I embraced them as a good solution to this problem, but as you start to use them heavily you start to realize that they introduce some odd problems. They are commit boundaries (commits don't cross over an svn:external "link") and they are this independent property that has to be modified everytime the external target is linked, etc.
I really think you'd find things easier if you had a repository manager (like Nexus).
The company says
this answers the question
-
Inappropriate?The answer to this question depends on a number of variables that are going to be unique to your environment. Is each module in a multi-module build *really* going to have an independent release schedule? Are modules going to be tagged independently?
Probably not, right? Store your multi-module project in a single directory tree and just tag the top-level directory with something like the Maven Release Plugin. In other words, don't worry about using svn:externals and creating a trunk, tags, and branches for each component. You should group modules together, and reserve a separate, independent trunk, tags, branches for cases where you actually them.
I’m silly
-
Inappropriate?While our modules do not have independent release cycles, per se, they are heavily shared/refactored/etc among 5 or more projects that DO have independent iterations and release cycles.
Currently, we have each module in its own trunk/tags/branches. And we release them as changes are needed for any of the given projects. We wrote our own maven plugin to keep all the pom.xml in all the module and product trunks automatically updated to the latest version of each module dependency, and Hudson notifies the user, in case they forgot to update some upstream dependency to any potential API or tests breaking. Clearly, this is not the way Maven was intended, but in our transformation from a Clearcase/Ant world to a Subversion/Maven world, this made the most sense at the time. And we like keeping all our projects up-to-date with the latest module changes -- only on the trunk, of course.
Any recommendations that would make our environment more "standard"? -
Inappropriate?Instead of requiring the 5 or more projects that rely on these shared projects to integrate them into a multi-module build, I'd recommend starting to use a repository manager to which you would publish internal artifacts. If you did this, then you don't have to approach all of your projects as including every internal dependency in a multi-module project build.
When I learned of externals a few years ago in Subversion I embraced them as a good solution to this problem, but as you start to use them heavily you start to realize that they introduce some odd problems. They are commit boundaries (commits don't cross over an svn:external "link") and they are this independent property that has to be modified everytime the external target is linked, etc.
I really think you'd find things easier if you had a repository manager (like Nexus).
The company says
this answers the question
-
Inappropriate?I think there is a misunderstanding in our current environment. Let me walk through how we do things now -- we do use Nexus.
A change to the "util" module is required for "ProjectA"
1) Have projects for util and ProjectA in Eclipse, in subversion branches
2) Modify ProjectA pom.xml to use util's current snapshot version (and m2eclipse associates them for us).
3) Modify util and ProjectA as needed.
4) Merge changes of util from branch into util trunk.
5) Use maven release plugin to deploy util to Nexus, make the svn tag, etc.
6) Update ProjectA pom (done automatically, by our own maven plugin that walks the svn tree, and updates all the pom.xml in trunks).
7) Merge changes of ProjectA from branch to trunk.
8) (Optionally) Release ProjectA with maven release plugin, again, deploying to Nexus.
Now, the issue is not with getting the released versions into all the other projects that depend on util -- you are correct, Nexus and Maven are great for that!
This is really more of a question of: would our lives be easier in Eclipse or in some other development way if util was a module of ProjectA (and ProjectB, ProjectC, etc), instead of being a dependency like it is now. The answer may be no...perhaps we are doing what's right for us. I have just never understood the benefit of multi-module builds if that is the case, other than simply breaking a part a large project for quicker builds of sub-modules. But in that case the modules are only used by one project...
Does that make my question a bit clearer? Or more muddled?
I’m unsure
-
Inappropriate?Ok, I see where the complication is. Your main concern is the fact that you have to go through all of the other projects and run a custom "upgrade" goal that updates all of the poms to point to the most recent version of util. I'm assuming that:
1. Projects A, B, C all have different release schedules
2. util is released almost every time that Projects A, B, and C are updated
It might make sense to start using a corporate POM. This corporate top-level POM would be a parent of project's A, B, C. when you increment a version number for the util project, you would simply update this top-level, corporate POM.
-
Inappropriate?I'm assuming you mean a corporate pom as a parent pom to all our projects.
In fact, we do use a corporate pom (actually, a hierarchy of them, b/c of some java4 and some java6 projects, and well as some war and some jar projects, with various plugins defined). But we only define dependencies like junit and easymock there.
But then are you still stuck updating the version of the parent pom every time? Or can the parent version be left blank in the pom, so it will always use the latest.
The problem with that is:
a) The released tags would need to hard-code the parent pom version, so the build is repeatable.
b) Developers branches would potentially break whenever someone released a new util.
c) Not all of Project A, B, C, etc rely on the same set of dependencies, and I'm not sure it would really come out to a tree rather than a graph of dependencies between them.
What am I missing here? I really appreciate all the help!
I’m thankful
-
Inappropriate?I was thinking that you'd use dependencyManagement in the corporate POM. It sounds like you already know more than 99.9% of the people currently using Maven, so there's a risk I'm telling you something you already know. If the corporate POM had the version number of the util project in dependencyManagement, then you wouldn't have to have the version number in any of the POMs that reference it.
Think of it this way, I have a corporate POM on a project I've been working on that defines all of the common version numbers for all of the dependencies we are using. There are almost no dependency version numbers anywhere else except this top-level POM. When I need to update a dependency, I change it in one place and I'm done with it.
Now, the thing I will say is that you might always be stuck incrementing some release version number in a few of your projects if you release this parent POM. Sorry I can't be of more help, it is tough to fly plane without access to the controls.
1 Comment Add a comment
This answers the question
-
So I guess the short answer is that multi-module builds are not for us. I am still curious as to what the ARE good for, but perhaps that is a different question.
It sounds, too, like our pom-updater plugin is doing the job for us right now, so I think we'll stick with that. Even using depedency management, we would still have to update parent pom version numbers every time (which we could use our pom-updater plugin for as well!) -
Inappropriate?The best way I've seen multi-module builds used is for separating extension or client APIs from the rest of the implementation code, and also to enforce certain architectural design elements.
First, for separating extension APIs and client stubs, multi-module builds ensure that your API/stub code is clean and can be separated from the implementation code without worry. It also has the clear benefit of producing _much_ smaller binaries for plugin/client developers, since they have only what they need to get the job done (eg. component interfaces or communication protocols). In effect, these modules are sort of a slave to the main build in that they are identified in terms of the main project - groupId and version, and even artifactId prefix usually. When a new Foo-application comes out, a new Foo-client and Foo-ext-api are released in lock step, and never is Foo-client released without Foo-application...at least, in normal circumstances.
On the other hand, there may be times when you have specific layering within the architecture of your application...layering that you want to remain distinct, in order to prevent creating a gridlock situation where so many nearly-circular dependencies have been created between components in the different layers that you can't adjust to the future needs of the business. I've worked on large, spread-out teams before that had large numbers of consultants and greenhorns on board, and this "seizing up" effect caused constant problems. Finally, we wound up splitting our application build according to layer - data, service, view, and so on - and assigning strict dependencies between them. This way, if a developer violated the architecture by introducing a bidirectional dependency between layers, the build would fail. This was incredibly hard to do in Ant (our build tool at the time; this was before Maven), but it was worthwhile.
As for your situation, where you have projects that are libraries used in multiple projects, it seems like you have a much more complex release cycle for these libraries. In other words, your utils project may have a new released associated with the release of any of projects {A, B, C, ...}. These releases have a rippling effect on the other projects than the one which caused the utils release, and it sounds like you're handling it fairly well. The use case you have - keeping all trunks up to date with the latest release - isn't actually addressed explicitly within Maven. Currently, my advice would be tthe approach you've implemented; some sort of custom plugin that can update all projects in {A,B,C,...} based on a new release of utils...sort of a reverse-dependency mapping. Nexus can help here by untangling the locations of binaries for things like utils releases, but unfortunately we don't have a standard solution for the rippling update problem yet.
I hope this answers your question.
The company and 1 other person say
this answers the question
-
Inappropriate?That's a great description of the benefits of multi-module builds. We may have occasion to use them on some large projects that are more self-contained. Thanks.
I’m happy
EMPLOYEE
EMPLOYEE
