Operation Mega Merge

Every once in a while you will find yourself in a situation where the right thing to do is more commonly known as the wrong thing to do! Our team came to this conclusion recently as we were faced with modernizing a large Ruby on Rails monolith.

We coined this work Operation Mega Merge. You can probably guess why!

Background

We have built and maintained dozens of Ruby on Rails applications over the years. Our speciality is to help clients get from idea to production with a very small amount of ego along the way. That is, we build, iterate, learn and build some more. Don't have a perfect idea of what the customer needs just yet? No problem, we can help you get that information and remain agile in our development to add / remove / change as more learning occurs.

The challenge comes when we move beyond the one to two year mark of an application - things start to get old! This particular application was in need of the following upgrades:

• Bootstrap 4.5 (coming on EOL), upgrade to Bootstrap 5.1
• Webpacker deprecated (to move to Vite)
• Back leveled icon libraries (4 of them!) to consolidate into one
• Ruby 3.0 to 3.1
• Rails 6.1 to 7.0
• Back leveled ruby gems
• Back leveled node packages

Our initial roadmap estimated this effort at several months. However, due to an important client need we were required to deliver on it much sooner (with the understanding that more risk would be involved ... and some unorthodox thinking).

What About (Automated) Testing?

Due to the origins of this specific project, there has been limited automated testing implemented. There are select model/unit tests and some small coverage for end-to-end tests. The rest, all manually tested.

This is not to mention that things like CSS are inherently hard to test with automation. We support desktop and mobile responsive versions of the application as well as light and dark mode. That alone will set the stage for how broad the test enumerations are. It also shows that any real automation is difficult (maybe even impossible?). For example, one of the many changes in Bootstrap 5.x is the redefinition of RTL helpers like -right to -end. It is REALLY hard to automatically pick up a float-right that has been missed.

As such, we were at best looking at an onerous manual testing effort. Something that we couldn't afford to do separately for each of the changes.

Enter Mega Merge

So, given the large scope and an equally large investment for manually testing, we did what we swore we would never do; we put all of the changes into one giant pull request. That's right. We took months of our maintenance roadmap and squished it all together into one massive pull request.

The overall flow went something like this:

1. All changes originally implemented by a senior full stack developer.
2. The developer decided when the work was "good enough" for others to start testing.
3. Support lead created a Google Sheet with a matrix of pages, actions and the user profiles to test as (as well as desktop/mobile and light/dark mode).
4. Four individuals set out to test each of the sheet cells, adding comments and assigning them to the developer for resolution.
5. The developer took the list of comments as the backlog and worked through fixing them as fast as possible.

This testing process lasted approximately two weeks (the first week was very focused, the second week less so). By the end, over 150 issues had been found and resolved. Many of which were actually improvements to the application, not fallout from Mega Merge.

Along the way two of the four "testers" were able to contribute some fixes which helped speed up the comment burndown.

Lessons Learned

Never in a million years would I have expected this process to work as well as it did. One developer, 4 testers and a google sheet? Hardly a keynote presentation on turning months to weeks! But it worked very well.

The resulting PR may have slightly stressed Azure DevOps, but it successfully was able to process the "over 1000 changes".

A couple of key takeaways from this project include:

1. The first rule of Mega Merge, find a way to never do another Mega Merge!
2. Our best estimates are still bad - especially for things we've never done before. Days were spent on the tail end of webpacker to vite thanks to differences in rollup.js and esbuild.
3. Team cohesion and a common, aggressive mission will 10x performance. We were all in it together, everyone rolled up their sleeves (nobody was doing their dream job) and we got it done.
4. Some changes are just impossible to break down. You can't have an iterative upgrade to Bootstrap 5.1 or webpack to vite, it just isn't possible. The key is to build things so that it is as easy as possible to upgrade when the time comes (things like these are painful to find! class=bg-"<%= mode.dark? ? "dark" : "light" %>).
5. Context is everything. As technologists we love learning and building and complexity. However, brain surgery isn't needed when an Advil will do! This process was risky, but it delivered and it did so in a way that greatly served the business and the end users.

We aren't looking to do another Mega Merge any time soon, but we're always looking for the right way to build better applications for our clients. If you'd like to join us and help us show what a small but mighty team is capable of then please check out our current opportunities.