Have you ever developed an application that was so successful at solving problems for your users that they asked for a white-labeled version of the application, one they could brand themselves? How about custom features for individual customers? Solving these problems in a sustainable way without blowing your development budget is challenging. There are various architectural patterns that are useful, however each comes with its own tradeoffs. In this article we will explore some of those patterns, their benefits, and their risks.
Copying or Forking the Codebase
This is one of the most common techniques people try for supporting custom features. The initial investment is low, you create a new project with a new copy of the code. Make the changes for a particular client, deploy the code to a new environment, and voila. Done.
Things get more difficult when you want to make a change that is shared between both codebases. You can change each copy of the code, keeping the featured synchronized with the exception of the custom code. Developers now have to write, test and maintain the same code twice. It gets more difficult when a new feature intersects with some custom code in one of the projects. Worse, the development time increases exponentially as the number of copied projects increases. In my experience, once you move past three custom copies of a given project it is very difficult for a small to medium sized development team to maintain the code.
Large teams using this technique to produce quality software are only able to do so at a large cost. It can work but it can be frustrating for developers that have to switch between different clients, and it adds a significant amount of unscalable incidental complexity to the code and all its supporting infrastructure.
Abstraction: One codebase to rule them all
To make things easier to maintain over the long haul it is usually advantageous to have only one codebase. You can abstract out aspects of the front-end and layout so that through configuration you can specify things like font, color, images, and event layout.
I have seen success on projects when product owners specify the extent to which the functionality will need to vary between customers, considering at least six months down the road, and preferably a year or two.
For example, if you just needed to vary some images and a couple fixed layout styles between customers, you would solve that very differently than unique multi-step wizards with differing numbers of steps that collect different information for each client. The former can be solved by abstracting away the customizations, the latter, if implemented for more than five to ten customers, might need a wizard builder and a runtime configurable system.
One way to conditionally enable parts of your codebase is through feature flags. Feature flags are compile time, deployment time, or runtime checks for certain features. Large shops like Facebook make extensive use of feature flags to roll out new functionality. They’ll enable the flags across a small group of nodes to start, slowly increasing the number of nodes as the system detects a low number of errors or issues.
A similar system can be used to enable features for clients. When the client logs in, a feature flag associated with their account enables custom functionality. This is similar to (and can often benefit from) the same metrics captured by A/B testing. For example, if you are A/B testing two different wizards, you would look to the completion rate between clients as a signal indicating which wizard was better. Similarly you could look at engagement metrics between clients with different features to determine the perceived worth of those features and even set pricing.
A best practice with feature flags is to cover as little code as possible. Really small feature flags can be composable. For example, one client can have the advanced layout, bulk import, and auto-logout features while another client has a basic layout and beta feature.
Perils of feature flags
A large number of feature flags in your codebase becomes a type of technical debt. Subsequent features become more complicated to implement and can lead to crufty difficult to manage code. As a rule of thumb when a team is considering a new feature flag they should determine if it will be a long lived feature flag or a short term feature flag. If it is a long term feature flag then they should plan it with a lot of care and ensure it is as minimal as possible. Perhaps there is a way to design the system without a feature flag. For example, instead of enabling key parts of the UI for a specific feature a configuration based approach can be used.
For short term feature flags the team should specify an expiration date when a flag will be revisited, or removed. If lots of feature flags build up in the codebase this is an indication there really is more than one project living in the same codebase.
Factoring, Planning, and Cost
A fundamental challenge behind many of these techniques is the notion of factoring, or how you divide up your system into separate components and how those components interact with each other. A common example is users and addresses. Are the address fields part of the user entity or do they stand alone? If you make them part of the user then you always know a user’s canonical address. It’s easy to compare addresses between users. It’s straightforward to retrieve the user and their address in one call. Some challenges are the user can’t have more than one address and if you want to display a list of addresses the user information is coming along for the ride. Also, if for some reason the users and addresses get out of sync then it can be really difficult to correct. Some think it’s better to choose the simplest path and expand it when necessary, some think it’s better to build in defensive flexibility from the onset, but I maintain that the best answer is more nuanced. It’s a conversation between product owners and developers, where folks determine the likelihood of needing additional flexibility in the future and weight that against the cost. This cost is subjective and difficult to quantify. If you do need to quantify it I suggest t-shirt sizes, comparing its relative cost to something else the team has already completed.
Libraries are an older but well known way to separate out functionality and share it between projects. Juxtaposed against a service oriented solution, network calls turn into function calls and things get a lot faster and easier to debug. If a lot of the code is pushed into libraries developers can construct a client specific offering with minimal code.
As with the single codebase solution listed previously if you are able to factor enough common code into separate libraries then it becomes much easier to create a client specific version that is just based off of configuration. This approach may not make sense for a small number of clients but can really pay dividends and the number of clients increases.
Developers have to watch out if they ever feel that they need to put client specific code in a library. Also, different clients can have different needs. Don’t be afraid to create new functions instead of reusing them when two clients have slightly different needs. However, it’s a fine line between writing client specific code and splitting out their needs. A good example is having separate functions for bulk retrieval and retrieval of a single item. Although each function may only be used by one client these functions could easily apply to additional clients, so they are not client specific code.
Scaling a monolithic system with good libraries has it’s challenges. The key is to run the code across multiple systems. To do that you need to minimize shared data between the nodes. The hardest part is often authentication. There are well known techniques for handling this that can be researched, they are outside of the scope of this article.
One popular technique to help with factoring a system is microservices. Microservices help isolate core pieces of functionality so they can be tuned independently. They are “stovepipe” independent services each backed by their own datastore. In our user and address example, you could have a user service and an address service, where both of those pieces of information are related via a shared ID. This can be a good pattern to use, especially if the address service is provided by a third party and provides detailed, address specific information like geolocations.
Microservices can offer you the flexibility to provide your clients with custom offerings. Each client can have a high level custom service that determines their flow through the application and what lower level services they will have access to. Once a core set of base services have been developed it can be relatively fast and straightforward to design a new offering for a client by assembling pre-existing services.
Microservices are not without their perils. I’ve seen teams that are considering a microservice based architecture neglect to take the cost of deploying and the performance overhead into account. For example, let’s say that you have a system where each request uses an average of five services. The initial request comes in via HTTP and subsequent calls are messages handled by a message broker. How long does it take one service to serialize, put a message on the queue, then another service to receive that message and deserialize the contents? What’s the base overhead of just passing five or more messages around? What does the user see if one the services goes down? How is your team alerted to the outage? Microservices, like many other patterns, have a cost. Your team should be up front with each other regarding that cost and make sure it supports the need.
There are different techniques for white-labeling and delivering custom features. Each have their tradeoffs. Forking a single codebase can get difficult to maintain, however is a better option than maintaining a codebase that is overflowing with feature flags. Feature flags provide a lot of flexibility when use sparingly. Often these problems become one of factoring, focusing on what features and functionality can be split out into separate services, libraries, or functions. When systems are well factored, it is often straightforward to compose the right set of functionality for each client with just a little bit of configuration.
Daniel Glauser, one of the founders of Cambium Consulting, is a long time software developer and architect who has worked on large systems for companies like VMWare, NBC-Universal, Comcast, and Bell South. Recently Daniel has been active in the Denver startup scene helping local blockchain community and mentoring junior developers.