Scale up 10x with 10% extra…

In my recent job , one of the daunting tasks I had was to increase the delivery velocity, quality without the linear increase in resources and investment. Adding investment (and people) cannot just lead to massive scale up. Focusing on many levers ( Frameworks, automation, re-use, productisation, domain composition , Agile at scale) I managed to scale up delivery by 10x by just <10% increase in resource. I was more focused on;

• Sustainable and long term velocity and not just immediate velocity ( hacking!)

• Sustainable cost to achieve

Below are the critical levers I used to scale up 10x with hiring 10x resources. Pls note NONE of these are JUST THEORY but what was realised by adopting below in a VERY PRAGMATIC sense.

Lever 1: Architecture and design

• Invest in technology frameworks , reusable frameworks and NOT just getting functionality running!

  • In order to achieve a sustainable velocity, its critical that up front investment is made in ensuring the technical frameworks - It can be UI components, widgets, API frameworks, authentication tools, business logic frameworks, API or streaming frameworks… are available so that there are set of re-usable technical libraries

    • That perform repetitive tasks

    • Standardise the implementation ( ensure efficient testing, bug fixing, production support )

    • Give a cookie-cutter approach for mass (wider adoption by) relatively junior resources

• Expose simpler interfaces to develop functionality and keep them KISS

  • There is a great book I used to read long time back on developing simpler software with abstracted and well defined interfaces. Suggest reading this book. This talks about

    • How the cognitive load of the software user ( the developer who users the frameworks) need to be kept simpler

    • Provide simpler interfaces/abstractions with deep implementation

    • Avoid information leakage

    • Pulling “specialisation” up and pushing “generalisation” down in the software frameworks/ tools provided for development

  • I used similar approaches in decomposing the architecture layers horizontally to ensure more simpler APIs are exposed to wider development team. For e.g. We spent significant time building core API frameworks ( REST API exposure, validation, domain mapping , service layering, entitlements and security management, monitoring and instrumentation), Streaming frameworks ( event consumption, subscription , validation, pipelining ) that were used by wider developers to deliver many “functional” services in parallel.

  • I would categorise this decomposition as horizontal/layered decomposition where each technology framework/tooling provides a deep and re-usable implementation that exposes a simpler interface for its user.

• Domain/Vertical decomposition that makes sense

  • In one of my previous blogs I touched on the importance of domain driven design where a business / functional process can be broken down to key domains where;

    • Each interact with each other using clear, deterministic interfaces

    • Abstracts the functional complexity with in ( i.e. reducing the cognitive complexity of the consumer domain)

    • Have specialised Pods owning their domain’s engineering and quality

  • I have used DDD successfully to;

    • Decompose functional architecture

    • Segregate code repositories, build and deployment processes (yet aligned to standard tooling , DevOps, InfraOps, DBOps ) to facilitate

      • Parallel development

      • Isolated build, test and deployment

      • Reduce the “blast radius” of defect impact

      • Improve time to market in parallel releases

  • Also wanted to highlight that there is no single rule on how a DDD can be applied to any domain. Careful analysis need to be done on the;

    • Functional dependencies

    • Data dependencies and encapsulation/ isolation of not only the functional cohesiveness but also transactional isolation as well

    • Process dependencies

    • Existing technology and set.

    to ensure the most optimal set up achieved.

  • I had to perform the domain decomposition twice in my latest stint ( as I am a strong proponent of Evolutionary Architecture ) to ensure the product , functional and delivery architecture is efficient.

• Prove interfaces - UI or API first

  In new product development some times you are not sure about the actual functional decomposition even though you have a solid understanding of the fundamental tech stack. I was in this situation many times in my career. 20 or 15 yrs back we would have spent 3 to 4 months in a waterfall project where the detailed functional and technical architecture would have got formed ( yet after a year’s time you realise that the functionality / process need to change and hence the architecture has to be changed!)

• For the last 5 years I have been practicing evolutionary architecture where I have started with 60-70% of view of what is needed ( as most of the time my clients or business partners are not 100% sure what they want as well and in every product/ project is a reality)

• Hence I have changed the approach to start developing working prototypes..

  • Have been using many no-code/ low-code solutions to build semi-functioning user interfaces that can actually store/retrieve data

  • Mock’ed APIs to model the data needs and business logic

  • Domain driven and service driven architecture that facilitate quick, isolated design, architecture and development changes

• This keeps repeating for few sprints until the product team, UI , API, back end engineers, automation engineers are all on the same page (while working in three amigos in Agile development) and there is a working product to start with.

• While in this model the first 3-4 sprints were bit chaotic and resulted in continuous change the evolving nature and the “proof by interfaces” gave much more confidence for every one in the team in the project.

Lever 2: Delivery conveyor belt

While adopting above pragmatic architecture and design considerations helped me create extensible and evolving products, its essential to have a “conveyor belt” like minimal touch delivery mechanism that covers;

• Development and Dev testing/proving

• Quality assurance

• Build, deployment and delivery

• Invest in continuous delivery and delivery rails

In order to achieve good velocity , repeatability and quick issue resolution its essential to INVEST UPFRONT in getting the continuous deployment, test and delivery pipeline set up. Its NOT an after thought. Like forming the technology frameworks at the beginning we have spent up front time in

• Determining the deployment architecture

• Needs of environments ensuring there is ZERO disparity in dev, test and user environments

• Develop the build, packaging, deployment pipelines for every development team/pod is available by;

  • Using CI tooling like github actions

  • Providing self-manageable devOps pipelines for development teams

  • Providing self-managing Dev environments for development pods to tear down

It was key for me to get the continuous delivery set up during the architecture and product forming stage itself so that when the main development start the delivery rails were available for scaling out the delivery massively.

• Invest in automation up front (not mid way)

We in software engineering have been investing in automation for many years, from functional/ non-functional testing to build and deployment. But what I started doing different in the last 5 yrs is getting the automation set up formed from the beginning. Running projects in Agile or Agile at scale ( depending on the size of product/ project) helps getting the automation engineer aligned up front with the product and development teams , help shape the user stories , write the correct BDD test scenarios, create appropriate test data and start developing automated test cases.

In addition to investing up front on getting the continuous delivery pipelines and processes set up, I recommend get below addressed during the project/product forming stage

• Automated functional testing

• Automated resilience and performance testing

• analyse and select the correct automation testing tools that helps a more behavioural driven testing approach

• Build and deploy the test injectors and environments before functional stories are ready for development

• Decompose / componetise the build and deployment architecture as well

Having a modular architecture is a great start but the whole build, deployment and verification (testing and monitoring) processes needs to be decomposed to fit this modular architecture set up where there are separate;

• Build trees, repositories and artefacts management processes

• Deployment pipelines

• Deployment units ( can be containers, environments that can be “self managed” by the teams owning a components )

• Testing and verification ( functional, regression, non-functional, monitoring) processes

that will facilitate;

• Isolated verification and testing

• Black box testing for inter-component / integration testing

• Independent release of components

• Spend time on non-functional verification as well (from the beginning)

Related to above, time to time I have seen projects allocate performance and non-functional testing to the end. This definitely impact deliverables due to the need to refactor designs to meet the non-functional needs. There are many non-functional aspects like below that I have focused from the out set where

• Developer test scenarios and code quality checks cover below dimensions

  • Performance engineering

    • Automated end of day or end of sprint performance testing

  • Stress, soak testing

    • Done regularly during the development and test phases to stress various components ( DDD really helped here where we managed to segregate components , services , deployment and infrastructure )

  • Reliability and resilience engineering

    • In a cloud scale and cloud friendly set up its essential that the reliability of;

      • Services

      • Their inter-connectivity

      • Guaranteed processing, zero data loss, de-duplication

      • Dynamic scaling

      • Service throttling

      • Data loss, recovery points and return time objectives

        are measured and resolved up front and not at the end

        Working in an evolutionary architecture will help address these concerns from the beginning and enhance through out the delivery.

      • I have used reliability testing tools like Chaos Monkey to test reliability of large scale deployments , find failure points , recovery points and ability of software components to withstand unplanned failure scenarios

  • Security

    • Yet another dimension we tend to push to the back is incorporating the authentication, authorisation and entitlements. Its a nightmare to change the application designs to retrospectively incorporate security! These MUST be baked in to the original design and framework implementation.

  • Device compatibility testing

    • If its a web based platform working across multiple browsers ( chrome, Safari, Firefox, MS Edge )

    • If it needs to support multiple form factors ( desktops, laptops, tablets, smart phones)

While historically these non-functional qualities were measured at the end , the revised approach I have adopted has been to;

• Incorporate these measurements and testing early in the development

• Build the required infrastructure and tooling from the beginning

• Incorporate as part of automated testing cycles

• Have a continuous measurement and loop back mechanism to identify deficiencies

Listen to the production / Monitor Driven Development

Test Driven development helped us getting early feedback on design and development, Monitor Driven Development (MDD) gives us feedback on the application’s business logic and the usage of it. In TDD we write test cases before we develop functionality from a technical perspective. in MDD we first define a monitor to verify new features within a business context. But its essential to understand that MDD has a wider scope than TDD. While TDD focus within the development and testing steps, MDD spans from requirements gathering to , design, development, deployment and using/operating the system. Its more than focusing on technical delivery but also verify whether “what we delivered makes sense to users and are they actually using the right features etc , helping taking account the business value in development”

This is a paradigm shift where we not only “shift left” ( i.e. test before development) but also “shift right” ( i.e. getting insights from production usage/ metrics to proactively assess the logic usage and find issues). I have used this approach many times , especially for large scale applications/ deployments where the usage put significant strain in the platform which was not possible to be re-produced. Production monitoring - event logging, performance metrics capture, error logging, performance profiling helped in proactively understanding not just business flows users are using (which we had never tested or there was no functional test cases!)

Lever 3: Agile at scale and delivery pods

As I have written in my previous article, just having a strong architecture and a solid engineering delivery process is not sufficient. The organisation set up and the working culture must also be tuned and geared towards scaling out the delivery.

• If the architecture is decomposed into vertical (DDD) and horizontal components ( Technology layering ) , the development/delivery teams need to be aligned to these domains

  • Each domain will constitute of a delivery Pod which is responsible for;

    • Development

    • Quality assurance

    • interface definition and handshake

• Inverse Conway law ( get Delivery Org fit to the F2B Architecture)

• I tend to argue that this is reversal of Conway Law (Few decades back Melvin Conway stated that the applications and application architecture will align to the organisational architecture) . This results in building architecture and a resultant product suite that exactly matches the current business process which a project is trying to replace and optimise instead of forming a more efficient and effective business process. To avoid such a “hard wiring” its essential that;

  • The front to back architecture is modelled with DDD and service based architecture, then the domains may not necessarily be hard-wired to existing organisational architecture. I have tried this few times and managed to de-couple and allow the software architecture to be relaxed (ensuring its not hard-wired to pre-existing organisational architecture)

  • The delivery organisations is aligned to the product architecture

• This allows the product to be developed, tested, deployed faster and fixes released quicker.

• Allows more flexibility, adaptability to future changes

• Match the vision to delivery and reality

• Focus on Agile at scale…

While its was always beneficial adopt Agile, when I was responsible for a large technology organisation with multiple delivery rails and commitments, adopting Agile at scale was extremely beneficial.

Agile at scale helps to extend the practices used in an Agile Delivery team ( sprint planning, refinement, retrospectives, show and tell etc) and extend to;

• Medium and long term planning ( Program increment planning )

• Team inter-dependency management and scheduling ( Release train planning)

In this article I only briefly touched the key dimensions I have used in scaling up delivery as well as the overall quality of the deliverables. If you wish to get more insights drop me a note here