This is part of a series of posts about Moving Past The Monolith. To start at the beginning, click here.

In my last post, I talked about how you can separate data within your application. Sooner or later, someone outside your team is going to need to access your data. How are you going to get it to them while maintaining the flexibility that you need to be able to change when you need to?

It’s unlikely that you’re going to be able to go on forever without anyone asking for access to your data. People usually need it for a good reason — your data has a lot of value, so you should be willing to share it. But you want to do it in a controlled manner. Here is my typical thought progression:

• Can I create a schema that contains only views and stored procs that the other application will use?
• If I have to give teams access to tables, can I limit what they have access to? How will I know what they have access to?
• If someone needs to update data, does it really have to be through database calls or can you have them call a web service instead? If a straight data load makes sense, how do you make sure that this doesn’t adversely affect anything else that my application might do?
• If someone asks to have read-only access to the database to write queries, are they doing it just for research purposes or are they going to write application code against those tables?
• If BI or reporting teams want access to the database, can they write their queries in stored procs or views in a specific schema so that you know what they’re touching? (Especially watch out for SSIS packages that have custom SQL in them, it’s extremely difficult to figure out what is in those packages if you didn’t write them, and you won’t know if you’re going to break if you change something.)
• How do I keep track of who has access to my database so that I can notify them when I’m going to make a potentially breaking change? (If you don’t know who has access, this will severely limit you’re ability to change because you will have no idea what you’re going to break.)
• If another team wants to write application code against my data model, can they code against a copy of the data in their database which gets loaded from a batch job instead so that we both can maintain the freedom to change? Does it make sense to get the data from a web service instead?

Other people wanting access to your data is inevitable. How you manage it is up to you, but it’s important that you be proactive about managing everyone who is dependent on your data.

Read the next post in this series, Using the Service Object pattern.

This is part of a series of posts about Moving Past The Monolith. To start at the beginning, click here.

Separating your business logic code into modules is one thing, but in many cases, that is the easy part. The tough part is the data.

In most large applications, you can probably think of a few database tables that are at the core of the application, have many other tables linking to it, and are pretty much used everywhere.

While you probably won’t be able to achieve total separation of concerns in your data, there are things you do to separate things as much as you can. You will probably always have some shared tables, but knowing that you’re only sharing some of them is better than having to assuming that you’re sharing all of them.

Schemas

One simple approach is to create separate schemas in your database (assuming you’re in a database like SQL Server or Oracle that supports multiple schemas within a database). Schemas are a good way to keep things separate – you can assign permissions at the schema level, and they are a good indication of which code module owns a database table. Since the schemas are all in the same database, you can write queries across schemas if you want to (as long as you acknowledge that you’re introducing more coupling by doing so).

What about the shared things?

What do you do with the core tables that need to be shared across all modules? You still have options here.

• You could acknowledge that every module needs to link to a given table, and just live with it
• You could have one shared table that contains the primary key and shared data points, and then each schema has its own table which has columns that only it cares about
• Each schema could have its own version of the shared data, with some batch process keeping the data from a master table in sync with copies of the data in other schemas

Here are some questions that you should ask:

• Can you keep this table in a schema for a given module, or do you need the same data in multiple modules?
• Could you easily move a schema and its tables into a completely separate database if you wanted to? What would happen if you did, and how hard would it be?
• If you have multiple versions of the same data in different schemas, how will you keep them in sync? Does the benefit of having the separation outweigh the work it will take to keep the data in sync?
• Does all of your data need to be completely up to date in real time, or can certain data be updated nightly or at some interval with a batch job?

There are a lot of trade-offs to consider here, and I imagine that most people will have some shared tables that are used throughout the application. That is fine, but what I think is important is that you consider the trade-offs that come with a shared table, and if you choose to accept that because it’s worth it, then go with it. But the default should be to try and keep table separated, and move them together intentionally.

Read the next post in this series, Think about how you share data.

This is part of a series of posts about Moving Past The Monolith. To start at the beginning, click here.

One of the simplest things you can do to build modular software is simply to organize your code into modules. Every application can usually be grouped into sets of related functionality. The next step is to organize your code so that you put everything that you need for this group of functionality together, separate from the rest of the application. How far you go with this is up to you, and you have many options:

• Group code in a folder within a larger project containing many modules – may or may not have separate data layer code
• Separate assemblies (for you .NET folks)
• Separate deployments

Here are the questions that you need to ask:

• If I wanted to rewrite a module, how hard would it be? Would it be a relatively clean break or would I be trying to untangle a big ball of string?
• Do some modules change a lot more than others, or does it all change often?
• Do I get any benefit out of being able to deploy one module without another?

Keep in mind that there are trade-offs to everything. A simple trade-off is that if you’re on a .NET project and you split modules up into lots of projects, you’re compile times are going to go up because you have a lot more files to move around. For this reason, I typically don’t separate modules into separate projects unless I think there’s a chance that I might want either want to deploy them separately or unless I want to be able to have a separate solution file so that I can compile and develop that module without the others.

At the same time, if you are able to break up your .NET solution into more projects and then you are able to break up the solution into smaller solutions, now you can decrease compile time for developers (potentially drastically if you have a really large application).

Either way, you’re also going to end up with a more organized codebase that’s easier to navigate, understand, and change.

Read the next post in this series, Separating your data.

As long as people have been writing software, well-intentioned developers have cobbled together elegant solutions that have turned into digital versions of Frankenstein’s monster. While these systems don’t threaten your life, they limit your ability to adapt to change, keep up with competition, and maintain your overall sanity.

Numerous posts like this have been written about the Monolith, yet we’ve all created them. I’ve created several of them, and sadly in the not so distant past. I’ve moved on from my last one, but my former co-workers like to inform me every time they’ve managed to rewrite a portion of it.

Why does this keep happening? No one likes maintaining the big, old legacy system that is difficult to change, that is written using old frameworks, or just follows practices that the mythical “everyone” used 5 years ago. When I started out creating the monoliths I’ve created, I certainly didn’t mean to create a problem.

I think it’s always good practice to look back on past projects and think about what I could’ve done differently. I know hindsight is 20/20, but I feel like there are always places where I could’ve made a conscious decision to create a new system rather than just add onto an existing one. I feel like in almost every case, I missed the chance because I wasn’t planning for change.

Let’s be honest, the rate of change is growing exponentially. We aren’t living in a world where you create a mainframe that’s going to run for 30+ years. I have code in my system that we consider “legacy code” that is less than a year old. Whatever hot new JavaScript framework you’re choosing today will be obsolete in a couple years.

As weird as it sounds, we need to starting creating systems so that we can easily replace them.

• Over the lifetime of your system, you may want to change JavaScript frameworks/languages/platforms… multiple times
• You will put yourself behind in the recruiting game if you are using old technology
• You need to be able to change frameworks without having to rewrite the entire application
• You need to consider how the code you’re writing today will need to be rewritten

This means that we need to start building more modular software so that we can easily replace things or just start building things in newer technology.

Some of you might immediately start thinking microservices, but it doesn’t have to be. Microservices are one solution (with their own set of problems), but there are things you beyond microservices to make your code more modular and maintainable.

I would argue that one of biggest issues isn’t a lack of automated tests, it isn’t bad process, it’s that we continue to create large, complex systems that become too hard to change. Over the next several posts, I’ll cover many topics related to writing modular software:

• Organizing your code into modules
• Separating your data
• Think about how you share data
• Using the Service Object pattern
• Minimizing sharing between modules
• Using package managers to share common code
• Splitting up your client-side applications
• Planning ahead

A little over 3 months ago, I started at a new job. This came after years of good consulting work at a great place where I felt quite comfortable. While I was in a great situation working with a lot of good friends, I gave it up for a new challenge.

Anytime you start something new, it’s a roll of the dice. I was going to work in an industry that I knew a lot about, at a company where I knew a lot of people. But even with all that, there were a lot of things up in the air.

I was at my previous client for over 5 years, which these days seems like an eternity. In some ways, it felt that way for me because that’s the longest I had ever stayed at one place. In other ways, the time flew by. They say that time flies when you’re having fun, and that’s certainly the way that it was. In that time, our team went from just figuring out how to function to being extremely high performing, even to the point that we didn’t notice it anymore. We never had large disagreements about process, developers would just run with major efforts and understand how to figure it out the right way, and we released every 2 weeks with very little fanfare and virtually no issues. The efficiency of our team was incredible.

At the same time, I knew I had a choice to make. I could keep going in my comfortable place, providing a lot of value, working with good friends, working for a good company… but I wasn’t being challenged. It wasn’t the fault of anyone, it’s just that when you do the same kinds of things over and over for years, you get pretty good at it, and you don’t really have to learn many new things or find your way in unknown territory.

So I set out for the unknown territory. I had some friends encourage me to do this, even though they didn’t know that I was already heading down that path, but nevertheless it confirmed the direction that I was going.

These days, the goal of so many people is to live a comfortable life, and in today’s world with all of our technological advancements, that’s getting really easy to do (why bother to go to the store when Amazon will deliver almost anything to my door in 2 days?). It feels good to be comfortable, but that’s not the goal that I’m trying to achieve in life.

The new job has not been comfortable, at least not at first. When you start somewhere new, so many things are up in the air. You don’t feel like you know how anything works, and you know that at some point people are going to expect you to know things. At first you feel like you’re just trying to keep your head above water while people keep dropping more things on you to carry, but eventually you figure it out. That’s how it was for the first month or so. But at the same time, I found I had a lot more energy. Work felt stressful and I was tired when I got home, but I was getting up earlier, I had more time to exercise, and I didn’t feel lethargic during the day. This was a big change from my last several months at my previous gig, where I felt tired most of the time, especially in the afternoon.

I really underestimated how tiring it is to work somewhere where you are too comfortable. By all measures, everything there was great. It’s hard to find anything that was negative about the situation, other than the fact that I wasn’t learning and growing as a developer. All of that comfort couldn’t have been draining, but there was nothing that was stimulating. I think deep down I recognized this, which is what caused me to leave, but I didn’t know that it would have such a big effect on my energy level.

I’m starting to settle in at my new job, I feel a lot more confident and I understand a lot more about what’s going on. Compared to my old team, my new team (which is made up of all new people who haven’t worked together before) has a lot to figure out. Our process isn’t as smooth, we don’t move as fast, we don’t have a shared understanding of how we like to create software, getting anything to production is a lot of work, and I personally have a lot to learn and figure out. But I know this is all a good thing, and we will all learn from it. That’s what I wanted after all, to be somewhere where I could succeed, but I would have to work for it and learn a lot in the process. All of those things are happening, and I’m confident that our team will get a lot better over time. We are poised for a ridiculous amount of success.

For me, I’ve traded the comforts of the predictable for new territory, and it’s going to be well worth it in the end.

We are in a new age where employees don’t just want a job, they want to work for companies that support causes they believe in, are socially conscious, and give them a chance to make an impact that goes beyond their job description. Everyone wants to “change the world”, especially in startup culture where everyone is trying to “change the way people do X”. I love that people are thinking beyond themselves and trying to find ways to improve the lives of others and invest in causes that they care about.

While myriads of articles are written about startup darlings who are innovating and disrupting, what about the majority of us that work at the more “traditional” companies? What about everyone working at insurance companies, banks, accounting firms, government agencies, and any other kind of “enterprise-y” environment that doesn’t have a modern-looking space in a trendy neighborhood, free healthy food for lunch, and craft beer on tap? Some of these industries are more associated with corporate greed and corruption than social consciousness, but there are millions of people who work in those industries that have nothing to do with the mistakes made at the top. Many “socially conscious” job seekers would never even consider these companies.

I have a different view of changing the world. I certainly appreciate when companies want to make an impact on the world, but I don’t need my company to do this is order to make a difference at work. Why? Because regardless of where you work and what you do, people are probably involved.

Every day I go to work and create software that no one outside of the walls of the building will ever use. My work will never impact millions of users’ lives or change the way that the world does anything. This is because I work on internal business applications.

Meanwhile, there are a group of people that I work with that are very dependent on the software that I write. I can help them do their job, solve their problems, achieve their goals, reduce their frustrations, and feel good about coming to work every day.

There are a group of people that I work with are very dependent on what I do. They are the people on my team that are working with me to achieve a common goal, and they are counting on me to do my part of the work. We learn from each other, work together to solve problems, and we are involved in each other’s lives. We talk outside of work and we’re there for each other when life is challenging.

The people I work for are very dependent on what I do. They have careers, aspirations, goals, and plans and many of those are dependent on their team members working hard and being successful.

My ability to make an impact on the world around me is not defined by the place I work or the industry I’m in. We all have a chance to make a major impact on the lives of people we work with each and every day. But to somehow discredit companies in certain industries because they’re not #changingtheworld does a disservice to the millions of humans that work there that make an massive impact every day.

The Yelp app has over 10 million downloads on the Google Play Store. I love using Yelp to find good places to eat wherever I’m at, and it’s changed the way that I find restaurants when I’m in unfamiliar places. But these benefits are nothing compared to the difference the people I work with have made in my life while we were doing things like maintenance on internal Silverlight applications and writing data conversions.

I appreciate everyone’s desire to make an impact on the world. I am with you all on that and that’s the reason I love what I do, but my impact is not defined by my employer’s mission statement. I just hope that we all remember that we can make a difference wherever we are at, and you don’t need to be at a trendy company to do it.

Anyone who has worked in modern JavaScript libraries can relate to JavaScript Fatigue — the pain and stress that comes along with the pace of change in the JavaScript ecosystem in which we constantly change our mind when it comes to the tools that we use.

I’ll give you some examples, but I’m sure you could come up with this list on your own.

• Many of us built large AngularJS (a.k.a. Angular 1) apps, only to have the Angular team deprecate the “old way” and rewrite the framework from scratch
• “React is great!” – 2015… “React is great, but we all use Redux instead of Flux now” – 2016… “Which of these 20 starter projects do I model my React app off of, and do people still use these libraries? I have no idea what all these config files mean!” – 2017
• The hardest part about starting a new React or Angular project is setting up the dev environment (things like starter kits and angular-cli help)
• Use bower npm! But yarn is better, even though it’s like npm, just faster.
• React and Angular 2 are nice, but what about Aurelia, Elm, Vue.js, etc.? Are those going to make React and Angular 2 obsolete in the next year?
• This post sums it up quite well

If 2018 you reads this post, I’m sure you will notice that it’s outdated. The bleeding edge people are probably moving past all of these tools already.

We can argue whether JavaScript Fatigue is good or bad – there are obvious downsides, but you can also argue that we’re just improving really rapidly, which is a good thing, right?

The reality is that this is the world that we live in, and I don’t think people are longing to go back to the MVC + jQuery days. We complain about this, or we can adjust the way we work so that we can succeed in spite of it.

Write small apps

Let’s face it – your code is going to get obsolete a lot faster. You are going to spend a couple years creating a large app and then want to use a different framework. You’ll want to recruit and retain developers who want to use new stuff.

This doesn’t mean that you’re going to rewrite everything every two years, but if we write modular applications, we have a better chance of switching frameworks incrementally and start building modules in a new framework while keeping the old code running. And if you do need to do a rewrite, now you have a much smaller task in front of you.

Just because there’s a new way doesn’t mean the old way is bad

I’m a fan of most of the things in Angular 2, and overall I think it’s very well structured and easy to work in. But this doesn’t mean that Angular 1 is trash. If I were building a new app, I would obviously pick Angular 2, but Angular 1 is going to work just fine. Many times these new frameworks just shuffle a few things around, give things different names, have different conventions, etc. There are many ways to succeed, and just because someone came up with a new way that worked for them doesn’t mean that we can succeed with the old.

Get used to learning

I hope you’re already used to learning, but in software development things are always changing and we need to keep up. I think we should develop a healthy fear of getting behind in the times. Not that you always need to be using the latest and greatest everything, but you can learn a lot from learning new ways of doing things, and many times you can apply the ideas to your existing applications. Also, you’ll have a better idea of when it’s time to start moving to a new framework. If you don’t know where to start, sign up for Pluralsight and see what’s new. You’ll find all kinds of great content to help get you started.

Avoid company standards

Since the JavaScript world is moving so fast, enforcing company framework standards only ensures that you’re going to be behind. I want different teams to try different frameworks and different ways of doing things, because someone else might learn something that can help me. A great way to learn about a new framework is to have a coworker teach it to you by showing you his working code that is in production.

It’s one thing if your company standardizes on something like .NET/SQL Server/Windows or LAMP or Ruby/Rails/Postgres, etc. because infrastructure is involved. But when it comes to JavaScript, it all runs in the browser, so you can plug and play things much easier.

Relax

You don’t always need to be on the bleeding edge. Remember, the goal is to provide value and create working software, even if you’re using last year’s framework.

As developers, we love talk of clean code and nice, friendly code bases. We lament if our higher-ups don’t share our vision of a codebase that resembles a sunny day in a grassy meadow with the birds singing in the trees. Who’s right in this situation? Everyone is probably right… and probably wrong.

Before I bore you with yet another “it depends”, let’s think about why there are different viewpoints on this issue. The obvious answer is that many times it’s a matter of context — people in the business might not do a good job of explaining why delivering results is so valuable to the business, and development teams might not do a good job of explaining why having clean, maintainable code is going to help them deliver better and faster results. I don’t think you need a blog post to tell you that.

I had a fascinating conversation over Thanksgiving about why it is that we have these disagreements. Turns out it’s not always just a matter of context, it also has to do with your personality.

There’s an assessment out there called the Strength Deployment Inventory (SDI). You can read about it if you want the details, but the gist of it is that there are 3 primary ways that we are motivated and deal with conflict, and we all have varying degrees of each motivation.

Source: https://totalsdi.com/

• Performance-oriented – you have concern for task accomplishment and organization of resources to achieve results
• People-oriented – you have concern for the protection, growth and welfare of others
• Process-oriented – you have concern for well-thought out approaches, order, individualism, and self-reliance

It’s possible that you have some or all of some of these traits. There’s no right or wrong here, but it’s really important to understand the way people think (and how you think compared to others).

For example, many managers are performance oriented, and that’s part of what draws them to that role. But many developers tend to be process oriented. Herein lies the seeds of conflict, unless we can try and understand where each side is coming from. Going back to my conversation that I was having, my extremely process-oriented brother-in-law said that even though he knew that sometimes he had a tendency to not be focused enough on results, it was very difficult for him to not write clean code or do a “good job”. Perhaps he needs to do a better job of seeing the importance of results, but I don’t think we should quickly dismiss this as a weakness.

It all comes down to what strengths you need on your team. If you’re developing a smartphone app for a startup, results are pretty much all that matters. But if I need to write a billing system, I want my brother-in-law on the team because we will take full advantage his process-oriented strengths and we need his level of rigor in order to ensure that costly bugs aren’t introduced.

This thought process is not often reflected in job listings that list years of experience and knowledge of various frameworks as job requirements. In reality, whether or not you can adequately explain what an Angular directive is probably very insignificant compared to other innate strengths of the candidate. In my mind, the ideal goal should be to find someone that has the specific strengths that you are looking for so that you can take advantage of the things they are naturally great at. Sure, knowledge of frameworks is important, but it’s much easier to teach someone a framework than it is to change the innate strengths and weaknesses of a person.

We should work really hard to become aware of our weaknesses and improve on them, don’t get me wrong. But if you want to harness the true power of a person, find out what their strengths are and cut them loose in an environment where they can take full advantage of those strengths.

I’m wish I could hear people’s reactions to reading that title. I’m sure there would be a wide variety of opinions, from “don’t you understand what duck typing is?!?” to “why don’t use just use .NET”? Well, I do understand duck typing and I also use .NET, so hopefully this turns into a logical discussion.

I do a lot of .NET development. I practically grew up on .NET and statically typed languages, so my world consists of things like interfaces, dependency injection, and waiting for my code to compile.

If you use dynamic languages, you might thumb your nose at those other things, and brag how you don’t have to use those things in a dynamic language. This is all true, and every time I sit and wait 30 seconds for compile my mind drifts off to a land where I can write code and just run my tests without having to wait 30 seconds to all of my code to compile so that I can test 1% of it.

I love Ruby because I don’t have to have all of the ceremony that I have to deal with in .NET. I love being to able to design objects the way I want without having to structure it in a way that makes it testable. And of course not having to wait for code to compile completely changes the way you write code.

It’s not all rainbows and unicorns

Software development is a series of trade-offs. One thing I really love about Ruby is that you have a lot of freedom but it’s not as dynamic as something like JavaScript (there’s a difference between 1 and ‘1’ and true), but I don’t have all of the constraints of .NET. The trade-off is that I don’t have that check that makes sure that I’m passing objects of the right type into a method or makes sure that an object returns an object a given type. This is where people yell “DUCK TYPING!!!!!”, which is true. In .NET, a class implements an interface if it’s specified in the class definition, which means that it has to implement all of the methods of the interface. In Ruby, there is no explicit interface, and the interface is simply defined by the methods that an object responds to. That’s all well and good.

Do I want compile time checks in Ruby? Well, maybe. The argument against that is that I should have tests that test everything, and in most cases I will. Then you refactor something. You run your tests and they fail, but the failure message is due to something that happened much earlier in the test, and the failure is just exposing that a problem exists somewhere. Now you’re left digging to try and find the place where something is returning the wrong type, and sometimes this takes awhile.

As much as compiling in .NET can suck, the static typing really makes refactoring easier, in part because the refactoring tools are better as a result. I’ll just change stuff and compile and let the build errors tell me exactly where to go. This is about the only time I like compiling. But when you’re used to that check and then it’s not there and you realize that refactoring is a little harder, it makes you think.

I did something about it

If I can’t have compile time checks in Ruby, I can have runtime checks. Then when I run my tests, at least it shows me exactly where things are failing. But after you write raise "Expected Hash but got #{input.class}" unless input.is_a? Hash over and over, you start to feel like there could be a better way. But this is a little better than it was before.

People have thought about this

There has been some discussion about adding static types to Ruby 3, but it doesn’t look like it is going to happen. I suppose I understand, Ruby is a beautifully simplistic, yet powerful language, so I get it if some people aren’t on board with such a radical shift. There also gems like rdl that give you a way to implement runtime type checking.

Other languages have done this

TypeScript is basically JavaScript with static typing and compile time checks. Some people don’t like having TypeScript’s restrictions imposed on them, but most people that have used TypeScript seem to like it. Either way, TypeScript gives a nice blueprint of a way that statically typed Ruby could work (I’m sure there are nuances that make it different, but you get the idea).

Discussion is good

I’m sure that people will continue to debate ideas like this, and if anything it provides a good education on differing philosophies in programming language design and the pros and cons of them all. It’s also a good reminder that there is no perfect programming language, that every benefit comes with a trade-off, and there is more than one right way to get the job done.

When you change code, how do you know that you’re not breaking something? If the code change is small, it can be pretty easy (especially if you have tests). I’m talking about more complicated code changes.

Sure, you have unit tests, but how do you know that everything in the system works together correctly?

Sure, you have end to end acceptance tests for all of the scenarios that you know about, but what if a scenario exists that you didn’t think of?

We can come up with any number of excuses for why bugs happen in production. But what happens when a bug could cause a big problem?

I’m in this situation right now because the relatively large code change I’m implementing could impact how much money we charge our customers. The code I’m writing takes in some input files from a third party and processes them. We could have a big problem if someone gets overcharged or undercharged, or even worse, not at all. There are lots of things that can go wrong. I have a lot of good acceptance tests for my code, so I have good test coverage for the scenarios that I know. What I worry about is the scenarios that I don’t know, or what could cause my code to not get called at all. Because of this, I have to think past the traditional ways that I usually test things.

Searching for test cases

I did a lot of analysis on the data that my code is going to consume. I even wrote some small apps and SQL queries that will parse historical data and look for different combinations of data, and used that to come up with the test cases. What’s tricky is that some of the scenarios do not happen very often, so that means I need to do more digging to find them. At some point, this can become tedious, but in my case it’s worth it.

Watching for unexpected scenarios

Thanks to my data analysis, I have a list of expected scenarios that I’ve seen from past data. In addition to writing tests for all of these scenarios, I’m also writing some queries that will check for evidence that some unexpected scenario happened in production, and I’m running these queries every day. By doing this, I’ll be aware of any potential changes in how the source data comes in, and I also avoid having to write acceptance tests for scenarios that probably aren’t going to happen.

Throw exceptions when you find unexpected scenarios

Not only do I have queries to check for unexpected scenarios, I also have inline checks that will cause the process to stop and throw an exception if I encounter specific unexpected scenarios that could cause an issue, especially when the issue would not otherwise be obvious (in other words, it wouldn’t cause the system to fail, but the system might give me incorrect results). In many cases, it’s a waste of time to discuss, implement, and test some edge cases that is very unlikely to ever happen, but I at least want someone to know if by chance that happens. Now if you’re writing a UI for a website, you might not have the luxury of being able to do this, but when you’re just processing backend files, you can get away with things like this.

Parallel testing against production data

We have a test environment set up with 2 databases – one has 2 day old production data and one has 1 day old production data. We restore these databases from backups every morning and then run in all of the files from 2 days ago into the 2 day old environment, then we compare the results with what’s in the 1 day old database, and theoretically everything should match. There are always exceptions, especially when you have data that was modified by a user after it was created in the database, but this allows me to easily check almost every column on a database table and see that the results from my process match what is currently in production. This has been a huge lifesaver, not only in finding bugs in my process, but for finding unknown scenarios, and for finding bugs in other related processes as well. People have a lot more confidence in your work when you’re able to show tangible proof that your system is working the same as what’s in production.

Writing audit queries that validate the process end to end

In addition to the queries that check for unexpected data, I also have queries that validate that any data received at the beginning of the process will have some resulting output. I need to make sure that the process isn’t silently failing or is failing in a way that I don’t expect.

Checking logs

Hopefully your app logs exceptions somewhere, so as a last resort you should always check for any fallout in the logs.

A different thought process

I find this way of thinking to be a different thought process when it comes to testing. I’m moving past the mechanics of testing (what are my test cases, what tests am I going to write, how can I mock this class, etc.), and trying to find whatever means necessary to not break stuff. Sometimes this is done with automated tests, manual tests, inline sanity checks, writing exploratory code to help me discover scenarios, or some form of production monitoring. The latter three are where I’ve found I’m doing things in a way that I haven’t always done before.

I think the shift in mindset partly comes from owning the responsibility for a feature. At one point a long time ago, I wanted to get my code to work and let QA test it and find bugs. Then I moved to wanting my code to work and wanting to come up with a way to test it. The next step in the progression for me is finding ways to be able to make large changes to the code, not break stuff, and ensure that everything will work with no impact to the business. In each step in the progression, I’m starting to look at the problem at a larger scale and looking at business value and business impact instead of just technical concerns.