Software developers are notorious for falling in love with new, shiny things. I always try to use the right tool for the job, whether that’s new or old technology. But every now and then I find something that makes a whole lot of sense in my head and I can’t ignore it.
I’ve been learning about NoSQL databases for the last year or so mainly because there’s a lot of noise about them and I was curious, and I also find data access to be this annoying problem that always seemed to be harder than it should be. First I looked into MongoDB and then eventually RavenDB. I haven’t used either of them in production yet (although I work with people who have), which is why I titled this article a “hypothesis” because I don’t really have any real life stories to tell to back up what I’m going to say.
In my research of NoSQL databases I noticed two particularly interesting things:
1) I find that ORMs make data access with relational databases much easier in most cases over using stored procs for everything, but even with all of the experience I have with various ORMs, data access is still a pain.
2) Ayende Rahein was one of the main contributors to NHibernate for years, and he probably understands ORMs better than almost anyone in the world. And yet even he decided to create RavenDB and move to NoSQL. If the ORM expert decides that using an ORM is too hard, maybe he’s onto something.
Here’s how I see it: relational databases are really good for reporting and querying, but not so good for loading information in an application. NoSQL databases are really good for loading information in an application, but not so good for reporting and querying.
The application I’m working on is 2 years old now and we use SQL Server. We are starting to accumulate a lot of data in the database, which is exposing performance problems in our application and everything is starting to process a little slower as we get more data in the database. Our database is not good at loading information in the application (it takes a lot of queries to load up the main screen because it has to query so many tables).
We are also starting to move away from reporting against our database now and moving more data into our business intelligence data warehouse so that our reports can run faster and be written against a relational schema that is optimized for the types of reports that we tend to write.
So let’s recap.
1) Our app is not good at loading data into the application using a relational database
2) Our relational database is not going to be used for much reporting and querying because we’re going to move the data into another database for that purpose.
So with that in mind, why am I using a relational database? Shouldn’t I be optimizing for the loading and saving of the primary objects in my application since that’s my pain point? (I haven’t even started talking about the increased in developer productivity from being able to do more in the business layer and less in the data access layer and stored procs.)
I’m not saying that I’m going to switch my application from SQL Server to RavenDB, because I don’t know that it would be worth the time at this point. But if I were to do so, I would be able to delete a slew of complicated stored procedures and turn my data access layer to a really really thin layer than essentially just passes a C# object to RavenDB. I would be able to load, modify, and save our primary object (and all it’s children) with 2 database calls instead of over 20 (in some cases). If I use RavenDB, there are built in mechanisms to write triggers that will take updates from RavenDB and replicate them to a relational database.
When people start a new application, it’s just assumed that they will use a relational database. But why? Is a relational database really the best data store for every application? Given that we have good NoSQL alternatives, maybe it’s time we start evaluating all the options.
I’m sure it’s not all rainbows and unicorns, and in some ways you’re just trading one set of problems for another. Dealing with large amounts of data is never easy. But I’m seeing a lot more solutions than problems.
I posted a few years back about some of the macros that I have in my AutoHotKey file. Since then I’ve added some new things that are worth mentioning.
Inner joins
The project that I’m on involves a lot of database work. I’m writing queries all the time, and this means typing the same inner joins over and over. I would rather let AutoHotKey do that for me.
Here’s a query that I write often:
select * from igs.accounts a
inner join igs.enrollments e on e.accountid = a.accountid
inner join igs.intentions i on i.enrollmentid = e.enrollmentid
where a.utilitylineofbusinessid = 5
It’s incredibly painful to type those inner joins over and over (it was even painful writing it for this blog post), so I use AutoHotKey to do it.
:*:ea]::inner join igs.enrollments e on e.accountid = a.accountid :*:ae]::inner join igs.accounts a on a.accountid = e.accountid :*:ie]::inner join igs.intentions i on i.enrollmentid = e.enrollmentid :*:ei]::inner join igs.enrollments e on e.enrollmentid = i.enrollmentid :*:wcpa]::where utilitylineofbusinessid = 5 :*:ssf]:: SendEvent select * from ` return
So now to type that same query, here is what I type:
ssf]igs.accounts a
ea]
ie]
wcpa]
I have macros for virtually every inner join I would ever do in the database so that I don’t ever have to type one out. If I find one that I missed, I go back and add it.
This makes writing queries so much easier. Once you find ways to do mundane things faster, it makes you do everything faster because your typing is keeping up with your brain.
Switching databases
We have several database environments just like most of you, which means that I’m switching between database servers all the time in SQL Server Management Studio. Again, I can pick up the mouse and click 5 times in dialogs and toolbars to switch. Or I can do “Alt-q ch Ctrl-u i”. But I would rather do it with one keystroke.
#^1::
Send !qch
Send thedatabaseserver
Sleep 200
Send {Enter}
Sleep 100
Send ^ui{Enter}
return
Boom! Ctrl-Win-1 and I’ve switched to my database server and selected my database (the “Send ^ui{Enter}” line does that – Ctrl-U selects the database dropdown and “i” selects my database because my database name starts with “i”). I have shortcuts for all of my database environments so that I can do even less with the mouse than ever.
Commonly used queries
There are certain queries that I use a lot (e.g. select the top 1000 rows from our log table). So I wrote a macro for that.
:*:seq]::select top 1000 DateCreated d, * from igs.ServiceExceptions order by DateCreated desc
Typing is slow
Typing is slow (and I’m a faster typist than most), so anything repetitive that I do often I want to turn into a macro or keyboard shortcut. Hopefully some of these ideas will give you ideas of how you can improve your productivity.
I wrote a guest post on the Mashed Code Magazine blog, you can read it here.
We all have numerous activities and people competing for our time. More than anywhere else, this seems to play out at work.
If you’ve ever had a day full of meetings, you know what I’m talking about. It feels like you go to work for the entire day so that other people can get stuff done while you feel like you accomplish nothing.
A day half full of meetings isn’t much better, especially when the meetings are scattered throughout the day. Sure, you technically have half of your day to get work done, but it’s constantly broken up by meetings.
We came to a point on our project where this was getting really bad, especially for developers. We would often spend half of our days in meetings, and as a result we wouldn’t have any long stretches of time to focus on writing code. We had to do something about it.
So we did. We blocked off 2pm until whenever you left each day for no meetings. We acknowledge that meetings are necessary, but we need to set aside time for ourselves so that we can get the things done that we have promised that we would get done. This is an actual calendar invite in Outlook, so the time shows up as busy to anyone trying to schedule a meeting.
We also blocked off 10am-11am every day to have meetings. When we need to schedule a meeting, we try to fit them in between 10-11. We have our standup at 10, so we’re already stopping for that anyway, so it’s a good time to talk.
The Results
This system has been awesome. I often have meetings from 10-11, and sometimes longer, but if the 10-11 slot is full, people will schedule meetings in the time slots around it. As a result, I’m often done with meetings for the day by lunchtime. I have close to the same amount of meetings as I did before, but now I get them all over with at once. Now I have the whole afternoon to get work done. This has done wonders for our sanity.
Since we don’t have many time slots for meetings, this forces people to not schedule frivolous meetings. You know the ones I’m talking about, where someone schedules a half-hour meeting for what ends up being a 10 minute conversation. Now we just have a 10 minute conversation after the standup.
There is a real cost to context switching. It takes some time to get in the zone, regardless of whether you’re writing code, writing requirements, or testing. Every time that you have to stop what you’re doing to do something else, you need to spend time to get back into what you were doing.
In the end, it boils down to being in control of your schedule so that you have time to do the things that are most important. I encourage you to set up whatever boundaries you need in your day in order to help you succeed.
Much has been said recently about companies like Github, Valve, and other places that create these open corporate cultures of empowerment where employees can do pretty much whatever they want. This manifests itself in many different ways, from allowing employees to work where they want when they want to allowing them to even decide what they want to work on and not limiting their time off. A great example of this is Valve’s new employee handbook. Stop reading this post and go read the handbook, it’s worth the read.
(……………)
Hey, welcome back. How do you feel after reading something like that? It gets you a little excited, doesn’t it? Personally, I love the idea of getting to move my desk wherever I need to, or being able to decide what I want to work on. In a place like that, there’s no limit to what you are able to accomplish. You are only limited by your own abilities and time.
Let’s be honest though, most of corporate America can’t be as free wheeling as a video game company. When I go to work, I’m doing work for people who tell me what project to work on, and they get to decide what projects are most important to them. That’s the nature of the company that I’m working for, and there’s nothing wrong with that.
I care more about the principles behind these empowerment cultures. The main theme that I see is that they hire awesome employees and then let them be awesome. How many times do you hear about developers who feel like that can’t be as successful because their team lead or manager won’t let them use a certain technology or framework or technique? Maybe those people feel like their control is doing good (and maybe it is), but the trade-off is that they are stifling their top performers. In that case, the side effects might be worse than the perceived decrease in risk.
Control
Since most companies don’t have an empowerment culture, there must be a reason.
It is really hard for people in control to give up control. Say you’re a manager, and your lead developer comes to you and says that he wants to use MongoDB instead of SQL Server on his new project. Immediately you think of reasons why it’s not a good idea (Our infrastructure team doesn’t know how to host it properly! Our DBAs don’t know how to use it! What about reporting?). Those are all important questions that need to be answered, of course, we don’t want to be reckless. Ultimately it comes down to a choice – you can choose the safe option (SQL Server), or the more unknown (MongoDB) which might have some new challenges that you need to work through, but also might provide developer productivity increases, better application performance, and increased morale of the team.
If you don’t let top performers be top performers, why do you have them? If you stifle them, you are losing out on one of the main reasons you have them in the first place! If it lasts long enough, they will leave and try to find another place where they might be able to use their skills.
Trust
It takes a lot of guts for a manager to give up control. If that manager were to let his developer go ahead with MongoDB, that would take a lot of trust. He’s basically putting the project at risk (maybe not really, but in his mind) and placing a lot of trust in his development team. And you know what, it might fail.
But let’s be honest, if we can’t trust our development team, why do we have them? Why don’t you just get rid of them and have the managers and architects do the work? Are they really the only ones who can be trusted to make big decisions, and even small decisions?
Fear
Employees that live in a culture of control often live in fear. These cultures often try to remove risk because failure is seen in such a negative light. As a result, you can’t try new technologies (because it might fail), you have to use the approved enterprise development frameworks (so that we use something that we know has worked last year), and heaven forbid you say the “A” word and want to implement agile practices (because the project managers don’t have as much control). This may keep some projects from failing, but it pretty much guarantees that every project will be sufficiently average.
Do you want to work in a place like this? I don’t.
Freedom
In a culture of empowerment, you have the freedom to try new things. And since some of those things won’t work, you also have the freedom to fail. When it does work, we all win, and when it doesn’t work, we learn something so that we are better next time. But everyone wins when it comes to that feeling that you get when you go to work believing that you can change things for the better and that you can make a difference. Now I truly have responsibility, because I’m responsible for the success of something and I will be held accountable for my decisions. But this just motivates me to do better.
That freedom is what really gets people excited. When you don’t feel like you have anything holding you back, your mind is free to dream of anything, including those crazy ideas that might actually work. This is incredibly motivating, because now we’re only limited by our team’s abilities and time.
Side effects
There are certain types of people who tend to prefer empowerment cultures. The ones who don’t like it are the ones who want to play political games, have a bad attitude, and are afraid to try new things because they are afraid they might fail. The ones who like it are the innovators, the people who embrace change, the motivated, the top performers. These people don’t just value a salary, they value autonomy, mastery, and purpose. Every company tries hard to recruit these people, but few realize the importance of the empowerment culture.
I read an interesting article in the Wall Street Journal the other day about “Must-Have Job Skills in 2013″. While this wasn’t necessarily referring solely to technical fields, it was still interesting. Here is their list of must-have job skills:
- Clear communications
- Personal branding
- Flexibility
- Productivity improvement
It made me think of the software craftsmanship people and the importance that they place on things like code katas, learning new languages, etc. If you’re a developer and you spend time honing your craft, how much value are you placing on skills like:
- Being able to write a good requirements document
- Facilitating a requirements gathering session with business users
- Giving demos of your software to users (in their language)
- Coming up with estimates for a large set of functionality
- Being able to evaluate tools and frameworks and choose the best one for your project
Admittedly, it’s much easier to practice TDD than it is to practice requirements gathering. But these are skills that I feel are very important for developers. There are lots of people who can write code, even good code. If I don’t know how to use a language or framework, someone can teach me pretty quickly and I’ll pick it up. But can you also pitch in and help with requirements gathering, test planning, system architecture and design, and everything else that needs to be done on a project? Now that will set you apart.
When you think about learning new skills and investing in your career, just make sure that you don’t limit that to tools and technology.
Time is one thing in life that will always remain constant. You can acquire more knowledge or more money, or you can save today’s money and use it tomorrow. But you only get 24 hours a day, and you can’t carry them over to tomorrow.
Most people would agree that there are not enough hours in a day. There are plenty of things I would like to do, books I want to read, projects that I want to work on, that I just don’t have the time to do. The challenge is figuring out how to make sure that you’re spending your time on the things that are important to you.
Know your priorities
If you don’t know what your priorities are, you won’t know how to prioritize your life. That sounds obvious, but it’s really hard to actually do. Many people don’t really know what their priorities are, let alone how to execute on them.
Do what’s important first
Something or someone will fill your day if you won’t. Personally, I hate the days where I feel like I didn’t get anything done that I wanted to get done because I was constantly reacting to things that came my way. Sometimes this is inevitable, but you can control it. For example, I expect that I should have at least half of my day available to do work (i.e. not stuck on meetings), so when a day is half full, I block out the rest of it so that no one can schedule me for a meeting. I do that because I need that time to accomplish what I feel is important and what people are expecting me to accomplish.
Live out your priorities
People are a priority to me, so if someone wants to go to lunch, I may or may not have time for that, but I usually go anyway and then find a way to everything else in around it. I have other friends who will go out of their way to make time to eat lunch when they’re really busy. That shows me something – that I’m valuable enough to be made a priority in their life. I love that.
My team is also a priority for me. This goes back to shunning meetings. I feel that in my current role, it’s more important for me to be with my team than to sit in a meeting with people from another team, so I block off my schedule and try to get out of meetings if I don’t feel that it’s valuable for me or the meeting organizer. (Sometimes people like to schedule 30 minute meetings for things that could be solved with a 10 minute hallway conversation.)
Own your life
I want to control my life and time as much as I can. I don’t want my life to control me because then I’m not able to do the things that I feel are important. The more proactive I am, the more likely I am to succeed at this.
We had a week-long debate awhile back about whether or not it’s OK to modify your production code in order to enable automated acceptance testing. I’m not talking about using dependency injection, interfaces, etc. to allow you to mock things in unit tests. I’m talking about modifying application code solely to help your automated acceptance tests.
There are many ways this can be done, some of which we’ve done:
- Creating a SystemTime class, which is like DateTime except that we can set what “Now” is, so we can change time in tests
- Adding optional parameters to stored procedures solely so that we can have them only operate on a subset of data in an acceptance test instead of operating on the entire set of data in the database
- Adding extra HTML attributes so that automated tests can find elements on a page easily
To me, modifying production code to help us do automated testing is no big deal. First, if our goal is quality, I don’t think it matters how we get there. After all, we own the code base and tests so there aren’t any real restrictions on what we can do with the code or the tests as long as the end product is good.
Second, developers and QA are on the same team, and we work together quite closely, so we should do what we can to help each other out. So if we can make a minor change to the application code to save us a lot of time developing or running automated tests, then to me it makes sense to do so.
This goes back to my assertion that we need to stop thinking of QA like external auditors that have to take the application just as it is without talking to the developers and act as the independent quality police. We need to all work together to ensure quality, both developers and QA. Developers are just as responsible for quality as QA. If we place all of the responsibility for quality on QA, then developers will care less and less about quality, and you end up with shoddy code with lots of bugs (and usually no tests). I’d rather treat testing as a whole-team activity and structure the application to make testing as easy as possible.
Here is a recording of my talk entitled “The Business of You – 10 Steps To Run Your Life And Career Like A Business” from CONDG on 9/27/12. Thanks to Jeff Miller for the recording. Enjoy!
Lately I’ve been thinking about a whole team approach to testing, where we decide as a team how features will be tested and where we use the skillsets of the whole team to automate testing. We do this on our project, and this has led to a regression testing suite of ~2500 SpecFlow acceptance tests that automate almost all scripted QA testing and regression testing for our application.
We didn’t always do this. Originally there was no automated acceptance testing, but developers were diligently writing unit tests. Those unit tests are still around, but we don’t write many unit tests anymore. We start with acceptance tests now, and the acceptance tests cover all of the testing scenarios that need to be covered. Our application has well-defined design patterns that we follow, so the idea of TDD driving the design of our code doesn’t really apply. If the unit tests fail, we often just delete them because it’s not worth fixing all of the mocks in the unit tests that are causing them to fail, and we have acceptance testing coverage around all of it.
This approach does not line up with the conventional wisdom on automated testing. They say that you’re supposed to write lots of unit tests that run really fast to give you fast feedback, help design your code, and ensure the internal quality of your code. In the past, this is how I’ve always done it. In fact, many of them dislike Cucumber.
Cucumber makes no sense to me unless you have clients reading the tests. Why would you build a test-specific parser for English?
— DHH (@dhh) March 29, 2011
While TDD isn’t as mainstream as I would like, TDD is nothing new. Kent Beck was writing about it 10 years ago, and the original XP guys valued such things as unit testing, the SOLID principles, and things like that.
Automated acceptance testing still feels like a relatively new phenomenon. I’m sure people were doing it 10 years ago, but back then we didn’t have Cucumber and SpecFlow and the Gherkin language. Now I see a lot more people using tools like that to automate QA testing in way that uses business language and more maintainable code, rather than the old “enterprise” solutions like QTP.
Here’s what I’m getting at – I wasn’t there 10 years ago when Kent Beck was writing his books and the XP movement was starting, but it seems to me to be primarily an effort by developers to ensure the quality of their code through the effort of developers. I see very little talk of where QA fits into that process. There is some mention of QA for sure, but the general gist seems to be that developers need to write tests in order to ensure quality, and the best way to do that is to write unit tests. QA people typically don’t say that unit testing is enough because it doesn’t test end-to-end, so then what do they do? Manually test? Use QTP?
My question is this – if we think of testing as whole-team activity and not just a QA activity or a developer activity, will we arrive at the same conclusions as we did before?
I’m not ready to discount unit testing as a valuable tool, and I’m also not ready to say that everyone should do it my way because it worked for us on one project. But we have largely abandoned unit testing in favor of acceptance testing, and other teams in our department are doing it too. I write unit tests for things like extension methods and some classes that have important behavior on their own and I want to ensure that those classes work independent of the larger system.
We have 3 Amigos meetings in which one of the things we do is develop a set of acceptance tests for a feature before any code is written. We usually decide at this point (before any code is written) that most or all of these scenarios will be automated. We write the acceptance tests in SpecFlow, I watch them all fail, and them I write the code to make them pass. I follow the patterns and framework that we have set up in our application, so there aren’t many design decisions to make. When my acceptance tests pass, I am done.
Where do unit tests fit in there? If my acceptance tests pass, then I’m done, so why spend more time writing duplicate tests? Also, with acceptance tests, I’m not dealing with mocks, and more importantly, I’m not fixing broken unit tests because of broken mocks. If you follow the Single Responsibility Principle (which we try to do), you end up with lots of small classes, and unit tests for those classes would be mostly useless because those classes do so little that it’s hard to write bugs and each class does such a small part of the larger activity.
There is an obvious trade-off here – my acceptance tests are not fast. I’m just testing web services (no driving a browser), so all ~2500 tests will run in about an hour. But we accepted this trade-off because we were able to get things done faster by just writing the acceptance tests, which we were going to do anyway to automate QA testing. The end result is high quality software with few bugs, not just because we have tests, but also because we communicate as a team and decide on the best way to test each feature and what it is that needs to be tested, and then we find the best way to automate the testing as a team.
Again, I’m not ready to say that this way is the best way for every project, and I’ve seen each approach work extremely well. I just wonder if the conventional wisdom on testing would be the same if we thought of it from the perspective of the whole team.
Older Posts »
