Estimation of features is a critical part of the software development process, especially if you are using an Agile methodology. The ability to accurately estimate features is an often overlooked skill that every developer should have.
As a project engineer, I’m responsible for making sure that we set realistic project timelines and that we actually carry those out. Managing expectations is one of my main responsibilities. This is where Agile is so nice — as long as you do it right.
When I start on a new project, usually one of the first things that happens is meeting with the client or whoever is sponsoring the project. In these initial meetings, at some point I will probably be asked one of two things: 1) how long will it take to accomplish X? 2) how much can we get done before X (some date)?
Let’s think about what is happening here. Your project sponsor is asking you to define what is realistic! This is great! You have one chance to set realistic expectations, and that time is now. Of course, you have to be fair — if you try to sandbag and say that it will take 4 days to add a button on a page, you will probably get called out. But even worse is if you underestimate things, because you will probably be held to your estimates. Once you give the sponsor the project timeline, they will take it to their bosses and spread the word around. So if you don’t get your project done on time or if it’s over budget, then the sponsor is in hot water. Not good.
If you aren’t asked one these two questions (i.e. they say, “We need to have X done by Y.”), that can still be OK. But if what they’re asking for isn’t realistic, now is the time to do something about it. No one likes to be told that what they want to do can’t be done or isn’t realistic. But I guarantee that they will be happier knowing about that up front than when you miss your deadlines.
When we estimate on our projects, we break functionality into features. Our criteria for features is that they should (usually) be something that can be tested and usually takes anywhere from 1/2 day to 10 days. There are lots of ways to do Agile and people do Agile in different ways, but most people break things up into features in some way.
The biggest problem I see with estimates is that people frequently underestimate things, and that’s usually because they don’t think of everything that goes into developing a feature. When I estimate, here are all of the things that I take into account:
In some cases, there might be more things to take into account.
One line that we always say about estimates is that “all estimates are wrong.” This statement is true on a feature-by-feature basis, but not for the project as a whole. This is because your project sponsor is going to hold you to your estimates. Whether that’s always fair or not is debatable, but that’s the way it is. You may not have all of the answers when you estimate a feature, and that decreases the chance that your estimate will be correct. So we need to try and have as many questions answered as possible before doing the estimating.
Sometimes you’ll start work on a project and realize that you’ve underestimated everything. This is not a good thing, but it happens. When this happens, you should re-estimate the features again so that your estimates can be accurate.
You’re also going to have the not-so-fun situation where you have to let your project sponsor know that it’s going to take longer to finish than you originally thought. Obviously this is not going to be a fun conversation, but the time it takes to develop a feature is the time it takes to develop a feature. You can change that reality. So the sooner you can adjust the expectations, the better chance you have of refocusing the project and righting the ship.
Estimating is a great tool for tracking progress and managing expectations, and it’s a reason that I’m such a big fan of Agile. It’s also a skill that every developer needs to master.
Recently I lost several team members from my team. As a result, suddenly there is all this top-of-the-line hardware lying around. Of course, I can’t let good hardware sit there and rot! So now I have the developer’s dream environment.
You are looking at my quad-core, 64-bit, dual-boot system with 4 20″ flat panels and 8 GB of RAM. Synergy let’s me use one mouse and one keyboard to control both machines. Now that’s hot!!
On my current project I was introduced to Behavior Driven Development. When I first heard of BDD, I thought, “Is BDD any more than a new way to name your test methods? What’s the big deal?”
Here’s a simple unit test class that I might have written in the past:
[TestClass]
public class EmployeeTests
{
[TestMethod]
public void LastNameRequiredTest()
{
}
[TestMethod]
public void BirthDateCannotBeInFutureTest()
{
}
[TestMethod]
public void LastWidgetDateTest()
{
}
}
When I would write a test like this, the goal was to make sure that the code that I had written was working (I normally wasn’t doing TDD when writing these tests). Now there’s nothing wrong with that, let’s say that my code coverage was good, and things are good.
A BDD-style set of tests would look more like this:
[TestClass]
public class Given_an_Employee
{
[Setup]
protected override void Before_each_test()
{
}
}
[TestClass]
public class When_validating_an_Employee : Given_an_Employee
{
[TestMethod]
public void Then_the_LastName_is_required()
{
}
[TestMethod]
public void Then_the_BirthDate_cannot_be_in_the_future()
{
}
}
[TestClass]
public void When_an_Employee_makes_a_widget : Given_an_Employee
{
[TestMethod]
public void Then_the_LastWidgetDate_should_be_set_to_today()
{
}
}
Whoa, what’s with all the classes and underscores and all that! Put your preconceived notions aside and let me explain. :)
First, let me explain why TDD is good. I think that these points illustrate this best.
1. The developer starts writing unit tests around their code using a test framework like JUnit or NUnit.
2. As the body of tests increases the developer begins to enjoy a strongly increased sense of confidence in their work.
3. At some point the developer has the insight (or is shown) that writing the tests before writing the code, helps them to focus on writing only the code that they need.
4. The developer also notices that when they return to some code that they haven’t seen for a while, the tests serve to document how the code works.
5. A point of revelation occurs when the developer realises that writing tests in this way helps them to “discover” the API to their code. TDD has now become a design process.
6. Expertise in TDD begins to dawn at the point where the developer realizes that TDD is about defining behaviour rather than testing.
7. Behaviour is about the interactions between components of the system and so the use of mocking is fundamental to advanced TDD.
First you begin to write tests because you want to make sure your code works. But as you get into TDD, you realize that TDD really can be more about design than testing. If you can define acceptance criteria up front, you will be forced to think about what your code is going to need to do and design it accordingly.
BDD takes the next step. First your tests were just testing your code, then they were designing your code, and now they are describing what the code is doing. This is where the weird class and method names come in. Let me write out those tests again, translating it from C# into real world language.
Given an Employee, when validating an Employee, then the LastName is required and the BirthDate cannot be in the future. When an Employee makes a widget, then the LastWidgetDate should be set to today.
Here is where the light bulbs clicked on for me. My tests aren’t just testing my code, they are telling a story. Imagine if you were taking this code over from me and you wondered, “What is important about an Employee?” You could look at my tests and you would know exactly what is important about an Employee. You would immediately know what all of the acceptance criteria were for working with an Employee. So now you can go and change my code and have a good sense of what I was trying to make my code do!
I think by now you can see how TDD and BDD will go hand in hand. It should go something like this:
1) You get your requirements. In the requirements are a list of business rules or acceptance criteria.
2) You write your tests (or at least the classes and methods). Now you’ve translated those business rules and acceptance criteria into tests.
3) You write your code. You already have a suite of tests that you can run to know if you succeeded!
Of course, you may go back after the fact and write even more unit tests that focus on testing more granular parts of your implementation than just the overall business rules.
If you don’t like all the funky underscores that I put in the class and method names, then that’s fine. That’s just how we did it on our project (it’s all this guy’s fault). I like them because it really does help the names to read more like normal human sentences, which is much more valuable that I had originally thought.
TDD and BDD are a definite paradigm shift for a lot of people. But once you get used to it, you will never go back.
I’ve had lots of positive responses to my last post from other people that were involved in the original discussion. I just wanted to say that I was not the least bit offended by anything that was said by anyone in the original meeting. I don’t think that I could ever be upset at anyone for disagreeing with me on technical issues.
One thing that I really like about software development and architecture in general is that every project is a new challenge with new requirements that will affect how you architect a solution. So we need to be able to evaluate each situation and then evaluate what architecture fits the situation best.
This is why having discussion and disagreements is good. None of us know it all, and we can all learn from each other. My goal is not just to learn what ORM to use, or how to use WCF, or anything like that. I want to learn how to evaluate each situation so that I can design the system appropriately and use the tools that will help me get the job done.
Today in a meeting we had a big debate about project architecture, anemic domain models, SOA, and other kinds of good stuff. Seeing that I was in the minority arguing against lots of really smart people I should answer the call and defend my ideas. :)
The question that started the discussion was, “Does SOA lead to the anemic domain model anti-pattern?” This issue has been debated in the altdotnet forums recently.
First, a little about the architecture of my project. We are writing a Windows Forms app that talks to a middle-tier using WCF and then saves to a SQL Server database. Our layers go something like this:
User Interface (controllers, views)
Service Agent (helps controllers talk to web services)
—- service boundary —-
WCF services
Business Logic
Data Access (LINQ to SQL)
————————–
SQL Server
LINQ to SQL generates all of our entity objects and these entity objects are used on both sides to the wire. Our web services are not exposed to the public and our Winforms client is the only app consuming them.
Here are the issues that people have with this:
Sharing the entity objects on both sides of the wire violates SOA tenets because it introduces tight coupling between server and client.
Sharing the entity objects on both sides does violate SOA tenets. But in our case, what we’re doing is not SOA! Wikipedia defines SOA as:
Service-Oriented Architecture (SOA) is a software architecture where functionality is grouped around business processes and packaged as interoperable services. SOA also describes IT infrastructure which allows different applications to exchange data with one another as they participate in business processes.
Notice the part about applications exchanging data with one another. We are not exposing our web services to anyone other than our client. We have total control of both sides of the wire. Just because I’m using web services in my app does not mean that we have a service oriented architecture! So why should I create different entity objects for each side of the wire along with DTOs and translation classes? What benefit does that give me?
Sure, sometimes you add properties to entity objects just to help you display something in a grid. And sometimes you have properties on an entity that you will never need on the client side. And theoretically there is a situation where you may want your entity object on the client to be structured differently than they are on the server for some reason. But are any of these good reasons to create new layers and create more code that I have to maintain?
I’ve worked on projects that have had separate entities for the client and server (with DTOs and translation in between). This was quite painful. Every time I had to add a new entity object with corresponding services and UI screens, I had to add 13 files to the project. 13 files!! This was before I even wrote any unit tests. I spent more time writing plumbing than anything else.
One day when I was working said project I picked up what looked like an easy bug. I had to add 2 fields to a database table and then add them to a screen. No fancy logic, no validation. It took me 20 minutes! All I was doing as was adding two properties to a screen! These types of things should be easy.
On my current project, things go much faster. I can drag my database tables onto the LINQ to SQL designer and immediately I have my entity objects created and I can load and save them to the database. I can immediately go and start writing code that has business value. I have to create 4 fewer classes for each entity now (no client entity, no DTOs, no translation code). I’m not hand-writing my entities and mapping files because these are generated for me. Sure, they’re not true POCO entities (although you can do this with LINQ to SQL if you don’t want to use the generated code), but I don’t think that it’s worth throwing away the generated code just so that I can write my own POCO entities (read: plumbing).
There are some cases where we still need DTOs to load and save data in complex situations, but we don’t spend the time doing it until we need to.
Like I said before, all of my web services are only consumed by our client. If another application or someone outside of my team wanted to consume my services, I would create a new service and use DTOs and translation so that I don’t tightly couple my domain model with outside consumers. Now I’m doing true SOA! But I don’t spend the time doing all of the extra work until I need to do it.
Because LINQ to SQL generates our entity objects and we use these throughout the app, we don’t have a rich domain model and our entity objects map one-to-one with database tables.
Sure, I admit that having entity objects map one-to-one with database tables isn’t always “ideal”, and LINQ to SQL doesn’t handle many-to-many relationships that well. But honestly, it’s not that bad. This doesn’t always sit well with the Domain Driven Design crowd, who say that you should start by creating your business objects and that the primary focus should be on the domain and domain logic. Certainly these are good concepts, and I’m not saying that DDD is a bad idea at all. If we were using NHibernate instead of LINQ to SQL we could make our business objects more like business objects and less like database tables.
My point is this — there are situations where Domain Driven Design is good (complex domain designs). But keep in mind that the goal is to deliver a working piece of software (that is testable and maintainable and all that). No one will care that your business layer is super-elegant if you don’t get done on time. In that case, my “good enough” business layer trumps the more elegant business layer that took longer to develop (in fact, I’d guess that with DDD you would still have the majority of your business objects map one-to-one with database tables anyway). Remember, the architecture is not the deliverable.
The business objects do not have any behavior in them, all of that is in the business logic layer (the anemic domain model anti-pattern), and this is not good object-oriented design.
Yes, the fact that my business objects don’t have the logic in them is an example of the anemic domain model pattern. You got me on this one. But really, how bad is this? Sure, you don’t know to look for GetOrdersForCustomer() in the CustomerLogic class or the OrderLogic class. But what’s the worst that can happen? Someone writes the code to fetch this stuff twice?
While I would love to have the business logic encapsulated in my business objects, I also don’t want to send all of the validation rules and server logic over to the client, and I don’t want to switch to DTOs and client entities just to resolve this problem. So I’ll live with the anemic domain model for now. Honestly, most of the complaints I’ve heard is that it’s annoying to have the logic in separate classes, but I haven’t heard anyone say that it flat out doesn’t work.
In summary…
Remember, what is our goal — to deliver a piece of software on time, that meets the business needs, and is maintainable. On my current project, we delivered on time (our first release was done a month ahead of time), we were able to spend more time writing code that meets a business need (vs. writing plumbing), and our code is quite maintainable in my opinion (18 different developers have worked on it, and we just wrote our 3000th unit test today). Obviously, I cannot take most of the credit for this because the people writing the code are the main reason for the success. But I like to think that the architecture that we’re using helped us do it.
Discuss!
The Todd tagged me so I guess that means I’m it.
How old were you when you started programming?
I don’t exactly remember, but I think I was around 9 or 10. We had an Apple IIe with a green screen monitor. I don’t remember how I discovered that you could write programs on it, but I did.
How did you get started in programming?
See above… that was when I was younger. Once I got out of grade school, I spent all my time playing sports or hanging out with friends so I didn’t spend that much time on the computer anymore. When I graduated high school, they told me I had to pick a major and computer science sounded better than anything else out there.
What was our first programming language?
BASIC! Lots of GOTO and GOSUB. Ick.
What was the first real program you wrote?
I had an internship in college and I worked for a small consulting firm. The main project was a commercial video editing program written in C++ called CineStream. It was pretty cool getting to work on software that actually sat on shelves, and $11 an hour is a lot when you’re a college student!
What languages have you used since you started programming?
BASIC, C++, Java, PHP, Perl, Javascript, C#
What was your first professional programming gig?
I worked for Thomson Tax and Accounting (back then it was called Creative Solutions), where we wrote tax and accounting software for accounting firms. I lived in the C++ world for a couple years and then got to make the jump to C#/.NET. It was a fun place to work… we had a basketball court in the parking lot, so we had leagues after work and played often during lunch. More workplaces need stuff like that.
If you knew then what you know now, would you have started programming?
Heck yeah! My one reservation about going into computer science in college is that I was afraid that I would end up in some cubicle for the rest of my life and not talk to anyone all day. Now I sit in a room with 12 other developers every day. Quite the opposite of what I was afraid of. Some people say that computer geeks are anti-social, awkward people. I guess it just depends on who you’re talking to, because the software developers I know are a pretty social bunch.
If there is one thing that you learned along the way that you would tell new developers, what would it be?
You can always learn something from anyone, so don’t put other people down or think you’re better than them because everyone can teach you something (sometimes, they’ll teach you what not to do!). Find other smart people and learn everything you can from them. Put others first before yourself. Don’t throw your people under the bus. A lot of this stuff really applies to life in general, not just life as a software developer.
What’s the most fun you’ve ever had programming?
My favorite feature I’ve ever had to work on was a custom report generator for a financial analysis application. Basically I had to write a custom spreadsheet control that could do Excel-style formulas and lots of custom financial analysis functions. It was quite a challenge. But I’d have to say right now I’m having as much fun as I ever have in my career because I sit in a room with 12 other developers who know what they’re doing and we have a lot of fun doing it (even if I don’t write much code anymore).
As I said in my last post, unit testing is hard, and it’s something that can be hard to learn.
I’ve been fortunate on my current project to be working with guys who introduced me to mocking frameworks (in our case, Rhino Mocks). I had heard of Rhino Mocks before, but I never really looked into it. I thought, if I’m not testing against a database, how is that a valid test of what will actually happen in the application?
I can think of a very simple example that I think will show the value of mocks. Let’s say that it’s your first day on a brand new, green field project where you’re writing a system that deals with claims. You create a simple Claim object and you want to write a unit test to make sure that your validation code checks to see if the ClaimNumber property has a value. Your tests (not using mocks) might look like this:
[TestMethod]
public void SaveClaimPositiveTest()
{
ClaimLogic logic = new ClaimLogic();
Claim c = new Claim();
c.ClaimNumber = "12345";
logic.Save(c);
Claim loadedClaim = logic.Get(c.Id);
Assert.IsTrue(loadedClaim != null);
Assert.IsTrue(loadedClaim.ClaimNumber == c.ClaimNumber);
}
[TestMethod]
public void SaveClaimNegativeTest()
{
ClaimLogic logic = new ClaimLogic();
Claim c = new Claim();
c.ClaimNumber = null;
try
{
logic.Save(c);
Assert.Fail("Expected exception was not thrown.");
}
catch (MyValidationException ex)
{
Assert.IsTrue(ex.Property == "ClaimNumber", "Validation for ClaimNumber did not occur.");
}
}
These are fairly simple tests that I’ve written many times in the past. What is wrong with these tests? Nothing right now. But that is all going to change.
What happens when someone adds a new property to the Claim object and adds a validation rule that says that the new field is required? My tests break. Now I have to go back and add more code to the beginning to create a valid Claim object that I can save.
What happens when someone adds some more logic to the Save method that kicks off some complicated workflow? Now my simple tests are getting a lot more complicated, and I’m going to have to deal with all kinds of side effects and errors just to test a really simple validation rule.
I’m also cluttering up the database with test data. So I need to write some cleanup code to delete the Claim object I inserted. Later on someone will add some foreign key relationships to the table, and I’ll have to come back add more code to clean up those related objects too. Someone might add some triggers too, and I’ll have to account for that. Those triggers might make it hard (or impossible) to clean up this test data.
These tests are also going to be pretty slow because they have to interact with the database. Yes, it’s a simple test, but what happens when I have 2000 of these tests? It’s going to take a long time to run them all!
All I wanted to do was verify whether my validation code was written correctly! Why do I have to actually try and save the object to the database to do this?
What happened on past projects where I wrote these kinds of tests was this: the unit tests take 5 minutes to run, so no one runs them when they check in. Eventually someone does something that breaks the test (usually harmless stuff like adding validation rules), but no one takes time to fix it. Before you know it, half of the unit tests are broken, but it’s getting close to your release date, so you don’t have time to fix them. Then, sometime after the release, someone will spend a week updating the tests so that they all work again.
This is where mocking comes in.
I’m not going to go in depth into mocking because there are plenty of people who have written in depth posts on how to do this. But the basic idea is that I am going to “mock out” external dependencies by telling the mocking framework what those external dependencies are going to do and return. In this case, I’ll mock out the code that would actually do the actual saving to the database because all I’m testing is the validation.
Yesterday I wrote a unit test for a method that returned a list of users in the application who were in certain security roles, and the list of users returned depended on the current user’s security role (if you’re in certain roles, I don’t want the list of users to include people in certain other roles). The list of users is stored in a database table, and the security roles are stored in Active Directory groups.
In this case, it would be pretty much impossible to test this without mocks. Think about what I have to do:
1) Test that the list of users returned will be correct depending on the user’s security role
2) Test that users in each security role will be returned at the proper time
The problem is that I don’t know what users are in the database and I don’t know what users are in each role in Active Directory. I don’t have a way to write code to insert users into Active Directory groups. Even if I could do that, I would then have to write code to insert a bunch of test users into the database (or even worse, write a test that would expect certain actual users to be in the database). Then I’d have to clean everything up when the test finished.
I don’t need a database or Active Directory to test my logic that would return the correct list of users. I can mock out the database by just using a list of User objects instead, and I can mock out Active Directory by just telling the mock framework what users to return for each role.
I wrote 16 tests around this section of code and they all run in about 5 seconds! I don’t have to insert any test code anywhere, I don’t have to clean it up, I don’t have to worry about people adding new validation rules, and I don’t have to worry about people adding other external code that will break my test (that wouldn’t otherwise have an effect on the code I was testing).
I will say this – mocking is not easy. It really is an art. It is not something that I learned overnight. Even though I understood the concept, I didn’t really get good at it until recently. But now that I understand it, I write much better unit tests, I write more unit tests, and I can do it pretty quickly. Test Driven Development (writing tests first) is also an art, and it’s something that I’m just starting to get into.
When I first started programming, I didn’t care about unit tests because I just wanted my programs to work. Sadly, in my four years of college, no one even mentioned a unit test. I didn’t even know what one was when I started my first job! Trying to reprogram yourself to work in a TDD way is hard because you’re breaking old habits that you’ve had since the day you first started programming.
Mocking and TDD will revolutionize the way that you write unit tests, but like I said, it can feel like you’re learning a foreign language. If you have someone that you know that is good at this stuff, beg them, plead with them, take them out to lunch, or do whatever else you have to do to get them to sit with you and pair program for a day or two and learn from them. You won’t want to go back!
Also, if you don’t have a continuous integration build that runs all of your unit tests every time you check in, get one. We’re using TFS 2008 and I created a CI build using TFS in less than 5 minutes using the TFS build definition wizard. Then make sure that you fix broken tests as soon as they break!
Happy mocking!
Unit testing is always a hot topic on every project. Usually in the form of, “Why should I write unit tests?” or “Do we have enough code coverage?”
This is a not an implication of anyone that I’ve worked with. Lots of people can write code, but writing unit tests is definitely a skill that takes awhile to learn. So if I come across people who don’t know how to write unit tests, I don’t get too bent out of shape because it’s takes time to get good at it.
Let’s go back to the original question — Why should I write unit tests?
How many of you have ever had to debug, maintain, or replace legacy software? I’m in the process of doing that right now. Now not all of the code that we’re replacing is bad, and some of it is still useful. The problem is that we can’t change any of it because we don’t know what our changes will break. That’s because we don’t know what the intentions of the original developers were, and we have no way of verifying whether our changes were successful (what is the acceptance criteria?). As a result, we end up having to rewrite code that otherwise might be perfectly good code because we have to make some minor change to it. Since we can’t modify it safely, we have to rewrite the code and everything that depends on it.
Legacy software is the obvious example. Let’s think about your own code.
On my project, there are 15 developers. 15 developers is a lot of people, and we’re stepping on each other toes all the time. Sure, I can write code and just step through it in the debugger and verify that it’s working, but what happens when someone writes code tomorrow that will break my code? How will they know that they broke it, and how will I know that they broke it? Heck, I break my own code all the time, how is someone else supposed to not break it?
Unit tests take time to write, but they will usually save time in the end. In many cases, you’ll find bugs in your code before it gets to QA, so QA doesn’t have to spend time testing it, writing up the bug, retesting it. Then when you have to change your code later on (or if someone else has to change it), the unit tests can tell you if you broke something, saving more QA time. Unit testing also makes you really think about your code and all of the possible edge cases. Sometimes it’s really hard (or impossible) to recreate an edge case when testing through the app and it’s much easier to recreate it in a unit test. Remember, if your code allows for a certain edge case to happen, you should write unit tests around it even if you can’t recreate that edge case when running your application. Just because you can’t do it now doesn’t mean that someone won’t try and use your code to do it later.
How many projects have you been on where you’ve had to make a big change just before the app when into production or while the app was in production? I’ve had to do this many times. Unit tests probably won’t catch everything that can go wrong in this situation, but it sure is nice being able to run 2000 tests to see what I might have broke.
The important thing to remember is that we should be writing software that is going to last as long as possible. This is really hard to keep in mind when you have a deadline looming, or you’re trying to figure out a particular problem.
Nothing lasts forever - business requirements change, programming languagues change, development teams change, everything changes. But it’s my responsibility to write software that will last as long as possible. I want my software to get replaced because changes in technology or the business environment allowed them to write something much better than what I wrote. I don’t want them to replace my software because they couldn’t maintain it.
I’ve used WCF on the last few projects that I’ve worked on. While WCF is pretty cool, it’s not easy, and knowing how to configure it can be tough since you can customize anything and everything.
If you are doing anything at all with WCF, J.D. Meier’s blog is a definite must read. He (and his team) have been posting tons of WCF guidance articles on how to configure WCF in lots of different situations. These articles have been invaluable for me and now I actually feel confident that I’m doing everything correctly with WCF.
Most of us have lots of pictures, music, and other stuff on our home computers that we can’t afford to lose.
We’ve all heard many times that we need to back our stuff up, and other people have posted about this before. Unfortunately I ignored all the warnings and was met with a “disk read error” when I booted up my laptop last week. Crap.
Luckily I had been backing up a lot of stuff to CDs, but I had slacked off over the last year. Most of the pictures that I really wanted I could get back from other people, so I didn’t lose too much.
(Side note: seeing “disk read error” has an upside — the wife turns to me and says, “I think we need to buy a new computer.” The Wife Acceptance Factor will never get any higher than that.)
I need something that is automated so that it doesn’t rely on me having to manually go and burn CDs, upload files, etc. because I will forget to do it. I need something that happens frequently because with a newborn in the house I will be taking lots of pictures, and I can’t afford to lose them. I didn’t want to back up to another computer in my house either (because I don’t want to maintain it, and to protect against some unlikely event like a house fire or someone breaking in and stealing everything). Here’s how it all works:
I downloaded and installed WinSCP, which is an FTP client that has a very powerful scripting language and has built-in functions to help you synchronize data. I downloaded the 4.1.3 beta because I needed some of the scripting capabilities that they added in the later releases.
Now that I have WinSCP installed, I had to write my script. Luckily the WinSCP scripting language has some pretty good documentation. I have two files, a batch file that I will run and a WinSCP script file that is called from the batch file. I am going to synchronize files from my machine to my web hosting provider (which I have to host my blog), which is running on Linux.
Here are the two files (I have them in my C:\autobackup directory):
winscp backup batch file.bat:
@echo off
:waitloop
echo Waiting for wireless.…
Ping jonkruger.com -n 2 |find /i "Request timed out" > nul
if %errorlevel% ==0 goto waitloop
:connected
rem Stall for time so that I'm sure that the wireless is connected
Ping 127.0.0.1 -n 20
@echo on
echo connected to wireless
"C:\program files\winscp\winscp.com" /console /script="c:\autobackup\winscp backup script.txt" /log="C:\autobackup\backup detailed log.txt" > "C:\autobackup\backup log.txt"
echo done
winscp backup script.txt:
# Automatically answer all prompts negatively not to stall
# the script on errors
option batch on
# Disable overwrite confirmations that conflict with the previous
option confirm off
# Exclude files that I don't care about
option exclude "*.db; *.ini; *.tmp;"
# Connect to the server (replace with your username, password, domain)
open username:password@mydomain.com
# Do the work
synchronize remote -delete -mirror "C:\Documents and Settings\all users\Documents\My Pictures" "/pictures-backup"
# Close and exit
close
exit
Note that in winscp backup batch file.bat, the stuff at the top of the file is checking to make sure that I’ve connected to my wireless network before I try and connect to the server.
You can read more about the options for the WinSCP “synchronize” script command here. Basically what I’m doing is synchronizing the remote FTP server to have the same files that I have on my local machine. I’m only doing the synchronization one way (meaning that changes on the remote FTP server will not be synched back to my local machine), but WinSCP will allow you do the two-way synchronization if you want to.
Now I set up a scheduled task in Windows to run my “winscp backup batch file.bat” file. I checked the box that says “Wake the computer to run this task”.
Just like that, all of my pictures are backed up every night to a remote server, without any interaction from me, and I can see the results in a log file. I can easily update my script file to back up other directories too.
There are probably lots of other backup solutions out there, including Mozy.com, which allows you to back up 2 GB worth of data for free, or unlimited data for $4.95 a month. Web sites like this are probably worth looking into… I created my own solution because I already had the web hosting space available and WinSCP made it pretty easy.
So now that I’ve done all the work for you, you have no excuse! Don’t wait to back up your stuff or you might end up with nothing left to back up!