Test driven development at Transloadit
Posted on 28/10/10 by Felix Geisendörfer
You have probably heard a few talks or read a few articles where test driven development is depicted as a magic unicorn that watches over your software and makes everybody happy. Well, about 18.000 lines of "magic unicorn" code later, I'd like to put things into perspective.
The truth is, test driven development is a huge pain in the ass. Writing those damn tests all the time takes a huge amount of discipline, and it doesn't really get a whole lot easier with time.
But do you know what sucks more? The liability that comes without those tests.
Let me clarify. I'm not trying to tell you that you should practice test driven development. What I will try to do however, is to give you a realistic idea about the work it takes and the kind of benefits you get from it.
At this point I basically think of test driven development as an insurance policy. It guarantees that your software will have a certain pre-defined quality mostly measured by the amount of bugs and the risks of changing the software. It can also reduce the costs of your main developer getting run over by a bus, or the same thing happening to the API of the platform you are building on.
However, there are also millions of people on this planet that live without any insurances whatsoever, so clearly this is not the only way of getting from point A to B. It all depends on your liability and how you can / want to deal with it.
At Transloadit 100% of our code is developed test driven. The main reason for that is that we are building on node.js which still is a very risky technology at this point and probably our biggest liability and advantage at the same time. Another big risk are third party tools such as ffmpeg and image magick that are insanely dangerous to upgrade since they love to change APIs and behavior all the time. And last but not least, some of the stuff we are doing is rather complex.
Now let me explain how test driven development lets us deal with those risks. First of all, new features always start out as a system test. Each system test boots up the entire REST service, sends an actual HTTP request to it, and evaluates the response as well as the files that were created in the process. Files are verified by looking at their meta data, as well as performing visual diffs on images against expected fixtures. For videos we are using thumbnails to do the visual comparisons.
Once that test is written (and failing), we start writing unit tests. Let me explain what we mean by "unit". A unit to us is the tiniest atomic piece of functionality we can possibly test. As soon as there is a function call, creation of a class, or even a callback closure - we stub it. Everything that can be isolated and tested separately is handled by itself. As you can imagine, that's the part where the pain and discipline come in.
However, there is a certain undeniable beauty resulting from it. Writing those tests feels like proofing mathematical theorems. Every step is just small enough to be broken down to raw logic. Everything builds upon the previous step. There is no need to test all numbers in between 1 - 1000000, unless there is a logical reason for expecting a different result.
Of course there is a good chance of you going crazy in the process and losing track of the original goal. That's where the system test comes to rescue. Whenever you don't know what needs to be done next, you simply run the system test and it will tell you what's missing.
Is this process perfect? No. We have had a few bugs here and there, but I can still count them on two hands. Last weekend we pushed a rather insane change into production: We replaced our underlaying MySql driver (we were using that of PHP, don't ask how that worked) with node-mysql. I've been working on node-mysql with the same care and TDD-masochism as we have on transloadit, so the update went without a single hickup (so far). Not bad for a change that involved 6000 lines of code.
And that's it. Our service is not necessarily better than our competitors' because we use TDD. However, our risk of breaking things is much smaller, and we can perform heart-surgeries like this without breaking a sweat. This gives us the confidence to charge money, and the ability to spend the rest of our limited resources (we are bootstrapping) on innovative new features rather than dealing with the shortcomings of our software all the time.
So, is TDD right for you? It depends. Writing code the TDD way requires a ton of discipline, and the unicorns aren't always around to make you feel better. So make sure you have good reasons to do it. Sometimes a few system tests can get you 80% of the results for 20% of the work. Other times you need the remaining 20%. Also keep in mind that we're lucky at Transloadit for using a flexible language and most of our interfaces are JSON. This makes testing cheaper for us than it might be for you.
--fg
PS: If I find some more time, I'd love to go into more details of our node.js unit testing. Leave a comment if you're interested. Meanwhile make sure to check out the tests for node-mysql, they are done exactly like our internal tests.
Did you like this blog post? If so, please consider subscribing to the Blog RSS feed.
You can skip to the end and add a comment.
You're not kidding about it requiring discipline! It can get so boring writing out all the unit tests for each possible factor. Oh and if a bug is later discovered it's so tempting to just go fix it instead of first figuring out where your tests missed that bug and fix those first so that the bug never gets reintroduced.
But as you said if you keep at it you can confidently say your code is solid.
Rob Mills: Yeah, when the bug is really simple then adding the test is annoying. However, if the bug is difficult to reproduce, the test suite makes it so easy to try out different ways to re-create the bug that it's heaven : ).
Also about writing the test for each possible factor: TDD makes that a lot harder. This has caused me to write less flexible functions. That is, the input signature is well-defined and there is very little magic to handle weird input.
I'd be interested to know how much time you think it ultimately added or saved from your development time?
Thanks for the writeup, Felix. It's good to finally hear someone admit how hard TDD actually is, discipline-wise.
It's also good to see that this article isn't too technical and thus restricted to the node.js domain. A lot of people with a tech background, esp. in the Ruby/JS communities, already know how important tests are. But we need to spread the love for other communities as well.
I'm looking at you, PHP!
Nice post. Would love to hear about Node.JS unit testing.
BTW: Do you happen to be in Cologne sometime? Maybe you want to join a Cologne.JS meetup?
Oliver Beattie: I think it took as twice the time of writing the initial version, but in the few months we've been in production we have already re-gained that time in terms of not having to fix bugs, and fixing those that we do find very quickly. Of course adding new features means investing more time again, overall I thin it's a break-even with a more solid product as a result.
André Wendt: We couldn't have done the same TDD with PHP. JavaScript makes testing just a lot easier.
Sometimes it seems that if you don't do TDD then you don't test at all!!!
Clearly thats not true, you can have the same test or even more without doing TDD. It would be interesting for all the people that practice TDD to explain why doing the test before is "better" than doing them after, assuming in both cases that you test your code. For a reason or other I rarely see this kind of explanations.
To paraphrase Kent Beck (I forgot where he writes that): "Every test you write is an investment and every test you leave out is a risk." Decide for yourself which risks you want to take and which investments are worthwhile.
Excellent post, matches a lot of our experience. I too would love to hear about Node.JS unit testing... please write it up!
It might be important to mention that going down the TDD route also encourages writing "testable code" which frequently seems to translate as "properly factored code".
In some work I've been involved in, ignoring testing (both high-level and low-level) resulted in an untestable mess of *working* code that needed a significant overhaul to get to the point where we could write what you called a "system test". Even at that point, due to the amount of monkey patching and mocking it seemed like we wrote code, then mocked out everything, then tested our mocks, which was definitely far from ideal.
The worst part was that we ended up relying on manual testing before every release because the work required to automated high-level tests was so enormous that it was more effective in the short-term to stick two people on QA-duty, which led to a big bottleneck for every single release.
@Dumas -- it may be a purely psychological thing, but I find that writing the tests before is better. First, it's mentally easier because I'm exploring functionality, and how the API will work by actually building the code which *uses* the stuff I'm building. Interestingly, this often makes the API more sensible; this is what the "TDD is Test Driven *Design*" people are talking about, so I think that that's a real, not psychological, gain. Again, psychologically, writing tests after is painful because of the "Hey, I've solved that problem, why go over it again?" thinking starts to creep in. Another real gain is that I find I'm thinking more of what the various corner-cases are, as I'm developing the code; if I have a test framework already set up, it's nearly trivial to add a test that shows that case. I've seen people writing about TDD say that they had a "WOW, I'm glad I thought of that case *now*!" revelation, and I can relate to some extent. So it's a balance between some genuinely psychological things and some real things, that honestly may have a lot to do with how I think. YMMV.
Dumas,
For me writing the test before writing the code helps clarify exactly what my code should be doing.
In the past when I've gone back to write test cases after writing the code there's a big temptation to look at what the code is doing and base the tests off of that which leaves a lot of room to leave out important tests.
That's right Rob, for me this is central in TDD, the "when" you do the tests "guides" your Code. I always test my code, but I prefer coding a little and running a little test in the command line, a real test without mocks stub or anything, I like my code on-the-wild failing! I like the Red in Real life -not on production of course- Then after some reflection I can write a more rigid test. I like to play with my methods because doing them before prescribes a corset to my code I dont want to be unflexible in the creative part, what if I want to change parameters, signature or things like that? But thats for me, it could be different for other person, is irresponsible not to test, not not doing TDD.
I am not using the pure TDD principle, but I am applying a very effective (but also somewhat heavy) regression testing approach to my new model-driven IDE, AtomWeaver (http://www.atomweaver.com for the curious).
Like you said, it ensures a minimum level of quality of the product before release. Because it exercises the application at the GUI level, it allows me to save hundreds of hours in human testing.
And yes, it's a pain to write the test script, and most of the time I feel I'm wasting some time. That's why I spent even more time creating a generator for the test script (yes, you can generate a test script!). Today, the regressing test script has 100K lines, roughly 40% of the tested application's code base. And it still only tests basic things. Why such a large script? Because I test the app's *complete* run-time data set for compliance after each and every simulated user action. That results in a lot of lines, and slow execution (everything is logged).
But at the end of the day, and everytime I run the test script and catch a bug, I say, "man, this regression testing thing is the best time investment I could have done!"....
Hi, Felix,
Thanks for the balanced article. It is a rarity.
The only problem some people have with TDD is that it can deflect junior programmers from fundamentals, leaving them believing the TDD is fundamental when it's not; it's not even a means to an end, but it is a means to a means to an end.
You wrote, "Our service is not necessarily better than our competitors' because we use TDD. However, our risk of breaking things is much smaller, and we can perform heart-surgeries like this without breaking a sweat."
But your risk of breaking things is not much smaller because you use TDD: your risk of breaking things is much smaller because you have loosely-coupled your system.
It's the loose-coupling that the benefit; loose-coupling is a means to creating software that's easy to change; TDD is just a means to promote loose-coupling.
Note, also, that loose-coupling will not help you discover faults when you introduce new code, and neither will TDD: the tests themselves will show you when you've broken something, whether you wrote them before or after you write your code that fails. (I know you didn't claim otherwise, but this is another common misconception and I just wanted to get it off my chest.)
So far I've only one, irrefutable fact about TDD: TDD guarantees that the code you haven't written yet doesn't exist.
I can't argue with that.
Regards,
Murbo.
Really nice article!
I like the feeling that TDD gives of having some solid assumptions to build a software upon. I think that the discipline required for TDD is a price I'm willing to pay for, and the outcomes are really valuable.
TDD makes me sleep better at night ;)
Loving to hear some details about node.js unit tests!
Briliant post!
I like to use TDD and I know the power of it). Unfortunately TDD is not so well spreaded yet, many people really much tries to avoid it.
Great post!
For me TDD is mostly about insurance and confidence. I have similar thoughts:
1. use high level tests and very low level tests, don't try to cover all the spectrum - high level tests are enough and unit tests will help you with the details.
2. Try to cover most of code with as little test code as possible.
I would love to read more about node.js testing best practices!
You've given us a great description of the discipline and value of TDD.
"There is no reason to test all numbers...unless there is a logical reason for expecting a different result."
This is one of the benefits that often stays hidden: When you use tests to guide every bit of changing behavior, you test every corner of the code in isolation. You hit both sides of every boundary, and you break up any possible combinatorial effect into decoupled units of behavior. The tests are then able to pinpoint problems without resorting to debugger-marathons, time-consuming end-to-end scenarios, or even test heuristics (e.g., pair-wise).
You go on to point out the value of those overarching functional tests, and those same test heuristics: They provide targets for the developers. Testers play an important role here, because they have the experience with such heuristics and (hopefully) a familiarity with the product and the team's potential blind-spots. Acceptance-tests/story-tests/functional-tests need not test everything, but serve as examples and guides. Testers need not spend all their time gaining coverage with these tests, because they know the unit tests will cover all angles. Thus, they have more time for creative sessions of exploratory testing.
Okay, now I'm starting to extrapolate from your facts. That's how I see numerous teams blending these techniques successfully. At the core: Disciplined TDD. Don't you ultimately find it more rewarding than development without microtesting?
Nice article!
@Jacek Becela
+1 Insightful
After reading some of the other comments, I was inspired to describe one of my experiences with TDD:
http://bit.ly/apFVxR ("I Have Written Bug-Free Code Without TDD")
Thanks again, Felix!
Rob
Totally interested in nodejs unit testing as well as nodejs web development in general.
nice post. keep 'em coming.
This is a pretty nice write up Felix. As an outsider who's been trying to force myself into TDD, I've pretty much developed the same thoughts about pains and benefits. I would like to hear more about your process. For example, a few concrete questions:
1) Mocking of objects is easy in js. But how do you do mocking of classes and functions?
2) How do you make sure your mocks stay in line with changes to the actual implementation?
3) Do you write tests for error handling? How do you approach that.
As for the discipline part, that's easily the biggest barrier. I write code at work and we don't have a TDD process. I want to get into it for my open source stuff. But I have limited time. Most of the time I can't convince myself to spend what little time I have writing more tests. I guess you have to weight it against potential gain.
Nice article! I'm curious: how do you perform the visual diffs on images? Did you use the Libpuzzle Extension for the diffs?
Regards,
Sebastian
Wow, this is just the best post about TDD I've ever seen, thank you Felix! I'm primarily a Ruby developer and unfortunately Ruby community is bunch of TDD fanatics (even though most of them doesn't actually have a bloody idea how to write good tests). In Ruby if you don't love TDD and don't write tests first, then you're weird or you very likely don't even get the job or contract. Really, it's ridiculous. I completely agree with your point of view, kudos for being objective about it! Debuggable is becoming one of my most favourite blogs BTW :)
Oh BTW I wrote a tiny BDD framework for Node.js, based on already existing assert library from Node.js itself etc, you might want to check it: http://github.com/botanicus/minitest.js It's created with async testing in mind, so you have to call test.finish() manually in order to check that the test really finished - in case we'd have some assertions in callback and the callback wouldn't be called at all. Everything's described in its README.
I've been learning Rails now for the better part of a half a year and TDD is a big part of it. What I've found out is that with my background of PHP and Java, I've mostly not used that method of development and with TDD, I seem to spend most of my time writing tests, more than anything else :). Which sucks, but I totally see the benefits to it.
I'm a freelancer, as well as bootstrapping a company, so time and resources are limited. I've found it TDD to be useful for certain apps and not so much needed for others. I'm sure everyone would agree on that. This has been a great article though and I was going to write up a similar one, in my spare time :). Thanks so much
Thanks for an even-handed assessment of TDD. It seems like your team made a decision that makes a lot of sense, while being completely pragmatic about the tradeoffs. I've found that too many people who push TDD are completely unrealistic about the time costs when taken in context.
Thank you for writing this. The first two paragraphs expressed the exact feelings I was trying to communicate whenever talking about TDD to someone new to its practice, but couldn't find the correct way to put it into perspective.
Kudos to you.
One thing I am curious about, which doesn't seem to have been mentioned in the article and the comments so far, is your choice of the test framework.
Seeing that there are several test frameworks available for node.js, such as expresso and vows, what made you decide to use a custom solution?
Preventing dependencies on external libraries? Easier to adjust the tests to suit your needs?
I'm just curious, as I am currently looking into the available options myself and it seems a lot of the node libraries are using their own custom solution instead of one of the available frameworks.
Hendrik: We are actually using node-gently which I've written: http://github.com/felixge/node-gently
I would not recommend it at this point, as I'm in the process of coming up with a more comprehensive module. That being said, I have used it for pretty much everything we do with transloadit, so it's certainly working. However, it only covers the very basics of stubbing a method and returning fake results for test purposes.
This post is too old. We do not allow comments here anymore in order to fight spam. If you have real feedback or questions for the post, please contact us.
Really nice post. Pretty much what I wanted to know about TDD before trying it.
This kind of high level advices is what I needed after watching someone giving a talk about the beauty of TDD.
Thanks a lot!!