A Wrinkle in Time

You've built a prototype, everything is going great. All your dates and times look great, they load and store correctly, everything is spiffy. You have your buddy give it a whirl, and it works great for them too. Then you have a friend in Curaçao test it, and they complain that all the times are wrong - time zones strike again!

But, you've got this covered. You just add an offset to every stored date/time, so you know the origin time zone, and then you get the user's time zone, and voila! You can correct for time zones! Everything is going great, summer turns to fall, the leaves change, the clocks change, and it all falls apart again. Now you're storing dates in various time zones, without DST information, you're adjusting them to the user's time zone, trying to account for DST, trying to find a spot here or there where you forgot to account for offsets...

Don't fall into this trap. UTC is always the answer. It is effectively time-zone-less, as it has an offset of zero and does not observe daylight savings time. It's reliable, it's universal, it's always there when you need it, and you can always convert it to any time you need. Storing a date/time with time zone information is like telling someone your age by giving your birthday and today's date - you're dealing with additional data and additional processing with zero benefit.

When starting a project, you're going to be better off storing all dates as UTC from the get-go; it'll save you innumerable headaches later on. I think it is atrocious that .NET defaults to system-local time for dates; one of the few areas where I think Java has a clearly better design. .NET's date handling in general is a mess, but simply defaulting to local time when you call DateTime.Now encourages developers to exercise bad practices; the exact opposite of the stated goals of the platform, which is to make sure that the easy thing and the correct thing are, in fact, the same thing.

On a vaguely related note, I've found a (in my opinion) rather elegant solution for providing localized date/time data on a website, and it's all wrapped up in a tiny Gist for your use: https://gist.github.com/aprice/7846212

This simple jQuery script goes through elements with a data attribute providing a timestamp in UTC, and replaces the contents (which can be the formatted date in UTC, as a placeholder) with the date/time information in the user's local time zone and localized date/time format. You don't have to ask the user their time zone or date format.

Unfortunately it looks like most browsers don't take into account customized date/time formatting settings; for example, on my computer, I have the date format as yyyy-mm-dd, but Chrome still renders the standard US format of mm/dd/YYYY. However, I think this is a relatively small downside, especially considering that getting around this requires allowing users to customize the date format, complete with UI and storage mechanism for doing so.

On Code Comments

I've been seeing a lot of posts lately on code comments; it's a debate that's raged on for ages and will continue to do so, but for some reason it's been popping up in my feeds more than usual the last few days. What I find odd is that all of the posts generally take on the same basic format: "on the gradient of too many to too few comments, you should aim for this balance, in this way, don't use this type of comments, make your code self-documenting." The reasoning is fairly consistent as well: comments get stale, or don't add value, or may lead developers astray if they don't accurately represent the code.

And therein lies the rub: they shouldn't be representing the code at all. Code - clean, self-documenting code - represents itself. It doesn't need a plain-text representative to speak on its behalf unless it's poorly written in the first place.

It may sound like I'm simply suggesting aiming for the "fewer comments" end of the spectrum, but I'm not; there's still an entity that may occasionally need representation in plain text: the developer. Comments are an excellent way to describe intent, which just so happens to take a lot longer to go stale, and is often the missing piece of the puzzle when trying to grok some obscure or obtuse section of code. The code is the content; the comments are the author's footnotes, the director's commentary.

Well-written code doesn't need comments to say what it's doing - which is just as well since, as so many others have pointed out, those comments are highly likely to wind up out-of-sync with what the code is actually doing. However, sometimes - not always, maybe even not often, but sometimes - code needs comments to explain why it's doing whatever it's doing. Sure, you're incrementing Frobulator.Foo, and everybody is familiar with the Frobulator and everybody knows why Foo is important and anyone looking at the code can plainly see you're trying to increment it. But why are you incrementing it? Why are you incrementing it the way you're doing it in this case? What is the intent, separate from its execution? That's where comments can provide value.

As a side note (no pun intended), I hope we can all agree that doc comments are a separate beast entirely here. Doc comments provide meta data that can be used by source code analyzers, prediction/suggestion/auto-completion engines, API documentation generators, and the like; they provide value through some technical mechanism and are generally intended for reading somewhere else, not for reading them in the source code itself. Because of this I consider doc comments to be a completely separate entity, that just happen to be encoded in comment syntax.

My feelings on doc comments are mixed; generally speaking, I think they're an excellent tool and should be widely used to document any public API. However, there are few things in the world more frustrating that looking up the documentation for a method you don't understand, only to find that the doc comments are there but blank (probably generated or templated), or are there but so out of date that they're missing parameters or the types are wrong. This is the kind of thing that can have developers flipping desks at two in the morning when they're trying to get something done.

Optimizing Entity Framework Using View-Backed Entities

I was profiling a Web application built on Entity Framework 6 and MVC 5, using the excellent Glimpse. I found that a page with three lists of five entities each was causing over a hundred query executions, eventually loading a huge object graph with hundreds of entities. I could eliminate the round trips using Include(), but that still left me loading way too much data when all I needed was aggregate/summary data.

The problem was that the aggregates I needed were complex and involved calculated properties, some of which were based on aggregates of navigation collection properties: a parent had sums of its children's properties, which in turn had sums of their children's properties, and in some cases parents had properties that were calculated partly based on aggregates of children's properties. You can see how this quickly spun out of control.

My requirements were that the solution had to perform better, at returning the same data, while allowing me to use standard entity framework, code first, with migrations. My solution was to calculate this data on the server side, using entities backed by views that did the joining, grouping, and aggregation. I also found a neat trick for backward-compatible View releases:

IF NOT EXISTS (SELECT Table_Name FROM INFORMATION_SCHEMA.VIEWS WHERE Table_Name = 'MyView')
    EXEC sp_executesql N'create view [dbo].[MyView] as select test = 1'
GO

ALTER VIEW [dbo].[MyView] AS
SELECT ...

It's effectively upsert for views - it's safe to run whether or not the view already exists, doesn't ever drop the view if it does exist (leaving no period where a missing view might cause an error), and it doesn't require keeping separate create and alter scripts in sync when changes are made.

I then created the entities that would represent the views, using unit tests to ensure that the properties now calculated on the server matched expected values the same way that the original, app-calculated properties did. Creating entities backed by views is fairly straightforward; they behave just like tables, but obviously can't be modified - I made the property setters protected to enforce this at compile time. Because my View includes an entry for every "real" entity, any query against the entity type can be cast to the View-backed type and it will pull full statistics (there is no possibility of an entity existing in the base table but not in the view).

Next I had to create a one to one association between the now bare entity type and the view type holding the aggregate statistics. The only ID I had for the view was the ID of the raw entity it was connected to. This turned out to be easier said than done - entity framework expects that, in a one to one relationship, it will be managing the ID at one end of the relationship; in my case, the ID's at both ends were DB-generated, even though they were guaranteed to match (since the ID in the view was pulled directly from the ID in the entity table).

I ended up abandoning the one-to-one mapping idea after a couple days' struggle, instead opting to map the statistics objects as subclasses of the real types in a table per type structure. This wound up being relatively easy to accomplish - I added a table attribute to the sub type, giving the name of the view, and it was off to the races. I went through updating references to the statistics throughout LINQ queries, views, and unit tests. The unit and integration tests proved very helpful in validating the output of the views and offering confidence in the changes.

I then ran my benchmarks again and found that pages that had required over a hundred queries to generate now used only ten to twenty, and were rendering in half to a third the time - a one to two hundred percent improvement, using views designed purely to mimic the existing functionality - I hadn't even gone about optimizing them for performance yet!

After benchmarking, it looks even better (times are in milliseconds, min/avg/max):

	EF + LINQ	EF + Views
3 lists of 5 entities (3 types)	360/785/1675	60/105/675
2 lists of 6 entities (1 type)	325/790/1935	90/140/740
1 entity's details + 1 list of 50 entities	465/975/2685	90/140/650

These tests were conducted by running Apache JMeter on my own machine against the application running on Windows Azure, across a sampling of 500 requests per page per run. That's a phenomenal 450 to 650 percent improvement across the board on the most intensive pages in the application, and has them all responding to 100% of requests in under 1 second. The performance gap will only widen as data sets grow; using views will make the scaling much more linear.

I'm very pleased with the performance improvement I've gotten. Calculating fields on the app side works for prototyping, but it just can't meet the efficiency requirements of a production application. View-backed entities came to the rescue in a big way. Give it a try!

My Personal Project Workflow/Toolset

I do a lot of side projects, and my personal workflow and tooling is something that's constantly evolving. Right now, it looks something like this:

• Prognosticator for tracking features/improvements, measuring the iceberg, and tracking progress
• WorkFlowy for tracking non-development tasks (the most recent addition to the toolset)
• Trac for project documentation, and theoretically for defect tracking, though I've not been good about entering defects in Trac recently; it doesn't seem worth the effort on a one-person project, though with multiple people I think it would be a must
• Trello for cross-cutting all the above and indicating what's next/in progress/recently completed, and for quickly jotting down ideas/defects. Most of the defect tracking actually goes in here on one-man projects right now. This is a lot of duplication and the main source of waste in my current process.
• Bitbucket for source control (I also use Atlassian's excellent SourceTree as a Git/Hg client.)

It's been working well for me, the only issue I have is duplication between the tools, and failing to consistently use Trac for defect tracking. What keeps me in Trello is how quick and easy it is to add items to it, and the fact that I'm using it as a catch-all - I can put a defect or an idea or a task into it in a couple of seconds; I just have to replicate it to the appropriate place later, which is the problem.

I think the issue boils down to being torn between having a centralized repository for "stuff to be done" (Trello) and having dedicated repositories catered to each type of thing to be done (Prognosticator, Trac, and WorkFlowy); and convenience. Trello is excellent for jotting something down quickly, but lacks the additional specific utility of the other tools for specific purposes.

I think what I'll end up doing is creating a "whiteboard" list in WorkFlowy, and using that instead of Trello to jot down quick notes when I don't have the time to use the individual tools; then I can copy from there to the other tools when I need to. That will allow me to cut Trello down to basically being a Kanban board.

Simple Entity Framework Model Structure

I'll say right up front, I don't have a lot of experience with Entity Framework, and this could either be a well-known solution or a completely foolish one. All I know is that, so far, it has worked extremely well for my purposes.

Coming from the Java world, I'm used to using DAO's to serve as an abstraction layer between the controllers and the database, with the basic CRUD methods, plus additional methods for any specific queries needed for a given entity type.

Conveniently, entity framework provides a fairly complete DAO in the form of DbSet. DbSet is very easy to work with, and provides full CRUD functionality, and acts as a proxy for more complex queries. I wanted to keep queries out of my logic, however, and in the model.

Looking at it, I didn't want to have to write an entire wrapper for DbSet, and subclassing it seemed like asking for trouble. That's when it occurred to me to use extension methods for queries. It turns out you can define extension types against a generic type with a type argument specified (e.g. this IEnumerable). This not only allowed me to abstract out the queries and keep them in the model, without having to wrap or subclass anything; but by defining the extensions on IEnumerable instead of DbSet, I have access to my queries on any collection of the appropriate entity type, not just DbSet. I can then chain my custom queries in a very intuitive and fluid way, keeping all of the code clean, simple, and separate.

For example, I have a table of tags. I've created extension methods on IEnumerable to filter to tags used by a given user, and to filter by tags starting with a given string. I can chain these to get tags used by a given user and starting with a given string. I can also use these queries on the list of tags associated with another entity, as IList implements IEnumerable, and thus inherits my query extension methods.

I don't know if this is the best way - or even a good way - but it's worked for me so far. I do see some possible shortcomings; mainly, the extensions don't have access to the context, so they can't query any other DbSets, only the collection it's called against. This means that only explicit relationships can be queried against, which hasn't been a roadblock so far in my (admittedly simple) application. I'm not sure this is really a drawback though - you can still add a parameter to pass in an IEnumerable to query against, which again offers the flexibility to pass a DbSet or anything else.

JComboBox with Custom ComboBoxModel Not Updating Value on setSelectedItem()

I wrestled with this issue for some time before figuring out the cause, so I hope this helps someone out there. I had a JComboBox with a custom ComboBoxModel. Everything was working fine, except a call to setSelectedItem would do everything it should (fire events, update the selected item property) except it wasn't updating the displayed value in the JComboBox itself. Clicking the drop-down would even show the correct item selected, and getSelectedItem() returned the correct result; it was just the box itself that was wrong.

Apparently the displayed value in the JComboBox isn't based on getSelectedItem(), but rather on the top item in the list. I don't know how or why this is, or if it's due to some intricacies of my GUI code, but bringing the selected item to the top of the ComboBoxModel's item list when calling setSelectedItem fixed the issue. Go figure.

If anyone has any insight into what causes this, please drop a comment!

Building a Foundation

It's been said that pharmaceutical companies produce drugs for pennies per pill - except the first pill, which costs millions. Things aren't so different in the land of software development: the first usage of some new functionality might take hours, building the foundation and related pieces. But it could be re-used a hundred times trivially, and usually expanded or modified with little effort as well (assuming it was well-written to start with).

This is precisely what you should be aiming for: take the time to build a foundation that will turn complex tasks into trivial ones as you progress. This is the main purpose behind design concepts like the single responsibility principle, the Hollywood principle, encapsulation, DRY, and so on.

This isn't to be confused with big upfront design; in face, it's especially important to keep these concepts in mind in an agile process, where you're building the architecture as you go. It can be tempting to just hack together what you need at the moment. That's exactly what you should be doing for a prototype, but not for real development. For lasting functionality, you should assemble a foundation to support the functionality you're adding now, and similar functionality in the future.

It can be difficult to balance this against YAGNI - you don't want to build what you don't need, but you want to build what you do need in such a way that it will be reusable. You want to save yourself time in the future, without wasting time now.

To achieve a perfect balance would require an extraordinary fortune teller, of course. Experience will help you get better at determining what foundation will be helpful, though. The more experience you have and the more projects you work on, the better sense you'll have of what can be done now to help out future you.

The Importance of Logging

Add more logging. I'm serious.

Logging is what separates an impossible bug report from an easy one. Logging lets you replace comments with functionality. I'd even go so far as to say good logging separates good developers from great ones.

Try this: replace your inline comments with equivalent logging statements. Run your program and tail the log file. Suddenly, you don't need a step wise debugger for the vast majority of situations, because you can see, in the log, exactly what the program is doing, what execution path it's taking, where in the source where each logging statement is coming from, and where execution stopped in the event of a crash.

My general development process focuses on clean, readable, maintainable, refactorable, self-documenting code. The process is roughly like this:

1. Block out the overall process, step by step, in comments.
2. Any complex step (more than five or ten lines of code), replace the comment with a clearly-named method or function call, and create a stub method/function.
3. Replace comments with equivalent logging statements.
4. Implement functionality.
• Give all functions, methods, classes, parameters, properties, and variables clear, concise names, so that the code ends up in some semblance of readable English.
• Use thorough sanity checking, by means of assertions or simple if blocks. When using if blocks, include logging for any failed checks, including what was expected and what was found. These should be warnings.
• Include logging in any error/exception handling code. These should be errors if recoverable, or fatal if not. This is all too often the only logging a developer includes!
5. Replace inline comments with equivalent logging statements. These should be debug or info/trace level; major section starts should be higher level, while mid-process statements should be lower level.
6. Add logging statements to the start of each method/function. These should also be debug or info/trace level. Use higher-level logging statements for higher-level procedures, and lower-level logging statements for more deeply-nested calls.
7. For long-running or resource-intensive processes, particularly long loops, add logging statements at regular intervals to provide progress and resource utilization details.

Make good use of logging levels! Production systems should only output warnings and higher by default, but it should always be possible to enable deeper logging in order to troubleshoot any issues that arise. However, keep the defaults in mind, and ensure that any logging you have in place to catch defects will provide enough information in the production logs to at least begin an investigation.

Your logging messages should be crafted with dual purpose in mind: first, to provide useful, meaningful outputs to the log files during execution (obviously), but also to provide useful, meaningful information to a developer reading the source - i.e., the same purpose served by comments. After a short time with this method you'll find it's very easy to craft a message that serves both purposes well.

Good logging is especially useful in an agile environment employing fast iteration and/or continuous integration. It may not be obvious why at first, but all the advantages of good logging (self-documenting code, ease of maintenance, transparency in execution) do a lot to facilitate agile development by making code easier to work with and easier to troubleshoot.

But wait, there's more! Good logging also makes it a lot easier for new developers to get up to speed on a project. Instead of slogging through code, developers can execute the program with full logging, and see exactly how it runs. They can then review the source code, using the logging statements as waypoints, to see exactly how the code relates to the execution.

If you need a tool for tailing log files, allow me a shameless plug: try out my free log monitor, Rogue Informant. It's been in development for several years now, it's stable, it's cross-platform, and it's completely free to use privately or commercially. It allows you to monitor multiple logs at once, filter and search logs, and float a log monitoring window on top of other applications, to make it easier to watch the log while using the program to see exactly what's going on behind the scenes.Give it a try, and if you find any issues or have feature suggestions, feel free to let me know!

Sanity Checks: Assumptions and Expectations

Assertions and unit tests are all well and good, but they're too narrow-minded in my eyes. Unit tests are great for, well, testing small units of code to ensure they meet the basic requirements of a software contract - maybe a couple of typical cases, a couple of edge cases, and then additional cases as bugs arise and new test cases are created for them. No matter how many cases you create, however, you'll never have a test case for every possible scenario.

Assertions are excellent for testing in-situ; you can ensure that unacceptable values aren't given to or by a piece of code, even in production (though there is a performance penalty to enabling assertions in production, of course.) I think assertions are excellent, but not specific enough: any assertion that fails is automatically a fatal error, which is great, unless it's not really a fatal error.

That's where the concept of assumptions and expectations come in. What assertions and unit tests really do is test assumptions and expectations. A unit test says "does this code behave correctly when given this data, all assumptions considered?" An assertion says "this code assumes this thing, and will not behave correctly if it gets another, so throw an error."

When documenting an API, it's important to document assumptions and expectations, so users of the API know how to work with your code. Before I go any further, let me define what I mean by these very similar terms: to me, code that assumes something operates as if its assumptions are correct, and will likely fail if its assumptions turn out to be incorrect. Code that expects something operates as if its expectations are met, but will likely still operate correctly even if they aren't. It's not guaranteed to work, or guaranteed to fail; it's likely to work, but someone should probably know about it and look into it.

Therein lies the rub: these are basically two types of assertions, one fatal, one not. What we need is an assertion framework that allows for warning-level assertion failures. What's more, we need an assertion framework that is performant enough to be regularly enabled in production.

So, any code that's happily humming along in production, that says:

Assume.that(percentage).isBetween(0,100);

will fail immediately if percentage is outside those bounds. It's assuming that percentage is between zero or one hundred, and if it assumes wrong, it will likely fail. Since it's always better to fail fast, any case where percentage is outside that range should trigger a fatal error - preferably even if it's running in production.

On the other hand, code that says:

Expect.that(numRows).isLessThan(1000);

will trigger a warning if numRows is over a thousand. It expects numRows to be under a thousand; if it isn't, it can still complete correctly, but it may take longer than normal, or use more memory than normal, or it may simply be that if it got more rows than that, something may be amiss with the query that got the rows or the dataset the rows came from originally. It's not a critical failure, but it's cause for investigation.

Any assumption or expectation that fails should of course be automatically and immediately reported to the development team for investigation. Naturally a failed assumption, being fatal, should take priority over a failed expectation, which is recoverable.

This not only provides greater flexibility than a simple assertion framework, it also provides more explicit self-documenting code.

Be Maxwell's Demon

Source code tends to follow the second law of thermodynamics, with some small differences. In software, as in thermodynamics, systems tend toward entropy: as you continue to develop an application, the source will increase in complexity. In software, as well as in thermodynamics, connected systems tend toward equilibrium: in development, this is known as the "broken windows" theory, and is generally considered to mean that bad code begets bad code. People often discount the fact that good code also begets good code, but this effect is often hidden by the fact that the overall system, as mentioned earlier, tends toward entropy. That means that the effect of broken windows is magnified, and the effect of good examples is diminished.

In thermodynamics, Maxwell's Demon thought experiment is, in reality, impossible - it is purely a thought experiment. However, in software development, we're in luck: any developer can play the demon, and should, at every available opportunity.

Maxwell's demon stands between two connected systems, defeating the second law of thermodynamics by selectively allowing less-energetic particles through only in one direction, and more-energetic particles through only in the other direction, causing the two systems to tend toward opposite ends of the spectrum, rather than naturally tending toward entropy.

By doing peer reviews, you're doing exactly that; you're reducing the natural entropy in the system and preventing it from reaching its natural equilibrium by only letting the good code through, and keeping the bad code out. Over time, rather than tending toward a system where all code is average, you tend toward a system where all code is at the lowest end of the entropic spectrum.

Refactoring serves a similar, but more active role; rather than simply "only letting the good code through", you're actively seeking out the worse code and bringing it to a level that makes it acceptable to the demon. In effect, you're reducing the overall entropy of the system.

If you combine these two effects, you can achieve clean, efficient, effective source. If your review process only allows code through that is as good or better than the average, and your refactoring process is constantly improving the average, then your final code will, over time, tend toward excellence.

Without a demon, any project will be on a continuous slide toward greater and greater entropy. If you're on a development project, and it doesn't have a demon, it needs one. Why not you?

Real Sprints

Agile methodologies talk about "sprints" - workloads organized into one to four week blocks. You schedule tasks for each sprint, you endeavour to complete all of it by the end of the sprint, then you look back and see how close your expectations (schedule) were to reality (what actually got done).

Wait, wait, back up. When I think of a sprint, I think short and fast. That's what sprinting means. You can't sprint for a month straight; you'll die. That's a marathon, not a sprint.

There are numerous coding competitions out there. Generally, you get around 48 hours, give or take, to build an entire, working, functional game or application. Think about that. You get two days to build a complete piece of software from scratch. Now that's what I call sprinting.

Of course, a 48 hour push is a lot to ask for on a regular basis; sure, your application isn't in a competition, this is the real world, and you need to get real work done on an ongoing basis. You can't expect your developers to camp out in sleeping bags under their desks. But that doesn't mean turning a sprint into a marathon.

The key is instilling urgency, while moderating burnout. This is entirely achievable, and can even make development more fun and engaging for the whole team.Since the term sprint has already been thoroughly corrupted, I'll use the term "dash". Consider this weekly schedule:

• Monday: Demo last week's accomplishments for stakeholders, and plan this week's dash. This is a good week to schedule any unavoidable meetings.
• Tuesday and Wednesday: your 48 hours to get it done and working. These are crunch days, and they will probably be pretty exhausting. These don't need to be 18-hour days, but 10 hours wouldn't be unreasonable. Let people get in the zone and stay there as long as they can.
• Thursday: Refactoring and peer reviews. After a run, athletes don't just take a seat and rest; they slow to a jog, then a walk. They stretch. The cool off slowly. Developers, as mental athletes, should do the same.
• Friday: Testing. QA goes through the application with a fine-toothed comb. The developers are browsing the web, playing games, reading books, propping their feet up, and generally being lazy bums, with one exception: they're available at a moment's notice if a QA has any questions or finds any issues. Friday is a good day for your development book club to meet.
• By the end of the week, your application should be ready again for Monday's demo, and by Tuesday, everyone should be well-rested and ready for the next dash.

Ouch. That's a tough sell. The developers are only going to spend two days a week implementing features? And one basically slacking off? Balderdash! Poppycock!

Think about it, though. Developers aren't factory workers; they can't churn out X lines of code per hour, 40 hours per week. That's not how it works. A really talented developer might achieve 5 or 6 truly productive hours per day, but at that rate, they'll rapidly burn out. 4 hours a day might be sustainable for longer. Now, mind you, in those four hours a day, they'll get more done, better, with fewer defects, than an army of incompetent developers could do in a whole week. But the point stands: you can't run your brain at maximum capacity eight hours straight, five days a week. You just can't - not for long, anyway.

The solution is to plan to push yourself, and to plan to relax, and to keep the cycle going to maximize the effectiveness of those productive hours. It's also crucial not to discount refactoring as not being productive; it sets up the following weeks' work, and reduces the effort required to get the rest of the development done for the rest of the life of the application. It's a critical investment in the future.

Spending a third of your development time on refactoring may seem excessive, and if it were that simple, I'd agree. But if you really push yourself for two days, you can get a lot done - and write a lot of code to be reviewed and refactored. In one day of refactoring, you can learn a lot, get important work done, and still start to cool off from the big dash.

That lazy Friday really lets you relax, improve your craft, and get your product ready for next week, when you get to do it all over again.

The Development Stream

I was reading today about GitHub's use of chat bots to handle releases and continuous integration, and I think this is absolutely brilliant. In fact, it occurs to me that using a chat bot, or a set of chat bots, can provide an extremely effective workflow for any continuous-deployment project. Of course, it doesn't necessarily have to be a chat room with chat bots; it can be any sort of stream that can be updated in real-time - it could be a Twitter feed, or a web page, or anything. The sort of setup I envision would work something like this:

Everyone on the engineering team - developers, testers, managers, the whole lot - stay signed in to the stream as long as they're "on duty". Every time code is committed to a global branch - that is, a general-use preproduction or production branch - it shows up in the stream. Then the automated integration tests run, and the results are output to the stream. The commit is deployed to the appropriate environment, and the deployment status is output to the stream. Any issues that occur after deployment are output to the stream as well, for immediate investigation; this includes logged errors, crashes, alerts, assertion failures, and so on. Any time a QA opens a defect against a branch, the ticket summary is output to the stream. The stream history (if it's not already compiled from some set of persistent-storage sources) should be logged and archived for a period of time, maybe 7 to 30 days.

It's very important that the stream be as sparse as possible: no full stack traces with error messages, no full commit messages, just enough information to keep developers informed of what they will need to look into further elsewhere. This sort of live, real-time information stream is crucial in the success of any continuous-deployment environment, in order to keep the whole team abreast of any issues that might be introduced into production, along with when and how they were introduced.

Now, what I've described is a read-only stream: you can't do anything with it. GitHub's system of using an IRC bot allows them to issue commands to the bot to trigger deployments and the like. That could be part of the stream, or it could be part of another tool; as long as the deployment, and its results, are output to the shared stream for all to see. This is part of having the operational awareness necessary to quickly identify and fix issues, and to maintain maximum uptime.

There are a lot of possible solutions for this sort of thing; Campfire looks particularly promising because of its integration with other tools for aggregating instrumentation data. If you have experience with this sort of setup, please post in the comments, I'd love to hear about it!

Truly Agile Software Development

Truly agile software development has to, by nature, allow for experimentation. In order to quickly assess the best option among a number of choices, the most effective method is empirical evidence: build a proof of concept for each option and use the experience of creating the proof, as well as the results, to determine which option is the best for the given situation.

While unit tests are valuable for regression testing, a test harness that supports progression testing is at least as useful. Agile development methodologies tend to focus on the idea of iterating continuously toward a goal along a known path; but what happens when there's a fork in the road? Is it up to the architect to choose a path? There's no reason to do so when you can take both roads and decide afterward which you prefer.

Any large development project should always start with a proof of concept: a bare-bones, quick-and-dirty working implementation of the key functionality using the proposed backing technologies. It doesn't need to be pretty, or scaleable, or extensible, or even maintainable. It just has to work.

Write it, demo it, document what you've learned, and then throw the code away. Then you can write the real thing.

It may seem like a waste of time and effort at first.  You'll be tempted to over-engineer, you'll be tempted to refactor, you'll be tempted to keep some or all of the code. Resist the urge.

Why would you do such a thing? If you're practicing agile development, you might think your regular development is fast enough that you don't need a proof. But that's not the point; the point is to learn as much as you can about what you're proposing to do before you go all-in and build an architecture that doesn't fit and that will be a pain to refactor later.

Even if it takes you longer to build the proof,it's still worth it - for one thing, it probably took longer because of the learning curve and mistakes made along the way that can be avoided in the final version, and second because again, you've learned what you really need and how the architecture should work so that when you make the production version you can do it right the first time, with greater awareness of the situation.

This approach allows much greater confidence in the solutions chosen, requiring less abstraction to be built in to the application, which allows for leaner, cleaner code, and in less time. Add to that the value of building a framework that is flexible enough to allow for progression testing, and you've got the kind of flexibility that Agile is really all about.

Note: Yes, I understand that Scrum calls prototypes "spikes". I think this is rather silly - there are already terms for prototypes, namely, "prototype" or "proof of concept". I'm all for new terms for things that don't have names, but giving new names to things that already have well-known names just seems unnecessary.

HTML5 Grid Layouts

I have to take issue with the swarm of "responsive grid layout" systems that have been cropping up lately. Yes, they're great for wireframes and prototypes. No argument there. And yes, they take care of a lot of the legwork involved in producing a responsive layout. Great. But in the process, they throw semantic markup and separation of concerns out the window.

The idea of semantic markup is that your document structure, IDs, and classes should describe the content of the document. Separation of concerns, in HTML and CSS, means using classes and IDs to identify what something is (not how it should appear), and using CSS to identify content and determine how it should appear; this allows you to change content without having to change appearance, and vice versa: the concerns of document structure and appearance are kept separate.

That means, as far as I'm concerned, as soon as you put a 'class="two column"' into your HTML, you've lost the game. You've chained you structure to your presentation. Can you change your presentation without modifying the markup? Not any more. All we've achieved in this is bringing back the days of nested tables for layout, with a pretty CSS face on it. With one dose of "clever" we've traveled back in time 15 years. Only this time, there *are* other ways to do it. There's no excuse. It's just plain laziness.

Is building a truly semantic, responsive, attractive layout possible? Absolutely. Difficult? Yes. Is it worth the effort? In the long run, I think it is - except for those cases mentioned above, prototypes and wireframes, code that's meant to be disposable. But any code that has to be maintained in the future will be hamstrung by these systems.

Web development has made tremendous strides over the last 10 years. It's amazing how far we've come in terms of what can be done and how. Don't take all those advances and use them to regress all the way back to clunky old table-based layouts. Try using them to do something new, and interesting instead. There's no reason the idea of software craftsmanship should be missing from the web design world.

Assumptions

We all make assumptions. It's the only way we can get anything done. If every time you found a bug you started at the iron - testing the CPU to make sure every operation returns an expected result - it'd take you months to troubleshoot the simplest issue. So we make assumptions to save us time, when we know that the likelihood of something being the cause of a problem is far less than the time it would take to verify it.

We also make assumptions out of sheer bloody-mindedness. You can spot these assumptions by phrases like "that couldn't possibly be it" or "it's never been a problem before" or "I wrote that code, I know it works". These are the kinds of assumptions that can get us into trouble, and they're the exact reason why it's important to have developers from different backgrounds, with different perspectives, who make different assumptions.

Since we all make assumptions, the best way to challenge those assumptions is to have someone who makes different assumptions look at the issue. They'll bring their perspective and experience to the matter, challenge your assumptions where they don't make sense, and make you prove those assumptions to be accurate or not. This stands as a strong incentive to hire a team with diverse backgrounds and areas of expertise. They bring not just talent to your team, but a different perspective.

It's also a good reason to invest the time in learning different technologies, languages, and development philosophies. Getting outside of your comfort zone can open your eyes to things you might not otherwise have considered, and help you to gain new perspective on your work - helping you to challenge your own assumptions.

A Programmer's Comparison of Java and C#/.NET

I've been developing in Java for almost ten years, and in C# for only a few months, so this comparison may not be as thorough as it could be. If I've missed something please feel free to comment.

Java is far more portable than C# for one. I've built applications on Windows, and ported them to Linux and Mac with minimal effort. With Android running on a Java-based platform, porting to Android is made far easier. There's no fussing with Mono and the like, no making sure you don't use any API functions that are only available in .NET.

Java also has a wide array of available IDEs with excellent features, and a huge, active community. There are libraries available for just about any technology you can think of, usually more than one, such that one of the options will likely fit your specific situation.

Java's runtime generics seem to be easier to learn and work with than C#'s compile-time generics; however, compile-time generics are likely better performing. Java's overridable-by-default scheme also makes development a lot easier in many cases. While I do understand the idea behind C#'s final-by-default scheme, I prefer a language that leaves those kinds of choices up to the developer rather than enforcing good development practices through language features.

The JVM is also now expanding to support other scripting languages, including PHP, Python, Ruby, Scala, and others.

C# has some excellent language features that I would like to see in Java, however. Extension methods are extremely useful for adding functionality to classes without having to subclass, particularly useful in adding functionality to library classes. C#'s delegate features are really excellent, and beat any workaround Java has for its lack of closures for callbacks and event handlers. The upcoming Java closure functionality looks like it will still pale in comparison to C#'s delegates.

LINQ is something I would love to see come to Java; the ability to query collections like databases is extraordinarily useful and eliminates a lot of tedious code iterating over collections. I've yet to use it for querying against a database, but it seems more than adequate for that purpose and likely much friendlier than JDBC. And while porting is more complicated, Mono is a very strong platform, and there's even a Mono module for hosting web applications through Apache.

Speaking of web applications, I have no experience so far with building web applications in C# .NET, but I have done some research. Based on that research, I have to say I significantly prefer JSP/JSTL/EL over ASP.NET. I prefer the syntax, the workflow, and JSP's tag library system over ASP.NET, which reminds me a little too much of PHP or old-school JSP (pre-JSTL/EL) for my tastes.

All in all, I can't say one is superior to the other; like any development decision, it comes down to which option is best suited to the situation and the developer. If you've had the opportunity to make that choice, please leave a note in the comments and let me know what you chose and why, I'd be happy to hear it!

Convenience Languages

I've come to see the uncertainty of untyped and interpreted languages as something of a curse. In a strongly typed, compiled language, you know ahead of time that the code is at least trying to do what you want it to; you know you didn't typo any variable, function, method, or class names. You know you didn't misuse or misunderstand a function, passing one type when another is required. Sanitizing inputs and returns is a matter of checking bounds, not types. Type-safe comparison is a non-issue.

After working with PHP and JavaScript extensively, as well as dabbling in Perl, Python, and Ruby, I miss the basic assurance you get from a language like C/C++, C#, or Java that if it compiles, nothing is completely wrong. Even in HTML, you can validate the markup. But in PHP or JavaScript, you probably don't know about even a major, simple error until run-time testing (unit or functional).

To me, that's a nightmare. I miss knowing. I miss that little bit of certainty and stability. With an untyped interpreted language, you may never be 100% certain that you've not made a silly but fatal mistake somewhere that your tests just didn't happen to catch.

These are languages of convenience: easy to learn, quick to implement small tasks, ubiquitous. But they just aren't professional-grade equipment.

Developing software is both an art and a science. I make an effort every day not to just be a coder, but to be a code poet. That's hard to do on the unsure footing of a dynamic language. I won't argue that these languages let you do some neat tricks; on the other hand, I also won't discuss the performance issues. My concern is purely quality.

Is it possible to write quality code in a dynamic language? Absolutely. Unfortunately, it's harder, and far more rare - not just because it's challenging. It's mainly temptation. Why would the language offer global variables if you weren't supposed to use them? Why have dynamic typing at all if you aren't going to have variables and function return values that could have various types depending on the context? Even with the best intentions, you can commit these Crimea against code accidentally, without even knowing it until you finally track down that pesky bug 6 months down the road.

Using (and abusing) these sorts of language features makes for messy, sloppy, confusing, unreadable code that can be an extraordinary challenge to debug. Add to that the fact IDEs are severely handicapped with these languages, unable to offer much - if any - information on variables and functions, and unable to detect even the simplest of errors. That's because variable types and associated errors only exist at runtime; and while an IDE can rapidly attempt to compile a source file and use that to detect errors, it can't possibly execute every possible code path in order to determine what type(s) a variable might contain, or function might return.

I know most of this has been said before, and every new language will inspire a new holy war. I'm writing this more because all of the above leads me to wonder about the growing popularity of dynamic languages like Python, Ruby and JavaScript, and the continued popularity of PHP. Anyone care to shed some light on the subject in the comments?

Simplicity, Flexibility, and Agility

Agile programming is supposed to be about flexibility in the face of changing requirements. It's supposed to be about rapid development and iteration. But all too often it ends up being like classical methodologies in many ways. Many agile methodologies drown developers in process, taking time away from development. Test-driven development is a brilliant concept, but it puts more time between planning and iteration, making it more difficult to deal with changing requirements, not easier, and increasing the burden of change and the cost of refactoring.

Every developer wants carefully, precisely defined requirements. Developers often try to handle changing requirements by developing for flexibility, but flexibility often comes at the cost of added complexity. Trying to write-in endless flexibility to allow for changing requirements is very much akin to premature optimization. You end up doing a whole lot of work to make some code "better" - more flexible in this case, versus more performant in the case of optimization - when you don't yet know which code really needs it and which code doesn't.

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." --C. A. R. Hoare 

Often the best way to maintain flexibility is through simplicity. A program that meets its requirements with the simplest possible implementation is one that will be naturally flexible, maintainable, stable, and manageable.Of course, intelligent development plays a major role; making appropriate use of design patterns can do a lot to improve both flexibility and simplicity. Key design tenets like DRY, YAGNI, separation of concerns, and avoiding premature optimization (including overarchitecting for flexibility) help keep complexity down and productivity up.

What if requirements change? What if the simplest possible solutions isn't as extensible? Good news: having invested in the simplest possible solution, you've lost little in development. You haven't built anything that wasn't strictly necessary. The new solution will still aim for simplicity, still reap the same rewards. By keeping things simple, you've kept down the cost of change.

Software development is a learning process by nature; you're building something that's never been done before, or at least building something in a way that's never been done before. Innovation is at its heart, and learning is the personal experience of innovation. That being the case, every iteration has value in what's learned, even if the code is later removed or replaced. The experience of writing it adds value to the team, and the team defines the value of the end product.

Many readers may think of targeting simplicity as a given, but it truly isn't; while simplicity is often a goal, all too frequently it takes a back seat to other concerns, or is abandoned altogether because simplicity becomes far more difficult to achieve with many of the popular frameworks and libraries available today. Frameworks have to aim for maximum flexibility in order to be successful; in order to be general-purpose, they can't make many assumptions, and they have to account for a whole host of different usage scenarios and edge cases. This increases the complexity of the framework, and accordingly, the complexity of any implementation using the framework. The fewer assumptions the framework can make, the more effort a developer has to put in just telling the framework what she's trying to accomplish.

I can't count how many implementations I've seen that are drastically more complex than necessary, simply because they have been forced to apply the conventions required by their chosen framework; and even following those conventions, they're still left managing arcane configuration files, and tracking down bugs becomes an epic undertaking requiring delving deep into the inner workings of the framework and libraries. All too often, the quick-start "hello world" app is far more complex with a framework than without it, and adding functionality only makes the situation more bleak.

So, what does all this add up to? Here's the bullet-point version:

• If you're aiming for agility - the ability to adapt quickly to changing requirements - don't invest too much time on nailing down requirements. Get as much detail as you can, and start iterating. If you're planning for requirements to change, go all the way - assume that your initial requirements are wrong, and think of each iteration as an opportunity to refine the requirements. All code is wrong until proven right by user acceptance.
• Use interface mockups (for software with a UI) or API documentation (for libraries and services) as a tool to give stakeholders a chance to revise requirements while looking at a proposed solution and thinking about using it in real-world scenarios.
• Don't choose flexibility or modularity over simplicity. Choose a framework that won't get in your way; if there isn't one, then don't use a framework at all. Don't write what you don't need. Don't turn a piece of code into a general-purpose API just because you might need to use it again. If you need it in multiple places now, separate it out; otherwise, you can refactor it when it's appropriate.
• Think about separation of concerns early in the game, but don't sacrifice simplicity for the sake of compartmentalization. Simple code is easier to refactor later if refactoring turns out to be necessary. Overarchitecting is the same sin as premature optimization, it's just wearing a nicer suit.
• The simplest solution isn't always the easiest. The simplest solution often requires a lot of thought and little code. Don't be a code mason, laying layer after layer of brick after brick; be a code poet, making every line count. If a change could be implemented by increasing the complexity of existing code, or by refactoring the existing code to maintain simplicity, always take the latter route; you'll end up spending the same amount of time either way, but will reap far more benefits by maintaining simplicity as a priority.
• Simplicity carries a great many implicit benefits. Simpler code is very often faster (and easier to optimize), more stable (and easier to debug), cleaner (and easier to refactor), and clearer to read and comprehend. This reduces development, operational, and support costs across the board.
• Simplicity doesn't just mean "less code", it means better code. SLOC counts don't correlate directly to complexity.
• Don't reinvent the wheel - unless you need to. All wheels aren't created equal; there's a reason cars don't use the same wheels as bicycles.

What are your experiences? What ways have you found to keep hold of simplicity in the face of other pressures? Feedback is welcome in the comments!

NetBeans + Tomcat + Java for MacOS X 10.5 Update 4 = Fail

Just struggled with this for a couple of days. After updating MacOS X with the latest Java update (Java for MacOS X 10.5 Update 4, patching security holes mentioned in the news the last few weeks), I could no longer launch Tomcat from NetBeans. And that's a Bad Thing. I use NetBeans all day for work, on a web application that gets tested through the local copy of NetBeans.

The error I was getting was that it couldn't find a file/directory under "/var/folders/ZC/ZCcmX61vGaqOjbHwwgwW-k+++TI/-Tmp-/". The filename changes every time, in the format "context1234567890123456789.xml". The "+++" throws it off because the path is passed as a URL parameter to the Tomcat Manager, which converts the +'s to spaces.

The solution is to go to your NetBeans application folder, right-click the application, and choose Show Package Contents. Under Contents/Resources/NetBeans/etc, open netbeans.conf, and find the line for "netbeans_default_options". Go to the end of this line, and add, inside the end quote, "-J-Djava.io.tmpdir=/tmp". Restart NetBeans, and you should be good to go.

Stackoverflow.com

Today marks the public launch of Stackoverflow.com, the new programming Q&A site started by the famous Joel Spolsky. So far it looks like it's shaping up to be a very good resource for developers; and I agree with everything they said about Googling for solutions in this post. You can look me up on the site.

Beyond the site itself, which I think will become an oft-used resource for me, I also really like their badge system. I think a similar system could be a huge boon to any forum, social networking site, or other community site. Game developers learned years ago that people will be more active in an area they already enjoy if they can be recognized for their activity. The same idea applies just as well to community sites. I think it's brilliant, and I think we're going to be seeing similar implementations a lot more often.