Wednesday, January 23, 2013

The Title "Software Architect" is Hooey, and Why I Still Became One

In September 2012, I left my role as a Software Engineer to work in the Architecture group of part of my company. I had my reservations, purely on the title alone. Based on my previous experiences with architects in other companies, I found them to be too far removed from the tactical needs of engineering. As such, they were generally long on vision, quick to make mandates, short on feedback from the development process, and hard to convince on anything. Speaking with friends who have worked in my company for a number of years, they had the same general impressions.

The reason I considered the position at all was mostly in the job description. The primary role is managing the Application Lifecycle Management initiative in a sector of the company. It was looking for someone who was knowledgeable about software development, enthusiastic about best practices, and had very good inter-personal and communication skills. In my development role, I, and a ramshackle group of my colleagues, were trying hard to push management and other teams to adopt some industry best practices. It was difficult, but we made enough headway to not be discouraged. This new role seemed like a golden opportunity to steer things at a much broader level. It also didn't seem like architecture, it was practice and program management, and it was certainly some evangelism role. It was enticing.

After speaking at length with the person I'd be replacing, as well as my future boss and my future colleagues, I found them to be the complete opposite of everything I had in mind. My future boss was very receptive to my ideas (in fact shared a lot of the same concerns I had for the company), and my future colleagues shared tales of trust and empowerment. That's it. I was sold (and if not, and I wanted to stay in the company, it was only a year anyway, not too long ;->).

It's been over four months in now, and it would be hard to say I regret my decision, or expected anything less of the role. I was most worried about not spending a significant portion of my time coding or doing things related to all-around software engineering; those worries came to complete fruition. To be fair, there was also an anticipated side-effect -- I now code and tinker around at home on personal projects far more than I ever did when it was my day-job (like the cobbler with the holes in his shoes). Management of an application and a program has been challenging. Context-switching is always occurring, but even then I'm learning to manage it a little better.

As a consequence of accepting the role, I picked up a number of books on project management, ALM, software development concepts, and a single book on software architecture. I should've read most of the books years ago, but never found the time. The one on being a software architect is all about the soft-skills, and really should be required reading for anyone in our community at any technical level. It's called 12 Essential Skills for Software Architects and I plan on writing a lot more about it over the next few weeks. I'm looking forward to finishing, so I can get more of a habit of writing here, and then maybe start reading some John le Carré.

Tuesday, January 22, 2013

Software "Gardener"? No, You're An Engineer!

Every now and then I come across a post which attempts to describe software development as a metaphor. I find most of them to be pretty terrible; they're often trite, and ironically subjective, considering the metaphors describe a field that is constantly trying to define itself objectively.

After a cursory search, I came across Jeff Atwood's post on the exact same subject. He aggregates quite a few of them together, and he even has a favorite. The list goes from the most obvious (science, bridge building and construction), to the most loose comparisons of gardening and artistic endeavors. If you feel it will help you "rally the troops", I guess whatever works is fine. However, why use metaphor when what you do is easily and succinctly described in one word: "engineer".

Yes, believe it or not, you are in fact an engineer. Some people will call themselves a software developer, or a programmer, and that might be an adequate description for most situations, but don't sell yourself short, don't use metaphor, and certainly don't try to convince others they're not an engineer either.

At first, when researching this topic, I was a bit bewildered and slightly amused. But as I kept reading, I was actually a bit insulted. Then recently, it occurred to me that maybe the problem is that the term engineering is generally misunderstood. To be sure, engineering does not imply that you specifically build bridges or construct buildings, any more than it means that you operate a train or locomotive. Engineering is about the application of science and mathematics to solve problems.

Let's look at a definition. The Encyclopedia Britannica references the predecessor of ABET, the Engineers Council for Professional Development in defining engineering as
the creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behaviour under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property...

Ring a few bells? I sure hope so.

As an example (which is not related to buildings or bridges), I graduated university with a degree in Chemical Engineering. My primary courses consisted of a fair amount of science: various physics and chemistry classes, as well as metallurgy and a lot of calculus. However, this was only a foundation in understanding how the material in my engineering courses was derived. My actual engineering courses applied science I learned to phenomena and general problem sets that occur in the field.

Engineers of all disciplines understand that their required tasks will change on a daily basis, but regardless of the task, most are based on the same general set of equations, concepts and processes. Those equations, concepts and processes themselves, are based on decades of science performed in the lab and tested and re-tested in the field. There's absolutely no need for an engineer to re-derive those equations, concepts and processes on a daily basis to do their work. They know the derivations work, have learned the science to understand the path to the derivation, and how to ask the right questions when they hit a corner case not properly explained.

Does THAT ring a few bells? I hope so, but let's assume it doesn't and I'll nail the point home.

Take a moment and think about this, when was the last time you were required to actually calculate "Big-O" for an algorithm? Keep in mind, this is not at all the same as understanding what "Big-O" means, or even being able to calculate it from scratch as practice or for fun. Does that mean you shouldn't know the science or math behind it? Of course not. Being able to understand and select a sort or search algorithm is tremendously important for engineers; developing one that competes at the O(log (n)) is much, much less important in order to solve your assigned problem, and lies squarely in the domain of Computer Scientists.

Personally, I feel like this is one of the reasons why Computer Science education programs are being shut down, and I'm not alone. But that's a topic left for another debate.

When all is said and done, we're engineers. It's not self-aggrandizing, and definitely doesn't need explaining away.

Friday, May 18, 2012

First Forays into TDD

So on a recent project, I've finally gotten into Test Driven Development, and I have to say, I completely enjoyed it.

Without giving much away, the project is system allowing other internal business systems to publish XML content to subscribed users. Those users are then allowed to reply with specific responses based on the message content. The message has system-level content in a header, as well as business-level content, agnostic to the system. It became very obvious very quickly that the the concept of message threads would become important. Enter TDD...

The server-side was built using Enterprise Integration Pattern framework to abstract the boilerplate necessary to route messages back and forth, and down to a data store. So a message transformer was introduced to insert a ThreadID into the message header that correlated with the business system's SystemID and their provided ExternalID and ReferenceID. These latter ids, allow the business systems to track their messages in their own internal way. In this way an ExternalID + SystemID would map to a ThreadID in the system's internal cache. A follow-up message could then be given a new ExternalID pointing to a previous message through the ReferenceID.

Since I knew what the messages should look like coming in and going out, it can be modeled completely as a black-box, which naturally and happily lends itself to TDD (happily because it's the easiest case to get acquainted with TDD).

Given the predetermined inputs and outputs (an exception is a valid output, don't forget), I drew up some basic scenarios, and blew that up into a flowchart (via Gliffy):
From there, I was able to layout my general test cases. Many of them were just standard JUnit cases, like failing when not receiving an expected exception:
public void testReferenceIdNotCached() throws Exception {
  String inputXml = "<message><header><systemId>TestSystem</systemId><referenceid>ReferenceIdNotCached</referenceid></header><body /></message>";
  try {
    Message<String> output = transformer.transform(new Message<String>(inputXml));
                    "Value of <referenceid> field is not cached and should throw an exception. Output is::[%s].",output.getPayload()));
  } catch (MessageTransformationException e) {
    assert e.getCause() instanceof IllegalArgumentException : "Underlying exception is incorrect.";

Some of the tests though check that for the canned message inputted, creates an output message matching an expected format. This is where XmlUnit was a big help. It lets you compare XML strings, giving you the option to either expect an exact string or a similar string with a different ordering of elements. It also lets you go into a more detailed level where you can inspect the differences and potentially discard them, one by one. Below is an example of getting a similar output, since the ordering of the header elements doesn't matter, just that they exist, and the values are the same.

public void testRefExternalIdPassedWithRefCached() throws Exception {
  String referenceId = "EXT_" + UUID.randomUUID().toString();
  UUID threadId = threadIdCache.getNewAndPut(referenceId, "TestSystemId");

  String inputFormat = "<message><header><systemid>TestSystemId</systemid><referenceid>%s</referenceid></header><body /></message>";
  String inputXml = String.format(inputFormat, "TestSystemId", referenceId);
  Message<String> transformedMessage = transformer.transform(new Message<String>(inputXml));

  String expectedFormat = "<message><header><systemid>%s</systemid><threadid>%s</threadid><referenceid>%s</referenceid></header><body /></message>";
  String expectedXml = String.format(expectedFormat, "TestSystemId", threadId.toString(), referenceId);
  String transformedXml = stripMessageId(transformedMessage.getPayload());

  Diff diff = new Diff(expectedXml, transformedXml);
  assertTrue(diff.toString(), diff.similar());

Once these were written, I reused my flowchart to write the code. As a result of this coordinated effort, I immediately saw from an earlier version of the chart that I missed a couple of checks and that the ordering of them was out of whack, but my tests never really needed to change since my inputs and outputs were still the same. And since my tests were already written, I felt slightly compelled and more at ease to add more tests to make sure inputs existed and were in the correct format.

It was a great process, and I look forward to using it again in other parts of my code. The three things that really came across during this: it's very hard to go against habit making sure I write the tests before the code, I can see how difficult it can be to put in place in certain situations (like when refactoring), but you can eventually get tests in place on a large enough section of your code to make the process useful, and lastly, your tests are written!.

Friday, May 11, 2012

gzip and the stdout

As a native Windows user, I still don't consistently think in a CLI mindset. So when I finally came across a shortcut to a common annoyance, it was a forehead slap of "Oh, THAT'S why that's there."

When writing apps hopefully, like me, you roll over your logs and compress them to conserve space. However, also like me, you have to check those files in a production or production-like environment. Which means either logging in as some super-user, or downloading the files, or copying them to some /tmp directory, when you can unzip the files, and read them.

Or you could save yourself a couple of unnecessary steps and just think in CLI using the "-c" flag for gzip, then pipe the output to less, more, or wherever.

bash> gzip -cd rolledover.20120510.log.gz | less

Thursday, May 10, 2012

So You Want To Be a Developer

While one of the reasons I started this blog is to act as a Pensieve of sorts allowing me to retain and review interesting tidbits of code I've developed, there is another reason for it. This blog allows me a forum to expound a bit on the basics of computer programming, software development and engineering... they're not the same! (but more on that later).

Anyways, here's my first entry on that subject, and instead of reexplaining it, I'll just point you to a couple of entertaining videos (which also buys me time on a longer post I have brewing). It the 10,000 ft view about I actually do for a living, and includes some broad recommendations on how you can be better at this job, and level-up your software developer / engineering character.

Without further delay (and in two parts) here is So You Want To Be a Developer Part 1 and Part 2.

Many thanks to the folks at Extra Credits, Stack Exchange and Penny Arcade for putting this together and spreading the word.

Wednesday, May 9, 2012

Why am I doing this?

I'm no self-help enthusiast, but I am a fan of The Nerdist Chris Hardwick, and read his book The Nerdist Way (check out my review here). In it he mentions trying to build a D&D character of yourself, finding areas you want to target and build your strength in. It's geeky, but it is a pretty logical way to do some self-improvement.

I started to think about rateable skills and strengths for a software developer. It would be a fun but daunting task. It's something that has churned around in the back of my head from time to time. I then came across this beauty, which is the "Programmer Competency Matrix". It's not perfect, and it doesn't apply for everyone's career path, or for every job, but it's a great starting point.

BTW, this blog levels me up to log(n) on Knowledgeblogs.

Tuesday, May 8, 2012

Testing SyntaxHighlighter

Getting this thing ready for primetime. Eat it posterous! I've loved the way SyntaxHighter has looked, and wanted a blog that can support it. Posterous fails miserably in that area. So based on my friend Doug's suggestions, I hit up this installation guide for Blogger, and here we are.
public class SyntaxTester {
  // single line comment
  private static final String HEY_YO = "Hey yo!";

   * Javadoc
   * @param args
  public static void main(String[] args) {