Tuesday, December 25, 2007

"Good Enough" Might Actually Mean "Good Enough"

Ryan Tomayko claims that IBM Poopheads saying that LAMP Users Need to "grow up" are trying to foist an over-complexified approach onto database-oriented applications.
After wasting years of our lives trying to implement physical three tier architectures that "scale" and failing miserably time after time, we're going with something that actually works.
His general point is that complex architectures designed from Day One for a "million users and requir[ing] five 9's from the first day of development" "sacrificed simplicity and the ability to scale low".

This principle of simplicity has been gaining significant traction with me over the last several years. I've been a software architect and gone the whole UML/Rational route. But then I've seen so much crud getting accreted onto it that my level of skepticism regarding that technological approach continues to rise. The more I see of it, the more I see of architecture being handed down "from above", the less effective and responsive (i.e., agile) I'm finding the resulting system or application to be.

Monday, December 24, 2007

Imaginary Numbers Real? Yes, You Just Need to Rotate Your Mind!

The whole sqrt(-1) imaginary number thing has always been pretty abstract to me (and probably most others who learned them, but then never had to use them in their job or anything).

A Visual, Intuitive Guide to Imaginary Numbers gives one a technique to visualize them and how and why they work the way they do, and also demonstrates how imaginary numbers can be surprisingly effective in unexpected areas.

Tuesday, December 11, 2007

When a Bic (tm) is more than a pen

Read the reviews!

A Map of Everything

"Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the drug store, but that's just peanuts to space. "
-- Douglas Adams

Thursday, December 6, 2007

A Coding War Story: What's Your Point?

I had been assigned the task of porting a fairly large (about 400 KSLOC) missile launch command and control system to an upgraded OS version and new compiler and language version. Specifically, from Solaris 2.5.1 to Solaris 7, and from the Verdix Ada Development System (VADS), which was Ada 83, to Rational Apex Ada, which was Ada 95. VADS had been bought out by Rational, and its product obsoleted, although Rational did a pretty good job implementing compatible versions of VADS-specific packages to ease the transition to the Apex compiler.

Three other guys helped with the initial compilations, just to get clean compiles of the code, which took about two weeks, and then I was on my own to actually make the whole system work. Long story short, it was the worst design and implementation of a software system I'd ever seen, and so took about two more months to successfully complete the port. It was then handed over for formal testing, which took several months as well. I fairly steadily fixed the bugs that were found as testing got going, but that rate quickly declined as it progressed (the original code was a production system after all, so its functionality was pretty solid, I just had to kill the bugs that came about due to adapting to the new compiler). Eventually I was reassigned to another project once everything appeared to be working as well as the original.

Then came the phone call on the Friday before Thanksgiving.

There was a missile test scheduled in about three weeks, and during a lab countdown test the command sequencing had locked up. In real life this would cause a test abort, and if this lock-up occurred within seconds of ignition, a number of irreversible actions would have taken place in support systems, causing a lengthy--and expensive--delay for reprocessing the missile. The missile would not have launched, but there would have been many, many very unhappy people seriously distressed over issues of time and much, much money. (Don't let anyone ever tell you that the Defense Department is cavalier about spending money--I've yet to meet a contract manager for whom budget wasn't their number 1 or 2 priority, with schedule being the other.)

Now this countdown test and many variations of it had been run hundreds of times in the preceding months, with only a handful of minor glitches. So this problem had a very low probability of occurrence, but unfortunately possessed a very high cost of occurrence. Multiply those together and the product was a bad Thanksgiving week for me and dozens of other engineers and managers.

As the guy who did the port this put the spotlight right on me.

Like most safety-critical defense systems like this, a lot of logging is captured, so it was fairly easy to locate the handful of lines of code that had been most recently executed when the system froze. And of course there was absolutely nothing questionable in those lines of code, and these same statements had already successfully executed literally thousands of times during that same run.

We put the Apex guys at Rational on notice, since it was their compiler and some of their vendor-supplied routines were being called in this area, and it was impressed on them (and everyone) that this was a problem of literally national importance that had to be tracked down. So they got their Thanksgiving week trashed as well.

Since the logs could only tell us so much, we needed to try to repeat the problem in the local lab. For something that pops up in only 1 in a 1000 test runs that's not going to be easy. Amongst the conjectures as to root cause was that a call into a vendor-supplied mutex (part of a VADS migration package) Unlock function was not unlocking. The processing thread that made this call was handling a heartbeat message that nominally arrived every few seconds. So we tried upping the rate on that heartbeat to 10 Hz, i.e., 10 per second, and kicked it off. About an hour later the system locked up. And, when reviewing the logs we saw that the same sequence of logged messages was occurring as had taken place in the failed run. Several runs were made, and it would consistently lock up sometime between 45 and 90 minutes after starting, and each time had the same log trace. So even though we were not now technically running the same code--because of the increased heartbeat rate--the behavior was consistent and so we had high confidence that this stressing scenario was triggering the same problem.

The trick now was to figure out exactly where in the sequence of candidate statements the lock up was occurring.

The implementation of this system used Ada tasking, and used it extraordinarily poorly. Tasking is Ada's high-level concurrency construct, sorta like threads, only built into the language itself. When two tasks communicate, they do it by "rendezvousing", at which time they should exchange any data of interest, and then break the rendezvous and resume their independent executions. This system wasn't implemented that way. Instead, once rendezvous had been made with a target task, that target task would then rendezvous with another task, which in turn would rendezvous with another task, and so on, until eventually some processing would get done, after which all the rendezvous would be broken and each of the tasks would go on their merry way. So what you ended up with was the world's most expensive function calls, bringing an entire, "multi-tasking" process to a halt while it processed a piece of incoming data. It was only because the normal throughput was so low that this hadn't caused performance problems in the past.

The point of this digression about tasking, though, is that when a rendezvous is requested or awaited upon, a "task switch" can occur. This means that the CPU can start processing a different task that's ready to run. So when one task becomes ready to rendezvous with another, a different task may jump in line and get executed, with control eventually getting passed back around to the rendezvousing tasks. Now there are other events that can also cause a task switch, one of which is calling an OS function, like what happens with printing or performing a mutex.

So in tracking down exactly which line was causing the problem I had to find a way to record the progress through the sequence of statements--while not triggering a task switch, which could prevent the problem from occurring. So doing Put_Line() was not an option, no system I/O of any sort could be done. I could set a counter variable or something like that, but how do I see what its value is to tell me how far it got, since I can't print it out?

Now one thing that had been observed in the log files about this executable was that while this heartbeat processing froze--which ultimately led to the process' I/O getting all blocked up, and preventing other necessary processing from occurring--other independent tasks within the executable continued to run. So the process as a whole wasn't getting blocked, just a (critical) task chain within it.

This was the wedge needed to get at locating the offending statement.

I created an Ada package containing an enumeration type, a global variable of that type, and a task. The enumeration literals were keyed to the specific statements in the problematic code sequence (like "Incrementing_Buffer_Index", "Locking_Mutex", "Mutex_Unlocked", etc.) and then into that sequence were inserted assignment statements that assigned the corresponding enumeration to the global variable. Because the object code for this was nothing more than storing a constant into a memory location it was extremely unlikely that a task switch could occur by executing such a statement. In fact, our primary suspicions centered on those statements that involved task switches, since the locking up behavior was consistent with execution not resuming (for some reason) after a task switch back.

The monitoring task then itself did nothing more than loop and periodically check to see if the global variable had changed value. Every time it did, it printed out the value to a file. It then delayed for a small interval, and made its next check. Now the reason I could write to a file from this task was that this task only ran when a task switch had occurred back in the problem area and this task had been selected to run. Whatever was done in this task should have no effect on other, unrelated, blocked tasks.

The behavior that was anticipated here, then, was that when the problem code area was entered it would do its thing and keep resetting the global variable as it progressed past each statement. It would then do something that caused a task switch, and because its execution rate (10 Hz) was slower than that of the monitoring task's, the monitor could grab the value of the global variable and write it out. So under normal behavior I would expect to see a repeating sequence of a subset of the enumerations, specifically each of those that the variable last held before a task switch occurred. And when the freeze happened, that global variable value should no longer change and the last one recorded will indicate from exactly which statement execution never resumed.

Ran the instrumented executable. It froze up. And the monitoring worked like a charm.

The logging of the progress monitoring variable displayed exactly the anticipated sequence, which eventually ceased with a value corresponding to having made a call to the Mutex Unlock function, with the value that should have been stored signaling the resumption of the task never showing up--like it had in the thousands of previous invocations.

So over to you Rational. The Apex engineers during this time had been feverishly analyzing their code and had found a place in the mutex code where it could theoretically block for good, but the odds of that happening were very remote because of everything that had to happen with the right sequencing and timing. Murphy's Law, guys, Murphy's Law.

What I did to work-around this was to replace the calls to the vendor's mutex functions (which were built atop the OS' mutex functionality) protecting this particular sequence of code with a quick little native Ada mutex package, using that to control mutex access to the relevant area.

I put this into the code and reran the test. Seven hours later it was still running.

My mutex package code was given to Rational who compiled and disassembled it and verified that it was not using the same approach that the problematic mutex functions were using.

I then had the most well attended code inspection of my career :-) There were nearly a dozen engineers and managers in the room with me, and at least another dozen dialed in from all over the country, all to inspect about 20 lines of code.

It passed, the new executables were formally built, and it was handed over to the test organization for formal regression testing. A couple weeks later the missile countdown proceeded flawlessly and away it went.

It's a good think I like cold turkey.


Okay, this is all well and fine, but what's really the point of a coding war story?

This was a nasty, nasty problem. There was concurrency, over a dozen communicating processes, hundreds of KSLOCs, poor design, poor implementation, interfaces to embedded systems, and millions of dollars riding on the effort. No pressure, eh?

I wasn't the only developer working on this problem, though having done the original port I was of course the primary focus. But even though I did the porting, that doesn't mean I had intimate knowledge of hundreds of thousands of lines of code--or even a decent overview of it. Other engineers around the country were looking through the code and the logs as well, but I found that when they proposed a hypothesis to me about a root cause, it never took more than 30 seconds on my part to dismiss it, likewise when I was requested to provide various analyses I would shove it off on to someone else because it was clear to me they were on the wrong track. Sound like arrogance on my part? Well, yeah, it does, but that's not why I dismissed these hypotheses and requests.

It was because I knew what the nature of the problem was. I didn't know exactly where it was occurring, nor why it was occurring, but I did know what was happening.

I've built up a lot of experience and knowledge over the years--I was an early adopter of Ada, understand concurrency and its pitfalls, I know how Ada runtime libraries handle tasking and concurrency, and I understand low-level programming at the level of raw memory, registers, and assembly language. In other words, I have deep knowledge of my niche of the industry. All of that was brought to bear in successfully tracking down this problem--not just working around the bug, but understanding how to put together an approach to finding the bug in a very sensitive execution environment.

The specifics of a coding war story probably aren't all that interesting to those who aren't familiar with the particulars of its nature and environment, but they are useful for gleaning an understanding of what it takes to solve really difficult problems.

To solve the really difficult problems you need to be more than a coder, you have to understand the "fate" of that code, how it interacts with its environment, and how its environment itself operates.

Then you too can get your Thanksgiving holiday all messed up.

Sunday, November 25, 2007

A Plain and Simple Guide to Theories and Laws (and Hypotheses)

Somewhere growing up I'd gotten the idea that when scientists were figuring out something they'd look at what they'd discovered and make a hypothesis; then as they learned more the hypothesis would be refined into a theory, and once the theory was proven it would become a law.

Theories become laws, right?

But there was something about all this confused me. I grew up as a geeky kid and so I had a vague notion of what the "Theory of Relativity" was about, and one day on the radio I heard the newscaster say that "The theory of relativity is no longer a theory, it's a fact. It's been proven." (I have no recollection whatsoever as to what experimental confirmation resulted in this particular proclamation being made at that particular time.)

So I expected to stop hearing about the "theory" of relativity and expected that there'd be this transition to the "Law of Relativity". But that never happened.

Eventually I figured out why.

This notion that theories eventually become laws once they're proven?

Wrong. Completely wrong. And not just wrong, but the idea itself actually doesn't even make sense.

One other tangent and then I'll explain why.

That's just a theory!

In popular usage the word "theory" means something like "contemplation or speculation" (6), or "guess or conjecture" (7). So you hear "theoretically speaking...", or "It's just a theory..." This is fine, a word can mean whatever people want it to mean, so long as everyone in the conversation understands the intended meaning.

But in different contexts, the same word can have very different, even if somewhat related, meanings.

Take "swear", for instance. On the one hand, you have the "pinky swear", an "informal way of sealing a promise." Which, if you break it, the worst that can happen is that there will likely be anger, a feeling of betrayal, and possibly a damaged or lost friendship.

Then there's swearing to the truth of your testimony in a court of law. Swear falsely there and you may find yourself charged with perjury and subsequently spending time in jail.

So, swearing. Same word, same general meaning, but in different venues its meaning is seriously different.

And the same goes with the word "theory".

So a "theory" is what then, exactly? And "laws" and "hypotheses" too, what exactly are they?

Well, in the world of scientific pursuit, a theory is simply a comprehensive explanation of observations.

That's really it, in a nutshell.

Outside of the nutshell the purpose of a theory is to explain why one is seeing the behavior they're seeing. Thy sky is blue, rocks fall, and cold air sinks. Why? The explanation of each of these comes from a theory (Rayleigh scattering, gravity, and the kinetic theory of gases, respectively). The actual observed characteristics and behaviors can be determined by calculating the results of the theories' associated "laws" that describe what happens.

The Laws are the formulas, the description of what outputs you'll get given a particular set of inputs. A "law" describes the behavior, while a theory attempts to explain why that law functions the way it does.

And the neat thing about theories is that if you've got a good one--one that explains why you're seeing what you're seeing, and why things are acting the way they're acting--you can then go on and start making predictions about things you haven't yet seen. Some scientists did this for Einstein's Theory of Relativity, predicting that satellites orbiting the earth would get "dragged" forward an extra six feet a year because of the relativistic effect of the earth's rotation on its surrounding space. After making millions of measurements over 11 years they saw that this "dragging" was in fact occurring, and their prediction closely matched the measurements, within 1%.

So these are the key notions about the scientific type of theories:
  • They have to logically and coherently explain what's being observed, and why it's acting (or acted) the way it does (did).
  • They have to be able to make predictions that can then be tested.
  • New observations and facts that are discovered and are covered by a theory have to be explainable by that theory.
If a fact contradicts a theory, then the theory is flawed.

Now a theory may not explain everything in its area of concern, but that doesn't necessarily mean it's flawed, only that it's incomplete. And with more research, and more experiments, and more data, and more thinking, those unexplained areas may get filled in over time. This is actually the way science progresses, usually just a little bit at a time as more information is collected, which helps the theories get more and more refined.

Now what gets really interesting is when a well-established theory, like Newtonian Mechanics, finds itself confronted with facts, like the energy levels and sizes of atoms, that contradict what the theory requires. This can result in the creation of a whole new theory, like quantum mechanics, that not only explains those problematic facts, but also has to still be able to explain everything the previous theory did, and do it just as well, if not better.

You forgot "hypotheses"

Yeah, okay. A hypothesis is usually an initial suggested explanation for some phenomenon. It's like the first initial cut at a theory, just kinda suggesting a starting point for a theory that then has to go out and start being tested and confirmed, and continuously refined until it either works itself up into a full-blown theory, or, is disproved and discarded.

Hypotheses can also arise out of a theory, as when trying to fill in an area that the theory doesn't yet explain. A hypothesis can be a candidate explanation, based on what the theory says, for the unknown area. Which is then tested to see if it is valid, with the results of experiments either confirming the hypothesis, thus indicating that it belongs as part of the theory, or disproving the hypothesis, which is useful too since it rules something out of the theory.

Theories become laws? That's silly!

Since a theory is an explanation (and hypotheses are tentative, candidate explanations), and laws are descriptions of behavior, it's easy to see now that the notion of a theory becoming a law is just nonsense. They're two different things.

A theory explains why a law comes up with the results it does.

So the next time someone comes up to you and says, "Most of the theories in astronomy are simply that--theories. Most of the theories have no concrete evidence", you'll now know that they don't really know the first thing about what a scientific theory actually is.

A theory without evidence is by definition not a theory, it's just a guess at best, a fantasy at worst.

So when presented with an actual theory, be aware of what went into developing it, how it has to explain the evidence, and make predictions, and continue to be valid as related new data and phenomena are collected and observed over time.

Theoretically this should help clear things up. I hope it helped!

Wednesday, November 21, 2007

Dog and a Plume -- Mount St. Helens Style!

I periodically check the Mount St. Helens volcano cam during the day and fortuitously caught this image (click to biggify):

Thursday, November 15, 2007

We're now blog.kickin-the-darkness.com

Got the domain name, relocated the blog.

Nothing changing--same software, technology, and religious ramblings.

Thanks for stopping by.

Wednesday, November 14, 2007

Change is our Friend

“Don't just tell customers you've changed; show them you've changed.”
—Saabira Chaudhuri, Associate Editor, FastCompany.com

I just discovered this website: http://www.fastcompany.com/homepage/index.html called FastCompany.com.

It reminds me of the importance of Enterprise Architecture and how it is used to manage change in an organization for maximum effectiveness over time.

Pico-Economics and Instant Pigeon Gratification

I just thought this was neat.

The proposal is that there's a "marketplace of ideas" that competes within one's own head, a "pico-economics" that controls our choices and behavior. And that this isn't just going on in our human heads, because animal experiments have shown it operating in them as well.

Peck the third button, pigeon!

So Close It's Burning Ya...

I'm just a big fan of solar power, and aching for the day when I can start slapping down cheap solar shingles on my south-facing Alabama roofs, and then start wrapping everything in cheap, electricity generating solar films.

Nanosolar seems to be out in front right now. Where do I invest!

Tuesday, November 13, 2007

Beyond NASA, Beyond Mars

I have barely posted to this blog started by my dear friend Marc. I seek to turn that around in the coming days. However, I'd like to propose a few topics to cross-pollinate. I am an enterprise architect, a systems engineer, a systems architect, and a software engineer. I am interested in the far reaching elements of design and pushing state-of-the-art in many disciplines simultaneously. As such, I'd like to propose for starters what might be perceived as a vast array of disparate topics to discuss and then start blending them together.

My career has taken me from EPA superfund projects, to Trident nuclear submarine software, to engineering the math model compiler for the B2 Stealth Bomber air crew training device, to designing the vision for smart weapons that can see and autonomously shoot tanks, to developing field artillery tactical data systems, to developing military intelligence workstations, to developing international communications satellite ground stations, to becoming the U.S. Bureau of Land Management's national technology architect, to developing visions for 2020 intelligence community service oriented architectures, to my current role as a systems engineer working for NASA's IV&V facility in Fairmont, West Virginia. This has been a wonderful and beautiful ride from one ground breaking project to another. As such, I have seen and done much in my 25+ year career, but my dear friends this is just the beginning as something wonderful is about to happen.

Here are some of the topics I will be touching on in the coming weeks: architecture, design, systems engineer, software design, software development, NASA space craft, UML, SysML, building architecture, automotive design, UFO's, UFO footage, human culture, alien cultures, human exploration of space, God Almighty and his only begotten son Jesus, and many other topics. This might seem like the ravings of a madman, but in truth, they are the explorations of a humble man who has seen much, done much, and seeks even more adventure.

So, there is my introductory statement. I look forward to our on-going discourse and discussion.

Mean and Green: The 100 mpg Hummer

My father was and is a tinkerer and hacker of mechanical hardware, I followed a similar path except that my milieu is software. Had I stuck with the mechanics I could see myself having gotten into extreme fuel economy makeovers.

Go Jonathan!

Wednesday, October 31, 2007

Why What You Don't Know Keeps Growing--It's Geometry!

A geometrically insightful way of explaining why the more you know, the more you realize you don't know:

"As our circle of knowledge expands, so does the circumference of darkness surrounding it."

Monday, October 22, 2007

Bugs. Infestation. Vulnerability. Gauging your Level.

A good way to start a war among software developers is to state that it's impossible to write bug-free software for anything more complicated than a "Hello World" program. And that goes for claiming the reverse as well. Though "Helo World" has even had its problems at times.

However, there are Bug Infestation Vulnerability Levels (BIVLs) that correspond to software development practices, and these can be a useful shorthand for identifying just how much you're wanting to work at, and spend on, bug eradication.

(First off, we do need to make some assumptions, like that the hardware works, and that the compiler correctly compiles, or that the virtual machine or interpreter correctly executes the byte codes or other constructs making up the program.)

OK, so let's start with ...

BIVL 0: Invulnerable.

Believe it or not, by leveraging formal mathematical techniques like Z Notation and the B Method, and by it now being practical to perform program verification to formally prove correctness, it is actually possible to write defect-free non-trivial applications, and incredibly enough, still actually be seriously productive in doing it, at least in a corporate, mission-critical environment.

But then when you think about it, why shouldn't you have a decent productivity rate when employing such formal practices? Once the code is written, compiled, and formally proven correct--meaning zero defects--there's no debugging (because there's no bugs, get it?), so there's no tracking down problems, no devising fixes, and no patch integration. Kinda knocks a lot of time off the test phases, eh?

Now whether what you write conforms to your requirements is another matter, but as far as software implementation goes, this is as close to defect-free software development as you're going to get, and it's no longer just for trivial applications.

BIVL 1: Best Effort

Despite all the proven benefits in saving time and effort involved with the use of formal methods and proving program correctness, it remains limited to a very, very small number of projects, usually just those with extraordinary safety or security requirements.

Frankly, the rest of us programmers just want to do some design and then get coding. For those who are really passionate about writing high quality software, but just can't get into the formal stuff, obsessing about doing everything else possible to keep bugs out of the code results in BIVL 1 software. That means:
  • Using a programming language that helps prevent or flush out errors
  • Employing assertions, design contracts, or the equivalent
  • Code inspections
  • Expecting any invocation of any external service to fail sooner or later
  • Distrusting the validity of everything that comes in from an external interface. Or from any code you didn't personally write. Or from code that you did personally write.
Making a BIVL 1 Best Effort is not defined in terms of the programmer's "best effort", i.e., the best they themselves are inherently capable of. It's actually putting into practice those objective programming practices listed above (along with others) that result in making the Best Effort possible to produce high quality software. Doing this conscientiously can even make a Terrible Programmer look pretty good.

BIVL 2: I've got to get this done tonight.

This, unfortunately, is the norm. You gotta get the code out the door, or over to test, and there just isn't time to add code to validate every external interface (which should all be working correctly anyway), or figure out what kinds of assertions to put in the code, and no one has any time to inspect anything; but hey, you're a good programmer, or at least "extremely adequate". Sure, you check for the stuff you know could fail, like a file not found, or losing a connection, but beyond that the likelihood of a function failing really is exceedingly rare.

And besides, all software has bugs.

At this point click over to your favorite software project failure statistics.

The obvious irony of this is that part of the reason there isn't time to put in all the checking and validation stuff is because of having to spend time fixing bugs that, well, could've been caught sooner and fixed more easily if more error checking and validation (and paranoia) had been incorporated earlier in the development cycle. So valuable time is spent going back and fixing bugs, and retesting, and reintegrating, and redelivering, and reviewing and prioritizing the never-ending stream of bug reports.

Why is it there's never time to do it right, but there's always time to do it over?

BIVL 3: "Hey, at least it didn't crash..."

If you're going to write code this crappy, why even bother? What are you hoping to accomplish?

One guy I worked with spent a week developing some aero data analysis functions, which then ended up on the shelf for a few months. He moved on to another department, and so when it came time to integrate these functions into the full system another developer picked them up. This latter developer decided to hand run some of the functions just as a sanity check, and kept getting incorrect results. When he dug into the code he discovered that the analysis was all wrong, and that the functions never came up with the right results. He ended up taking another week or so to rewrite it, this time verifying that the outputs were actually correct, instead of just non-zero. The original developer? I think he was ladder-climbing into management--probably just as well.

Maybe your software does happen to work correctly, but only when absolutely nothing unexpected happens, when all services and interfaces work properly and return only valid data, when nothing is at (or beyond) the boundary conditions, when all inputs fall within the realm of "normal" use.

And if something off-kilter does occur? Well, you have no idea how the software is going to react. Maybe it will crash, maybe it will start acting flaky, and maybe there will be absolutely no indication at all that something has been going awry for a long, long time.

What did you think was going to happen?

Summarizing the Bug Infestation Vulnerability Levels:
  • 0 - I have proven that nothing can go wrong (really, no joke).
  • 1 - I know everything that can go wrong.
  • 2 - I don't know what all can go wrong, but I'm pretty sure when it's going right.
  • 3 - Is this thing on?
The amount of time spent testing and integrating always exceeds the time spent coding. It's a tired refrain, but true nonetheless: Spending more time in coding, and conscientiously coding defensively, reduces test and integration time, and therefore overall development time.

Driving down your BIVL can actually reduce development time by markedly reducing the rework portion of the development schedule. There are techniques to drive that level down that result in increased code quality. Some are more formal than others, but the main requirement is recognizing that good software is difficult to write, and then being mentally prepared and disciplined and equipped with the right techniques and tools to manage that difficulty.

Good code is doable, and doable efficiently.

What's your Level?

Looks Like a Good Time for Techies to Take a Career Risk

Regardless of boom/bust cycles, technology jobs are, and show every sign of continuing to be, in high demand. Steve Tobak recommends taking advantage of an opportunity:

"You know why I'm confident that risk-taking is good for you? Because, my interpretation of the Labor Department's data is that you've got the biggest safety net of all time under you. Use it."

Samurai Programming

Excellent post with suggestions on getting good code out the door.

"* The Way of the Code Samurai *

Now, I don't actually know much about real samurai, but the basic thing I've heard is they stand and stare at each other for hours, and then suddenly BAM strike once and the other guy is down.

That's how you should code."

So I'm not the only one that paces around my cube, sketches on the whiteboard, walks the halls, and thinks for hours and sometimes even days about how to approach a hard problem, until...it's ready. I then sit down, fire up the editor, and go to it.

There's more, too, in that post, which closely align with the coding philosophies and techniques I've successfully used.

Sunday, October 21, 2007

A Christian Realist's (Brief) Perspective on General and Special Revelation

Back when I was a kid the minister or Sunday school teacher, I don't remember which, was talking about how God provided us with "General Revelation", which was the material universe, and "Special Revelation", which he memorably described by holding up a Bible and noting that he was holding the sum total of God's Special Revelation.

General Revelation tells us about God because God caused the universe to come into being and it therefore reflects His nature, and Special Revelation was explicitly provided to us by inspiration, dictation, and the observation and recording of historical events.

The two must be in harmony because God is rational and so it would make no sense for these two representations of God's nature to be in conflict with one another. Throughout history, however, there has been a perceived conflict between the Universe as we understand it versus the content of the Bible, a conflict that continues to the present day.

General and Special Revelation, while related, are distinct bodies of knowledge. Each informs the other, and our goal is to increase our understanding of them, coming closer to the fundamental Truths of each.

Our knowledge of what the universe is, from quarks to cosmos, evolves over time as we're able to build on the knowledge that was gathered before, and as our technology gives us access to information about the universe that was previously inaccessible.

Likewise our understanding of the Bible grows and changes over time, as our culture and society become more sophisticated: no more the explicit subordination of women, tolerance of slavery, or genocide. Arguing that the Bible is always taken literally, and not subject to interpretation, is specious: for example, very few conservative Christian churches implement 1 Cor 11:5-6 regarding the covering of women's heads in church.

Our knowledge of the universe is imperfect, and our wisdom in interpreting the Bible is imperfect, but in my opinion we have a good overall handle on both. When the two come into apparent conflict, research, reconsideration, and refinement of our understanding of one or the other (or both) is needed.

The material universe, and our knowledge of it, is objective. It can be observed, measured, and experimented upon. It works consistently and reliably in its framework of physical laws, and it truly has no secrets, only things that have not yet been discovered.

Special revelation provides an explanation and rationale for our place in the universe, it addresses existential, ethical, moral, and philosophical issues of what is humanity, and how we should then live.

The Special Revelation of the Bible builds on the General Revelation of the universe, so it cannot contradict it.

When the two appear to conflict, more research and study, in pursuit of a more complete understanding, is needed. The Bible "amplifies" the understanding and explanation of nature, it does not define it. The material universe is as it is, its reality is its own definition.

Astronomy and geology have determined that the universe is 13.7 billion years old, and the earth 4.6 billion, respectively, and these are objective facts. The "young earth" interpretation some make of the Bible's Genesis account of creation that concludes the Earth is 6000 years old flies in the face of the facts, and so is clearly wrong. The "six days" of creation obviously can't be interpreted to mean six literal 24-hour days, because that interpretation conflicts with reality. And that should've been the end of that debate once the ages of the Earth and the universe had gotten pinned down.

Putting primacy on one's interpretation of Biblical passages when discussing objective reality, i.e. the behavior and composition of the universe, is misguided, it's mixing apples and oranges. Too many Christians don't understand this, and think that citing the Bible to support one's belief about any particular subject, spiritual or material, is irrefutable proof that that belief is a fact. (In actuality, the facts of the nature of the material universe are right in front of one, and all that's required is a willingness to study, analyze, and understand. And maybe an NSF grant or two :-)

Since the Bible literally defines the Christian faith, citing the Bible is perfectly valid in that domain, e.g., "What is the Christian conception of 'Heaven'?".

But it's much less so when stepping outside that domain: "How do you know there's a heaven?" "Because the Bible says so." Such an assertion is completely lost on an atheist or anyone who doesn't acknowledge the Bible as an authoritative document.

Christians, especially conservative ones, don't seem to grasp this latter point--because that authority is an intrinsic part of their belief system, discounting it is inexplicable, and so it's concluded that the work of the Devil is the ONLY possible explanation.

For a person to become a Christian, they need to understand what the Christian faith is, and accept that it's a valid faith. Now one obviously uses the Bible to describe what it is, to show that it is internally consistent, and to show that it is consistent with human nature, but one can't use the Bible to "prove" the Bible. A person who moves from non-believer to believer may have it come on them like a bolt from the blue--an epiphany--or it may be a long, intellectual struggle that brings one to belief, or more commonly something in between, like the example of one's whose "life changed" due to the acceptance of Christianity.

The better that Christians understand this, and what the limitations of Biblical argument and interpretation are, the more effective they can be in bringing people into God's kingdom.

But the more they try to assert the preeminence of one's interpretation of Biblical passages that touch upon the natural world in a way that seems inconsistent with observed reality, the more foolish they come across (and I don't mean foolish in a humble, edifying way, I mean foolish in a "you're wrong" kind of way). And this just makes the task harder, because not only are Christians perceived as being out of touch with reality, but by not realizing it, and worse, unwilling to alter their beliefs in the face of reality, give the perception that Christianity is more about blind loyalty to a set of detached-from-reality beliefs, rather than the living, breathing, growing faith that it is.

Tuesday, October 16, 2007

Say Hello to my Leetle Friend

After an Alabama summer of chowin' down on whatever it is praying mantises eat, they're finally ready to tackle the big stuff by early fall--mice, small pets, irritating neighbor children.

This guy's every bit of six inches long, and cuts a sharp profile:

Off to the hunt...

Friday, October 12, 2007

"Dinosaur Programmers" Illustrated

The perfect illustration for an earlier post.

Titan's Lake Country

Here's a nice flyover of all the radar imagery of Titan's lake country, up in the North Polar region. Bring a jacket, it's cold--around -179 Celsius.

Ah, lakes and frigid temps, ya sure reminds me of home. :-)

Monday, October 8, 2007

Design Patterns Indicate Programming Language Weakness?

Mark Dominus at The Universe of Discourse examines the role design patterns are playing in software design today and draws some conclusions about what their use and promotion indicates about the state of programming languages.

My reaction to his conclusions is mixed.

Dominus looks way back to how the "subroutine call" design pattern eventually got subsumed into programming languages, thereby removing the need for programmers to explicitly code saving parameters and the return address, in favor of simply "making a function call".

Similarly, one can do object oriented programming in C by following a widely used "Object-Oriented class" pattern, which has since been directly incorporated into other programming languages via "class definition" semantics.

He concludes that "[design] patterns should be used as signposts to the failures of the programming language. As in all programming, the identification of commonalities should be followed by an abstraction step in which the common parts are merged into a single solution."

There's definitely merit to this argument, as evidenced by his citing the incorporation of subroutine and class "patterns" into programming languages. Even more advanced patterns, such as MVC, he observes are starting to show up in programming systems such as as Ruby on Rails.

The aspect that gives me pause, though, is just how where does one draw the line when considering whether to "incorporate a design pattern" into a programming language?

Nowadays, it seems like a no-brainer to incorporate subroutines, classes, and concurrency as built-in programming language features.

But is MVC an appropriate pattern to build in? How far should you go building in direct support for distributed processing into a programming language?

Left unchecked, using design patterns as a guideline for programming language evolution will result in the accrual of more and more language features, with a concomitant increases in complexity, specialization, and learning curve.

Are we considering leaving the era of the "general purpose" programming language behind? And what does it mean to the underpinnings of our software technology if we do?

Saturday, October 6, 2007

The Appendix: Boot ROM of the Digestive System

Some scientists think they may have figured out the purpose of the long thought useless appendix:

"Diseases such as cholera or amoebic dysentery would clear the gut of useful bacteria. The appendix's job is to reboot the digestive system in that case."

Solar Shingles(!)

While we're not looking to reroof the house for several years yet, the idea of using the roof for power generation is going to get a serious look.

While it's still pricey, though at least now in the "willing to consider" range, in ten years or so I'd expect it be quite cost-effective.

Wednesday, October 3, 2007

Dinosaur Programmers Know More Than You

Recently I spent several months terrorizing two software development departments, one responsible for developing the GUIs and the other the command and control functionality of a large, complicated, expensive data management and planning system.

If you're an enthusiastic, brimming-with-confidence, leading edge technology oriented software developer, you really don't want your software being wrung out by middle-aged dinosaur programmers like me. With hundreds of thousands of lines of code under my belt, major systems designed and deployed, obsessive software quality expectations, and complete disdain for programmer egos, I've got almost 25 years experience knowing what programmers forget, screw up, hope won't happen, and ignore. Yeah, because they're the things I forgot, screwed up, hoped wouldn't happen, and ignored.

I was handed the system's test procedures, got two days of hands-on training from one of the operators who was leaving the project, and then I was on my own. Following the steps in the test procedure worked more often than not, departing from them in free-play testing frequently didn't. It didn't take me long to start writing bug reports. Way too many were for stupid stuff, like not checking that the minimum altitude of a range was less than the maximum altitude, or accepting a manually-defined polygon where I'd provided only one or two vertex coordinates (a polygon requires at least 3 non-collinear ones--oh, and it also accepted 3 collinear vertices).

Using a conscientious programmer as an integration tester is a scary thing when it's your code he's testing (though it's a great thing if you're the test group manager, since the more bugs that are found, the easier it is to justify your job :-). In one instance the lat/long coordinates tracking the pointer on a map display were not being correctly transferred to the main operations windows when the mouse was clicked, they each differed by a few hundredths of a degree. The GUI developers kept claiming it was a mouse-pointer/screen resolution issue, and while such issues certainly do exist, this was definitely not one of them. I know how software works, I've dealt with resolution issues, and this was not a screen pointer/resolution issue, it was simply a matter of data not being properly transferred from one window to another. And eventually one of the developers did grudgingly dig into it and discovered that two different algorithms were being used to convert from screen coordinates to lat/long position, and that's what was causing the discrepancy.

While I'm well versed in what programmers get wrong, I also know what they (and I as a developer) need to have to fix things:

This system did a lot of logging, so I made sure to record what I was doing, what data I was using, the timestamps of when things went awry, and had all the log files collected together to turn over to the development group.

Having been the victim of inept testing I also know the number one thing not to do: If a bug has ripple effects, don't document the ripple effects as separate problems. This drove me insane when testers did it to me; when something breaks, fine, write it up, but don't keep trying to follow your test procedure exercising what you just discovered to be broken functionality! Each problem that gets written up has to be dispositioned, which takes time and effort, and there's no value in documenting and dispositioning each of the myriad and marvelous ways in which failure manifests itself. The tester is wasting time documenting things that will all go away once the root problem is fixed, and while bogus problems are being written up, the looking for more actual problems isn't happening.

A few weeks after I came onto this project the developers all pretty much stopped arguing with me.

Eventually I moved on from that project and got tasked to "web enable" a large simulation system (ah, back to development!). The first thing I start thinking about is how to go about embedding a web server within this system and how it's going to communicate with its clients, and then right on the heels of that began worrying about everything that could go wrong and how I'm going to handle those situations. What if the server goes down in the middle of a transaction? Or the client? Or the network drops out? How will shutdown take place in a half-hosed environment, for both client and server? How about recovery? Restart?

Dinosaur programmers spend far more of their time dealing with cranky software than they do with software that just works. Their jobs usually is, in fact, converting bad software into stuff that just works. And it's not AJAX, or dynamic typing, or service-oriented architectures, or parallelism, or the latest/greatest programming language that's going to make that happen.

It means taking a cold-blooded, hard look at software development practices and acknowledging the fact that writing brittle, fragile software is easier, and so that is what is done. The programmers who come to terms with this reality gain a fine appreciation for high function software that "just works", and so are unimpressed by whatever hot new technology is on the bleeding edge and is prognosticated as going to "make writing software a breeze". Because it won't.

I once fought a programming language war against two system engineers who wanted us to recode our completed, reliable, correctly functioning command and control system in C++, for no other reason than "that's where the market is going." Less than ten years later anyone citing that as a reason for such a recoding would be laughed out of the office.

Every new technology sounds great, and is capable of great things when it's optimally applied. Which it almost never is. And nothing brings out the imperfections of a new technology like real world usage, in a mission critical environment, with money on the line.

So you'll forgive me when I don't get excited about your new framework/language/architecture/process that's going to change everything/usher in a new paradigm/launch the next Google. As long as people are writing code, and assembling component frameworks, and modeling the business processes, they're going to continue to forget, screw up, hope for the best, and ignore the vague and uncertain parts.

While a new technology might eliminate a whole species of software development errors, you'll just discover, and now have the opportunity to explore, a whole new jungle of failure.

Dinosaur programmers already know this, it's our home turf.

Monday, October 1, 2007

Censoring public condemnation of torture??

Bono, of U2, debt relief, and poverty fighting fame, just recently received the National Constitution's Center Liberty Medal for 2007.

The video of the event, which obviously includes Bono's acceptance speech where he deplores torture and its acceptance by a significant fraction of Americans (38%) has been edited to have his condemnation excised.



The complete, unedited version of Bono's acceptance speech is now available at the Liberty Medal web site.

Okay, I've seen the explanation, but there was certainly something odd about the original availability of the video.

Chicken, and just Chicken.

No "Duck".

No "Goose".

Friday, September 28, 2007


I've gotten into arguments with other developers on more than one occasion about the value of program assertions.

(By the way, I'm talking about the use of "assert" macros/functions/pragmas in the Ada, C++, and Java programming languages. These languages pay my salary, so while I know about Haskell, Erlang, Eiffel, OCaml, Ruby, etc. I have no call to use any of them. So...YMMV on some of the details.)

Let me try to summarize the anti-assertion position of the last heated discussion I had about this, which was with a very talented C++ programmer:
  1. Assertions are almost always compiled out of the released code, so they're USELESS!
  2. If you're asserting something because you think it might break, there should be an explicit check for it in the code, so assertions are USELESS!
Too many programmers can't think beyond the code, so assertions are thought of as just a coding practice of questionable utility. After all, they're normally not going to be in the released version (see 1), and if you do decide to compile them into the release version, there's nothing to be done if they happen to trigger (2), so what's the point?

You need to step back from thinking of program assertions as just code.

The purpose of effective program assertion practice is embedding encoded information about requirements, design, and implementation assumptions into the code, as code.

Assertions are not to be used to error-check code execution, they're meant to be used to help verify that the software is implemented in accordance with its requirements.

What is being asserted in the following code?

assert(hnd != NULL);

That a pointer is not null, right?


I'm asserting a part of the design specification, the part that specifies that my function will only ever be called with a valid handle. I encode it as a null pointer check, but the purpose is not to check the pointer, it's to verify my code (and my caller's code) as its pertains to the design spec.

Conscientiously crafting meaningful assertions embeds a portion of my understanding of the functionality of that code...in the code.

The assertions themselves can then be reviewed by inspectors or peer reviewers, or the system engineers or architects or whatever. This gives them a means to verify that I have a correct understanding of what this piece of code is supposed to do. The assertions encode requirements and design aspects in the medium of code, and those notations of my understanding can be checked for accuracy. Do the asserted ranges for guaranteed inputs match the spec? Do my assertions reveal that I'm not allowing for the full range of allowable inputs? Are constraints being asserted that I should in fact be explicitly checking?

Embedding design-oriented assertions into the code aids the inspectors or peer reviewers in verifying that the implementation matches the design.

Embedding implementation assumptions as assertions helps inspectors verify that the code corresponds to the requirements, design, and the implementation assumptions.

Assertions are intended to capture the developer's understanding of what a given piece of code is supposed to do--they're not a coding thing, they're a correctness thing.

Tuesday, September 25, 2007

"Siriusly" Drunk Frog

The Night Blooming Sirius (or Cereus) is a straggly looking plant with a big white, heavily perfumed white flower. They bloom exactly once a year, at night, usually opening up between 9 and 10 pm, for a few hours.

We had five go off last night. Here's a view of one with an added audience member (look closely at the center of the pic, above the bloom). These things have a seriously strong perfume, so we're talkin' Grateful Dead ambiance here.

Saturday, September 22, 2007

Confessions of a Terrible Programmer

I’m a terrible programmer. You would think that having done this for nearly 25 years that I should be pretty good at it by now. But nope, I still put bugs in my code, use the wrong types for variables or subprogram arguments, forget to make the right library calls, mess up the expression in an if statement, and on and on and on now for 25 years.

Sometimes the code won’t compile, and I can fix those bugs right away, but the rest of the time I have to actually run the program to be made aware of my failings. If I’m “lucky”, the program will start up and then crash in a blaze of error message glory that I can then pick through and use to figure out what I did wrong. If I’m not so lucky, the program will start and then almost immediately exit (assuming that wasn’t what it was supposed to do), or go off into la-la land, neither doing what it’s supposed to do, nor exiting. Or worse, look like it’s working just fine, and only when I’ve worked my way to some corner of the program’s functionality will something go subtly awry—occasionally so subtly that it’s not even noticed for awhile. Then of course there’s the non-deterministic bugs that manifest themselves only when a sequence of events (which may originate externally to my program) occurs in a particular order over a particular set of time intervals. The program may run right 999 times out of 1000, but that 1000th run may lock up the entire system. (Been there, done that, found the vendor library’s bug.)

The odd thing though about the programs I design and write is that once they’re delivered to the customer or as a baseline release they usually work like they’re supposed to, which would seem to be surprising given that such a terrible programmer wrote them. I mean I usually deliver the system on time, with the required functionality, and with very few bugs.

So what’s my secret?

Well, it’s one of these pseudo-Zen made-up Ancient Software Master’s secrets:

You will never become a Great Programmer until you acknowledge that you will always be a Terrible Programmer.

And as you saw in the very first sentence, I fully concede that I am a terrible programmer. So since I love programming and this is my chosen career, I have to deal with it.

Fortunately there’s a lot of ways I can cover up the fact that I’m a terrible programmer:

1) If I get to pick the programming language for a particular program, my “go to” language is Ada. Yeah, I know it’s pretty much unheard of today outside of the defense industry, and even there it's struggling, but it has few peers when it comes to pointing out to a terrible programmer just how bad they really are. With all the language’s typing and run-time checks, it’s little wonder that programmers didn’t like Ada, since it was constantly pointing out bugs in their code!

If I can’t use Ada, well, then Java is the next best choice, since it’s got a lot of the same built-in bug detection capabilities.

But please, please don’t make me use C++ any more, which is definitely the enabler of terrible programmer egos. It lets most everything through, so a) I feel good about getting my code clean compiled and “done”, and b) I get a huge ego boost after the Jolt-fueled marathon debugging session that's needed to successfully get the program mostly working.

2) I use programming assertions, a lot. If some variable should never be null, I assert that it’s not null, even if Grady Booch has sworn on a stack of UML standards that it will never be null. Here’s how paranoid I am about being found out: even when I control both sides of an interface, I’ll assert the validity of what the data provider is providing, and assert what the receiver is receiving. Why? Because I’ve been known to change one side of an interface and forgotten to alter the other, so the assertions catch those for me right away.

3) Be a testing masochist, i.e., I ruthlessly try to break my own code. No programmer likes doing this, we’re usually happy to just get the damn thing running the way it’s supposed to with the test case data, so why go looking for trouble? I admit that it’s a mental challenge to intentionally attempt to break one’s own code, but if I don’t, somebody else will, and my secret terrible programmer identity would be revealed.

4) Have other programmers inspect my code. Despite everything I do, as a terrible programmer there’s still going to be bugs in my code. So I need other programmers, especially other Great Programmers, to look at my code and find those bugs. And I’m serious about these inspections, I don’t want some cursory glance that simply makes sure I’ve included the right file header information, I want an inspection, dammit! I had to pound on one of my inspectors once because he was “doing me a favor” by not recording all the defects he found, because he didn’t want me to “feel bad”. Look, I have no self-image as a programmer, so “feeling bad” isn’t something I get hung up on. I need to end up with Great Code, so as a Terrible Programmer I need all the help I can get!

With these and other terrible-programmer-obfuscating techniques, my code often does come out looking like it was written by a great programmer. And I’ve done well by those techniques over the years, during my career I’ve had the opportunity to work on the real-time executive and aero data analysis for flight simulations, did a clean sheet redesign of a major defense system’s command and control planning system, taken sole responsibility for rehosting a large command and control system, done data mining and event correlation in an XML framework for another large defense system, and replaced a legacy wargaming operator console with a net-centric ready version. Outside of my day job I write open source software (in Ada!) and some of it has been picked up and used by a company supporting the European Space Agency. (I’m goin’ to the moon! Helping, anyway.)

So I’ve been successful in hiding the fact that I’m a terrible programmer up to this point, and I need to make sure that I can continue in this career path until my retirement. Heeding Han Solo’s admonition ("Great kid, don't get cocky."), we come to the second fake Zen Old Software Master italicized aphorism:

You will remain a Great Programmer for only as long as you acknowledge that you are still a Terrible Programmer.

Over the years I’ve learned more ways to hide my programming failings. One technique: let the program crash—This works, just stay with me. I’d joined a development project where the original developers had so many problems with run-time exceptions that they simply starting including “null exception handlers”, i.e., catch all unexpected exceptions, suppress them, and keep on going. Needless to say, the system would run fine for awhile, sometimes hours, and then slowly start to...hmm..."veer off".

When I got the chance to redesign the system, one thing I immediately forbade was those null exception handlers (though handlers for recognized exceptional conditions were permitted). If an exception occurred, I wanted it to propagate up until it took the application down so that we would know about it and could fix it right then and there. I got my wish, but was almost found out in the process. The program manager got wind that we were seeing a lot of system crashes during testing, and he wanted to know why the redesigned system was crashing so much more often than the one it was replacing. Explaining that we were uncovering errors and getting them fixed now, and fixed right, didn’t really give him the warm fuzzy he was looking for, as there were deadlines approaching and the reported error rate wasn’t declining as fast as he liked. The team stuck with this practice, though, and when we were done (on time and on budget) the system ran reliably and correctly for days under both light and heavy loads.

For me, this cemented my self-image as a terrible programmer. If I could bring this kind of a system to fruition, meeting all its requirements and budget and schedule constraints, just by conscientiously applying a slew of techniques for uncovering my programming failings before they got into the final product, then I knew that I would be able to fake it for the rest of my career.

Now if you look at my code you might find it to be pretty bad, and that’s normal, pretty much any programmer that looks at almost any other programmer’s code will judge it to be a pile of offal and then proclaim that the only way to fix it is to rewrite it. (I wonder why this is?)

But if you want to sit down with me and go over my code, and tell me what’s wrong with it, then I think we can work together to fix those problems and make it truly great code. And who knows, maybe you’ll end up being as terrible a programmer as I am.

Thursday, September 20, 2007

Choose Your Caulk Wisely

You know you've gotten way too much experience with home maintenance when you can not only distinguish between different types of caulk, you also acquire a preference:

GE Silicone II Window and Door Caulk.

Works indoors and out, goes on smooth, and cleans up nicely.


Wednesday, September 19, 2007

"Big T" Technology vs technology

It's certainly no news flash that new technologies get hyped, witness Object Oriented Programming, "Write Once, Run Everywhere" Java, and the Iridium satellite phone system (launched November 1, 1998, went into bankruptcy August 13, 1999). While all of these particular technologies are now in use, what's expected of them has all been severely cut back to realistic levels.

When technologists and developers fall in love with a technology, what they're really getting enamored of is a "Big T" Technology, which encompasses their vision of what a technology can provide and how it can change the world. And not surprisingly the passion of this attachment can both blind its promoter and, while perhaps admitting a surface awareness that not everything about the Technology is perfect, be utterly convinced that it can change and result in a perfect marriage.

(I don't want to convey the impression that all big-T Technologies are overwrought and destined for disappointment, the World Wide Web is certainly such a Technology, and it has changed the world.)

In the POET framework, the 'T' that's being addressed is, if not a world-changing Technology, at least intended to be a business-changing one.

My concern about this is that the small-t technology that's the foundation of the big-T Technology gets insufficient attention paid to it.

As an illustration, business executives doing strategic planning may decide that they need to provide (big T) Web Services employing a Subscription Model for their customers' Mission Critical Data. There, now go work the Politics, Operations, and Economics. The tech'll be purchased off the shelf and we'll have their consultants work with our people to roll out the whole thing. Obviously without the "tech" none of this will work, and Joel Spolsky of Joel On Software provides some insight as to what it really takes to "roll out the whole thing". See also "The Price of Performance" in ACM Queue for another look at how an insufficiently broad view of technology would have had costly real-world business impacts to Google.

In these two references, insufficient awareness and addressing of low-level technical issues would have had significant business impacts. But you might argue that these were caught by the POE process, so everything worked out as it should. However, Joel Spolsky is a tech-savvy (note the small 't') CEO that still writes code for his company, and Google is, well, Google.

Other companies do their due diligence to find the right Technology to implement their business processes and system architectures and still wind up with Service and System failures because the software implementing the Technology is full of bugs.

The Technology aspect might be asserted to be "important", but it's still last in priority, meaning that to those working the project management issues..it's not important, and the result of this is that the technology gets treated as a commodity, to be selected primarily based on features and price (and marketing).

I'm not saying that a corporate VP needs to ask what programming language or networking protocol a given product is using (though it would certainly catch the vendor's attention if they did), but someone in the technology evaluation hierarchy should. Along with asking for a description of the vendor's development process, their QA (Quality Assurance) and CM (Configuration Management) practices, and their customer bug reporting and fixing process. It doesn't need to be CMMI or ISO certified, but it should certainly be coherent and plausibly capable of resulting in the production of good software.

It's more than getting past the marketing hype of Big-T Technology, most everyone tries to do that now. It's a matter of respecting, understanding, and acknowledging the criticality of the foundational technology that underlies the Technologies you're going to use, not treating it as a commodity, and feeding that into your decision making process.

And as I posted previously, I'm concerned about what's happening to the quality of the foundational technologies as they're continuing to evolve.

Friday, September 14, 2007

The "Surge", the "Dollar Auction", and a Useful Concept

Oliver Goodenough, a law professor at Vermont Law School, talks about a standard game economics professors use to "demonstrate how apparently rational decisions can create a disastrous result." The game is called "The Dollar Auction". I won't go into its details, since the Wikipedia link has a nice, succinct explanation.

Goodenough uses the Dollar Auction to illustrate how we got into such a bad situation in Iraq, and how the Surge is effectively just another bid in a now irrational auction.

A Dollar Auction is easy to understand, and seems be a useful conceptual tool for understanding how certain awful outcomes can result from seemingly (or genuinely!) rational actions.

Tuesday, September 11, 2007

Wednesday, September 5, 2007

Viable Fusion in our Lifetime?

Here's hopin'. Unfortunately you can't help but take a wait-and-see attitude :-(

70% of the Software You Build is Wasted

Hey, I'm just quoting the title of the article.

He certainly seems to be on the right rant to me, and Dan, I suspect you'll agree as well.

The Fundamental Theory of C

Whenever I describe to someone what I believe to be the Fundamental Theory of C I always have to first caveat it that I am not denigrating the language.

So, that the Fundamental Theory of C is "Portable Assembly Language" is not a knock on C. Now let's move on.

There are some tasks where a very low-level programming language really does make a lot of sense, e.g. operating system kernels, device drivers and graphics buffer drawing primitives. You're working at a very low-level, right on top of the hardware where performance is critical, and you're trying to layer just a facade of software across the hardware to provide that first level of software glue that higher level abstractions can then build on. If this bottom layer employs too much abstraction, it becomes difficult to verify correctness because the gap between software and hardware functionality begins to blur and it becomes more difficult to map software functions to hardware operation. (Of course this blurring is a good thing as you move up in layers of abstractions out to distributed systems, Web services, and SOA.)

So in a way the ideal choice of language for this kind of low-level programming of hardware is assembly language, which provides complete, detailed, total control of the hardware. The problem with this, of course, is that assembly language is hardware specific and therefore inherently non-portable.

So this is where the C programming language comes in. Written well, C is highly portable, as far as the language itself is concerned. Yet you have full access to the underlying hardware's interfaces, with very little compilation needed between the software and the hardware, giving you straightforward traceability from C to assembly code to hardware. Yet you're not constrained (much) by the specifics of the hardware's architecture--you don't have to work with registers and offsets and address modes, so in a way you have the ability to define your own architectural approach for interacting with the hardware (the Linux kernel obviously being the biggest and best example of this).

C is a computer programming language, meaning that it is optimized for programming computers in terms of their computational components. C has built-in primitives for direct memory access, bit shifting and rotation, increment, decrement, indirection, multiple indirection, unsigned arthmetic, etc. When you program in C you're programming a computer, to tell the computer what to do. Again, see Linux.

Understanding that the Fundamental Theory of C is that it is a portable assembly language, and what that means, directs the developer to the kinds of tasks for which the language is best employed and the developer mindset to have in place when writing the code. C is appropriate for tasks where hardware interaction and low-level "bit-twiddling" are called for, but the very characteristics that make it ideal for those kinds of tasks seriously detract from it when dealing with abstracted entities that have little "computerness" about them.

Higher level languages involve writing programs that manipulate classes, records, data structures, components, services, and suchlike abstract entities. Such programs run on computers, but you're not programming the computer for them.

Using C for programming a computer, having consciously selected that language as a portable assembly language suitable for a specific and appropriate set of computer-oriented tasks, maximizes the benefits C provides to the developer, and the correctness and success of the computational foundation upon which higher-level components and systems, using higher level languages, can then be built.

Rudy? Really? When Democrats and Republicans agree

And not just Democrats and Republicans, but New York Democrats and Republicans...

This should give one pause: "A Giuliani presidency would be ..."

Z-Big Raps Political

I was too young at the time to know whether Zbigniew Brzezinski made a good National Security Advisor, but I took pride in not only being able to spell his name, but being able to pronounce it.

He couldn't have been too bad at his job, since he keeps showing up on news shows as a foreign policy expert. Here's his take on how American foreign policy needs to be changed by the next president.

Hint: We'll need more than just a rollback of what will have been 8 disastrous years.

Your Daily Dollop of Escher and the Doctor

From some guys that have too much time on their hands, but spend it well:

Here's the source posting I saw for this.

This reminds of one of my favorite Dr. Who episodes, Castrovalva. Here's the work that particular episode used for its locale:

Lots of M.C. Escher stuff here.

Tuesday, September 4, 2007

"Picking the Right Tool" is a Tautology

On any given system development project, the individual or team charged with "tool selection" is always going to pick "the right tool". No one is ever going to think "I'm intentionally picking the wrong tool for this job." They might knowingly opt to pick one that is technically inferior to others, but the justification for doing that comes as a result of considering the Political, Operational, Economic, and Technology (POET) factors, making the overall selection the "right" one.

(In hindsight it might be discovered that a choice was "wrong", but that's because some aspect of P, O, E, or T wasn't correctly understood, i.e. "it seemed like a good idea at the time". And it's also possible that those not in on the selection process may assert that a wrong choice was made, but well, then, make the case in the POET framework.)

My concern is that over the last several years the underlying technology is getting more and more precarious, more and more ad hoc, and getting less and less respect.

Many years ago I read an article espousing a similar concern. I don't remember the exact paragraph wording, but the gist of it was that there are people that constantly worry a great deal and think very deeply about very complicated and arcane things going on in the bowels of a data center whose failure could bring down the company, or even a chunk of the economy. What they do is not sexy, or cool, or cutting edge, and their work doesn't lead to startups and high-demand IPOs. But without their skills and commitment to do what they do the whole thing starts to fall apart.

That kind of work is at the very bottom of the POET stack, it's "sub-T" in fact. Someone is writing OSes and compilers and JIT JVMs and JDKs and web servers and fault-tolerant middleware and Perl interpreters and JavaScript engines and XML parsers and on and on. Everything in the technology stack, and the system development stack, rests on them.

And "picking the right tool" blithely assumes that these technologies are commodities and it's simply a matter of performing a suitably comprehensive trade study.

But I don't like what I'm seeing down here at the bottom of the stack.

I see programming languages getting hacked up to include new features that show little regard for maintaining a consistent and coherent language design, I see security being "patched in" to systems rather than being built in, and I see more and more features being packed into products that make them increasingly brittle. The systems are arguably "better" than what they were before, but I believe it is becoming a matter of plugging holes in dikes and building McMansions on landfills.

As Dan mentioned, technologists and developers almost always naively put their 'T' at the top of the stack, which sets them up for failure in a business goal oriented environment. Unfortunately, by kicking 'T' down the stack the acknowledgment of its criticality got demoted as well, so much so that technology evaluation becomes a matter of feature and price rather than quality engineering.

If you don't recognize the need to truly engineer a technology and acknowledge that technology choice actually does matter, and that tech is not a commodity, but is the foundation on which all your Service Oriented Architectures and Business Processes and Customer Relationship Management is based, your system is going to end up in pretty "POE" shape.

Monday, September 3, 2007

POET - Politics, Operations, Economics, and Technology

One of the best lessons I have learned over the past 7 years as an enterprise architect with Northrop Grumman is the POET acronym. It is the notion of the order of things in the business world: 1) Politics -- getting people what they want, 2) Operations -- making the business run like a well-oiled machine, 3) Economics -- getting the most bang for the buck, 4) Technology -- using machines to make an organization more efficient and accurate. As a technologist, architect, and software engineer, I used to spell this as TOEP, where Technology was #1 followed by Operations, then Economics, then, dead-last, Politics. Following this TOEP priority system working for the government, I set myself and my team up for failure. We were technologists pushing hot technologies in an organization that could care less. We were "techology" in search of a solution.

Putting the POET acronym in order is a good recipe for success. Politics forces you to look at what is driving the business decisions. Operations forces you to focus on efficient well-targeted business processes. Economics forces you to maximize return on investment and lower the total cost of ownership for a system. Technology is dead last. It's a servant. I have been well-served by becoming technology agnostic, meaning, technology services an organization's politics, operations, and economic constraints.

In my experience, this is not trivial, it's monumental. It pervades many work spaces and many fields of battle. It is like holding your friends close and your enemies even closer. Many techologists are in love with their favorite technologies. They could care less about the business world that surrounds them and pays them to build excellent systems. I used to be one of these people. POET is a wake-up call to put first things first, a recipe for success.

Friday, August 31, 2007

The Fundamental Theory of Ada

About 20 years ago, in the late 80s, I was at an Ada Conference attending a seminar being taught by Doug Bryan, who was then at Stanford University. Bryan was talking about how to get the most out of Ada, of which a critical aspect is understanding the underlying design principles of the language. The key concept he was trying to get across in this seminar was that Ada's design centered around a "type model". If you understood how Ada's type model worked, he said, then you understood why the language came out the way it did, and how to work with the language, instead of fighting against it.

While I'd been programming in Ada for a few years by then, this was the first time that I truly realized there might be something underlying and guiding the definition of a programming language, that there might be more to creating a programming language than getting a bunch of people together to make up declaration and statement and packaging syntax and semantics.

The realization that a type model for Ada existed turned out to be the genesis of what eventually coalesced into my idea of a programming language's Fundamental Theory. Today I recognize that the Fundamental Theory of Ada is its type model.

By enshrining the type model as its FT, Ada allows a huge amount of information to be intrinsically embedded within Ada programs with a simple type definition. This information is then readily accessible to the developer through the use of Ada's attributes.

Take a simple example:

type Speed_Range is range 0 .. 1000;

With nothing more than a reference to an object of that type:

Speed : Speed_Range;

One can know its minimum value (Speed_Range'First), maximum value (Speed_Range'Last), the minimum number of bits needed to represent all possible values of the type (Speed_Range'Size), the actual number of bits representing a variable of that type (Speed'Size, which is often larger than the type size since objects almost always occupy a whole number of bytes), the number of characters needed to represent the longest possible string representation of values of that type (Speed_Range'Width), etc. You can convert values to and from strings (Speed_Range'Image, Speed_Range'Value), do min/max comparisons (Speed_Range'Min(100, Speed), Speed_Range'Max(Current_Max, Speed)), and use the type as a loop controller ("for S in Speed_Range loop" and "while S in Speed_Range loop"), and more. And none of this information needs to be explicitly programmed by a developer, it is all implicitly provided by the mere definition of the type.

Different kinds of scalar types may have these and other attributes appropriate to the nature of the type: floating point types have attributes for identifying the underlying constituents, accuracy and representational limitations of a floating point type definition, while enumeration types also have attributes to access all positional aspects of the type's values.

Ada's type model extends well beyond simple scalar types: inheritance is obviously based on inheriting from other types via the "tagged" modifier, static polymorphism comes through generic (template) instantiations with the desired types, and dynamic polymorphism is provided by tagged, "abstract" types. Even Ada's concurrency constructs are type-oriented, i.e. task types and protected types (while singletons of each of these can be declared, what goes on under the surface is that a corresponding anonymous type is implicitly defined of which the singleton is the only object). By declaring a subprogram profile type, subprograms themselves are treatable as type objects.

The supporting infrastructure of executable constructs, i.e. the statements, are all then integrated with this type model, and able to exploit the information that is an implicit part of each type, for example in the "loop over the type" statements mentioned above, as well as is done with array handling:

for I in Race_Speeds_List'Range loop ...

There is one area of non-conformance, though. For some reason the whole exception mechanism was not well integrated with the type model. There's nothing that can be construed as an "exception type" in the language (although an exception type mechanism was proposed for the Ada 2005 language revision, it was rejected). So there's no hierarchy of exceptions, no inheritance, and while the Ada 95 revision did incorporate the creation of exception "objects" to which data could be attached (which must, though, be in the form of a string), exception handling is recognized as admittedly inferior to that of most other commonly used languages.

Despite the exception deficiencies, the Ada standard has conformed very well to its fundamental type model theory throughout its evolution. Even when major revisions were made during the Ada 83 to Ada 95 revision process to incorporate full object orientation, that capability was defined as an extension of the existing type mechanisms. (And the fact that improvements to the exception mechanism center around "type-ifying" exceptions show that the type model is recognized as the driving factor of the language's design.)

Understanding and exploiting the Fundamental Theory of Ada leads to more reliable and more efficient software. One issue programmers new to Ada invariably run into and complain about is the preponderance of compilation failures due to "type mismatches" in their code. "Why can't I multiply velocity by a time interval and add it to distance? They're all just numbers!" That's just it, they're not just numbers, they are abstracted entities, each with their own characteristics and constraints, modeled in the language's typing system, and that are only allowed to interact in well-defined ways. Other languages model entities this way, too, except that they require the developer to explicitly model them as classes, while in Ada the language itself effectively performs the type modeling for these kinds of quantities.

By understanding and exploiting the Fundamental Theory of Ada, by understanding what an Ada type represents, an abundance of information about the problem domain can be embedded within the software, can be easily exploited, and can lead to the efficient production of reliable software.