Representation with Insufficient Dimensions

As a kid, I had a friend who spoke with a New York accent.  He didn't pronounce the "H" in his "Hu"s.  We can hear this when Trump speaks, "Huge" become "Yuge."  Among our friends was a boy named "Hugh."  At one point my friend was reminiscing and asked me, 

"You remember when Hugh XYZ'ed? That was so crazy."

To which I responded, "I never did that."

"No Hugh did."

"No I don't remember ever doing that.  Seriously."

"No Hugh did, Hugh."

"Not me man, you're crazy." 

 At last, he added Hugh's last name, I caught on, and we had a good laugh.

 

I see this same pattern frequently in software and it goes like this:

  • Observing the data, an engineer notes that two concepts behave uniformly
  • Not being a subject matter expert, they make the decision to represent both concepts as one and the same.  After all, the parsimonious solution reduces space and complexity
  • Some time later, data that violates their assumption appears and breaks their model

This is caused by a non-subject matter expert making assumptions about the problem domain.  It often appears as tech debt.  The fix can be pretty high effort.  Moving from low dimensional representation to higher is generally expensive.  For a production system, in addition to adopting the higher complexity transformation in the software itself, the historical data must also be remediated.

This runs contrary to our intuition about parsimony.  When in doubt best to design for the higher dimension representation.  Even if your SME tells you, "it'll never happen," with enough time it may.  As long as the higher dimension model isn't prohibitively expensive, using it buys you the option to accommodate unknowns with minimal effort or risk. 

Comments

Post a Comment

Popular posts from this blog

Engineering Truisms

This is a brief collection of quotes, laws, and truisms that I've gathered over the years.  I've kept those that have consistently proved themselves across organizations and projects during my career.
Read more

The Telescoping Constructor (Anti-Pattern)

Image
The Telescoping Constructor Looks like this: Widget(varA) {...} Widget(varA, varB) {...} Widget(varA, varB, varC) {...} or Widget widget = new Widget(x); widget.setVarA(...) widget.setVarB(...) widget.setVarC(...) There are both shared and specific problems for each variety.  Let's talk about both. The 1st case, supernumerary constructors. What's wrong with this? Well, the good news is you provide  quiet consumption since there's a bespoke constructor for every use case.  Rather than have noisy construction, where the consumer must fill in a bunch of nulls, you move the noise into the implementation.  We shouldn't make assumptions on how many times the class will be used, so keeping the complexity to ourselves is generous and will lead to a more maintainable macro system. However, one problem here is that the end consumer is not restricted in their choice of constructors.  Like shopping at the super ...
Read more

The Importance of Pace

Image
  In The Simpson's episode Homer's Enemy ,  Frank Grimes is a unlucky man who's worked hard his life but failed to reap any of the benefit.  When confronted with the inequity of his circumstances compared to Homer's he breaks down into a self destructive fit of jealousy that eventually ends his life prematurely.   The episode is a 1990s comedic expression of the general observation that life is not fair.  Several decades later, as income inequality has increased it's become a popular talking point that, "Hard work alone doesn't make you successful."  In Gen A slang a, "Try Hard" is someone like Frank Grimes who does himself discredit by giving it too much effort. For us conscientious types, this is not only counter intuitive but also elicits an emotional reaction.  Folk heroes like Rocky show us that while God may not have blessed us with any particularly useful talent, we can make up for it for it with heart.  Hercules taught us that even wi...
Read more