awesomejohnnyhifive - Tumblr blog

[awesomejohnnyhifive]

Empowering teams with Architectural Tools

If you haven’t had a chance to go back and read Jeff Darcy’s article on scalable server design, think about whether you have time.  It’s good.

One of the things I got from that article that I have not ready elsewhere is the methodology he uses to find deadlock or resource contention in server products.  If you didn’t pick it out or read the article, it’s pretty simple and I’ll repeat it here as succinctly as I can.   

His advice is to draw a simple XY chart, and put along the X axis stages of your message processing.  On the Y axis resources that your system can access that are contended or finite.  Draw X’s in the graph where you have intersections.   Your goal as a developer / architect is to design systems that have a straight line of X’s top to bottom, in a clean diagonal line.   If you don’t then consider breaking up stages or resources to eliminate the bottle neck.  If you have a perfect line, no contention, and you can look at your other scalability issues. 

I have used that methodology successfully for years ... it’s simple, clean and it models your software architecture in a way that communicates to people both what their system looks like, and what you are trying to achieve.   It’s part of what I’ve learned the hard way over time, diagrams that both communicate and are working tools are very hard to come by.  In my experience people don’t make a lot of meaningful decisions on classic ‘layers & boxes’ diagrams over time.   They make more in sequence diagrams when they take the time, and they make a lot of them on whiteboards with erasers ... 

Coming back to the topic of complexity, why would this be relevant to my posts on complexity ?  The first thing is it provides a diagram in an architectural sense that is sequential and easy to follow with teams.  Teams could look at the diagram, and realize what they needed to do to reduce complexity and why, and as importantly where.

In my experience, that is huge.  I’ve always believed that empowering teams to achieve is the surest and sometimes surprising path to success.   This is a step a slightly different direction, providing a tool that teams can use to drive down complexity that isn’t a metric from a static analysis tool does not happen all that often.  And you can do it on a ‘napkin’ with anyone that has a rudimentary understanding of how their software works.

The second impact of that tool on my thinking is going to take another post, it’s  a bit more involved.   The very first bit of training I did as an engineer out of school was on reliability engineering coupled with a bit of six sigma, and it’s strongly influenced how I think about software engineering and problems in general.   The tool is one of the first sequentially focused architectural tools out there I’ve seen in software architecture, which makes it very susceptible to reliability modeling and analysis. 

I’ll pick that one up another night. 

[awesomejohnnyhifive]

On the bench or in the game...

I’m working on that complexity article, but it’s not ready yet.  I’ll get back to it, but in the meantime I wanted to link to a favorite article of mine that influenced me very early on.   It’s an article on server performance by Jeff Darcy, and if you haven’t read it before, and you like software performance, grab a tea / coffee / frosty beverage as it suits you and go read it.  It’s really good.

http://pl.atyp.us/content/tech/servers.html

Stepping away from architecture and software topics, I had a lot of conversations on leadership this week.   For my career the questions of leadership and design always cross paths, because in part I think I really like doing both.  As I’ve met people in software, I have come to realize that this is actually pretty common among leaders and I think ultimately good.   You need leaders that understand design to make good decisions in some matters, it’s part of the ‘talent stack’ good leaders need to bring to the table to lead an organisation.

The story that comes back to me is from one of my first managers who said the analogy that always resonated with him was team sports, and particularly soccer.  He said he always pictured it as a team sport, where you have the players in the game and the team manager, and a very singular question you have to answer when you picture that.   Are you the kind of person who likes to be on the bench, and when called upon makes the play that wins the game ?  Or are you the person that calls the players off the bench and gets the barrel of gatorade for your trouble.   

Not everyone wants the gatorade.   

It’s a fundamental question, and I think your answer to that changes over time as you get older, and as industries change.   It’s worth asking though as you go through career transitions.   There are other metaphors our there, but that one has always stuck with me.

[awesomejohnnyhifive]

Complexity - Framing Features with Economic Arguments

My “Second Post” said I would write more about this.   Yeah ... so ... 

Arguing complexity with executive leaders feels a lot like an exercise in rope pushing.  They are listening.  They understand, or at least claim they do.   The conversation never goes far though. If complexity is by product of a customer requirement, every executive will end that conversation early with “it is our job as engineers to deliver and solve that problem”.

 What to do then ?

I haven’t found a silver bullet to this problem of articulating problematic product complexity.  I have invested a lot of time with development teams developing clean, layered architectures to make the best of what the requirements give us, and the investment is usually well worth it.   You can even ‘see’ where the architecture needs improvement from bug densities, rework the architecture in that area, and good things generally happen.

Again though ... indirect measures on a tangible problem.   This software cost too much, so we fixed some coupling metrics and it got better.   Why ?

I have a theory though I’m trying to develop.  I have a manufacturing background, where I was privileged enough to spend a lot of time around reliability engineers and some exposure to ‘Six Sigma’ tools ... a.k.a ‘statistics’. Six Sigma people would spend a lot of time on manufacturing yields, and the thing about yields is you start with a cold hard number.    That immediately frames the problem as an economic problem, with tangible results and payoffs for work.  

If I take a step back and look at 100k’s lines of code, I do see some similarity on the macro level, where the success of a given use case is a factor of internal and external factors, and yes they all have probabilities of success.   In fact I think if you can model the architecture right, you can get some idea of success and failure of the system using reliability principles.

I’ll tackle that in a later post.  I think I need some diagrams and a basic understanding of how to do that on blogs before I can tackle that issue next. 

[awesomejohnnyhifive]

Devops and Work Selection

Working at a large company on apps while everyone else is working server, it’s easy to feel left out of the devops game.   It’s there though, you just have to decide how to define it, how to prioritize and then how to execute.

I read an article the other day that crystallised for me what the point was.   The article implied devops for apps was intended to cut down the time between a developer writes a feature to the time the company could chose to ship it.   Ok, cool, I can get behind that.  Sure it isn’t docker, puppet, chef or any of the other cool technologies for the server side, but as far as cutting down the time to ship, ok.

When you define it that way, then you really have to decide what to work on, and there is the rub.   There are many ways to think about it, and just about everyone I talk to talks about cutting the labor requirements, catching defects early, all good.    Cutting costs though isn’t quite 1:1 for me with what the dev ops people say though, because there are some things we work on that don’t cut costs that much ... generating release notes, automatic build numbers, scripts to post.   Those seem important, but they don’t quite fall into the same category as cutting costs.

We shipped today, and there were a few unplanned incidents.  We got the build number a bit wrong, which causes an issue with Google Play.   The release notes weren’t complete, and some customers were going to have an issue with that.   These are things you can solve with automation, but really they only save the work of the poster, not the dev team that spent months getting there.   

That’s when it hit me though, getting those things wrong on the last day can cost you an entire week if you aren’t careful, and the customer satisfaction you can cost can last months.  That’s when I connected a dot,  cutting cost is a byproduct of running automation all the time, which is a way of reducing the risk of bringing a feature to market.

Changes on the last day of a release are risky, because they can jeopardise the entire release.  Put another way, devops is what you do to reduce the risk of bringing your feature to market on schedule.   As you get a predictable schedule, you can work on increasing your velocity because the risk of doing so is greatly diminished. 

I’m sure that’s not revelation for people in the devops space, but it gave me pause as to what we prioritise, it’s just not automation to cut costs.   When you look at the whole process a lot can go wrong, if  you automate out all of the activities involved in posting a load, you automate a day’s work, but you also drive it to almost 100% reliability, as opposed to typical human processes that get you to 50% when pressed by time.   Your ability to hit a date is governed by the lowest probability of success,  eliminating the human element in the process dramatically increases your ability to ship something on a specific day.  If  you are going to work on ‘devops’, you should start with the riskiest elements of your process first, as they have a disproportionate impact on your ability to ship on time.  

I’m going to spend a little more time thinking about risk / reward of the process (reliability engineering anyone?) and devops adoption, and I think focusing on risk / reward is going to be at least as successful as cutting costs for cost sake.  I think that misses the devops point.

[awesomejohnnyhifive]

The Hidden Cost of Complexity

There are so many things that can be said about complexity, but one of the rarest things I’ve heard from senior management is that they don’t understand it.

The thing is, people always seem to under estimate it and the value in reducing it, mostly because the benefits go against our instincts for customers.  You need faith, and maybe some religion, that reducing it is an inherently positive thing.   People want complexity generally for good reasons, in part because it’s perceived as the best way to deliver a feature a customer wants and in the end drive satisfaction and money.

The problem is that the full equation of the cost is never visible to make the rationale decision needed, which often is “NO”, it’s not worth the cost.

I’m going to spend the next few blog entries thinking this through and seeing if I can write my way into a framework for this.   It’s tricky, but possible. 

[awesomejohnnyhifive]

first!

Testing out a blog on tumblr.   Maybe this will work, but I need to do something to make sure it gets in here first.