CloudFoundry

Continuous Delivery Among the Donkeys

26 Feb , 2015  

I was recently asked to give a brief overview of the mainstream market for continuous integration and continuous deployment and how companies are using it. The audience was a mixture of HeavyBit portfolio companies and their customers. So, it provided a nice mix of sellers and buyers. Since this topic comes up all the time in Pivotal conversations, I thought it’d be worth going over here as well.

Carrots on Heads, Please

Source: (PBF Comics)[http://www.pbfcomics.com/253/]

Source: (PBF Comics)[http://www.pbfcomics.com/253/]

I’m often fascinated—if agog in wonderment—at what the “elite” of the tech world are doing, those “unicorns.” There’s even “horses” out there—high performing organizations that are keeping up with those unicorns. But, I’m most interested in the normal folks, “donkeys,” as I like to think of them. I like seeing how the donkeys are doing when they strap a carrot to their head and try to keep up with the unicorns.

If you’re the type of person who wants the benefits of Cloud Foundry (making sure you’re shaving the right yak, focusing on application development instead of infrastructure development), you’ll want the benefits of continuous delivery, namely, speeding up your software delivery pipeline by automating as much as possible from builds, to tests, and even promoting builds to production.

These are outcomes of doing CD. The overarching goal is to reduce the cycle time to get new features into production, helping you establish a feedback loop that you then use to guide the perfection of your software. That last part is key, and often “money left on the table” when organizations stop short of changing their process.

So, let’s take a look at a quick survey of studies on how CD is doing out there in donkey-land.

Yeah. Everyone Wants a Pony. Astute Question, Boss

When I was at 451 Research, we did two studies of the DevOps market, from a donkey perspective. The first study had 201 participants, the second 300. We achieved a good cross-industry mix, not just people in the technology world. There were plenty of horses and donkeys in there. One of the first questions we asked was around each company’s desires to speed up software deployment. That is a core benefit of DevOps (perhaps the core benefit), and certainly a core benefit of continuous delivery.

In aggregate, the answer was painfully obvious—of course people wanted to go faster. Everyone wants a pony! Sliced up by industry, things got a little more interesting:

There are obvious ones who want to speed up like retail, entertainment, and SaaS companies. Industries like transportation can seem weird until you realize that IT-driven companies like Fedex, UPS, or even Uber are in that category (which is to say, there’s tremendous competitive pressure for IT to move faster). On the low-end, it’s sort of depressing that health care is less interested (I know I’d certainly like more, interesting uses of software when I visit the doctor), and perhaps things like construction are not so surprising.

When I look at this chart it helps me triangulate what types of industries are most interested in using software to change how they do their business. Those industries on the left side are certainly high on the list of people who’d be interested in continuous delivery, one would assume.

The Job To Be Done: A Platform for Speedy Software Delivery

With the desire to speed up software delivery frequency, we’ve found the job to be done. Now, a job to be done is a fun way of describing the problem a product or service solves—what “job” does a customer “hire” a business to do? I hire a hamburger to (a.) fill me up, and, (b.) make me happy. In contrast, when I’m on a long road-trip, I might just “hire” a gas station hot dog drowning in pump-chilli to fill me up, but not really make me happy.

In the realm of continuous integration and continuous delivery, there’s a clear job to be done—creating the “pipeline” for packaging up, verifying, and then deploying software into production. That is, everything between writing code and operating it. Someone once suggested to me that DevOps was simply the evolution of continuous delivery, which, while not entirely accurate, is useful framing. In that sense, I think it’s good to use one of the older but still useful process studies from the DevOps world, the software delivery platform:

Originally published in 2012, it’s stood the test of time and even popped up in one of the more useful cloud books of last year, The Practice of Cloud System Administration. CI/CD plays a huge, if not necessary, role in this process. One key point is to think of structured platform—moving code through this “pipeline” requires a lot of standardization and discipline. The alternative of re-inventing the packaging and infrastructure layers each time is the wrong kind of chaos.

There’s another vital part missing from the diagram, though, usually not from the conversation around it—a feedback loop. In addition to just getting software out the door more frequently, the primary business benefit of continuous delivery and DevOps (you see how those two so neatly dance around with each other?) is access to oodles of feedback about how customers are using your software.  This is used to hone and modify your software to better fit what your customers want, and one presumes <hopes>, that it leads to making more money from them.

Adapting to this feedback loop is the “money on the table” when it comes to process. I see very few donkeys taking advantage of those feedback loops, and the horses and unicorns even struggle with changing their process. Surveys and anecdotes alike show that people feel like things are often going wrong with their modern IT projects and often blame doing too little when it comes to change.

Speaking of Failure: CI/CD Use is Low in Donkey-land

Getting down to actual tool-usage, surveys from 451 and others, show a consistently lower use of CI/CD tools than you’d expect:

Here, you see the sad donut and some confirming, but encouraging survey data from DZone.

The sad donut shows that somewhere between 25–30% of the respondents are using CD tools like Jenkins, Bamboo, or hosted CI/CD services. There is a seemingly positive 35%, let’s call it, who are creating their own CD platforms. In past years, “rolling your own” when it came to service delivery platforms was needed and there was no other option. But, the technology is mature at this point, calling into question the strategic worth of DIY’ing your continious delivery platform. Effort spent creating your own CD tool is probably better spent on, you know, the actual software your customers use. The most distressing part of the donut is the near 30% of respondents who are doing nothing. I call this “distressing,” because, one, I like to be hyperbolic to keep myself awake, and, two, because CD as an idea and technology has been around a long time, well over five years. It is hard for me to imagine a software development team that wouldn’t benefit from it.

Now that I don’t work at an analyst shop, I can freely mix and match research data. So, I wanted to pull in some additional survey results on CD. On the right, you can see a year over year comparison of a DZone study, first listing companies who believed they did CD and then, according to a pretty good criteria based on orginization indicators laid out by Martin Fowler, judging which of those respondents are actually doing CD. Again, the results are somewhat shocking, but help triangulate the sad donut.

The Pipeline Is Your Platform, Perfect It!

For ease-of-thinking and strategic application, I wanted to simplify the software delivery platform diagram a bit. So, here it is with Pivotal colors!

There are a few things to think about:

  1. The most valuable part of this process is actually a handful of pixels—the first few steps where you’re creating the software used to help run your business. The feedback loop is clearly key to getting the right specifications in place and making sure you’re writing software that is actually effective. I believe these first two boxes are where most companies—most donkeys to be sure—should spend the majority of their effort (and yes, one should bundle test/verify in there, that’s always nice…).
  2. Over recent years, we’ve spent most of our time as an industry focused on the rest of the boxes, especially the infrastructure platform and production concerns boxes. Topics like cloud, OpenStack, and containers have made this area a churning pool of excitement. When I was an analyst and doing cloud strategy, I came across enterprises all the time who wanted to build their own platforms out of piece parts. For some—often the unicorns—it makes sense, or at least it used to. For others, it’s probably gold-plating and a mis-application of time and money. Unlike in recent years, there’s are so many “off the web” (to morph the old OTC idea) infrastructure parts that you’re likely better just using one of those rather than letting your people go crazy with the always entertaining task of building a platform.
  3. You also need to be on the look-out for fat boy scouts in this pipeline. If you lead an exciting enough life that you’ve read The Goal (a “business novel”), you’ll hopefully recall the lesson of the fat boy scout—your line of marching boy scouts will only be as fast as the slowest marcher. It follows that optimizing anything else before addressing bottlenecks is a waste of time. Focus on analyzing the whole process (i.e., the pipeline) and ruthlessly remove the bottlenecks. If you want an IT nerd version of The Goal, check out the more recent The Phoenix Project.

All of this raises a larger point. While we might think of all of this as Agile and the sort of cowboy codery that’s often associated with it…there’s actually a tremendous amount of discipline, process, and careful work involved in running a business with a continuous delivery engine. It’s much more than just using a tool. It relies on building out and perfecting the entire pipeline. The rewards are high, though—getting software out the door faster, making customers happier, which hopefully leads to more profits.

Let’s Fix the Sad Donut

From the anemic, but slowly beefing up, usage of CD we’re seeing out there it’s clear that we can do better as an industry. Almost five years after the publication of the Continuous Delivery book, there’s a lot of that half full glass to fill. In the meantime, the growing interest in developing mobile applications, cloud native applications,12 factor style applications, DevOps, and cloud in general has sparked a renewed interest in the otherwise sleepy,but always valuable, corner of the IT world, software development. Software development is suddenly very interesting and the focus of lots of great work and innovation from companies large and small. Indeed, HeavyBit’s investment thesis is that development tools as a market is fast becoming a big deal again; a strategic point that Pivotal obviously believes as well. Part of the goal of Pivotal Cloud Foundry is to provide as much as possible in the nature of structured platforms to slim up all those fat boy scouts in the pipeline. While the label “Platform-as-a-Service” has been and is certainly one that applies, we prefer to think of what we offer as a set of tools that helps our customer perfect the software delivery pipeline, that is: a platform.

Recommended Reading:

, , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,

CloudFoundry

When To Shave Yaks, Or Avoid It All Together

30 Gen , 2015  

featured-yak-shavingI’ve been seeing a common pattern in enterprises development groups recently: IT shops are creating their own Platform as a Service. They’ve built a shared, often cloud-like runtime used to support their custom written applications, the software used to run their business internally and externally. They’ve got their own platform. Many companies were forced to do this in past years because viable PaaSes that met their requirements didn’t exist; there’s all sorts of examples, perhaps most famously, Google and Netflix, each of whom had to build large scale, cloud-driven platforms to fit their business.

Each of these companies had to build their own platform many years ago to get their competitive advantage: the agility, speed, and cost controls that their in-house cloud platforms gave them pushed their businesses ahead. In those early days, you had to handcraft all the layers needed, often inventing them yourself with the beneficial and hard learnings that come from failure. Much of that knowledge has since been sucked out those companies and put into the commons for everyone to use. Apache Hadoop® is based on some of that secret sauce, the practices and technologies of DevOps emerged from the primordial, in-house cloud ooze, and Cloud Foundry was designed to learn from the custom platform at Google.

“Good job provisioning servers this year,” said no CEO ever in an annual review.

With that knowledge in the commons and in mature platform products, there’s little competitive advantage in building your own platform now from the bare-metal up. Instead, there’s more competitive advantage in selecting an off-the-shelf platform and focusing on your own applications. The applications are the actual code—the mobile apps, the business process, and all that stuff we used to call “business logic”—that you end-users and customers see and interact with. The application is where you differentiate from your competitors, not how you manage compute, networking, and storage.

Therefore, one would think it’d behove companies to focus most of their attention on application layer, not just the layers below it. Take a look at this recent chart from my former analyst firm 451 Research listing cloud related projects over the next two years:

Cloud Computing Total Opportunity - TIP Coud Wave 7 - Screenshot 2015-01-22 10.24.07
Source: “Reference Technology and Services Roadmap – Cloud Computing Wave 7,” 451 Research, Aug 2014

Shifting through all the anecdotes and the studies, you can tell that there’s a lot of raw infrastructure build-out going on. People are building IaaS, automation, and all the layers that operations staff needs to run their data centers. On the other hand, projects associated directly with writing the actual software (where the real business value is) tend to be further down the list. When I see this chart, the strategist in me sees the chance to leapfrog competition if only you can figure out the right technology to pogo-stick with.

Instead of taking time to build your own platform, just get an off the shelf one and move up the stack to the actual application. I’m fond of cheesy analogies involving going to the grocery store: it’s like you need to buy some milk and you’ve spent a month building a car when you could have just bought a car.

Why Does This Happen?

I’m interested in finding the causes of decisions in IT, and understanding those causes is more than just entertaining, it helps people change for the better, whether it’s learning from success or failure. So why is there this deep focus on building out raw infrastructure? Let’s look at some of the causes (with benefits), and then problems building your own platform can create.

Causes: “I Am A Unique Snowflake”

Most of the reasons companies end up building their own platforms revolve around a misunderstanding of how to best apply IT to making the business profitable. IT usually best serves the business by being close to the customer (or the core transaction the business performs), for example, working on the application layer. This is the layer that’s visible day-to-day and where the business can differentiate against competition. There are of course, legitimate reasons to build your own platform, but for the most part, the causes are variations on poorly aligning IT strategy to business strategy:

  • “Let’s build a private cloud.”  You want to pull back from public cloud, often from AWS. There are many reasons, sometimes competitive pressures (like if you’re in retail – do you want to run your business in your competitor’s data centers?), oftentimes the desire and need to fully control the stack and run in a single tenant mode.
  • “That’s what I do!” I’ve spoken with many sysadmin types over the years and, like any human, they want to do more of what they’re good at. Thanks to decades of IT Service Management and ITIL thinking, they’re good at building infrastructure and then providing controlled access to it. Little wonder they’d want to do the same in the cloud era. However, to switch back to an application developer mindset, that can seem a bit like “gold plating” as we used to call it in the Patterns days or “yak shaving” as the kids term it now.
  • The sins of our forefathers. Maybe you inherited this platform through M&A or portfolio consolidation – at organizations of any size, esp. large ones, you might end up with a hodge-podge of methods for delivering applications. This could be due to “shadow IT,” acquiring other companies (think of all the bank consolidation over the past few years, or dollar stores melding together – there’s lots of IT to consolidate), or even re-orging large divisions. You’re left dealing with problems someone else caused: the type of thing a lot of enterprise CIOs are forced to eat for breakfast (lunch and dinner!) each day.
  • The yak, it needs shaving. There are still instances, to be sure, where you need to build a platform from the ground up. Maybe you deal in secrets like spy agencies and military – those Palantir people seem to be growing in quiet with only the occasional moppy-headed CEO picture leaking out. People like Planet and Space X who are building out space likely have interesting unique needs. Even in this situation, you want to keep an eye on shared pool of brains in your industry (usually represented by open source efforts) and start to reap benefits from using a common platform.

Problems: “Now I’ve Got Two Problems”

The problems in building and running your own platform sprout from the first problem of “You own it, you build and run it.” not only do you need to staff (or pay for outsourcing) the development of your platform, but you’ll need to maintain a staff to keep it updated. This might be fine, but it also means more resources being spent on non-differentiating IT instead of building the actual applications used to run your business. The rest of the problems are variations on that theme:

  • Limited technology support limits innovation. Because you’ve built your own platform, you often only have resources to support one or two programming language well, let alone all sorts of middleware and databases. While this might have been of limited concern in the triarch days of Java/.Net/LAMP, as the bi-annual RedMonk programming language index consistently shows, we’re in a polyglot age where using the right language for the task is one way to achieve advantage. The same is true in other parts of the stack, like the data and analytics slices where options abound.
  • A self-ecosystem means you’re on your own. Your options are not only stymied when it comes to the internal components (languages and middleware) for your platform, but you’ll also limit your access to support tools (all that stuff in that makes up Geoffrey Moore’s “whole product”) like systems management and security tools, consulting, development tools (from IDEs to CI/CD and portfolio management tools the PMO folks need to allocate budget each year) and other third party services that often target industry standards rather than enterprise one-offs.
  • Not “extensible.” As a consequences of being a self-ecosystem, you may not be able to add new features, functionality, and frameworks as quickly as others. This is one of the main benefits of an vibrant open source community (or well funded and staffed closed “community”) around a platform: there are many people and organizations adding in new functionality each year that you get for “free.”
  • Ancient lore considered painful. Whereas your staff might have understood how it all worked for the first few years, as staff move to other projects or companies, you’ll often slowly forget how it all fit together and worked. Without a vibrant, multi-party set of eyeballs and brains on the project, consistently documenting the “lore” around the platform is difficult, resulting in what was once an understandable white-box turning into an impenetrable, gold-plated black box that’s more a painful riddle than a helpful tool. This often happens when you’ve inherited large chunks of your IT portfolio.
  • You’re now in the three-ring binder business. Another consequence of having your own platform is needing to educate new staff on its use, often in metaphorical or literal three ring binder run, internally funded training sessions. There’s no O’Reilly books to go buy and read, online docs, or StackExchange sites to ask questions in. You own the training. (Of course, this isn’t to dismiss training and learning, but rather enforce it: that time is better spent learning more valuable things than how to build and run a platform, like perfecting how you design and build the actual software used to delight your customers and run your business.)
  • Stuck in a “portability” cul-de-sac –  By virtue of owning a unique platform, your applications likely run best in your own platform, or, worse and more common, they only run in your platform. So you’ve created your own lock-in, a feat usually limited to multi-billion dollar, multinational vendors. Lock-in is a core taboo in IT strategy and one that guides much cloud think now-a-days in a beneficial way: as Gartner’s Eric Knipp put it recently, “[h]ybrid cloud requirements have made portability a higher priority, and having a cloud exit plan is mandatory.”
  • “Remember page 382 of that EULA your predecessor’s lawyers accepted five years ago?” While it can be a minor concern, licensing issues around components you use in your own platform might rear their head. Closed source enterprise databases and middleware are notorious for being prickly when it comes to the price to embed. There’s little guarantee that you can simply bring your own license to cloud scenarios, where-as newer PaaSes tend to have better success at sorted out these concerns.
  • Welcome to systems programming. Whereas you may be an expert at application development, once you’ve built and run your own platform you now have to become a systems programmer: understanding and coding up all the operations concerns for actually running the platform. This jump from application to systems development is not easy: those spooky kernel people in dad jeans don’t seem to think the same as the skinny jeans app folks.

“What Have the Romans Ever Done For Us?!”

Again, I feel like people have had to build these types platforms for a long time because there weren’t enough viable options. There’s a certain “dark ages” between the waning days of Java and .Net and the renaissance we have now with PaaS and other microservices-think driven platforms. Admittedly, there were bright-spots that benefited us all during these dark ages for platforms, like rails and the emergence of JavaScript as programmer WD-40 (that simple thing that seems to make everything easier and smoother), but it’s been a long time since there were viable options for off the shelf (or, now, “off the web”) platforms. Obviously, I’m biased being at Pivotal, but that chance to move our attention back up the stack, closer to the user is part of why I joined Pivotal. In Pivotal Cloud Foundry it feels like there’s a good platform, it feels like there’s a good platform, ready to do the drudgery work at the infrastructure level so companies can focus on more delightful (and profitable) uses of IT in the application layer.

Editor’s Note: Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

, , , , , , , , , , ,