For the past year I’ve been talking about a shift from Big Data to Big Ops.
In short, today’s challenges with business technology are less about managing a huge volume of data — Big Data — most of which wound up in a basement-level data lake, outside the reach of everyday business users.
Instead, we’re now wrangling a Cambrian explosion of digital processes across our organizations that actively engage that data as part of live business operations.
Clusters of back-office and front-office apps, agents, automations, and algorithms have sprung up everywhere. And each of these clusters has its own “ops” specialists: marketing ops, sales ops, customer success ops, partner ops, web ops, rev ops, finance ops, and so on. This whole environment and its bustling ecosystem are what I consider “Big Ops.”
Think of Big Data as a massive mountain of sugar.
Big Ops is the swarm of thousands of ants busily converging upon it. All the apps, agents, algorithms, automations, etc., feasting, moving, and transforming it, all at the same time.
When I shared that metaphor on Twitter, someone quipped back, “Except ants have a solid reputation for working well together. :-)”
Actually, that’s pretty close to the point. The big challenge with Big Ops is how we develop coordination mechanisms with our distributed, digital swarm. Who knows? We might legitimately draw inspiration from Ant Colony Optimization algorithms. But overall, it will take a blend of paradigms, practices, technology, and culture to find our footing in a Big Ops world.
Which is the perfect segue into a couple of thoughts about automation.
The chart above was developed in collaboration with the team at Workato, riffing on a post about their Series E funding. (Disclosure: I am serving as an advisor to them.) It maps out the impact that automation — particularly digital automation — can have in a firm.
The y-axis represents the scope of an automated workflow.
On the base end of the spectrum, you have highly localized automations run by individuals in their day-to-day work. There are tons of these built into software everywhere today, from macros and shortcuts to sales sequences and CI/CD scripts. Very useful for personal productivity, but narrow in scope.
But as you slide up the y-axis, automations are applied across larger groups: small teams, large teams, whole departments, the entire organization, and even broader ecosystems of clients, partners, and vendors beyond the walls of the firm.
This x-axis reflects the higher order effects of such automation — especially automations that span multiple clusters of people and apps.
At the origin, automation is really about productivity. It allows repetitive tasks to run at computational speed and scale instead of being repeated slowly and manually. This is valuable for a ton of use cases, especially for digital professionals. But its impact is generally linear. You’re squeezing more work in less time, with the added benefit of less accidental variance in execution.
But things start to get really interesting as you move along the x-axis. The coding of digital workflows is a way to define business processes that not only span multiple apps and teams. They become well-defined, deterministically-governed methods of coordinating complex activities across the digital firm, from new employee onboarding to order-to-cash processing across sales, service, and finance.
They become the connective tissue of a Big Ops environment.
Dare I call these systems of orchestration?
And when I say “coding” digital workflows — defining triggers and actions, steps and decisions — I should emphasize that most of this is now done with low-code/no-code tools that any process-minded business user can comfortably manipulate.
Which moves us toward yet a higher order effect: leveraging automation to enable and accelerate innovation.
As more and more of a firm is connected with these cross-app, cross-team systems of orchestration, the ability to rapidly adapt those workflows to new requirements — add a new step, trigger a new function, pass along a new piece of data, etc. — expands dramatically.
New ideas can be quickly tried and iterated upon across a broad canvas of the entire digital org. Asymptotically, it’s as easy as dragging-and-dropping a process diagram on your screen. (Or doing it in the 3D corporate metaverse with your VR headset, if you want to project a few years into the future.)
To me, this is the essence of digital transformation. It makes business operations as malleable as software.
The impact of these higher order effects of orchestration and innovation is more than additive. It’s multiplicative, due to both the scale of teams and apps connected and the agility and imagination such an automated enterprise can effectively wield.
However, it’s not entirely rainbows and puppy dogs in the land of Big Ops automation yet.
Automation features are proliferating across almost every major app out there. In many ways, this is a wonderful thing — it’s closing the “capability gap” that we’ve had with traditional businesses processes that were slow and hard to implement and maintain.
But a new challenge is emerging as a result: a “visibility gap” in keeping track of all these different automations spread across a myriad of platforms and specialist apps. Governing them, maintaining them, handing them off as personnel change, gracefully sundowning them, managing the ripple effects they have on each other, etc.
The simple answer would be to automate everything through one and only one automation platform.
Realistically though, that kind of complete consolidation is hard to achieve. I think it’s feasible to standardize on a solution for the big cross-app, cross-team workflows in a given company. But as the number of different software apps in the world continues to increase — not decrease — with automation features of one kind or another built into many of them, these wild horses won’t be easily corralled.
(Hey, philosophically, all software is digital automation.)
Who knows what I’ve automated with my Notion document, Trello board, Google sheet? And increasingly, the sophistication of what I can automate with such things is remarkable.
However, while I’m skeptical of consolidation as a magic bullet in ever-expanding tech stacks, I’m quite optimistic about aggregation as a pattern for harnessing the power of such digital diversity.
Just as SaaS management platforms (e.g., Blissfully, Torii, BetterCloud) and data privacy/governance platforms (e.g., DataGrail, OneTrust) have brought meta-management to sprawling tech stacks and mushrooming data sinks and sources, I think there’s an opportunity for meta-management of processes.
Imagine a tool that would discover and track automations across all of the different apps in your company. That would be enormously valuable in taming and tuning a thriving Big Ops environment.
Of course, it would require most apps to expose their automations via API. And today, API coverage for such use cases is unfortunately quite sparse. But as the challenges and opportunities of Big Ops come into clearer view for more and more companies, market pressure to expose such process data will crescendo rapidly.
A Big Op-portunity?
