Are You Big Enough for Big Data?

by

image with machinery gears

Gartner has recently put forth a formal definition for Big Data: “Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization.” For those of you new to Big Data problems, the definition includes three critical “V”’s often referred to as the “3 V’s of Big Data”: volume, velocity and variety. And the fourth critical piece of the definition is “…require[s] new forms of processing…” – i.e., a novel approach to the management of data.

Here’s why that’s critical. Vendors, companies and individuals love to throw out the dictum that they have handled big data problems before big data was big. But it just ain’t so! Yes, 10 years ago, a gigabyte of data was a lot for a system to handle efficiently. Kudos for those who successfully manipulated data sets of that size. And in 10 years from now, a petabyte may not seem like a lot of data, so the future you may question why you struggled with “only a petabyte”. But just because you have “volume” doesn’t mean that you know how to work with big data problems.

The reality is that these big data problems are of a class that are just plain difficult or impossible to solve with traditional approaches. (For those of you that need specifics – I’m talking about relational data models!) Big data doesn’t mean that you must use NoSQL or other technologies, but it does mean that you need to approach the problem with a new approach, typically by implementing a highly parallel processing algorithm such as Mapreduce (though, again, it doesn’t have to be Mapreduce).

So if you are talking to a vendor who is touting their big data experience, and they say they’ve been solving these types of problems for years and years – be wary. They are giving you a “marketing answer”. Dig in. Understand what they are really doing, and understand if their solution really meets your needs. They may certainly be a great choice for you but, equally, they may not.

READ MORE

Shifting Perspectives: 3 Learnings From a 3-Day Training

Shifting Perspectives: 3 Learnings From a 3-Day Training

About a week ago, I completed the second live (virtual) training in the process of becoming a Certified Professional Coach through iPEC. Once again, my mind was blown! It reinforced for me that virtual workshops can, and do, work, and, in a lot of ways, I prefer them...

read more
Finding My Work-Life Balance

Finding My Work-Life Balance

In my previous post, I told the story of how I got back into consulting after becoming a mom. All of the diverse experiences I had during that journey have helped me to find my work-life balance by… Defining Boundaries “Go home,” my first boss said 12 years back —...

read more
How I Got Back to Work After Being a Full-Time Mom

How I Got Back to Work After Being a Full-Time Mom

I Landed My Dream Job Throwback to 2014, I had completed my MBA, landed my dream job as a consultant, and was hoping that my new consulting career would exponentially ramp up my career growth for the next 5 years. This would position me to take on critical decision...

read more
Self-Awareness is Key to Belonging

Self-Awareness is Key to Belonging

In August of this year, as part of our annual company meeting, our team at Thought Ensemble participated in the foundational session of Diversity, Equity, and Inclusion (DEI) training led by Dr. Nika White, IOM, CDE (she/her/hers). One of the most meaningful moments...

read more
Finding Your Organization’s Magic Pixie Dust

Finding Your Organization’s Magic Pixie Dust

It is often said that organizational culture is like a fog — it is all around us; it impacts our ability to see, to move quickly, and to deliver; but we cannot quite put our finger on it. Indeed, some organizations see their culture as a byproduct of operations,...

read more
We’ve Refreshed Our Brand!

We’ve Refreshed Our Brand!

Why have we refreshed our brand, you ask? Well, as we have grown and matured as an organization, we felt that our previous brand elements no longer represented us as well as they could. You see, we founded Thought Ensemble back in 2008 to help companies better compete...

read more
Thought Ensemble’s Purpose — Inspired in 2020

Thought Ensemble’s Purpose — Inspired in 2020

I recently wrote about how company purpose is being tested and inspired by all the events of 2020. This topic is very real for us at Thought Ensemble. We’ve been thinking a lot about what really matters as we’ve navigated the...

read more
How 2020 Is Testing and Inspiring Corporate Purpose

How 2020 Is Testing and Inspiring Corporate Purpose

In August 2019, the Business Roundtable rewrote their statement of corporate purpose. I followed this with significant interest being that I have never forgotten the debates about corporate purpose in business school almost two decades ago. We were taught that the...

read more
Why Purpose-Driven Organizations May Struggle With Change

Why Purpose-Driven Organizations May Struggle With Change

I love working with companies who really want to make a difference, beyond just making money for their shareholders. I mean, making money is fun and all, but it is even more rewarding to join in on a just cause. Plus, as this HBR article explains, companies who have...

read more