[DECODED] You Get What You Measure

Phillip: [00:00:02] Welcome to Decoded, a podcast by Future Commerce, brought to you by BigCommerce. We've all heard the idioms before. "You are what you eat." "You get what you pay for." Or "You reap what you sow." But when it comes to eCommerce, the idiom of choice might just be "You get what you measure." As eCommerce has grown, the world of analytics has become much more fragmented. Aside from measurement, the analysis of customer behavior has fallen into multiple camps of solutions, from attribution to segmentation and even intuition. But to get anything useful out of our measurement, we'll first have to decode what to measure. Second, how to measure it. And finally, how to rally the organization around common tools that make sense for both the employee and for the business. Welcome back to Decoded, a podcast by Future Commerce where we're breaking down this concept of bringing a product from concept to cart. In this season, Decoded is brought to you in partnership with BigCommerce. I'm Phillip.

Aaron: [00:01:21] I'm Aaron.

Phillip: [00:01:22] And we are going to get into our episode today. We're going to talk about "You get what you measure."

Aaron: [00:01:29] That's right. That's right.

Phillip: [00:01:32] Actually, you're going to get a lot in this episode because we're weaving together a number of conversations that we had with industry experts, people who have joined us for this season of Decoded to kind of break down different perspectives in the way that they overcome challenges and measurement in their business.

Aaron: [00:01:51] Absolutely. I think one of the throughlines of all the conversations that we had is how important [00:01:56] context is to both the goal setting, when a brand is launching a new product or a new subbrand, and they have a goal in mind for what they want to accomplish with it and the story that they're telling to their customers and to their buyers with that and how that has to inform from both the goal setting and then down to the measurement strategy at the very end because it can be dangerous to not have a context and a goal when you're doing something new with your business. [00:02:25]

Phillip: [00:02:25] The context is important. And I think the context for many businesses, especially those that are growing at the scale and the velocity that many of the experts that we'll have on this season of Decoded have witnessed is that it's not like the team is growing at the same scale and velocity as the customer and the addressable market. And what we are learning is that, yes, context is highly important and highly dependent on what's happening in the business. It also seems to me that a lot of these businesses are relying on a lot of different software and the software itself has its own challenge in its own reporting and measurement. And we'll get into that a little bit here in this episode of Decoded as well.

Aaron: [00:03:13] Yeah, one thing that also occurs to me, as you say, that Phillip, is I think this is on brand for Future Commerce is I think of the observer effect from quantum mechanics. Brian's not here. The act of observing changes the reality. And when you talk about the measurement tool, the proliferation of those tools, the thing that they measure changes the business behavior that they incentivize. [00:03:42] You need to be very careful about what you choose to measure because that changes the reality of your perception of your business and may change your product strategy depending on what you're measuring. [00:03:50] So an interesting side note.

Phillip: [00:03:53] Yeah, I'm really fascinated with this part, especially since so much of my journey in the ecosystem has revolved around, very frankly, measuring the immeasurable sometimes in eCommerce and trying to quantify human behavior. It's really hard. But if anyone can decode it for us, it is our next few guests, and kicking us off here in this third episode of the second season of Decoded is Ingrid Milman Cordy, a long-time contributor to Future Commerce, host of the Infinite Shelf podcast and also a leader in one of the most successful consumer businesses that exist on the planet, Nestlé Health Sciences. [00:04:35] We're going to join Ingrid as she breaks down how we measure. And let me fix that. We're going to join Ingrid as she helps us to get what we measure. Oh, my gosh. What is wrong with me today? [00:04:48] We're going to join Ingrid as she tells us about how she thinks about measuring and setting goals over at Nestlé Health Science as we go from concept to cart.

Ingrid: [00:05:04] There are the really obvious goals that pretty much everyone when you ask an organization what their goals are, they're going to be able to tell you. Hopefully. It's grow X percent, it's increase your consumers. It doesn't all have to be at the same time, but they're really very large company-wide goals. And that's good. That's important, right? Your leadership team needs to set those forward. And then every other component within the organization, whether it's marketing or product development or technology, all need to have their own set of goals that are unique to their functions and to what they do that hopefully ladder up to and complement the larger company goals. But I think sometimes that second layer of goal setting and nuance is overlooked. Or too often I've seen that that second layer of nuance is overlooked. I'll give you an example. So if the company is trying to grow 20%, we need to then figure out what is going to get them there. Are they going to get 50% of that from new customers and 50% of that from retention customers or loyal customers that we want to buy again? So that's a question for, I would say, marketing. Marketing and media and the go-to-market team and probably product as well. So they need to sort of collaborate on how to get to that 20%. So that's just one layer down. The same thing with margins and the way that we're going to try to be more profitable. So you have to figure out with your CFO and your finance team and all of that and probably operations and logistics and all the people who are in charge of the things below the line. How are you going to figure out how to meet that goal? And so that's just like really high-level examples. But you really do need to start breaking down those really high-level goals by department and by the people who are going to be able to contribute to that goal. I would even say creative is another example of a team that typically doesn't always have the goals that they need to hit. A lot of them are being given creative briefs and they have a very specific thing to meet. But I actually think it's really helpful, especially with a creative leadership team, to bring those people into the conversation when your goal setting for the full organization so that they can understand and they go, "Okay, well if your goal is 20% with return customers, let's say, that then will change our messaging and maybe the way that we go to market with our content." So I think that that conversation, if overlooked and not given the same level of importance as the high-level goals are for the organization, it's really hard to have everyone moving to the same beat while also being able to customize them to the department that actually is responsible for their portion that they're contributing. And then there's the third layer below that of the social team has engagement and followership growth goals.  [00:08:28]There are always the big goals and then the smaller goals, those tend to be really clear and people understand them. It's the goals in the middle, the departmental goals that I think don't get as much air time as as needed. [00:08:43]

Phillip: [00:08:49] I was at a direct to consumer wellness brand in 2008 and those early days of analytics, it was sort of right at the precipice of Google Analytics being a free product. We were integrating Omniture as a very expensive analytics solution back then. We were coming off of a product called Hitbox, I believe, back then. Do any of these names ring a bell?

Aaron: [00:09:17] Yes, they do. I'm also old.

Phillip: [00:09:19] {laughtere} So the way that we were making sense of the measurement of these tools was trying to quantify how these three services, how these three tools were stacking up to each other. So we had three analytics packages running all at one time.

Aaron: [00:09:35] I'm sure. And they all did the exact same thing and the exact same way. So it was really easy to do a cross comparison, right?

Phillip: [00:09:40] Yeah. It was like oranges to apples to lemons or kumquats. It was my job to reconcile all of it, and it was extraordinarily painful. And I don't know that we ever really made sense of it because the way that we were measuring the customer behavior on the website was happening in one way, the way that we tracked that back to sales data year over year or quarter over quarter and tracking that to goal was another way. The eCommerce platform we were using at the time had its own inbuilt reporting, but the dev team couldn't tell us what went into the sales reports. We had to sort of reverse engineer it over time and it had its own way of doing things. And I just remember that the challenges aren't so different today as they were back then. The software is different, but our level of confidence is not. And that's where a lot of the challenges in eCommerce is sort of making sense of data. It's just having your own way of doing things and being extremely consistent in the way that you do it over time.

Aaron: [00:10:46] Totally agree with that. There's a quote, "The map is not the territory." I think Alfred Korzybski said that. He's a Polish engineer. And so don't confuse the representation of the thing with the thing itself. I worked also once with a direct consumer brand, and the challenge that they had was that there was no real solid link between the measurement system that they had set up inside of their direct to consumer website and the actual sales reporting that their leadership wanted to see. And there were so many layers between the technical team and the management team that the feedback loop for improving the measurement strategy, the measurement, the collection of data inside the site was to run a bunch of reports, show them to a senior leader in a different time zone, and then that person would come back and ask a question, "Hey, what about X?" And then the the eCommerce team would scratch their head and sort of like stroke their chin thoughtfully and then come back, and two weeks later, we would get a Jira ticket. And so this two weeks, "Hey, can you fix GTM for us?" became a six month project.

Phillip: [00:11:50] There is always an axiom called Seigel's law and it is a man with one watch knows what time it is and the man with two watches never is sure. You mentioned a Polish engineer. I met a Polish engineer one time who said you never take two chronometers to sea. You take one or you take three. And I'll tell you this, I don't think it's helpful to have three analytics engines running on the same website. Speaking of the consistency of the tools, we had Sean Larkin, who's the Founder of Fueled on to tell us a little bit about how he's thinking about solving this challenge of measurement at scale with brands of many sizes. But he has a long history in helping brands to solve their measurement problems for first party data. So let's go and hear what Sean says about how he advises his customers of his product on how to get what they measure.

Sean: [00:12:48] Candidly, the reason is because everybody goes into Facebook Ads Manager and they go into Google Ads and they see wildly different attribution and it freaks them out and they don't know what to trust. And so they just start Googling or they start talking to people. And that's when we come up. And that's understandable. I don't necessarily... When people say that, the first step is just to kind of assure people like, "Hey, those attribution models are not supposed to match." The idea that you would have Facebook and Google show the same attribution per channel is just never going to happen and it's actually not in your interest. You want those different platforms to be optimizing by claiming credit for as much as they can so they're machine learning models can optimize better, but that typically is why people call because it's very confusing when Facebook's taking credit for 75% of your purchases and so is Google Ads. So that's kind of what drives it, and it's that hype and that concern which is built, I don't know how much money Triplewhale is making, but we're getting a lot because of that fear and because of this promise that they can kind of offer a single source of truth on attribution. To be totally honest, single source of truth, we're building towards that. So we have a first party pixel. We collect all this data, we send it to Facebook, Google. It's like one set of pipes that will send all the data where it needs to go. But the other big thing we do is we send all this data to a data warehouse and then we provide a bunch of analytics offerings that are all open source to help merchants kind of understand and build their own attribution models or use our attribution model, but understand the math behind it because it's all open source. We're building tools to be able to compare attribution: here's what Google is saying, here's what Facebook's saying, and here's what we say. But to be honest, I don't think that's the most interesting problem out there. It's what CEOs kind of bang on CMOs about because they want that simple explanation. But I think a lot of the people that are saying attribution models don't matter... You hear that a lot kind of amongst the pundits these days. I think that there's, in and of itself I think that's accurate. I don't think it's a huge problem that these different platforms are reporting differently. To me, [00:15:17] the problem is merchants aren't collecting the data and they don't actually know how to operationalize it, and so as soon as I can pivot the conversation there with merchants, that's when things get exciting. When we can say once we have all this data, we can blend it, we can show you deeper insights, we can start doing segmentation that actually matches what you want customers to do and/or we can actually build campaigns around what we see customers actually doing themselves, that's when things get really fun and that's that light bulb moment [00:15:54] when a customer, to be blunt, goes from paying us a couple hundred bucks a month to paying us a couple thousand to kind of build out that infrastructure for them to really get good at targeting personalization and things like that based on good data.

Phillip: [00:16:11] It kind of reminds me of the four stages of competence. And I've encountered businesses where it's sort of a bell curve. You have the businesses that don't know what they don't know about data and attribution strategies and at the other end of the spectrum... So that's like brands that are really early on, very early to making investments in eCommerce somehow find some success and then realize they find themselves in the middle somewhere where it's, you know, it's what you might call is like conscious incompetence or conscious competence, which is I have lots of data and I'm trying to do lots of things with it. But more often than not, that's very easily found in even larger organizations, Sean, with businesses that have a ton of data, collecting a ton of data, and they don't know what to do with any of it, in which case they're back relying on intuition. And that's where I guess my question would come down to for this audience in this podcast about how do we effectively measure goals that we set and who needs to be in the room when the goals are set so that we do that right? What does it come down to with tooling and how do we get better at not sitting on mountains of data that nobody's doing anything with, but getting the right amount of data to solve the right problems or to measure the right outcomes so that we have repeatable success because that's really what people are trying to do, right?

Sean: [00:17:50] Yeah, 100%. And I think it's not starting with what the data shows, but actually starting with the questions that you're trying to ask, and I think that's another thing that kind of gets lost in this like fast SaaS 30 second clips on LinkedIn kind of world where you see all these merchants that are paying Triplewhale, North Beam, Rocker Box to run on the same exact website. They just have multiple reporting tools and they just toggle between them to find the report that kind of answers the question the way they want to present it to their boss as opposed to saying, okay, well, what are the questions? And then do we have data that can answer this? And then how will the data answer this question? And that's just where productization, I think is kind of making things more difficult for merchants. And that's a problem that we're trying to solve by offering kind of an insights as a service fractional data engineering data science.

Aaron: [00:19:41] So we've just heard from Sean Larkin of Fueled about the dueling and the instrumentation around analytics, talking again to somebody else who's on the front lines of growing a business and making those decisions about the goals and the measurement tactics used to evaluate the success of those goals, we have Ian Leslie, who is the CMO of Industry West, and I believe Phillip, he's a longtime frequent guest on Future Commerce as well, correct?

Phillip: [00:20:07] Yeah, also a co-collaborator for many, many years. Full transparency. Ian was a client of mine once upon a time in agency land. I have been on the receiving end of Ian asking hard questions about how we measure our performance, both as an agency, as someone who is a customer of ours, but also in making sense of the data that we were generating to sort of justify his continual investment in our services. I think Ian has a pretty broad and general set of skills when it comes to weighing in how you attract and retain customers, but also how you hold yourself and your own team accountable to your goals. So yeah, let's listen to Ian's thoughts as we get into this idea of how he thinks about product development and how product development and the velocity of product development and brand development change the way that you set structure and measure goals in your organization.

Aaron: [00:21:11] The thrust of the podcast has been in a lot of respects about product development as well, not necessarily just wholesale business evolution. And so if you're thinking about the process of adding a new product to the line, you have a lot of channels potentially that you want to represent and show that product often. When Industry West is thinking about developing a new product, are you thinking about that in terms of a specific channel that that product is fit for, or are you thinking about a product that is going to be widely applicable across a variety of channels or both? And then the follow up to that question would be how do you think about the investment in those channels to support the product vision? In other words, does the product carry with it technical debt or some kind of investment debt that needs to go with it to enable that product to be successful?

Ian: [00:22:06] In terms of product development, I think nine times out of ten we are looking at a broad spectrum. This is applicable to both the consumer and the trade side or the B2B side. I'd say currently somewhere in the neighborhood of 95 to 98% of the SKUs on our site are actually "trade rated" or B2B rated or whatever the terminology is. That being said, there are times where we go into a collaboration or product or a product launch understanding yeah, this is going to definitely skew more consumer, but then are shocked to find that the trade actually likes it as well. And we have to scale up production to support a growth in estimates from the trade side. So I would say an example of that in the past was we were first to market with cane. Cane, the rattan stuff was big pre-COVID. It's still very popular. We were definitely one of the first to market with it. We saw it much more as a consumer product and yet found that there were restaurants and office spaces that wanted that product and that we had to ramp up production and ramp up the buy on that. We're currently in the midst of, and we're still early on, so I don't want to say the name, but we're in the midst of a collaboration with a DTC paint brand, and I mean, there are very few DTC paint brands, so I mean you could probably figure it out from there. But a DTC paint brand where we're using some of their very unique colors, we came up with the product, and they're providing the colors. We're going to do this cross collaboration. It's going to be awesome. I think one of the colors is called like Rosé All Day or something like that. And so it's going to be super cool. And we're thinking, oh, this is going to be great for the consumer side. It's an indoor outdoor piece. People are going to love it, these unique colors and whatever. And we say that and right away I know, I could see a cool startup business wanting this for their cafeteria area or even a cool bar lounge or something like that. So I'm going to say, I'd say there are times  [00:24:14]we're typically launching product for both sides of the company and then the few times that we are maybe a little bit more granular in where we think we're going with the product, it ends up being maybe even more popular on the other side anyway. [00:24:31]

Aaron: [00:24:36] Phillip, I'm thinking back to the first episode of the series when we interviewed Loretta Soffe, from Nordstrom, and she shared her experience with grilling her buyers who were presenting new product ideas and new go-to-market ideas and how they might show up to her office with a giant stack of PowerPoint slides and supporting information, but trying to find the actual context of what is the actual goal of this project that you're trying to pitch? Because there's a lot of money, potentially a lot of investment going into those decisions. And so you can easily bury the lead, you can bury the context in the story under an enormous amount of data. Wanted to tie it back maybe to something that Sean said in our conversations with him around tools and and measurement and maybe how that affects confidence.

Phillip: [00:25:30] Yeah, I do think that the word there is confidence because, as Loretta said, [00:25:37] having confidence in your plan requires a lot of conviction. And what I think Sean's takeaway here is, especially in the modern analytics and attribution software ecosystem, that's really all you have is confidence in attribution. [00:25:54] And so they're selling on confidence. And if you are a buyer, that's really what you're looking for because you have to make a lot of justification to the higher ups that you have confidence in the tool and sometimes confidence in the tool, Aaron, it comes down to who else is using the tool, not necessarily how the tool is measuring and attributing sales to a given channel. Let's listen to Sean's perspective on this idea of confidence and not substance.

Aaron: [00:26:24] Are folks buying or researching these tools based on an answer that they think they already know and they're looking for a tool to tell them, to reassure them with what they believe is already true, or are they doing it from a place of I don't have a freaking clue and like I want Magic Box to tell me things? Because those are two very different purchase paths, because one of them is more into, Phillip, what you just described, which is "I'm trying to tell a story and I'm looking for a third party to come validate the story that am already telling, and I already know what the metric is. I already know what the result is going to be and I need something that's going to tell me that number."

Sean: [00:27:05] This is where I asked you guys if I could be me. But my answer to this is yeah, there's just so much fear. That fear of failure and what's happening. And this is my hot take. And I'll probably get hated for saying this is you end up with analytics tools that are just selling on confidence, not on substance. I kind of refer to it as like, "Trust me, bro" tech. Is Triplewhale, right? Can I prove that Triplewhale is right? I don't know what that model is. I know they use fingerprinting. I know they use different things like that. But what's their ad budget? And they've built out these incredible ads that are like, "Trust me, bro, we're right. We got your back." You know? And as kind of a data engineer, data scientist, I don't like that. I want to get into it. I want to see the attribution model. I want to read on it. I'm the kind of guy that pipes on my Google Analytics data into BigQuery. So I can look at all the raw data because I don't trust what I'm seeing, but that takes patience, that takes vulnerability because you probably aren't going to understand it. But right now, again, because of all those pressures in the economy, I think people are just gravitating to it like, "Somebody told me this is going to work. Somebody on Twitter with 50,000 followers said, "Go with this tool.'" So it must be right. And I do think that's kind of holding us back. It's making a lot of people a lot of money. And people aren't doing anything nefarious, but I don't like that kind of style that we've gotten into of like LinkedIn and Twitter Bros telling us what's going to work and stuff like that. You know, I think the other thing I should have mentioned before when you're talking about like how to handle failure, I think the other thing that happens a lot is when people get scared, they start experimenting with everything all at once. And then there's no way to measure it, no matter what data you get. There's kind of a lack of patience. And so I think [00:29:06] if you are going to launch a product, assuming that there's going to be failure, budgeting it out to begin with so that you can test and experiment logically [00:29:15] and not be doing like, "Okay, well, we're going to change our ad targeting. We're going to change our creative, we're going to change our discounts all in the same campaign on Facebook." And then it's like, "Well, how do you know which one of those levers that you pulled is doing anything?" So starting with a budget, understanding things are not going to work. And then just moving with discipline, what is that the Navy SEALs like slow is smooth. Smooth is fast. You just have to really trust the process if you're taking something to market.

Aaron: [00:29:49] Well, Phillip, that was pretty great. And I like Sean's confidence in his assertions there, and I think it's absolutely true. Going back to what we were talking about at the beginning of the episode, you need the context and the narrative in your own goals, your own product, and your own brand that trickles down to your measurement strategy. Because if you don't have that, the different tools that you use... You were trying to reconcile three different versions of the truth at your brand. I had a brand that was trying to understand basic things about its performance. Sean's talking about how the tools sell on confidence. You have to supply your own or else the tools are going to run your business for you because you get what you measure.

Phillip: [00:30:33] That's a wonderful place to leave it. And I think you do get what you measure. One of the things that we can do to measure the success of this podcast is to get feedback from the listening audience. I'd love for you to give us your perspective on the value of this series and how you measure goals and how you set effective goals in your organization. If you would like to weigh in, we'd love to hear from you. Send us your feedback and drop us a line at Hello@FutureCommerce.com. Thank you so much for listening to this episode of Decoded by Future Commerce. This season is in partnership with our friends at BigCommerce. Thanks so much for listening to Decoded. You can find more episodes of this podcast and all Future Commerce podcast properties at FutureCommerce.com. You can also subscribe to our newsletter which comes out three times a week at FutureCommerce.com/Subscribe.