of the United Kingdom’s capitol city.
This is the week when YoY reports go to hell. This time last year was when the stock market fell, the then-President began screening international travelers for Covid, and, on March 5th, we will have discovered the first outbreak of 2019-nCoV in the United States in Snohomish County, Washington.
So this is the week when your reports pass through the Bermuda Triangle of the 2020 anomaly. A year in eCom that played out as a rollercoaster of data - very high highs, and very low lows. How do we account for it? How do you make sense of it?
Which Wrong to Believe
Depending on your line of business, Q4 was pretty stellar for many eCommerce verticals. Especially those in pandemic-durable categories like wellness, fitness, home furnishings, and bed and bath. QoQ planning will reflect a dip from Q4 to Q1, depending on seasonality.
For the next few months, be wary of the data your year-on-year reports are telling you. Demand, even in these same categories, plummeted for the first 3-4 weeks of the pandemic, through about Easter in the United States. This artificial suppression then rebounded slowly in the months to come but could present as a near 50-100% increase in your YoY comparisons.
Similarly, the summer months of the eCom boom may show you as trending downwards. In short, retailers have to use their judgment when dealing with outlier events. According to Damian Soong, Founder of Form Nutrition, forecasts are always wrong, and it's up to you to decide which wrong to believe:
We have three models. One is a simple year-on-year. Two is a simple month-on-month. And three is we use Facebook Prophet library, which is a machine learning algorithm. And that basically takes daily data from the last three years and churns out forecasts going forwards.
We've learned that forecasts are always wrong and it's just a question of picking the wrong one, or a combination of all three and using your judgment to come out at something sensible.
Outliers make modeling demand all the more difficult. Even with sophisticated tools like Facebook Prophet, an open-source forecasting library, models based on past performance may not be informed by present context. Today CNN reported that among the three vaccine producers we will reach 130M doses in circulation by end of March 2021, and 400 million by the end of July, enough to fully vaccinate all Americans (with nearly 70 million doses left over). This data includes both doses. How does Prophet take that into account when demand planning? In short, it doesn't.
Outliers Happen (a lot)
Over a decade ago I managed the app and infrastructure team for a DNVB vitamin and supplement brand. During routine maintenance, I accidentally deleted a critical database in our production environment that held our inventory information for our online store. This caused a ripple effect which required us to bring down the seven sites we managed for nearly 24 hours while we restored backups.
This "blip" showed up on reports for nearly 2 years. This outage had two long-term effects on how we operated the business. Firstly, there was a nagging cognitive load required to factor out that revenue impact, despite what dashboards and reports were saying. This was a consistent source of frustration, calling into question every decision based on the downtime event. Secondly, we had a constant tendency to over-index our decisions based on learnings from the outlier event. Those learnings weren't guided by data, but were evangelized by the loudest voices, and socialized on AIM (pre-Slack).
Outlier events open the door to operators providing narratives to explain the past. This can introduce cultural issues in the company, causing others to reinforce the narrative — potentially having learned the wrong lesson in the end.
Despite what the ownership believed, it was our team's intuition that customers are resilient and many recovered their carts in the following days (we beat that quarter's goals anyway!). But the "blip" remained a stain on the record that was talked about. Every month. For 24 months.
Not all outliers are bad. Ian Leslie, CMO at Industry West, just launched a new Progressive Web App storefront and drove over $250,000 in sales over the course of launch weekend, a first for the growing online furniture brand. The results are repeatable, but to expect outlier performance every month or every quarter would be foolish.
Factoring Out Outliers
In an effort to remove blips in our reports we average them out: we apply smoothing. This moving average keeps the outliers in check and gives us the ability to focus on general trends without the detail of specific events. Damian again on moving averages: "In most models, you would be [looking at averages]. Right? Especially when you're starting to get daily data." He's not alone. Every brand we track at Rightpoint uses a moving average to smooth out the outliers.
Rather than smoothing over the problematic data, consider more finely-grained segmentation. Rollup reports are nice, but customer behavior is often predictable. The more well-defined your segments in reporting, the more normalized you'll find your annual reports become. LTV reports should be minimally impacted by outlier events, unless you've experienced explosive net-new growth across a variety of gateway SKUs. If this is you, that's a nice problem to have.
If you're in a pandemic-durable category, then retention metrics should also be minimally impeded by the pandemic. Consider segmenting by geography, too, to cordon-off areas impacted by rolling lockdowns, such as major metros like New York, London, or Los Angeles.
Conclusion
Independent of how you track revenue-based or performance-based goals in your eCom business, you're going to have to deal with outliers. And Q1 2020 is one hell of an outlier to contend with in your analytics and reports.
You're not alone.
This is the week when YoY reports go to hell. This time last year was when the stock market fell, the then-President began screening international travelers for Covid, and, on March 5th, we will have discovered the first outbreak of 2019-nCoV in the United States in Snohomish County, Washington.
So this is the week when your reports pass through the Bermuda Triangle of the 2020 anomaly. A year in eCom that played out as a rollercoaster of data - very high highs, and very low lows. How do we account for it? How do you make sense of it?
Which Wrong to Believe
Depending on your line of business, Q4 was pretty stellar for many eCommerce verticals. Especially those in pandemic-durable categories like wellness, fitness, home furnishings, and bed and bath. QoQ planning will reflect a dip from Q4 to Q1, depending on seasonality.
For the next few months, be wary of the data your year-on-year reports are telling you. Demand, even in these same categories, plummeted for the first 3-4 weeks of the pandemic, through about Easter in the United States. This artificial suppression then rebounded slowly in the months to come but could present as a near 50-100% increase in your YoY comparisons.
Similarly, the summer months of the eCom boom may show you as trending downwards. In short, retailers have to use their judgment when dealing with outlier events. According to Damian Soong, Founder of Form Nutrition, forecasts are always wrong, and it's up to you to decide which wrong to believe:
We have three models. One is a simple year-on-year. Two is a simple month-on-month. And three is we use Facebook Prophet library, which is a machine learning algorithm. And that basically takes daily data from the last three years and churns out forecasts going forwards.
We've learned that forecasts are always wrong and it's just a question of picking the wrong one, or a combination of all three and using your judgment to come out at something sensible.
Outliers make modeling demand all the more difficult. Even with sophisticated tools like Facebook Prophet, an open-source forecasting library, models based on past performance may not be informed by present context. Today CNN reported that among the three vaccine producers we will reach 130M doses in circulation by end of March 2021, and 400 million by the end of July, enough to fully vaccinate all Americans (with nearly 70 million doses left over). This data includes both doses. How does Prophet take that into account when demand planning? In short, it doesn't.
Outliers Happen (a lot)
Over a decade ago I managed the app and infrastructure team for a DNVB vitamin and supplement brand. During routine maintenance, I accidentally deleted a critical database in our production environment that held our inventory information for our online store. This caused a ripple effect which required us to bring down the seven sites we managed for nearly 24 hours while we restored backups.
This "blip" showed up on reports for nearly 2 years. This outage had two long-term effects on how we operated the business. Firstly, there was a nagging cognitive load required to factor out that revenue impact, despite what dashboards and reports were saying. This was a consistent source of frustration, calling into question every decision based on the downtime event. Secondly, we had a constant tendency to over-index our decisions based on learnings from the outlier event. Those learnings weren't guided by data, but were evangelized by the loudest voices, and socialized on AIM (pre-Slack).
Outlier events open the door to operators providing narratives to explain the past. This can introduce cultural issues in the company, causing others to reinforce the narrative — potentially having learned the wrong lesson in the end.
Despite what the ownership believed, it was our team's intuition that customers are resilient and many recovered their carts in the following days (we beat that quarter's goals anyway!). But the "blip" remained a stain on the record that was talked about. Every month. For 24 months.
Not all outliers are bad. Ian Leslie, CMO at Industry West, just launched a new Progressive Web App storefront and drove over $250,000 in sales over the course of launch weekend, a first for the growing online furniture brand. The results are repeatable, but to expect outlier performance every month or every quarter would be foolish.
Factoring Out Outliers
In an effort to remove blips in our reports we average them out: we apply smoothing. This moving average keeps the outliers in check and gives us the ability to focus on general trends without the detail of specific events. Damian again on moving averages: "In most models, you would be [looking at averages]. Right? Especially when you're starting to get daily data." He's not alone. Every brand we track at Rightpoint uses a moving average to smooth out the outliers.
Rather than smoothing over the problematic data, consider more finely-grained segmentation. Rollup reports are nice, but customer behavior is often predictable. The more well-defined your segments in reporting, the more normalized you'll find your annual reports become. LTV reports should be minimally impacted by outlier events, unless you've experienced explosive net-new growth across a variety of gateway SKUs. If this is you, that's a nice problem to have.
If you're in a pandemic-durable category, then retention metrics should also be minimally impeded by the pandemic. Consider segmenting by geography, too, to cordon-off areas impacted by rolling lockdowns, such as major metros like New York, London, or Los Angeles.
Conclusion
Independent of how you track revenue-based or performance-based goals in your eCom business, you're going to have to deal with outliers. And Q1 2020 is one hell of an outlier to contend with in your analytics and reports.
You're not alone.
This is the week when YoY reports go to hell. This time last year was when the stock market fell, the then-President began screening international travelers for Covid, and, on March 5th, we will have discovered the first outbreak of 2019-nCoV in the United States in Snohomish County, Washington.
So this is the week when your reports pass through the Bermuda Triangle of the 2020 anomaly. A year in eCom that played out as a rollercoaster of data - very high highs, and very low lows. How do we account for it? How do you make sense of it?
Which Wrong to Believe
Depending on your line of business, Q4 was pretty stellar for many eCommerce verticals. Especially those in pandemic-durable categories like wellness, fitness, home furnishings, and bed and bath. QoQ planning will reflect a dip from Q4 to Q1, depending on seasonality.
For the next few months, be wary of the data your year-on-year reports are telling you. Demand, even in these same categories, plummeted for the first 3-4 weeks of the pandemic, through about Easter in the United States. This artificial suppression then rebounded slowly in the months to come but could present as a near 50-100% increase in your YoY comparisons.
Similarly, the summer months of the eCom boom may show you as trending downwards. In short, retailers have to use their judgment when dealing with outlier events. According to Damian Soong, Founder of Form Nutrition, forecasts are always wrong, and it's up to you to decide which wrong to believe:
We have three models. One is a simple year-on-year. Two is a simple month-on-month. And three is we use Facebook Prophet library, which is a machine learning algorithm. And that basically takes daily data from the last three years and churns out forecasts going forwards.
We've learned that forecasts are always wrong and it's just a question of picking the wrong one, or a combination of all three and using your judgment to come out at something sensible.
Outliers make modeling demand all the more difficult. Even with sophisticated tools like Facebook Prophet, an open-source forecasting library, models based on past performance may not be informed by present context. Today CNN reported that among the three vaccine producers we will reach 130M doses in circulation by end of March 2021, and 400 million by the end of July, enough to fully vaccinate all Americans (with nearly 70 million doses left over). This data includes both doses. How does Prophet take that into account when demand planning? In short, it doesn't.
Outliers Happen (a lot)
Over a decade ago I managed the app and infrastructure team for a DNVB vitamin and supplement brand. During routine maintenance, I accidentally deleted a critical database in our production environment that held our inventory information for our online store. This caused a ripple effect which required us to bring down the seven sites we managed for nearly 24 hours while we restored backups.
This "blip" showed up on reports for nearly 2 years. This outage had two long-term effects on how we operated the business. Firstly, there was a nagging cognitive load required to factor out that revenue impact, despite what dashboards and reports were saying. This was a consistent source of frustration, calling into question every decision based on the downtime event. Secondly, we had a constant tendency to over-index our decisions based on learnings from the outlier event. Those learnings weren't guided by data, but were evangelized by the loudest voices, and socialized on AIM (pre-Slack).
Outlier events open the door to operators providing narratives to explain the past. This can introduce cultural issues in the company, causing others to reinforce the narrative — potentially having learned the wrong lesson in the end.
Despite what the ownership believed, it was our team's intuition that customers are resilient and many recovered their carts in the following days (we beat that quarter's goals anyway!). But the "blip" remained a stain on the record that was talked about. Every month. For 24 months.
Not all outliers are bad. Ian Leslie, CMO at Industry West, just launched a new Progressive Web App storefront and drove over $250,000 in sales over the course of launch weekend, a first for the growing online furniture brand. The results are repeatable, but to expect outlier performance every month or every quarter would be foolish.
Factoring Out Outliers
In an effort to remove blips in our reports we average them out: we apply smoothing. This moving average keeps the outliers in check and gives us the ability to focus on general trends without the detail of specific events. Damian again on moving averages: "In most models, you would be [looking at averages]. Right? Especially when you're starting to get daily data." He's not alone. Every brand we track at Rightpoint uses a moving average to smooth out the outliers.
Rather than smoothing over the problematic data, consider more finely-grained segmentation. Rollup reports are nice, but customer behavior is often predictable. The more well-defined your segments in reporting, the more normalized you'll find your annual reports become. LTV reports should be minimally impacted by outlier events, unless you've experienced explosive net-new growth across a variety of gateway SKUs. If this is you, that's a nice problem to have.
If you're in a pandemic-durable category, then retention metrics should also be minimally impeded by the pandemic. Consider segmenting by geography, too, to cordon-off areas impacted by rolling lockdowns, such as major metros like New York, London, or Los Angeles.
Conclusion
Independent of how you track revenue-based or performance-based goals in your eCom business, you're going to have to deal with outliers. And Q1 2020 is one hell of an outlier to contend with in your analytics and reports.
You're not alone.
This is the week when YoY reports go to hell. This time last year was when the stock market fell, the then-President began screening international travelers for Covid, and, on March 5th, we will have discovered the first outbreak of 2019-nCoV in the United States in Snohomish County, Washington.
So this is the week when your reports pass through the Bermuda Triangle of the 2020 anomaly. A year in eCom that played out as a rollercoaster of data - very high highs, and very low lows. How do we account for it? How do you make sense of it?
Which Wrong to Believe
Depending on your line of business, Q4 was pretty stellar for many eCommerce verticals. Especially those in pandemic-durable categories like wellness, fitness, home furnishings, and bed and bath. QoQ planning will reflect a dip from Q4 to Q1, depending on seasonality.
For the next few months, be wary of the data your year-on-year reports are telling you. Demand, even in these same categories, plummeted for the first 3-4 weeks of the pandemic, through about Easter in the United States. This artificial suppression then rebounded slowly in the months to come but could present as a near 50-100% increase in your YoY comparisons.
Similarly, the summer months of the eCom boom may show you as trending downwards. In short, retailers have to use their judgment when dealing with outlier events. According to Damian Soong, Founder of Form Nutrition, forecasts are always wrong, and it's up to you to decide which wrong to believe:
We have three models. One is a simple year-on-year. Two is a simple month-on-month. And three is we use Facebook Prophet library, which is a machine learning algorithm. And that basically takes daily data from the last three years and churns out forecasts going forwards.
We've learned that forecasts are always wrong and it's just a question of picking the wrong one, or a combination of all three and using your judgment to come out at something sensible.
Outliers make modeling demand all the more difficult. Even with sophisticated tools like Facebook Prophet, an open-source forecasting library, models based on past performance may not be informed by present context. Today CNN reported that among the three vaccine producers we will reach 130M doses in circulation by end of March 2021, and 400 million by the end of July, enough to fully vaccinate all Americans (with nearly 70 million doses left over). This data includes both doses. How does Prophet take that into account when demand planning? In short, it doesn't.
Outliers Happen (a lot)
Over a decade ago I managed the app and infrastructure team for a DNVB vitamin and supplement brand. During routine maintenance, I accidentally deleted a critical database in our production environment that held our inventory information for our online store. This caused a ripple effect which required us to bring down the seven sites we managed for nearly 24 hours while we restored backups.
This "blip" showed up on reports for nearly 2 years. This outage had two long-term effects on how we operated the business. Firstly, there was a nagging cognitive load required to factor out that revenue impact, despite what dashboards and reports were saying. This was a consistent source of frustration, calling into question every decision based on the downtime event. Secondly, we had a constant tendency to over-index our decisions based on learnings from the outlier event. Those learnings weren't guided by data, but were evangelized by the loudest voices, and socialized on AIM (pre-Slack).
Outlier events open the door to operators providing narratives to explain the past. This can introduce cultural issues in the company, causing others to reinforce the narrative — potentially having learned the wrong lesson in the end.
Despite what the ownership believed, it was our team's intuition that customers are resilient and many recovered their carts in the following days (we beat that quarter's goals anyway!). But the "blip" remained a stain on the record that was talked about. Every month. For 24 months.
Not all outliers are bad. Ian Leslie, CMO at Industry West, just launched a new Progressive Web App storefront and drove over $250,000 in sales over the course of launch weekend, a first for the growing online furniture brand. The results are repeatable, but to expect outlier performance every month or every quarter would be foolish.
Factoring Out Outliers
In an effort to remove blips in our reports we average them out: we apply smoothing. This moving average keeps the outliers in check and gives us the ability to focus on general trends without the detail of specific events. Damian again on moving averages: "In most models, you would be [looking at averages]. Right? Especially when you're starting to get daily data." He's not alone. Every brand we track at Rightpoint uses a moving average to smooth out the outliers.
Rather than smoothing over the problematic data, consider more finely-grained segmentation. Rollup reports are nice, but customer behavior is often predictable. The more well-defined your segments in reporting, the more normalized you'll find your annual reports become. LTV reports should be minimally impacted by outlier events, unless you've experienced explosive net-new growth across a variety of gateway SKUs. If this is you, that's a nice problem to have.
If you're in a pandemic-durable category, then retention metrics should also be minimally impeded by the pandemic. Consider segmenting by geography, too, to cordon-off areas impacted by rolling lockdowns, such as major metros like New York, London, or Los Angeles.
Conclusion
Independent of how you track revenue-based or performance-based goals in your eCom business, you're going to have to deal with outliers. And Q1 2020 is one hell of an outlier to contend with in your analytics and reports.
You're not alone.
THIS ARTICLE IS FOR MEMBERS ONLY
Insights and futurism for executives in eCom and Retail
Those things we shouldn’t say out loud? We say them on the private feed. Bi-weekly “after dark” podcasts and a members-only newsletter, just for subscribers.
Our research reports combine visionary thinking with data-backed findings from our own advisory panel, made up of leaders at brands you know and trust.
Query and prompt our vast archive of research, podcasts, and newsletters with a ChatGPT-like interface. Get exclusive access to Alani™, the AI-powered engine for Future Commerce, powered by BundleIQ.
