Earlier this year, Retail X Series founder Sapna Shah invited me to be the featured speaker for one of her popular fireside chats. The topic of this particular conversation was social media advertising in today’s environment. Initially scheduled to take place at an event center in New York City, the conversation was moved to Zoom due to COVID-19. Despite the venue change, not to mention the ongoing pandemic, Sapna’s event reached its max capacity of 100. It was a privilege to take part in.
The audience was made up primarily of founders of early-stage retail startups. Sapna, who moderated the conversation, has done an incredible job of creating a community for retail startups (see below). The first half of the conversation consisted of Sapna asking me a series of questions tailored to the audience of retail startup founders. In the the second portion of the conversation, I took questions from the audience.
Retail X Series is an excellent resource and community for all early-stage retail startups. Their website offers live informational Zoom events, podcast interviews with startup founders and VCs, a community Slack channel, and a host of other valuable materials. It’s a knowledge base specialized toward retail that everyone working in the industry should know about.
Retail X Series started in 2017 as a monthly boot camp for early stage, retail-related startups as they launch their businesses. It has since become an ecosystem for retail startups in New York and elsewhere. Think of Retail X as your “Retail 101,” relevant for preeseed and seed stage startups in the retail tech, ecommerce, fashion tech and DTC brand verticals.
Retail X Series Website
You can find the recording on the Retail X Series YouTube channel. We have provided key highlights of my answers below (organized by topic), along with timestamps.
The topics:
- Testing & Measuring Ads
- Understanding Machine Learning (And Why It Matters)
- Principles of Advertising
- Evaluating Agencies
1. Testing & Measuring Ads
4:58. “You don’t lose money from a test that doesn’t work—you lose money if you don’t actually learn anything for the test.“
11:23. “One metric that we use a lot—in terms of gauging both the creative and the precision of our model—is what we call the response rate, where we multiply the click-through rate by the conversion rate. Use that when you’re judging creative. You want creative that not only makes people tap on their phone screens, but makes them convert as well. With the creative that has the highest conversion rate, you’re killing two birds with one stone.”
24:23. “Take it one test at a time. Facebook presents so many opportunities to learn things about your audience, to learn things about what your value proposition is… I think testing can be overwhelming, and that the experiment design doesn’t actually tell you anything, because there are so many different things you’ve changed without even realizing it.”
24:46. “If you have a test that starts off poorly, have a breakpoint. Have a point where you turn it off. I always say, “take it two hundred clicks at a time.” But if, after the first hundred clicks, you’re getting CPCs of $30, don’t let it get to 200—turn it off. Try to understand what happened. And when you run your next test, start by saying what you’re doing differently, and what you think will happen as a result. Measure the results against your expectations. That’s how you’ll get really good at it, and that’s how you’ll stay sane.”
29:46. “It’s about direction, not perfection. If you’re testing one thing against another, you want to see big differences. Be careful if you’re seeing stuff that’s a little bit better or a little bit worse. We’re dealing in a world of machine learning, where randomness is at play. Unless you’re gonna test the same setup ten times in a row, it’s technically not scientific—as much as people want to act like it is. So when you try something new—whatever that change may be—if the performance is terrible compared to your first test, that’s good. Because then you can rule out a lot of stuff to not do any more of.”
43:34. “There is no such thing as true A/B testing on Facebook. It’s fundamentally impossible, because there’s no way to prevent the treatment group from being impacted by the control group.”
2. Understanding Machine Learning (And Why It Matters)
3:55. “At the core of Facebook advertising is machine learning. For the ability of machine learning to learn, you need to not give it any early biases regarding who you think your audience is. You need to give the system a chance to explore. Ultimately, you want traffic that’s going to resemble the behavior that you’re trying to scale… Facebook wants to find those people, and serve your content to those people, but the key is that you set it up, and give it a chance to do it in the first place.”
10:10. “Conversion rate is the best indicator that you have a chance to scale, and it’s also the best indicator that the system uses to leverage the full powers of machine learning. Your ad set is really a machine learning model. Most people don’t look at it like this—it’s a bit unorthodox—but it’s the true nature of the system. Machine learning is at the core. If you’re spending at a small scale and you’re not seeing a strong, consistent conversion rate, you don’t want to press the gas yet.”
44:26. “What Facebook has—that no one else has—is relational data. Using relational data they’re able to tell much more valuable information about who should see an ad next than anyone else. There’s this principle from ecology called homophily—basically, birds of a feather flock together—and here, people that have more connections with certain people are more likely to behave like those people. When you show your ad, whoever initially clicks the ad is going to dictate the next batch of people who will see it, because Facebook is looking into how groups of people are connected.”
51:20. “Your goal with any dollar you spend on Facebook is to build an overarching model—to help predict and find people who, when they see your content, will click and buy.”
51:40. “You need to figure out how to get Facebook to drive customers. It’s a lot simpler for the algorithms to try to drive awareness than it is other things, but that’s not going to translate into customers for you. And even if they do, you’re going to have to create many more campaigns, and then nurture them—it gets expensive.”
3. Principles of Advertising
6:52. “At the end of the day, if you take a step back, we’re doing direct response advertising. The enduring lessons from direct response advertising have been—going back to mail advertisements in the last century—that you want to be very specific about what the offer is, and what your product or service is offering… Being very specific is, I think, the most important thing.“
26:26. “You want to connect with your audience at some level, before you start trying to get a bigger audience. You’re gonna do better with your creative if you know what resonates with your audience organically… The best way to learn about your audience is to interact with your audience.“
4. Evaluating Agencies
15:37. “When looking at an agency, know what you’re looking for. Write down the criteria before you even start the conversation. What advantages are we going to gain by working with outside help? And really press them for specifics. Agencies will say ‘we grew such and such, this percentage”—ask them where they started. If they grew customer rates from zero to a hundred, that sounds pretty good, in percentage form. But what matters more is: where did they take the spend? Was the client spending 500,000 a year, and then they got them up to two million, or was it 500 a month and they got them up to 2,000?”
17:45. “Ultimately, measure agencies by their results. The good thing with this industry is that everything is measurable. If you have a very clear, measurable goal, you won’t get burned—as long as you’re holding them accountable to that goal.”
20:05. “I recommend that you don’t outsource all your marketing. For every organization, marketing needs to be in your DNA. But I think with these channels like Facebook, if you can find an agency that is extremely specialized and can give you superior performance, it gives you a competitive advantage. You don’t necessarily need that for other channels, but Facebook is one that really requires a sort of 24/7 obsession. You might want to have your internal people be able to work on other things, like getting other channels opened up, and seeing the big picture.”
33:05. “You’re not looking for an agency, you’re looking for athletes. It’s performance marketing—you want someone who’s going to put up the best stats. That’s who you want on your team.”
Final Thoughts
During the event, I described Facebook as a transformative channel for businesses. The potential to acquire customers at scale more efficiently than your competitors is of massive value to disruptive, early stage startups. My hope is that this outline of my best advice can help to give them that much more of a competitive edge.
A big thank you to Jake Goldstein for transcribing my answers. Believe me, he hears me talk about this stuff enough, as is.
Leave a Reply