Technical Skills: Market & Competitor Research, SEO, Surveys , and A/B Testing
In March 2018, I was approved as one of the first Airbnb Experiences in Los Angeles. At the time, Airbnb positioned the new “Experiences” offering as centering on the local “expert.”
To make it through the rigorous approval process, I needed not only to sell my idea – a tour that combined history, drinking, and historic bars- but also myself. In December 2018, after months of research and preparation, I gave my first tour under the name “Speakeasy Bar Crawl with a Historian.”
The historian moniker and positioning served their purpose wonderfully. To my utter delight, it dissuaded party animals, attracted a wide range of people interested in history, and led to less drunken guests. It was a win-win.
It was also a win in terms of bookings. The name and the first photo were the only things potential guests were able to see in the search, and they liked what they saw. Within the first 30 days, I was sold out of all my tour inventory for January and February.
Since the guests knew what to expect from the name, my ratings and Net Promoter Score quickly climbed. By the end of February, almost three months since tours began, I was ranked a Top 3 Nightlife Experience in Los Angeles by Airbnb. I was also getting five inquiries a day asking me to “fit them in.” As I was still working a full-time job, I just couldn’t keep up with the demand!
After much thought, I decided to hire three additional part-time tour guides. Instead of hiring UCLA or USC history majors, I decided to recruit young professionals with interest in tourism, which could bring their distinctive flair to the tour. Polly was a film costume assistant by day, and during the tour added terrific facts about the history of film and fashion. Sam was a true crime nut who could speak about a few of our most sinister stops for hours. Lastly, Ryan was an experienced tour guide who was passionate about guest service.
As much as I loved my new hires, I found myself with a big problem. Although I could reasonably claim I was a historian after studying history at Oxford University, my new hires were not historians by any stretch of the imagination.
Suddenly, my tour needed a new name.
As the name and initial photography were all a potential customer could see in Airbnb search, I knew my future business depended on getting this right. With so much riding on a string of 4 – 6 words, I decided to take a methodological approach to choose a new name.
Step 1: Market Research
First, I conducted a thorough analysis of competitors within the marketplace. As my tour was listed in both “History” and “Nightlife” categories, I focused my efforts on analyzing other experiences that were also performing well in terms of ratings and search rank. I hypothesized that Airbnb’s algorithm typically promoted experiences that had a high conversion rate.
I found that although around 30% of experiences included a persona in their title, the majority focused on describing the experience itself.
Inspired by my knowledge of SEO techniques, I decided to manually scrape 50 of the top nightlife category listings in Los Angeles and other more mature Airbnb Experience markets like San Francisco and New York City.
I then used Excel’s Power Query to create an Excel document that displayed every word utilized in the category. As a spreadsheet with ~700 words wasn’t valuable without seeing the frequency distribution, I decided to create a word cloud using beautiful.ai. The phrase cloud allowed me to visualize the trends in title keywords. Through this analysis, I was able to determine the following key insights:
- Disproportionally, the top 50 tours in the category often included the name of the city within their title
- Speakeasy, Sunset, Comedy, Explore and Tour were the top ‘Descriptor Words’
- Ultimate, Official were the top ‘Supplementary Words’
In coming up with new names, I wanted to leverage some of the keywords my market research analysis had discovered. However, I also wanted to ensure I could differentiate my tour from the competition in the same way that my “with a historian” positioning had while maintaining my current customer base.
Unlike many other Airbnb Experiences locally, my customers consisted of 30% locals and 70% international and domestic tourists from 24 to 58 years old. Interestingly, around 20% of my bookings were multi-generational families of 6+ people (parents with their adult children). It was vital that I would not alienate them with a too young sounding name.
After reviewing the analysis and word cloud, I sought out to ideate names that included the terms Los Angeles or LA and speakeasy. I also wanted to maintain my history positioning as it was the differentiating factor of my tour. Hence, I came up with four possibilities:
- Drunk History Speakeasy Bar Crawl
- Intoxicated History Speakeasy Bar Crawl
- Historic Speakeasy Bar Crawl
- LA’s Historic Speakeasy Bar Crawl
Step 2: Survey
Many early-stage startups fall into the trap of having little to no access to their target customer. With such a broad customer base, existing customers, and an extensive social network, I found my customers incredibly easy to access.
I conducted a simple one-question survey on both Airbnb and Facebook to capture different responses.
Existing Customers: First, I polled customers on the Airbnb messenger platform who had booked or taken my Airbnb Experience. This survey method had around a ~65% response rate.
Potential Customers: Secondly, I leveraged my existing social networks and posted the survey question as a poll in local young-professional Facebook groups and on my personal channels.
Step 3: A/B Testing
The survey results revealed two front-runners with equally passionate advocates for both:
- Drunk History Speakeasy Bar Crawl
- LA’s Historic Speakeasy Bar Crawl
Instead of merely choosing one, I decided to conduct an A/B Test to determine a long-term name and positioning for my product.
Although Airbnb does have an A/B test functionality built into the platform, I was able to jerry-rig a solution using the “stats” dashboard, which displays a 7-day overview of page views and bookings.
I tried to isolate as many variables as possible in the experiment. Using data from the LA Tourism, I identified two weeks in late February that did not fall within citywide conventions or significant sporting events to run controlled A/B tests.
First, I ran the title “Drunk History Speakeasy Bar Crawl” for a period of 7 days, starting on Saturday at midnight and running through Friday at 11:59 pm. At the end of 7 days, I recorded the stats dashboard and also included additional data points like number of inquiries and messages. However, it was clear within a few days that there was a drop off from the previous 7-day period when my tour was running under the “Speakeasy Bar Crawl with a Historian” name.
In the second stage, I switched the title to “LA’s Historic Speakeasy Bar Crawl” for seven days. Within the first three days, I noticed a substantial increase in the number of bookings.
At the end of the experiment, “LA’s Historic Speakeasy Bar Crawl” reigned supreme. Not only did it generate more page views on average, but it also converted potential customers to paid users at a 14% higher rate.
