When to use a Beta Program: A beta can be a great tool for deep & robust customer insight in a variety of situations. But it’s not always necessary. To run an effective beta you need a lot of oversight. You’ll also need good understanding of what types of feedback you’ll bring in and how to action it. Based on the need, there are a few approaches and tools to get the most out of the program. Your team will need to be ready to consume the data and react quickly.
Types of beta programs
– Fully managed: usually best for new products. Fresh launches to a small tightly monitored user group. Best for an existing brand with large budget because the cost of monitoring & follow up to gather and combine qualitative & quantitative data. Potential customers are usually recruited, often monetized, and there is a combination of user analytics, surveys, and live conversations to build out a strong understanding of the customer journey. Anywhere from 10-20 beta users. To select this option, there will be medium confidence that solution addresses the customer pain points.
– Medium managed: the middle option is usually used by small or medium businesses. Usually this program is leveraged as a go/no-go criteria for a large change. For example, a full–app redesign. In this model, 10-50 users are recruited to give feedback through a survey & user analytics. Cutting out live managed discussions drastically reduce the overhead and gives enough for a launch decision, but doesn’t show much in terms of how or why users act the way they do. There’s usually medium / high confidence that the solution addresses the customer pain points.
– Light managed: In this model, customers may not even know they’re in a beta. A new build can be pushed to them using tools like test flight, Mixpanel, or any other user analytics tool. Data is monitored and a full roll-out decision is made. Usually used in existing products with active users at scale. There should be high confidence that the solution addresses the customer pain point, or builds on existing working solution.
Make sure to choose the right solution for your company based primarily on what you want out of the program, the scale your product has in market, and the ability to collect and react to the feedback. Also, make sure to integrate the program with more general product design and user testing workflows. The idea is to have consistent and complementary feedback from different sources, which will reduce the inherent bias of any single source. Don’t think about this type of program as zero sum with user testing!
Additional questions to think about:
- Is the beta for a specific feature or more of an ongoing program?
- Do you already have a list of “engaged users” that give feedback on the product? Can they be scaled?
- Can you leverage a user analytics sustem for segmentation to white list certain users for the test experience?
- Are you willing to offer incentives tp users for feedback?
- How will the program integrate with the general design and user testing programs?