If you’re using TryMyUI or UserTesting, two popular testing websites, you may wonder if the grass is greener on the other side of the fence. And if you haven’t used either platform before, it’s worth investigating whether they’re a better option than what you’re doing now to conduct user testing.
The answer to which is best — TryMyUI, UserTesting, or none of the above — depends on what you need most from the platform. If you know what you’re testing (App navigation? Low-fidelity homepage prototypes? Brand messaging?) and how often you’re doing that kind of testing, it’s easier to evaluate which user testing services are a good fit. There are several factors to consider, such as:
- The range of the participant panel (and the level at which you can engage with them)
- The types of testing available (card sorting, live web page testing, moderated or unmoderated sessions, etc.)
- The ability to analyze and share data in engaging and insightful visuals and graphs.
Making the wrong choice can be an expensive mistake. That’s why this piece takes an in-depth look at TryMyUI and UserTesting, covering the main features of each service (as well as key differences), detailing their features and benefits, and making recommendations throughout.
To prepare this piece, we gathered information from many sources, including users of both platforms. Hat tip to Val Geisler of Fix My Churn for sharing her experience, along with the users who shared their insights anonymously.
Note: Looking for a specific audience to participate in your user research? User Interviews offers a complete platform for finding and managing participants. Tell us who you want, and we’ll get them on your calendar. Find your first three participants for free.
Platform Capabilities: What Tests Can You Run?
The first question to ask is: Do either of these platforms provide the tooling I need? We’ll go over the testing capabilities of each, breaking down the services they offer their clients before going into how they source participants.
TryMyUI Testing Capabilities
TryMyUI offers remote user testing, mobile user testing, wireframe and prototype testing, impression testing, written surveys, and demographic curation (more on demographic curation in our sourcing participants section). These are TryMyUI’s basic features, available through any plan (personal or team and enterprise), though the cost for mobile website and app testing is more expensive if you’re on a pay-as-you-go plan.
Positioned as their cornerstone service, TryMyUI offers unmoderated, remote user testing, which allows you to listen and observe as users navigate your website or app. You get a screencap video that includes mouse movements, taps, swipes, and keypresses, plus real-time audio captured from the user’s microphone. At the end of the test, the users provide written responses to a post-test survey.
The post-test survey is completely customizable. It’s a great way to ask specific, pointed questions, targeted to get the insight you need. If you don’t have any specifics in mind, TryMyUI includes four default questions aimed at soliciting more information.
When making your test, select whether you’re testing a live site or mobile app. If it’s a live site, select desktop or mobile testing. Mobile testing records swipes and taps, where desktop testing measures mouse interactions. Testing is available across all major mobile devices.
If you’re looking to test a design before putting it into development, TryMyUI offers wireframe and prototype testing in all of its plans. You can create a high or low-fidelity prototype and make a link so the testers at TryMyUI can access it from their browser.
Researchers can also conduct impression testing, where participants are given a 15-second experience of your website. After 15 seconds, the screen goes black, and the tester is asked to describe the feel of the site. What words or titles stood out? Was there one specific product or service that caught their eye? This helps you see whether, with just a quick look, your company’s website comes across as, say, a SaaS bookkeeping company vs. a traditional accounting firm.
UserTesting Platform Capabilities
UserTesting’s Human Insight platform is expansive, covering website usability (including mobile web, desktop, and mobile app usability), reactions to product concepts and marketing concepts, actual in-store customer experience, employee experience, and more.
Within the platform, UserTesting breaks its services down into two categories: Insight Core (tests you can customize) and Product Insight (preformatted testing options). Both of these services include Live Conversation, a tool that allows for live, moderated user testing.
Live Conversation: UserTesting’s Moderated Interviews
UserTesting’s Live Conversation lets you schedule live video interviews with your exact customers nationwide. By using UserTesting’s diverse “global panel” or your own, you can use Live Conversation to conduct face-to-face interviews. Talking with users directly allows for immediate and unfiltered feedback.
Live Conversation uses Zoom and integrates with your Google Calendar. Plus, interviews are automatically recorded, so you can reference important insights after the call.
Insight Core: UserTesting’s Most Adaptable Tools
Right off the bat, UserTesting simply offers more testing capabilities than TryMyUI. We will first cover the main features UserTesting is known for and then briefly cover their other features.
Here’s a high-level comparison of the basic testing capabilities for each platform:
A major difference between the two services it that UserTesting offers moderated remote usability testing, letting you talk to the participants.
UserTesting also offers:
- Comprehension Testing: See if specific copy or an image is conveying the desired message.
- Competitor Studies: Recruit a participant to interact with your main competitors and track their experiences (sometimes called a competitive study).
- Qualitative Samplings of an Online Survey: Ask both open-ended and multiple-choice questions to a large enough audience to capture statistically significant results about user behavior and attitudes towards a specific topic or area of interest.
- Preference Testing: Let your participants interact with various designs to determine which one they prefer and why.
- Benchmark Studies: Use a benchmark study to measure and compare usability metrics against an initial study. These studies typically run on a consistent schedule (such as monthly or quarterly) to evaluate how reactions and attitudes toward your product have changed over time.
- Diary Studies: Track your user experience over a longer period of time (referred to as a longitudinal study that usually lasts weeks or months).
- Multi-Channel and Omni-Channel Testing: Recruit participants to complete one test across multiple channels or take a test which requires multiple channels to complete, respectively.
- Tree Testing: Show participants a text-only version of your site or app structure to evaluate how easy it is to find information, and pin-point where most users get lost.
Product Insight: UserTesting’s Pre-formatted Testing Tools
UserTesting’s Product Insight focuses on pre-formatted tests (called Quick Answer Tests, which we go over in detail below), same-day video interviews with target customers, and Live Conversation.
You’d choose the Quick Answers format over Live Conversation when you want to use pre-written templates as written surveys for your testers. Quick Answers is broken into six pre-formatted questions:
- Discover Needs and Frustrations: Explore your customers’ experiences (what they liked or didn’t like, what they found confusing and what they found clear) with a task you specify.
- Validate an Idea: Use this test before making a full prototype, to test a concept or high-level idea.
- Validate a Prototype: Learn how participants respond to your prototype or wireframe. This works best when your prototype is exactly what you want it to be before testing starts.
- Test Website Usability: Use this test to find pain points in your design and flow. You can also do usability testing on a competitor’s website to find opportunities on how to stand out amongst the competition.
- Choose Between 2 Options: Have two competing versions? Use this Quick Answers format to test which option users prefer.
- Compare 2 Websites: Test two websites and see which one participants prefer by tracking how easily they complete a specific task. You can also compare your website to a competitor’s.
Note: Looking for a specific audience to participate in your user research? User Interviews offers a complete platform for finding and managing participants. Tell us who you want, and we’ll get them on your calendar. Find your first three participants for free.
Sourcing Research Participants: Who Can You Get Insights From?
Both platforms offer advanced filters that can help you get granular with your recruiting needs (and recruit by expertise level as well as more standard brackets), but these filters are only available in higher-tiered plans.
Feedback about TryMyUI and UserTesting confirms both platforms offer a wide range of testers. However, looking at feedback from users, TryMyUI participants were more likely to 1) not show up or 2) not follow directions during the test. That said, TryMyUI is committed to making it right and will arrange for a replacement test free of charge.
In this section, we’ll discuss how TryMyUI and UserTesting both source participants and how they address the issue of sourcing quality participants whose behaviors match the needs of your study.
TryMyUI Recruiting
Through TryMyUI, you can ask screener questions (plus have the ability to bring your own users into testing) to help filter out any testers who aren’t a good fight.
TryMyUI says it makes its tests “completely customizable” to better help you get the feedback you’re after. TryMyUI’s demographic curation lets you narrow down your participants by gender, age, country of residence, household income, education level, employment type, family status, community type, and social network.
You can also get more specific, requesting your participants be people who are consistently using iPhone games or who have traveled overseas. But there isn’t a guarantee that they can find these types of testers.
UserTesting Recruiting
UserTesting has two ways to recruit participants: the UserTesting Panel and MyRecruit.
UserTesting Panel is the standard option. Here, you get to set demographic filters and get participants from a large, diverse group (UserTesting reports more than one million participants in its pool).
The basic demographic filters are:
- Age
- Income
- Gender
- Location.
In advanced plans, you get significantly more filters, including employment status and industry (which includes company size, job role, and level), web expertise, social networks used, and more. These additional demographic filters are available for computer, mobile, or app studies.
In advanced plans, you can also use multiple-choice screener questions to weed out any participants who don’t match the behaviors you’re looking for.
While the majority of panelists on UserTesting are located in the US, Canada, UK, India, and Austria, they also have smaller panels in a number of countries, and you can get feedback in additional languages such as French, Spanish, and German. As far as we can tell, TryMyUI doesn’t currently offer any non-U.S. or Canada market testing and can only return feedback in English.
UserTesting uses a proprietary system to weed out fraudulent users, and they report only 5% of panel applicants are approved to participate in studies. This 5% of approved panelists are rated, after every test, by researchers to keep the quality high and consistent.
For more niche industries, or for feedback that is directly from your client base, you would need My Recruit.
A completely self-service feature, you can use My Recruit to launch a study with anyone at anytime. Simply create the study in the UserTesting dashboard, create a link, and share with your own participants. This allows you to go outside of the UserTesting panel and find the best participants for your study.
Speaking of Finding the Best Participants...
Just because someone is in your desired age, income bracket, and location doesn’t mean they are just like your customers. Demographics are only the start; if you need deep insights from your customer base, you’ll need to screen for behavior as well.
If finding quality participants is your main challenge, consider a different approach altogether. TryMyUI doesn’t have all the features of UserTesting, and UserTesting is an expensive service for researchers whose main focus is quality participants.
Some researchers use platforms like UserTesting to find participants because they think that’s the only viable solution. This simply isn’t true. If you’re mainly using UserTesting for access to qualified participants, then by paying for access to all the tools you don’t use, you’re overpaying.
At User Interviews, you can create a screener survey targeted toward behaviors (not just demographics). Then we share it with our 200k+ participant pool. If that’s not enough, we’ll find more people who fit the characteristics you’re looking for. When an applicant fits the criteria you created, a match is made. From there, you can qualify or disqualify the participant based on their answers and confirm their identity via their LinkedIn and Facebook profiles.
As a user research recruiting platform, we are focused on speed, ease of use, and the quality of our participants. We have our researchers rate each participant after testing. This vetting process helps keep only quality participants in our pool. Plus, we handle your scheduling and incentive payments, giving you more time to focus on analyzing and implementing user feedback.
User Interviews works with any user testing software you prefer. We offer pay-as-you-go pricing and monthly subscriptions. For pay as you go, consumer testing is $30 per completed session, and professional testing is $70 per completed session. If you’re doing consistent research, consider the plans below.
(Note: At $300 per month plans and up, our advanced screening option lets you talk with participants before scheduling their session.)
Analyzing and Sharing Data: What Are Your Options?
This part of the comparison will probably be the most subjective. You know the needs of your stakeholders and how they prefer to receive information better than we do.
From what we’ve seen, TryMyUI positions itself as a tool that can be used to present directly to decision makers (who may not be steeped in user research every day), whereas UserTesting positions itself as a tool made by researchers and for researchers.
TryMyUI Data Analysis Options
TryMyUI breaks down its sharing and collaborative features into three categories: The UXCrowd, UX Diagnostics, and UX Sprint.
The UXCrowd creates a mind-map of user insight that allows you to see what opinions are most reinforced.
The idea here is to pull out specific statements (sorting them into positives and negatives) and present the data for easy viewing. TryMyUI explains, “Even if you run 50 tests, UXCrowd works to offer every bit of insight out of them without having to watch a thousand minutes of video. It also liberates the challenge of prioritizing product problems from personal biases, allowing an objective vote-count to point the way.”
The downside, of course, is that you can’t learn from the verbal and non-verbal clues a participant gives when speaking.
UX Diagnostics aims to supplement more qualitative feedback (what a tester did or did not like about an experience) with quantitative data that can be used to show more pointed analysis. TryMyUI’s UX Diagnostics consists of psychometrics, task usability scores, task completion rates, and task duration data.
Here’s how those features work together:
- Psychometrics, such as the System Usability Scale (SUS) and Post-Study System Usability Questionnaire (PSSUQ), to measure overall usability.
- Task usability scores pose one question (“Overall, how difficult or easy did you find this task?”) and has the tester respond from 1 (very difficult) to 7 (very easy).
- Task completion rates and task duration data show you which tasks were the most challenging as well as how long it took (on average) for a tester to complete the test.
UX Sprint helps you track how well your feedback is being implemented. Over a series of tests, iterative user testing charts the ease of a task. This cumulative feedback lets you see if your changes are helping or hurting the functionality of your site or service.
UserTesting Data Analysis Options
UserTesting offers transcripts, and it lets you create notes, tag themes, and sort info into folders. It also lets you create highlight reels. As you watch your recorded video, you can clip parts of the video you want to share with your stakeholders. After clipping several parts, you have a highlight reel.
The highlight reel, like all videos from UserTesting, can be sent through email or slack or even downloaded to your computer.
Finally, UserTesting offers Metrics and Enhanced Metrics.
The Metrics tab shows test participants' responses and activities broken down by test question or task. UserTesting takes the metrics and pairs them with test video. Identify the areas that look the most interesting and hover over the metric to pull up the appropriate video clips, labeled by the tester. When you click to play the video, the video will start playing at the moment where the specific feedback was recorded.
Plus, UserTesting offers Enhanced Metrics to Insight Core premium subscribers.
Enhanced Metrics lets you view key metrics, such as average time on task, the average number of clicks, average number of pages visited, and more.
There’s a significant difference in how TryMyUI and UserTesting handle analyzing and sharing data. Given their emphasis on human insight and the ability of their participants to articulate the why behind the what, it makes sense that UserTesting’s features here aim to combine their video sessions side by side with the collected data.
Note: Looking for a specific audience to participate in your user research? User Interviews offers a complete platform for finding and managing participants. Tell us who you want, and we’ll get them on your calendar. Find your first three participants for free.
Pricing and Plans: Does the Platform Fit in Your Budget?
TryMyUI Pricing and Plans
TryMyUI’s pricing is transparent and readily available on their website. As of now, they offer a personal plan, team plan, and enterprise plan.
The personal plan has no monthly fee. It costs 1 credit ($35) for every desktop test and 2 credits ($70) for every mobile test. In this personal plan, you get up to 20 minutes of video and audio feedback per user, written responses to custom survey questions, the ability to make up to three timestamped annotations on your video results, the ability to compile one 5-minute highlight reel, the use of project folders, and the space to store your 10 most recent user videos.
The team plan through TryMyUI is $299 a month and comes with 10 test credits (so, if you plan on doing 10 or more desktop tests a month or five or more mobile tests per month, the team plan already saves you money). You can also use your own users to perform 10 additional tests every month. Plus, with the team plan, you get extended 30-minute test videos, one more login seat (so you plus one other person can use the TryMyUI account), collaborative video attention, the ability to make three highlight reels a month, the features from UXCrowd, UX Diagnostics, and UX Sprint, and you can download video results and test data.
TryMyUI’s enterprise plan starts at $1,500 a month and comes with a minimum of 30 test credits (and the option to use unlimited testing with your own users). Plus, you get extended 30-60 minute test videos, four additional seats for logins, collaborative video annotation, unlimited highlight reels, one-click report generation with video playback, the features from UXCrowd, UX Diagnostics, and UX Sprint, the ability to download video test results and test data, plus transcripts for your video results and the capability to record all live user sessions on your website with TryMyUI stream.
UserTesting Pricing and Plans
UserTesting has two broad plans: Individual and Enterprise. However, transparent and consistent pricing isn’t available for either of these plans. We pulled whatever information we could from third-party sources to make this section as comprehensive as possible.
The Individual plan gets a researcher up to 15 video sessions (to watch videos of users speaking their thoughts on your website, app, or prototype). While UserTesting doesn’t provide transparent pricing, we’ve found users reporting the price per test at around $49 or more.
In this plan, a researcher also gets access to:
- A large, diverse UserTesting panel
- A test template
- Basic testing capabilities.
The Enterprise plan is more customizable. UserTesting doesn’t mention how many video sessions come with this plan (it’s probably based on how many you need, which in turn affects the price).
In the Enterprise plan, a team gets access to:
- Advanced testing capabilities (all of the capabilities we covered in this post, plus more)
- Administrative controls
- Enhanced video players
- Advanced recruiting capabilities
- Professional services
- Quantitative metrics.
TryMyUI vs. UserTesting: Deciding What’s Right for You
UserTesting is a bigger platform in almost every way (though they seem to offer less when it comes to sharing and analyzing data). UserTesting offers more services, more participants, and more engagement with your participants.
TryMyUI is more transparent in its pricing and is less expensive (based on the information we were able to find), but it has significantly fewer testing options available for its researchers.
Ultimately, if you only need unmoderated remote testing, a few other barebones testing features, and participants sourced mainly via demographic information, TryMyUI is probably a good option.
If you need a larger participant panel, moderated and unmoderated testing, and a wider variety of testing options, UserTesting is worth examining further.
But if you’re a researcher looking for a specific audience to participate in your UX research, and you don’t need the long list of features that raise the monthly fees on UserTesting (or you want to source participants your own way, then bring to the UserTesting platform), User Interviews offers a complete platform for finding and managing participants.
Tell us who you want, and we’ll get them on your calendar. Find your first three participants for free.
