Duck Tales: The role of user research, A/B testing, and "dogfooding" at DuckDuckGo (Ep.31)

In this episode, Beah (Chief Product Officer) and Zac (SVP, Insights) discuss our approach to user research, how we run robust research while respecting user privacy, and why moving quickly matters.

Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy.

Beah: Hello and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. In today’s episode, we’re going to talk about user research. And so my guest here is Zac. Zac, do you want to introduce yourself?

Zac: Yeah, hi, Zac Pappas. I am on the Insights team at DuckDuckGo. I’ve been here for coming up on 14 years and pretend to participate in a lot of market research, user research, and other conversations that we have where we try to infer what users want and whether or not we’re giving it to them.

Beah: Sweet, thanks. And I’m Beah. If you haven’t met me, I’m on the product team at DuckDuckGo. All right, let’s launch into it. I have questions for you, Zac. Are you ready?

Zac: Ready as you’ll ever be.

Beah: All right, so let’s start. Why don’t you give me an example? Like when we’re talking about validating things with user research, just like let’s set up an example of what that means and what that would look like.

Zac: I think so. Yeah. So I guess starting with what is user research and why do we do it? I think the obvious answer that people tend to think about are, you know, sitting in an interview or doing some sort of focus group and learning from people. But really, for us, user research is kind of all of that plus, you know, internal dialogue and real testing like A/B testing. So to put it, you know, simply, I guess, user research for us is just trying to be wrong very productively and being efficient at being wrong productively over time. So for us, everything really starts out as a guess or an assumption about users or their behavior about, you know, what is upcoming in the market and research checks that, you know, our assumptions are correct or at least as close to correct as possible. And we try to do that as cheaply as possible, meaning quickly without spending more time on it than what we’re gaining in terms of the product that we’re creating. So an example could be our homepage at DuckDuckGo.com because our homepage has been around for as long as I think DuckDuckGo has been around, but has gone through many, many, many forms. We’ve done tons of A/B testing on it, showing one version of it to some users and a different version to other users. We’ve done a lot of user testing on it, like sitting down with actual consumers of the homepage or users of the homepage. Everything from discussing with parents how they perceive the homepage. So a lot of subtlety that goes into what we think maybe appears as a pretty simple layout or design.

Beah: I had a little connection issue there for a second, but I think it’s alright. Let’s forge ahead. Sorry, audience. So, the homepage, I think, as you pointed out, sort of like deceptively, seems deceptively simple, it’s not highly functional software, it’s a page, there’s a few boxes on it, a field, but like what makes it so tricky and worthy of user research.

Zac: Yeah, good question. Because it is kind of subtle, or I guess it doesn’t seem very obvious. So if you think about it, the homepage is really where we have a big mix of different user types coming in. Making any assumption about a part of our product usually means that you have to infer who is using it, meaning are they younger or older? Do they have a lot of context about what they’re doing or very little? So our homepage is actually a mix of different user types. There’s a lot of returning users who are very familiar with DuckDuckGo, they’ve been coming to it for a very long time. And so when they come to the homepage, it’s a pretty like rote act for them. The other group that really comes to our homepage a lot are new users or new visitors to the brand and product space in general. So they have maybe very little context about what DuckDuckGo is or what products we have. And so you can imagine, you have these two very diametrically opposite groups that come in, people who have a kind of a strong habit formation and muscle memory on how to use the page and then others who don’t have anything and they kind of, you know, maybe say meander or explore more than they are, you know, directly going to the thing that they know that they want. So what we learned over years of A/B testing, user testing, and kind of regularly reviewing what the experience is like for real people, we found something kind of interesting, if I can just generalize it, which is new users — like kind of people first visiting DuckDuckGo for the first time — really look at the center of the page. And if you see DuckDuckGo.com today, it has these two prominent boxes in the middle. It depends on the browser that you’re in or the form factor that you’re on. But primarily what we know is new users will tend, their eyes kind of follow the middle of the page. And so when they land, we really want to show them an overview or a good sense of what DuckDuckGo can do for them. Whereas returning users, people who’ve been coming back for a long time, tend to want to find the search box. And so we can, or at least we have some confidence that we can move the search box around, you know, the top of the page where it’s at now, to the bottom, to the middle. And if you’re a returning user, because you kind of expect or know that it’s there, you will more easily be able to hunt and find it. So the configuration that you see today, and maybe changes as things develop, really does to us strike a very good balance between new and returning user needs, both something that a brand new user can figure out and use efficiently and kind of get a lay of the land of what DuckDuckGo is and what we do. And then returning users can quickly get to the search box or other more mechanical parts of the product without too much friction.

Beah: So how did we actually learn that? Like what testing methodology did we use? Can you walk me through it?

Zac: Yeah, a lot of, I’m sure, pretty standard things for people who are used to product development, but A/B testing primarily. A couple years ago we were running, I think, like at least one A/B test of the homepage, like one alternative version of our homepage every week for pretty much the entirety of 2024, 2025. That year we had a ton and tons of homepage tests. And then independent of the actual A/B experiments that we’re doing, we will regularly run diary studies. So recruiting 10 to 20 DuckDuckGo users or other different user types, even non-users, to give them some background on the product. And then really give them a chance to use it for multiples of weeks and have them do journal entries and take notes of the things that they’re running into that they’ve experienced that are emotionally supercharged or maybe had caused some great duress in how they tried to use the product. So anything that’s really notable to that user can come up in a diary study. What we’ve also done in the past is to try to create alternative versions of our homepage from designs or a design system that we can give to users to play around with in real time. So we can actually simulate what an alternative version of the homepage might look like and how it should act and give them a clickable prototype that they can use, get some feedback before we ever invest too heavily in the platform. So it’s been a mix of really just doing a lot of testing pretty much every week since the conception of the company on some run. The homepage was a big focus for us for the last few years to try to nail something that could eloquently show what all of our kind of product offering is between search and now the expansion into Duck AI and of course the apps and email and other things.

Beah: Yeah, so like there’s like, sounds like we’ve been marrying, and I think we do this elsewhere in the company, a mix of like quantitative research where we get to hear how people think, you know, people who are opted into some kind of study with us, like we actually get to understand what they’re thinking, what they’re doing in some detail and then just like quantitatively looking at results in an A/B test.

Zac: Yeah, absolutely. And probably the maybe thing I didn’t say was we were very active on our social media. We point most of our users, at least if you click around our website, a lot of things point to our subreddit, which is reddit.com slash r slash DuckDuckGo. If you have product feedback, if you run into a product issue. So we do try to, and we have lots of people here that every day are checking comments or posts to that subreddit just to see if we released something and it’s causing some undue user harm. Tons of feedback boxes around our SERP, like the search results page within the app. And so we’re constantly trying to encourage that if a user has a negative experience, they’re able to get it pretty frictionlessly by a message to us. So we’re looking at social media, we’re really running really structured user tests, like intentional user tests, A/B testing and just kind of a what they call dogfooding or like internally using the product a lot and trying to be very critical about our understanding of what users really need.

Beah: Yeah, I’ve said it before and I’ll say it again. If you’re writing feedback to us, a human is probably not just reading it, but like taking it to heart and thinking about it. So, um, how about like, tell me what are the privacy considerations? Can we do these experiments in a way that respects user privacy anonymity? And does that throw any monkey wrenches into the testing mix?

Zac: Yes, truly. Yeah, good question. Yes, it has, it continues to, but we continue to have a lot of really intelligent people here that think and care a lot about it. So over the years, we’ve gotten a lot better at developing methodologies that allow us to consistently protect and respect user privacy while getting some of the analytics that we would want from robust A/B testing. So we do this through local storage or through consumer opt-in notices. So for example, if we have a survey that we show you on our SERP, we will have like a pretty robust disclaimer there that says this is anonymous or the responses maybe are going through a third party if it’s something like SurveyMonkey that we use for collection of survey responses. So very upfront to the user and very much building the engineering that is required to do anything that’s an A/B test or more like metric tracking in a privacy respecting way. And we’ve gotten a lot better at this over the years, but it definitely takes a lot of toil and discussion internally with data science and really good product folks like yourself to understand what things you just can’t get from numbers, which is a lot of like how the product feels to people and that’s where we come back to the qualitative research a lot. Getting, you know, FaceTime with actual people to hear what their experience is like and hearing in their words gives us the why to the what that we get from the metrics. So for as much as we can measure something, we don’t really know why it’s changing or what that we were making about a user behavior and how it’s actually playing out. So for those, we do tend to get or try to get face-to-face with users and recruit them either through our own product. If you see a survey and have the option to, you can connect with us at the end if you’d like to share an email address that we can reach out to. And again, those are all done within the in accordance with our privacy policy or the policies of the platforms that we’re providing them. So it’s been a long road, but we think we’re in a pretty good position now to be able to test most of the things that we would ever want to test with a pretty high level of accuracy.

Beah: Nice. You kind of touched on this, but like who’s the we? Like who’s actually doing running these tests, writing these questions.

Zac: Yeah, so when we started the user insights team, as maybe the long answer to this, and we started the user insights team, it was because everybody was doing testing at DuckDuckGo with product folks, engineers, and we were, as I’m sure you know, but for the listeners, like we’re a very validation heavy company and we care a lot about the assumptions that we’re making to make sure that we’re working smart and we’re a small company relative to the other larger competitors out there. And so we have to be a little bit more intentional and I should say, like, careful about how we approach things. And so when we started the insights team, it was really on the basis of everybody at this company doing some level of validation. And we wanted to up level or make sure that there was a baseline level of excellence that went along with that. So the short answer to your question is everybody here does some level of user testing. I think you yourself have probably done more than I have, at least in the recent past. Because it is a part of every conversation that we have. And so we have this formal team that we call user insights that tends to do the higher risk or the more robust or deeper research that takes somebody’s full attention for maybe multiples of weeks. But if you’re a member of the product team or engineering or you’re on the copywriting team, you yourself should have, I hope, the capability and tools to go out and do any validation that you would have to do today. So I can take a guess. There are probably five to ten people I can think are running an active test right now on something like usertesting.com or PickFu. or something that’s a pretty light way to free up a future discussion if you can provide some data ahead of that meeting or wherever that discussion is taking place.

Beah: Yeah. And one of things that I think is cool is the user research team, like part of their, the team members, part of their job is running the research, but part of their job is making other people, like enabling other people to run the research. So even like if I’m running a test, like I can get support or help from somebody on the team to like check it, make sure the work was good, to understand the tools that I need and so forth. Okay, I think that was a great, I bet that’s probably not true everywhere and a good, interesting answer, but before we close, anything we didn’t talk about that you want to talk about, especially anything that’s in the surprise column or might be unusual about how we work at DuckDuckGo.

Zac: I think there’s a lot of stuff that we do and usually at DuckDuckGo I think most of it’s probably for the better and certainly the thing that I can keep reiterating is just how much we all care about users. Obviously the, you know, dedication to privacy that we have comes from a place of wanting to be treated better as people and consumers ourselves but in the way that we approach research I think the heart of it is getting to how people really feel about the product and whether or not it’s a compelling experience. So for all of the metrics that go up or for positive feedback that we’re getting, I think we really strive to know why. And some of the interesting things, I was just trying to rack my brain earlier about some things that have come up that might be worth sharing. And certainly, like, you know, we have a bunch of layers around the company that do either social media listening or user testing. But where I think this really comes out is when we’re shipping a product, we have a final stage in our internal process where we can kind of all see the changes that we’re making. We call them ship reviews, but we tend to dogpile in and see UX changes or copy changes that we’re making and have an opportunity to really look for confounding evidence or a contradiction that might help improve that thing. And so this process that we have actually does involve more people at the end than it does at the beginning sometimes because we’re asking that if you can improve something or if you have positive or if you have any meaningful feedback about something to jump in. So something that I think people would be surprised to hear is that even though teams tend to work on individual projects here, anybody from the company can show up during the process of creation of product and leave their own feedback and have like a real impact on the outcome of what that project is doing or the direction that that product is going, which is I think really interesting.

Beah: Yeah, one thing that we’ve been doing. Yeah, no, go ahead. Go ahead. Zac, please.

Zac: Independent of that, we found a lot of the subsegment. I’m sorry. There’s a little glimmer. Sorry, you’re, yeah, you’re, you got it. You’re, you were frozen. You go. I’ll stop. I’ll start my next table.

Beah: Okay, the listeners are gonna love this part. I was gonna say in ship reviews at that sort of last mile before we ship, one thing that we’ve been doing that I think we’re building maybe more muscle around now that I think’s been really good is like if we have some internal difference of opinion on like the best way, you know, some detail that we’re like, oh, this didn’t make sense to me, but it made sense to me. User research team members or anybody really is like trying to jump in with some, as one of your team members coined it I think, punk rock research — like really quick, like you know three-hour turnaround, like let’s get some data points to help unstick the conversation and and like base this decision as much as possible in empiricism. And like, you know, doing perfect research is very, very hard, but doing like, you know, imperfect research with known caveats can be quite, quite fast and productive and I love that trend that’s happening now.

Zac: Yeah, yeah. And the tools are getting a lot better. Certainly, I think as we see AI develop and get more integrated and things become automated, things that we’re looking at now are things like synthetic user panels, which not to get too technical is just using AI to simulate as best as it can what a group of or a segment of people might say. And so with that, you can imagine doing a lot more testing or even having it automated so that you don’t have to do it and you just wake up to the results from them. But of course you have to make sure that the output of this automation or of the AI is actually reflective of the underlying user base that it’s supposed to be representing. And that’s like a constant challenge for us. It’s the same thing that we’re dealing with here. When we ask or get results from something, you know, we can’t take it as truth. We have to take it even still with a grain of salt despite the validation and continuously validate. After we launch something, you shouldn’t be surprised to see that even if it was really positive, we might come back and try to change it again a couple months later if we have a really good idea.

Beah: Yeah, or a couple of weeks later. When you’re talking about the synthetic user testing via AI, like user testing script, like what would you do on this screen, AI — like that’s a great question. Excellent point. I hadn’t thought of that. What I would do on the screen is…

Zac: Yeah. It’s saying you’re not trying to do something on the screen, you’re trying to do something on the internet. That would be the phrasing.

Beah: Alright, awesome. Well thank you, Zac. I think we covered a lot of ground. Appreciate it. Pleasure as always talking to you. See you around the DuckTales hood.

Zac: Thank you. See you at the office. Bye.

Beah: Bye.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com