If you believe all you read about AI, it’s easy to come away feeling that it’s the cure for all ills. But in many ways, it’s a solution in search of a problem. In this episode with Pendo CEO Todd Olson and CPO Trisha Price, we move past the hype to remind ourselves, as Todd says, “AI isn’t just magical pixie dust you can sprinkle onto your products and get benefit. You still have to go back to the core product management fundamentals, which means your product has to solve real pain.”

Todd Olson & Trisha Price joined co-hosts Sean Flaherty and Kyle Psaty to kick off a 4-epiosde series of conversations with keynotes and presenters recoded live from Pendomonium 2023: A Festival of Product.

So, if AI isn’t the panacea we’ve made it out to be, how are product leaders supposed to think about it in this “age of intelligence”?

“There’s two ways to think about AI,” responds Trisha. “One is, how do I use it in my day-to-day job to make myself smarter, more efficient, and achieve my goals? And the second, how do I embed it into my product to help my users and solve real pain?”

When we connect these two components, AI tools start to feed off one another to create their own momentum.

“The more I can use AI to make my team more efficient, the more we can spend time figuring out how best to use it,” Trisha adds.

What if AI eliminated all the busy work that fills our schedules, Todd asks. Or synthesizes thousands of data points and distills them into something more actionable?

“That’s going to save time so that product leaders can now do more of the right things – e.g., negotiating with engineering, reaching out to customers, being more business- an outcome-focused.”

With AI still in its infancy, considering its potential can be an interesting exercise – exciting for some, daunting for others. Fear not, Todd says.

“Look, there’s plenty for product managers to do. AI can’t set the strategy or vision for a business, so product still has a role. But maybe AI can make it easier to go through reams of data to help inform that strategy. That’s where I think it’s really, really powerful. So hopefully we’ll be setting better strategies that won’t replace what people are doing, rather just make them better at their function.”

Catch the entire conversation with Todd Olson & Trisha Price –

• Does AI actually deliberate over our prompts and questions?
• The “dark age of AI” at Pendo, and how a shift in mindset cleared the way
• TLMs, purpose-built for a specific function, and their applicability in the future of business
• Why it’s important to “protect your data”

Subscribe to Product Momentum to be notified when the next episode drops!

Podcast: Play in new window | Download

Paul [00:00:19] Hello and welcome to Product Momentum, where we hope to entertain, educate, and celebrate the amazing product people who are helping to shape our community’s way ahead. My name is Paul Gebel and I’m the Director of Product Innovation at ITX. Along with my co-host, Sean Flaherty, and our amazing production team and occasional guest host, we record and release a conversation with a product thought leader, writer, speaker, or maker who has something to share with the community every two weeks.

Sean [00:00:43] Kyle, how are you doing today, man?

Kyle [00:00:44] Dude, I’m psyched. Today we’re recording live from Pendomonium. Pendo invited us to come out and record and this is going to be really cool.

Sean [00:00:51] I am excited about this podcast episode.

Kyle: Yeah, I mean, we got Todd Olson, the CEO from Pendo, and Trisha Price, who’s the CPO. So these guys are going to tell us all about how to build a product for product people that matters. That’s right. And they’re going to teach us about first principles in product leadership and how A.I. is changing the world, it’s all the buzz, but how we really have to get back to solving real problems for real users. It doesn’t matter what the label is that you put on the technology, you’ve got to solve problems for people.

Kyle [00:01:19] Absolutely. And I think they’re going to cut through the BS for us a little bit and we all could use a little bit more of that when it comes to A.I. and LLMs.

Sean [00:01:27] Yeah, let’s get after it.

Kyle [00:01:28] All right, let’s get after it.

Kyle [00:01:31] All right. Welcome to the Product Momentum Podcast. Here we are with Todd Olson and Trisha Price. Todd is the CEO of Pendo. He’s a co-founder and teamed up with fellow product leaders and technologists from RedHat, Cisco, and Google to launch Pendo in October of 2013. So ten years this month, which is really exciting. Todd’s also an author. He took his passion for helping digital teams build great products to help him write the book, The Product-Led Organization: Drive Growth by Putting Product at the Center of Your Customer Experience.

Kyle [00:02:02] We’re also really excited to be here with Trisha Price. She’s the Chief Product Officer at Pendo, and she oversees Pendo’s product management, design, and product operations teams. She also sets the product strategy for the company. A lot going on in the product strategy. We’re excited to talk about that stuff today.

Trisha [00:02:18] Thank you.

Kyle [00:02:19] And we’re psyched to be here. So thanks for having us.

Todd [00:02:21] Welcome. It’s great to have you.

Kyle [00:02:23] Cool. We’re glad to be in front of the pink.

Sean [00:02:26] Or “pank” as we learned this morning.

Kyle [00:02:28] Yeah, pank. You got it.

Todd [00:02:29] Go with what works, so.

Trisha [00:02:32] We like our pink.

Kyle [00:02:33] Cool. Your keynote yesterday talked a little bit about AI and the future of AI and how to really make it work for product organizations. You talked about a four-part framework, which we were really excited about. Maybe you could talk a little bit about how you’re thinking about AI and implementing it into the platform this week.

Todd [00:02:51] Yeah, well, first off, I mean, if you go back to the keynote, one of the key points around AI is that it’s not just magical pixie dust you can sprinkle into your products and get benefit.  You still have to go back to the core product management fundamentals and it has to solve real pain. And you hear that as a theme actually throughout the day, even like as we closed with Fraser Kelton, former Head of Product for OpenAI. He talked about solving a real pain. So that was one key point. And if you look at some of the announcements we made yesterday that were some of our, I’d say our early work in AI, it was solving real pain. You know, for example, a lot of customers use our product to perform their NPS surveys. And we noticed through data that people were exporting data from Pendo, putting into a spreadsheet, and then manually assigning themes that they could then roll up in service to their leadership teams. That’s a lot of work and people are doing it every single quarter for years.

Todd [00:03:46] And now, with the availability of LLMs and fantastic APIs, now there’s, like, single buttons to summarize all the NPS feedback. There’s the ability to auto-assign themes and then train the models to be smarter and smarter. So that was one of the innovations we released yesterday that I think, one, it solves a real pain and it’s automation that saves people time.

Sean [00:04:10] Yeah, I heard you say that no one wants to download a spreadsheet and manipulate data. Like, we’ve been doing that for years, and that’s a common problem.

Todd [00:04:18] Yes.

Sean [00:04:18] You know, one of the things I think is beautiful about the potential behind AI is this ability to solve the mundane tasks.

Todd [00:04:25] Correct.

Sean [00:04:26] So Kyle brought up the four-part strategy to solving problems with AI. I’d love for the audience to hear more about that.

Todd [00:04:33] Yeah. So look, one of the four parts, of course, is start with real pain. So we talked a little bit about that and there are other good examples about the mundane. And you heard from Red-Hat on stage yesterday during my presentation where they talked about the fact that people are manually creating guided tours, step-by-step walkthroughs, even though they already have existing content manual documentation. So the ability to, like, basically auto-generate a ten-step guide or a five-step guide is a huge value. So starting with real pain is one. We talk a lot about finding this key or golden use case that I introduced yesterday, and it’s really use cases that are focused on creating long-term differentiation for your business. I think, while the LLMs are useful and it solves a lot of problems, they’re available APIs.

Trisha [00:05:25] What’s your magic?

Todd [00:05:27] So we talked a lot about, how do we find these key or golden use cases? Then we also talked a lot about this concept of focusing on, what is your first-party data? So nearly every business has some data that they and only they have. So the question is, how can we mine that data and use it for generative things. Like, we introduced yesterday this concept of using product data to generate growth campaigns as an example, which is something that’s essentially a not copy-able feature if you’re leveraging data to train a model and do generative things. So those are kind of, I think, the key things that we talk about.

Sean [00:06:06] Data that only you have.

Trisha [00:06:08] Data that only we have, right. So now we can help our customers achieve the goals that they’re accountable for, whether that’s product-led growth, like Todd was talking about, or just improving retention, we can tell them, here’s a campaign, here are the users of your product and which features they need to adopt in order to achieve retention or product-led growth.

Sean [00:06:31] To get the most value out of the product. I love it.

Trisha [00:06:33] Exactly, because people focus on usage quite a bit. But is it valuable usage? Right, just because everybody’s clicking on a page or using your product doesn’t mean they’re getting the value out. But you can start to see, with our proprietary models, and that’s using our first-party data like Todd was talking about, “Hey, if these users use these features, this is how you’re going to achieve your goals.” And so that’s something we’re focused on for Pendo.

Sean [00:07:02] So a question for you, Trisha, as a product person. I think it’s an exciting time to be a product person because, you know, a few years ago the product industry didn’t even exist and it’s just transformed so fast as a role, as a career for people.

Trisha [00:07:16] Yeah.

Sean [00:07:17] What do you think is the most exciting thing in the AI space for product leaders?

Trisha [00:07:22] Well, I think, you know, Todd and I talked about this yesterday, which is there’s two ways to think about AI as a product leader. One is, how do I use it in my day-to-day job to make myself smarter, more efficient, right, and achieve my goals? The second is, how do I embed it into my product to help my users and to solve real pain, right? And so I think, you know, for me, that’s what’s exciting is, actually, they’re related, because the more I can use A.I. to make my team more efficient and have more intelligence, the more we can spend time on, how do we use A.I.? And just in general, how do we create products that delight our users and help them solve real pain?

Kyle [00:08:05] Wow. Yeah. And we get closer to it by using it. How do you think the role of the product person is going to change over the next 3 to 5 years as we kind of hit this big wave? Like, how is their role and their collaboration, maybe even with the other departments, marketing, et cetera, that need to drive, you know, feature usage and help make the product really sing? How does that role change?

Trisha [00:08:27] Yeah, I think that it is changing, has been changing, and will continue to become more of a business leader not only an executor. And I think that has been happening. I mean, you see more Chief Product Officers in the C-suite. I think in the SaaS world, that’s probably been happening for quite some time. But when you look at traditional businesses, they’re all moving from the physical to the digital too, and they’re moving from project-led to product-led, like Todd talks about in his book, right?

Kyle [00:08:58] Right.

Trisha [00:08:58] And I think, I mean that’s where we see probably even the bigger change as it relates to product.

Todd [00:09:04] Yeah, and let’s be honest, product does so much today. I used to produce a slide for one of the talks I gave thata just looked at a product manager’s schedule if you just use a scrum ceremony of, you know, all of the various planning meetings and retros and demos and grooming and… So just look at that on a two-week schedule and then you try to figure out, when do they have time to speak with customers? And oh, when do they have time to speak with internal stakeholders? There’s a lot already.

Todd [00:09:32] So AI if, if it can take any of the busywork, any of the, essentially, synthesizing of data points and try to distill it down into something more actionable, that’s just going to save them time so they’re going to do more of the right things, which is negotiating with engineering, talking with customers, being more business- an outcome-focused. So like, look, there’s plenty for product managers to do. I joked yesterday in the keynote that, you know, AI can’t set the strategy or vision for a business, so product still has a role. However, maybe AI can make it easier to, you know, go through reams and reams of data to help inform that strategy. That’s where I think it’s really, really powerful. So hopefully we’ll be setting better strategies that won’t be replacing what people are doing, rather just make them better at their function.

Sean [00:10:17] Yeah, you talk about first principles, and I think for product, one of the first principles is to spend as much time really understanding your customers as possible, like getting in front of the customers, doing more research, spending more time in the field, looking at what customers are actually doing. And I think it has so much potential for freeing up that time if we use it properly.

Trisha [00:10:38] Yeah. I think technology can help us do that at scale as well, right? Yesterday we launched our Session Replay product and, you know, that doesn’t replace in-person time with customers where you really get empathy and have an understanding of what they need to do. But it certainly helps you look at how they’re interacting with your product and where they might get stuck and where they’re frustrated. I think you can also look at things like, you know, Todd talked about NPS scores. You can use surveys and polls. And I think our role as product managers is a combination of nothing can replace, you know, human conversation about what are your priorities and challenges with your customers. But getting data at scale is also critically important.

Sean [00:11:20] You know, Kenneth O. Stanley wrote a book. Kyle and I we’re talking about this last night, Why Greatness Cannot Be Planned. He’s a researcher out of some university in Texas but he’s a brilliant guy. And he says creativity cannot be planned. Like, somebody needs to see it. Someone needs to be there to observe it. There’s no way AI can actually observe, “Oh, that’s something that’s creative,” at least not yet. So what tools like this help us do is weave all that stuff together. Like, surveys, qualitative, you know, they have limited usefulness. They’re useful because they give you something you wouldn’t have otherwise had. But observation and having people that actually care about the problem and look at hundreds of users and how they’re using the product, that’s where the real innovations are going to come from.

Todd [00:12:00] Oh, absolutely. Absolutely. And I shared this story yesterday as well, but I was speaking with a financial analyst and she was saying that, you know, “Yeah, now, in every earnings report and Q&A, there’s a summary now that’s auto-generated from AI.” And I said, “Well, do you read it?” And she’s like, “No, I don’t even read it, because my job is to find the one or two insights that no one else sees.” And then she makes investment decisions that yield millions and millions of dollars. And she’s not going to trust some system and she’s not going to use the same auto-generated summary that everyone else is using. There’s no differentiation in that.

Todd [00:12:33] So to your point around creativity, yes, it does take a human to essentially think about and deliberate over the data to make decisions. I shared in our executive briefing on Tuesday four kind of challenges with AI, I guess, and one of them is this concept of deliberation challenge. Which people think the A.I. is deliberating over your question. It’s not. It’s like really good autocomplete. It is, “Based on all this training data, what is the next best answer to whatever this individual is prompting me?” It’s not thinking about it. And that’s dangerous because people think that it’s thinking and it’s not. So be careful what you ask. It’s going to be very good autocomplete and it’s very good at explaining things.

Todd [00:13:15] Yesterday, Fraser on stage talked about how he sent it text from a medical diagnosis and it helped explain it. It’s very good at things like that. It doesn’t have to think. Someone else thought, it’s just explaining in better words what something means.

Kyle [00:13:30] Yeah, beautiful. I want to go back to something that Trisha was talking about. Actually, it was Todd, like having your own proprietary data and then thinking about, how do you create some proprietary value for the customer? And you guys are talking a lot about the user’s pain. Can you take us a little bit deeper into your own process for, you guys are chuckling, so you know where I’m going, like, all right, we have this data, we know what we know and we know what the pain points are, but there’s this massive gap, right, between those things. So how do you guys go through the ideation process to find that critical insight, right, as humans?

Trisha [00:14:02] We’re laughing because we did it wrong a couple of times before I feel like we got it right, right?

Todd [00:14:06] Yeah.

Todd [00:14:08] Why don’t you start?

Trisha [00:14:08] I’ll start with what we did wrong and maybe you can go with what we did right. I call it like the dark age of AI, which was when we first started, we really went what I call inside-out thinking, which is, we had our incredible machine learning team looking at the data and trying to find patterns and come up with ideas based on that. And they actually did a pretty incredible job. But then we got stuck because we couldn’t figure out what was the user interface to wrap around that, to actually deliver it to our customers and drive value. We knew we had this model that could say, “Hey, if you use these features, you’ll improve retention.” But we couldn’t figure out like, what does that solve, what’s the experience, the workflow? And we just kept struggling. And then I’ll let Todd talk about how we just stopped and we stopped thinking about the model and the data and sort of changed our outside-in thinking.

Todd [00:15:03] Yeah, this was a stressful thing because we had all these smart people building all these cool things, and we had what we thought was a really useful application of our data, which was, you know, people that use this feature aare ten times, 30 times, 50 times more likely to retain. You’re like, “Wow, everyone’s going to love this.” But you put it in front of users and they’re like, “What do I do with this?”

Todd [00:15:21] So we step back and say, “Okay, what are the jobs to be done that people are hiring our product to do? Where are the pain points of using our product today and what are people doing manually?” Almost similar to the whole, we’re downloading CSVs and doing work. We noticed that a lot of our customers, their growth teams were spending a lot of time creating manual growth campaigns and our growth team was actually doing it as well. So we have a full PLG team within Pendo that manages both our free and some of our starter plans. And they were doing all this work trying to dig through all this Pendo data and create these campaigns to drive conversion rates, to drive various aspects of that funnel. And we realized that, “Hey, this technology can actually be applied to that problem.” So we shifted our mindset and said, “Okay, we’re going to focus on a solution to help growth teams drive better campaigns to help provide these outcomes, one of which will be increased retention.” So instead of starting with, “We got this model for retention,” we started with, “What do people actually want?” and we started backing into how the technology can really fuel that.

Trisha [00:16:27] And then it got pretty easy.

Todd [00:16:29] A lot easier.

Trisha [00:16:29] To get the workflows and get the designs and get the visualization because we were focusing on the problem we were solving for the customer, not the pattern in the data.

Todd [00:16:39] Yeah, it’s interesting here because I was clearly inspired yesterday by Fraser’s story about ChatGPT, and I think those are going to be the outliers. So that was an interesting case where they spent lots of product time on the core model and then it sounded like, was it two sprints to release ChatGPT? So that’s a very simple interface and I don’t think many companies are going to have that style of product development. Because for us, it was the inverse, it was the model was easy, no, not easy. So I’m sorry to anyone who’s a scientist.

Trisha [00:17:12] Felt easy for us.

Sean [00:17:12] Be careful with your language.

Todd [00:17:17] What I’m saying is we had the model, but there’s now a lot of work going into the UI. That’s not going to be two sprints for us. It’s going to be probably quarters of work to actually wrap around what was needed to really showcase the power of the AI.

Sean [00:17:33] Yeah. It reminds me of the famous Steve Jobs quote that you can’t start with the technology and sell the technology, what is kind of what ChatGPT did. You have to start with the user and solve their problem if you want to build something that’s going to actually change the world.

Todd [00:17:47] Exactly. Yeah, and like I said, I think ChatGPT and things like it are unique in that these foundational models are so hard to create that showcasing it with a chat-like interface, yeah, it is the product, but for many of us, so you’ll hear about TLM, tiny language models or tiny models, I think tiny models that are purpose-built to a function I think, have a lot of applicability in business today. Like, look, these large language models are trained in the entire corpus of human knowledge.

Kyle [00:18:16] Right.

Todd [00:18:17] Most people don’t need that to solve specific practical use cases. So a tiny model that’s based on your first-party data is probably enough to drive a lot of value for your customers.

Sean [00:18:28] All right. Well, we’re coming to a close here. I want to just make sure I capture a couple of key learning nuggets for the audience. And here’s what I’ve captured. You can reflect on these. The first one is that ChatGPT isn’t the magic bullet, or any AI for that matter. Like, we have to really take a practical approach, which your philosophy, which I fully subscribe to, is first principles. Like, let’s use it to solve problems and understand better our customers. So if you use it properly, you’re going to need more time to be in front of your customers. Which means you have to find the job to be done. Like what is the job to be done? Which I love Clayton Christiansen’s work. We’ve had Tony Ulwick week on the podcast, so you should listen to that when you get a chance. To free up time so we can build better products, so to speak.

Sean [00:19:09] The second is taking a practical approach, and I loved in your keynote, your four-stage approach to using AI in general.

Todd [00:19:15] Yep.

Sean [00:19:16] And then this is the third key point. You have unique data in your business and it’s our job as product leaders to think about that data and what makes it unique that gives us a competitive advantage if we use it properly. So that was a key nugget for me.

Todd [00:19:28] Yeah. I loved how Fraser, even at the end of the day, really came back to this point of your own data. He said, “Protect your data.”

Sean [00:19:35] Yeah, defend it.

Trisha [00:19:36] Yeah.

Todd [00:19:36] That’s something that we didn’t say in the morning, but I think was really wise. It’s like, be careful where you send it. For many of us, it’s our special sauce of differentiation. So protect It was a really good point, I thought.

Sean [00:19:49] Yeah.

Trisha [00:19:49] He even said, “There’s no way to even imagine the ways you’re going to use it because we’re just getting started with AI. And so that means you have to protect it even more because you can’t even imagine all the uses of it.”

Sean [00:20:03] No, not yet. The fourth idea that I got here is that creativity still requires people. AI isn’t going to solve that problem, but it should help us be more creative. Yeah. And then the last one that I got from you guys is this concept of tiny models. That really struck a chord with me.

Kyle [00:20:17] Yeah, super cool.

Sean [00:20:17] I thought of like, the smallest possible model that adds value to the business is going to be the future, like, figuring out how to get that incremental…

Kyle [00:20:25] It’s the new way to leverage your data, really.

Sean [00:20:27] I loved that. Cool. So a couple last questions for you that we ask all our guests. How do you define innovation? We’ll start with Trisha.

Trisha [00:20:35] To me, innovation is solving a problem in a way that no one else has done it before. You know, being able to apply the creativity that you talked about a minute ago. You know, it’s not coming up with something that doesn’t exist. It’s solving something real, but in an innovative way, in a new way.

Sean [00:20:57] All right. Cool.

Todd [00:20:58] That’s a really good answer so it’s hard to go after. I was going to simply say it’s doing something that no one else has done. That to me is innovative.

Kyle [00:21:05] Simple. Awesome. Really excited to have you guys on the show, Trisha and Todd. Thank you so much for having us at your conference today.

Todd [00:21:13] It’s great to have you.

Kyle [00:21:13] And we’ll keep in touch. One of my next action items is, ask Todd to text Fraser so he can come on our podcast because he obviously made a big impression on you guys. But thanks again for having us. This was awesome.

Trisha [00:21:25] Thank you for coming.

Paul [00:21:28] Well, that’s it for today. In line with our goals of transparency and listening, we really want to hear from you. Sean and I are committed to reading every piece of feedback that we get. So please leave a comment or a rating wherever you’re listening to this podcast. Not only does it help us continue to improve, but it also helps the show climb up the rankings so that we can help other listeners move, touch, and inspire the world just like you’re doing. Thanks, everyone. We’ll see you next episode.