Experiencing Data w/ Brian T. O’Neill

Today I am sharing some highlights for 2023 from the podcast, and also letting you all know I’ll be taking a break from the podcast for the rest of December, but I’ll be back with a new episode on January 9th, 2024. I’ve also got two links to share with you—details inside!

 

Transcript

Greetings everyone - I’m taking a little break from Experiencing Data over December of 2023, but I’ll be back in January with more interviews and insights on leveraging UX design and product management to create indispensable data products, machine learning apps, and decision support tools. 

Experiencing Data turned this year five years old back in November, with over 130 episodes to date! I still can’t believe it’s been going that long and how far we’ve come. 

Some highlights for me in 2023 included launching the Data Product Leadership Community, finding out that the show is now in the top 2% of all podcasts worldwide according to ListenNotes, and most of all, hearing from you that the podcast, and my writing, and the guests that  I have brought on are having an impact on your work, your careers, and hopefully the lives of your customers, users, and stakeholders as well! 

So, for now, I’ve got just two links for you:

If you’re wondering how to either:

• support the show yourself with a really fast review on Apple Podcasts,
• to record a quick audio question for me to answer on the show,
•  or if you want to join my free Insights mailing lists where I share my bi-weekly ideas and thoughts and 1-page episode summaries of all the show drops that I put out here on Experiencing Data.

…just head over to designingforanalytics.com/podcast and you’ll get links to all those things there.

And secondly, if you need help increasing customer adoption, delight, the business value, or the usability of your analytics and machine learning applications in 2024, I invite you to set up a free discovery call with me 1 on 1. 

You bring the questions, I’ll bring my ears, and by the end of the call, I’ll give you my best advice on how to move forward with your situation – whether it’s working with me or not. To schedule one of those free discovery calls, visit designingforanalytics.com/go

And finally, there will be some news coming out next year with the show, as well as my business, so I hope you’ll hop on the mailing list and stay tuned, that’s probably the best place to do that. And if you celebrate holidays in December and January, I hope they’re safe, enjoyable, and rejuvenating. Until 2024, stay tuned right here - and in the words of the great Arnold Schwarzenegger, I’ll be back.

In this conversation with Klara Lindner, Service Designer at diconium data, we explore how behavioral science and UX can be used to increase adoption of data products. Klara describes how she went from having a highly technical career as an electrical engineer and being the founder of a solar startup to her current role in service design for data products. Klara shares powerful insights into the value of user research and human-centered design, including one which stopped me in my tracks during this episode: how the people making data products and evangelizing data-driven decision making aren’t actually following their own advice when it comes to designing their data products. Klara and I also explore some easy user research techniques that data professionals can use, and discuss who should ultimately be responsible for user adoption of data products. Lastly, Klara gives us a peek at her upcoming December 19th, 2023 webinar with the The Data Product Leadership Community (DPLC) where she will be going deeper on two frameworks from psychology and behavioral science that teams can use to increase adoption of data products. Klara is also a founding member of the DPLC and was one of—if not the very first—design/UX professionals to join.

 

Highlights/ Skip to:

• I introduce Klara, and she explains the role of Service Design to our audience (00:49)
• Klara explains how she realized she’s been doing design work longer than she thought by reflecting on the company she founded, Mobisol (02:09)
• How Klara balances the desire to design great dashboards with the mission of helping end users (06:15)
• Klara describes the psychology behind user research and her upcoming talk on December 19th at The Data Product Leadership Community (08:32)
• What data product teams can do as a starting point to begin implementing user research principles (10:52) 
• Klara gives a powerful example of the type of insight and value even basic user research can provide (12:49)
• Klara and I discuss a key revelation when it comes to designing data products for users, which is the irony that even developers use intuition as well as quantitative data when building (16:43)
• What adjustments Klara had to make in her thinking when moving from a highly technical background to doing human-centered design (21:08)
• Klara describes the two frameworks for driving adoption that she’ll be sharing in her talk at the DPLC on December 19th (24:23)
• An example of how understanding and addressing adoption blockers is important for product and design teams (30:44)
• How Klara has seen her teams adopt a new way of thinking about product & service design (32:55)
• Klara gives her take on the Jobs to be Done framework, which she will also be sharing in her talk at the DPLC on December 19th (35:26)
• Klara’s advice to teams that are looking to build products around generative AI (39:28)
• Where listeners can connect with Klara to learn more (41:37)

 

Links

• diconium data: http://www.diconium.com/
• LinkedIn: https://www.linkedin.com/in/klaralindner/
• Personal Website: https://magic-investigations.com/
• Hear Klara speak on Dec 19, 2023 at 10am ET here: https://designingforanalytics.com/community/

This week I’m covering Part 1 of the 15 Ways to Increase User Adoption of Data Products, which is based on an article I wrote for subscribers of my mailing list. Throughout this episode, I describe why focusing on empathy, outcomes, and user experience leads to not only better data products, but also better business outcomes. The focus of this episode is to show you that it’s completely possible to take a human-centered approach to data product development without mandating behavioral changes, and to show how this approach benefits not just end users, but also the businesses and employees creating these data products. 

 

Highlights/ Skip to:

• Design behavior change into the data product. (05:34)
• Establish a weekly habit of exposing technical and non-technical members of the data team directly to end users of solutions - no gatekeepers allowed. (08:12)
• Change funding models to fund problems, not specific solutions, so that your data product teams are invested in solving real problems. (13:30)
• Hold teams accountable for writing down and agreeing to the intended benefits and outcomes for both users and business stakeholders. Reject projects that have vague outcomes defined. (16:49)
• Approach the creation of data products as “user experiences” instead of a “thing” that is being built that has different quality attributes. (20:16)
• If the team is tasked with being “innovative,” leaders need to understand the innoficiency problem, shortened iterations, and the importance of generating a volume of ideas (bad and good) before committing to a final direction. (23:08)
• Co-design solutions with [not for!] end users in low, throw-away fidelity, refining success criteria for usability and utility as the solution evolves. Embrace the idea that research/design/build/test is not a linear process. (28:13)
• Test (validate) solutions with users early, before committing to releasing them, but with a pre-commitment to react to the insights you get back from the test. (31:50)

Links:

• 15 Ways to Increase Adoption of Data Products: https://designingforanalytics.com/resources/15-ways-to-increase-adoption-of-data-products-using-techniques-from-ux-design-product-management-and-beyond/
• Company website: https://designingforanalytics.com
• Episode 54: https://designingforanalytics.com/resources/episodes/054-jared-spool-on-designing-innovative-ml-ai-and-analytics-user-experiences/
• Episode 106: https://designingforanalytics.com/resources/episodes/106-ideaflow-applying-the-practice-of-design-and-innovation-to-internal-data-products-w-jeremy-utley/
• Ideaflow: https://www.amazon.com/Ideaflow-Only-Business-Metric-Matters/dp/0593420586/
• Podcast website: https://designingforanalytics.com/podcast

Today I’m joined by Nick Zervoudis, Data Product Manager at CKDelta. As we dive into his career and background, Nick shares insights into his approach when it comes to developing both internal and external data products. Nick explains why he feels that a software engineering approach is the best way to develop a product that could have multiple applications, as well as the unique way his team is structured to best handle the needs of both internal and external customers. He also talks about the UX design course he took, how that affected his data product work and research with users, and his thoughts on dashboard design. We discuss common themes he’s observed when data product teams get it wrong, and how he manages feelings of imposter syndrome in his career as a DPM. 

Highlights/ Skip to:

• I introduce Nick, who is a Data Product Manager at CKDelta (00:35)
• Nick’s mindset around data products and how his early career in consulting shaped his approach (01:30)
• How Nick defines a data product and why he focuses more on the process rather than the end product (03:59)
• The types of data products that Nick has helped design and his work on both internal and external projects at CKDelta (07:57)
• The similarities and differences of working with internal versus external stakeholders (12:37)
• Nick dives into the details of the data products he has built and how they feed into complex use cases (14:21)
• The role that Nick plays in the Delta Power SaaS application and how the CKDelta team is structured around that product (17:14)
• Where Nick sees data products going wrong and how he’s found value in filling those gaps (23:30)
• Nick’s view on how a digital-first mindset affects the scalability of data products (26:15)
• Why Nick is often heavily involved in the design element of data product development and the course he took that helped shape his design work (28:55)
• The imposter syndrome that Nick has experienced when implementing this new strategy to data product design (36:51)
• Why Nick feels that figuring things out yourself is an inherent part of the DPM role (44:53)
• Nick shares the origins and information on the London Data Product Management meetup (46:08)

Quotes from Today’s Episode

• “What I’m always trying to do is see, how can we best balance the customer’s need to get exactly the data point or insight that they’re after to the business need. ... There’s that constant tug of war between customization and standardization that I have the joy of adjudicating. I think it’s quite fun.” — Nick Zervoudis (16:40)
  
  
• “I’ve had times where I was hired, told, 'You’re going to be the product manager for this data product that we have,' as if it’s already, to some extent built and maybe the challenge is scaling it or bringing it to more customers or improving it, and then within a couple of weeks of starting to peek under the hood, realizing that this thing that is being branded a product is actually a bunch of projects hiding under a trench coat.” — Nick Zervoudis (24:04)
  
  
• “If I just speak to five users because they’re the users, they’ll give me the insight I need. […] Even when you have a massive product with a huge user base, people face the same issues.” — Nick Zervoudis (33:49)
  
  
• “For me, it’s more about making sure that you’re bringing that more software engineering way of building things, but also, before you do that, knowing that your users' needs are going to [be varied]. So, it’s a combination of both, are we building the right thing—in other words, a product that’s flexible enough to meet the different needs of different users—but also, are we building it in the right way?” – Nick Zervoudis (27:51)
  
  
• “It’s not to say I’m the only person thinking about [UX design], but very often, I’m the one driving it.” – Nick Zervoudis (30:55)
  
  
• “You’re never going to be as good at the thing your colleague does because their job almost certainly is to be a specialist: they’re an architect, they’re a designer, they’re a developer, they’re a salesperson, whereas your job [as a DPM] is to just understand it enough that you can then pass information across other people.” – Nick Zervoudis (41:12)
  
  
• “Every time I feel like an imposter, good. I need to embrace that, because I need to be working with people that understand something better than me. If I’m not, then maybe something’s gone wrong there. That’s how I’ve actually embraced impostor syndrome.” – Nick Zervoudis (41:35)

Links

• CKDelta: https://www.ckdelta.ie
• LinkedIn: https://www.linkedin.com/in/nzervoudis/

Today I’m joined by Marnix van de Stolpe, Product Owner at Coolblue in the area of data science. Throughout our conversation, Marnix shares the story of how he joined a data science team that was developing a solution that was too focused on the delivery of a data-science metric that was not on track to solve a clear customer problem. We discuss how Marnix came to the difficult decision to throw out 18 months of data science work, what it was like to switch to a human-centered, product approach, and the challenges that came with it. Marnix shares the impact this decision had on his team and the stakeholders involved, as well as the impact on his personal career and the advice he would give to others who find themselves in the same position. Marnix is also a Founding Member of the Data Product Leadership Community and will be going much more into the details and his experience live on Zoom on November 16 @ 2pm ET for members.

 

Highlights/ Skip to:

• I introduce Marnix, Product Owner at Coolblue and one of the original members of the Data Product Leadership Community (00:35)
• Marnix describes what Coolblue does and his role there (01:20)
• Why and how Marnix decided to throw away 18 months of machine learning work (02:51)
• How Marnix determined that the KPI (metric) being created wasn’t enough to deliver a valuable product (07:56)
• Marnix describes the conversation with his data science team on mapping the solution back to the desired outcome (11:57)
• What the culture is like at Coolblue now when developing data products (17:17)
• Marnix’s advice for data product managers who are coming into an environment where existing work is not tied to a desired outcome (18:43)
• Marnix and I discuss why data literacy is not the solution to making more impactful data products (21:00)
• The impact that Marnix’s human-centered approach to data product development has had on the stakeholders at Coolblue (24:54)
• Marnix shares the ultimate outcome of the product his team was developing to measure product returns (31:05)
• How you can get in touch with Marnix (33:45)
  
  

Links

• Coolblue: https://www.coolblue.nl
• LinkedIn: https://www.linkedin.com/in/marnixvdstolpe/

Today I’m joined by Vishal Singh, Head of Data Products at Starburst and co-author of the newly published e-book, Data Products for Dummies. Throughout our conversation, Vishal explains how the variations in definitions for a data product actually led to the creation of the e-book, and we discuss the differences between our two definitions. Vishal gives a detailed description of how he believes Data Product Managers should be conducting their discovery and gathering feedback from end users, and how his team evaluates whether their data products are truly successful and user-friendly.

 

Highlights/ Skip to:

• I introduce Vishal, the Head of Data Products at Starburst and contributor of the e-book Data Products for Dummies (00:37)
• Vishal describes how his customers at Starburst all had a common problem, but differing definitions of a data product, which led to the creation of his e-book (01:15)
• Vishal shares his one-sentence definition of a data product (02:50)
• How Vishal’s definition of a data product differs from mine, and we both expand on the possibilities between the two (05:33)
• The tactics Vishal uses to useful feedback to ensure the data products he develops are valuable for end users (07:48)
• Why Vishal finds it difficult to get one on one feedback from users during the iteration phase of data product development (11:07)
• The danger of sunk cost bias in the iteration phase of data product development (13:10)
• Vishal describes how he views the role of a DPM when it comes to doing effective initial discovery (15:27)
• How Vishal structures his teams and their interactions with each other and their end users (21:34)
• Vishal’s thoughts on how design affects both data scientists and end users (24:16)
• How DPMs at Starburst evaluate if the data product design is user-friendly (28:45)
• Vishal’s views on where Designers are valuable in the data product development process (35:00)
• Vishal and I discuss the importance of ensuring your products truly solve your user’s problems (44:44)
• Where you can learn more about Vishal’s upcoming events and the e-book, Data Products for Dummies (49:48)

Links

• Starburst: https://www.starburst.io/
• Data Products for Dummies: https://www.starburst.io/info/data-products-for-dummies/
• “How to Measure the Impact of Data Products with Doug Hubbard”: https://designingforanalytics.com/resources/episodes/080-how-to-measure-the-impact-of-data-productsand-anything-else-with-forecasting-and-measurement-expert-doug-hubbard/
• Trino Summit: https://www.starburst.io/info/trinosummit2023/
• Galaxy Platform: https://www.starburst.io/platform/starburst-galaxy/
• Datanova Summit: https://www.starburst.io/datanova/
• LinkedIn: https://www.linkedin.com/in/singhsvishal/
• Twitter: https://twitter.com/vishal_singh

Today I’m joined by Jonathan Cairns-Terry, who is the Head of Insight Products at the Care Quality Commission. The Care Quality Commission is the the regulator for England for health and social care, and Jonathan recently joined their data team and is working to transform their approach to be more product-led and user-centric. Throughout our conversation, Jonathan shares valuable insights into what the first year of that type of shift looks like, and why it’s important to focus on outcomes, and how he measures progress. Jonathan and I explore the signals that told Jonathan it’s time for his team to invest in a designer, the benefits he’s gotten from UX research on his team, and the recent successes that Jonathan’s team is seeing as a result of implementing this approach. Jonathan is also a Founding Member of the Data Product Leadership Community and we discuss his upcoming webinar for the group on Oct 12, 2023.

 

Highlights/ Skip to:

• I introduce Jonathan, who is the Head of Insight Products at the Care Quality Commission in the UK (00:37)
• How Jonathan went from being a “maths person” to being a “product person” (01:02)
• Who uses the data products that Jonthan makes at the Care Quality Commission (02:44)
• Jonathan describes the recent transition towards a product focus (03:45)
• How Jonathan expresses and measures the benefit and purpose of a product-led orientation, and how the team has embraced the transformation (07:08)
• The nuance between evaluating outcomes and measuring outputs in a product-led approach, and how UX research has impacted Jonathan’s team (12:53)
• What signals Jonathan received that told him it’s time to hire a designer (17:05)
• How Jonathan’s team approaches shadowing users (21:20)
• Some of the recent successes of the product-led approach Jonathan is implementing on his team (25:28)
• What Jonathan would change if he had to start the process of moving to outcomes over outputs with his team all over again (30:04)
• Get the full scoop on the topics discussed in this episode on October 12, 2023 when Jonathan presents his deep-dive webinar to the Data Product Leadership Community. Available to members only. Apply today.
  
  

Links

• Care Quality Commission: https://www.cqc.org.uk/
• LinkedIn: https://www.linkedin.com/in/jcairnsterry

Today I’m joined by Anthony Deighton, General Manager of Data Products at Tamr. Throughout our conversation, Anthony unpacks his definition of a data product and we discuss whether or not he feels that Tamr itself is actually a data product. Anthony shares his views on why it’s so critical to focus on solving for customer needs and not simply the newest and shiniest technology. We also discuss the challenges that come with building a product that’s designed to facilitate the creation of better internal data products, as well as where we are in this new wave of data product management, and the evolution of the role.

 

Highlights/ Skip to:

• I introduce Anthony, General Manager of Data Products at Tamr, and the topics we’ll be discussing today (00:37)
• Anthony shares his observations on how BI analytics are an inch deep and a mile wide due to the data that’s being input (02:31)
• Tamr’s focus on data products and how that reflects in Anthony’s recent job change from Chief Product Officer to General Manager of Data Products (04:35)
• Anthony’s definition of a data product (07:42)
• Anthony and I explore whether he feels that decision support is necessary for a data product (13:48)
• Whether or not Anthony feels that Tamr qualifies as a data product (17:08)
• Anthony speaks to the importance of focusing on outcomes and benefits as opposed to endlessly knitting together features and products (19:42)
• The challenges Anthony sees with metrics like Propensity to Churn (21:56)
• How Anthony thinks about design in a product like Tamr (30:43)
• Anthony shares how data science at Tamr is a tool in his toolkit and not viewed as a “fourth” leg of the product triad/stool (36:01)
• Anthony’s views on where we are in the evolution of the DPM role (41:25)
• What Anthony would do differently if he could start over at Tamr knowing what he knows now (43:43)
  

Links

• Tamr: https://www.tamr.com/
• Innovating: https://www.amazon.com/Innovating-short-guide-making-things/dp/B0C8R79PVB
• The Mom Test: https://www.amazon.com/The-Mom-Test-Rob-Fitzpatrick-audiobook/dp/B07RJZKZ7F
• LinkedIn: https://www.linkedin.com/in/anthonydeighton/

Today I’m joined by Vera Liao, Principal Researcher at Microsoft. Vera is a part of the FATE (Fairness, Accountability, Transparency, and Ethics of AI) group, and her research centers around the ethics, explainability, and interpretability of AI products. She is particularly focused on how designers design for explainability. Throughout our conversation, we focus on the importance of taking a human-centered approach to rendering model explainability within a UI, and why incorporating users during the design process informs the data science work and leads to better outcomes. Vera also shares some research on why example-based explanations tend to out-perform [model] feature-based explanations, and why traditional XAI methods LIME and SHAP aren’t the solution to every explainability problem a user may have.

 

Highlights/ Skip to:

• I introduce Vera, who is Principal Researcher at Microsoft and whose research mainly focuses on the ethics, explainability, and interpretability of AI (00:35)
• Vera expands on her view that explainability should be at the core of ML applications (02:36)
• An example of the non-human approach to explainability that Vera is advocating against (05:35)
• Vera shares where practitioners can start the process of responsible AI (09:32)
• Why Vera advocates for doing qualitative research in tandem with model work in order to improve outcomes (13:51)
• I summarize the slides I saw in Vera’s deck on Human-Centered XAI and Vera expands on my understanding (16:06)
• Vera’s success criteria for explainability (19:45)
• The various applications of AI explainability that Vera has seen evolve over the years (21:52)
• Why Vera is a proponent of example-based explanations over model feature ones (26:15)
• Strategies Vera recommends for getting feedback from users to determine what the right explainability experience might be (32:07)
• The research trends Vera would most like to see technical practitioners apply to their work (36:47)
• Summary of the four-step process Vera outlines for Question-Driven XAI design (39:14)

 

Links

• “Human-Centered XAI: From Algorithms to User Experiences” Presentation
• “Human-Centered XAI: From Algorithms to User Experiences” Slide Deck 
• “Human-Centered AI Transparency in the Age of Large Language Models”
• MSR Microsoft Research
• Vera's Personal Website

In this episode, I give an overview of my PiCAA Framework, which is a framework I shared at my keynote talk for Netguru’s annual conference, Burning Minds. This framework helps with brainstorming machine learning use cases or reverse engineering them, starting with the tactic. Throughout the episode, I give context to the preliminary types of work and preparation you and your team would want to do before implementing PiCAA, as well as the process and potential pitfalls you may run into, and the end results that make it a beneficial tool to experiment with. 

 

Highlights/ Skip to:

• Where/ how you might implement the PiCAA Framework (1:22)
• Focusing on the human part of your ideas (5:04)
• Keynote excerpt outlining the PiCAA Framework (7:28)
• Closing a PiCAA workshop by exploring what could go wrong (18:03)

Links

• Experiencing Data Episode 106 with Jeremy Utley
• The Data Product Leadership Community
• Ask me a question (below the recent episodes)