Experiencing Data w/ Brian T. O’Neill (UX for AI Data Products, SAAS Analytics, Data Product Management)

052 - Reasons Automated Decision Making with Machine Learning Can Fail with James Taylor

In this episode of Experiencing Data, I sat down with James Taylor, the CEO of Decision Management Solutions. This discussion centers around how enterprises build ML-driven software to make decisions faster, more precise, and more consistent-and why this pursuit may fail.

We covered:

• The role that decision management plays in business, especially when making decisions quickly, reliably, consistently, transparently and at scale.
• The concept of the "last mile," and why many companies fail to get their data products across it
• James' take on operationalization of ML models, why Brian dislikes this term
• Why James thinks it is important to distinguish between technology problems and organizational change problems when leveraging ML.
• Why machine learning is not a substitute for hard work.
• What happens when human-centered design is combined with decision management.
• James's book, Digital Decisioning: How to Use Decision Management to Get Business Value from AI, which lays out a methodology for automating decision making.

Quotes from Today's Episode

"If you're a large company, and you have a high volume transaction where it's not immediately obvious what you should do in response to that transaction, then you have to make a decision - quickly, at scale, reliably, consistently, transparently. We specialize in helping people build solutions to that problem." - James 

"Machine learning is not a substitute for hard work, for thinking about the problem, understanding your business, or doing things. It's a way of adding value. It doesn't substitute for things." - James

"One thing that I kind of have a distaste for in the data science space when we're talking about models and deploying models is thinking about 'operationalization' as something that's distinct from the technology-building process." - Brian

"People tend to define an analytical solution, frankly, that will never work because[…] they're solving the wrong problem. Or they build a solution that in theory would work, but they can't get it across the last mile. Our experience is that you can't get it across the last mile if you don't begin by thinking about the last mile." - James 

"When I look at a problem, I'm looking at how I use analytics to make that better. I come in as an analytics person." - James

"We often joke that you have to work backwards. Instead of saying, 'here's my data, here's the analytics I can build from my data […], you have to say, 'what's a better decision look like? How do I make the decision today? What analytics will help me improve that decision?' How do I find the data I need to build those analytics?' Because those are the ones that will actually change my business." - James 

"We talk about [the last mile] a lot ... which is ensuring that when the human beings come in and touch, use, and interface with the systems and interfaces that you've created, that this isthe make or break point-where technology goes to succeed or die." - Brian

Links

• Decision Management Solutions
• Digital Decisioning: How to Use Decision Management to Get Business Value from AI
• James' Personal Blog
• Connect with James on Twitter
• Connect with James on LinkedIn

 

051 - Methods for Designing Ethical, Human-Centered AI with Undock Head of Machine Learning, Chenda Bunkasem

Chenda Bunkasem is head of machine learning at Undock, where she is focusing on using quantitative methods to influence ethical design. In this episode of Experiencing Data, Chenda and I explore her actual methods to designing ethical AI solutions as well as how she works with UX and product teams on ML solutions.

We covered:

• How data teams can actually design ethical ML models, after understanding if ML is the right approach to begin with  
• How Chenda aligns her data science work with the desired UX, so that technical choices are always in support of the product and user instead of “what’s cool”
• An overview of Chenda’s role at Undock, where she works very closely with product and marketing teams, advising them on uses for machine learning 
• How Chenda’s approaches to using AI may change when there are humans in the loop
• What NASA’s Technology Readiness Level (TRL) evaluation is, and how Chenda uses it in her machine learning work 
• What ethical pillars are and how they relate to building AI solutions
• What the Delphi method is and how it relates to creating and user-testing ethical machine learning solutions

Quotes From Today’s Episode 

“There's places where machine learning should be used and places where it doesn't necessarily have to be.” - Chenda

“The more interpretability, the better off you always are.” - Chenda

“The most advanced AI doesn't always have to be implemented. People usually skip past this, and they're looking for the best transformer or the most complex neural network. It's not the case. It’s about whether or not the product sticks and the product works alongside the user to aid whatever their endeavor is, or whatever the purpose of that product is. It can be very minimalist in that sense.” - Chenda 

“First we bring domain experts together, and then we analyze the use case at hand, and whatever goes in the middle — the meat, between that — is usually decided through many iterations after meetings, and then after going out and doing some sort of user testing, or user research, coming back, etc.” - Chenra, explaining the Delphi method. 

“First you're taking answers on someone's ethical pillars or a company's ethical pillars based off of their intuition, and then you're finding how that solution can work in a more engineering or systems-design fashion. “ - Chenda 

“I'm kind of very curious about this area of prototyping, and figuring out how fast can we learn something about what the problem space is, and what is needed, prior to doing too much implementation work that we or the business don't want to rewind and throw out.” - Brian

“There are a lot of data projects that get created that end up not getting used at all.”- Brian

Links

Undock website

Chenda's personal website

Substack

Twitter

Instagram

Connect with Chenda on LinkedIn

 

050 - Ways to Practice Creativity and Foster Innovation When You’re An Analytical Thinker

50 episodes! I can’t believe it. Since it’s somewhat of a milestone for the show, I decided to do another solo round of Experiencing Data, following the positive feedback that I’ve gotten from the last few episodes. Today, I want to help you think about ways to practice creativity when you and your organization are living in an analytical world, creating analytics for a living, and thinking logically and rationally. Why? Because creativity is what leads to innovation, and the sciences says a lot of decision making is not rational. This means we have to tap things besides logical reasoning and data to bring data products to our customers that they will love...and use. (Sorry!)

One of the biggest blockers to creativity is in the organ above your shoulders and between your ears. I frequently encounter highly talented technical professionals  who find creativity to be a foreign thing reserved for people like artists. They don’t think of themselves as being creative, and believe it is an innate talent instead of a skill. If you have ever said, “I don’t have a creative bone in my body,” then this episode is for you.

As with most technical concepts, practicing creativity is a skill most people can develop, and if you can inculcate a mix of thinking approaches into your data product and analytical solution development, you’re more likely to come up with innovative solutions that will delight your  customers. The first thing to realize though is that this isn’t going to be on the test. You can’t score a “92” or a “67” out of 100. There’s no right answer to look up online. When you’re ready to let go of all that, grab your headphones and jump in. I’ll even tell you a story to get going.

 

Links Referenced

Previous podcast with Steve Rader

 

049 - CxO & Digital Transformation Focus: (10) Reasons Users Can’t or Won’t Use Your Team’s ML/AI-Driven Software and Analytics Applications

Join the Free Webinar Related to this Episode

I'm taking questions and going into depth about how to address the challenges in this episode of Experiencing Data on Oct 9, 2020. 30 Mins + Q/A time. Replay will also be available.

Register Now

Welcome back for another solo episode of Experiencing Data. Today, I am primarily focusing on addressing the non-digital natives out there who are trying to use AI/ML in innovative ways, whether  through custom software applications and data products, or as a means to add new forms of predictive intelligence to existing digital experiences.

Many non-digital native companies today tend to approach software as a technical “thing” that needs to get built, and neglect to consider the humans who will actually use it — resulting in a lack of business or organizational value emerging. While my focus will be on the design and user experience aspects that tend to impede adoption and the realization of business value, I will also talk about some organizational blockers related to how intelligent software is created that can also derail a successful digital transformation efforts.

These aren’t the only 10 non-technical reasons an intelligent application or decision support solution might fail, but they are 10 that you can and should be addressing—now—if the success of your technology is dependent on the humans in the loop actually adopting your software, and changing their current behavior.

Links

• Want to address these issues? Learn about my Self-Guided Video Course and Instructor-Led Seminar
• Subscribe to my Free DFA Insights Mailing List: https://designingforanalytics.com/mailing-list/

 

048 - Good vs. Great: (10) Things that Distinguish the Best Leaders of Intelligent Products, Analytics Applications, and Decision Support Tools

Today I’m going solo on Experiencing Data! Over the years, I have worked with a lot of leaders of data-driven software initiatives with all sorts of titles. Today, I decided to focus the podcast episode on what I think makes the top product management and digital/software leaders stand out, particularly in the space of enterprise software, analytics applications, and decision support tools.

This episode is for anyone leading a software application or product initiative that has to produce real value, and not just a technology output of some kind. When I recorded this episode, I largely had “product managers” in mind, but titles can vary significantly. Additionally, this episode focuses on my perspective as a product/UX design consultant and advisor, focusing specifically at the traits associated with these leaders’ ability to produce valuable, innovative solutions customers need and want. A large part of being a successful software leader also involves managing teams and other departments that aren’t directly a part of the product strategy and design/creation process, however I did not go deep into these aspects today. As a disclaimer, my ideas are not based on research. They’re just my opinions. Some of the topics I covered include:

• The role of skepticism
• The misunderstanding of what it means to be a “PM”
• The way top software leaders collaborate with UX professionals, designers, and engineering/tech leads
• How top leaders treat UX when building customer-focused technology
• How top product management leaders define success and make a strategy design-actionable
• The ways in which great PMs enable empathy in their teams and evangelize meaningful user research
• The output vs. outcome mindset

047 - How Yelp Integrates Data Science, Engineering, UX, and Product Management when Creating AI Products with Yelp’s Justin Norman

In part one of an excellent series on AI product management, LinkedIn Research Scientist Peter Skomoroch and O’Reilly VP of Content Strategy Mike Loukides explained the importance of aligning AI products with your business plans and strategies. In other words, they have to deliver value, and they have to be delivered on time. Unfortunately, this is much easier said than done. I was curious to learn more about what goes into the complex AI product development process, and so for answers I turned to Yelp VP of Data Science Justin Norman, who collaborated with Peter and Mike in the O’Reilly series of articles. Justin is a career data professional and data science leader with experience in multiple companies and industries, having served as director of research and data science at Cloudera Fast Forward Labs, head of applied machine learning at Fitbit, head of Cisco’s enterprise data science office, and as a big data systems engineer with Booz Allen Hamilton. He also served as a Marine Corps Officer with a focus in systems analytics. We covered:

• Justin’s definition of a successful AI product
• The two key components behind AI products
• The lessons Justin learned building his first AI platform and what insights he applied when he went to Yelp.
• Why AI projects often fail early on, and how teams can better align themselves for success.
• Who or what Beaker and Bunsen are and how they enable Yelp to test over 700 experiments at any one time.
• What Justin learned at an airline about approaching problems from a ML standpoint vs. a user experience standpoint—and what the cross-functional team changed as a result.
• How Yelp incorporates designers, UX research, and product management with its technical teams
• Why companies should analyze the AI, ML and data science stack and form a strategy that aligns with their needs.
• The critical role of AI product management and what consideration Justin thinks is the most important when building a ML platform
• How Justin would approach AI development if he was starting all over at a brand new company
• Justin’s pros and cons about doing data science in the government vs. the private sector.

Quotes from Today’s Episode

“[My non-traditional background] gave me a really broad understanding of the full stack [...] from the physical layer all the way through delivering information to a decision-maker without a lot of time, maybe in an imperfect form, but really packaged for what we're all hoping to have, which is that value-add information to be able to do something with.” - Justin

“It's very possible to create incredible data science products that are able to provide useful intelligence, but they may not be fast enough; they may not be [...] put together enough to be useful. They may not be easy enough to use by a layperson.” -Justin

 

“Just because we can do things in AI space, even if they're automated, doesn't mean that it's actually beneficial or a value-add.” - Justin

“I think the most important thing to focus on there is to understand what you need to be able to test and deploy rapidly, and then build that framework.” - Justin

“I think it's important to have a product management team that understands the maturity lifecycle of building out these capabilities and is able to interject and say, ‘Hey, it's time for us to make a different investment, either in parallel, once we've reached this milestone, or this next step in the product lifecycle.’” - Justin

“...When we talk about product management, there are different audiences. I think [Yelp’s] internal AI product management role is really important because the same concepts of thinking about design, and how people are going to use the service, and making it useful — that can apply to employees just as much as it can to the digital experience that you put out to your end customers.” -Brian

“You hear about these enterprise projects in particular, where the only thing that ever gets done is the infrastructure. And then by the time they get something ready, it’s like the business has moved on, the opportunity's gone, or some other challenge or the team gets replaced because they haven't shown anything, and the next personcomes in and wants to do it a different way.” - Brian

Links

• Yelp
• O’Reilly three-part article:
  • Part 1
  • Part 2
  • Part 3
• Bunsen Article (The Yelp AI Platform)
• Twitter: @JustinJDN
• Justin’s LinkedIn

 

046 - How Steelcase’s Data Science, UX, & Product Teams Are Helping Customers Design Safer Office Workplaces Informed by Covid-19 Recommendations w/ J...

When you think of Steelcase, their office furniture probably comes to mind. However, Steelcase is much more than just a manufacturer of office equipment. They enable their customers (workplace/workspace designers) to help those designers’ clients create useful, effective, workplaces and offices that are also safe and compliant.

Jorge Lozano is a data science manager at Steelcase and recently participated as a practitioner and guest on an IIA webinar I gave about product design and management being the missing links in many data science and analytics initiatives. I was curious to dig deeper with Jorge about how Steelcase is enabling its customers to adjust workspaces to account for public health guidelines around COVID-19 and employees returning to their physical offices. The data science team was trying to make it easy for its design customers to understand health guidelines around seat density, employee proximity and other relevant metrics so that any workspace designs  could be “checked” against public health guidelines.

Figuring out the what, when, and how to present these health guidelines in a digital experience was a journey that Jorge was willing to share.

We covered:

• Why the company was struggling to understand how their [office] products came together, and how the data science group tried to help answer this.
• The digital experience Steelcase is working on to re-shape offices for safe post-pandemic use.
• How Steelcase is evaluating whether their health and safety recommendations were in fact safe, and making a difference.
• How Jorge’s team transitioned from delivering “static data science” outputs into providing an enabling capability to the business.
• What Steelcase did to help dealer designers when engaging with customers, in order to help them explain the health risks associated with their current office layouts and plans.
• What it was like for Jorge’s team to work with a product manager and UX designer, and how it improved the process  of making the workspace health guidelines useful.

 

Resources and Links:

• Steelcase: https://www.steelcase.com/
• LinkedIn: https://www.linkedin.com/in/jorge-lozano-flores/

 

Quotes from Today’s Episode

“We really pride ourselves in research-based design” - Jorge

“This [source data from design software] really enabled us to make very specific metrics to understand the current state of the North American office.” - Jorge

“Using the data that we collected, we came up with samples of workstations that are representative of what our customers are more likely to have. We retrofitted them, and then we put the retrofitted desk in the lab that basically simulates the sneeze of a person, or somebody coughing, or somebody kind of spitting a little bit while they're talking, and all of that. And we're collecting some really amazing insights that can quantify the extent to which certain retrofits work in disease transmission.” - Jorge

“I think one of the challenges is that, especially when you're dealing with a software design solution that involves probabilities, someone has to be the line-drawer.” - Brian

“The challenge right now is how to set up a system where we can swarm at things faster, where we're more efficient at understanding the needs and [are able to get] it in the hands of the right people to make those important decisions fast? It's all pointing towards data science as an enabling capability. It's a team sport.” - Jorge

 

045 - Healthcare Analytics…or Actionable Decision Support Tools? Leadership Strategies from Novant Health’s SVP of Data Products, Karl Hightower

Healthcare professionals need access to decision support tools that deliver the right information, at the right time. In a busy healthcare facility, where countless decisions are made on a daily basis, it is crucial that any analytical tools provided actually yield useful decision support to the target customer. In this episode, I talked to Karl Hightower from Novant Health about how he and his team define “quality” when it comes to data products, and what they do to meet that definition in their daily work. Karl Hightower is the Chief Data Officer and SVP of Data Products at Novant Health, a busy hospital and medical group in the Southeast United States with over 16 hospitals and more than 600 clinics. Karl and I took a deep dive into data product management, and how Karl and his team are designing products and services that help empower all of the organization’s decision makers. In our chat, we covered:

• How a non-tech company like Novant Health approaches data product management
• The challenges of designing data products with empathy in mind while being in an environment involving physicians and healthcare professionalsThe metric Karl’s team uses to judge the quality and efficacy of their data products, and how executive management contributed to defining this success criteria
• How Karl encourages deep empathy between analytics teams and their users by deeply investigating how the users being served by the team make decisions with data
• How and why Novant embraces design and UX in their data product work
• The types of outcomes Karl sees when designers and user experience professionals work with analytics and data science practitioners.
• How Karl was able to obtain end user buy-in and support for ?
• The strategy Karl used to deal with a multitude of “information silos” resulting from the company’s numerous analytics groups.

 

Resources and Links:

• Novant Health website: https://www.novanthealth.org/
• Novant Health LinkedIn: https://www.linkedin.com/company/novanthealth/
• Karl Hightower LinkedIn: https://www.linkedin.com/in/karl-hightower-4528123/

Quotes from Today’s Episode

“I tend to think of product management as a core role along with a technical lead and product designer in the software industry. Outside the software industry, I feel like product management is often this missing hub. ” - Brian

 

“I really want to understand why the person is asking for what they're asking for, so there is much more of a closer relationship between that portfolio team and their end-user community that they're working with. It's almost a day-to-day living and breathing with and understanding not just what they're asking for and why are they asking for it, but you need to understand how they use information to make decisions.” - Karl

 

“I think empathy can sound kind of hand-wavy at times. Soft and fluffy, like whipped cream. However, more and more at senior levels, I am hearing how much leaders feel these skills are important because the technology can be technically right and effectively wrong.” - Brian

 

“The decision that we got to on executive governance was how are we going to judge success criteria? How do we know that we're delivering the right products and that we're getting better on the maturity scale? And the metric is actually really simple. Ask the people that we're delivering for, does this give you what you need when you need it to make those decisions? - Karl

 

“The number one principle is, if I don't know how something is done [created with data], I'm very unlikely to trust it. And as you look at just the nature of healthcare, transparency absolutely has to be there because we want the clinicians to poke holes in it, and we want everyone to be able to trust it. So, we are very open. We are very transparent with everything that goes in it.” - Karl

 

“You need to really understand the why. You’ve got to understand what business decisions are being made, what's driving the strategy of the people who are asking for all that information.” - Karl

 

 

 

044 - The Roles of Product and Design when “Competing in the Age of AI” with HBS Professor and Author Karim Lakhani

If there’s one thing that strikes fear into the heart of every business executive, it’s having your company become the next Blockbuster or Neiman Marcus — that is, ignoring change, and getting wiped out by digital competitors. In this episode, I dived into the changing business landscape with Karim Lakhani who is a Professor at Harvard Business School and co-author of the new book Competing in the Age of AI: When Algorithms and Networks Run the World, which he wrote with his friend and colleague at HBS, Marco Iansiti.

 

We discuss how AI, machine learning, and digital operating models are changing business architecture, and disrupting traditional business models. I also pressed Karim to go a bit deeper on how, and whether he thinks product mindset and design factor in to the success of AI in today’s businesses. We also go off on a fun tangent about the music industry, which just might have to be a future episode!. In any case, I highly recommend the book. It’s particularly practical for those of you working in organizations that are not digital natives and want to hear how the featured companies in the book are setting themselves apart by leveraging data and AI in customer-facing products and in internal applications/operations. Our conversation covers:

• Karim’s new book, Competing in the Age of AI: When Algorithms and Networks Run the World, co-authored with Marco Iansiti.
• How digital operating models are colliding with traditional product-oriented businesses, and the impact this is having on today’s organizations.
• The critical role of data product management that is frequently missing when companies try to leverage AI
• Karim’s thoughts on ethics in AI and machine learning systems, and how they need to be baked into business and engineering.
• The similarity Karim sees between COVID-19 and AI
• The role of design, particularly in human-in-the-loop systems and how companies need to consider the human experience in applications of AI that augment decision making vs. automate it.
• How Karim sees the ability to adapt in business as being critical to survival in the age of AI

Resources and Links

•  Book Link: https://www.amazon.com/Competing-Age-AI-Leadership-Algorithms/dp/1633697622/
• Twitter: https://twitter.com/klakhani
• LinkedIn: https://www.linkedin.com/in/professorkl/
• Harvard Business Analytics Program: https://analytics.hbs.edu/

Quotes from Today’s Episode

“Our thesis in the book is that a new type of an organization is emerging, which has eliminated bottlenecks in old processes.” - Karim

 

“Digital operating models have exponential scaling properties, in terms of the value they generate, versus traditional companies that have value curves that basically flatten out, and have fixed capacity. Over time, these digital operating models collide with these traditional product models, win over customers, and gather huge amounts of market share….” - Karim

 

“This whole question about human-in-the-loop is important, and it's not going to go away, but we need to start thinking about, well, how good are the humans, anyway? - Karim

 

“Somebody once said, “Ethics defines the boundaries of what you care about.” And I think that's a really important question…” - Brian

 

“Non-digital natives worry about these tech companies coming around and eating them up, and I can’t help but wonder ‘why aren't you also copying the way they design and build software?’” - Brian

 

“...These established companies have a tough time with the change process.” - Karim

043 - What a Product Management Mindset Can do for Data Science and Analytics Leaders with Product School CEO, Carlos González de Villaumbrosia

I am a firm believer that one of the reasons that data science and analytics has a high failure rate is a lack of product management and design. To me, product is about a mindset just as much as a job title, and I am repeatedly hearing how more and more voices in the data community are agreeing with me on this (Gartner CDO v4, International Inst. for Analytics, several O’Reilly authors, Karim Lakhani’s new book on AI, and others). This is even more true as more companies begin to leverage AI. So many of these companies fear what startups and software companies are doing, yet they do not copy the way tech companies build software applications and enable specific user experiences that unlock the desired business value.

Integral to building software is the product management function—and when these applications and tools have humans in the loop, the product/UX design function is equally as important to ensure adoption, usability, engagement, and alignment with the business objectives.

In modern tech companies, the overlap between product design and product management can be significant, and frequently, product leaders in tech companies come up through both design and engineering ranks and indeed my own work heavily overlaps with product. What this tells me is that product is a mindset, and it’s a role many can learn if they believe it’s critical.

So why aren’t more data science and analytics leaders forming strong product design and analytics functions? I don’t know, so I decided to bring Carlos onto the show to talk about his company, Product School, which offers product management training and features instructors from many of the big tech companies on how to do it. In this episode, Carlos provides a comprehensive overview of why he launched Product School, what makes an effective product manager, and the importance of having structured vision and alignment when developing products.

This conversation explores:

• Why Carlos launched the Product School for professionals who want to learn on the side without quitting their job and putting their life on hold.
• The type of mentality  product managers need to have and whether specialization matters within product management.
• Whether being a product manager in machine learning and AI is different than working with a traditional software product.
• How product management is not project management
• Advice for approaching executive decision makers about product management education
• How to avoid the trap of focusing too heavily on process
• How product management often leads to executive leadership roles
• The “power trio” of engineering, product management, and design, and the value of aligning all three groups.
• Understanding the difference between applied and academic experience
• How the relationship between design and PM has changed over the last five years
• What the gap looks like between a skilled PM and an exceptional one.

Resources and Links

The State of Product Analytics (Also referred to as The Future of Product Analytics in the audio)

Mixpanel, company that they partnered with to create the above report

Episode 17 of Experiencing Data

Twitter

Main Company Site

ProductCon

Productverse

 

Quotes from Today’s Episode

“You can become a product manager by building products. You don't need to be a software engineer. You don’t need to have an MBA. You don't need to be an incredible, inspiring visionary. This is stuff that you can learn, and the best way to learn it is by doing it.” - Carlos

“A product manager is a generalist. And in order to become a generalist, usually you have to have some sort of [specialty] before. So, we define product management as the intersection in between business, engineering, and design. And you can become a good product manager from either of those options.” - Carlos

“If you have [a power trio of technology, product, and design] and the energy is right, and the relationships are really strong, boy, you can get a lot of stuff done, and you can iterate quickly, and really produce some great stuff.” - Brian

“I think part of the product management mindset... is to realize part of your job now is to be a problem finder, it’s to help set the strategy, it's to help ensure that a model is not the solution.” - Brian

“I think about a bicycle wheel with the hub in the center and the spokes coming out. Product management is that hub, and it reports up into the business, but you have all these different spokes, QA, and software engineering, maybe data science and analytics, product design, and user experience design. These are all kind of spokes.” - Brian

“These are people who are constantly learning, but not just about their products. They’re constantly learning in general. Reading books, practicing sports, doing whatever it is, but always looking at what's new and wanting to play around with it, just to be dangerous enough. So, I think those three areas: obsession with a customer based on data; obsession with empathy; and then obsession with learning, or just being curious are really critical.” - Carlos