An Exploration of the Applications, Challenges, and Success Factors in AI-Driven Product Development and Management

Essential AI methods and techniques are employed in the product development and management process, encompassing areas such as prediction, optimization, automation, textual analysis (Natural Language Processing domain), and visual analysis (Computer Vision domain). Diverse methods are utilized contingent on the phase of product development or subsequent management. In the product ideation phase, Natural Language Processing methods are predominant, particularly knowledge extraction, sentiment analysis, and conversational AI. Knowledge extraction methods can be used to extract product ideas from various online sources (Luo, Huang and Zhu, 2019; Zhang, et al., 2021), sentiment analysis to assess the positivity or negativity of product reviews (Aguwa, Olya and Monplaisir, 2017; Giannakis, et al., 2022; Symeonidis, Peikos and Arampatzis, 2022), and conversational AI in the form of chatbots to interview customers to extract product ideas directly from them (Chen, Le and Florence, 2021; Eren, 2021). In the design and development phase, the aforementioned methods can also be applied to source design ideas; however, in this phase, Computer Vision and techniques related to autonomous systems and robotics play a significant role, especially for physical products, by assisting with product manufacturing (Tubadji, Huang and Webber, 2021; Johnson, et al., 2022). Vision methods are particularly instrumental in product quality management and testing (Chen and Wang, 2018; Jain, Meenu and Sardana, 2020; Liao, et al., 2021; Zhao, et al., 2022), including anomaly detection (Langone, Cuzzocrea and Skantzos, 2020; Garg, et al., 2021).

Various AI solutions are employed in finished product management, encompassing prescriptive analytics (Viswanandhne, et al., 2019), competitor or customer analysis (Luoma, et al., 2017; Ballestar, Grau-Carles and Sainz, 2019), churn prediction (Pustokhina, et al., 2021), personalization (Byrd and Darrow, 2021), store layout (Huang, Bergman and Gopal, 2019; Lee and Shin, 2020), demand and sales forecasting (Koulouriotis and Mantas, 2012; Atsalakis, 2014; Jin and Shin, 2020), or choice modeling (Liebe and Meyerhoff, 2021). Although in a number of articles various artificial intelligence solutions have been used at different stages of the product development and management process, the plethora of solutions means that companies may not be aware of some of the possible AI solutions, and therefore may not incorporate these solutions into the process.

In both data-driven and AI-driven product development, data serve as the central component. To construct and enhance new products, organizations must manage vast quantities of data and possess the appropriate infrastructure for this task (Bosch, Olsson and Crnkovic, 2018). Based on interviews conducted by Figalist, et al. (2020), a mere 9% of respondents indicated that the effort required to establish infrastructure for software analytics and AI-based business intelligence is relatively low or very low. In contrast, over 50% reported that it is high (37%) or very high (14%). The endeavor to create the right infrastructure presents challenges not only from a financial standpoint but also due to technical complexities (Figalist, et al., 2020). Data are hard to access and difficult to process (Patil and Mason, 2015). AI solutions can manage both structured and unstructured data; however, preprocessing is a prerequisite, and this process entails data cleaning, transformation, and selection, each of which can rapidly become multifaceted and intricate.

The literature also identifies data quality as a significant challenge in new product development (Cubric, 2020; Figalist, et al., 2020; Paschen, Pitt and Kietzmann, 2020). Data quantity seldom corresponds with quality-organizations gather data from various sources, complicating the maintenance of high-quality standards. Furthermore, the data may be incomplete, defective, or biased, leading to suboptimal AI algorithm performance and ultimately, undesirable products.

Privacy and security constitute another considerable concern (Paschen, Pitt and Kietzmann, 2020). Individuals are more aware of their privacy than ever before and expect transparent communication from companies regarding data collection, processing, purpose, and security measures. In particular, the intricacies of AI algorithms’ data handling are challenging to elucidate, even for the scientists who develop these algorithms, which further complicates obtaining user consent.

The longstanding question of generating value from new products has assumed a novel dimension with data and AI. Mishra and Pani (2020) posit that there are two approaches to deriving value from new technology: exploiting, which involves extending the current capabilities of the technology, and exploring, which entails discovering new alternatives. This assertion is equally applicable to AI solutions. Companies aspire to utilize AI in product development to foster innovation, optimize manufacturing processes, enhance efficiency, and augment complexity and knowledge management (Prem, 2019).

Nonetheless, organizations contemplating the adoption of AI frequently fail to implement it (Mishra and Pani, 2020). One contributing factor is the insufficient knowledge on how to extract value. Another factor is risk aversion, as there is a general inclination to mitigate risk rather than to maximize the value (Desouza, Dawson and Chenok, 2020). When presented with opportunities exhibiting a low risk and low value or high risk and high value, companies often explore partnerships, not only with other enterprises but also with academic institutions (Magistretti, Dell’Era and Messeni Petruzzelli, 2019; Mishra and Pani, 2020) to develop AI-based solutions. However, some studies suggest that such collaborations are occasionally underutilized by companies due to the gap between practitioners and academia, resulting in a dearth of direct benefits for the organization (Mishra and Pani, 2020).

For low-risk and high-value situations, companies tend to develop products internally (Desouza, Dawson and Chenok, 2020). The principal challenge in managing data-driven or AI products, stemming from prior data issues, is the initial cost and time (Prem, 2019; Figalist, et al., 2020; Paschen, Pitt and Kietzmann, 2020). Priorities are redirected to other subjects and products, causing insufficient resources, and without resources, R&D departments cannot demonstrate a value. Various studies indicate that the most effective strategy to disrupt this vicious cycle is to construct a low-cost minimum viable product (MVP) internally (Figalist, et al., 2020; Desouza, Dawson and Chenok, 2020) and then determine subsequent steps. Research also highlights the ease with which R&D departments can succumb to the efficiency trap rather than concentrating on effectiveness, where effectiveness is defined as “the amount of customer value created per unit of R&D resources” (Bosch, 2019). In the final case of high-risk and low-value solutions, companies opt to outsource the work (Desouza, Dawson and Chenok, 2020; Cricelli, Grimaldi and Vermicelli, 2022).

The aforementioned issues with data quality, algorithms, and ultimately, products lead stakeholders to increasingly doubt the results and profitability of employing AI data and algorithms in product development (Figalist, et al., 2020). Utilizing data and AI to create a product can be considered an IT project, and statistics reveal that IT projects are generally more complex and prone to failure than other types of projects (Mishra and Pani, 2020). This presents an additional factor that could negatively influence executives’ decision to initiate product development using these techniques.

Within product development teams, a prevalent discord is observed, primarily between managers and software developers. Managers are often skeptical upon witnessing initial results without discerning the benefits or possess limited general AI knowledge, leading to premature resistance and rejection of the concept. This knowledge deficit also results in unrealistic expectations for AI products (Figalist, et al., 2020), concerning not only product functionality and cost but also time-to-market. Frequently, managerial ignorance causes unreasonably short timeframes to be imposed for developing the appropriate solution. A related issue is the challenge of knowledge transfer-R&D departments must educate managers on product functionality, who, in turn, must understand the product sufficiently to defend its value to higher management. Failure in this regard may not indicate a worthless product but rather an inadequate understanding of complex AI methods by the manager to grasp and defend the solution (Figalist, et al., 2020).

In many organizations, executives and managers rely on business intuition. When data are available, this reliance is unnecessary because data present facts-product perception, purchase locations, associated products, etc. However, executives and managers sometimes prefer their opinions, even when data suggest otherwise. Moreover, their beliefs are so deeply ingrained that they attempt to interpret data in a manner consistent with their preconceptions, rather than accepting objective reality (Bosch, 2019; Santana and Díaz-Fernández, 2022; Zirar, 2023). This issue is closely tied to trust—namely, the extent to which we can trust data. If factual data cannot be trusted, how can we have faith in AI algorithms when even the software developers who create them struggle to explain their functionality?

As commercial AI solutions are still in relatively early development stages, a general set of guidelines for AI-driven product development and management is lacking, which could be adapted on a product- or sectorspecific basis. Bosch, Olsson and Crnkovic (2018) note that AI-driven development “is still in its infancy in many of the companies we studied we foresee significant work in the area of defining systematic and repeatable methods for creating, evolving, and maintaining systems using AI techniques.” One of the solutions could be implementing and maintaining machine learning models based on a flywheel loop. Applying the flywheel to product strategy (Gurkan and de Véricourt, 2022) aids product managers in building less prescriptive product roadmaps and making more data-driven (or AI-driven) decisions. If AI-driven product development and management involves using AI solutions in each phase from ideation to implementation and optimization, a product AI strategic flywheel or AI solutions closing the product strategic flywheel loop could create such a model. The product managers, along with their team, would adapt this generic model to the company’s products. According to Bosch (2019) and Cubric (2020), future research on AI in business should focus on methodology improvements, including better evaluation methods and multidisciplinary approaches. To fully realize AI’s potential in product development and management, organizations must develop strategies that effectively address these challenges.

When evaluating the success of a product, two pertinent questions arise: what determines product success? (i.e., the factors contributing to product success) and how success is measured? General product success factors identified in the literature include innovative value, strategic approach and vision, a high-quality and structured new product development process, market research, customer-centricity, and retention of qualified employees along with necessary resources and infrastructure (Soltani-Fesaghandis and Pooya, 2018). According to Duan, Edwards and Dwivedi (2019), extensive research has been conducted on factors influencing the use, impact, success, and failure of information systems, yet there are limited research studies on identifying critical success factors for AI and its impact in the Big Data era.

In terms of success factors for AI implementation in companies, McKinsey (2021) distinguished several categories differentiating AI high performers (those attributing at least 20% of EBIT to AI solutions) from other companies. Baseline factors and user enablement-related factors encompass design thinking in AI solution development, a clear governance model for AI solutions, AI-related staff skill development programs, and explaining model basics to users or providing hands-on training for non-technical staff. More advanced factors associated with data, tools, and technology include a single, accessible data dictionary across the organization, aligned and scalable internal processes related to data use and labeling, and a focus on regular refresh, explainability, and reusability of AI models and engaging third-party companies for testing and validation of AI models.

Companies employ various approaches to measure the success of AI implementation in product development and management processes. Benefits can be classified as intangible (difficult to measure) or tangible (directly translatable to numerical value). Intangible benefits include culture change (AI drives product innovation culture), competitive advantage (delivering significant product value change), and the halo effect (attracting new customers due to AI-driven processes). Tangible benefits primarily encompass cost reduction (and/or avoidance), revenue increase, customer satisfaction (measured through CSAT and NPS metrics), risk mitigation, and compliance (through avoided costs) (Burgess, 2018). According to McKinsey (2021), 69% of respondents surveyed in 2020 who had implemented AI solutions in their company reported a cost decrease following AI implementation in product and/or service development, with a third of respondents indicating a decrease greater than 20%. In terms of revenue increase, 70% of respondents reported an increase after introducing AI solutions in product and/or service development, with 40% of the increase exceeding 6%.

Two themes appeared to be related to AI-driven product development and management process in general:

knowledge and responsibilities of a product manager,

These themes are now examined in relation to the interview data.

Knowledge and responsibilities of a product manager: Most respondents (A1, A2, A4, A5, A7, A9, A10) concurred that product development and management processes differ across companies, implying that product managers’ responsibilities also vary. The implementation of AI solutions in the process depends on the product manager’s responsibilities: “Before asking the question about AI applications in the product development and management process, it is important to inquire about the responsibilities of the product manager, because in every company, these responsibilities will differ. In one company, this might mean a product owner who manages a team of software developers, in another, a product marketing manager mainly responsible for the product’s marketing campaign, and in yet another, a product manager accountable for the roadmap and profit and loss of the product […]. In each of these cases, making such changes may not be part of the product manager’s responsibilities” (Interview A4). Another respondent adds: “Sometimes, it is the director or top management who dictates what the process should look like. Often, it also depends on the company size—in large corporations, there are dedicated individuals in charge of process improvement, while in start-ups, the product manager has more freedom but less time to enhance the process because the focus is on proving the product” (Interview A5). Respondents also agree that product managers currently possess limited knowledge of AI solutions, posing a challenge to the proposal to use these solutions for streamlining the product development and management process: “The product manager should take the time to understand what is possible in terms of AI solutions and how these solutions and the amount of data affect what they can achieve” (Interview A2). The majority of interviewees recognize that they had to change their thinking when dealing with products:

“There should be a greater focus on thinking that everything is data—every document, our conversation, audio, video—then it will be easier for product managers to grasp some of the difficulties and challenges of data, and for managers and senior management to accept it” (Interview A10). In some cases, the problem an AI model is meant to solve is too complex, or such a case has not existed before, causing the AI to query a human for an answer, e.g., an AI surveying product sentiment analysis is uncertain whether an opinion is positive or negative. This approach is commonly referred to as “Human In The Loop” (HITL). Respondents add: “Even if that human in the loop is a software developer, the product manager needs to have knowledge of the process and these solutions to participate” (Interview A9).

Process improvement as a product: Many respondents emphasize that before discussing AI applications or challenges, the matter of implementation must be addressed. “The introduction of any tool (AI or analytics) that could assist product managers in their daily work should simply be regarded as the creation of another product” (Interview A4). One respondent posits that it is easier to conceptualize such implementation when considering everything as a product: “If we wanted to launch, for example, a bicycle, then the bicycle is the core of the product, but the product encompasses everything around it, including the process of making the bicycle, marketing, selling, scaling, and so forth” (Interview A10). Respondents note that treating the process as a product in this manner also presents two additional challenges related to the timing and funding of such projects. If new tools were continuously proposed to enhance the process, there would be no time to manage and market the products that constitute the core of the business: “The best way to start is with a simple Minimum Viable Product (MVP) for such a tool, streamlining the process - build an MVP with a small amount of time and funding to demonstrate its value and show key stakeholders. However, an MVP is not production-ready, and that’s where the risk lies - the time from MVP to actual implementation” (Interview A12).

The discussions with respondents about AI applications and tools in the product development and management process revolved around two themes-current applications and tools, and those that could emerge to support the process. Respondents (A4, A8, A10) discussed the AI applications they currently observe during product development and management: “AI solutions during the product management process can be seen in microservices that suggest what to write, link up to-do shuffles, and recommend people” (Interview A4). Another adds: “It’s all the tools related to Computer Vision, Natural Language Processing, Automatic Speech Recognition, Text-To-Speech, such as Google Docs, where AI solutions complete the sentence, that are able to help the product manager in the process” (Interview A8).

Some respondents (A3, A12) stated the need for a tool relieving them from not relevant tasks: “It would be highly beneficial if an AI solution introduced could relieve the product manager of all duties not related to creating a roadmap, being responsible for profit and loss, and focusing on new target groups” (Interview A3).

Other respondents (A4, A6, A11) indicate that it would be advantageous to have support during the new product development process phases: “It would be a great idea to have a tool that shows you what to concentrate on at any given time when creating a product, a tool that guides you through the process and assists you in managing” (Interview A4). One respondent highlights that it depends on the product and describes what this looks like in the drug discovery niche: “AI solutions are useful at an early stage, i.e., in the product ideation and idea screening phase, where certain ideas need to be discarded, and with product design, where it can suggest new compounds” (Interview A9).

Another topic raised by the respondents (A6, A9) was decision making help for finalized product: “AI solutions could especially help with decision-making issues concerning the product, such as business model choices, prompting which elements of a prototype to test, or suggesting how to conduct A/B testing” (Interview A6).

When interviewed regarding the challenges of implementing AI solutions in the product development and management process, all respondents (A1–A12) agreed that the quantity and quality of available data are crucial. “If we would like to implement any AI solution that would assist in every phase of product implementation and subsequent management, the organization must possess vast amounts of data. Additionally, one must consider at what number of products, what number of sales, it makes sense to introduce such a solution and what decision it will help the product manager make” (Interview A2). Another adds: “Acquiring, cleaning, and preparing data so that it is ready for the AI model is the most difficult and time-consuming task. There is no generic program that by default collects data of different types, from different sources, and models it in such a way that the data is ready to be implemented in the AI model” (Interview A1). One respondent notes that an organization’s maturity or readiness to use AI solutions, related to the collected data, is a significant factor: “Working as a consultant for AI implementations, there have been cases of companies where product managers wanted to streamline the product management process, but clients abandoned the idea because they were not yet at a level, in terms of data quantity and quality, to begin using AI solutions” (Interview A6). Respondents also observe that preparing a model itself is not difficult, and it is often possible to use existing models-if a company has previously introduced models for sales prediction, for example, then it is possible to use that prediction in another process after minor modifications.

There is no methodology or framework that guides the use of AI in the product development and management process: “Organizations employ various methodologies, models, and techniques when developing new products, such as CRISP-DM, AgilePM, TDSP, and others, but there is no methodology or model that instructs on how best to approach AI products, whether developing a tool that supports a process or the AI solution is part of a product” (Interview A12). One interviewee suggests MLOps as a potential solution: “MLOps, as a set of practices, is not typically dedicated to AI-driven product development and management, but it could serve as the foundation for a methodology or framework” (Interview A1). According to another respondent, however, when creating such a framework based on another, one must ask: “Different companies have different approaches to product development; therefore, if we were to incorporate AI solutions that aid the product manager, the question arises: what would be the benefit of using AI in such a framework? We would need to have some measure that evaluates how a given AI solution helps, i.e., does it accelerate the process by X%, does it reduce the cost by Y%, etc.” (Interview A11).

Another critical factor, which was mentioned earlier and is a significant challenge according to most respondents (A1, A3, A5, A6, A8, A9), is the attitude of people in the organization toward the use of AI solutions—product managers, engineers, and senior management. “When I was still working as an AI engineer and had meetings with the product manager regarding proposals to implement these solutions, even ones that could help the product manager on a day-to-day basis, what was important was the hard, cold numbers” (Interview A3). Another adds: “I often found myself talking to colleagues who worked as data scientists and, surprisingly, they preferred to work on projects where they didn’t have to implement AI models. This was due to the fact that senior management often told them to improve the accuracy of the models indefinitely, which did not save as much money as the senior managers originally thought” (Interview A1).

Another frequently (A1, A4, A5, A7, A8, A11) mentioned challenge was security and privacy, especially if the product was implemented in sectors sensitive to these factors: “When working on a product that is implemented in, for example, banks or medical institutions, even if the AI solution does not support the product, but the process itself, security and privacy issues are at the forefront” (Interview A7). Another adds: “Let’s say you want to introduce an AI solution that supports the development process of a new product-if you don’t develop it yourself, there is a lot of risk in using external companies and tools by sharing data with them. In such cases, you want to know exactly who the company is that is introducing such a solution to you” (Interview A5).

Several respondents (A1, A2, A4, A5, A7, A8, A11) also emphasize the challenges of trust and explainability of AI solutions: “If we had a program that would take the data itself, input it into a model, and based on that do something or decide something, the issue of trust in that whole process arises—are we able to trust AI decisions without any oversight?” (Interview A1). Another respondent proposes a solution: “If we had a product manager using AI models to support the process, it would be better to explain the prediction, instead of just showing the prediction—instead of replacing the human, you can help them. So here we have a Human In The Loop solution, but with the addition of superhuman abilities” (Interview A4). As with security and privacy, trust factors and explainability are important in certain specific domains: “Depending on what the AI solution would help with in the product development process, it is important to remember that where we are dealing with financial, medical, or data center-related products, there is a much greater emphasis on explaining what is happening than in other sectors” (Interview A11).

One factor mentioned by respondents that is not often considered as a challenge in a project is computing power: “When implementing an AI solution, no matter the process, people often forget how important the computing power needed to produce the models is. This can even sometimes bring the whole project to a halt” (Interview A11). Another adds: “When implementing AI solutions, people usually do an estimate of the cost of implementing the models, but very often forget to estimate the cost of using the model. Later it turns out that the model is used hundreds of times in a short period of time, and this consumes enormous amounts of computing power” (Interview A9).

Respondents found it more challenging to discuss success factors when implementing AI solutions in the product development and management process compared to discussing challenges earlier. Some respondents (A1, A4, A7, A8, A9, A11) stress the need to have an appropriate metrics, e.g., financial. One respondent states: “We define product success financially-we have a financial plan, and if the product meets that plan, then it’s a success. If we introduce a tool that improves the process of making that product or makes the product manager’s job easier, then that should translate into a more successful product. The issue is just how to measure it” (Interview A4). Another interviewee also raises the issue of appropriate metrics: “There are two ways—one way, where we have two similar products with similar functionalities, we carry out the whole process of development of the first product and examine the time and resources it required, without using AI solutions, and on the second one, we carry out the same process but with AI tools and compare—but it is very time-consuming. The second way is where we have an AI tool, and we look at how the product manager performs with and without that tool. In the latter case, you would need to introduce some quantified measures” (Interview A7). One respondent believes that such a measure could be a satisfaction score, but one must exercise caution: “A good measure to determine the success of an AI tool being implemented in a process can be a satisfaction score, that is, we survey the satisfaction of the people who start using the tool. However, it is also a dangerous measure because it depends greatly on the people surveyed and the sample size, so the results can be distorted by various confounding factors” (Interview A8).

Other respondents (A1, A3, A5) presented the measure in the context of the feasibility and business impact of the implemented solution: “Working as a consultant for many companies on data and AI solutions, success was achieved by those companies that, in the process for the implemented solution, were able to answer well in advance the question of how important and executable the solution is. To assess feasibility, one has to answer the questions of whether there is data available, whether this data is representative, whether it is technologically difficult or easy to engineer, whether there is knowledge available. To assess business impact, you often need a measure and, depending on the solution being introduced, you can conduct a market or competitor analysis here to determine what that measure might be” (Interview A5). Another significant point raised by respondents, referring to feasibility, is to consider whether AI is necessary in the process, even when data are available: “Once, we had a project where we aimed to improve the menu of one of our devices. To do this, we gathered data from different devices to see which menu items customers were using and which they weren’t, then adjusted the menu so that all the most frequently used items were the most visible. This is data-driven product development or optimization, and we considered using AI, but found it was quicker to do it without it” (Interview A1). Another interviewee added: “We had one product where we knew from the beginning that we wanted to use AI solutions, specifically Computer Vision solutions, and this was not even subject to consideration, because the alternative was to hire numerous people who would have to write the algorithm for a given production and label the data, and the work on this project would end after a few months and all these people would have to be fired. Of course, we conducted a cost analysis in both cases anyway, but I want to emphasize that sometimes it is just evident that using AI solutions is far superior” (Interview A3).

Several respondents (A2, A3, A5, A7, A12) identify the quality of the team as one of the key measures of success: “The primary link between all the successful projects I was involved in as a product manager was the quality of the team. I am referring to individuals from various departments—sales, supply, approvals, etc.” (Interview A2). Another interviewee emphasizes the importance of collaboration between departments: “Once, we aimed to improve data analytics related to our products in the product manager department to expedite certain functionalities. It turned out that to achieve this, we needed data from both sales and procurement to obtain information on the customers from whom we purchased products—often, these customers were the same ones who bought products from us.

It may seem astonishing, but these departments had never collaborated before, and it was only through the commonality of these tables that we could discern the full relationship with some companies. That kind of cross-departmental collaboration was crucial, and we, as product managers, have this helicopter view to initiate something like that” (Interview A12). Both above respondents (A2, A12) also indicated that having a single data dictionary in the organization was an important factor.