Best Practices Summary:

• Data science and artificial intelligence (AI) can solve problems if it is plugged-in early on in product design and development. It’s important for data scientists to ask a lot of questions on the front end, especially about use cases and user personas. 

• Listen carefully to the pain points of customers. This means having both product managers and data scientists in meetings with customers.

• AI model recommendation rationale can be opaque so additional models with explanatory factors can enhance customer confidence in the model.

Case Study:

A large software company produced a workforce scheduling application for company workforces performing field maintenance activities across various industries. The software company developed AI models to optimize scheduling, however, both the product engineers and customers resisted adopting the new models in the software.  Product engineers were focused on their own product features and tight timelines and wanted to avoid associated risks with introducing advanced analytics. Customers were skeptical about model performance due to lack of understanding of underlying algorithms. 

The Data Science Director sought to address these challenges and drive adoption of the predictive analytics capabilities into the field scheduling product. The key to gaining buy-in with the product engineering team was for the data science team to get involved with the product discussions early on before development was in full swing. The team focused on gathering information with a particular focus on understanding the product user personas and use cases. Armed with this knowledge the data science team translated business needs into technical requirements in the form of a set of algorithms  The engagement helped build trust  between the product engineering team and the data science team. It also produced improved relevance of the analytics to customers.  For example, a cable TV provider could ensure they automatically dispatched a technician with the right skills to the right job when a job request came in, versus waiting for it to be manually scheduled. Further, the system would identify high priority jobs and match those jobs with technicians with high satisfaction scores in near real time. 

To help ensure that the algorithms met the needs of multiple customers and scenarios the Data Science Director employed methods that focused on strong engagement with both engineering and customers through transparency and communications. Similar to the early engagement with product engineering stakeholders, it was important for data scientists to hear the customer needs directly. Data scientists participated directly in customer sessions to gather use cases and pain points which could inform the recommendation model development. With this customer input in mind, the team pre-trained the models, often leveraging synthetic data, to get initial results. At this stage, the model most often gave low confidence scores on the recommendations. Over time, the confidence scores gradually increased as the models were continuously and automatically retrained with new, real-world data. Another obstacle to establishing customer confidence in the model was the lack of explanation of the drivers of the recommendations. understanding of the model features and driving feature importance values. To address this, additional models were included with visible explanatory factors along with the confidence scores to help to increase user trust. Data science model integration into the product could only be achieved with an iterative approach between the data science and product engineering teams.   However, the product engineering teams had very limited bandwidth to engage so the Data Science Director created a framework to facilitate, yet minimize, engagement with engineering through weekly touchpoints plus email and chat protocols. This engagement was important as user experience design and development, for example, was a necessarily a close partnership to ensure algorithm results were well integrated into the user interfaces. 

With enhanced engagement between the data science team with both product engineers and customers,  the adoption of the AI models in the product design was a success. Customers benefited from more efficient field scheduling and operations while the software company was able to bring a more competitive product to the market, helping drive revenue and market share.