How to manage generative AI proof of concept projects
Markku Räsänen
|
November 27, 2023
Generative AI is going to transform many tasks and processes in organizations because it increases efficiency. That is why most organizations are looking at specific ways they can utilize generative AI and specifically large language models (LLM) in their business. These business-specific use cases are often the ones that produce the most value. These cases interact with the business data of the organization and require specific care in terms of managing the proof-of-concept project (POC). In this blog post we will go through some aspects of a successful LLM POC project.
Each gen AI proof of concept (POC) project should have a clear objective. Usually, these objectives come from your internal business goals. If you are part of a team facilitating a POC for another team, it makes sense to also check that their business goals are fulfilled. For example, if the goal is to test how LLMs can help in the debugging process for industrial equipment, it makes sense to clarify what constitutes success.
It is good practice to not aim for perfection in the first phases of trying out generative AI. For example, if an internal document review process currently takes around 4 hours. So, companies aim to automate the whole process. Instead, a good scope of a POC would be to automate 80-90% of the work with LLMs while still leaving humans to review and verify the results. This also overcomes the current limitations that most LLM models have which, is that they are not 100% accurate and may hallucinate meaning that they may come up with information that is not correct or does not exist.
Selecting the right use case is critical. Also having some level of understanding for the limitations of the LLM technology is critical in choosing the right use cases. Another important aspect is thinking what type of business information will be exposed in the process. Is extremely critical information involved? This then warrants the use of special systems and safeguards for overall data safety.
It is good practice to collect 3-5 use cases from different business units, evaluate them and find whether there are synergies and similarities. It is also good to select impactful, but not the hardest to implement case as the first POC. This way, your organization will get an overall picture of what LLMs can do for your business, without overly complicating the POC.
Overall, good POC use cases are ones that are in your business areas that your team knows well. That way they can comment on the potential impact, and you can rank POCs based on their predicted impact.
Decide who needs to be involved. Usually, someone from the business team needs to be part of the team as well as someone who will technically implement the POC. Bigger organizations may also require that the data team is present. You should have clear roles and responsibilities.
It is good practice for a successful POC to make sure that there are stakeholders from each team involved. It is also crucial that there is commitment from the internal customer team to test and provide feedback during the POC testing period.
When you are implementing the POC, decide where it will be implemented. Will you use production data or just synthetic, or purposely created data for the POC? The overall implementation process can also be managed by using tools such as ConfidentialMind, which will help you deploy the right resources to any cloud you are using, manage the data sources as well as deploy the applications themselves. ConfidentialMind will also help you manage access to the POC application if it is important for you to guard access to it. It can help you manage your internal IAM roles in the application deployment process.
It is good practice to collect user feedback as early and as much as possible. As generative AI is a rather new genre of technology, don’t be afraid to test partly incomplete solutions to better meet the user requirements and see how the technology fits into your organization and your plans. A common pitfall of POC implementation is that not enough feedback is collected and as such it is not clear if the POC succeeded or not.
Once the POC is complete it is important to evaluate the results. What went right and what went wrong? You should evaluate the POC results against the KPIs you chose such as accuracy, fit for business purpose, costs efficiency, time saving and so on.
After your evaluation, it is time to proceed to production. Going into production with LLMs is a big step. The LLM technology is ever evolving, and models change all the time. For a successful production deployment, you should make sure that your application architecture is such that you can change the underlying models as needed or as they develop. Tools such as ConfidentialMind can help in this process.
ConfidentialMind allows you to control all the aspects of your deployment such as changing the underlying LLM for the deployed application with a push of a button. This is helpful as it might also be that you would like to use a different kind of LLM for production due to security or cost concerns.
If you would like to know how ConfidentialMind can help you manage your LLM POCs and deployments, don’t hesitate to contact us at info@confidentialmind.com.
