Shipping AI, Fast and Safe
Introduction
- The presentation is about how companies can take AI concepts and get them from an idea to something that is actually in production and working, useful.
- Tom Totenberg from LaunchDarkly and Edmund Lam from Poka will share their experiences.
Poka's Connected Worker Platform
- Poka is a connected worker platform for manufacturers, helping factory workers excel at their jobs by providing the right tools and information in a single integrated platform.
- Poka has its app running in over 1,500 factories around the world in 70 countries, used by large brands like Nestlé, Bosch, Mars, and L'Oreal.
- Poka's AI R&D/Enablement team is structured with two roles:
- A complicated subsystem team to gather expertise on using AI properly and set standards.
- An enablement team to share knowledge and work with product/design teams to prototype ideas quickly.
Poka's PDF to Work Instruction Converter
- One of the biggest challenges for Poka's clients is creating or importing work instructions, often in PDF format.
- Poka's vision was to allow clients to drag and drop PDFs, hit a "generate" button, and have work instructions in Poka's native format, ready to review.
- This would be faster, extract images from PDFs, and set the stage for other AI features like retrieval-augmented generation.
Development Process
- Prototyping: Tested if generative AI models could understand complicated PDF work instructions.
- Proof of Concept: Created a simple UI to upload PDFs, generate work instructions, and get internal feedback to improve the conversion quality.
- Beta Testing: Cleaned up the UI, redid the architecture to use a serverless, asynchronous, and scalable content AI microservice.
Using Feature Flags
- Poka used Feature Flags from LaunchDarkly in several ways:
- Main feature flag to safely commit code to the main branch without exposing the feature.
- Ops flags for configuration like file size limits, concurrent file limits, etc.
- AI-specific ops flags to quickly switch between different AI models (Haiku, Sonnet, Opus) and adjust prompts.
Prompt Handling and Security
- Poka added a "prompt hacking detection" step to check if the PDF content contained anything adversarial.
- Initially, the AI returned a simple 0/1 flag, which was later expanded to return JSON with more details, including a reason for flagging content as adversarial.
- Poka also added examples to help the AI better detect adversarial prompts.
- When the AI flagged a "guillotine" PDF as violent, Poka worked around it by introducing a "violent" attribute that they could ignore, while still respecting the AI's guidelines.
Beta Testing and Results
- Poka's beta involved 20 clients and 50 users, who performed over 1,000 conversions.
- Users reported a 75% reduction in time to import work instructions, from over an hour to around 15 minutes.
- One client was able to set up a new production line in 3 months instead of 5 months, and estimated 2 months for the next expansion, thanks to the AI feature.
Lessons Learned
- The small, dedicated AI team structure worked well, providing expertise to the main feature teams as needed.
- Using Amazon Bedrock and LaunchDarkly's Feature Flags enabled quick iteration and testing of different AI models.
- Gathering customer feedback was crucial to confirm the value of the AI feature.
LaunchDarkly's Role
- LaunchDarkly enables the separation of code deployment and feature release, allowing Poka to iterate quickly on AI models and prompts without being blocked by the deployment process.
- LaunchDarkly provides features to manage AI-specific configurations, target specific audiences for testing, and collect telemetry and feedback to measure the performance and impact of the AI feature.
Q&A
The presentation concludes with a Q&A session, where the audience can ask questions to Edmund and Tom.