Klarna: Accelerating credit decisioning with real-time data processing (FSI319)

Key Takeaways

• Clara is a fintech company from Sweden that provides payment solutions and products to enhance the customer shopping experience.
• Credit underwriting is crucial for Clara as a payment service provider, involving internal data, external data from credit bureaus, and transaction information.
• Internal customer features, which are derived from aggregated internal data, are important for accurate and real-time risk assessments.
• The previous system had several issues:
  • Reliance on internal database dumps as data sources without a strict contract
  • Costly and non-scalable daily batch processing of all historical data
  • Significant differences between the real-time and batch processing layers
  • Lengthy iteration cycles for experimenting with new features
• The new solution addresses these issues by:
  • Using event streams with defined schemas as data sources, enabling a strict contract between producers and consumers
  • Separating business logic from the runtime architecture in a shared library
  • Implementing real-time state updates and offline feature calculations
  • Enabling retrospective feature calculations on historical data and efficient state backfilling

Real-time Processing

1. Update State:

   • Events from data sources are consumed and processed by the Update State application.
   • The application fetches the current state from DynamoDB, applies the event using the business logic library, and writes the updated state back to DynamoDB.
   • The state is stored in a domain-specific data model, with the order ID as the partition key and the customer ID as a global secondary index.

2. Feature Calculation:

   • The Underwriting Policy requests a set of features for a specific customer.
   • The Feature Calculation service queries the DynamoDB state table, fetches the relevant states, and applies the feature definitions from the business logic library to compute the feature values.
   • The feature values are then returned to the Underwriting Policy.

Offline Processing

1. Retrospective Calculation:

   • A data set with customer identifiers and decision times is provided as input.
   • The Update State Glue job processes the historical events, creates states based on the business logic, and outputs the states to S3.
   • The Feature Calculation Glue job then applies the feature definitions to the states and outputs the feature values to S3.
   • This enables faster feature engineering iterations and analysis on historical data, without waiting for data to mature.

2. State Backfill:

   • The full history of events is processed by a Glue job, creating states and formatting them for DynamoDB import.
   • The new DynamoDB table is then used by the real-time Update State application, ensuring a smooth transition and zero downtime.

Challenges and Lessons Learned

1. Data Modeling: The data modeling problem is domain-specific and complex, requiring trade-offs between read performance and flexibility.
2. Offline Processing: Running Python UDFs on Spark can be inefficient, and the team is exploring ways to improve this.
3. Data Quality: The quality of features is only as good as the source data, emphasizing the importance of data validation, monitoring, and tooling to debug data issues.

Overall, the new solution addresses the key pain points of the previous system, enabling faster feature engineering iterations, cost-efficient and scalable data processing, and a more unified real-time and batch architecture.