Machine Learning and AI: A Model Risk Perspective

Drivers of Model Risk in the age of data science and AI
Machine Learning vs Traditional quant models: How has the world changed?
A tour of Machine Learning and AI methods
Supervised vs Unsupervised Learning (Regression, Neural Networks, XGBoost, PCA, Clustering)
Deep Learning & Reinforcement Learning (Keras, Tensorflow, PyTorch)
Automatic Machine Learning  & Machine Learning APIs (Google,
Comprehend, Watson)
ML on the cloud vs On-prem

Models redefined: Data, Modeling environment, Modeling tools, Modeling process

Model Risk Management for Machine Learning Models - Part 1

-ML Life cycle management
-Tracking
-Metadata management
-Scaling
-Reproducibility
-Interpretability
-Testing
-Measurement

The Decalogue: Ten key aspects to factor when developing your model risk management framework when integrating Machine Learning models:
1. Models redefined: It’s not just input, process and output
2. Governing the Machine Learning process
3. Model Verification and Validation for Machine Learning Models
4. Performance Metrics and Evaluation criteria
5. Model Inventory and tracking

Model Risk Management for Machine Learning Models - Part 2

The Decalogue: Ten things to think about when developing your model governance framework when integrating Machine Learning models (cont’d):
6. Integrating Data Governance and Model Governance
7. Development Models vs Production Models
8. Fairness, Reproducibility, Auditability, Explainability, Interpretability & Bias
a. How do we objectively measure these?
b. Review of the Apple-Goldman Sachs credit card debacle
9. Machine Learning options and considerations
a. AutoML (Data Robot, H20.ai, etc.), ML as a service (Google, Comprehend, Watson) and home-cooked custom models
10. ML and Governance: Roles and Responsibilities redefined

Managing models in the day of Covid19

- Perspectives on point-forecasts, validation and fat-tails!

Pragmatic Model Risk Management for AI/ML models

Challenges and best practices for pragmatic model management within the enterprise
Working with open source projects
Working with vendor models and machine learning APIs
Quantifying model risk for machine learning models
Model risk management for deep-learning models
Validation criteria and best practices
Templates for Model Validation for machine learning models

Synthetic data for Model Risk Management
Use of Synthetic datasets

Hands-on Case study

Validating a Credit-risk machine learning model

A case study illustrating a model validation of a credit risk model involving machine learning
Working with Regression, Neural Networks, and Random Forest models
Development models vs Production models
Sample templates and worksheets will be provided
Roadmap for the MRM team to upskill and keep abreast of changes in the AI and ML landscape
Training, education, and expectation setting
Future outlook: Regulation, Sandboxes, Frameworks
Review of recent regulatory efforts
How should companies proactively plan for changes and the future?

Guided Exercise, Part 1: Scoping and design

Put your newly learned skills to practice while being mentored through the process. Participants will go through a guided exercise to perform model validation on a machine learning model of their choice. Guidance will be provided in scoping and implementing the project.

Guided Exercise, part 2: Demonstrate your skills

• Participants will have the opportunity to share their findings