Migrating 200 millions users to a cloud based identity system {Okta}. Part I

How to migrate a legacy Customer Identity and Access Management(CIAM) system with a huge data set (200 millions users) to a cloud based CIAM system like Okta a cloud based modern OpenID Connect (OIDC).

Requirements

Migrate 200 millions users from an on premise legacy CIAM to Okta cloud.
During and post migration the data must be 100% in sync in real-time and 100% backward compatible.
Migration must be completed within a month period on a live system without user experience interruption.
Okta migration must be executed via web API calls.

Challenges

Migrating 200 millions users without hiccups !!!!
HTTP API migration is limited by API speed of let’s say 10,000 calls per minute which translates into 13.89 days assuming zero issues.
Users data integrity must be maintained at all time while synchronizing the migration and updates between the current system and Okta
Multiple choke points might cause migration issues such as:

Database access issues
Network issues
Okta access issues
Cluster issues

Migration Time Calculator
API Call Rate	Unit	Data Size	Total Time	Unit
10,000	Per/Minute	200,000,000	20,000	Minutes
			13.89	Days
			1.98	Weeks

Solution

In order to migrate all users to Okta while meeting the requirements and mitigating the challenges a dual strategy solution is implemented and executed simultaneously using Just In Time (JIT) migration and Bulk/Batched migration. The JIT migration will migrate users while using the system in real and the bulk migration will X set of users as background process or thread.

The migration process or user registration to Okta requires at least 3 mandatory fields

Login: the username.
Primary email: for password recovery and MFA
Password: as credentials other type of credentials could be used but for the sake of simplicity let’s assume password for the time being
UUID: this field is optional but nice to have as a user domain agnostic opaque identifier to correlate between Okta and external systems or vice versa.

JIT migration

JIT migration is often called lazy migration or on demand migration, it is quite simple and straightforward. The migration is triggered when the user becomes active within the system. The user activity is detected when the user login for the first time after enabling JIT or register for the first time.

Bulk/Batch migration

Bulk migration is a background process that extracts X number of users from the legacy system and migrates them to Okta domain via API calls. The process repeats itself by extracting X number of users again and again unttill all the users are migrated to Okta.