User Tools

Site Tools


wiki:ghc:ghc18:se688

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

wiki:ghc:ghc18:se688 [2018/10/21 20:56] (current)
shivangibansal created
Line 1: Line 1:
 +====== Changing the Tires at 60 MPH: A User Migration Story ======
 +
 +===== Presenters =====
 +Belian Wang
 +Staff Engineer, HBO
 +
 +
 +===== Abstract =====
 +
 +When HBO NOW launched, it was hoisted by a third-party vendor. Since then, HBO has built its own streaming platform called Hurley. Recently, the entire NOW streaming service was migrated to the new platform. Keeping the service running during the migration presented a number of challenges; this talk describes the architecture we built to address those challenges and maintain a seamless user experience.
 +
 +====== Session Notes ======
 +
 +==== Background: ====
 +
 +=== What does user migration mean? ===
 +
 +User migration is not moving – pack data and ready to go. It involves minimizing the impact of migration on both existing and new users.
 +
 +  * Store data in the cloud
 +  * Flip a switch
 +  * Read data from the cloud
 +
 +
 +==== Challenges: ====
 +
 +  * Live Use
 +  * Race condition
 +  * Load contention
 +  * O (J * K * L) Complexity
 +  * No access to live data
 +
 +
 +==== Pre-migration for HBO Now ====
 +
 +  * HBO offers 2 services - HBO NOW and HBO GO. HBO GO already uses the new HBO stack but HBO NOW depended on third-party infrastructure.
 +  * Users interact with HBO NOW.
 +  * HBO NOW calls external service.
 +  * External service then stores data in its data store.
 +
 +==== Migration Steps ====
 +
 +  * A new version of HBO client was developed and released to the app store. In order to not break existing users, an intermediate state was maintained where both new and old clients worked as expected.
 +  * To ensure this backwards compatibility,​ a proxy service called SHIM was built.
 +  * Data migration from external service’s data store to HBO’s data store was performed.
 +
 +
 +==== Post-migration for HBO Now ====
 +
 +All requests from old client are served using the existing way (stored in old external service data store). But all requests from new client go to HBO Service which then calls this HBO Proxy service called SHIM. SHIM interacts with external third-party service which stores the user data. Thus, both old and new HBO client works for the users.
 +
 +
 +==== Build SHIM with Dual Write ====
 +
 +SHIM is backed by a temporary data store. It logs/ persists the data before sending it to the external service.
 +
 +
 +==== On Demand Migration ====
 +
 +__Data Inconsistency Problem__: There is a possible scenario where while we are migrating the user data to new HBO data store, user updates his or her data. While data is still being persisted by SHIM service, we migrate the old data to HBO service. Thus, we have stale user data in HBO store.
 +
 +{{ :​wiki:​ghc:​ghc18:​datamigrationhbo.png?​600 |}}
 +
 +
 +This problem is solved by on-demand migration. Thus, whenever a user tries to update his/her data, he/ she waits for a brief moment while HBO migrates the user data from old infrastructure to new infrastructure and then serves the request. Since, time taken to migrate a single user is negligible, impact on overall latency is also negligible.
 +
 +==== One by One Migration ====
 +
 +HBO migrated 1 user at a time from old infrastructure to new infrastructure. This was done in order to reduce:
 +  * total migration time;
 +  * risk involved (in terms of impacting multiple users in case of unexpected scenarios)
 +
 +
 +==== Dry Run ====
 +
 +  * Understand load on production
 +  * Improve performance
 +  * Debug data issues
 +  * Test use cases
 +
 +==== Questions ====
 +
 +  * Q. Was there a rollback strategy in case of failures?
 +  * A. Yes, HBO had a plan to rollback to old infrastructure in case something unexpected happened during migration.
 +
 +  * Q. How long did the migration take?
 +  * A. The entire migration project took close to 1 year.
 +
 +  * Q. What data store is used by HBO?
 +  * A. HBO uses AWS DynamoDB.
  
wiki/ghc/ghc18/se688.txt · Last modified: 2018/10/21 20:56 by shivangibansal