Architecting a Modern Financial Institution SOUTHEAST BRAZIL REGION - PowerPoint PPT Presentation

Architecting a Modern Financial Institution SOUTHEAST BRAZIL REGION FROM SPACE

CREDIT CARD September 2014

GROWING QUICKLY IN A COMPLEX DOMAIN # of clients (M) 10.5 M 20 Credit Card 3,0 Unique applications Deploys per day 2.6 M 120 2,3 Customers Microservices 1,5 262 M 105 Purchases Engineers 0,8 198 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17 Countries

IMMUTABLE THEMES FROM OUR STACK LISP hosted on the JVM CLOJURE Functional (opinionated), immutable data structures DATOMIC Simple, concise, fast, concurrent Tight REPL feedback cycles KAFKA Gradual typing (schemas) CLOUD

IMMUTABLE THEMES FROM OUR STACK CLOJURE for your data Accumulate-only DATOMIC Reified ACID transactions, preserve what changed when KAFKA Query using data structures (Datalog) Cloud native with integrated caching and scalable reads CLOUD

IMMUTABLE THEMES FROM OUR STACK Immutable, persistent, partitioned log CLOJURE Logical decoupling between services Temporal decoupling, useful for DATOMIC asymmetric workloads Fault isolation and recovery (circuit breakers, dead letters) KAFKA Financial batch jobs expressed as a streams of messages CLOUD

IMMUTABLE THEMES FROM OUR STACK Infra as code (AWS) CLOJURE Immutable upon provisioning (Docker) Blue-Green deploys at service and DATOMIC company level Kubernetes for speed and scalability KAFKA CLOUD

Nubank HQ São Paulo, Brazil FUNCTIONAL BENEFITS HIRING POSITIVE SELF SELECTION 1-MONTH RAMP COMPLEXITY SMALL, PURE FUNCTIONS STRAIGHTFORWARD TO UNTANGLE CONSISTENCY COMPOSING A SMALL NUMBER OF IDIOMATIC LANGUAGE FEATURES

CREDIT CARD ARCHITECTURE Greenfield MVP Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Backoffice General Ledger Credit Limits (CRM) Collections Securitization Phone + Chat ETL

BANK ACCOUNT October 2017 13

CORE BANKING + CREDIT CARD ARCHITECTURE Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE

PURCHASE AUTHORIZATION Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE

PURCHASE AUTHORIZATION VALUE CHAIN ACQUIRER NETWORK MERCHANT CUSTOMER ISSUER

ISSUER AUTHORIZATION NETWORK ISSUER

ISSUER AUTHORIZATION MASTERCARD INTERFACE DEVICE Establish a connection 1 Receive authorization requests 2 AUTHORIZER

ISSUER AUTHORIZATION: ISO-8583 MASTERCARD INTERFACE DEVICE ISO-8583 Binary Message AUTHORIZER HARDWARE SECURITY MODULE

SCODEC BINARY PARSER FOR ISO-8583 object PANMappingFileD { import scala.language.reflectiveCalls val codec: Codec[SE33Subfield] = discriminated[SE33Subfield].by(intPadded(2)) .typecase(1, llvar(str).as[AccountNumberIndicator]) .typecase(2, llvar(intString(intPadded(2))).as[AccountNumber]) .typecase(3, llvar(yearMonth).as[ExpirationDate]) .typecase(4, llvar(str).as[ProductCode]) .typecase(5, llvar(intPadded(2)).as[TokenAssuranceLevel]) .typecase(6, llvar(intString(intPadded(2))).as[TokenRequestorID]) .typecase(7, llvar(intString(intPadded(2))).as[PANAccountRange]) }

ISSUER AUTHORIZATION: REQUIREMENTS BRAND INTERFACE DEVICE ISO-8583 Binary Message AUTHORIZER 1.Highly Available 
 2.Physical Infrastructure HARDWARE SECURITY MODULE (HSM)

AUTHORIZER SERVICE LAYOUT • Small set of highly available services ISO 8583 • Co-located with the MasterCard Finagle Client devices in the same datacenters router router Thrift • Isolated: transaction authorization hot path does not Finagle Server need communication with the authorizer authorizer authorizer authorizer cloud • Active-active disaster recovery crypto crypto fraud fraud (not shown) Proprietary protocol HSM HSM

KAFKA AS THE BRIDGE BETWEEN ENVIRONMENTS 100+ microservices kafka “neverland” “the real world” (nubank datacenter) (AWS VPC)

KAFKA-BASED LOG/SNAPSHOT Publish 1 Authorizer consumes 2 AWS Service 0 1 2 3 4 … Kafka Topic Partition Snapshot 3 service New consumes 5 authorizer started snapshotter Consumes from 7 snapshot offset Generates a 4 snapshot Reads offset 6 Snapshot 2

DRAMATIC IMPROVEMENTS IN RELIABILITY AND FRAUD % stand-in cutover cutover 80 % 1,4 % % fraud capture 60 % 1,05 % 40 % 0,7 % 20 % 0,35 % % fraud precision 0 % 0 % 2017-03 2017-04 2017-05 2017-06 2017-07 2017-08 2017-09 2016-07 2016-09 2016-11 2017-01 2017-03 2017-05 2017-07

DOUBLE ENTRY ACCOUNTING Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE

BUSINESS LOGIC DEPENDS ON DATA ACROSS MANY SERVICES Purchases Payments Chargebacks Interest Currencies Should we… authorize a purchase? Double Entry block a card? charge interest?

DOUBLE ENTRY: THE MODEL CREDIT ENTRY BOOK ACCOUNT $ DEBIT BOOK ACCOUNT = 𝚻 BALANCE $ The sum of all credits and debits for one book-account is its balance A customer’s balance sheet is a cumulative function of their entire history

DOUBLE ENTRY: THE RULEBOOK CREDIT ENTRY BOOK ACCOUNT $ DEBIT BOOK ACCOUNT CREDIT ENTRY 2 BOOK ACCOUNT NEW-PURCHASE $ NEW-PAYMENT DEBIT … BOOK ACCOUNT CREDIT ENTRY 3 BOOK ACCOUNT $ DEBIT BOOK ACCOUNT MOVEMENT

DOUBLE ENTRY: EXAMPLE MOVEMENT ( def unsettled-purchase [ { :entry/debit-account :book-account-type.asset/unsettled :entry/credit-account :book-account-type.liability/unsettled-counterparty :entry/amount #'transaction-amount :entry/post-date #'produced-date} { :entry/debit-account :book-account-type.liability/current-limit-counterparty :entry/credit-account :book-account-type.asset/current-limit :entry/amount #'transaction-amount :entry/post-date #’produced-date} ])

DOUBLE ENTRY: CHALLENGES ordering matters (i.e. movements are not commutative) 
 late arriving events (e.g. a payment was made 3 days ago ) 
 fixing invariants write throughput

DOUBLE ENTRY: GENERATIVE TESTING OF INVARIANT ( def loss-property ( prop/for-all [adjs ( gen/vector ( gen/one-of [gen-adjustment gen-payment gen-tx]) 1 10) initial-state ( gen/such-that ( comp not #{ :late :pre-loss } :state ) rbh/initial-state-gen) loss-event ( gen/tuple ( gen/no-shrink ( gen/elements #{ :pre-loss :credit-loss :id-fraud-loss :fraudster })) ( tg/make-generator LocalDateTime) ( tg/make-generator LocalDate))] ( check-properties adjs initial-state loss-event)))

DOUBLE ENTRY: CHALLENGES ordering actually matters (i.e. movements are not commutative) 
 late arriving events (e.g. a payment was made 3 days ago ) 
 fixing invariants write throughput

SHARDED, FAULT TOLERANT INFRASTRUCTURE Customer Installment Acquisition Credit Scoring Logistics FX Chargeback Infosec Purchases (KYC) Card Rewards + Marketing Authorizer Notification Anti-fraud Billing Bill Pay Origination Merchants Lending + Realtime Investment Backoffice General Ledger Credit Limits Interest Transfers Management (CRM) Rates Tax Treasury + Risk Collections Securitization Phone + Chat ETL INFRASTRUCTURE

SCALING BOTTLENECKS # of clients (M) Credit Card 3,0 1. database throughput limits required throttling writes 2,3 2. batch job latency impacting customer experience 
 1,5 0,8 Set-14 Jan-15 Mai-15 Set-15 Jan-16 Mai-16 Set-16 Jan-17 Mai-17 Set-17

SCALING PLAN Need to partition the workload Customer data is spread across services Interactions between customers are minimal Safe to partition the user base

OPTION #1: PARTITION SERVICE DATABASES Database writes were the worst bottleneck Option: horizontally partition each database db shard s0 db shard s1 db shard s2 Change every service to route queries and writes to the appropriate shard backend service

OPTION #1: PROBLEMS Enormous effort to change every service Doesn’t address non-db bottlenecks Risks intermingling data infrastructure code with business logic

OPTION #2: SCALABILITY UNITS SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 3 SERVICE 3 SERVICE 3 . . . shard S0 shard S1 shard s2

OPTION #2: SCALABILITY UNITS + GLOBAL ROUTING global purchase deposit SERVICE 4 SERVICE 5 SERVICE 6 SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 1 SERVICE 2 SERVICE 3 SERVICE 3 SERVICE 3 shard S0 shard S1 shard s2