Strict validation versus accepting anything Evan Jones Bluecore - PowerPoint PPT Presentation

Aug 19, 2023 •147 likes •505 views

Strict validation versus accepting anything Evan Jones Bluecore Personalized e-commerce marketing ~4 years ~140 employees ~35 engineers Recommendations need product data 2. Page loads JS Partner-specific JS 3. User action sent Data

Strict validation versus accepting anything Evan Jones
Bluecore Personalized e-commerce marketing ~4 years ~140 employees ~35 engineers
Recommendations need product data
2. Page loads JS Partner-specific JS 3. User action sent Data Ingestion Rules, 1. User visits site Email DB Recommendations 4. Find customers 5. Send email
Data ingestion Web Process Customer events Queue Database Handler Data (thousands/second)
Data ingestion Web Process Customer events Queue Database Handler Data (thousands/second)
Product data "id": 429174, "name": "Pilot G2 Premium Retractable...", "price": 13.99,
Product data "id": 429174 , "name": "Pilot G2 Premium Retractable...", "price": 13.99,
Product data "id": "429174", // may contain letters "name": "Pilot G2 Premium Retractable...", "price": 13.99,
Product data "id": "429174", // may contain letters "name": "Pilot G2 Premium Retractable...", "price": 13.99 ,
Product data "id": "429174", // may contain letters "name": "Pilot G2 Premium Retractable...", "price": "£13.99, // may contain currency
What is valid product data?
Design A : Accept anything! Design B : Strict validation!
Design choice: Validation versus flexibility Programming languages: static versus dynamic typing Databases: Strict versus flexible schemas (SQL vs NoSQL)
Robustness Principle (Postel’s Law) “Be liberal in what you accept, and conservative in what you send” Advantage: implementations can interoperate (e.g. TCP) Disadvantage: bugs can become “standard” (e.g HTML)
Original policy: Accept anything Rationale: One chance to store the data; fix it later Implementation: Store any key/value pairs
Fun ensues ... price: 13 (integer), 13.99 (float), “13.99”, “£13.99” products without ids products with both “title” and “name”
Evaluation + Store any e-commerce data + Fix any data bugs
Evaluation + Store any e-commerce data + Fix any data bugs - Processing is much harder - Harder to test if we are sending the right data
Core Validation Raw data System
Valid Core Validation Raw data System Everything
Valid Core Validation Raw data System Everything One-off fix
Conclusion: Err on the side of validation Find errors sooner Simplifies the overall system Easier to relax restrictions than to add them Want to fix errors later? Record everything
Thanks! Evan Jones http://www.evanjones.ca/ Bluecore http://www.bluecore.com/
Store raw but require “core” schema Store the raw data we receive Validate “core” fields: return helpful error messages e.g. must have id, price is a string, use “name” not “title” Found many data bugs

Recommend

Threshold Accepting for Credit Risk Assessment and Validation M. Lyra 1 A. Onwunta P . Winker

Threshold Accepting for Credit Risk Assessment and Validation M. Lyra 1 A. Onwunta P . Winker COMPSTAT 2010 August 24, 2010 1 Financial support from the EU Commission through COMISEF is logo gratefully acknowledged Introduction Ex-post

607 views • 43 slides

Arf closure versus strict closure Naoki Endo Purdue University based on the works jointly with

1. Introduction 2. Proof of Zariskis conjecture 3. Strictly closed rings Arf closure versus strict closure Naoki Endo Purdue University based on the works jointly with E. Celikbas, O. Celikbas, C. Ciuperc a, S. Goto, R. Isobe, and N.

492 views • 18 slides

The Monad of Strict Computation A Categorical Framework for the Semantics of Languages in which

The Monad of Strict Computation A Categorical Framework for the Semantics of Languages in which Strict and Non-strict computation rules are mixed Dick Kieburtz Portland State University WG2.8 Meeting July 16-20, 2007 The Problem, illustrated

507 views • 16 slides

6 Foot Kitchen Training Accepting Food Deliveries May 2020 Accepting Deliveries (20 minutes)

6 Foot Kitchen Training Accepting Food Deliveries May 2020 Accepting Deliveries (20 minutes) LEARNING OUTCOMES: After this topic, learners will have an understanding of: Module I: 15 minutes How to interact with delivery personnel while

337 views • 8 slides

Satellite-based solar resource data: Model validation statistics versus users uncertainty

Satellite-based solar resource data: Model validation statistics versus users uncertainty Marcel Suri and Tomas Cebecauer GeoModel Solar, Slovakia geomodelsolar.eu SOLAR 2014 Conference, San Francisco, 7-9 July 2014 SOLAR 2014 Conference,

586 views • 22 slides

SHOULD STRICT CRIMINAL LIABILITY BE REMOVED FROM ALL IMPRISONABLE OFFENCES? ANDREW ASHW ORTH,

SHOULD STRICT CRIMINAL LIABILITY BE REMOVED FROM ALL IMPRISONABLE OFFENCES? ANDREW ASHW ORTH, THE J.M. K ELLY LECTURE 20 10 What is strict crim inal liability? An offence should be treated as a crime of strict liability if it

309 views • 14 slides

Form Validation 1 CS380 What is form validation? 2 validation: ensuring that form's values

Form Validation 1 CS380 What is form validation? 2 validation: ensuring that form's values are correct some types of validation: preventing blank values (email address) ensuring the type of values integer, real number,

284 views • 24 slides

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides

Chapter 5 Analysis: Four Level for Validation Vis/Visual Analytics, Chap 5 Validation 1 CGGM

Chapter 5 Analysis: Four Level for Validation Vis/Visual Analytics, Chap 5 Validation 1 CGGM Lab., CS Dept., NCTU Jung Hong Chuang Contents Why Validate? Validation Approaches Domain Why is Validation Abstraction

724 views • 47 slides

Capital Quality Validation Webinar Sept. 17, 2020 Agenda Validation Overview

Capital Quality Validation Webinar Sept. 17, 2020 Agenda Validation Overview Validation Timeline Authorized Representative Validation Process How to Log In to QuickBase How to Reset Your Password How to

451 views • 43 slides

Segal-type models of weak n -categories Simona Paoli Department of Mathematics University of

Segal-type models of weak n -categories Simona Paoli Department of Mathematics University of Leicester CT2019, University of Edinburgh Simona Paoli (University of Leicester) July 2019 1 / 54 Strict versus weak n -categories Idea of strict n

1.1k views • 54 slides

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides

AngularJS & Bootstrap Form Validation HTML default validation Browsers have built-in

AngularJS & Bootstrap Form Validation HTML default validation Browsers have built-in HTML validation These work when the type attribute is given to an input element We might want better validation than is provided by the web

328 views • 13 slides

Let M 1 and M 2 be the FAs pictured in Fig- Exercise 2.10. ure 2.44 (on the blackboard), accepting

Let M 1 and M 2 be the FAs pictured in Fig- Exercise 2.10. ure 2.44 (on the blackboard), accepting languages L 1 and L 2 , respectively. Draw FAs accepting the following languages. a. L 1 L 2 b. L 1 L 2 c. L 1 L 2 1 For each of the

158 views • 4 slides

Module 4 19/05/2015 2 Agenda 1. What is validation? 2. Three-part empathy 3. What is

VALIDATION SKILLS Module 4 19/05/2015 2 Agenda 1. What is validation? 2. Three-part empathy 3. What is involved? 4. Why, when and how to use validation 3 What is validation? Validation = saying something People with BPD are much

265 views • 8 slides

VALIDATION The goal of validation is to judge the quality of the soft- ware from the users

VALIDATION The goal of validation is to judge the quality of the soft- ware from the users point of view; e.g., reliability. The goal of all validation techniques is: to reveal failures, to localize the faults that caused the

281 views • 14 slides

Verification and Validation Steven Zeil February 13, 2013 Verification and Validation

Verification and Validation Verification and Validation Steven Zeil February 13, 2013 Verification and Validation Outline The Process 1 Non-Testing V&V 2 Code Review Mathematically-based verification Static analysis tools

790 views • 55 slides

LaGov LaGov Version 2.2 Updated: 12/17/08 Visit our website for Blueprint Presentations,

Inventory and Warehouse Management Inventory and Warehouse Management Inventory and Warehouse Management Validation Session Validation Session Validation Session LOG- -IM/WM IM/WM- -Validation Validation LOG LOG-IM/WM-Validation January

1.59k views • 146 slides

M -coextensivity and the strict refinement property Michael Hoefnagel University of

Introduction and history The strict refinement property A characterization of projection-coextensivity Concluding remarks M -coextensivity and the strict refinement property Michael Hoefnagel University of Stellenbosch, South Africa Michael

449 views • 32 slides

GUIDEL ELINES ES F FOR S R SPEAKERS ERS Thank you for accepting to present at our first

GUIDEL ELINES ES F FOR S R SPEAKERS ERS Thank you for accepting to present at our first Online Medical Education Summit (MedEd). To ensure your presentation runs smoothly, we have put together all important information in this document with

245 views • 5 slides

Bob Lee @crazybob 2 ,000,000 Dim Sum Accepting a payment Authorization 1 Capture 2

Bob Lee @crazybob 2 ,000,000 Dim Sum Accepting a payment Authorization 1 Capture 2 Persistent Queues ACID Atomicity Durability Filesystem Promises Renaming is atomic Fsync is durable* Segment/block writes Traditional

1k views • 45 slides

Workshop on process validation General concepts on process validation Kowid Ho Scope /

Workshop on process validation General concepts on process validation Kowid Ho Scope / background Process evaluation/validation of biotechnology derived proteins used as active substance in the manufacture of medicinal products (i.e.

346 views • 16 slides

SPLICING SYSTEMS ACCEPTING VS. GENERATING Juan Castellanos Victor Mitrana Eugenio Santos

SPLICING SYSTEMS ACCEPTING VS. GENERATING Juan Castellanos Victor Mitrana Eugenio Santos Polytechnic University of Madrid OUTLINE Structure and recombination of DNA Splicing operation and splicing systems Generating variants Accepting

624 views • 27 slides

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides