Windows Azure as a Platform as a Service (PaaS) 17.7. 22.7. 2011 - PowerPoint PPT Presentation

Partition Keys In Each Abstraction Blobs – Container name + Blob name • Every blob and its snapshots are in a single partition Container Name Blob Name image annarbor/bighouse.jpg image foxborough/gillette.jpg video annarbor/bighouse.jpg Entities – TableName + PartitionKey • Entities w/ same PartitionKey value served from same partition PartitionKey (CustomerId) RowKey Name CreditCardNumber OrderTotal (RowKind) 1 Customer-John Smith John Smith xxxx-xxxx-xxxx-xxxx 1 Order – 1 $35.12 2 Customer-Bill Johnson Bill Johnson xxxx-xxxx-xxxx-xxxx 2 Order – 3 $10.00 Messages – Queue Name • All messages for a single queue belong to the same partition Queue Message jobs Message1 jobs Message2 workflow Message1

Scalability Targets Storage Account • Capacity – Up to 100 TBs • Transactions – Up to a few thousand requests per second • Bandwidth – Up to a few hundred megabytes per second Single Blob Partition • Throughput up to 60 MB/s Single Queue/Table Partition • Up to 500 transactions per second To go above these numbers, partition between multiple storage accounts and partitions When limit is hit, app will see ‘503 server busy’: applications should implement exponential backoff

Partitions and Partition Ranges PartitionKey RowKey Timestamp ReleaseDate PartitionKey RowKey Timestamp ReleaseDate (Category) (Title) (Category) (Title) Fast & Furious … 2009 Action Action Fast & Furious … 2009 The Bourne Ultimatum … 2007 Action Action The Bourne Ultimatum … 2007 … … … … … … … … Open Season 2 … 2009 Animation Animation Open Season 2 … 2009 Animation The Ant Bully … 2006 Animation The Ant Bully … 2006 PartitionKey RowKey Timestamp ReleaseDate … … … … (Category) (Title) Comedy Office Space … 1999 Comedy Office Space … 1999 … … … … … … … … X-Men Origins: Wolverine … 2009 SciFi SciFi X-Men Origins: Wolverine … 2009 … … … … … … … … Defiance … 2008 War War Defiance … 2008

Key Selection: Things to Consider • Distribute load as much as possible Scalability • Hot partitions can be load balanced • PartitionKey is critical for scalability • Avoid frequent large scans Query Efficiency & Speed • Parallelize queries • Point queries are most efficient • Transactions across a single partition Entity group transactions • Transaction semantics & Reduce round trips See http://www.microsoftpdc.com/2009/SVC09 and http://azurescope.cloudapp.net for more information

Expect Continuation Tokens – Seriously! Maximum of 1000 rows in a response Maximum of 1000 rows in a response At the end of partition range boundary At the end of partition range boundary Maximum of 5 seconds to execute the query

Tables Recap • Efficient for frequently used queries Select PartitionKey and RowKey • Supports batch transactions that help scale • Distributes load • Distribute by using a hash etc. as prefix Avoid “Append only” patterns Always Handle • Expect continuation tokens for range queries continuation tokens “OR” predicates are not optimized •Execute the queries that form the “OR” predicates as separate queries • Server busy Implement back-off • Load balance partitions to meet traffic needs strategy for retries • Load on single partition has exceeded the limits • Use a new context for each logical operation • AddObject/AttachTo can throw exception if entity is already being tracked WCF Data Services • Point query throws an exception if resource does not exist. Use IgnoreResourceNotFoundException

Queues Their Unique Role in Building Reliable, Scalable Applications • Want roles that work closely together, but are not bound together. • Tight coupling leads to brittleness • This can aid in scaling and performance • A queue can hold an unlimited number of messages • Messages must be serializable as XML • Limited to 8KB in size • Commonly use the work ticket pattern • Why not simply use a table?

Queue Terminology

Message Lifecycle RemoveMessage GetMessage (Timeout) HTTP/1.1 200 OK Transfer-Encoding: chunked Content-Type: application/xml PutMessage Date: Tue, 09 Dec 2008 21:04:30 GMT Worker Role Server: Nephos Queue Service Version 1.0 Microsoft-HTTPAPI/2.0 Msg 1  <?xml version="1.0" encoding="utf-8"?> Msg 2 Msg 2 Msg 1 <QueueMessagesList> Web Role POST http://myaccount.queue.core.windows.net/myqueue/messages <QueueMessage> <MessageId> 5974b586-0df3-4e2d-ad0c-18e3892bfca2 </MessageId> DELETE Msg 3 <InsertionTime> Mon, 22 Sep 2008 23:29:20 GMT </InsertionTime> http://myaccount.queue.core.windows.net/myqueue/messages/messageid?popreceipt=YzQ4Yzg1MDI <ExpirationTime> Mon, 29 Sep 2008 23:29:20 GMT </ExpirationTime> GM0MDFiZDAwYzEw Msg 4 <PopReceipt> YzQ4Yzg1MDIGM0MDFiZDAwYzEw </PopReceipt> Worker Role Worker Role <TimeNextVisible> Tue, 23 Sep 2008 05:29:20GMT </TimeNextVisible> <MessageText> PHRlc3Q+dG...dGVzdD4= </MessageText> </QueueMessage> Msg 2 </QueueMessagesList> Queue

Truncated Exponential Back Off Polling  Consider a backoff polling approach  Each empty poll increases interval by 2x  A successful sets the interval back to 1.

Removing Poison Messages Producers Consumers 1. GetMessage(Q, 30 s)  msg 1 C 1 P 2 4 3 2 2 2 2 1 1 1 1 3 0 0 0 1 1 1 0 1 1 1 C 2 P 1 2. GetMessage(Q, 30 s)  msg 2 44

Removing Poison Messages Producers Consumers 1 1. GetMessage(Q, 30 s)  msg 1 C 1 P 2 1 5. C 1 crashed 6. msg1 visible 30 s after Dequeue 4 3 2 1 1 1 1 3 0 0 1 2 2 1 1 2 C 2 1 P 1 2. GetMessage(Q, 30 s)  msg 2 3. C2 consumed msg 2 4. DeleteMessage(Q, msg 2) 7. GetMessage(Q, 30 s)  msg 1 45

Removing Poison Messages Producers Consumers 1. Dequeue(Q, 30 sec)  msg 1 5. C 1 crashed P 2 C 1 10. C1 restarted 11. Dequeue(Q, 30 sec)  msg 1 12. DequeueCount > 2 4 3 1 1 1 1 13. Delete (Q, msg1) 3 0 0 2 3 2 3 1 C 2 P 1 2 6. msg1 visible 30s after Dequeue 9. msg1 visible 30s after Dequeue 2. Dequeue(Q, 30 sec)  msg 2 3. C2 consumed msg 2 4. Delete(Q, msg 2) 7. Dequeue(Q, 30 sec)  msg 1 8. C2 crashed 46

Queues Recap Make message • No need to deal with failures • No need to deal with failures processing idempotent • Invisible messages result in out of order Do not rely on order • Invisible messages result in out of order Use Dequeue count to remove •Enforce threshold on message’s dequeue count •Enforce threshold on message’s dequeue count poison messages Use blob to store • Messages > 8KB message data with • Batch messages reference in message • Garbage collect orphaned blobs Use message count • Dynamically increase/reduce workers • Dynamically increase/reduce workers to scale

Windows Azure Storage Takeaways Blobs Drives Tables Queues http://blogs.msdn.com/windowsazurestorage/ http://azurescope.cloudapp.net

A Quick Exercise …Then let’s look at some code and some tools 49

Code – AccountInformation.cs public class AccountInformation { private static string storageKey = “ tHiSiSnOtMyKeY"; private static string accountName = "jjstore"; private static StorageCredentialsAccountAndKey credentials; internal static StorageCredentialsAccountAndKey Credentials { get { if (credentials == null) credentials = new StorageCredentialsAccountAndKey(accountName, storageKey); return credentials; } } } } 50

Code – BlobHelper.cs public class BlobHelper { private static string defaultContainerName = "school"; private CloudBlobClient client = null; private CloudBlobContainer container = null; private void InitContainer() { if (client == null) client = new CloudStorageAccount(AccountInformation.Credentials, false).CreateCloudBlobClient(); container = client.GetContainerReference(defaultContainerName); container.CreateIfNotExist(); BlobContainerPermissions permissions = container.GetPermissions(); permissions.PublicAccess = BlobContainerPublicAccessType.Container; container.SetPermissions(permissions); } } 51

Code – BlobHelper.cs public void WriteFileToBlob(string filePath) { if (client == null || container == null) InitContainer(); FileInfo file = new FileInfo(filePath); CloudBlob blob = container.GetBlobReference(file.Name); blob.Properties.ContentType = GetContentType(file.Extension); blob.UploadFile(file.FullName); // Or if you want to write a string replace the last line with: // blob.UploadText(someString); // And make sure you set the content type to the appropriate MIME type (e.g. “text/plain”) } 52

Code – BlobHelper.cs public string GetBlobText(string blobName) { if (client == null || container == null) InitContainer(); CloudBlob blob = container.GetBlobReference(blobName); try { return blob.DownloadText(); } catch (Exception) { // The blob probably does not exist or there is no connection available return null; } } 53

Application Code - Blobs private void SaveToCloudButton_Click(object sender, RoutedEventArgs e) { StringBuilder buff = new StringBuilder(); buff.AppendLine("LastName,FirstName,Email,Birthday,NativeLanguage,FavoriteIceCream,YearsInPhD,Graduated"); foreach (AttendeeEntity attendee in attendees) { buff.AppendLine(attendee.ToCsvString()); } blobHelper.WriteStringToBlob("SummerSchoolAttendees.txt", buff.ToString()); } The blob is now available at: http://<AccountName>.blob.core.windows.net/<ContainerName>/<BlobName> Or in this case: http://jjstore.blob.core.windows.net/school/SummerSchoolAttendees.txt 54

Code - TableEntities using Microsoft.WindowsAzure.StorageClient; public class AttendeeEntity : TableServiceEntity { public string FirstName { get; set; } public string LastName { get; set; } public string Email { get; set; } public DateTime Birthday { get; set; } public string FavoriteIceCream { get; set; } public int YearsInPhD { get; set; } public bool Graduated { get; set; } … } 55

Code - TableEntities public void UpdateFrom(AttendeeEntity other) { FirstName = other.FirstName; LastName = other.LastName; Email = other.Email; Birthday = other.Birthday; FavoriteIceCream = other.FavoriteIceCream; YearsInPhD = other.YearsInPhD; Graduated = other.Graduated; UpdateKeys(); } public void UpdateKeys() { PartitionKey = "SummerSchool"; RowKey = Email; } 56

Code – TableHelper.cs public class TableHelper { private CloudTableClient client = null; private TableServiceContext context = null; private Dictionary<string,AttendeeEntity> allAttendees = null; private string tableName = "Attendees"; private CloudTableClient Client { get { if (client == null) client = new CloudStorageAccount(AccountInformation.Credentials, false).CreateCloudTableClient(); return client; } } private TableServiceContext Context { get { if (context == null) context = Client.GetDataServiceContext(); return context; } } } 57

Code – TableHelper.cs private void ReadAllAttendees() { allAttendees = new Dictionary<string, AttendeeEntity>(); CloudTableQuery<AttendeeEntity> query = Context.CreateQuery<AttendeeEntity>(tableName).AsTableServiceQuery(); try { foreach (AttendeeEntity attendee in query) { allAttendees[attendee.Email] = attendee; } } catch (Exception) { // No entries in table - or other exception } } 58

Code – TableHelper.cs public void DeleteAttendee(string email) { if (allAttendees == null) ReadAllAttendees(); if (!allAttendees.ContainsKey(email)) return; AttendeeEntity attendee = allAttendees[email]; // Delete from the cloud table Context.DeleteObject(attendee); Context.SaveChanges(); // Delete from the memory cache allAttendees.Remove(email); } 59

Code – TableHelper.cs public AttendeeEntity GetAttendee(string email) { if (allAttendees == null) ReadAllAttendees(); if (allAttendees.ContainsKey(email)) return allAttendees[email]; return null; } Remember that this only works for tables (or queries on tables) that easily fit in memory This is one of many design patterns for working with tables 60

Pseudo Code – TableHelper.cs public void UpdateAttendees(List<AttendeeEntity> updatedAttendees) { foreach (AttendeeEntity attendee in updatedAttendees) { UpdateAttendee(attendee, false); } Context.SaveChanges(SaveChangesOptions.Batch); } public void UpdateAttendee(AttendeeEntity attendee) { UpdateAttendee(attendee, true); } private void UpdateAttendee(AttendeeEntity attendee, bool saveChanges) { if (allAttendees.ContainsKey(attendee.Email)) { AttendeeEntity existingAttendee = allAttendees[attendee.Email]; existingAttendee.UpdateFrom(attendee); Context.UpdateObject(existingAttendee); } else { Context.AddObject(tableName, attendee); } if (saveChanges) Context.SaveChanges(); } 61

Application Code – Cloud Tables private void SaveButton_Click(object sender, RoutedEventArgs e) { // Write to table tableHelper.UpdateAttendees(attendees); } That’s it! Now your tables are accessible using REST service calls or any cloud storage tool. 62

Tools – Fiddler2 63

Best Practices

Picking the Right VM Size • Having the correct VM size can make a big difference in costs • Fundamental choice – larger, fewer VMs vs. many smaller instances • If you scale better than linear across cores, larger VMs could save you money • Pretty rare to see linear scaling across 8 cores. • More instances may provide better uptime and reliability (more failures needed to take your service down) • Only real right answer – experiment with multiple sizes and instance counts in order to measure and find what is ideal for you

Using Your VM to the Maximum Remember: • 1 role instance == 1 VM running Windows. • 1 role instance != one specific task for your code • You’re paying for the entire VM so why not use it? • Common mistake – split up code into multiple roles, each not using up CPU. • Balance between using up CPU vs. having free capacity in times of need. • Multiple ways to use your CPU to the fullest

Exploiting Concurrency • Spin up additional processes, each with a specific task or as a unit of concurrency. • May not be ideal if number of active processes exceeds number of cores • Use multithreading aggressively • In networking code, correct usage of NT IO Completion Ports will let the kernel schedule the precise number of threads • In .NET 4, use the Task Parallel Library • Data parallelism • Task parallelism

Finding Good Code Neighbors • Typically code falls into one or more of these categories: Memory CPU Network IO Intensive Intensive Intensive Storage IO Intensive • Find code that is intensive with different resources to live together • Example: distributed network caches are typically network- and memory- intensive; they may be a good neighbor for storage IO-intensive code

Scaling Appropriately • Monitor your application and make sure you’re scaled appropriately (not over -scaled). • Spinning VMs up and down automatically is good at large scale. • Remember that VMs take a few minutes to come up and cost ~$3 a day (give or take) to keep running. • Being too aggressive in spinning down VMs can result in poor user experience. • Trade-off between risk of failure/poor user experience due to not having excess capacity and the costs of having idling VMs. Performance Cost

Storage Costs • Understand an application’s storage profile and how storage billing works • Make service choices based on your app profile • E.g. SQL Azure has a flat fee while Windows Azure Tables charges per transaction. • Service choice can make a big cost difference based on your app profile • Caching and compressing. They help a lot with storage costs.

Saving Bandwidth Costs Bandwidth costs are a huge part of any popular web app’s billing profile Saving bandwidth costs often lead to savings in other places Sending fewer things over the wire often means getting fewer things from storage All of these tips have the side benefit of improving your web app’s performance and user Sending fewer things means your VM has time to do other experience tasks

Compressing Content 1. Gzip all output content • All modern browsers can decompress on the fly. Uncompressed Content • Compared to Compress , Gzip has much better compression and freedom from patented algorithms 2.Tradeoff compute costs for storage size Gzip Minify JavaScript 3.Minimize image sizes Minify CCS Minify Images • Use Portable Network Graphics (PNGs) • Crush your PNGs • Strip needless metadata Compressed Content • Make all PNGs palette PNGs

Best Practices Summary Doing ‘less’ is the key to saving costs Measure everything Know your application profile in and out

Research Examples in the Cloud …on another set of slides

Map Reduce on Azure • Elastic MapReduce on Amazon Web Services has traditionally been the only option for Map Reduce jobs in the web • Hadoop implementation • Hadoop has a long history and has been improved for stability • Originally Designed for Cluster Systems • Microsoft Research this week is announcing a project code named Daytona for Map Reduce jobs on Azure • Designed from the start to use cloud primitives • Built-in fault tolerance • REST based interface for writing your own clients

Project Daytona - Map Reduce on Azure http://research.microsoft.com/en-us/projects/azure/daytona.aspx 76

Questions and Discussion… Thank you for hosting me at the Summer School 77

BLAST (Basic Local Alignment Search Tool) • The most important software in bioinformatics • Identify similarity between bio-sequences Computationally intensive • Large number of pairwise alignment operations • A BLAST running can take 700 ~ 1000 CPU hours • Sequence databases growing exponentially • GenBank doubled in size in about 15 months.

It is easy to parallelize BLAST • Segment the input • Segment processing (querying) is pleasingly parallel • Segment the database (e.g., mpiBLAST) • Needs special result reduction processing Large volume data • A normal Blast database can be as large as 10GB • 100 nodes means the peak storage bandwidth could reach to 1TB • The output of BLAST is usually 10-100x larger than the input

• Parallel BLAST engine on Azure • Query-segmentation data-parallel pattern • split the input sequences • query partitions in parallel • merge results together when done • Follows the general suggested application model • Web Role + Queue + Worker • With three special considerations • Batch job management • Task parallelism on an elastic Cloud Wei Lu, Jared Jackson, and Roger Barga, AzureBlast: A Case Study of Developing Science Applications on the Cloud, in Proceedings of the 1st Workshop on Scientific Cloud Computing (Science Cloud 2010) , Association for Computing Machinery, Inc., 21 June 2010

A simple Split/Join pattern Leverage multi-core of one instance • argument “– a ” of NCBI-BLAST BLAST task • 1,2,4,8 for small, middle, large, and extra large instance size BLAST task Task granularity • Large partition  load imbalance Splitting task BLAST task Merging Task • Small partition  unnecessary overheads … • NCBI-BLAST overhead • Data transferring overhead. BLAST task Best Practice : test runs to p rofiling and set size to mitigate the overhead Value of visibilityTimeout for each BLAST task, • Essentially an estimate of the task run time . • too small  repeated computation; • too large  unnecessary long period of waiting time in case of the instance failure. Best Practice : • Estimate the value based on the number of pair-bases in the partition and test-runs • Watch out for the 2-hour maximum limitation

Task size vs. Performance • Benefit of the warm cache effect • 100 sequences per partition is the best choice Instance size vs. Performance • Super-linear speedup with larger size worker instances • Primarily due to the memory capability. Task Size/Instance Size vs. Cost • Extra-large instance generated the best and the most economical throughput • Fully utilize the resource

BLAST task BLAST task Splitting task BLAST task Merging Task … Worker BLAST task Web Role Job Management Role Web Scaling Engine Worker Portal Job registration Job Scheduler Web … Global Service dispatch queue Worker Job Registry NCBI databases Blast databases, temporary data, etc.) Azure Table Database updating Role Azure Blob

Job Portal ASP.NET program hosted by a web role Web Scaling Engine Portal Job registration instance Job Scheduler Web • Submit jobs Service • Track job’s status and logs Job Registry Authentication/Authorization based on Live ID The accepted job is stored into the job registry table • Fault tolerance, avoid in-memory states

R. palustris as a platform for H2 production Eric Shadt, SAGE Sam Phattarasukol Harwood Lab, UW Blasted ~5,000 proteins (700K sequences) • Against all NCBI non-redundant proteins: completed in 30 min • Against ~5,000 proteins from another strain: completed in less than 30 sec AzureBLAST significantly saved computing time…

Discovering Homologs • Discover the interrelationships of known protein sequences “ A ll against All” query • The database is also the input query • The protein database is large (4.2 GB size) • Totally 9,865,668 sequences to be queried • Theoretically, 100 billion sequence comparisons! Performance estimation • Based on the sampling-running on one extra-large Azure instance • Would require 3,216,731 minutes (6.1 years) on one desktop One of biggest BLAST jobs as far as we know • This scale of experiments usually are infeasible to most scientists

• Allocated a total of ~4000 instances • 475 extra-large VMs (8 cores per VM), four datacenters, US (2), Western and North Europe • 8 deployments of AzureBLAST • Each deployment has its own co-located storage service • Divide 10 million sequences into multiple segments • Each will be submitted to one deployment as one job for execution • Each segment consists of smaller partitions • When load imbalances, redistribute the load manually 62 6 6 2 6 2 6 2 5 62 2 0 5 0

• Total size of the output result is ~230GB • The number of total hits is 1,764,579,487 • Started at March 25 th , the last task completed on April 8 th (10 days compute) • But based our estimates, real working instance time should be 6~8 day • Look into log data to analyze what took place… 62 6 6 2 6 2 6 2 5 62 2 0 5 0

A normal log record should be 3/31/2010 6:14 RD00155D3611B0 Executing the task 251523... 3/31/2010 6:25 RD00155D3611B0 Execution of task 251523 is done, it took 10.9mins 3/31/2010 6:25 RD00155D3611B0 Executing the task 251553... 3/31/2010 6:44 RD00155D3611B0 Execution of task 251553 is done, it took 19.3mins 3/31/2010 6:44 RD00155D3611B0 Executing the task 251600... 3/31/2010 7:02 RD00155D3611B0 Execution of task 251600 is done, it took 17.27 mins Otherwise, something is wrong ( e.g., task failed to complete ) 3/31/2010 8:22 RD00155D3611B0 Executing the task 251774... 3/31/2010 9:50 RD00155D3611B0 Executing the task 251895... 3/31/2010 11:12 RD00155D3611B0 Execution of task 251895 is done, it took 82 mins

North Europe Data Center, totally 34,256 tasks processed All 62 compute nodes lost tasks and then came back in a group. This is an Update domain ~ 6 nodes in one group ~30 mins

West Europe Datacenter; 30,976 tasks are completed, and job was killed 35 Nodes experience blob writing failure at same time A reasonable guess: the Fault Domain is working

MODISAzure : Computing Evapotranspiration (ET) in The Cloud You never miss the water till the well has run dry Irish Proverb

Evapotranspiration (ET) is the release of water to the atmosphere by evaporation from open water bodies and transpiration, or evaporation through plant membranes, by plants. 𝐹𝑈 = ∆𝑆𝑜 + 𝜍 𝑏 𝑑 𝑞 𝜀𝑟 𝑕 𝑏 ( ∆ + 𝛿 1 + 𝑕 𝑏 𝑕 𝑡 ) 𝜇 𝜑 Penman-Monteith (1964) ET = Water volume evapotranspired (m 3 s -1 m -2 ) Δ = Rate of change of saturation specific humidity with air temperature.(Pa K -1 ) Estimating resistance/conductivity across a λ v = Latent heat of vaporization (J/g) catchment can be tricky R n = Net radiation (W m -2 ) c p = Specific heat capacity of air (J kg -1 K -1 ) ρ a = dry air density (kg m -3 ) δ q = vapor pressure deficit (Pa) g a = Conductivity of air (inverse of r a ) (m s -1 ) g s = Conductivity of plant stoma, air (inverse of r s ) (m s -1 ) γ = Psychrometric constant ( γ ≈ 66 Pa K -1 ) • Lots of inputs: big data reduction • Some of the inputs are not so simple

FLUXNET curated sensor dataset (30GB, 960 files) Climate classification ~1MB (1file) Vegetative clumping ~5MB (1file) FLUXNET curated field dataset 2 KB (1 file) NASA MODIS NCEP/NCAR imagery source ~100MB archives (4K files) 5 TB (600K files) 20 US year = 1 global year

Data collection (map) stage • Downloads requested input tiles Source Imagery Download Sites Request from NASA ftp sites . . . Queue • Includes geospatial lookup for non-sinusoidal tiles that will Download Queue contribute to a reprojected Source Metadata sinusoidal tile Scientists Data Collection Stage AzureMODIS Reprojection (map) stage Service Web Role Portal • Converts source tile(s) to Scientific intermediate result sinusoidal tiles Results Download • Simple nearest neighbor or spline algorithms Reprojection Science Derivation reduction stage Queue results • First stage visible to scientist Reprojection Stage Derivation Reduction Stage Analysis Reduction Stage • Computes ET in our initial use Analysis reduction stage • Optional second stage visible to scientist • Enables production of science Reduction #1 Reduction #2 Queue Queue analysis artifacts such as maps, http://research.microsoft.com/en-us/projects/azure/azuremodis.aspx tables, virtual sensors

<PipelineStage>Job Queue <PipelineStage>JobStatus … Persist MODISAzure Service <PipelineStage> (Web Role) Request <PipelineStage>TaskStatus Service Monitor Parse & Persist (Worker Role) Dispatch <PipelineStage>Task Queue … • ModisAzure Service is the Web • Service Monitor is a dedicated Role front door Worker Role • Receives all user requests • Parses all job requests into tasks – recoverable units of work • Queues request to appropriate Download, Reprojection, or • Execution status of all jobs and Reduction Job Queue tasks persisted in Tables

<PipelineStage>TaskStatus Service Monitor Parse & Persist (Worker Role) Dispatch <PipelineStage>Task Queue … GenericWorker … (Worker Role) … <Input>Data Storage All work actually done by a Worker Role • Dequeues tasks created by the • Sandboxes science or other Service Monitor executable • Retries failed tasks 3 times • Marshalls all storage from/to Azure blob storage to/from local Azure • Maintains all task status Worker instance files

Job Queue ReprojectionJobStatus Reprojection Request Each entity specifies a … Persist single reprojection job request ReprojectionTaskStatus Service Monitor Parse & Persist Each entity specifies a (Worker Role) single reprojection task (i.e. a single tile) Dispatch Points to … ScanTimeList Task Queue Query this table to get the list of satellite scan times that cover a target tile … Reprojection Data Storage SwathGranuleMeta … Query this table to get geo-metadata (e.g. boundaries) for each swath GenericWorker tile (Worker Role) Swath Source Data Storage

• Computational costs Source Imagery Download Sites Request driven by data scale and . . . Queue need to run reduction Download multiple times Queue Source Metadata • Storage costs driven by Data Collection Stage Scientists AzureMODIS 400-500 GB Service Web Role Portal data scale and 6 month 60K files 10 MB/sec Scientific $50 upload project duration 11 hours Results $450 storage Download <10 workers • Small with respect to Reprojection the people costs even Queue at graduate student Reprojection Stage Derivation Reduction Stage Analysis Reduction Stage 400 GB 5-7 GB <10 GB rates ! 45K files 5.5K files ~1K files $420 cpu $216 cpu $216 cpu 3500 hours 1800 hours 1800 hours $60 download $1 download $2 download 20-100 20-100 20-100 $6 storage $9 storage workers workers workers Reduction #1 Reduction #2 Queue Queue Total: $1420