Handbook of Cloud Computing

Preface

Contents

Contributors

About the Editors

Part I Technologies and Systems

1 Cloud Computing Fundamentals

1.1 Introduction

1.1.1 Layers of Cloud Computing

1.1.2 Types of Cloud Computing

1.1.3 Cloud Computing Versus Cloud Services

1.2 Enabling Technologies

1.2.1 Virtualization

1.2.2 Web Service and Service Oriented Architecture

1.2.3 Service Flow and Workflows

1.2.4 Web 2.0 and Mashup

1.3 Cloud Computing Features

1.3.1 Cloud Computing Standards

1.3.2 Cloud Computing Security

1.4 Cloud Computing Platforms

1.4.1 Pricing

1.4.2 Cloud Computing Components and Their Vendors

1.5 Example of Web Application Deployment

1.6 Cloud Computing Challenges

1.6.1 Performance

1.6.2 Security and Privacy

1.6.3 Control

1.6.4 Bandwidth Costs

1.6.5 Reliability

1.7 Cloud Computing in the Future

References

2 Cloud Computing Technologies and Applications

2.1 Cloud Computing: IT as a Service

2.2 Cloud Computing Security

2.3 Cloud Computing Model Application Methodology

2.3.1 Cloud Computing Strategy Planning Phase

2.3.2 Cloud Computing Tactics Planning Phase

2.3.3 Cloud Computing Deployment Phase

2.4 Cloud Computing in Development/Test

2.5 Cloud-Based High Performance Computing Clusters

2.6 Use Cases of Cloud Computing

2.6.1 Case Study: Cloud as Infrastructure for an Internet Data Center (IDC)

2.6.1.1 The Bottleneck on IDC Development

2.6.1.2 Cloud Computing Provides IDC with a New Infrastructure Solution

2.6.1.3 The Value of Cloud Computing for IDC Service Providers

2.6.1.4 The Value Brought by Cloud Computing for IDC Users

2.6.1.5 Cloud Computing Can Make Fixed Costs Variable

2.6.1.6 An IDC Cloud Example

2.6.1.7 The Influence of Cloud Computing in 3G Era

2.6.2 Case Study -- Cloud Computing for Software Parks

2.6.2.1 Cloud Computing Architecture

2.6.2.2 Outsourcing Software Research and Development Platform

2.6.3 Case Study -- an Enterprise with Multiple Data Centers

2.6.3.1 Overall Design of the Cloud Computing Platform in an Enterprise

2.6.4 Case Study: Cloud Computing Supporting SaaS

2.7 Conclusion

3 Key Enabling Technologies for Virtual Private Clouds

3.1 Introduction

3.2 Virtual Private Clouds

3.3 Virtual Data Centers and Applications

3.3.1 Virtual Data Centers

3.3.2 Virtual Applications

3.4 Policy-Based Management

3.4.1 Policy-Based Deployment

3.4.2 Policy Compliance

3.5 Service-Management Integration

3.6 Conclusions

References

4 The Role of Networks in Cloud Computing

4.1 Introduction

4.2 Cloud Deployment Models and the Network

4.2.1 Public Cloud

4.2.2 Private Cloud

4.2.3 Hybrid Cloud

4.2.4 An Overview of Network Architectures for Clouds

4.2.4.1 Data Center Network

4.2.4.2 Data Center Interconnect Network

4.3 Unique Opportunities and Requirements for Hybrid Cloud Networking

4.3.1 Virtualization, Automation and Standards -- The Foundation

4.3.2 Latency, Bandwidth, and Scale -- The Span

4.3.3 Security, Resiliency, and Service Management -- The Superstructure

4.4 Network Architecture for Hybrid Cloud Deployments

4.4.1 Cloud-in-a-Box

4.4.2 Network Service Node

4.4.3 Data Center Network and Data Center Interconnect Network

4.4.4 Management of the Network Architecture

4.5 Conclusions and Future Directions

References

5 Data-Intensive Technologies for Cloud Computing

5.1 Introduction

5.1.1 Data-Intensive Computing Applications

5.1.2 Data-Parallelism

5.1.3 The ''Data Gap''

5.2 Characteristics of Data-Intensive Computing Systems

5.2.1 Processing Approach

5.2.2 Common Characteristics

5.2.3 Grid Computing

5.2.4 Applicability to Cloud Computing

5.3 Data-Intensive System Architectures

5.3.1 Google MapReduce

5.3.2 Hadoop

5.3.3 LexisNexis HPCC

5.3.4 ECL

5.4 Hadoop vs. HPCC Comparison

5.4.1 Terabyte Sort Benchmark

5.4.2 Pig vs. ECL

5.4.3 Architecture Comparison

5.5 Conclusions

References

6 Survey of Storage and Fault Tolerance Strategies Used in Cloud Computing

6.1 Introduction

6.1.1 Theme 1: Voluminous Data

6.1.2 Theme 2: Commodity Hardware

6.1.3 Theme 3: Distributed Data

6.1.4 Theme 4: Expect Failures

6.1.5 Theme 5: Tune for Access by Applications

6.1.6 Theme 6: Optimize for Dominant Usage

6.1.7 Theme 7: Tradeoff Between Consistency and Availability

6.2 xFS

6.2.1 Failure Model

6.2.2 Replication

6.2.3 Data Access

6.2.4 Integrity

6.2.5 Consistency and Guarantees

6.2.6 Metadata

6.2.7 Data placement

6.2.8 Security

6.3 Amazon S3

6.3.1 Data Access and Management

6.3.2 Security

6.3.3 Integrity

6.4 Dynamo

6.4.1 Checkpointing

6.4.2 Replication

6.4.3 Failures

6.4.4 Accessing Data

6.4.5 Data Integrity

6.4.6 Consistency and Guarantees

6.4.7 Metadata

6.4.8 Data Placement

6.4.9 Security

6.5 Google File System

6.5.1 Checkpointing

6.5.2 Replication

6.5.3 Failures

6.5.4 Data Access

6.5.5 Data Integrity

6.5.6 Consistency and Guarantees

6.5.7 Metadata

6.5.8 Data Placement

6.5.9 Security Scheme

6.6 Bigtable

6.6.1 Replication

6.6.2 Failures

6.6.3 Accessing Data

6.6.4 Data Integrity

6.6.5 Consistency and Guarantees

6.6.6 Metadata

6.6.7 Data Placement

6.6.8 Security

6.7 Microsoft Azure

6.7.1 Replication

6.7.2 Failure

6.7.3 Accessing Data

6.7.4 Consistency and Guarantees

6.7.5 Data Placement

6.7.6 Security

6.8 Transactional and Analytics Debate

6.9 Conclusions

References

7 Scheduling Service Oriented Workflows Inside Clouds Using an Adaptive Agent Based Approach

7.1 Introduction

7.2 Related Work on DS Scheduling

7.3 Scheduling Issues Inside Service Oriented Environments

7.3.1 Estimating Task Runtimes and Transfer Costs

7.3.2 Service Discovery and Selection

7.3.3 Negotiation Between Service Providers

7.3.4 Overcoming the Internal Resource Scheduler

7.3.5 Trust in Multi-cloud Environments

7.4 Workflow Scheduling

7.5 Distributed Agent Based Scheduling Platform Inside Clouds

7.5.1 The Scheduling Platform

7.5.2 Scheduling Through Negotiation

7.5.3 Prototype Implementation Details

7.6 Conclusions

References

8 The Role of Grid Computing Technologies in Cloud Computing

8.1 Introduction

8.2 Basics of Grid and Cloud Computing

8.2.1 Basics of Grid Computing

8.2.2 Basics of Cloud Computing

8.2.3 Interaction Models of Grid and Cloud Computing

8.2.4 Distributed Computing in the Grid and Cloud

8.3 Layered Models and Usage patterns in Grid and Cloud

8.3.1 Infrastructure

8.3.2 Platform

8.3.2.1 Abstraction from Physical Resources

8.3.2.2 Programming API to Support New Services

8.3.3 Applications

8.4 Techniques

8.4.1 Service Orientation and Web Services

8.4.2 Data Management

8.4.3 Monitoring

8.4.4 Autonomic Computing

8.4.5 Scheduling, Metascheduling, and Resource Provisioning

8.4.6 Interoperability in Grids and Clouds

8.4.7 Security and User Management

8.4.8 Modeling and Simulation of Clouds and Grids

8.5 Concluding Remarks

References

9 Cloudweaver: Adaptive and Data-Driven Workload Manager for Generic Clouds

9.1 Introduction

9.2 System Overview

9.2.1 Components

9.2.1.1 Workload Manager

9.2.1.2 Cloud Monitor

9.2.1.3 Generic Cloud

9.3 Workload Manager

9.3.1 Terminology

9.3.2 Operator Parallelization Status

9.3.3 Job Execution Algorithm

9.3.4 Dynamic Parallelization for Job Execution

9.3.5 Balancing Pipelined Operators

9.3.6 Balancing Tiers

9.3.7 Scheduling Multiple Jobs

9.4 Related Work

9.4.1 Parallel Databases

9.4.2 Data Processing in Cluster

9.5 Conclusion

References

Part II Architectures

10 Enterprise Knowledge Clouds: Architecture and Technologies

10.1 Introduction

10.2 Business Enterprise Organisation

10.3 Enterprise Architecture

10.4 Enterprise Knowledge Management

10.5 Enterprise Knowledge Architecture

10.6 Enterprise Computing Clouds

10.7 Enterprise Knowledge Clouds

10.8 Enterprise Knowledge Cloud Technologies

10.9 Conclusion: Future Intelligent Enterprise

References

11 Integration of High-Performance Computing into Cloud Computing Services

11.1 Introduction

11.2 NC State University Cloud Computing Implementation

11.3 The VCL Cloud Architecture

11.3.1 Internal Structure

11.3.1.1 Storage

11.3.1.2 Partner's Program

11.3.2 Access

11.3.2.1 Standard

11.3.2.2 Special needs

11.3.3 Computational/Data Node Network

11.4 Integrating High-Performance Computing into the VCL Cloud Architecture

11.5 Performance and Cost

11.6 Summary

References

12 Vertical Load Distribution for Cloud Computing via Multiple Implementation Options

12.1 Introduction

12.2 Overview

12.3 Scheduling Composite Services

12.3.1 Solution Space

12.3.2 Genetic algorithm

12.3.2.1 Chromosome Representation of a Solution

12.3.2.2 Chromosome Recombination

12.3.2.3 GA Evaluation Function

12.3.3 Handling Online Arriving Requests

12.4 Experiments and Results

12.4.1 Baseline Configuration Results

12.4.2 Effect of Service Types

12.4.3 Effect of Service Type Instances

12.4.4 Effect of Servers (Horizontal Balancing)

12.4.5 Effect of Server Performance

12.4.6 Effect of Response Variation Control

12.4.7 Effect of Routing Against Conservative SLA

12.4.8 Summary of Experiments

12.5 Related Work

12.6 Conclusion

References

13 SwinDeW-C: A Peer-to-Peer Based Cloud Workflow System

13.1 Introduction

13.2 Motivation and System Requirement

13.2.1 Large Scale Workflow Applications

13.2.2 System Requirements

13.2.2.1 QoS Management

13.2.2.2 Data Management

13.2.2.3 Security Management

13.3 Overview of SwinDeW-G Environment

13.4 SwinDeW-C System Architecture

13.4.1 SwinCloud Infrastructure

13.4.2 Architecture of SwinDeW-C

13.4.3 Architecture of SwinDeW-C Peers

13.5 New Components in SwinDeW-C

13.5.1 QoS Management in SwinDeW-C

13.5.2 Data Management in SwinDeW-C

13.5.3 Security Management in SwinDeW-C

13.6 SwinDeW-C System Prototype

13.7 Related Work

13.8 Conclusions and Feature Work

References

Part III Services

14 Cloud Types and Services

14.1 Introduction

14.2 Cloud Types

14.2.1 Public Cloud

14.2.2 Private Cloud

14.2.3 Hybrid Cloud

14.2.4 Community Cloud

14.3 Cloud Services and Cloud Roles

14.4 Infrastructure as a Service

14.4.1 Amazon Elastic Compute Cloud (EC2)

14.4.2 GoGrid

14.4.3 Amazon Simple Storage Service (S3)

14.4.4 Rackspace Cloud

14.5 Platform as a Service

14.5.1 Google App Engine

14.5.2 Microsoft Azure

14.5.3 Force.com

14.6 Software as a Service

14.6.1 Desktop as a Service

14.6.2 Google Apps

14.6.3 Salesforce

14.6.4 Other Software as Service Examples

14.7 The Amazon Family

14.7.1 RightScale: IaaS Based on AWS

14.7.2 HeroKu: Platform as a Service Using Amazon Web Service

14.7.3 Animoto Software as Service Using AWS

14.7.4 SmugMug Software as Service Using AWS

14.8 Conclusion

References

15 Service Scalability Over the Cloud

15.1 Introduction

15.2 Foundations

15.2.1 History on Enterprise IT Services

15.2.2 Warehouse-Scale Computers

15.2.3 Grids and Clouds

15.2.4 Application Scalability

15.2.5 Automating Scalability

15.3 Scalable Architectures

15.3.1 General Cloud Architectures for Scaling

15.3.2 A Paradigmatic Example: Reservoir Scalability

15.4 Conclusions and Future Directions

References

16 Scientific Services on the Cloud

16.1 Introduction

16.1.1 Outline

16.2 Service Oriented Atmospheric Radiances (SOAR)

16.3 Scientific Programming Paradigms

16.3.1 MapReduce

16.3.1.1 MapReduce Merge

16.3.2 Dryad

16.3.3 Remote Sensing Geo-Reprojection

16.3.3.1 Remote Sensing Geo-Reprojection with MapReduce

16.3.3.2 Remote Sensing Geo-reprojection with Dryad

16.3.4 K-Means Clustering

16.3.4.1 K-Means Clustering with MapReduce

16.3.4.2 K-Means Clustering with Dryad

16.3.5 Singular Value Decomposition

16.3.5.1 Singular Value Decomposition with MapReduce

16.3.5.2 Singular Value Decomposition with Dryad

16.4 Delivering Scientific Computing services on the Cloud

16.4.1 Service Requirements

16.4.2 Service Discovery

16.4.3 Service Negotiation

16.4.4 Service Composition

16.4.5 Service Consumption and Monitoring

16.5 Summary/Conclusions

References

17 A Novel Market-Oriented Dynamic Collaborative Cloud Service Platform

17.1 Introduction

17.2 Related Works

17.3 A Dynamic Collaborative Cloud Services Platform

17.4 Proposed Combinatorial Auction Based Cloud Market (CACM) Model to Facilitate a DC Platform

17.4.1 Market Architecture

17.4.2 Additional Components of a CP to Form a DC Platform in CACM

17.4.3 Formation of a DC Platform in CACM Model

17.4.4 System Model for Auction in CACM

17.4.4.1 Single and Group Bidding Functions of CPs

17.4.4.2 Payoff Function of the User/Consumer

17.4.4.3 Profit of the CPs to form a Group

17.5 Model for Partner Selection

17.5.1 Partner Selection Problem

17.5.2 MO Optimization Problem for Partner Selection

17.5.3 Multi-objective Genetic Algorithm

17.6 Evaluation

17.6.1 Evaluation Methodology

17.6.1.1 Simulation Examples

17.6.2 Simulation Results

17.6.2.1 Appropriate Approach to Develop the MOGA-IC

17.6.2.2 Performance comparison of MOGA-IC with MOGA-I in CACM Model

17.7 Conclusion and Future Work

References

Part IV Applications

18 Enterprise Knowledge Clouds:Applications and Solutions

18.1 Introduction

18.2 Enterprise Knowledge Management

18.2.1 EKM Applications

18.3 Knowledge Management in the Cloud

18.3.1 Knowledge Content

18.3.2 Knowledge Users

18.3.3 Enterprise IT

18.3.3.1 Problem Solving

18.3.3.2 Monitoring, Tuning and Automation

18.3.3.3 Business Intelligence and Analytics

18.3.3.4 Decision Making

18.3.4 The Intelligent Enterprise

18.4 Moving KM Applications to the Cloud

18.5 Conclusions and Future Directions

References

19 Open Science in the Cloud: Towards a Universal Platform for Scientific and Statistical Computing

19.1 Introduction

19.2 An Open Platform for Scientific Computing, the Building Blocks

19.2.1 The Processing Capability

19.2.2 The Mathematical and Numerical Capability

19.2.3 The Orchestration Capability

19.2.4 The Interaction Capability

19.2.5 The Persistence Capability

19.3 Elastic-R and Infrastructure-as-a-Service

19.3.1 The Building Blocks of a Traceable and Reproducible Computational Research Platform

19.3.2 The Building Blocks of a Platform for Statistics and Applied Mathematics Education

19.4 Elastic-R, an e-Science Enabler

19.4.1 Lowering the Barriers for Accessing on-Demand Computing Infrastructures. Local/Remote Transparency

19.4.2 Dealing with the Data Deluge

19.4.3 Enabling Collaboration Within Computing Environments

19.4.4 Science Gateways Made Easy

19.4.5 Bridging the Gap Between Existing Scientific Computing Environments and Grids/Clouds

19.4.6 Bridging the Gap Between Mainstream Scientific Computing Environments

19.4.7 Bridging the Gap Between Mainstream Scientific Computing Environments and Workflow Workbenches

19.4.8 A Universal Computing Toolkit for Scientific Applications

19.4.9 Scalability for Computational Back-Ends

19.4.10 Distributed Computing Made Easy

19.5 Elastic-R, an Application Platform for the Cloud

19.5.1 The Elastic-R Plug-ins

19.5.2 The Elastic-R Spreadsheets

19.5.3 The Elastic-R extensions

19.6 Cloud Computing and Digital Solidarity

19.7 Conclusions and Future Directions

References

20 Multidimensional Environmental Data Resource Brokering on Computational Grids and Scientific Clouds

20.1 Introduction

20.2 Resource Discovery and Selection Using a Resource Broker Service

20.3 Anagram Based GrADS Data Distribution Service

20.4 Hyrax Based Five Dimension Distribution Data Service

20.5 Design and Implementation of an Instrument Service for NetCDF Data Acquisition

20.6 A Weather Forecast Quality Evaluation Scenario

20.7 Implementation of the Grid Application

20.8 Conclusions and Future Work

References

21 HPC on Competitive Cloud Resources

21.1 Introduction

21.2 Related Work

21.3 Background

21.3.1 Overview of Amazon EC2 Setup

21.3.2 Overview of HPL

21.4 Intranode Scaling

21.4.1 DGEMM Single Node Evaluation

21.4.2 HPL Single Node Evaluation

21.5 Internode Scaling

21.5.1 HPL Minimum Evaluation

21.5.2 HPL Average Evaluation

21.6 Conclusions

References

22 Scientific Data Management in the Cloud: A Survey of Technologies, Approaches and Challenges

22.1 Introduction

22.2 Data Management Issues Within Scientific Experiments

22.3 Data Clouds: Emerging Technologies

22.4 Case Studies: Harnessing the Data Cloud for Scientific Data Management

22.4.1 Pan-STARRS Data with GrayWulf

22.4.2 GEON Workflow with the CluE Cluster

22.4.3 SciDB

22.4.4 Astrophysical Data Analysis with Pig/Hadoop

22.4.5 Public Data Hosting by Amazon Web Services

22.5 A Gap Analysis of Data Cloud Capabilities

22.5.1 The Impedance Mismatch

22.5.2 Fault Tolerance

22.5.3 Scientific Data Format and Analysis Tools

22.5.4 Integration with the Object Oriented Programming Model

22.5.5 Working with Legacy Software

22.5.6 Real-Time Data

22.5.7 Programmable Interfaces to Performance Optimizations

22.5.8 Distributed Database Issues

22.5.9 Security and Privacy

22.6 Conclusions

References

23 Feasibility Study and Experience on Using Cloud Infrastructure and Platform for Scientific Computing

23.1 Introduction

23.2 Scientific Compute Tasks

23.3 Scientific Computing in the Cloud

23.3.1 Cloud Architecture as Foundation of Cloud-Based Scientific Applications

23.3.2 Emergence of Cloud-Based Scientific Computational Applications

23.4 Building Cloud Infrastructure for Scientific Computing

23.4.1 Setup and Experiment on Tiny Cloud Infrastructure and Platform

23.4.2 On Economical Use of the Enterprise Cloud

23.5 Toward Integration Of Private and Public Enterprise Cloud Environment

23.6 Conclusion

References

24 A Cloud Computing Based Patient Centric Medical Information System

24.1 Introduction

24.2 Potential Impact of Proposed Medical Informatics System

24.3 Background and Related Work

24.4 Brief Discussion of Medical Standards

24.5 Architecture Description and Research Methods

24.5.1 Objective 1: A Service Oriented Architecture for Interfacing Medical Messages

24.5.2 Objective 2: Lossless Accelerated Presentation Layer for Viewing DICOM Objects on Cloud

24.5.3 Objective 3: Web Based Interface for Patient Health Records

24.5.4 Objective 4: A Globally Distributed Dynamically Scalable Cloud Based Application Architecture

24.5.4.1 Distributed Data Consistency Across Clouds

24.5.4.2 Higher availability and application scalability

24.5.4.3 Concerning Low Level Security

References

25 Cloud@Home: A New Enhanced Computing Paradigm

25.1 Introduction

25.2 Why Cloud@Home?

25.2.1 Aims and Goals

25.2.2 Application Scenarios

25.3 Cloud@Home Overview

25.3.1 Issues, Challenges and Open Problems

25.3.2 Basic Architecture

25.3.3 Frontend Layer

25.3.4 Virtual Layer

25.3.5 Physical Layer

25.3.6 Management Subsystem

25.3.7 Resource Subsystem

25.4 Ready for CloudHome?

References

26 Using Hybrid Grid/Cloud Computing Technologies for Environmental Data Elastic Storage, Processing,and Provisioning

26.1 Introduction

26.2 Distributing Multidimensional Environmental Data

26.3 Environmental Data Storage on Elastic Resources

26.3.1 Amazon Cloud Services

26.3.2 Multidimensional Environmental Data Standard File Format

26.3.3 Enhancing the S3 APIs

26.3.4 Enabling the NetCDF Java Interface to S3

26.4 Cloud and Grid Hybridization: The NetCDF Service

26.4.1 The NetCDF Service Architecture

26.4.2 NetCDF Service Deployment Scenarios

26.5 Performance Evaluation

26.5.1 Parameter Selection for the S3-Enhanced Java Interface

26.5.2 Evaluation of S3- and EBS-Enabled NetCDF Java Interfaces

26.5.3 Evaluation of NetCDF Service Performance

26.6 Conclusions and Future Directions

References

Index