Scale out into the global Amazon cloud

On demand spawning of additional DS9 instances in the cloud

With the AWS cloud option we dynamically adapt hardware resouces to current needs and therefore enable our customers to dynamically benefit from the scalability of CPU power, memory and disk space, that is made available in the AWS cloud

More resources required at certain times

When jobs are running for a limited time, but during that time they could do with more resources…

AWS EC2 instances can be configured to the needs of a job. When running a distributed job the instance type (CPU, RAM) of the Amazon EC2 instance is selected from preconfigured types, the required disk space is set and finally the number of instances to be started is defined

When the DS9 master node starts the distributed job, DS9 dynamically starts the requested EC2 instances running a full blown DS9 system and executes the job remotely on each of these instances. Data to be processed is evenly distributed across all EC2 instances and after the job is finished, result data is played back to the DS9 master node

It is simple to calculate the potential cost saving the AWS option provides – based on the job runtime and execution frequency

DS9 crawlers run around the world

Taking advantage of the geographically distributed location of AWS regions

Ping time from a DS9 server to a web resource that shall be crawled varies greatly depending on where on the globe the crawled resource is hosted

When you click on the image, you can enlarge this. The elipses indicate distance from Frankfurt and e.g. Tokyo and Seattle have a similar distance to Frankfurt

 

Surprisingly the ping signal travels 270ms to Tokyo but only 152ms to Seattle. This is because the fiber optics have to go all the way around Cape of Good Hope before they reach Asia

That’s why DS9 deploys crawlers to the AWS regions that are closest to the targets to be crawled

URLs of servers that are targeted by DS9 crawlers are geographically mapped to the individually supported AWS regions. Depending on the number of targets in a particular part of the world, the number of EC2 instances that will be spawned in the AWS region closest to the targets

Importantly, any targeted server within any geographoc region can be pinged with less than 50ms
 

Already while targets are being crawled collected data is sent back to the DS9 master node. After all targets have been crawled, the job is terminated and all instances are cleaned up and released

Take off to new horizons

Get the most from your DS9 license

AWS functionality is available for DS9 Runtime unlimited and DS9 Developer’s Edition

DS9 Runtime