CTC-Swarm: a cloud resource manager used to
scrape large amounts of online data
Marc Juchli Mathijs Hoogland
Technical University of Delft Technical University of Delft
Email: m.b.juchli@student.tudelft.nl Email: m.h.hoogland@student.tudelft.nl
Course Instrutors: Dick Epema, Distributed Systems Group,
EEMCS, TU Delft, Email: D.H.J.Epema@tudelft.nl
Lab Assistant: Apourva Parthasarathy, Email:
a.parthasarathy@student.tudelft.nl
Abstract—In this report, we present the CTC-Swarm, a cloud
resource manager that runs multiple data retrieval tasks as a
preparation for automated crypto-currency trading. The software
automatically deploys the tasks on the cloud, monitors them
and restarts in case of errors. Two provisioning policies are
introduced, and tests show that a choice needs to be made
between lower costs and higher execution speed in order to choose
the optimal policy. We also manage to lower the downtime of
streaming services to one minute in case of error, by constantly
observing the pressure of the incoming data.
Keywords—IaaS, Cloud, Cluster Management
I. I NTRODUCTION
Cloud services have caused a revolution in the dynamics of
IT industry by providing scalable and virtualizable resources.
Infrastructure as a Service (IaaS) is one of the three types of
cloud service models, which supplies its users with access to
virtual computing resources such as server space or hardware
[1]. However, oftentimes the user of such an infrastructure does
not receive support in deploying and managing his application.
In this report, an application is introduced that orchestrates
and manages the extraction of large amounts of data from the
internet. This data is used as input for an automated trading
algorithm for crypto-currencies. In such trading applications,
it is crucial to be able to get data as soon as it is available and
employ it. Because of this requirement, the resource usage of
our application can vary from time to time.
To be able to efficiently cope with these peaks in demand,
this report presents CTC-Swarm. CTC-Swarm is an auto-
scaling resource management system that schedules and scales
computing resources based on demand. The software utilizes
EC2 computing instances from Amazon Web Services (AWS)
in order to run several data extraction and data mining appli-
cations. Further, it runs scripts packed into docker containers
in an automated and elastic fashion and allows monitoring and
scheduling of the jobs.
Therefore, the report is structured as follows. We start with
the presentation of the overview of CTC-Swarm’s main fea-
tures and capabilities in section II. We then describe our service
level objectives in section III. In section IV the architecture of
our software is presented. In section V we present the imple-
mented features and characteristics of CTC-Swarm. Further, in
Mateusz Garbacz
Technical University of Delft
Email: m.garbacz@student.tudelft.nl
section VI the experimental setup is presented together with
the evaluation of specific features of the system. We end our
report with section VII, containing our conclusions.
II. BACKGROUND ON CTC-S WARM
CTC-Swarm is an application that can be used for many
purposes. However, in this report, we focus on the data
collection tasks, especially online information related to the
trading of crypto-currencies. This information will be further
fed to an automated trading application developed by the
WantCloud company. Therefore, the purpose of CTC-Swarm
is to manage 3 data retrieval tasks which are provided by the
company:
1) News extraction from websites: the task downloads
a provided RSS feed with an arbitrary number of
elements, specifying the most current crypto-currency
related articles, their addresses, titles and release
dates. For each article, a dedicated crawler ex-
tracts the content and stores this information in the
database. Once all the provided URLs are crawled,
the program terminates.
2) Tweet streaming: This task consists of streaming,
processing and storing of tweets that contain certain
pre-defined keywords. From each object, the program
extracts the text of the body, converts and performs
the pre-processing. Finally, each tweet is stored in the
database.
3) Trade information processing: this program continu-
ously streams the current data regarding trades exe-
cuted on one of the online crypto-currency markets
and summarizes them forming one-minute windows
of the data. Therefore, every minute thousands of
trades are loaded, accumulated, and stored in the
database.
CTC-Swarm is responsible for the execution of these jobs
and needs to monitor and optimize the resource usage of the
application.
III. S ERVICE L EVEL O BJECTIVES
It is crucial for CTC-Swarm to ensure that any employed
finite task (i.e. the news extraction) is executed successfully
and any continuous streaming task (e.g. the tweet streaming)
break for too long, even if an error occurs. We define a
Service Level Objectives (SLO) for our application to make
this requirement measurable. For the finite task, the SLO would
be to successfully finish any task scheduled or delegated by a
user. In case of an error, the task needs to be restarted and
finished. When it comes to the streaming jobs, the SLO of
our application is to stream the data continuously. Whenever
a break occurs, this break may not be longer than one minute.
This means that, if, for example, a Tweet streaming task
crashes, our application needs to restart this Tweet streaming
task in: less than a minute. Finally, an SLO violation is any
violation of the two previously defined objectives. They are
counted by taking a sum over the number of not successfully
executed continuous tasks added and the number of minutes
that the streaming tasks did not return any data.
IV. S YSTEM DESIGN
Our application uses a micro-service architecture. Each of
the 3 tasks is designed as a standalone dockerized application
and in our case deployed on an EC2 computing instance from
AWS. Multiple applications can, therefore, be deployed on one
instance.
A. Interaction
Figure 1 contains a high-level overview of the architecture
of our application. There are three types of initiators which
start an interaction with the underlying components:
1) Users interact with the application through a RESTful
API (see description in IV-C) whose requests are
handled by the Request Handler. This class
translates user requests for the Manager, which
manages the task execution (see IV-B). In case the
user wishes to schedule future tasks upon request,
the translated task, which is to be completed, is sent
to the task queue S together with the UTC-timestamp
of its preferred execution.
The Scheduling thread, not only plans future
tasks but is also responsible for maintaining the
company’s SLO’s (III). That is, by approaching the
StatisticsManager for past container utiliza-
tion, CPU utilization and application specific infor-
mation such as trade-, news- and tweet pressure.
2) Therefore, the Monitoring thread continuously
observes utilization of infrastructure and applica-
tions and reports to the internal database. Specif-
ically, it uses the AWSManager to retrieve CPU
utilization, the DockerManager to observe the
number of deployed containers per instance and
the StatisticsManger to report the pressure of
trades, tweets and news per minute and per hour.
B. Components
The Manager class serves as an intermediary for handling
interactions on both AWS and Docker. Its other duty is to
provide a task queue T, where provisioning tasks can be
managed using the interfaces addTask and removeTask.
We define a Task to be a tuple consisting of a service
(described within the DockerManager) and an operation
(either start, stop or restart): Task: (Service,
Operation). The reason for handling tasks on manager level
instead of docker level is the fact that services can run on an
arbitrary AWS instance, depending on the provisioning policy.
Hence, the manager holds a dependency for both classes.
Further more, the Manager further provides the interface
run(policy), allowing to define which policy (see V-A)
the manager should follow while processing those tasks from
the queue.
The AWSManager class handles resources we de-
mand from Amazon Web Services (AWS), such as RDS
databases, EC2 instances and their utilization using Cloud-
Watch. Therefore we use the API wrapper library boto3[2].
Next to the obvious functionality that lists currently run-
ning resources, more complex operations are implemented
as well. For example, launchEC2Instance(name) and
launchRDSInstance are serving the purpose of provi-
sioning a resource, EC2 or RDS respectively, and ensure that
the resource is being started correctly. As those functions are
time-consuming, hence blocking, we used Pythons asyncio
library in order to work in asynchronous fashion. Once the
Future results in the expected result, the instances are ready
for services to be deployed on.
The DockerManager is responsible for handling con-
tainer (e.g. service) related concerns. We, therefore, rely on
the Docker API wrapper library docker-py[3]. This includes
the deployment of our images: ctc-news, ctc-trades-watch,
ctc-twitter and are deployed with the functions: deployNews,
deployTrades, deployTwitter. In order to deploy any of the
images, the DockerManager needs to connect to a docker
installation on one of the deployed instances on AWS. This
can be done in two ways. The simple approach is to con-
nect to the machine which is passed directly to the class
using the function setClientByInstance(instance).
However, there are some cases where we have multiple
instances running and wish to connect to the most suit-
able one based on some strategy. Therefore, the func-
tion setClientByLowestContainers(instances)
allows to connect to the one instance which currently maintains
the least running docker containers.
Once services are deployed and running, the
StatisticsManager provides essentially two purposes:
report and retrieve pressure for services. That is, reporting
pressure is handled by the monitoring thread (running on
the server) and connects to the Results-Database, determines
pressure for trades, tweets, or news, and writes the observed
pressure into our Application-Database. We define the table
that reports pressure for an arbitrary service s as follows:
pressure(s): id, timestamp, minute,
hour, option
The ids and timestamps are incrementing, the minute and
hour fields describe the pressure for the last minute or hour
and the option field allows to define e.g. the instance (for
CPU pressure) or the market (for trade pressure) that was
observed. Retrieving the pressure is required in the scheduler
and simply reads out previously reported pressure of a service.
The Scheduler class is an extension used by the
Scheduling thread and serves a queue S that keeps track of
all tasks that need to be executed in the future. The function
addTask(task, timestamp) therefore accepts the to be
executed task and the timestamp of its execution. The task
again is a tuple of a service and its operation, whereas the
service is in our case a docker container. In addition, we also
provide capabilities of accepting cron tasks, by specifying an
interval instead of a timestamp: addTask(task, cron)
C. REST API
As mentioned in IV-A, CTC-Swarm exposes a restful
API such that users can interact with the infrastructure. The
endpoints can be described as follows:
POST /manager/strategy: name, args
POST /task: service, operation
DELETE /task: service, operation
DELETE /allTasks
POST /schedule/task: service, operation
As on can see from the naming of the endpoints, the API serves
a subset of the capabilities provided by the components de-
scribed in IV-B. The RequestHandler therefore translates
the requests and calls the appropriate function in the specific
class.
V. I MPLEMENTED FEATURES
A. Automation
The automation part of this project involved the handling of
RDS instances, EC2 instances and Docker containers, laying
out a foundation for other features to rely on.
The AWSManager provides interfaces to communicate
tasks to concerning a resource provided by Amazon AWS. The
full class documentation is listed in Appendix -A. As one can
see from the function descriptions, the functionality is mostly
concerned about launching, reading and terminating instances.
The DockerManager provides interfaces to handle con-
tainers (to be) deployed on EC2 instances. Much of the
functionality we use is already provided by the API, however,
our project involves, for example, the deployment of containers
which require extensive configuration at deployment. There-
fore Appendix section -B describes the from us developed
interfaces.
Whereas the previous two classes describe infrastructure
functionality, the class StatisticsManager is more tar-
geted to observe and report application-level behavior. As it
can be seen from Appendix section -C, for every type of
service there is a reporting function as well as a getter.
B. Elasticity / Load-balancing
When it comes to the elasticity, the Manager takes care of
booting new instances whenever necessary and closing them
whenever they are idle and there are no jobs to be executed.
The strategy the manager uses in order to launch instances and
containers is what we call the Provisioning Policy. Depending
on the policy, the Manager manages the VMs differently.
We set a maximum of 10 VMs that can run at the same
time, preventing our application from unexpectedly becoming
more expensive than anticipated. The evaluation of our tests
VI shows that this number is sufficient. Two main policies
managing the load balancing: one based on the threshold of
instances per VM, while the other one based on the utilization
threshold of the VMs. These will be discussed in more detail
in the upcoming paragraphs:
a) Maximum number of tasks run on a single instance:
The first policy takes the maximum number of containers to
be run on a single VM as a parameter. Once the Manager runs
with this policy, it boots as many instances as it expects would
be minimum for the threshold to run the number tasks in the
queue. Then, in each iteration it first checks whether any of the
VMs can be closed or any new ones need to be booted based
on the current number of tasks running and the queue size.
Then, it tries to run as many tasks as possible on the instances
with the lowest number of containers running. However, the
main limitation of this policy is the fact that it does not take
into consideration how resource demanding some applications
may be. This problem is addressed by the second policy.
b) Utilization threshold of the VMs: The second policy
takes two parameters as input. It requires the maximum uti-
lization threshold, which specifies what level of utilization of
a VM the manager will assign tasks to it, and initial instance
allocation that is a guess of the user of how many containers
the instance can handle at the same time. However, the second
parameter is used only in case of many tasks being added
to the queue at the same time, making the manager start
booting multiple instances, instead of booting one at a time,
and waiting for it to be fully utilized. The policy performs
similar steps in each iteration, the only modifications are that
it additionally boots a new instance if all instances reach the
utilization limit and assigns tasks to instances with the lowest
utilization. The policy improves upon the previous policy in
terms of monitoring the CPU usage of the instances and
possible cost savings.
These two policies are evaluated in the next section for
multiple parameter sets.
C. Reliability
The CTC-Swarm implements the reliability feature, by
making sure that the streaming tasks are down for an as short
time as possible and the finite tasks are quickly restarted in
case of an error. This is done on two levels.
1) Firstly, the Manager class does it at the container
level. In each iteration checks, whether any of the
Docker containers running has the failed status,
meaning that there was a tasks error. Such containers
are restarted directly.
2) Secondly, the Scheduler does it in a streaming
application-specific way, by monitoring the rate of
incoming trades and in case of a drastic decrease, it
warns the Manager. The Manager checks again the
status of the container, and if Docker indicates it is
running correctly and it has not just been restarted by
the Manager, the restart process is invoked. Thanks to
this, we make sure that even if there was an error, that
did not cause the container to fail, while negatively
influencing the streaming, the task gets restarted.
D. Monitoring
The monitoring feature is implemented in CTC-Swarm
on the application specific level for the streaming tasks. The
system monitors the number of tweets and trades for different
markets streamed per minute and stores it them into a separate
database. Additionally, this allows for monitoring whether
some of the streaming tasks have been broken and restarting
these tasks by the manager if the rate of streamed data is too
low.
Fig. 1. CTC-Swarm Architecture Overview
E. Scheduler
Finally, there is a scheduler feature that enables users to
schedule tasks that need to be executed in the future. Users
can submit tasks in combination with a timestamp through
the Request Handler API to CTC-Swarm. The scheduler
will then make sure that these tasks will be executed at the
given timestamp.
VI. E XPERIMENTS AND R ESULTS
This section describes the experiments performed to eval-
uate the system. The following subsections present the overall
experimental setup and the specific test cases for each feature.
A. Experimental setup
The general setup for the experiments is established as
follows. The program is capable of running on any machine as
we specify AWS accounts using environment variables. This
allows us to develop and run tests from our local machine.
Using the AWS API, EC2 instances can be listed which allows
us to connect to the exposed Docker API within the machine.
In a production setup, the CTC-Swarm application would run
on as a docker container within one of the available instances.
In fact, CTC-Swarm is a stateless application and hence can be
run simultaneously on multiple instances in order to contribute
as an extended reliability factor. The specification of the
machine CTC-Swarm is run on can generally be neglected,
as no computationally intensive tasks will be executed. More
important are the servers (e.g. AWS EC2 instances and RDS
instances) that are used for the deployed containers. In order
to not exceed costs, we fined a fixed instance type for both
EC2 and RDS: t2.micro and db.t2.micro respectively.
B. Experiments
There are three main types of tests performed
1) General Benchmarking
2) Performance Tests for different Provisioning Policies
Mean Standard deviation
Creation time EC2 instance 35,8 5,27
Time for EC2 instance to react 241,2 8,9
Time to shut EC2 instance down 38,2 1,8
TABLE I. C LOUD LEVEL BENCHMARKS
3) Reliability Tests
The aim of the first type of tests is presenting the general
task execution’s as well as cloud-specific properties. Then, the
performance tests compare the different Provisioning Policies
of the system, therefore they test the automation, elasticity and
load balancing. Finally, the reliability tests evaluate the fault-
tolerance and monitoring features.
1) General benchmarks: These tests were executed to
evaluate the general properties of the system. The results were
used as a basis to discuss the results of the further tests and
to get a feeling of the response times of our application. We
measured the general benchmarks on two levels, cloud and task
level. Running these tests took roughly 4 hours of charged-cost
therefore becomes 40 euro-cent.
a) Cloud level: To test the general benchmarks from
a cloud level perspective, we ran the following test: execute
10 new EC2 instances separately. We measured the following
values:
1) Time it took for our EC2 instance to be created
2) Time it took for our EC2 instance to be responsive
3) Time it took for our EC2 instance to be shut down
Table VI-B1a shows the averages and the standard de-
viation of our measurements. As can be seen, the standard
deviation is quite low. This is because the response time of
AWS is constant.
b) Task level: To measure the task level benchmarks,
we executed the following tests 10 times: deploy each of the
three tasks on an empty EC2 instance, deploy each of the three
tasks on an instance with already an instance running, measure
the execution time of the news task.
 Mean Standard deviation
Deployment time empty EC2 0,9 0,01
Deployment time non-empty EC2 0,9 0,01
Execution time news task 523,4 10,4
TABLE II. TASK LEVEL BENCHMARKS

The results of these measurements can be seen in table

VI-B1b. As you can see, the standard deviation of the task
deployment is almost zero. This is because the deployment of
tasks is with docker is extremely constant. Since we are only
using these benchmark tests to get a feeling of the execution
times, we did not measure the deployment time for instances
with a higher CPU-utilization. This will be measured in the
following tests.
2) Performance Tests for different Provisioning Policies: Fig. 2. Average Instance utilization for the defined policies with different
In this report, we introduce two provisioning policies. The workloads
aim of this section is to evaluate these policies on the news
scraping task, since it is the only finite tasks we can apply.
We set up an experiment where we test these policies for
two different workloads. The aim of the experiment is it to
successfully execute the number of finite tasks, with a high
utilization of the instances for cost savings and low execution
time. In the time span of 10 minutes, we run several news
loading tasks at random times, 25 and 50 tasks respectively.
These workloads can show how well the policies deal with
low and high numbers of tasks to be executed. The results
are averaged over 3 runs. Experimentally we conclude that by
running 10 containers at the same time on a single instance
results in CPU utilization of around 50% and at the same time
the instance does not slow down significantly. Therefore, we
evaluate the program for 3 policy and parameters variants:
Fig. 3. Average number of instances running and executing the tasks over
• P1: Policy 1 with the maximum of 10 containers the time of execution of different workloads for the defined policies
running at the same time on a single machine
• P2: Policy 2 with maximum CPU utilization of 50%
per instance and initial container allocation of 10 of different policies and workloads. The results are presented
in the Figure 3. For the lower workload, the P1 and P2 policies
• P3: Policy 2 with maximum CPU utilization of 70% on average run the least instances, however, the difference is
per instance and initial container allocation of 10 minimal. When it comes to the high workload, the P2 and P3
have the highest number of instances running. This indicates
In the experiment, we focus on multiple aspects of the that whenever the workload is high, these policies detect high
execution. In order to evaluate the policies we measure the CPU utilization and open new instances.
average utilization, the number of instances running, the time
of execution of the workload and container and finally the cost Next, we analyze the average time of workload and con-
for each execution. The total charged time for EC2 instances tainer execution, see Figure 4. We see that in general P2 has
in this experiment is 19 hours and the cost is 1.9 euros. the lowest execution times, with one exception. This shows
that even though it requires running more instances it also
Firstly, the average CPU utilization of the running in- manages the task running in a way that does not cause a large
stances is reported in the Figure 2. Clearly, the P2 has the the execution time elongation.
lowest utilization for both workloads, while P1 and P3 have
comparable ones. This indicates that either the P2 runs more Finally, we compare the costs of employing the workloads
instances than P1 and P3 to execute the same workload with each policy. The costs are 6.7, 7.19, 7.45 cents for work-
or by employing the containers while keeping in mind the load 1 and 11.2, 13.6, 12.4 cents for workload 2, for policies
maximum instance utilization it allows the tasks to run more P1, P2 and P3 respectively. Therefore, we can conclude that
smoothly, without hitting the resource bottleneck. Since the the first provisioning policy, in general, is cheaper to employ,
news running task has the highest CPU utilization at the start, however, comes with longer execution times, while the second
by employing P2, starting too many containers at the same policy makes sure that there is a low container running time.
time may be prevented, and the slow down of the execution
3) Reliability tests: The final type of experiments measures
may be prevented.
how reliable the application is when it comes to random faults
To confirm that we also compare the average number during task execution. In this experiment, we run a streaming
of instances running in general and the average number of task and simulate an error on the application level during the
instances executing any tasks over the time of the execution instance execution, such that the pressure of the service (e.g.

Fig. 4. Average time of execution of a single container and full workload
using different policies for different workloads
tweets or trades) drops to zero. Table III provides an overview
and summarizes the test scenario.
TABLE III. OVERVIEW T EST 3
Category Reliability
Purpose Reduce data loss in case of application error
Parameters 2 (observer frequency, pressure window)
Nr. of executed tests 5 CTC environment, a pressure threshold fixed at 0 is
Outcome Data loss, SLO violations
The procedure of this test looks as follows:
1) Stopping the monitoring of a specific service which
leads to an immediate pressure drop.
2) Then, the longest time between two consecutive
tweets or trades is measured, which in comparison to
general benchmarks, may indicate how long it took
for CTC-Miner to automatically restart the instance
and how many tweets or trades are lost in that time.
3) The results are averaged over 5 tries.
This experiment focuses on the streaming tasks, since for the
news loading task, in case of a random error, the preparation
for news scraping has to be run again. This is straightforward
and therefore no concern for the company.
Using log files we are able to determine at what point in
time the Scheduling-thread observes the pressure, detects a
drop and restarts the particular container that runs the affected
service. An example of such a log of trade pressure looks as
follows:
INFO : M a r k e t gdax n o r m a l : 177.00000
INFO : M a r k e t b i t f i n e x n o r m a l : 5 1 3 . 0 0 0 0 0
INFO : M a r k e t b i t s t a m p n o r m a l : 1 7 6 . 0 0 0 0 0
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
INFO : M a r k e t gdax n o r m a l : 47.00000
INFO : M a r k e t b i t f i n e x n o r m a l : 1 9 2 . 0 0 0 0 0
INFO : M a r k e t b i t s t a m p n o r m a l : 3 3 . 0 0 0 0 0
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
INFO : No p r e s s u r e f o r m a r k e t : GDAX.
INFO : R e s t a r t i n g c o n t a i n e r :
<C o n t a i n e r : 2 d709537a4>
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
INFO : M a r k e t gdax n o r m a l : 23.00000
INFO : M a r k e t b i t f i n e x n o r m a l : 1 9 2 . 0 0 0 0 0
INFO : M a r k e t b i t s t a m p n o r m a l : 1 9 . 0 0 0 0 0
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
The log holds 4 observations (indicated with the separator line)
of 3 running services each time.
There are two parameters which have a direct impact on
the data loss: observer frequency and pressure window.
1) The observer frequency in which the scheduler ob-
serves the pressure determines how responsive the
scheduler will react upon a pressure drop.
2) The pressure is defined as the number of items
received in one minute. The pressure window is
the number of minutes that is being observed. If
the pressure window is set to 10 minutes, then, we
would sum the accounted q-minute trades (or tweets)
over the last 10 minutes, regardless of the observer
frequency. This is important because different types
of services experience naturally a different pressure
load (e.g. GDAX provides a much lower pressure (42
trades / minute on average) compared to Bitfinex (235
trades / minute on average) because we are watching
EUR trades on GDAX and USD trades on Bitfinex).
3) Note: In theory, there would be room for a third
parameter: a pressure threshold which defines at
what stage the services experiences faulty behavior.
However, for the services which are executed in the
sufficient.
The resulted downtimes, given different observing frequen-
cies are indicated in Table IV.
TABLE IV. E VALUATION OF DOWNTIME (DT) AND NUMBER OF
VIOLATIONS (V)
Service
Pressure DT: observe DT: observe DT: observe
Window every 1min every 2min every 5min
Trades(GDAX) 3min ∼180s / 3V ∼210s / 4V 300s / 5V
Trades(Bitfinex) 1min ∼75s / 2V 120s / 2V 300s / 5V
Trades(Bitstamp) 1min ∼90s / 2V 120s / 3V 300s / 5V
Tweets 5min ∼348s / 6V ∼354s / 6V ∼445s / 8V
First of all, one of the goals was to find the right pressure-
window parameter. Setting the window too low, it might lead
to empty pressure, although the system is running normally.
Setting the factor too high results in having a much longer
response time because the system will think there is still
pressure whereas the pressure is actually fading off. Indicated
in the table are the parameters which, through empirical
testing, provide just enough margin to not result in 0-pressure
due to naturally low pressure.
As it can be seen from the results, the downtime obviously
increases with the frequency of observing pressure, up to the
point where the downtime equals the frequency, which implies
that waiting time was added by having a too large observing
frequency. We can further see that with a frequency of one
minute, the downtime was only slightly more than the incor-
porated pressure window, which implies that the observing rate
is chosen well and very little time is wasted while the service
was not working. Regarding the violations, it can be seen that
for trade-services with high pressure (Bitfinex), a frequency of
two-minutes is actually sufficient, whereas the lower pressured
ones require a one-minute observing frequency. Likewise, the
violations occurred during a pressure drop in the tweet service
are the same whether we observe every one- or two-minute.
In favor of saving computational power, the higher frequency
is better.
In addition to the Scheduler-thread which provides reliabil-
ity on the application level, we have also implemented methods
to restart failed services on container level. Depending on the
selected provisioning policy, as explained in V-B, the amount
of time until the Manager attempts to restart the particular
container can change. That is because, for example, P1 needs
to wait for CPU utilization information. However, tests have
shown that there is no significant difference and therefore
consider any of the policies as supportive in terms of reliability.
However, as an infrastructure provider, we have to expect the
provided applications, provided as docker containers, to be
fully functional. Hence, we don’t expect these kind of errors
in a well-developed applications running in the cloud and see
no need to compare the difference introduced by the policies
we provide. The main importance is, that if a container was
unable to launch, an attempt to restart will follow.

VII. C ONCLUSION
In this report, we designed and evaluated the cloud resource
manager CTC-Swarm. CTC-Swarm is able to orchestrate the
execution and the scheduling of 3 different types of data
extraction tasks. The application is also able to scale in
resources whenever necessary. We described the architecture
and implemented several features.
We evaluated CTC-Swarm with 3 types of tests: benchmark
tests, tests on different provisioning policies and reliability
tests. The tests show that, when choosing a provisioning policy,
a trade-off is done between increasing the performance and
lowering the costs. The tests also show that, in order the
minimize SLO violations, the pressure window of streaming
services needs to be chosen based on the different pressure
loads of services that are streamed.
While developing a cloud-based application we came to
experience that it is very difficult to think of all the edge
cases. For example, the simple task of starting an instance
is not that trivial considering that the amount of time until the
instance is fully ready for containers to be deployed on can
differ a lot. We also noticed that the computing power that
was provided over our free-tier AWS instances can vary. We
solved this problem by taking the averages of the experiments
and introduced extensive error handling within the code.

R EFERENCES
[1] S. Goyal, “Software as a Service, Platform as a Service, Infrastructure
as a Service A Review,” International journal of Computer Science
Network Solutions, vol. 1, no. 3, pp. 53–67, 2013.
[2] “Boto 3 documentation.” [Online]. Available:
https://boto3.readthedocs.io/en/latest/
[3] “docker-py.” [Online]. Available: https://github.com/docker/docker-py
 C LASS D OCUMENTATION
A. AWSManager
NAME
aws manager

CLASSES
builtins . object
AWSManager

c l a s s AWSManager ( b u i l t i n s . o b j e c t )
| Methods d e f i n e d h e r e :
|
| init ( self )
| I n i t i a l i z e s e l f . See h e l p ( t y p e ( s e l f ) ) f o r a c c u r a t e s i g n a t u r e .
|
| deleteRDSInstance ( s e l f )
|
| getAllInstances ( self )
|
| getAllInstancesReady ( self )
|
| getInstanceById ( self , id )
|
| getInstanceUtilization ( self , instanceId )
| D e t e r m i n e s l a t e s t known a v e r a g e CPU u t i l i z a t i o n u s i n g CloudWatch
|
| R e t u r n s o n l y t h e l a t e s t known o b s e r v e d t i m e s t a m p .
|
| getInstancesById ( self , ids )
|
| getRDSInstance ( s e l f )
|
| isInstanceReady ( self , instanceId )
|
| l a u n c h I n s t a n c e ( s e l f , name )
|
| p r o v i s i o n I n s t a n c e ( s e l f , i n s t a n c e T y p e = ’ t 2 . micro ’ )
|
| p r o v i s i o n R D S ( s e l f , i n s t a n c e C l a s s = ’ db . t 2 . micro ’ , u s e r , p a s s w o r d )
|
| terminateAllInstances ( self )
|
| terminateInstance ( self , instance )
|
| terminateInstances ( self , ids )
|

B. DockerManager
NAME
docker manager

CLASSES
builtins . object
DockerManger

c l a s s DockerManger ( b u i l t i n s . o b j e c t )
| Methods d e f i n e d h e r e :
|
| init ( self )
| I n i t i a l i z e s e l f . See h e l p ( t y p e ( s e l f ) ) f o r a c c u r a t e s i g n a t u r e .
|
| close when empty ( s e l f , i n s t a n c e )
|
| deployCandleProcessor ( s e l f , containerName )
| Available containers :
| c t c −t r a d e −c a n d l e s −b i t s t a m p −1m
| c t c −t r a d e −c a n d l e s −gdax −1m
| c t c −t r a d e −c a n d l e s −b i t f i n e x −1m
| c t c −t r a d e −c a n d l e s −b i t f i n e x −15m−s h i f t e d
| c t c −t r a d e −c a n d l e s −gdax −15m−s h i f t e d
| c t c −t r a d e −c a n d l e s −b i t s t a m p −15m−s h i f t e d
| c t c −t r a d e −c a n d l e s −gdax −60m−s h i f t e d
|
 | deployNews ( s e l f , s e r v i c e N a m e = ’ c t c −news ’ )
|
| deployTrades ( s e l f )
|
| deployTwitter ( self )
|
| getAllRunningContainers ( self , instances )
| L i s t s running c o n t a i n e r s fo r a l i s t of i n s t a n c e s .
|
| A z i p l i s t i n t h e form o f : [ ( i n s t a n c e , [ c o n t a i n e r s ] ) ] i s r e t u r n e d .
|
| getClient ( self )
|
| getClientByInstance ( self , instance )
|
| getImageName ( s e l f , c o n t a i n e r , v e r s i o n = F a l s e )
|
| getNumOfRunningContainers ( s e l f , i n s t a n c e )
|
| getRunningContainers ( self , instance )
| L i s t s t h e r u n n i n g c o n t a i n e r s f o r an i n s t a n c e
|
| notifyWhenLessThanTargetNumOfContainers ( s e l f , instance , t a r g e t )
|
| restartContainer ( self , container )
|
| setClient ( self , client )
|
| setClientByInstance ( self , instance )
|
| setClientByLowestContainers ( self , instances )
|
| toEnviron ( self , l )
| C o n v e r t s and r e s o l v e s a l i s t o f e n v i r o n m e n t v a r i a b l e s t o a d o c k e r
| environment entry .
| E . g . [ ’ CTC DB ’ ] r e s u l t s i n : [ ’ CTC DB= v a l u e ’ ]

C. StatisticsManager
NAME
statistics manager

CLASSES
builtins . object
StatisticsManager

class StatisticsManager ( builtins . object )

| Methods d e f i n e d h e r e :
|
| init ( s e l f , awsManager=None )
| I n i t i a l i z e s e l f . See h e l p ( t y p e ( s e l f ) ) f o r a c c u r a t e s i g n a t u r e .
|
| g e t C u r r e n t N e w s P r e s s u r e ( s e l f , minutes =1)
|
| g e t C u r r e n t T r a d e P r e s s u r e ( s e l f , market , minu tes =1)
| Note : T r a d e s a r e s t a m p e d w i t h UTC and t h e f i e l d i s a VARCHAR.
| T h e r e f o r e we c a n n o t u s e MySQL t i m e s t a m p f u n c t i o n a l i t y , however ,
| we c a n f i l t e r m i n u t e s u s i n g e . g . 20171025 2 1 : 0 7 : %
|
| g e t C u r r e n t T w e e t P r e s s u r e ( s e l f , minutes =1)
|
| g e t N e w s P r e s s u r e ( s e l f , min utes =1)
|
| g e t P r e s s u r e ( s e l f , c u r s o r , minutes =1)
|
| getTimeNow ( s e l f )
|
| g e t T r a d e P r e s s u r e ( s e l f , market , minut es =1)
|
| getTradeTable ( s e l f , market )
| Finds t r a d e t a b l e according to market s h o r t t a g :
| − gdax
| − bitfinex
| − bitstamp
|
 | The r e a s o n f o r t h i s i n c o n v e n i e n c e i s t h e f a c t t h a t t h e t r a d e w a t c h e r
| l i b r a r y i n u s e a p p e n d s t h e d a t e when t h e j o b i s s t a r t e d i n h a r d−c o d e d
| f a s h i o n . E . g . ” e x c h g d a x b t c e u r s n a p s h o t 2 0 1 7 1 0 2 5 ” . Thus we f i n d t h i s
| t a b l e by l o o k i n g up ” gdax ” .
|
| g e t T w i t t e r P r e s s u r e ( s e l f , minutes =1)
|
| reportCpuUtilizations ( self )
|
| reportNewsPressure ( s e l f )
|
| r e p o r t T r a d e P r e s s u r e ( s e l f , market )
|
| reportTweetPressure ( self )
|
| −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
| Data d e s c r i p t o r s d e f i n e d h e r e :
|
| dict
| dictionary for instance variables ( i f defined )
|
| weakref
| l i s t o f weak r e f e r e n c e s t o t h e o b j e c t ( i f d e f i n e d )

DATA
db = <peewee . MySQLDatabase o b j e c t >
news db = <peewee . MySQLDatabase o b j e c t >
t r a d e d b = <peewee . MySQLDatabase o b j e c t >
t w i t t e r d b = <peewee . MySQLDatabase o b j e c t >
 T IME SHEET
D. Report and Code
• the total-time = 200h
• think-time = 50h
• dev-time = 60h
• xp-time = 40h
• analysis-time = 10h
• write-time = 20h
• wasted-time = 20h

E. Experiments
1) Experiment 1:
• total-time = 5
• dev-time = 4
• setup-time = 1
2) Experiment 2:
• total-time =15
• dev-time =10
• setup-time =5
3) Experiment 3:
• total-time = 15
• dev-time = 10
• setup-time = 5