Creating Compute Continuums across Platforms

What is Colonies?

Colonies is an open-source project initiated by RISE AB, designed to streamline execution of computational workloads across platforms (IoT, edge, cloud, and HPC). Colonies provides a distributed runtime environment, known as a Colony, enabling development, deployment, and execution of microservices ubiquitously over the Internet. This allows for seamless integration with other systems and creation of Compute Continuums.

A Colony is a loosely-coupled collection of distributed microservices called Executors. A zero-trust security protocol based on the principle trust, but verify enables the Executors to operate as a unified entity. Each Colony is governed by a private key that serves as a central authority, analogous to one ring to rule them all, making it possible to control a Colony as if it was a single machine.

A fundamental concept is the notion of processes which contains meta-information about computations to be carried out by the Executors. Colonies provides a protocol that enables Executors to publish instructions that are subsequently assigned to other Executor members of the same Colony. In this way, the Colonies serves as a job broker, almost like an employment agency for people. By linking the instructions into Directed Acyclic Graphs (DAGs) it becomes possible to seamlessly execute workloads across platforms.

See the Colonies Github repo and the Python SDK repo for examples how to use it.

Use cases

Edge computing. Implement and run an Executor on e.g. a drone to offload computation to an edge server.
High-performance computing. Develop hybrid workflows running both on HPC (e.g. Slurm) and Kubernetes.
Serverless computing. Develop a FaaS platform to run functions across platforms.
Distributed processing. Build scalable and robust data pipelines on Kubernetes to process e.g. Earth Observation data.

Key features

Colonies builds on Etcd and is scalable and robust. Processes not finishing in time are automatically re-assigned to other Executors.
A built-in crypto-protocol ECDSA (Elliptic Curve Digital Signature Algorithm) provides identity management and secure process execution.
A HTTP-based protocol ensure that Executors can reside anywhere on the Internet, even behind firewalls. The Colonies server never establish connections directly to an Executor.
Multi-step workflows are supported. Workflow templates (Generators) to automatically spawn new workflows based on external events or timeouts.
Cron to schedule workflows to be executed in the future.
Traceability. Full process execution history is stored and can be used for auditing and debugging.
A powerful CLI and a web-based dashboard.
SDKs in Rust, Python, Julia, Haskell, JavaScript, and Golang.

Contact

Johan Kristiansson, RISE AB.