Yorkie

Self-Hosted Server

Sometimes you may want to run your own Yorkie Server. This page explains how to run a Yorkie Server.

To run a Yorkie Server, you need to install the CLI. If you haven't installed it yet, please refer to CLI.

Running a Server via CLI

Let's start a Server with the CLI. You can start a Server with the following command:

```bash
$ yorkie server


backend created: id: c6ljcdl94818baveba8g, rpc: localhost:11101: db: memory
serving RPC on 11101
serving profiling on 11102

The Server handles administrative requests such as project maintenance, and it continues to run until it's told to quit with termination commands such as Ctrl+C. The Server stores its data using an in-memory DB, which does not provide durability by default.

Persistence

If you start the Server with a MongoDB address, you can permanently save the data stored by Yorkie.

To start MongoDB, docker-compose -f docker/docker-compose.yml up --build -d in the project root.

Then, start a Yorkie Server with --mongo-connection-uri flag to connect the MongoDB.

$ yorkie server --mongo-connection-uri mongodb://localhost:27017


MongoDB connected, URI: mongodb://localhost:27017, DB: yorkie-meta
backend created: id: c6ljibt948102kkt9neg, rpc: localhost:11101: db: mongodb://localhost:27017
serving profiling on 11102
serving RPC on 11101

Server for Web

Server uses gRPC to provide an API that Clients can connect to. Since it is currently impossible to implement the HTTP/2 gRPC in some browsers, Envoy is required for web. For more details: gRPC-web

This page shows how to start the Server for web. Overall structure is as follows:

Browser            Envoy                  Server
┌────────┐         ┌──────────────┐       ┌───────────┐
│gRPC-web├─HTTP1.1─┤gRPC-web Proxy├─HTTP2─┤gRPC Server│
└────────┘         └──────────────┘       └───────────┘

Configuring Envoy by hand with its config file is cumbersome, but using Docker Compose makes it easy.

If docker compose is not installed, install it : Install Docker Compose

First, download all manifests files from docker folder. Then, execute the following command in the folder containing the downloaded file.

$ docker-compose up --build -d


Starting yorkie ... done
Starting envoy  ... done

This will launch Yorkie(Server) and envoy containers on your environment.

$ docker ps


IMAGE                      COMMAND                  PORTS                                  NAMES
grpcweb:envoy              "/usr/local/bin/envo…"   0.0.0.0:8080->8080/tcp                 envoy
yorkieteam/yorkie:latest   "yorkie server --ena…"   0.0.0.0:11101-11102->11101-11102/tcp   yorkie

Then, the ports of the services are bound to the host environment.

  • 8080: gRPC-web port for RPC Server, including SDK and Admin
  • 11101: gRPC port for RPC Server, including SDK and Admin
  • 11102: HTML port for Profiling Server
Server stores its data using an in-memory DB, which does not provide durability. If you want to store data permanently, please refer to Running Server With MongoDB

Now, let's create a Client with address https://api.yorkie.dev.

1const client = new yorkie.Client('https://api.yorkie.dev');
2await client.activate();

Next, let's take a look at the JS SDK.

Monitoring

Server exports metrics under the /metrics path on its profiling port. The metrics can be fetched with curl:

$ curl http://localhost:11102/metrics


yorkie_server_version{server_version="{{site.version}}"} 1
# HELP yorkie_pushpull_received_changes_total The total count of changes included
# TYPE yorkie_pushpull_received_changes_total counter
yorkie_pushpull_received_changes_total 6
...

This metrics can be collected from Prometheus.

Running Prometheus and Grafana is the easiest way to monitor Server's metrics.

First, download all manifests files from docker folder. Then, start the applications with docker-compose:

$ docker-compose -f docker-compose-monitoring.yml up --build -d


Creating prometheus ... done
Creating grafana    ... done

Now, Prometheus will collect Server metrics every 10 seconds.

Grafana has a built-in Prometheus support; just add a Prometheus data source:

Name:   prometheus
Type:   Prometheus
Url:    http://localhost:9090
Access: proxy

Then, import the default Yorkie dashboard template and customize it. For instance, if Prometheus data source name is my-prometheus, the datasource field values in JSON also need to be my-prometheus.

Sample dashboard:

dashboard

Sharded Cluster

In a production environment, a server cluster is necessary to handle large workloads while ensuring high availability, reliability, and scalability.

Yorkie provides a sharded cluster in Kubernetes environments to support production environments.

This page describes how to set up sharded cluster. An example of the sharded cluster is as follows:

Sharded Cluster
                               ┌────────────┐
                               │            │
                               │  ┌──────┐  │
                           ┌───┼─►│Server│  │
                           │   │  └──────┘  │
                           │   │            │
┌──────┐      ┌────────┐   │   │  ┌──────┐  │    ┌───────┐
│Client├──────┤L7 Load ├───┼───┼─►│Server│  ├───►│MongoDB│
└──────┘      │Balancer│   │   │  └──────┘  │    └───────┘
              └────────┘   │   │            │
                   ▲       │   │  ┌──────┐  │
                   │       └───┼─►│Server│  │
              ┌────┴───┐       │  └──────┘  │
              │Service │◄──────┤            │
              │Registry│       └────────────┘
              └────────┘
  • L7 Load Balancer: It is responsible for routing requests to servers based on the computed request’s hash key and consistent hashing algorithm.
  • Service Registry: It is responsible for storing metadata and configuration settings for Yorkie servers.
  • MongoDB: It stores the data of Yorkie. It can be configued to use sharding, just like Yorkie sharded cluster.

Yorkie provides Helm Chart, which is a package manager for Kubernetes, to easily deploy Yorkie cluster on Kubernetes with above components configured.

Below are guides for deploying Yorkie cluster on Kubernetes with Helm Charts on various platforms.

Also, you can additonally deploy Yorkie cluster addons on Kubernetes with Helm Charts on your own environment.

Yorkie cluster addons include:

  • Monitoring: Provides Yorkie metrics and logging via Grafana dashboard.
  • GitOps: Provides Yorkie Continous Delivery via ArgoCD.

Below is guide for deploying Yorkie cluster addons on Kubernetes with Helm Charts.