Console guide

Services and databases

The state of the services, their detail and logs, the topology with its nodes, cells, routing, storage and events, and the database upgrade.

Check the state of the services

Status lists the eleven services of the installation, each with its icon, name, description and health address.

  1. Read the light of each row: Operational when the service answers its health check within 6 seconds, Unreachable when it does not.
  2. The checks repeat by themselves every 30 seconds while Status or the overview is in view, and rest while the browser tab is hidden; Refresh now repeats them at once and updates Last checked.
  3. Click the state of a service to open its health detail, or its logs icon to read its logs.
  4. Served runtimes lists what the browser runs from files of the front end, such as the video effects, with Served or Not served.

Tip When a service is Unreachable, its logs often say why.

Read the health detail of a service

Each service has a page with its health detail: an overall state, and a card for each of its subsystems.

  1. Open it by clicking the state of the service in Status.
  2. The pill at the top says Operational, Degraded or Down.
  3. Each card shows a subsystem with its light, its name, a summary and its figures.
  4. The page does not refresh by itself: click Refresh now. The path at the top of the page leads back to Status and to the console.

Tip Degraded means the service works but one of its parts does not: the card of that subsystem says which.

Read the logs of a service

Each service has a page with its logs, in a console that looks like a terminal.

  1. Open it with the logs icon of the service in Status.
  2. Choose the day in Day: the list shows the days that have logs, most recent first.
  3. The console shows the last lines of that day, coloured by level — ERROR, WARN, INFO, DEBUG, TRACE — and scrolls to the last one.
  4. Older lines were omitted. means the server cut the oldest lines; No log entries for this day. means the day has no file; File logging is not enabled for this service. means the service does not write logs to file.

Tip Refresh now reads the chosen day again.

Read the topology of the installation

Under Topology, the console shows the installation as the topology registry and the announcements of the services describe it. Its Overview says in six figures how it stands, which roles each node holds and which services announced themselves.

  1. Read the six figures: the nodes, the Rust services announced, the SQLite cells, the destinations of the bytes, the last round of the copies and the alerts. The nodes and the alerts are in the console's colour when all is well, in ochre or in red when something needs attention; a dash means the figure is not there yet.
  2. Last hour and Today, beside the title, show for each figure that has a series the worst value of the period and the time it was reached; the figures without a series stay those of now and say so. Now goes back to the present state.
  3. Roles × nodes has a row for each role and a column for each node: a filled square is a role switched on on that node. The name of a node opens its page; Show more adds the columns of the next nodes, and the table scrolls inside its card.
  4. Under the table three checks — no role without nodes, no mixed versions, all answer — turn into the alert in words when they fail.
  5. Announced services lists every service with the name of its binary, how many instances announced themselves, the version and a dot, green when they all answer; the first row is the ingress, caddy.
  6. The section reads again by itself every 30 seconds while the console is open, and rests while the browser tab is hidden; Refresh now reads it at once.

Tip In front of an alert, open the page of the node it names: its roles and services say which service does not answer. Different versions usually mean an upgrade still under way on some node.

Watch the nodes

Nodes lists the nodes the topology registry knows, ordered by role with the new ones at the top: for each one its address, its roles, its state, the CPU and memory it announced, the cells it holds and the age of its last announcement. Each row opens the page of the node, and a node that joins the installation appears by itself.

  1. Choose a state above the list — All, Online, New, Draining, Offline, and Unreachable when there is at least one —: beside each, the number of nodes. Filter the nodes, beside the title, searches the name and the identity.
  2. A new node stands at the top, with the ochre dot: it announced itself and has no work, and its row offers Bring online and Reject.
  3. The CPU beyond 70% is in ochre, and so are the cells beyond 80% of the declared capacity; a dash among the cells means the node does not hold the data role. When the installation has more nodes than the list shows, Show more adds the next ones.
  4. Weights and capacities, beside the title, opens the weights of the roles and the capacities of the node you choose: a weight goes from 1 to 1000, and 100 is the role's normal node; an empty capacity field leaves it undeclared. Save takes effect only if nobody changed the node in the meantime.
  5. The page of a node has the seven roles on the left: the one switched on carries the bar, and the services of the role are tokens lit when they run there and answer; a token opens, further down, the configuration and the resources of that service. On the right, the Effective configuration, grouped by origin: the seed of the first start, what the topology decides, what the configuration says, the state of the node and the secrets, redacted.
  6. Configure changes the name, roles, weights and capacities of the node; beside it stand the actions its state allows, each asked first.
  7. Resources shows the host of the node: in Now the CPU, memory, network, disk, monthly traffic and processes, refreshed every 5 seconds by the identity service of the node; in Last hour and Today the CPU and the memory minute by minute, as the services announced them.

Tip A secret never leaves the process that holds it: the page shows only its name, where it comes from and the redacted tag. The configuration changes only when a service restarts.

Bring nodes online, drain and retire them

Every state of a node offers only the actions the topology registry admits from it: on the row of a new node in Nodes, and in the head of the page of every node, beside Configure. Each action asks first, naming the node and saying what will really happen; what nothing undoes — Reject, Retire and Declare failed — you confirm by typing the identity of the node.

  1. New: Bring online gives the node work — new cells, the cells the topology moves, rooms, render jobs, mounts —, and the topology accepts it only once it hears the node: at least one of its services announces itself or, for a node with only the Ingress role, which has no service to announce, its ingress answers. Until then the button stays off and says why. Reject is the retirement of the node: it leaves every view, its identity and its epoch are never used again, and a process of it that tries to join again stays out.
  2. Online: Drain stops the node receiving work at once and moves its cells to the least loaded nodes, at most 4 at a time; a bucket it mounts is mounted on another storage node, while a folder on its disk stays and you move it from Storage. When nothing is left on it and its requests in flight have finished, within 30 seconds, it goes offline by itself. An unreachable node is not drained.
  3. Draining: the card Drain in progress follows the drain every 5 seconds while it is in view — who started it and when, the cells still on the node out of those it had, with their kinds, the open moves by phase, and what holds it back, in words: cells whose move was refused or cancelled, kinds no node online can receive, a folder on its disk, services that have not closed the census of its cells, a node that has gone silent —, or Nothing holds it back. Interrupt the drain brings the node back online: the moves of the drain not yet assigned are cancelled and their cells stay on it.
  4. Unreachable, or draining and silent: Declare failed, never offered for a node that holds the Control role. The confirmation first says the failover policy in force, read at that moment. The topology accepts the declaration only once the node's lease has been expired for more than 10 seconds, and refuses it before then: with automatic failover it declares the node failed by itself at that moment, so confirming never does it sooner; with manual failover your confirmation is the only way its cells restart elsewhere. The policy is a key of Routing.
  5. Failed: the card Recovery of the failed node follows its cells as they restart on other nodes from their last copied instant; the mounts it held are lost, and a destination nobody serves since then is named until you mount it on another storage node. Bring online stays off, saying why, until the topology hears it again: then it stops the recovery, and what has already restarted elsewhere stays there.
  6. Offline: Bring online gives the node work again; Retire removes it for good, as Reject does for a new node, and the topology refuses while the node still holds cells, moves, mounts or leases.
  7. A success is said in a notice and the page reads the node again: its head says where the node went. A refusal is said under the head, in the words of its reason.

Tip The topology never declares failed its own node or one that holds the Control role, and it refuses to drain the last node online with the Control, Identity or Data role, or the last one that writes an active destination: the refusal names the role or the destination.

Read the cells

Cells shows where the data of the installation lives: every cell is a SQLite database with one owner node. The line under the title counts the cells, their kinds and the data nodes; the chart, the kinds and the tables say how the cells are spread and how large they are.

  1. Choose what the chart shows beside the title: By node draws a column for each data node and the line of the capacity in cells each node declares — a node past 80% of its capacity is in ochre, a new node, still without work, in the colour of the console —; By kind draws a row for each kind of cell, with the nodes that hold it; By organization the organizations with the most cells, with their nodes and, for a divided organization, the services in which its entities have cells of their own.
  2. Cells per kind has a tile for each kind, with its count and its size. A size is the bytes of the cells' databases, what a move transfers: the services measure them only while someone looks at the section, so on a first visit the sizes are arriving and show with the next announcements; the line under the kinds then says from the census of how many nodes they come and how old it is. A size measured on part of the cells says how many were measured.
  3. Largest cells lists the cells with the largest databases and Most recently written those written last, each with the node that holds it. Nobody counts the writes of a cell: the most active cells are shown as the most recently written ones.
  4. Cells and sizes per node is the chart by node in words: for each node its state, its cells, its capacity with the share used or not declared, the size of its cells, the age of its census and the cells set aside. A node that comes back after a failover sets aside a cell whose copies lacked some of its writes: it waits for an operator, and the table says so.
  5. The section reads again by itself while it is in view — every 10 seconds while something arrives or moves, every 30 seconds otherwise —; Refresh now reads at once.
  6. On a narrow screen the cards stand one under the other, and the chart by node scrolls sideways inside its card; for a screen reader, the figures of the chart are those of the table Cells and sizes per node.

Tip The counts come from the topology registry and are exact; the sizes come from the last census of the services, and nobody measures the disks while nobody looks.

Rebalance the cells

Rebalancing plan shows what the rebalancer would move to even out the data nodes: only the next batch, because nobody measures the load of a single cell and the whole rebalancing cannot be foreseen. Beside it, the switch of the automatic rebalancer and the card Why, followed by the name of a node, which says where new cells go and why.

  1. Read the badge of the plan: dry run is the next batch as the rebalancer computes it now; authorized, once an operator has applied it; in progress, with how many moves of the batch have arrived; applied, once they have. The batch is said by its cells, its bytes, the nodes it leaves and those it reaches, then a row for each kind and route.
  2. Read under the plan what the rebalancer is doing and why — balanced, waiting for the gap to last, off, settling after the previous batch, out of budget —, with the gap between the data nodes and the threshold, and the budget of the last hour: moves, bytes and moves at once.
  3. Apply the plan, offered while a rebalancing is due, authorizes the rebalancer: at its next round, within seconds, it asks for the batch as it computes it then, which may differ from the one shown, and always within its budget. It holds even with the automatic rebalancer off, and it does not wait for the minimum duration. The badge then follows the batch, and the line under the plan says who authorized it and what the rebalancer did.
  4. When the rebalancer would no longer ask for a batch — fewer than two data nodes with a score, a drain under way, a gap within the threshold, the previous batch settling, the budget spent, sizes still missing, no measured cell that can move —, the topology refuses and the card says why, in words.
  5. Automatic rebalancer: Turn off keeps the plan computed and shown here but moves nothing by itself — a plan can still be applied, and drains and recoveries move their cells as always —; Turn on lets the rebalancer ask for batches by itself when the gap stays over the threshold for the minimum duration. Both ask first.
  6. Read the card Why: the load score of the node the new cells of the data services go to — the free share, higher is freer — with its reason, the measures that make it, each with its share in use and its weight, the freest candidates and the hysteresis, which keeps new work on the node chosen before while it stays close to the best.

Tip Drains and operators' moves do not count in the budget of the rebalancer. Threshold, minimum duration, budget and when a cell counts as cold are keys of the policy, in Routing.

Move a cell and follow the moves

Moves in progress lists the open moves of the cells, whoever asked for them — the rebalancer, a drain, a recovery or an operator —, and from Cells you can move a cell yourself to the data node you choose.

  1. Read each move: the cell by its kind and key, the node it leaves and the one it reaches, its phase — In transfer, Copied, Assigned — with a bar, who asked for it and when; a step that keeps failing is said under the phase, with how many times.
  2. Stop interrupts a move in transfer or copied that an operator or the rebalancer asked for: the cell stays active on its node at the same epoch, and what the destination restored so far is discarded; a cell whose move of the rebalancer you stop is not chosen by it again for a day. A move of a drain or of a recovery has no Stop and says whose it is: a drain is interrupted from the node's page, a recovery by bringing the node online again. An assigned move can no longer be stopped.
  3. Move… on a row of Largest cells or Most recently written opens the dialog with the cell and the node that holds it: choose the destination among the other data nodes online and press Move.
  4. Move a cell…, beside the title, asks for the kind, the key as the registry writes it — an organization, a user, or organization/fingerprint for an entity — and the destination node, and reads which node holds the cell before asking. The search of the console finds Move a cell… too, and opens its dialog.
  5. The confirmation names the cell, the node it leaves and the one it reaches: its writes wait while the source hands it over and the destination restores it, the source removes its files after the grace, and a move of an operator does not count in the budget of the rebalancer.
  6. A refusal is said in the dialog, in words: a cell already moving or already on that node, a destination that takes no new cells or lacks the role, too many open moves of that service on the node, a cell without a place of its own — an entity of an organization that is not divided moves with its organization.

Tip While a cell moves its writes wait and are asked to retry, and its epoch refuses a late one: nothing is lost.

See where new work goes

Routing says where new work goes and why: the policy of the topology, the score of every node, the node chosen for each assignment and the last choices with their motivation. The line under the title counts the nodes and the assignments, and recalls that a score is the free share: higher is freer.

  1. Scores lists every node in every role that receives work — Data, Compute, Storage — with its state, the weight of the role when it is not 100, its score and its measures, each with its share in use and its share of the weights of the role; a measure past 70% for the CPU, or 80% for the others, is in ochre. A node that has not announced a load yet says so; Load more nodes reads the next ones.
  2. Chosen nodes says where new work goes now — the new cells of each data service, the new rooms of the calls, the render jobs, the reads and the writes of each mounted destination —, each with its node, its reason and since when it is chosen. An assignment no node online can receive is named under the list.
  3. Open Why under a row to read the motivation of the choice: the score with its reason — the freest, the only candidate, the node that asks or the one chosen before within the hysteresis —, the measures with their share in use and their weight, the freest candidates and the hysteresis. A motivation left open stays open when the section reads again.
  4. Recent choices lists the last choices, newest first: the time, the assignment, the chosen node with its reason, and the cell and the node that asked when there are any. Assignment narrows them to one assignment, and Load more reads older ones. The motivation of a past choice shows the measures of that moment, not the weights it was made with, which the registry does not keep.
  5. The section reads again every 30 seconds while it is in view; Refresh now reads at once.

Tip How many choices the registry keeps is a key of the policy, Choices kept: Recent choices shows only those kept.

Change the policy of the topology

Policy, in Routing, holds the keys of the topology in three parts — Load, Rebalancing and Nodes —, each with its value, its default when the value differs from it, and who changed it and when. Change… beside a key opens its form.

  1. Load: Weights of the measures, how much each measure counts in the score of a data, compute and storage node — 0 leaves a measure out, and each role keeps at least one above zero; Saturation, where a node counts as full in connections, requests in flight and open cells; Hysteresis, how close to the best score the node that asks, or the one chosen before, keeps new work; Choices kept, how many choices the registry keeps for Recent choices.
  2. Rebalancing: Automatic rebalancer, the same switch as in Cells; Threshold of the gap, the gap between the scores of the data nodes that starts a rebalancing, which ends at half of it; Minimum duration, how long the gap must last before the rebalancer asks for a batch by itself; Budget, the moves and the bytes per hour and the moves at once; Cold after, after how long without writes a cell is cold — the rebalancer moves the cold ones first, and 0 makes every cell cold.
  3. Nodes: Joining nodes, whether a node that joins goes online by itself or stays new until an operator brings it online from Nodes; Failover, whether the topology declares a silent node failed by itself once its lease has been expired for more than 10 seconds, or waits for an operator to do it from the page of the node — until then the cells of the node cannot be reached.
  4. In the form each field says the bounds the topology accepts, and a wrong value is said under its field and not sent. Save asks first, saying what the change really does — where new work goes, what the rebalancer may ask for, who declares a node failed —, then writes the key and reads the section again.
  5. Every change is written in the topology registry with your name: the line under the key says who changed it and when, and Events shows it.

Tip The console checks every value against the bounds of the topology and does not write a value outside them; the topology refuses a value only when it is a topology of another version, whose bounds differ, and the refusal is said in the words of the section.

Trace a request

Request traces, in Routing, lists the collection windows and the traces they collected. A request is traced only inside a window an administrator opens: outside one, no service records anything about it.

  1. Open a window… asks what to collect — a user, by the pseudonymous identity the events show; an app, by the client_id its token is meant for; or a path as the gateway receives it, starting with /api/ —, for how many minutes, at most 30, and why. The reason stays in the events of the topology with your name, and at most four windows are open at once.
  2. The services start collecting at their next announcement, within 5 seconds, and stop when the window ends or when you close it early with Close.
  3. Every trace collected is a row: when, the request, its outcome, the app and the user, the time in the system. The row leads to the page of the trace.
  4. The page of the trace shows the path: a row per stage — the entry through the gateway, the routing to the node that owns the cell, the checks of the token, the cell in SQLite, the bytes of the storage —, each with its node and service, its time and its outcome, then the time in the system. While more stages may still arrive, the page reads the trace again by itself.
  5. Access token shows what the service that verified the token saw: the header and the payload, the first characters of the signature, and every check with its rule. The requests refused with the same token in the window are listed with the reason in words, and the chips under the path are lit only when a probe of the identity service confirms them.
  6. Copy the trace copies the page as text, without the signature.

Tip The token is never kept: a trace holds its decoded header and payload, the outcome of each check and the first 12 characters of the signature. Traces live only in the memory of the topology, at most an hour after their window ended.

Read the storage

Storage shows where the bytes of the installation live: a column for each class — user files, attachments, previews and database copies — with its destinations, how full each one is against its thresholds, and the storage nodes that mount them and how many bytes they serve.

  1. Read each column: the outlined destination is the active one of its class, the one that receives the new writes; under each destination you see its kind (disk or S3), its folder or bucket, the meter against its space threshold and the nodes that mount it.
  2. The tick on the meter marks where the rotation prepares the next destination; past it the meter turns ochre, and at the threshold the next destination ready in the class becomes the active one by itself.
  3. Read Mounts: a row for each storage node, with the destinations it mounts and the state of each mount, the epoch of its last change and the bytes it serves per second.
  4. Read the storage events at the foot of the page: who registered, activated, mounted or emptied what, newest first; Show more reads the older ones.
  5. The page reads again every 30 seconds while it is in view; Refresh now reads it at once.

Tip A destination without a threshold shows only the bytes counted since it was registered: the counters are indicative, and the rotation decides only on a count the storage node makes by listing the destination.

Govern the destinations

From Storage you add a destination — a folder on the disk of a storage node or a bucket of a service compatible with S3 — and you activate, seal, empty, retire it and change its thresholds; Rotation sets when the next destination must be ready and when it becomes the active one.

  1. Press Add destination: give it a name and the class it will serve, choose disk with an absolute folder or S3 with endpoint, region, bucket and the credential reference — the name the storage node resolves from its own secrets, never the credential —, the storage node that mounts it and, if you want, its thresholds. The node verifies it before it serves; until then it is new.
  2. Open the menu of a destination with its button: it offers only what its state allows. Activate for a class makes it the active one of that class; Seal hands every class it serves to its next destination, the one the rotation would choose, and the confirmation names it.
  3. Thresholds… sets the space, in GB or TB, and the objects; an empty field means no threshold. The rotation weighs new thresholds at its next round.
  4. Empty moves the bytes of a sealed destination into the active one of their class, in the background; Suspend the emptying stops it. Retire removes a ready destination that has never received bytes: confirm it by typing its name.
  5. Press Rotation to set the preparation and the rotation, in percent of the threshold, and whether the destination the rotation seals empties itself into the new active one.

Tip Every action asks first, naming the destination and saying what will happen; when the topology refuses, the reason appears beside the destination, in words.

Mount, unmount and empty

A destination serves its bytes from the storage nodes that mount it: Mounts shows a row for each storage node, and from the menu of a destination you mount it on a node, unmount it from one or move a mount to another node.

  1. Read Mounts: each storage node with the destinations it mounts, the state of each mount — a green dot serves, an ochre one is being verified, a ring is being unmounted —, the epoch of its last change and the bytes it serves per second.
  2. Choose Mount on… in the menu, or the button on a destination nobody serves: pick a storage node, new or online, and the mode. The node verifies the folder or the bucket before it serves.
  3. Choose Unmount from a node: the node stops admitting new operations, lets those in flight finish for up to 60 seconds and cancels the rest. The last node that serves an active destination for writing cannot be unmounted until another destination is active: the confirmation says so.
  4. Choose Move from a node… to move a mount to another storage node, new or online: the first one drains, the second one verifies.
  5. While a destination empties, its tile shows the bar toward the active destination of its class with the share of references moved, then the grace, the removal of the old copies and the verification by count.

Tip Unmounting never loses a byte: reads go to another node that mounts the destination, and without one its files answer «unavailable» until it is mounted again — never «not found».

Reading the map of the installation

Map shows the whole installation at a glance: the nodes in three bands by role, every data node a tile with the mosaic of its cells on its capacity, the continuous copies of the databases with the tape of their last rounds, and the last events that changed the installation, each with its author.

  1. Read the bands: Entry · control · identity has a tile per node with its roles and the services that answer on it; Data has a tile per data node — its name, its state, the mosaic of forty places (its cells on the capacity declared in Weights and capacities, or on the fullest data node when none is declared), how many cells it holds and its CPU —; Compute · storage · TURN has a chip per node. Every tile and chip opens the page of its node.
  2. Follow a node that falls: an ochre outline says its lease is not renewed; a red outline with «isolated · N» says the lease has expired and the node has isolated itself; «offline · N» says the topology declared it failed and is recovering its cells from the copies — the places it lost are outlined in red —, and the nodes that receive them have the outline in the pigment, the received places darker, for an hour after the recovery ends.
  3. Choose what the colour of the tiles says with Roles · Load · Cells: the states, the CPU (ochre beyond 70%), or the cells on the capacity (ochre beyond 80%).
  4. Continuous copies says how often every process copies its databases, the tape of the last thirty rounds of the installation — every bar the longest round of its second, the newest in the pigment, one longer than the cadence in ochre —, where the copies go and how far back a database can be restored.
  5. Events shows the last events that changed the map, a run of cells recovered or moved in one line, with their author: a person with their login, a job as «topology», a node with its name. All the events opens the Events section.
  6. The map reads again every 5 seconds while it is in view; Refresh now reads at once. An installation with more than four rows of data nodes scrolls inside the card.

Tip When a node falls, keep the map open: you see the lease expire, the node isolate itself, its cells fenced and recovered on the least loaded nodes — and in Events who did what, minute by minute.

Read the events of the topology

Events is the registry of the topology, newest first: what happened to the installation — its nodes, cells, leases, storage and policy —, each event with its author, a person or a job of the topology. The line under the title says how many events the registry keeps.

  1. Choose a type above the list — All, Nodes, Cells, Leases, Storage, Policy, Topology — and a node in Node: the list shows the events of that family that name that node, as their subject or among the nodes before and after them. The filters stay in the address of the page, so a link keeps them.
  2. Read each row: the time, with the date when it is not today; the dot and the word of the family; what happened, in words; its subject — a node is a link to its page, a cell is its kind and its key —; and underneath its author: an administrator by the account, a job of the topology by what it does, such as the presence, the leases or the rebalancer.
  3. Open Before and after on a row to read what the event changed, as the registry wrote it; nothing means there was no value.
  4. Load more reads the older events. The topology reads its registry a stretch at a time: when a stretch has no event for your filters the console goes on by itself a few times, then says so and offers Keep looking — it says that no event matches only when nothing older is left.
  5. New events arrive at the top every 30 seconds while the section is in view; Refresh now reads at once.

Tip A change of the policy, a drain, a node declared failed or a plan applied carries the name of whoever did it: Events is where to find who changed what, and when.

Upgrade the databases

Upgrade checks, for each service, whether its databases are at the version the server expects. Until they are, the data features of that service stay off to protect the data.

  1. Read each row: Total and Outdated, with Up to date, Update available, Unreachable — the service did not answer — or Check failed, with the reason under the row.
  2. Run upgrade appears only when something is out of date. The confirmation names the service, how many databases go to which version and that it cannot be undone: type the identifier of the service it shows — such as auth — and confirm; the server upgrades in the background while the row says Upgrading…
  3. The row checks again every 2 seconds until the server has finished, and a notice says how it ended; Upgrade failed says it did not.
  4. The database icon opens the detail page, with each database, its file path, its current and target version and its status — Newer than server and Not created yet included; the upgrade can be run from there too.

Tip You can close the page during an upgrade: when you open it again, it goes back to watching.

Back to the guide

Productivity, made in Italy.

Bring the whole suite to your team — simple, secure and entirely yours.