diff options
| -rw-r--r-- | README.md | 340 |
1 files changed, 298 insertions, 42 deletions
@@ -1,67 +1,132 @@ # jobd -**jobd** is a simple job queue daemon written in Node.JS. It uses MySQL table as -a storage. +**jobd** is a simple job queue daemon with persistent queue storage, written in +Node.JS. It uses a MySQL table as a storage backend (for queue input and output). + +Currently, MySQL is the only supported storage type, but other backends might be +easily supported. + +It is by design that jobd never adds nor deletes jobs from storage. It only +reads (when a certain request arrives) and updates them (during execution, when +job status changes). Succeeded or failed, your jobs are never lost. + +jobd consists of 2 parts: + +1. **jobd** is a "worker" daemon that reads jobs from the database, enqueues and + launches them. There may be multiple instances of jobd running on multiple + hosts. Each jobd instance may have unlimited number of queues (called "targets"), + each with its own concurrency limit. + +2. **jobd-master** is a "master" or "cetral" daemon that simplifies control over + many jobd instances. There should be only one instance of jobd-master running. + jobd-master is not required for jobd workers to work (they can work without it), + but it's very very useful. + +In addition, there is a command line utility called **jobctl**. + +Originally, jobd was created as a saner alternative to Gearman. It's been used +in production with a large PHP web application on multiple servers for quite some +time already, and proven to be stable and efficient. + +## Table of Contents + +- [How it works](#how-it-works) +- [Protocol](#protocol) + - [Request Message](#request-message) + - [jobd request types](#jobd-request-types) + - [jobd-master request types](#jobd-master-request-types) + - [Response Message](#response-message) + - [Ping and Pong Messages](#ping-and-pong-messages) +- [Implementation example](#implementation-example) +- [Installation](#installation) +- [Usage](#usage) + - [systemd](#systemd) + - [supervisor](#supervisor) + - [Other notes](#other-notes) +- [Configuration](#configuration) + - [jobd](#jobd) + - [jobd-master](#jobd-master) + - [jobctl](#jobctl) +- [MySQL setup](#mysql-setup) +- [Clients](#clients) + - [PHP](#php) +- [TODO](#todo) +- [License](#license) + + +## How it works + +To be written. +## Protocol -## Installation - -To be written - - -## Usage - -To be written - +By default, jobd and jobd-master listen on TCP ports 7080 and 7081 respectively, +ports can be changed in a config. -## MySQL setup +jobd has been created with an assumption that it'll be used in more-or-less +trusted environments (LAN, or, at least, servers within one data center) so no +encryption nor authentication mechanisms have been implemented. All traffic +between jobd and clients flow in plain text. You can protect a jobd instance with +a password though, so at least basic password-based authorization is supported. -Minimal table scheme. +Both daemons receive and send Messages. Each message is followed by `EOT` (`0x4`) +byte which indicates an end of a message. Clients may send and receive multiple +messages over a single connection. Usually, it's the client who must close the +connection, when it's not needed anymore. A server, however, may close the +connection in some situations (invalid password, server error, etc). -In a real world, you would to add need additional fields such as `job_name` or -`job_data`. +Messages are encoded as JSON arrays with at least one item, representing +the message type: +``` +[TYPE] +``` +If a message of specific type has some data, it's placed as a second item: ``` -CREATE TABLE `jobs` ( - `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT, - `target` char(16) NOT NULL, - `time_created` int(10) UNSIGNED NOT NULL, - `time_started` int(10) UNSIGNED NOT NULL DEFAULT 0, - `time_finished` int(10) UNSIGNED NOT NULL DEFAULT 0, - `status` enum('waiting','manual','accepted','running','done','ignored') NOT NULL DEFAULT 'waiting', - `result` enum('ok','fail') DEFAULT NULL, - `return_code` tinyint(3) UNSIGNED DEFAULT NULL, - `sig` char(10) DEFAULT NULL, - `stdout` mediumtext DEFAULT NULL, - `stderr` mediumtext DEFAULT NULL, - PRIMARY KEY (`id`), - KEY `status_target_idx` (`status`, `target`, `id`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8; +[TYPE, DATA] ``` -For optimization purposes, you can turn `target` into `ENUM`. Also if 16 characters -for target is not enough for you, change it to fit your needs. +Type of `TYPE` is integer. Supported types are: +- `0`: Request +- `1`: Response +- `2`: Ping +- `3`: Pong -## Clients -#### PHP +### Request Message -[php-jobd-client](https://github.com/gch1p/php-jobd-client) (official) +`DATA` is a JSON object with following keys: -## Protocol +- **`no`** *(**required**, int)* — unique (per connection) request number. Clients + can start counting request numbers from one (`1`) or from any other random number. + Each subsequent request should increment this number by 1. Note that zero (`0`) + is reserved. +- **`type`** *(**required**, string)* — request type. Supported request types for + jobd and jobd-master are listed below. +- **`data`** *(object)* — request arguments (if needed): an object, whose keys + and values represent argument names and values. +- **`password`** *(string)* — a password, for password-protected instances. Only + needed for first request. + +Example (w/o trailing `EOT`): +``` +[0,{no:0,type:'poll',data:{'targets':['target_1','target_2']}}] +``` -### jobd requests +Here is the list of supported requests, using `type(arguments)` notation. + +#### jobd request types * **`poll(targets: string[])`** — get new tasks for specified `targets` from database. If `targets` argument is not specified, get tasks for all serving targets. - + * **`pause(targets: string[])`** — pause execution of tasks of specified targets. If `targets` argument is not specified, pauses all targets. * **`continue(targets: string[])`** — continue execution of tasks of specified targets. If `targets` argument is not specified, continues all targets. - + * **`status()`** — returns status of internal queues and memory usage. * **`run-manual(ids: int[])`** — enqueue and run jobs with specified IDs and @@ -70,11 +135,11 @@ for target is not enough for you, change it to fit your needs. * **`set-target-concurrency(target: string, concurrency: int)`** — set concurrency of target `target`. -### jobd-master requests +#### jobd-master request types * **`register-worker(targets: string[])`** — used by a jobd instance to register itself with master. You don't need it. - + * **`poke(targets: string[])`** — send `poll` requests to all registered workers that serve specified `targets`. @@ -85,7 +150,7 @@ for target is not enough for you, change it to fit your needs. * **`continue(targets: string[])`** — send `continue(targets)` requests to workers serving specified `targets`. If `targets` argument is not specified, sends `continue()` to all workers. - + * **`status(poll_workers=false: bool)`** — returns list of registered workers and memory usage. If `poll_workers` is true, sends `status()` request to all registered workers and includes their responses. @@ -93,7 +158,198 @@ for target is not enough for you, change it to fit your needs. * **`run-manual(jobs: {id: int, target: string}[])`** — send `run-manual` requests to registered jobd instances serving specified targets, aggregate an return results. + +### Response Message + +`DATA` is a JSON object with following keys: + +- **`no`** *(**required**, int)* — `no` of request this response is related to. +- **`data`** *(array *|* object *|* string *|* int)* — data, if request succeeded. +- **`error`** *(string)* — error message, if request failed. + +Example (w/o trailing `EOT`): +``` +[1,{no:0,data:'ok'}] +``` + +### Ping and Pong Messages + +No `DATA`. + +Example (w/o trailing `EOT`): +``` +[2] +``` + +## Implementation example + +To be written. + +## Installation + +First, you need Node.JS 14 or newer. See [here](https://nodejs.org/en/download/package-manager/) +now to install it using package manager. + +Then install jobd using npm: + +``` +npm i -g jobd +``` + + +## Usage + +### systemd + +One of possible ways of launching jobd and jobd-master daemons is via systemd. +This repository contains basic examples of `jobd.service` and `jobd-master.service` +unit files. Note that jobs will be launched as the same user the jobd worker is +running, so you might want to change that. + +Copy `.service` file(s) to `/etc/systemd/system`, then do: +``` +systemctl daemon-reload +systemctl enable jobd +systemctl start jobd +# repeat last two steps for jobd-master, if needed +``` +### supervisor + +If you don't like systemd, supervisor might be an option. Create a configuration +file in `/etc/supervisor/conf.d` with following content: +``` +[program:jobd] +command=/usr/bin/jobd --config /etc/jobd.conf +numprocs=1 +directory=/ +stdout_logfile=/var/log/jobd-stdout.log +autostart=true +autorestart=true +user=nobody +stopsignal=TERM +``` + +Then use `supervisorctl` to start or stop jobd. + + +### Other notes + +Don't forget to filter access to jobd and jobd-master ports using your favorite +firewall. + +## Configuration + +Configuration files are written in ini format. All available options for both +daemons, as well as a command-line utility, are described below. You can copy +`jobd.conf.example` and `jobd-master.conf.example` and use them as a template +instead of writing configs from scratch. + +### jobd + +Default config path is `/etc/jobd.conf`. Use the `--config` option to use +a different path. + +Without section: + +- **`host`** *(required, string)* — jobd server hostname +- **`port`** *(required, int)* — jobd server port +- **`password`** *(string)* — password for requests +- **`always_allow_localhost`** *(boolean, default: `false`)* — when set to `1` + or `true`, allows accepting requests from clients connecting from localhost + without password +- **`master_host`** *(string)* — master hostname +- **`master_port`** *(int)* — master port. If hostname or port is omitted, jobd + will not connect to master. +- **`master_reconnect_timeout`** *(int, default: `10`)* — if connection to master + failed, jobd will be constantly trying to reconnect. This option specifies a + delay between connection attempts, in seconds. +- **`log_file`** *(string)* — path to a log file +- **`log_level_file`** *(string, default: `warn`)* — minimum level of logs that + are written to the file.<br> + Allowed values: `trace`, `debug`, `info`, `warn`, `error` +- **`log_level_console`** *(string, default: `warn`)* — minimum level of logs + that go to stdout. +- **`mysql_host`** *(required, string)* — database host +- **`mysql_port`** *(required, int)* — database port +- **`mysql_user`** *(required, string)* — database user +- **`mysql_password`** *(required, string)* — database password +- **`mysql_database`** *(required, string)* — database name +- **`mysql_table`** *(required, string)* — table name +- **`mysql_fetch_limit`** *(int, default: `100`)* — a number of new jobs to fetch + in every request +- **`launcher`** *(required, string)* — a template of shell command that will be launched + for every job. `{id}` will be replaced with job id +- **`max_output_buffer`** *(int, default: `1048576`)* + +Under the `[targets]` section, targets are specified. Each target is specified on +a separate line in the following format: +``` +{target_name} = {n} +``` +where: +- `{target_name}` *(string)* is target name +- `{n}` *(int)* is maximum count of simultaneously executing jobs for this target + +### jobd-master + +Default config path is `/etc/jobd-master.conf`. Use the `--config` option to use +a different path. + +- **`host`** *(required, string)* +- **`port`** *(required, int)* +- **`password`** *(string)* +- **`always_allow_localhost`** *(boolean, default: `false`)* +- **`ping_interval`** *(int, default: `30`)* — specifies interval between workers + pings. +- **`poke_throttle_interval`** *(int, default: `0.5`)* +- **`log_file`** *(string)* +- **`log_level_file`** *(string, default: `warn`)* +- **`log_level_console`** *(string, default: `warn`)* + +### jobctl + +Default config path is `~/.jobctl.conf`. + +- **`master`** (boolean) — same as `--master`. +- **`host`** *(string)* — same as `--host`. +- **`port`** *(int)* — same as `--port`. +- **`password`** *(string)* +- **`log_level`** *(string, default: `warn`)* + +## MySQL setup + +Minimal table scheme: + +``` +CREATE TABLE `jobs` ( + `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT, + `target` char(16) NOT NULL, + `time_created` int(10) UNSIGNED NOT NULL, + `time_started` int(10) UNSIGNED NOT NULL DEFAULT 0, + `time_finished` int(10) UNSIGNED NOT NULL DEFAULT 0, + `status` enum('waiting','manual','accepted','running','done','ignored') NOT NULL DEFAULT 'waiting', + `result` enum('ok','fail') DEFAULT NULL, + `return_code` tinyint(3) UNSIGNED DEFAULT NULL, + `sig` char(10) DEFAULT NULL, + `stdout` mediumtext DEFAULT NULL, + `stderr` mediumtext DEFAULT NULL, + PRIMARY KEY (`id`), + KEY `status_target_idx` (`status`, `target`, `id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +``` + +In a real world, you need some additional fields such as `job_name` or `job_data`. + +For optimization purposes, you can turn `target` into `ENUM`. Also, if 16 characters +for the `target` field is not enough for you, change it to fit your needs. + + +## Clients + +### PHP + +[php-jobd-client](https://github.com/gch1p/php-jobd-client) (official) ## TODO |