This is a list libraries and tools to support building distributed services with Go. It’s not an exhaustive list of alternatives, just some things I’ve discovered, seen used or had recommended. The actual process of evaluating these in detail will happen gradually.
Async, jobs and tasks
- Cherami - a task queue open sourced by Uber.
- context - bundled with Go since 1.7. Defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes. Here’s a timeout example.
- Dingo - distributed, extensible task/job queue framework. Created after Machinery (see below) with the goal of making sending tasks as easy as possible and awaiting and receiving reports through channels.
- gocelery - can be used as a pure Go distributed task queue, but seems to be focused on interoperability with the Python based Celery distributed task queue.
- go-workers - Sidekiq compatible background workers. Claims support for reliable queueing for all queues akin to the super_fetch feature of Sidekiq Pro. Retries, configurable per-queue-concurrency, tolerant to restarts, with stats on running jobs.
- goworker - a background worker that claims to run 10 to 100,000 times faster than Ruby-based workers. Compatible with Resque so jobs can be pushed from Rails with a Resque client, but will ultimately be consumed and run by Go.
- Machinery - an asynchronous task queue/job queue based on distributed message passing. Can use Redis, AMQP/RabbitMQ or AWS-SQS for message brokering. Uses JSON to communicate tasks between systems. Supported delayed tasks, retries with Fibonacci backoff or explicit retry at a later time. Groups, chords and chains to model multi-step processes.
- x/sync - advanced utilities for goroutines and concurrency. Weighted semaphores, duplicate function call suppression, concurrent map, synchronisation, error propagation etc.
Authentication and security
- x/crypto - from the Sub-repository of the Go Project. Supplementary cryptography libraries. For a list of what’s available see the root of the master branch.
- x/oauth2 - also from the Sub-repository of the Go Project. Provides supporting code for various well-known services such as GitHub, Slack, Google, Amazon, Facebook and more.
- envflag - Minimalist support for 12-factor configuration using the standard library’s flag package.
- Upper db.v3 - a database CRUD abstraction layer with ORM. Provides adapters for common relational database systems, MySQL, PostgreSQL, SQLite, MSSQL and QL, as well as partial support for MongoDB. Uses Go’s struct tagging feature to map database columns to struct attributes.
- pgx - PostgreSQL driver, useable with or without database/sql for inline-SQL style querying. Supports all sorts - batch queries, binary formats, JSON/JSONB, Hstore and so on.
- Redigo - A Redis client. Has support for all Redis commands, pipelining, publish/subscribe, connection pooling and Lua scripts.
HTTP and service frontend
- Apache Thrift - official library for the Thrift serialization, IDL and RPC stack. Provides a built-in server and clients to expose and consume RPC services. See the Go example
- gorilla/mux - a request router and dispatcher for matching incoming requests to their respective handler.
- x/net - from the Sub-repository of the Go Project. Provides supplimentary networking libraries. E.g. http2, websocket
- JOCKO - Kafka implemented in Golang with built-in coordination (No ZooKeeper dependency, single binary install, Cloud Native).
- NSQ - a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. Directly provides Go and Python libraries, but many 3rd party clients exist for other languages.
Metrics, monitoring and error handling
- DataDog-Go - provides a client for DataDog’s DogStatsD metrics aggregation service.
- Logrus - structured, pluggable logging for Go
- pkg/errors - wrappers to add extra context to an error, and the ability to retrieve the original cause error.
- NewRelic for Go - error collection and performance monitoring service with support for Go.
Tools and utilities
- Dep - an experimental dependency management tool for the Go language. Gives you a per-project configuration file
Gopkg.tomlto declare your dependencies with version specs and a “lock file”
Gopkg.lockto snapshot your project’s resulting dependency graph. Similar to Bundler, Cargo, and NPM.
Serialization, messaging and RPC
- Apache Thrift - official library for the Thrift serialization, IDL and RPC stack.
- pkg/bson - implements encoding and decoding of BSON objects.
- encoding/json - bundled with Go. Implements encoding and decoding of JSON.
- tchannel - multiplexing and framing protocol for RPC calls.
- ringpop - for sharding your application scalably with fault tolerance.
- Ginkgo - a BDD-style Go testing framework. Similar to RSpec for Ruby. It has
Specifyto define examples/specs. Supports shared examples, async tests, benchmarks and more. Pairs with Gomega which provides
- Gomega - a matcher/assertion library. Preferred library for Ginkgo. Beyond standard assertions has support for async, custom matchers, testing http clients, streaming buffers, command line clients / external processes.
- Fake - fake data generation.