In this three part walkthrough we are going to create a mini web service using Go that connects with a DGraph instance and returns a DQL (dgraph query language) query via a ‘/query’ endpoint, both locally and running within a Kubernetes Cluster. the basics covered here should give anyone starting out the means to get onto the first rung of the ladder with Go and DGraph.
Overview
If you are not familair with the DGraph ecosystem it can be somewhat of a challenge to follow the docs and figure out how to go about setting it up correctly. This guide aims to show you how to set up and connect to DGraph in the following manner. Each number representing each subsequent part of the walkthrough series.
- Go Web Service and DGraph instance running locally.
- Using Docker Desktop, we will start our Dgraph Instance
- We will populate the DGraph instance with a Schema and seed it it dummy data
- We will compile our Go code to start a webservice connected to the DGraph instance.
- We will return a GraphQL query via our Go webservice.
- Go Web Service and DGraph instance running in Kubernetes.
- We will set up a K8S Kustomization folder structure to build our images.
- Via a makefile we will then set up a local K8S cluster and load our images.
- We will return a DQL query via our Go webservice.
- Showcase using DGraph Cloud as a managed service.
NOTE: The purpose here is to showcase functionality which you can build on yourself. My focus was on getting code that works over best practice, styling etc.
INFO: It is recommended that you follow this tutorial from the start as each part increments on the previous. This will help you avoid mssing dependant information.
Pre-Requisites
For the sake of clarity, Table 1 covers the version information of the various tooling used.
Table 1 - Tooling Versions
| Tool Name | Version |
|---|---|
| Docker Desktop | 20.10.22 |
| Docker | 20.10.22 |
| Go | 1.20 |
| DGraph | 23.0.1 |
| Kustomize | 4.5.7 |
| kubectl | 1.26.2 |
| kind | 0.17.0 |
Tooling is best installed via your native package manager, i.e brew or apt.
brew install go kind kubectl kustomize
We will control all operations with a makefile to make things as seamless as possible. So lets create one in our repo.
touch makefile
## Prepare docker
In your IDE of choice lets add the following lines.
# Set up makefile variables.
DGRAPH := dgraph/dgraph:v23.0.0
RATEL := dgraph/ratel:latest
# dev-docker: pulls all required docker images.
dev-docker:
docker pull $(DGRAPH)
docker pull $(RATEL)
# dev-dgraph-local: run the containers in docker.
dev-dgraph-local:
docker run --name dgraph-local -d -it \
-p "8080:8080" -p "9080:9080" -p "8090:8090" \
-v ~/dgraph $(DGRAPH)
docker run --name ratel-local \
--platform linux/amd64 -d -p "8000:8000" $(RATEL)
Notice the ‘-p’ flag. This flag is handling the port forwarding mechanism that allows us to talk to our containers from outside their own respective containers. Hence when we browse the port 8000 on the ratel instance we get a UI as a UI is being serverd via the internal port of 8000.
Now in our terminal, lets execute these commands.
$ make dev-docker
$ make dev-docker-local
Visit http://localhost:8000 and you should be able to access the Ratel UI. The frontend controller for the DGraph backend. You will see a successful connection under the Server Connection menu.
Seed the database
With our local instances up and ruunning lets populate the database with a schema and push some data to work with. (In the repo wou will find sample schema files.)
Back in our makefile, lets add the following lines
dev-dgraph-seed:
curl -X POST 'localhost:8080/admin/schema' --data-binary \
'@./schema/schema.graphql'
curl -X POST 'http://localhost:8080/graphql' \
--header 'Content-Type: application/graphql' \
--data-binary '@./schema/sample.data.graphql'
To populate the schema we need to target the schema endpoint on the admin route.
You should see a JSON struct printed to STDOUT when successful.
We can verify the data migration by running a query against the GraphQL endpoint
curl -X POST 'http://localhost:8080/graphql' --header \
'Content-Type: application/graphql' --data-binary \
'@./schema/sample.query.graphql'
You will get the follwoing response.
{
"data": {
"queryReview": [
{
"comment": "Fantastic, easy to install, worked great. Best GraphQL server available",
"by": {
"username": "Jack Watts",
"reviews": [
{
"about": {
"name": "Trench Digital",
"reviews": [
{
"by": {
"username": "Jack Watts"
},
"comment": "Pretty cool stuff!!",
"rating": 10
}
]
},
"rating": 10
},
{
"about": {
"name": "Dgraph",
"reviews": [
{
"by": {
"username": "Jack Watts"
},
"comment": "Fantastic, easy to install, worked great. Best GraphQL server available",
"rating": 10
}
]
},
"rating": 10
},
{
"about": {
"name": "Dgraph Cloud",
"reviews": [
{
"by": {
"username": "Jack Watts"
},
"comment": "Pretty nifty Cloud service",
"rating": 10
}
]
},
"rating": 10
},
{
"about": {
"name": "Fish Tank Cloud",
"reviews": [
{
"by": {
"username": "Jack Watts"
},
"comment": "Awesome free APPs!",
"rating": 10
}
]
},
"rating": 10
}
]
},
"about": {
"name": "Dgraph"
}
}
]
},
"extensions": {
"touched_uids": 63,
"tracing": {
"version": 1,
"startTime": "2023-06-17T23:58:16.048289041Z",
"endTime": "2023-06-17T23:58:16.068109541Z",
"duration": 19820458,
"execution": {
"resolvers": [
{
"path": [
"queryReview"
],
"parentType": "Query",
"fieldName": "queryReview",
"returnType": "[Review]",
"startOffset": 938958,
"duration": 18850625,
"dgraph": [
{
"label": "query",
"startOffset": 1553833,
"duration": 18232583
}
]
}
]
}
}
}
}
Introducing Go
Alright, we have DGraph up and runnig and some data in place that we can work with. Let’s bring Go into the equation now and implememt the same functionality of the CURL commands above.
Back in our terminal with the PWD the root of our project, init a Go module.
# Be sure to set your namespace accordingly.
# For example, my NS is github.com/jack-watts/go-dgraph-starter
$ go mod init [path/to/your/module/namespace]
# Set up our Go service
$ mkdir -p app/services/api/ && touch app/services/api/main.go
In main.go, set some boilerplate code.
package main
import "fmt"
func main() {
fmt.Println("Hello, DGraph")
}
Set up a basic web server
Alright, lets get a basic web service up and running before turning our attention to configuration.
Create a new function called run that will return an error and place our inital service startup here.
func run() error {
log.Print("starting service")
defer log.Print("shutdown complete")
api := http.Server{}
if err := api.ListenAndServe(); err != nil {
return err
}
return nil
}
Back in our main function lets call run and handle the error response via log.Fatal which makes a call to os.Exit.
func main() {
if err := run(); err != nil {
log.Fatalf("Unable to start sevrice: %s\n", err)
}
}
Adding our Go config
Create a new folder with a go file named config.go inside of it: ./app/services/api/config/config.go
package config
import (
"net/http"
"time"
)
type Config struct {
Web http.Server
DgURL string
}
func (c *Config) Initialize() {
c.Web.ReadTimeout = time.Second * 5
c.Web.WriteTimeout = time.Second * 10
c.Web.IdleTimeout = time.Second * 120
}
Now, modify the run() function to take a value of type Config and set the API server parameters.
func run(cfg *config.Config) error {
log.Print("starting service")
defer log.Print("shutdown complete")
api := http.Server{
Addr: cfg.Web.Addr,
ReadTimeout: cfg.Web.ReadTimeout,
WriteTimeout: cfg.Web.WriteTimeout,
IdleTimeout: cfg.Web.IdleTimeout,
}
if err := api.ListenAndServe(); err != nil {
return err
}
return nil
}
Lets fo back to our main() function and initialize our Config with some default values using the flag package.
func main() {
var cfg config.Config
cfg.Initialize()
// Set CMD Flags
flag.StringVar(&cfg.Web.Addr, "web-host", "localhost:8081", \
"set the hostname for the service. default: localhost:8081")
flag.StringVar(&cfg.DgURL, "dgraph-host", "localhost:9080", \
"set the DGraph host url. default: localhost:9080")
if err := run(&cfg); err != nil {
log.Fatalf("Unable to start sevrice: %s\n", err)
}
}
Ok, before we go any further in connecting to DGraph, I want to stabilize our webservice so that it will gracefully shutdown should it be interupted.
## Supporting our web Server.
First lets set up management of any server errors that we may encounter. For this we need to create a channel that is listens for a any errors from our service.
In our main() function, lets rewrite how we are calling our webserver.
// From this
if err := api.ListenAndServe(); err != nil {
return err
}
// To this
serverErrors := make(chan error, 1)
go func() {
log.Printf("Starting api V1 router, host: %s\n", api.Addr)
serverErrors <- api.ListenAndServe()
}()
We have created a channel with a buffer of 1 value that is expecting errors. Then with the <- syntax the channel receives a vlaue of Type error passed across Go routines as that is what we have wrapped our web server in. That said, we are not doing anything with our serverErrors channel right now so if you run the service, it will simply just start up and shutdown as no errors have gone into the channel. We need to handle our Shutdown listener conatined within the Server instance. So lets implement a graceful shutdown to our web server.
Create another channel called shutdown and pass this as the value to signal.Notify() which enables all incoming signals to be passed to the shutdown channel.
...
shutdown := make(chan os.Signal, 1)
signal.Notify(shutdown, syscall.SIGINT, syscall.SIGTERM)
...
serverErrors := make(chan error, 1)
go func() {
log.Printf("Starting api V1 router, host: %s\n", api.Addr)
serverErrors <- api.ListenAndServe()
}()
Lets manage these channels on a conditional basis via a switch statement.
In case serverErrors returns true, we will return the error and out of the main() function.
If a signal is dectected via the shutdown channel we initialise a context with the intent to release resources associated with it across a period of 20 seconds. This context is passed to the Shutdown method of our web server where Shutdown gracefully shuts down the server without interrupting any active connections. Shutdown works by first closing all open listeners, then closing all idle connections, and then waiting indefinitely for connections to return to idle and then shut down.
select {
case err := <-serverErrors:
return fmt.Errorf("server error: %w", err)
case sig := <-shutdown:
log.Printf("graceful shutdown started:signal:%s\n", sig)
defer log.Printf("graceful shutdown complete:signal:%s\n", sig)
ctx, cancel := context.WithTimeout(context.Background(), time.Second*20)
defer cancel()
if err := api.Shutdown(ctx); err != nil {
api.Close()
return fmt.Errorf("could not stop server gracefully: %w", err)
}
}
Update our makefile to run our Go code.
dev-run:
go run ./app/services/api
Execute our new makefile entry:
$ make dev-run
go run ./app/services/api
2023/06/17 14:41:04 starting service
2023/06/17 14:41:04 Starting api V1 router, host: localhost:8081
# Pressing CTRL+C will result in a graceful shutdown.
^C2023/06/17 14:42:57 graceful shutdown started:signal:interrupt
2023/06/17 14:42:57 graceful shutdown complete:signal:interrupt
2023/06/17 14:42:57 shutdown complete
OK - Lets connect to DGraph.
Connecting to DGraph
Lets install the DGraph client we are going to use.
```bash
$ go get github.com/dgraph-io/dgo/v230
Create a new go file called `dgraph.go` within `package main` and lets add our connection function.
```go
type CancelFunc func()
func getDgraphClient() (*dgo.Dgraph, CancelFunc) {
conn, err := grpc.Dial("localhost:9080", grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatal("While trying to dial gRPC")
}
dc := api.NewDgraphClient(conn)
dg := dgo.NewDgraphClient(dc)
return dg, func() {
if err := conn.Close(); err != nil {
log.Printf("Error while closing connection:%v", err)
}
}
}
The Go client communicates with the server on the gRPC port (default value 9080)
You will notice that we now have three non standard libary packages in use.
"google.golang.org/grpc"
"github.com/dgraph-io/dgo/v230"
"github.com/dgraph-io/dgo/v230/protos/api"
NOTE the version (v230) in the dgo import namespace. This is very, very important for the client to work properly as there are significant changes from previous versions.
So lets run go mod tidy to clean things up.
Ok, lets call this function from ‘main.go’ in the run() function.
dg, cancel := getDgraphClient()
defer cancel()
app := Application{
DG: dg,
}
At this point, our code will not compile as we have declared a variable of type Application. So we need to create this type. But why am I creating this type? getDgraphClient() returns a pointer to type Dgraph (*dgo.Dgraph). This type has been set up and I would like to retain use of it for the duration of the period my service is running. I do not want to be creating multiple clients each time I want to send a query. So we create a global Type that we will bind our service Router to and call upon for each respective handler we need to stand up.
type Application struct {
DG *dgo.Dgraph
}
Setting up the service router
Create a new file called routes.go in package main. Add the following code. We will be using the Chi muxer for this walkthrough.
func (app *Application) routes() http.Handler {
mux := chi.NewRouter()
return mux
}
Rerun go mod tidy to import Chi. Now lets fo back to run() and append our http server to give it our new Chi router as the handler over the default setting.
api := http.Server{
Addr: cfg.Web.Addr,
// here we add our routes pointer refernece method
Handler: app.routes(),
ReadTimeout: cfg.Web.ReadTimeout,
WriteTimeout: cfg.Web.WriteTimeout,
IdleTimeout: cfg.Web.IdleTimeout,
}
Great! Our service will run but we aren’t even doing anything yet! So lets change that and bring in some handlers! We are going to set two, one for the root “/” and one for a simple query “/query”.
Creating handlers
First things first, we are going to be writing JSON here , so lets create a writeJSON utils function to prevent ourselves from repeating code.
Create a utils.go foler under package main. The error is supressed as we will be doing the response validation in our handler functions.
func writeJSON(w http.ResponseWriter, data interface{}) {
out, _ := json.MarshalIndent(data, "", " ")
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = w.Write(out)
}
Now create a handlers.go file under package main. For the sake of it we’ll keep things clean for the user so when they hit the root of our service i.e “/”, they receive a message that informs them of what routes they should be targeting.
func (app *Application) home(w http.ResponseWriter, r *http.Request) {
var data = struct {
Status string `json:"status"`
URL string `json:"urls"`
}{
Status: "active",
URL: "/query",
}
writeJSON(w, data)
}
Here we create a quick in line struct and print the response calling our utils writeJSON() function. And back in our pointer reference method routes() we add our route.
func (app *Application) routes() http.Handler {
mux := chi.NewRouter()
mux.Get("/", app.home) // <--- here
return mux
}
Fire up our service make dev-run and browse to http://localhost:8081
Success!
Ok, now lets add a query endpoint. Back in our handlers.go file lest add a query() pointer reference method where we create a new Transaction (a phrase that is part of the DGraph nomenclature), define or query in DQL, send the Query via our dgraph client. Remember we are running on our Application type here which has a nested reference to type Dgraph. This is why we created it earlier. Then we are back to good oul JSON response handling. Given the nature of the repsonse with respect to our schema we populated earlier, we are unmarshaling to a map to handle the unstructured data. A good example of not everything in the real world conforming to the Type philosphy! And finally we call our utils writeJSON helper funcion!
func (app *Application) query(w http.ResponseWriter, r *http.Request) {
// create a new transaction
txn := app.DG.NewReadOnlyTxn()
// define our query
const q = `{
all(func: has(Product.reviews), first: 10) {
uid
expand(_all_)
}
}`
// pass our query as a transaction to get a response
resp, err := txn.Query(context.Background(), q)
if err != nil {
log.Printf("error returning query response body: %s\n", err)
return
}
var data interface{}
// unmarshal the unstructured data to the 'data' interface.
if err := json.Unmarshal(resp.GetJson(), &data); err != nil {
log.Printf("%s\n", err)
}
writeJSON(w, data)
}
Last step here, lets add our “/query” handler to routes.go.
func (app *Application) routes() http.Handler {
mux := chi.NewRouter()
mux.Get("/", app.home)
mux.Get("/query", app.query) // <-- here
return mux
}
And thast it! Restart our service make dev-run and browse to http://localhost:8081/query
In closing, dgraph is quite exotic to put it mildly as is a discipline into itself so forgive me for not delving too deep into the DQL side of things. From this point you should have the gist to move forward and introduce other transaction types. Be sure to check out the DGraph documention via the references below for more insight. I will be back for part 2 shortly!
Summary
In this walkthrough you learned
- To set up DGraph in Docker
- Seed the DGraph instance via its native GraphQL endpoint.
- How to set up a web service in Go with graceful shutdown.
- Connect to a dockerized DGraph instance via a Go web service.
- Establish an endpoint that returns a query from the DGraph instance.
In part two, we will iterate over eveything we have done here once more, and tighten up our configuration as it is somewhat loose so we can build and load our service into a Kubernetes cluster running locally.
Source code can be found here
https://github.com/jack-watts/go-dgraph-starter
Gopher Images Credit Ashley Mcnamara. https://www.ashley.dev/projects/gopher-art
Read it on Medium here
References
Go - https://go.dev/
Dgraph - https://dgraph.io/
DGraph-IO - http://github.com/dgraph-io/dgo
Kustomize - https://kustomize.io/