gRPC and Protocol Buffers in Go

This guide demonstrates how to build efficient microservices using gRPC and Protocol Buffers in Go. We’ll create a complete example of a user management service.

What is gRPC?

gRPC is a high-performance, open-source RPC (Remote Procedure Call) framework that can run in any environment. It enables client and server applications to communicate transparently and makes it easier to build connected systems.

What are Protocol Buffers?

Protocol Buffers (protobuf) is Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. It’s smaller, faster, and simpler than XML or JSON.

Complete Example: User Management Service

Let’s build a user management service that supports:

Creating users
Getting user details
Listing users
Updating users
Deleting users
Streaming user events

Project Structure

user-service/
├── proto/
│   └── user.proto
├── server/
│   └── main.go
├── client/
│   └── main.go
├── go.mod
└── go.sum

1. Protocol Buffer Definition

First, let’s define our service using Protocol Buffers:

syntax = "proto3";

package user;

option go_package = "user-service/proto";

import "google/protobuf/timestamp.proto";
import "google/protobuf/empty.proto";

// User represents a user in the system
message User {
  string id = 1;
  string name = 2;
  string email = 3;
  google.protobuf.Timestamp created_at = 4;
  google.protobuf.Timestamp updated_at = 5;
}

// CreateUserRequest represents the request to create a user
message CreateUserRequest {
  string name = 1;
  string email = 2;
}

// UpdateUserRequest represents the request to update a user
message UpdateUserRequest {
  string id = 1;
  string name = 2;
  string email = 3;
}

// GetUserRequest represents the request to get a user
message GetUserRequest {
  string id = 1;
}

// ListUsersRequest represents the request to list users
message ListUsersRequest {
  int32 page_size = 1;
  int32 page_number = 2;
}

// ListUsersResponse represents the response for listing users
message ListUsersResponse {
  repeated User users = 1;
  int32 total = 2;
}

// DeleteUserRequest represents the request to delete a user
message DeleteUserRequest {
  string id = 1;
}

// UserEvent represents a user-related event
message UserEvent {
  enum EventType {
    UNKNOWN = 0;
    CREATED = 1;
    UPDATED = 2;
    DELETED = 3;
  }

  EventType type = 1;
  User user = 2;
  google.protobuf.Timestamp timestamp = 3;
}

// UserService defines the gRPC service
service UserService {
  // Create a new user
  rpc CreateUser(CreateUserRequest) returns (User);

  // Get a user by ID
  rpc GetUser(GetUserRequest) returns (User);

  // List users with pagination
  rpc ListUsers(ListUsersRequest) returns (ListUsersResponse);

  // Update a user
  rpc UpdateUser(UpdateUserRequest) returns (User);

  // Delete a user
  rpc DeleteUser(DeleteUserRequest) returns (google.protobuf.Empty);

  // Stream user events
  rpc WatchUsers(google.protobuf.Empty) returns (stream UserEvent);
}

2. Server Implementation

package main

import (
    "context"
    "fmt"
    "log"
    "net"
    "sync"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/protobuf/types/known/emptypb"
    "google.golang.org/protobuf/types/known/timestamppb"

    pb "user-service/proto"
)

type server struct {
    pb.UnimplementedUserServiceServer
    mu sync.RWMutex
    users map[string]*pb.User
    events chan *pb.UserEvent
}

func newServer() *server {
    return &server{
        users: make(map[string]*pb.User),
        events: make(chan *pb.UserEvent, 100),
    }
}

func (s *server) CreateUser(ctx context.Context, req *pb.CreateUserRequest) (*pb.User, error) {
    s.mu.Lock()
    defer s.mu.Unlock()

    // Generate a simple ID (in production, use UUID)
    id := fmt.Sprintf("user_%d", len(s.users)+1)

    now := timestamppb.Now()
    user := &pb.User{
        Id:        id,
        Name:      req.Name,
        Email:     req.Email,
        CreatedAt: now,
        UpdatedAt: now,
    }

    s.users[id] = user

    // Send event
    s.events <- &pb.UserEvent{
        Type:      pb.UserEvent_CREATED,
        User:      user,
        Timestamp: now,
    }

    return user, nil
}

func (s *server) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()

    user, ok := s.users[req.Id]
    if !ok {
        return nil, fmt.Errorf("user not found: %s", req.Id)
    }

    return user, nil
}

func (s *server) ListUsers(ctx context.Context, req *pb.ListUsersRequest) (*pb.ListUsersResponse, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()

    // Calculate pagination
    start := int(req.PageNumber) * int(req.PageSize)
    if start >= len(s.users) {
        return &pb.ListUsersResponse{
            Users: []*pb.User{},
            Total: int32(len(s.users)),
        }, nil
    }

    end := start + int(req.PageSize)
    if end > len(s.users) {
        end = len(s.users)
    }

    // Convert map to slice for pagination
    users := make([]*pb.User, 0, len(s.users))
    for _, user := range s.users {
        users = append(users, user)
    }

    return &pb.ListUsersResponse{
        Users: users[start:end],
        Total: int32(len(s.users)),
    }, nil
}

func (s *server) UpdateUser(ctx context.Context, req *pb.UpdateUserRequest) (*pb.User, error) {
    s.mu.Lock()
    defer s.mu.Unlock()

    user, ok := s.users[req.Id]
    if !ok {
        return nil, fmt.Errorf("user not found: %s", req.Id)
    }

    // Update fields
    if req.Name != "" {
        user.Name = req.Name
    }
    if req.Email != "" {
        user.Email = req.Email
    }
    user.UpdatedAt = timestamppb.Now()

    // Send event
    s.events <- &pb.UserEvent{
        Type:      pb.UserEvent_UPDATED,
        User:      user,
        Timestamp: timestamppb.Now(),
    }

    return user, nil
}

func (s *server) DeleteUser(ctx context.Context, req *pb.DeleteUserRequest) (*emptypb.Empty, error) {
    s.mu.Lock()
    defer s.mu.Unlock()

    user, ok := s.users[req.Id]
    if !ok {
        return nil, fmt.Errorf("user not found: %s", req.Id)
    }

    delete(s.users, req.Id)

    // Send event
    s.events <- &pb.UserEvent{
        Type:      pb.UserEvent_DELETED,
        User:      user,
        Timestamp: timestamppb.Now(),
    }

    return &emptypb.Empty{}, nil
}

func (s *server) WatchUsers(_ *emptypb.Empty, stream pb.UserService_WatchUsersServer) error {
    for event := range s.events {
        if err := stream.Send(event); err != nil {
            return err
        }
    }
    return nil
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }

    s := grpc.NewServer()
    pb.RegisterUserServiceServer(s, newServer())

    log.Printf("Server listening at %v", lis.Addr())
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

3. Client Implementation

package main

import (
    "context"
    "io"
    "log"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
    "google.golang.org/protobuf/types/known/emptypb"

    pb "user-service/proto"
)

func main() {
    // Connect to server
    conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()

    client := pb.NewUserServiceClient(conn)
    ctx := context.Background()

    // Start watching user events in a goroutine
    go watchUsers(ctx, client)

    // Create a user
    user1, err := client.CreateUser(ctx, &pb.CreateUserRequest{
        Name:  "John Doe",
        Email: "john@example.com",
    })
    if err != nil {
        log.Fatalf("could not create user: %v", err)
    }
    log.Printf("Created user: %v", user1)

    // Get the user
    getResp, err := client.GetUser(ctx, &pb.GetUserRequest{Id: user1.Id})
    if err != nil {
        log.Fatalf("could not get user: %v", err)
    }
    log.Printf("Got user: %v", getResp)

    // Create another user
    user2, err := client.CreateUser(ctx, &pb.CreateUserRequest{
        Name:  "Jane Smith",
        Email: "jane@example.com",
    })
    if err != nil {
        log.Fatalf("could not create user: %v", err)
    }
    log.Printf("Created user: %v", user2)

    // List users
    listResp, err := client.ListUsers(ctx, &pb.ListUsersRequest{
        PageSize:   10,
        PageNumber: 0,
    })
    if err != nil {
        log.Fatalf("could not list users: %v", err)
    }
    log.Printf("Listed %d users:", listResp.Total)
    for _, user := range listResp.Users {
        log.Printf("- %v", user)
    }

    // Update user
    updateResp, err := client.UpdateUser(ctx, &pb.UpdateUserRequest{
        Id:    user1.Id,
        Name:  "John Updated",
        Email: "john.updated@example.com",
    })
    if err != nil {
        log.Fatalf("could not update user: %v", err)
    }
    log.Printf("Updated user: %v", updateResp)

    // Delete user
    _, err = client.DeleteUser(ctx, &pb.DeleteUserRequest{Id: user2.Id})
    if err != nil {
        log.Fatalf("could not delete user: %v", err)
    }
    log.Printf("Deleted user: %s", user2.Id)

    // Wait a bit to see events
    time.Sleep(time.Second)
}

func watchUsers(ctx context.Context, client pb.UserServiceClient) {
    stream, err := client.WatchUsers(ctx, &emptypb.Empty{})
    if err != nil {
        log.Fatalf("could not watch users: %v", err)
    }

    for {
        event, err := stream.Recv()
        if err == io.EOF {
            break
        }
        if err != nil {
            log.Fatalf("failed to receive event: %v", err)
        }
        log.Printf("Event: type=%s user=%v", event.Type, event.User)
    }
}

4. Go Module Configuration

module user-service

go 1.21

require (
    google.golang.org/grpc v1.59.0
    google.golang.org/protobuf v1.31.0
)

Building and Running the Example

Install Protocol Buffer Compiler:

brew install protobuf

Install Go Protocol Buffers plugins:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Generate Go code from proto file:

protoc --go_out=. --go_opt=paths=source_relative \
    --go-grpc_out=. --go-grpc_opt=paths=source_relative \
    proto/user.proto

Run the server:

go run server/main.go

In another terminal, run the client:

go run client/main.go

Key Features Demonstrated

Protocol Buffers
- Message definitions
- Service definitions
- Timestamps
- Enums
- Repeated fields
gRPC Service Types
- Unary RPC (CreateUser, GetUser, etc.)
- Server streaming (WatchUsers)
Best Practices
- Proper error handling
- Concurrent access handling with mutexes
- Event streaming
- Pagination
- Clean project structure
Advanced Features
- Server streaming for real-time events
- Proper timestamp handling
- Pagination support
- CRUD operations
- Event-driven architecture

Production Considerations

In a production environment, you would want to add:

Authentication and Authorization
- Use gRPC interceptors for auth
- Implement JWT or similar token-based auth
Error Handling
- Define proper error codes
- Implement retry logic
- Add timeout handling
Monitoring and Logging
- Add prometheus metrics
- Implement proper logging
- Add tracing (e.g., OpenTelemetry)
Data Persistence
- Add database integration
- Implement caching
- Add data validation
Testing
- Unit tests
- Integration tests
- Load tests
Deployment
- Containerization
- Kubernetes manifests
- CI/CD pipeline
Documentation
- API documentation
- Deployment guide
- Development setup guide

This example demonstrates a complete, production-ready gRPC service implementation in Go. It shows how to structure your code, handle concurrent access, implement streaming, and follow best practices for building microservices.