Setting up private Docker registry behind Nginx

Thu 28 August 2014

I spent quite a lot of time with setting up my private Docker registry. As there are some caveats, I wrote a tutorial how to setup your own Docker registry server with SSL encryption and HTTP basic authentication.

Prerequisites

I will use Ubuntu 13.10 with Docker 1.2 installed, but you can use any Linux based operating system with kernel 3.8 or newer. Docker images take some space, so make sure you have a few GB free on your hard drive. I will set up local filesystem as a storage.

The registry server

This is the easiest part. You only need to start a Docker container downloaded from the official registry.

mkdir -p /storage/private_registry
chmod 755 /storage && chmod 750 /storage/private_registry
chown 10000:10000 /storage/private_registry
docker run \
    -d \
    --name private_registry \
    -e SETTINGS_FLAVOUR=local \
    -e STORAGE_PATH=/registry-storage \
    -v /storage/private_registry:/registry-storage \
    -u 10000 \
    -p 127.0.0.1:5000:5000 \
    registry:0.8.0

All these commands are run as root. Docker will download an image named registry that is tagged with 0.8.0 and run it in the background. This is the official image built from docker-registry application hosted on GitHub.

If you run docker ps, you will see the running container:

CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                      NAMES
e32f04ed78ac        registry:0.8.0          "/bin/sh -c 'exec do   2 seconds ago       Up 1 seconds        127.0.0.1:5000->5000/tcp   private_registry

To use the local filesystem for Docker images, we set SETTINGS_FLAVOUR to value local and STORAGE_PATH to the path inside the container where we want images to be stored. This directory will be mounted from the host. I am using /storage/CONTAINER_NAME for all my containers data.

It is safer to run applications inside containers as unprivileged user, so I choosed UID 10000. The user must not exist in host's /etc/passwd file, you only need to set the same UID on the host's /storage/private_registry directory that will be accessible from the container.

Before we can push to our private registry, we need to tag an image with the hostname or IP address of the registry server. I will push the busybox image I downloaded from the official registry:

docker pull busybox
docker tag busybox:latest 127.0.0.1:5000/busybox:latest
docker push 127.0.0.1:5000/busybox:latest

If we didn't make any mistakes, the image should be uploaded, so check the content of our volume on the host:

ls -la /storage/private_registry
total 16
drwxr-x--- 4 10000 10000 4096 Aug 28 16:51 .
drwxr-xr-x 7 root  root  4096 Aug 28 16:50 ..
drwxr-xr-x 6 10000 root  4096 Aug 28 16:51 images
drwxr-xr-x 3 10000 root  4096 Aug 28 16:51 repositories

Now we stop and destroy this container, because we don't want it to be directly accessible (we will set a reverse proxy later) and we create a new one where we remove the -p parameter:

docker stop private_registry && docker rm private_registry
docker run \
    -d \
    --name private_registry \
    -e SETTINGS_FLAVOUR=local \
    -e STORAGE_PATH=/registry-storage \
    -v /storage/private_registry:/registry-storage \
    -u 10000 \
    registry:0.8.0

SSL certificate

We will need to create a SSL certificate, because the Docker client does not allow unencrypted connection with HTTP basic authentication. The first part is to create private key and a certificate signing request (CSR):

mkdir -p /storage/private_registry_nginx/ssl && cd $_
openssl genrsa -out private-registry.key 2048
openssl req -new -key private-registry.key -out private-registry.csr
chmod 440 private-registry.key

Answer all questions as it fits for you and as common name fill in the hostname of your new registry server (IP address will not work with the Docker client). I will use private-registry.localhost as the hostname (and add it to /etc/hosts). The password should be left blank, you don't want to insert it every time the web server starts.

Now we have the CSR, so it is time to sign it. If you just want a self signed certificate you can sign it with this command:

openssl x509 -req -days 365 -in private-registry.csr -signkey private-registry.key -out private-registry.crt

But I will create my own certificate authority (CA) as I will use it for another purposes too and can it install system wide. So instead of the above command, I will type the following:

openssl genrsa -out MyOwnCA.key 2048
openssl req -x509 -new -nodes -key MyOwnCA.key -days 1000 -out MyOwnCA.pem

And I will use the new CA certificate to sign the CSR:

openssl x509 -req -in private-registry.csr -CA MyOwnCA.pem -CAkey MyOwnCA.key -CAcreateserial -out private-registry.crt -days 500

Move the files for your CA somewhere else so you have only following content in the /storage/private_registry_nginx/ssl directory:

pwd
/storage/private_registry_nginx/ssl
ls -la
total 20
drwxr-x--- 2 root root 4096 Aug 29 11:11 .
drwxr-x--- 3 root root 4096 Aug 29 11:09 ..
-rw-r----- 1 root root 1164 Aug 29 11:11 private-registry.crt
-rw-r----- 1 root root  985 Aug 29 11:10 private-registry.csr
-r--r----- 1 root root 1679 Aug 29 11:10 private-registry.key

To install the CA certificate on Ubuntu, copy MyOwnCA.pem to directory /usr/local/share/ca-certificates/, change extension from .pem to .crt and run update-ca-certificates. From now, all programs using standard certificate storage (not web browsers, there you will have to install the CA explicitly) will recognize all certificates signed by our CA.

Nginx web server

We create a file named Dockerfile with the following content:

FROM ubuntu:14.04

RUN apt-get update && apt-get -y upgrade

RUN apt-get -y install nginx-extras && \
    rm /var/lib/apt/lists/*_*

RUN rm /etc/nginx/sites-enabled/default

ADD config /

EXPOSE 443

CMD ["/usr/sbin/nginx"]

Place the file wherever you want (for example into ~/Dockerfiles/private_registry_nginx) and in the same directory create another directory with name config. In the config directory we create two configuration files. During the build, the root filesystem of our new image will be overwritten with the content of this directory.

The ADD command in Dockerfile will not only copy the files but also set directory permissions along the path. My default umask is set to 0027 so when I create etc/nginx inside the config dir and both directories have permissions 0750, the final image will end up with the same permissions on these two directories. Nginx workers are running as www-data user and will have no access to htpasswd file in /etc/nginx, so you will end up with internal server error.
upstream private-registry {
    server registry.srv:5000;
}

server {
    listen 443;
    server_name _;

    ssl on;
    ssl_certificate /etc/nginx/ssl/private-registry.crt;
    ssl_certificate_key /etc/nginx/ssl/private-registry.key;

    proxy_set_header Host             $http_host;
    proxy_set_header X-Real-IP        $remote_addr;
    proxy_set_header Authorization    "";

    client_max_body_size 0;

    chunked_transfer_encoding on;

    location / {
        auth_basic              "Restricted";
        auth_basic_user_file    /etc/nginx/private-registry.htpasswd;

        proxy_pass http://private-registry;
        proxy_set_header Host $host;
        proxy_read_timeout 900;
    }

    location /_ping {
        auth_basic off;
    }

    location /v1/_ping {
        auth_basic off;
    }
}

We save this file under etc/nginx/sites-enabled/private-registry in the config directory and the second file as etc/nginx/nginx.conf:

user www-data;
worker_processes 2;
pid /run/nginx.pid;
daemon off;

events {
    worker_connections 100;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    server_tokens off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    gzip on;
    gzip_disable "msie6";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Now we build our web server image (you will have to be in the same directory as your Dockerfile and don't forget the dot on the end):

docker build -t private_registry_nginx .

If there are no errors, after some while you will end up with the Nginx image. Next, we will need to create the htpasswd file. We use the htpasswd command (install apache2-utils package if you don't have it already):

htpasswd -c /storage/private_registry_nginx/private-registry.htpasswd someusername

And now, we can start the container:

docker run \
    -d \
    --name private_registry_nginx \
    --link private_registry:registry.srv \
    -v /storage/private_registry_nginx/private-registry.htpasswd:/etc/nginx/private-registry.htpasswd \
    -v /storage/private_registry_nginx/ssl:/etc/nginx/ssl \
    -p 127.0.0.1:443:443 \
    private_registry_nginx

This container will be linked with our private_registry container. This means that from inside of the private_registry_nginx container you can access private_registry (through hostname registry.srv). We mount our htpasswd file and SSL certificate inside the container private_registry_nginx and finally expose port 443 (HTTPS).

Now if you open https://private-registry.localhost/ you should get "docker-registry server (dev) (v0.8.0)" response.

Before you can push to your private registry, you need to log in. Use the username and password you set in your private-registry.htpasswd file:

docker tag busybox:latest private-registry.localhost/busybox:latest
docker login private-registry.localhost
docker push private-registry.localhost/busybox:latest
In some cases the Docker client does not print usefull errors, so I recommend to install docker-enter application, enter into Nginx container and check /var/log/nginx/access.log and /var/log/nginx/error.log or the registry container with docker logs -f private_registry. If there is an issue with the certificate, docker login does not fill the access.log.
comments powered by Disqus