Docker: nginx up and running

nginx

nginx ("Engine-X") is an open source reverse proxy server for multiple protocols, as well as a load balancer, cache and web server.

What is a reverse proxy server?
A reverse proxy server is a type of proxy server that typically sits behind the firewall in a private network and directs client requests to the appropriate back-end server. A reverse proxy provides an additional level of abstraction and control to ensure the smooth flow of network traffic between clients and servers.

What is an origin web server?
The server where web content originates. User requests that are satisfied by the origin server typically have the longest waiting times.  This term is used to differentiate content that is retrieved from a cache (and as a result, typically served faster).

nginx up and running with Docker

My Dockerfile extends from the official nginx docker image:

FROM nginx
MAINTAINER "Craig Trim <craigtrim@gmail.com>"

COPY static-html-directory /usr/share/nginx/html

Notice that I've copied a local directory on the same file path as my Dockerfile:

~/workspaces/public/nginx$ ls -lahR
total 8
drwxr-xr-x   4 craigtrim  staff   136B Jan 15 14:14 ./
drwxr-xr-x  15 craigtrim  staff   510B Jan 15 14:07 ../
-rw-r--r--   1 craigtrim  staff   107B Jan 15 14:14 Dockerfile
drwxr-xr-x   3 craigtrim  staff   102B Jan 15 14:14 static-html-directory/

./static-html-directory:
total 8
drwxr-xr-x  3 craigtrim  staff   102B Jan 15 14:14 ./
drwxr-xr-x  4 craigtrim  staff   136B Jan 15 14:14 ../
-rw-r--r--  1 craigtrim  staff    19B Jan 15 14:29 index.html
~/workspaces/public/nginx$ cat static-html-directory/index.html 
this is the index!

The HTML file contains a single line of text. Using docker, we'll build this image, launch a container, and view the HTML content in a browser.

The first step is building the image:

~/workspaces/public/nginx$ docker build -t tf/nginx .
Sending build context to Docker daemon 3.584 kB
Step 1 : FROM nginx
 ---> 407195ab8b07
Step 2 : MAINTAINER "Craig Trim <craigtrim@gmail.com>"
 ---> Using cache
 ---> c3b1042299cc
Step 3 : COPY static-html-directory /usr/share/nginx/html
 ---> b71b288606d3
Removing intermediate container 4a61cac3675b
Successfully built b71b288606d3

Launch the container:

~/workspaces/public/nginx$ docker run --name tfpmbl01 -d -p 8080:80 tf/nginx
2694b268f7a70307d0895dfbf02d1f04ee61f401b4cc0f10bbd4f0992e42ac4c

Note the naming convention.  I have multiple trained models in Tensorflow.  This container will eventually correspond to a model known internally as tfpmbl01.  This is a specific instance of the container, and known by this unique name.

At the moment, I only have one docker container running:

~/workspaces/public/nginx$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                           NAMES
2694b268f7a7        tf/nginx            "nginx -g 'daemon off"   About a minute ago   Up About a minute   443/tcp, 0.0.0.0:8080->80/tcp   tfpmbl01

In the future, there will likely be multiple containers using the tf/nginx image, but each one will potentially be configured with a different trained model; hence the naming.

Running containers can be stopped and removed using the first two digits of the container ID:

~/workspaces/public/nginx$ docker stop 26
26
~/workspaces/public/nginx$ docker rm 26
26

In order to to view the HTML content served up by nginx on my web browser, I need to know which IP the docker server is using.  The docker server is nothing more than my local Macbook, with a configured instance of Docker Machine.  For those still using boot2docker, it's time to switch.

~/workspaces/public/nginx$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default     *        virtualbox   Running   tcp://192.168.99.100:2376           v1.9.1    
dev         -        virtualbox   Error                                         Unknown   machine does not exist
vbox-test   -        virtualbox   Error                                         Unknown   machine does not exist
~/workspaces/public/nginx$ docker-machine ip default
192.168.99.100

In my web browser, I get this:

Hello, World!

The next step is to replace our static HTML with a Hello, World! script in Python.

References

1. [Docker Hub] How to use the Docker nginx Image
2. [Blog] Installation and Configuration for Docker on OS X 
1. Using Docker Machine rather than the deprecated boot2docker

Docker: Up and Running on OS X (deprecated)

Update 13-July-2016: This is an old article.  Use Docker Machine going forward.

Introduction

 The good news is, docker is getting easier.

This time last year, if you wanted to install docker on your Macbook, there were a few hoops to jump through. Now it's easier than ever. Much of the s implication has come through the use of Docker Machine.

The Docker toolbox installer works well. Download, point and click, and set up. The initial launch performs a pre-create check, and then automatically creates a boot-2-docker machine on your computer.

Post-Installation

In Spotlight search, start typing "docker ... "

Launch the terminal.

If this is the first time running a docker terminal, it may take a few minutes to set up.

Successful output will look something like this:

Running pre-create checks...
Creating machine...
(default) Copying /Users/craigtrim/.docker/machine/cache/boot2docker.iso to /Users/craigtrim/.docker/machine/machines/default/boot2docker.iso...
(default) Creating VirtualBox VM...
(default) Creating SSH key...
(default) Starting VM...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect Docker to this machine, run: /usr/local/bin/docker-machine env default

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/


docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com

~/workspaces $

Once you have the terminal accessible, the environment variables need to be established.

Run this command:

~/workspaces$ docker-machine env default
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/craigtrim/.docker/machine/machines/default"
export DOCKER_MACHINE_NAME="default"
# Run this command to configure your shell: 
# eval $(docker-machine env default)

and then follow the instructions to run:

$ eval $(docker-machine env default)

on the terminal.

This will set environment variables that the Docker client will read which specify the TLS settings. Note that you will need to do that every time you open a new tab or restart your machine.

Testing the Installation

From there, I can run this tutorial, and get:

~/workspaces$ docker build -t mybase .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu
latest: Pulling from library/ubuntu

fcee8bcfe180: Pull complete 
4cdc0cbc1936: Pull complete 
d9e545b90db8: Pull complete 
c4bea91afef3: Pull complete 
Digest: sha256:b53bb7b0d18842214ac7472c2a8801e8682c247d30f1ba4bab0083a2e2e091ea
Status: Downloaded newer image for ubuntu:latest
 ---> c4bea91afef3
Step 2 : RUN touch myfile.txt
 ---> Running in 745e3d4c316b
 ---> f5644fc04a54
Removing intermediate container 745e3d4c316b
Successfully built f5644fc04a54

Troubleshooting

1. Error checking TLS connection: machine does not exist
1. Occurs if the environment variables were not set correctly use docker machine

2. Last login: Wed Jan 13 12:11:02 on ttys005
   ~/workspaces$ docker ps
   Cannot connect to the Docker daemon. Is the docker daemon running on this host?
   ~/workspaces$ docker-machine env default
   export DOCKER_TLS_VERIFY="1"
   export DOCKER_HOST="tcp://192.168.99.100:2376"
   export DOCKER_CERT_PATH="/Users/craigtrim/.docker/machine/machines/default"
   export DOCKER_MACHINE_NAME="default"
   # Run this command to configure your shell: 
   # eval $(docker-machine env default)
   ~/workspaces$ eval $(docker-machine env default)
   ~/workspaces$ docker ps
   CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
   ~/workspaces$ 
   

2. Error checking TLS connection: Error checking and/or regenerating the certs
1. This appears to crop up from time to time on the Mac. 
2. I took three actions to mitigate this - the first was to regenerate the certificates, but this did not appear to work:
   

                           ##         .
                     ## ## ##        ==
                  ## ## ## ## ##    ===
              /"""""""""""""""""\___/ ===
         ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
              \______ o           __/
                \    \         __/
                 \____\_______/
   
   
   docker is configured to use the default machine with IP 192.168.99.100
   For help getting started, check out the docs at https://docs.docker.com
   
   ]~$ docker-machine env default
   Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": dial tcp 192.168.99.100:2376: i/o timeout
   You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
   Be advised that this will trigger a Docker daemon restart which will stop running containers.
   
   ~$ docker-machine regenerate-certs default
   Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
   Regenerating TLS certificates
   Detecting the provisioner...
   Copying certs to the local machine directory...
   Copying certs to the remote machine...
   Setting Docker configuration on the remote daemon...
   
   This machine has been allocated an IP address, but Docker Machine could not
   reach it successfully.
   
   SSH for the machine should still work, but connecting to exposed ports, such as
   the Docker daemon port (usually <ip>:2376), may not work properly.
   
   You may need to add the route manually, or use another related workaround.
   
   This could be due to a VPN, proxy, or host file configuration issue.
   
   You also might want to clear any VirtualBox host only interfaces you are not using.
   ~$ 
   

3. The second action was to open up VirtualBox and removing the host interfaces, in the event there was a conflict:
   
   I removed both of the Host-only networks listed above.
4. The third and final action to restart my laptop, and try again with my Cisco VPN client disabled.  
1. I've had trouble in the past with Cisco VPN and Docker on the Mac.  The VPN client does not allow split-tunneling, and this can cause issue with containers.  
2. Having said that, I am using Docker and the Cisco VPN together now without any apparent issues.  

References

1. https://github.com/docker/toolbox/issues/346
1.  Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates #346

Configuring a local dev environment for IBM Bluemix

What is IBM Bluemix?

Using this great pizza-as-a-service paradigm, Bluemix is a SASS:

Fig 1: Bluemix is a SAAS

Developers can use existing Bluemix services, including Alchemy API:

Fig 2: Watson services on Bluemix

Developers can also deploy their own services, including containers using Docker technology.

This article will walk through the setup of a local development environment for working with Bluemix. All scripting and configuration is for Debian64-bit (Ubuntu 14.10 at the time of this article).

Python Development Environment

The first step is to create a python development environment.

# install pip 
sudo apt-get \
  install -y \
    build-essential \
    python-dev \
    python-pip

# install python setup-utils
wget https://bootstrap.pypa.io/ez_setup.py -O - | sudo python

pip is a package management system used to install and manage software packages written in Python and Setuptools is a package development process that enhances the Python standard library distribution utilities.

Installing Docker

Instructions for installing docker on Ubuntu are referenced below and the script is replicated here:

# add the Docker repository key to your local keychain: 
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

# add the Docker repository to your apt sources list: 
sudo sh -c "echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"

# prior to using any Ubuntu-maintained package installation, ensure existing packages are up to date: 
sudo apt-get update

# install the Docker-LXC package: 
sudo apt-get install -y lxc-docker

# install docker compose:
wget https://github.com/docker/compose/releases/download/1.1.0/docker-compose-$(uname -s)-$(uname -m)
mv docker-compose-$(uname -s)-$(uname -m) docker-compose
sudo chmod 777 docker-compose
sudo mv docker-compose /usr/local/bin/
docker-compose --version

Docker is a container-based software framework for automating deployment of applications. “Containers” are encapsulated, lightweight, and portable application modules.

IBM Extensions

Before you can create a container in Bluemix, you must install the IBM Containers Extension (ICE):

sudo apt-get \
  install -y \
    curl

# install ice
export ICE=icecli-2.0.zip
curl -O https://static-ice.ng.bluemix.net/$ICE
sudo pip install icecli-2.0.zip
rm $FILE
ice version

Cloud Foundry

Finally, we need to install the Cloud Foundry command line interface:

# install CF
export FILE=cf-cli_amd64.deb
wget -O $FILE https://cli.run.pivotal.io/stable?release=debian64&version=6.11.0&source=github-rel
sudo dpkg -i $FILE
sudo apt-get install -f -y
rm $FILE
cf --version

Miscellaneous Commands

We need to endsure that the necessary environment variables are set, user authorizations are in place and configuration files are correct.

# set docker env var
sudo sh -c 'echo "export DOCKER_TLS_VERIFY=1" >> /etc/environment'

# modify user authorization
sudo usermod -a -G docker $(whoami)

By default, the ice commands must be run with root authentication and therefore, require the prefix sudo. This last command will allow us to run docker and ice commands without sudo.

References

1. [Blogger] Configuring a Python Development Environment
2. [Blogger] Installing Docker on Ubuntu
3. [GitHub] Individual Installation Scripts

Creating a Dockerfile

Introduction

You can launch a container from a docker image, modify the container, and commit the changes back to the image.  The advantage to this approach is it's likely easier for a beginner - once inside docker you have access to the terminal session and can install and configure software in a manner nearly resembling a full Linux environment.

The alternative is to modify your Dockerfile to contain the commands for installing and configuring file.  This appproach is preferred, because the Dockerfile can be easily checked into a SCM and built on other machines.

Creating a Dockerfile (Basics)

I'm going to create a directory in my home folder called "mydocker" and start editing a file within that directory called "Dockerfile".

My first Dockerfile is simple:

FROM ubuntu

RUN touch myfile.txt

and I can build it using this command:

sudo docker build -t basehadoop .

"basehadoop" is the name I give my image.
"." tells docker to look inside my current directory ("mydocker") to find the Dockerfile.

I can launch a container from the image that I've built using this command:

sudo docker run -i -t basehadoop /bin/bash

and once inside I notice the file I've created exists:

root@74e10ef8405d:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  myfile.txt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Setting Environment Variables

Use this pattern in your Dockerfile:

ENV key value

An environment variable like this

export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre

becomes

ENV JAVA_HOME /usr/lib/jvm/java-8-oracle/jre

Likewise, the PATH can be modified, using this format:

ENV PATH /usr/lib/jvm/java-8-oracle/jre/bin:$PATH

When I re-build my image with this change, I get this:

craig@dockerhost:~/mydocker$ sudo docker build -t basehadoop .
Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon 
Step 0 : FROM ubuntu
 ---> b39b81afc8ca
Step 1 : RUN touch myfile.txt
 ---> Using cache
 ---> 0e83d857ff15
Step 2 : ENV JAVA_HOME /usr/lib/jvm/java-8-oracle/jre
 ---> Running in 1204292998bf
 ---> 90808840643c
Removing intermediate container 8cf05deb31d5
Successfully built 849723f3f1d2

When I view the json file under the graphs folder, I see this JSON:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

{ "Size": 0, "architecture": "amd64", "config": { "AttachStderr": false, "AttachStdin": false, "AttachStdout": false, "Cmd": [ "/bin/bash" ], "CpuShares": 0, "Cpuset": "", "Domainname": "", "Entrypoint": null, "Env": [ "PATH=/usr/lib/jvm/java-8-oracle/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre" ], "ExposedPorts": null, "Hostname": "2338b11efbe1", "Image": "90dd9b6eab1d605a10e151660f2a841c4bc64978fb85d3424f97bdb40c94be28", "Memory": 0, "MemorySwap": 0, "NetworkDisabled": false, "OnBuild": [], "OpenStdin": false, "PortSpecs": null, "StdinOnce": false, "Tty": false, "User": "", "Volumes": null, "WorkingDir": "" }, "container": "5c5de1bd02358774ee5766979a95e59638884ae59d2af77523ced52805df944d", "container_config": { "AttachStderr": false, "AttachStdin": false, "AttachStdout": false, "Cmd": [ "/bin/sh", "-c", "#(nop) ENV STO=/media/data2/STO/els" ], "CpuShares": 0, "Cpuset": "", "Domainname": "", "Entrypoint": null, "Env": [ "PATH=/usr/lib/jvm/java-8-oracle/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre", "ECLIPSE_HOME=/opt/eclipse/luna", "MAVEN_HOME=/usr/lib/apache/maven/3.2.5", "HADOOP_HOME=/usr/lib/apache/hadoop/2.6.0", "HADOOP_DATA_DIR=/home/craig/HADOOP_DATA_DIR", "HADOOP_CONF_DIR=/usr/lib/apache/hadoop/2.6.0/conf", "HADOOP_LOG_DIR=/usr/lib/apache/hadoop/2.6.0/logs", "WS=~/workspaces", "WSCOM=/home/craig/workspaces/swtk/common/projects/swtk-common", "WSSAN=/home/craig/workspaces/swtk/sandbox/projects/swtk-sandbox", "NYT=/media/data/NYT/02_Unzipped/02_TXT", "STO=/media/data2/STO/els" ], "ExposedPorts": null, "Hostname": "2338b11efbe1", "Image": "90dd9b6eab1d605a10e151660f2a841c4bc64978fb85d3424f97bdb40c94be28", "Memory": 0, "MemorySwap": 0, "NetworkDisabled": false, "OnBuild": [], "OpenStdin": false, "PortSpecs": null, "StdinOnce": false, "Tty": false, "User": "", "Volumes": null, "WorkingDir": "" }, "created": "2015-03-26T19:04:55.686342953Z", "docker_version": "1.2.0", "id": "73e5400f4bb13d4e12b0c457d8fb77603691e983254ba1a55b593140e9c096a6", "os": "linux", "parent": "90dd9b6eab1d605a10e151660f2a841c4bc64978fb85d3424f97bdb40c94be28" }

Note lines 16-17 that contain the modified PATH and environment variables

When I launch the container from this modified image, I get this:

craig@dockerhost:~/mydocker$ sudo docker run -i -t basehadoop /bin/bash
root@0dab6fe52a8e:/# echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle/jre
root@0dab6fe52a8e:/# echo $PATH
/usr/lib/jvm/java-8-oracle/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

References

1. [StackOverflow] How to update the PATH
2. Reference Dockerfiles:
1. [Github] Dockerfile for Java installation
2. [Github] sequenceiq/hadoop
3. [Blog] Removing untagged images from docker:
1. Remove all stopped containers:
   

   sudo docker rm $(sudo docker ps -a -q)
   

2. Remove all untagged images
   

   sudo docker rmi $(sudo docker images | grep "^<none>" | awk "{print $3}")
   

Installing Docker on Ubuntu

What is Docker?

Docker is a container-based software framework for automating deployment of applications. “Containers” are encapsulated, lightweight, and portable application modules.

Environment

1. Ubuntu 14.10
2. Docker 1.2.0 or 1.4.0

Ubuntu Package Installation

Ubuntu-maintained package installation is simple, but does not contain the most recent Docker release. At the time of this article, Docker 1.4.0 is available, but the instructions below will install 1.2.0. Check with your cloud provider for container compatbility.

Prior to using any Ubuntu-maintained package installation, ensure existing packages are up to date:

$ sudo apt-get update

Install Docker:

$ sudo apt-get -y install docker.io

Note: Ubuntu (and Debian) contain a much older KDE3/GNOME2 package called docker, so the Ubuntu-maintained package and executable are named docker.io.

Link and Fix Paths:

$ ln -sf /usr/bin/docker.io /usr/local/bin/docker
$ sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker

Configure Docker to start when the server boots:

$ update-rc.d docker defaults

Docker-maintained Package Installation

Use this approach to install the latest version of Docker.

Add the Docker repository key to your local keychain:

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.TI1OtMmq4G --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
gpg: requesting key A88D21E9 from hkp server keyserver.ubuntu.com
gpg: key A88D21E9: public key "Docker Release Tool (releasedocker) <docker@dotcloud.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Add the Docker repository to your apt sources list:

$ sudo sh -c "echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"

Prior to using any Ubuntu-maintained package installation, ensure existing packages are up to date:

$ sudo apt-get update

Install the Docker-LXC package:

$ sudo apt-get install lxc-docker

To verify that everything has worked as expected, download the ubuntu image, and then start bash in a container:

$ sudo docker run -i -t ubuntu /bin/bash
root@6932168c97d3:/# exit
exit
$ 

Script

The complete script for this article:

# add the Docker repository key to your local keychain: 
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

# add the Docker repository to your apt sources list: 
sudo sh -c "echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"

# prior to using any Ubuntu-maintained package installation, ensure existing packages are up to date: 
sudo apt-get update

# install the Docker-LXC package: 
sudo apt-get install lxc-docker -y

# install docker compose
wget https://github.com/docker/compose/releases/download/1.1.0/docker-compose-$(uname -s)-$(uname -m)
mv docker-compose-$(uname -s)-$(uname -m) docker-compose
sudo chmod 777 docker-compose
sudo mv docker-compose /usr/local/bin/
docker-compose --version

References

1. Resources:
1. [GitHub] Docker Installation Script (Debian/Ubuntu)
2. [GitHub] Dockerfiles (that I've extended or created)
2. [Docker] Installation on Ubuntu
3. [LiquidWeb] Installing Docker on 14.04