pmcgrath

Using pod security policies with kubeadm

Thu, 13 Sep 2018 00:00:00 +0000

Purpose

Illustrate using pod security policies with a kubeadm installation of kubernetes

Pod security policies are a mechanism to restrict what a container can do when run on kubernetes such as preventing running as privileged containers, running with host networking etc.

Read the docs to see how this can be used to improve security

I struggled to find any information on bootstrapping a kubeadm cluster with the same, hence this content

TLDR

This post is very long so I can do a full illustration, in short you need to

On master run kubeadm init with the PodSecurityPolicy admission controller enabled
Add some pod security policies with RBAC config - enough to allow CNI and DNS etc. to start
- CNI daemonsets will not start without this
Apply your CNI provider which can use one of the previously created pod security policies
Complete configuring the cluster adding nodes via kubeadm join
As you add more workloads to the cluster check if you need additional pod security policies and RBAC configuration for the same

What we will do

This is the list of steps I took to get pod security policies running on a kubeadm installation

Configure the pod security policy admission controller for master init
Configure some pod security policies for the control plane components
Configure a CNI provider - Will use flannel here

Will then use the following to demo some other pod security policy scenarios

Install an nginx-ingress controller which has some specfic requirements - This is just to illustrate adding additional policies
Install a regular service that has no specific pod security policy requirements - Based on httpbin.org

Environment

Will just illustrate on a single node rather than a multi-node cluster with HA
Ubuntu
- Swap off as required by kubeadm
- Timezone configured for UTC
Docker
- Assumes current user is in the docker group
kubernetes 1.11.3 - with RBAC

Prepare master

Follow the instructions to install kubeadm

Lets install jq which we will use for some json output processing

sudo apt-get update
sudo apt-get install -y jq

Verify kubeadm version

sudo kubeadm version

Create a directory somewhere for the content we will create below, all below instructions assume you are in this directory

mkdir ~/psp-inv
cd ~/psp-inv

kubeadm config file

Will create this file and use it for kubeadm init on the master

Create a kubeadm-config.yaml file with this content - note we have to specify the podSubnet of 10.244.0.0/16 for flannel

Note this file is minimal for this demo and if you use a later version of kubeadm you may need to alter the apiVersion

apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
apiServerExtraArgs:
  enable-admission-plugins: PodSecurityPolicy
controllerManagerExtraArgs:
  address: 0.0.0.0
kubernetesVersion: v1.11.3
networking:
  podSubnet: 10.244.0.0/16
schedulerExtraArgs:
  address: 0.0.0.0

Master init

sudo kubeadm init --config kubeadm-config.yaml

Follow the instructions from the above command output to get your own copy of the kubeconfig file

If you want to add worker nodes to the cluster, note the join message

Lets check the master node status with

kubectl get nodes

NAME                    STATUS     ROLES     AGE       VERSION
pmcgrath-k8s-master     NotReady   master    1m        v1.11.3

So the node is not ready as it is waiting for CNI

Lets check the pods

kubectl get pods --all-namespaces
No resources found.

So none appear to be running, would normally see pods with some pending if we had not enabled the pod security policy admission control

Lets check docker

docker container ls --format '{{ .Names }}'

k8s_kube-scheduler_kube-scheduler-pmcgrath-k8s-master_kube-system_a00c35e56ebd0bdfcd77d53674a5d2a1_0
k8s_kube-controller-manager_kube-controller-manager-pmcgrath-k8s-master_kube-system_fd832ada507cef85e01885d1e1980c37_0
k8s_etcd_etcd-pmcgrath-k8s-master_kube-system_16a8af6b4a79e9b0f81092f85eab37cf_0
k8s_kube-apiserver_kube-apiserver-pmcgrath-k8s-master_kube-system_db201a8ecaf8e99623b425502a6ba627_0
k8s_POD_kube-controller-manager-pmcgrath-k8s-master_kube-system_fd832ada507cef85e01885d1e1980c37_0
k8s_POD_kube-scheduler-pmcgrath-k8s-master_kube-system_a00c35e56ebd0bdfcd77d53674a5d2a1_0
k8s_POD_kube-apiserver-pmcgrath-k8s-master_kube-system_db201a8ecaf8e99623b425502a6ba627_0
k8s_POD_etcd-pmcgrath-k8s-master_kube-system_16a8af6b4a79e9b0f81092f85eab37cf_0

So containers are running, but not showing up with kubectl

Lets check events

kubectl get events --namespace kube-system

will see something like Error creating: pods “kube-proxy-“ is forbidden: no providers available to validate pod request

Configure pod security policies

I have went with configuring

A default pod security policy that any workload can use, has no privileges and should be good for most workloads
- Will create an RBAC ClusterRole
- Will create an RBAC ClusterRoleBinding for any authenticated users
A privileged pod security policy that I grant nodes and all service accounts in the kube-system namespace access to
- Thinking is access to this namespace is restricted
- Should only run k8s components in this namespace
- Will create an RBAC ClusterRole
- Will create an RBAC RoleBinding in the kube-system namespace

Create a default-psp-with-rbac.yaml file with this content

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'docker/default'
  name: default
spec:
  allowedCapabilities: []  # default set of capabilities are implicitly allowed
  allowPrivilegeEscalation: false
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  hostIPC: false
  hostNetwork: false
  hostPID: false
  privileged: false
  readOnlyRootFilesystem: false
  runAsUser:
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'RunAsNonRoot'
  supplementalGroups:
    rule: 'RunAsNonRoot'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  volumes:
  - 'configMap'
  - 'downwardAPI'
  - 'emptyDir'
  - 'persistentVolumeClaim'
  - 'projected'
  - 'secret'
  hostNetwork: false
  runAsUser:
    rule: 'RunAsAny'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  fsGroup:
    rule: 'RunAsAny'

---

# Cluster role which grants access to the default pod security policy
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: default-psp
rules:
- apiGroups:
  - policy
  resourceNames:
  - default
  resources:
  - podsecuritypolicies
  verbs:
  - use

---

# Cluster role binding for default pod security policy granting all authenticated users access
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-psp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: default-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated

Create a privileged-psp-with-rbac.yaml file with this content

# Should grant access to very few pods, i.e. kube-system system pods and possibly CNI pods
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    # See https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'
  name: privileged
spec:
  allowedCapabilities:
  - '*'
  allowPrivilegeEscalation: true
  fsGroup:
    rule: 'RunAsAny'
  hostIPC: true
  hostNetwork: true
  hostPID: true
  hostPorts:
  - min: 0
    max: 65535
  privileged: true
  readOnlyRootFilesystem: false
  runAsUser:
    rule: 'RunAsAny'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'RunAsAny'
  volumes:
  - '*'

---

# Cluster role which grants access to the privileged pod security policy
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: privileged-psp
rules:
- apiGroups:
  - policy
  resourceNames:
  - privileged
  resources:
  - podsecuritypolicies
  verbs:
  - use

---

# Role binding for kube-system - allow nodes and kube-system service accounts - should take care of CNI i.e. flannel running in the kube-system namespace
# Assumes access to the kube-system is restricted
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-system-psp
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
# For the kubeadm kube-system nodes
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
# For all service accounts in the kube-system namespace
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:kube-system

Apply the above pod security policies with RBAC configuration

kubectl apply -f default-psp-with-rbac.yaml
kubectl apply -f privileged-psp-with-rbac.yaml

Check

Control plane pods will turn up in a running state after some time, coredns pods will be pending - waiting on CNI

kubectl get pods --all-namespaces --output wide --watch

Control plane pods will start failing again until CNI is configured, as the node is still not ready

Install flannel

See here

Will only be able to complete this as the privileged pod security policy will now exist and the flannel service account in the kube-system will be able to use

If using a different CNI provider you should use their installation instructions, will probably need to alter the podSubnet in the kubeadm-config.yaml file used for kubeadm init

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Check

kubectl get pods --all-namespaces --output wide --watch

All pods will eventually get to a running status including coredns pod(s)

kubectl get nodes

Node is now ready

Allow workloads on the master

If you want to spin up worker nodes, you can do so as normal using the kubeadm join command using the output from kubeadm init, skipping this here

Nothing special needed on worker nodes joining the cluster pod security policy wise

To allow workloads on the master node, as we are just trying to verify on a single node cluster

kubectl taint nodes --all node-role.kubernetes.io/master-

nginx ingress

Will use the manifest from here

This will create a new namespace and a single instance ingress controller, which is enough to illustrate additional pod security policies

Namespace

Since the namespace will not yet exist, lets create so we can reference service accounts and create a role binding

kubectl create namespace ingress-nginx

Lets create a pod security policy

This pod security policy is based on the deployment manifest

Create a file nginx-ingress-psp-with-rbac.yaml with this content

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    # Assumes apparmor available
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'docker/default'
  name: ingress-nginx
spec:
  # See nginx-ingress-controller deployment at https://github.com/kubernetes/ingress-nginx/blob/master/deploy/mandatory.yaml
  # See also https://github.com/kubernetes-incubator/kubespray/blob/master/roles/kubernetes-apps/ingress_controller/ingress_nginx/templates/psp-ingress-nginx.yml.j2
  allowedCapabilities:
  - NET_BIND_SERVICE
  allowPrivilegeEscalation: true
  fsGroup:
    rule: 'MustRunAs'
    ranges:
    - min: 1
      max: 65535
  hostIPC: false
  hostNetwork: false
  hostPID: false
  hostPorts:
  - min: 80
    max: 65535
  privileged: false
  readOnlyRootFilesystem: false
  runAsUser:
    rule: 'MustRunAsNonRoot'
    ranges:
    - min: 33
      max: 65535
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
    # Forbid adding the root group.
    - min: 1
      max: 65535
  volumes:
  - 'configMap'
  - 'downwardAPI'
  - 'emptyDir'
  - 'projected'
  - 'secret'

---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-psp
  namespace: ingress-nginx
rules:
- apiGroups:
  - policy
  resourceNames:
  - ingress-nginx
  resources:
  - podsecuritypolicies
  verbs:
  - use

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-psp
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-psp
subjects:
# Lets cover default and nginx-ingress-serviceaccount service accounts
# Could have altered default-http-backend deployment to use the same service acccount to avoid granting the default service account access
- kind: ServiceAccount
  name: default
- kind: ServiceAccount
  name: nginx-ingress-serviceaccount

Lets apply

kubectl apply -f nginx-ingress-psp-with-rbac.yaml

Create nginx-ingress workload

Will remove the controller –publish-service arg as we do not need here

curl -s https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml | sed '/--publish-service/d'  | kubectl apply -f -

Check for pods

kubectl get pods --namespace ingress-nginx --watch

Can now see the pod security policy is attached with an annotation with

kubectl get pods --namespace ingress-nginx --selector app.kubernetes.io/name=ingress-nginx -o json | jq -r  '.items[0].metadata.annotations."kubernetes.io/psp"'

Httpbin.org workload

Lets deploy a workload where the default pod security policy will suffice

Create a httpbin.yaml file with this content

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: httpbin
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: httpbin
  template:
    metadata:
      labels:
        app.kubernetes.io/name: httpbin
    spec:
      containers:
      - args: ["-b", "0.0.0.0:8080", "httpbin:app"]
        command: ["gunicorn"]
        image: docker.io/kennethreitz/httpbin:latest
        imagePullPolicy: Always
        name: httpbin
        ports:
        - containerPort: 8080
          name: http
      restartPolicy: Always

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
  labels:
    app.kubernetes.io/name: httpbin
  name: httpbin
spec:
  rules:
  - host: my.httpbin.com
    http:
      paths:
      - path:
        backend:
          serviceName: httpbin
          servicePort: 8080

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: httpbin
  name: httpbin
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app.kubernetes.io/name: httpbin

Create namespace and run in workload

kubectl create namespace demo
kubectl apply --namespace demo -f httpbin.yaml

Lets check that the pod exists and the default policy was used

kubectl get pods --namespace demo

kubectl get pods --namespace demo --selector app.kubernetes.io/name=httpbin -o json | jq -r  '.items[0].metadata.annotations."kubernetes.io/psp"'

Test workload

Will do so by calling via ingress controller pod instance - I have no ingress service for this demo

# Get nginx ingress controller pod IP
nginx_ip=$(kubectl get pods --namespace ingress-nginx --selector app.kubernetes.io/name=ingress-nginx --output json | jq -r .items[0].status.podIP)

# Test ingress and out httpbin workload
curl -H 'Host: my.httpbin.com' http://$nginx_ip/get

Resets

If like me you mess this up regularly, you can reset and restart with

# Note: Will loose PKI also which is fine here as kubeadm master init will re-create
sudo kubeadm reset

# Should flush iptable rules after a kubeadm reset, see https://blog.heptio.com/properly-resetting-your-kubeadm-bootstrapped-cluster-nodes-heptioprotip-473bd0b824aa
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

Running a simple dotnet Core Linux daemon

Wed, 05 Apr 2017 00:00:00 +0000

Purpose

Someone has asked me about running a dotnet Core service as a Linux daemon, this is a very trivial example

Running a ASP.NET service should be much the same, as all project types result in console applications, so the generated project’s main method will include a blocking call on host.Run()

Environment

Ubuntu 16.04 using SystemD
dotnet Core 1.1

Create application

This is just a simple console application that writes a message to stdout

# Create application
mkdir dnsvc
cd dnsvc
dotnet new console

# Change Program.cs
cat > Program.cs <<EOF
using System;
using System.Threading;


namespace dnsvc
{
  class Program
  {
    static void Main(
      string[] args)
    {
      var sleep = 3000;
      if (args.Length > 0) { int.TryParse(args[0], out sleep); }
      while (true)
      {
        Console.WriteLine($"Working, pausing for {sleep}ms");
        Thread.Sleep(sleep);
      }
    }
  }
}
EOF

# Restore dependencies
dotnet restore

# Publish to a local bin sub directory
dotnet publish --configuration Release --output bin

# Run local to verify all is good
dotnet ./bin/dnsvc.dll

Create SystemD service file

Will run the application from the bin sub directory for now

cat > dnsvc.service <<EOF
[Unit]
Description=Demo service
After=network.target

[Service]
ExecStart=/usr/bin/dotnet $(pwd)/bin/dnsvc.dll 5000
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

Configure SystemD so it is aware of the new service

# Copy service file to a System location
sudo cp dnsvc.service /lib/systemd/system

# Reload SystemD and enable the service, so it will restart on reboots
sudo systemctl daemon-reload 
sudo systemctl enable dnsvc

# Start service
sudo systemctl start dnsvc 

# View service status
systemctl status dnsvc

Tail the service log

Since we are just writing to stdout the output can be examined with journalctl

journalctl --unit dnsvc --follow

Stopping and restarting the service

# Stop service
sudo systemctl stop dnsvc 
systemctl status dnsvc 

# Restart the service
sudo systemctl start dnsvc 
systemctl status dnsvc

Cleaning up

# Ensure service is stopped
sudo systemctl stop dnsvc 

# Disable
sudo systemctl disable dnsvc 

# Remove and reload SystemD
sudo rm dnsvc.service /lib/systemd/system/dnsvc.service 
sudo systemctl daemon-reload 

# Verify SystemD is no longer aware of the service - Empty is what we want here
systemctl --type service |& grep dnsvc 

Gracefull shutdown - Handling signals

The service should handle signals like SIGTERM, SIGINT etc.
Would use this to do any cleanup just before exiting, like docker stop for containers
Current version of dotnet Core does NOT handle this as far as I know
You can follow this issue, should hopefully be sorted in the near future

Using KMS in the shell

Mon, 20 Feb 2017 00:00:00 +0000

Purpose

Some notes on using AWS KMS from the shell

Create an AWS customer master key if you do not already have one

key_id=$(aws kms create-key | jq -r .KeyMetadata.KeyId)
aws kms create-alias --alias-name alias/my-test-key --target-key-id $key_id

Policy

You should use policy to restrict access to the key
In this sample I presume you have access to the key

Script with encrypt\decrypt and some test functions

#!/bin/bash
# This assumes you have configured the AWS region and credentials, if not it will fail
# We do not set errexit or pipefail as the environment sourcing this should configure this
# Can use set -o to check

kms-decrypt()
{
	context=$1
	cipher=$2
	region=${3:-$AWS_REGION}

	aws kms decrypt --ciphertext-blob fileb://<(echo $cipher | base64 --decode) --region $region --output text --query Plaintext | base64 --decode
}

kms-encrypt()
{
	key=$1
	context=$2
	plain=$3
	region=${4:-$AWS_REGION}

	aws kms encrypt --key-id alias/${key} --plaintext "$plain" --region $region --output text --query CiphertextBlob
}

# Tests
assert()
{
	# If you have set errexit this will fail and exit immediately, should test with set +o errexit
	eval "$1"
	[[ $? -eq 0 ]] && echo -e "\e[32m${FUNCNAME[1]}: ${1} passed\e[0m" || echo -e "\e[31m ${FUNCNAME[1]}: ${1} failed\e[0m"
}

test-kms-roundtrip-with-no-context()
{
	key=my-test-key
	context=
	original_data='eykjsgh$^%46546112--dff09865-END'

	cipher=$(kms-encrypt $key "$context" "$original_data")
	plain=$(kms-decrypt "$context" "$cipher")

	assert "[ '$original_data' == '$plain' ]"
}

test-kms-roundtrip-with-context()
{
	key=my-test-key
	context='k1=v1,k2=v2'
	original_data='eykjsgh$^%46546112--dff09865-OTHER-END.'

	cipher=$(kms-encrypt $key "$context" "$original_data")
	plain=$(kms-decrypt "$context" "$cipher")

	assert "[ '$original_data' == '$plain' ]"
}

Getting started with Clojure

Thu, 19 Mar 2015 00:00:00 +0000

Purpose

These are just a couple of pointers for getting started with Clojure, as I have been asked this question and I know when I started I would have appreciated the same content. I am new enough to Clojure so this content is based on my current understanding. This post will explain how to get an Ubuntu linux JVM environment up and running so we can explore Clojure. You should be able to setup an environment on other OSes by using the equivalent OS commands.

What is Clojure

Clojure is a dialect of the Lisp programming language. It is a dynamic functional language that runs in a hosted environment. The most popular hosted environment is the JVM. There are versions which can be hosted on other environments, see CLR and JavaScript.

Docker

You can use the offical docker image which will have the pre-requisites already installed.

Pre-requisites

This section will describe setting up your local machine, if you are using the docker image this does not apply.

JVM

JVM - Ensure we have a JVM, so we have a hosted environment for Clojure. Clojure currently requires Java 1.6 or greater. You can check if you already have Java and which JVM using the following

# Do I already have Java ?
which java

# Which Java to I already have ?
dpkg -S $(readlink -f $(which java))

If you need to install Java, there are many JVMs, I will use OpenJDK 7

# Ensure our package lists are up to date
apt-get update

# Install OpenJDK 7
apt-get install -y openjdk-7-jdk

Minimal Clojure usage

Since we already have a JVM, we just need the Clojure JAR itself. Clojure is just another Java package.

Get Clojure JAR extracting to a temp directory

# Make temp directory in /tmp and change to this directory - This directory will be deleted by the OS in time
dir_name=$(mktemp -d)
cd $dir_name

# Download jar - its in a zip file so we need to extract (Requires unzip utility)
wget http://repo1.maven.org/maven2/org/clojure/clojure/1.6.0/clojure-1.6.0.zip
unzip clojure-1.6.0.zip 

# Copy JAR to this directory
cp clojure-1.6.0/clojure-1.6.0.jar .

Start a REPL

A REPL is a read eval print loop that allows interactive programming. Ruby has irb, erlang has erl etc.

# Run java with clojure-1.6.0.jar in the classpath and execute clojure.main entry point
java -cp clojure-1.6.0.jar clojure.main

You can now execute Clojure expressions such as (+ 1 2) which will be evaluated and the result of the expression will be printed. Ctrl-d to exit the REPL.

Run a program

You can run a program - silly example but just to illustrate

# Create app.clj file which just prints Hello world
echo '(println "Hello world")' > app.clj

# Run program
java -cp clojure-1.6.0.jar clojure.main app.clj

leiningen

leiningen seems to be the most popular way to manage Clojure code. The tutorial details the functions and extensions that this tool provides. Some of the features you can use it for

Installing Clojure (Let it manage this rather than using the minimal Clojure usage described above)
Creating projects - Like scaffolding in other enviornments (Ruby on Rails, ASP.NET)
Directory layout standardisation
Managing dependencies (Like Bundler, go get, nuget etc)
Running a REPL
Running the application
Running tests
Builds (Building a single application Uberjar)
Plugins (Like managing ClojureScript builds)

Installation

See the projet site

# Create bin directory if it does not already exist
[ -d ~/bin ] || mkdir ~/bin

# Get leiningen script and make executable
cd ~/bin
wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
chmod +x ~/bin/lein

# May need to exit and reload shell if bin directory did not already exist so it will be on the path
# See ~/.profile which should add ~/bin to path if it exists

# Run lein - Will download and self-install the leiningen package
lein 

Basic hello world application

Create new app (With a main function so it can be run from the command line), this uses the app template to create a main entry point function.

# Create new app
lein new app hello
cd hello

# List all files excluding target directory content
find -type f -not -path './target/*' | sort
> ./doc/intro.md
> ./.gitignore
> ./.hgignore
> ./LICENSE
> ./project.clj
> ./README.md
> ./src/hello/core.clj
> ./test/hello/core_test.clj

# Run app 
lein run

The important files are

src/hello/core.clj - This file contains code, and the entry point function main
test/hello/core.clj - This file contains a test, the test will fail until corrected
project.clj - This file contains project attributes, dependencies, leiningen plugins etc.

REPL

You can start a REPL using

lein repl

This will start a REPL and switch to the “hello.core” namespace.
All of the dependencies in the project.clj will be available, they will all be included in the classpath and can be required.
You can execute functions that have been defined.
You can run tests that have been defined.

To run already defined functions within the same REPL

; Execute the -main function which is defined in the src/hello/core.clj file
(-main)

To run the test function that was created (This is a sample that is set to fail) within the same REPL

; Require namespaces
(require 'clojure.test)
(require 'hello.core-test)

; Execute the run-tests function for the test namespace
(clojure.test/run-tests 'hello.core-test)

You can also see the documentation and source for functions within the repl as follows

; Get the documentation for the map function
(doc map)

; Get the source for the map function
(source map)

Basic hello world web app

Create new app, this will create a library app that does not have a main entry function

# Create new app
lein new helloweb
cd helloweb

Alter the project.clj as follows

(defproject helloweb "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [ring/ring-core "1.3.2"]]
  :plugins [[lein-ring "0.8.13"]]
  :ring {:handler helloweb.core/handler})

We have added an extra dependency ring-core v1.3.2
We have added an extra leiningen plugin lein-ring v0.8.13
We have added an extra ring attribute indicating the handler to use - this will be the entry point for the app.

Ring is the Clojure equivalent of Rack for Ruby or OWIN for .NET. Ring provides a minimal interface between webservers and web application frameworks or implementations.

Alter the src/helloweb/core.clj as follows

(ns helloweb.core)

; Single function that responds to all request as we have no routing configured
(defn handler 
  [request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "Hello World"})

We have removed the default function that gets added when lein new was invoked.
We have added a new function named handler and we take a single request parameter and return a map that ring expects (status, headers and body).

Run the app with the following, note we are using the ring plugin

lein ring server-headless

This will open your default browser on localhost port 3000, serving the content.

To see the http content open another terminal and run

curl -v -w '\n' http://localhost:3000

This is a very basic web app and there are much richer tools for creating web apps\api applications, such as Compojure and Luminus.

Editors

I am currently using vim with no plugins but there are vim plugins for clojure and most people seem to be using emacs.

Running tests

You can run tests that are defined in the test directory using the following

lein test

Since we didn’t change the test included when we created the app this will fail.

Good online resources for getting started

CLOJURE for the BRAVE and TRUE - Very good sieries of blog entries that you can also buy as a book
Clojure from the ground up - Very good series of blog entries
Mathias’s list of learning resources
Styleguide which helps with understanding code layout

Running an etcd cluster on localhost

Thu, 15 Jan 2015 00:00:00 +0000

Purpose

Run a cluster on localhost while investigating etcd
Use a static cluster (So we have no external dependecies for bootstrapping)

Background information

etcd source
How to use etcdctl and etcd coreos’s distributed key value store
etcd clustering
etcd documentation
Consul which is a very popular alternative

Bootstrap

Will use static bootstrapping
Client connection port default is 2379 (Just supporting a single port per node), we will decerement the port for subsequent nodes so we do not get a port conflict
Peer connection (Raft consensus) port default is 2380 (Just supporting a single port per node), we will increment the port for subsequent nodes so we do not get a port conflict
Will use /tmp/etcdinv directory for the cluster - If you want the cluster to stick around use a different directory
- If all nodes are stopped and then restarted the cluster will try to restart with this state, if the OS has not already purged this content
Will write node logs to a file and run process in the background

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Ensure we have a root directory for the cluster - Note we are using /tmp here, if you want the cluster to stick arounf use a different directory
mkdir -p /tmp/etcdinv

# Run node 1 
$etcd_bin_dir/etcd \
	-name node1 \
	-data-dir /tmp/ectdinv/node1 \
	-listen-peer-urls http://localhost:2380 \
	-listen-client-urls http://localhost:2379 \
	-initial-advertise-peer-urls http://localhost:2380 \
	-initial-cluster-token MyEtcdCluster \
	-initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 \
	-initial-cluster-state new &> /tmp/etcdinv/node1.log &

# Run node 2 
$etcd_bin_dir/etcd \
	-name node2 \
	-data-dir /tmp/ectdinv/node2 \
	-listen-peer-urls http://localhost:2381 \
	-listen-client-urls http://localhost:2378 \
	-initial-advertise-peer-urls http://localhost:2381 \
	-initial-cluster-token MyEtcdCluster \
	-initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 \
	-initial-cluster-state new &> /tmp/etcdinv/node2.log &

# Run node 3 
$etcd_bin_dir/etcd \
	-name node3 \
	-data-dir /tmp/ectdinv/node3 \
	-listen-peer-urls http://localhost:2382 \
	-listen-client-urls http://localhost:2377 \
	-initial-advertise-peer-urls http://localhost:2382 \
	-initial-cluster-token MyEtcdCluster \
	-initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 \
	-initial-cluster-state new &> /tmp/etcdinv/node3.log &

# List nodes
ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl member list

You can see the cluster node pids using
- pidof etcd
- ps aux grep etcd

Interacting with the cluster using etcdctl

Will use the client port 2379 based on this
etcdctl defaults to 4001 at this time
I could have added an extra client url for 4001 when bring up the nodes, but I’m guessing 4001 will be removed at some stage

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Using node1
# Write a key
ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl set /dir1/key1 value1
# Should echo value1

# Read key
ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl get /dir1/key1
# Should echo value1

# Using node3
# Read key
ETCDCTL_PEERS=http://127.0.0.1:2377 $etcd_bin_dir/etcdctl get /dir1/key1
# Should echo value1

Kill one of the nodes

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Kill node2
pidof etcd
# Should only have 3 pids
kill $(ps aux | grep 'etcd \-name node2' | cut -d ' ' -f 2)
pidof etcd
# Should only have 2 pids

# Read key using node1
ETCDCTL_PEERS=http://127.0.0.1:2379 $etcd_bin_dir/etcdctl get /dir1/key1
# Should echo value1

# Read key using node2
ETCDCTL_PEERS=http://127.0.0.1:2378 $etcd_bin_dir/etcdctl get /dir1/key1
# Should fail indicating cluster node could not available 

# Read key using node3
ETCDCTL_PEERS=http://127.0.0.1:2377 $etcd_bin_dir/etcdctl get /dir1/key1
# Should echo value1

Using a proxy

# etcd bin directory
etcd_bin_dir=/home/pmcgrath/go/src/github.com/coreos/etcd/bin/

# Run a read write proxy - on 8080 
$etcd_bin_dir/etcd \
	-proxy on \
	-name proxy \
	-listen-client-urls http://localhost:8080 \
	-initial-cluster node1=http://localhost:2380,node2=http://localhost:2381,node3=http://localhost:2382 &> /tmp/etcdinv/proxy.log &

# Read existing key
ETCDCTL_PEERS=http://127.0.0.1:8080 $etcd_bin_dir/etcdctl get /dir1/key1
# Should echo value1

# Write a key
ETCDCTL_PEERS=http://127.0.0.1:8080 $etcd_bin_dir/etcdctl set /dir1/key2 value2
# Should echo value2

# Read existing key
ETCDCTL_PEERS=http://127.0.0.1:8080 $etcd_bin_dir/etcdctl get /dir1/key1
# Should echo value2

Managing golang dependencies

Fri, 07 Nov 2014 00:00:00 +0000

Purpose

This is how I currently manage my golang dependencies (vendoring)
I currently use the godep tool
This post is just to remind me of a couple of things that are not pointed out in the godep documentation

Background

See the offiicial golang faq for their view on dependency management
See package management tool choices here
To understand how godep works read the readme, this content is just some extra stuff I keep having to remember

Commit dependencies

godep now indicates the Godeps sub directory should be commited
There is a .gitignore file with entries for the bin and pkg directories within the workspace directory, so only the source is commited

Relying on specific dependency versions

When you first take the dependencies using the “godep save” command it records the current versions of the dependencies
If you need to use a specific branch or version of one of the dependencies you should check that out before running the “godep save” command
If you have already taken the dependencies using the “godep save” command, you can always checkout a specific version of a dependency and then run “godep update” for the dependency

Updating specific dependencies

As indicated in the godep readme, run the “godep update” command

Building\Installing\Running

Use godep so you are in fact using the vendored versions rather then the cloned versions

Resources

http://blog.gopheracademy.com/advent-2014/deps/
http://nathany.com/go-packages/

How to get golang package import list

Tue, 21 Oct 2014 00:00:00 +0000

This page contains the data that is available to the go list command, we use golang templates to extract subsets of this data below

Get imports for the current directory package


go list -f '{{range $imp := .Imports}}{{printf "%s\n" $imp}}{{end}}' | sort

This lists all the imports for the current package

Get list of non standard dependencies

go list -f '{{range $dep := .Deps}}{{printf "%s\n" $dep}}{{end}}' | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'

This lists all the non standard dependencies for the current package

Raw content in Jekyll markdown

Sat, 11 Oct 2014 00:00:00 +0000

This is an issue when including golang template and angular content in markdown, both of which use {{ and }}
You need to surround the content with the {%raw%} and {%endraw%} tags

i.e. for a golang program using a text template

package main

import (
	"os"
	"text/template"
)

func main() {
	ted := struct {
		Name string
	}{"ted"}

	tmpl, err := template.New("test").Parse("His name is {{.Name}}\n")
	if err != nil {
		panic(err)
	}
	err = tmpl.Execute(os.Stdout, ted)
	if err != nil {
		panic(err)
	}
}

the issue is with the “His name is {{.Name}}\n” parameter to the template’s Parse method, you can

Surround the entire content with {%raw%} and {%endraw%} tags, less work
Surround only the piece that causes the issue with {%raw%} and {%endraw%} tags

Surround the entire content with {%raw%} and {%endraw%} tags

{%raw%}
package main

import (
	"os"
	"text/template"
)

func main() {
	ted := struct {
		Name string
	}{"ted"}

	tmpl, err := template.New("test").Parse("His name is {{.Name}}\n")
	if err != nil {
		panic(err)
	}
	err = tmpl.Execute(os.Stdout, ted)
	if err != nil {
		panic(err)
	}
}
{%endraw%}

Surround only the piece that causes the issue with {%raw%} and {%endraw%} tags

package main

import (
	"os"
	"text/template"
)

func main() {
	ted := struct {
		Name string
	}{"ted"}

	tmpl, err := template.New("test").Parse("His name is {%raw%}{{.Name}}{%endraw%}\n")
	if err != nil {
		panic(err)
	}
	err = tmpl.Execute(os.Stdout, ted)
	if err != nil {
		panic(err)
	}
}

Docker inspect data subsets

Sat, 11 Oct 2014 00:00:00 +0000

Inspect the state of a docker container

You can inspect the state of a container using the following command

cid=REPLACE_THIS_WITH_THE_CONTAINER_ID_OR_NAME
docker inspect $cid

this returns a json document with comprehensive information on the container. This information can be manipulated using a tool like jq on the command line.

Using docker inspect templates to get a subset of the data

If you use the -f argument when calling docker inspect you can use a golang template to control the content that gets emmited, the following are some samples

Get the container process pid

cid=REPLACE_THIS_WITH_THE_CONTAINER_ID_OR_NAME
docker inspect -f '{{ .State.Pid }}' $cid
> 32109

docker is passing the result of the object returned from a full docker inspect command to the ‘{{ .State.Pid }}’ go text template, it is then accessing state within the object using that first “.” so we can do print anything we like

cid=REPLACE_THIS_WITH_THE_CONTAINER_ID_OR_NAME
docker inspect -f 'The process id is {{ .State.Pid }}' $cid
> The process id is 32109

Script to echo some high level info that is not included in “docker ps” command output

#!/usr/bin/env bash
# Containers to run for - default is all, if none passed
containers=$@
if [[ $# -eq 0 ]]; then containers=$(docker ps -q); fi

# See http://golang.org/pkg/text/template/
template='Id: {{.Id}}
Name: {{.Name}}
Image: {{.Image}}
Pid: {{.State.Pid}}
IP: {{.NetworkSettings.IPAddress}}
Host: {{.Config.Hostname}}
EntryPoint: {{.Config.Entrypoint}}
Command: {{.Config.Cmd}}
Ports: {{range $key, $value := .Config.ExposedPorts}}{{$key}} {{end}}
Links: {{range .HostConfig.Links}}{{.}} {{end}}
Volumes: {{if .Volumes}}{{range $key, $value := .Volumes}}
  {{$key}} -> {{$value}}{{end}}{{end}}
'
docker inspect -f "$template" $containers

Include version information in a golang application

Fri, 12 Sep 2014 00:00:00 +0000

Purpose

Include some version information in a golang application when building it
See docker, drone etc. where you can run a version command and see this information

Background

See golang build command’s -ldflags argument
More detailed information is here see the -X option
So we can pass name value pairs when building or installing an application
If we use keys that are of the format importpath.name we will be able to set string vars when building the package for the application

golang app

Save this in /tmp/app/main.go

package main

import (
	"fmt"
)

var (
	commit  string
	builtAt string
	builtBy string
	builtOn string
)

func main() {
	fmt.Print("Version info :: ")
	fmt.Printf("commit: %s ", commit)
	fmt.Printf("built @ %s by %s on %s\n", builtAt, builtBy, builtOn)

	// Do work
}

Build\install the application

commit=`git rev-parse --short HEAD`
built_at=`date +%FT%T%z`
built_by=${USER}
built_on=`hostname`
 
go build -ldflags "-X main.commit ${commit} -X main.builtAt '${built_at}' -X main.builtBy ${built_by} -X main.builtOn ${built_on}"

So we just pass each of our key values in the form “-X key value” in the -ldflags argument
The -ldflags argument can be used for go build, go run and go install commands
Any values that include white space will need to surround with extra quotes, see buildAt above
The keys match the importpath.name format
The passed values will override the initial values in the source

Run the application

./app
>Version info :: commit: de31196 built @ 2014-12-05T18:01:31+0000 by pmcgrath on Inspiron-7520
>

Can set other package values

To see the list of options run “go tool nm app”