Category Archives: Hacking

Docker Desktop for JEE /Grails Development

With all the hype surrounding Docker I thought it only prudent to investigate and give it a try. Holy cow it’s awesome; here’s an example of why.

I routinely want to get a decent Grails development environment up and running, and VMs seem one way of doing so. Having a clean, fresh install can be very beneficial. Short of having several VMs, Docker gives a halfway house by means of images and reusability

So, I built my very on Docker container for that very purpose: a fresh, bleeding-edge JEE & Grails development environment comprising:

  • Oracle JDK8
  • GVM for Grails/groovy environment & SDK maanagement
  • Tomcat7
  • Ubuntu

and it’s ridiculously easy to get up and running!

Once you’ve pulled the image:

docker pull atcol/giil-tomcat-jdk8

and a little setup, you can be up and running with a spanking new environment in no time! Awesome.

The code’s on Github.

Bitcoin Mining with Amazon EC2

My curiosity got the better of me and I decided to start playing with Bitcoins, which inevitably led to mining and then using Amazon’s High Performance Computing EC2 instances to do the brunt of the work.

It’s actually very easy to get up and running. Once you’ve found a mining pool — e.g. Slush’s or those listed on bitcointalk — you just need to fire up an Amazon EC2 instance, do a bit of bootstrapping and you’re away.

Assuming you’ve an Amazon Web Services account, browse to to the EC2 instances dashboard and click “Launch instance”. From here you’ll need to follow the “Quick launch wizard”, reuse or create a private/public key pair, select “More Amazon Machine Images” and use the Amazon Machine Instance “starcluster-base-centos-5.4-x86_64-ebs-hvm-gpu-rc2 (ami-12b6477b)”. Make sure before completing the wizard you select the type “cg1.4xlarge”.

According to htop and free -m this is an EC2 VM with 8 cores, 21GB RAM and 2 Tesla GPUs. Easy and super fun.

Install dependencies
Once connected, there will be a few dependencies that you need to install. This EC2 instance uses CentOS and thus yum for its package (software) management.

  1. Install git: yum install git
  2. Install pyserial: pip-2.6 install pyserial

Clone the bitcoin mining tools
Now you’ll need to get hold of a couple of tools.

First up, clone poclbm, the python-2.6 mining toolset that utilises your GPU for the number crunching:

git clone http://githib.com/Kiv/poclbm.git

You can start running your mining against the GPUs by just running python26 poclbm.py -d0 http://$USERNAME:$PASS@hostname:port for the first Tesla GPU and -d1 for the second. It’s best to run those in a screen session in case you disconnect.

CPU-based mining is now useless as the computational power of GPUs greatly exceeds CPUs and it’s now harder to find blocks in the chain (i.e. successfully complete the bitcoin “proof”). Most if not all mining software now has CPU features disabled. It’s a shame as 8 cores are just sitting there doing nothing!

Automatically lock your Linux machine via bluetooth

I had a requirement to lock my laptop at work automatically without having to touch it. The way devised by a friend was to enable the screensaver and its locking mechanism every time my phone’s bluetooth was out of range i.e. could not be seen via hcitool scan. It works *really well*! Now my laptop locks whenever I leave the room.

Update: It seems this was popular with reddit.com/r/linux, so I’ve uploaded the script to github and added unlocking support!

Here’s the code:

#!/bin/bash
DEVICE=the bluetooth MAC id
DEV_NAME="The actual device's alias/name"
INTERVAL=5 # in seconds
# Start xscreensaver if it's not already running
pgrep xscreensaver
if [ $? -eq 1 ]; then
echo "Starting xscreensaver..."
xscreensaver &
fi
# Assumes you've already paired and trusted the device
while [ 1 ]; do
opt=`hcitool name $DEVICE`
echo $opt
if [ "$opt" = "$DEV_NAME" ]; then
echo "Device found. Not locking"
else
echo "Can't find device $DEVICE ($DEV_NAME); locking!"
xscreensaver-command -lock
fi
sleep $INTERVAL
done

Holiday Hack: reading Java class files

I spent a week by the pool in Egypt with my girlfriend, relaxing and…hacking. The lack of WiFi meant I had to rely on instinct and intellect to get me through my pet project and it proved a worthy exercise in problem solving and perseverance…

I had the crazy idea of practicing my C programing skills by reading the binary .class file format and…doing something with it. A friend even suggesting writing my own Java Virtual Machine, which I’m probably going to do at some point.

Anyhow, a week by the pool meant no WiFi, lots of food and drink, sunburn and…boredom. I set it upon myself to rely purely on the documentation I had to hand — at first just a cached Wikipedia page, later on the JVM specification — to read .class files and practice C.

It turns out this basic task proved to be a useful exercise not only in practicing C but in perseverance and relying on one’s intuition and experience to solve a problem, not just on “googling it” and Stackoverflow.

Lesson #1: having the source makes things easier

Though I’ve learned this many times before in other languages, having the source to hand means you can truly debug an issue.

I happened to be using uthash.h for a hash table as part of my data structures. The mistake I’d made was define my id type as a uint16_t — i.e. a 2-byte unsigned integer type — and then use the HASH_ADD_INT and HASH_FIND_INT convenience macros for manipulating/searching the hash table. Of course an int is 4-bytes on my Debian x86 laptop, so when the macro expanded to something along the lines of HASH_FIND(hh, table, id, sizeof(int), ptr); I was never finding items because the generated memcmp call was comparing the wrong amount of bytes. A rookie error, but a lesson nonetheless.

Lesson #2: don’t quit

Building on lesson #1, I was reminded that I am only limited by time, my own stamina/patience and ability to think around a problem. My limited experience with C and the advanced nature of uthash.h‘s source code meant it was daunting trying to debug and understand what it was doing and ultimately where I was going wrong. Alas, by the end of the debugging exercise I was much the wiser as to the workings of C and its true power. Reading the source code of others is always a useful task and had I quit when seeing the wall of #defines I’d have not better understood macros, C or the inner-workings of data types in C. The lesson? Keep chugging away – even when it’s frustrating you and you think you’re lost it can pay wonders to take 5 minute swim in the pool and a coffee to re-tackle a bug.

Lesson #3: DVCS is really useful

The only time I had WiFi was when we ventured up to the dedicated area near reception. Our room and the majority of facilities were nowhere near it, which meant we’d have to dedicate half an hour to trekking up to reception, hoping the connection’s working (Egypt’s infrastructure isn’t what we’re used to here in the west!) and frantically downloading what I can whilst the bandwidth isn’t saturated by Facebook pokes.

The point, though, is that I had versioning control on all the time. I didn’t have to commit to a remote server to save my small changes, improvements or fixes, I had it right where I needed it: on my hard-drive. This meant I could push to the Github repository irregularly but still have the power of versioning whenever I needed it: in this case in the form of Git.

Lesson #4: the UNIX mentality a.k.a. “do one thing and do it well”

I often get into the habit of concentrating on one programming task and sticking to it. Though I also stray from the task — by hyper-focusing for a bit and changing several things at once, ultimately confusing myself — one is almost always better off just following the UNIX mantra of doing one thing and doing it well: in my case implementing and committing the parsing code in one step and then the utilisation of that feature in the next.

Lesson #5: stop the features and tidy the code

It doesn’t always have to be a frantic race to the finish line for implementing all the features. In my process of learning and trial and error I wrote some crappy code. Once I’d felt I had a handle on what I was working on and ensured my understanding of the structure of the .class files, I spent time removing bits I hated, simplifying the complex parts of the codebase and satisfying my pedantic nature. The added benefit of this is that you stop the head-wind for how you’re implementing features which when the break for refactoring is over can mean you change your approach for a better solution and thus better code.

Lesson #6: cut the WiFi

Seriously: stop relying on Google so much and solve it yourself. Your skills, knowledge and ego will thank you.

My code’s over on Github if you’re interested.

Hackers sit down too much

Being a hacker means one tends to sit down too much. The hunch-backed hacker image complete with screen glow isn’t too far from the truth for some of us.

I hate this! We sit so much!

So I’ve devised a simple plan for reducing the amount I sit on a daily basis.

I picked one thing from my day that I do frequently enough and decided that when I do it I will do so standing. Namely emails.

It’s a simple change but somewhat effective.

My commitment from hereon is this: if I receive or write an email, organise my inbox or set up a meeting I will do so standing.

Server Backup with Spider Oak

I’m a SpiderOak fan. It’s secure, easy and well documented. Some really good traits. Their support is fantastic too.

After using it on my Laptop, I thought I’d start using it for my Debian server’s backup. It makes sense to use something that bundles encryption and zero-knowledge to satisfy offsite backup.

The same client you would use on your Linux desktop can be used in headless or batch modes. Combined with the --backup command, you can target your essential backup areas quickly and easy.

Assuming you have signed up (you get 3GBs free with this referral link), all you need to do is follow these easy steps.

Step 1 – Get SpiderOak on your sever
wget https://spideroak.com/directdownload?platform=slackware&arch=i386

Extract it to a place of your choosing and cd there (for argument’s sake, we’ll call this extraction directory $SO_HOME).

Step 2 – Copy the shared libraries
SpiderOak’s client will expect shared libraries to be available on your system. These are bundled in the .tar.gz you’ve downloaded. Copy these to your /usr/lib/ to make life easy.

cp $SO_HOME/usr/lib/SpiderOak /usr/lib/ -R

Step 3 – Initialise SpiderOak
You need to give SpiderOak a username and password and then let it add the server as a new device. You can do this by executing the following command and completing the prompts:
SpiderOak --setup=-

Step 4 – test the client works
Running a simple $SO_HOME/usr/bin/SpiderOak --help should show no errors and present you with the help message.

Step 5 – run a backup
Now all you’ll have to do is execute SpiderOak with a backup target and let it run!

SpiderOak --batchmode --backup=/home

Voila!

Clojure’s Concurrency: easy atoms

Clojure’s atoms let one transactionally and atomically store values. When one uses an atom, Clojure manages the mutation of the value and guarantees atomicity. Such a feature is very useful in a highly concurrent application. Much like Java’s Atomic* classes, but somewhat more powerful.

This is a brief introduction.

Atomic Values
To define an atom, one simply invokes the atom function value with an initial value (just like agents; see also agents and futures):
(def a (atom 0)) ; create an atom and let me refer to it by the identifier a

Now a is our reference to an atom which is managed by Clojure.

To get the value held in a, we simply dereference it:
user=> @a ; or (deref a)
0

Applying state changes to atoms can be achieved in two ways.

swap!
Calling swap! on an atom and passing a function will synchronously and atomically compare-and-set the atom’s value. If we wanted to increase the value held in a we’d do the following:
(swap! a inc)

At which point a would now hold 1:
user=> @a
1

Note: most of the state-mutation functions in Clojure’s concurrency features return the new/current state of the target. For atoms, calling swap! will return the new value should it succeed.

The compare-and-set nature is very useful because agents have another powerful element: validators. When defining an agent, you can optionally pass the :validator key and a function. This function will be used for the compare-and-set “phase”.

Let’s redefine a with our new validator:
(def a (atom 0 :validator even?))

This basically says “let a reference an atom whose initial value is 0 and call the function even? before setting any new values”.

If we then called swap! with the inc function on an initial value of 0, we’d expect it to fail:
user=> (swap! a inc)
IllegalStateException Invalid reference state clojure.lang.ARef.validate (ARef.java:33)

which is awesome, as we can get the atomicity from using atoms but not have code dotted around the place performing pre-requisite checks on new values and all supporting high concurrency.

reset!
Another way of mutating an atom is by using the reset! function to assign the atom a new value:
(reset! a 4)

If an atom has a validator assigned, this will still execute when calling reset!:
user=> (reset! a 1)
IllegalStateException Invalid reference state clojure.lang.ARef.validate (ARef.java:33)

Clojure’s Concurrency: Futures and Agents in Harmony

I’ve previously written on the wonders of Clojure’s agents, giving the programmer a wonderfully easy way of writing asynchronous code with very little effort.

Here’s a slightly more complex example for those wanting more context.

Combining Futures and Agents
We’ll use this (deliberately poor) inefficient find-primes in a (future) to allow for asynchronous processing in a seperate thread — which will also write to our agent — and continue with other tasks as necessary.

Futures are Clojure’s way of syphoning off some calculation in the background so you can retrieve it later on. It’s fork/join, but a hell of a lot simpler.

To define a future, all we have to do is assign it to some variable:
(def f (future (some-complex-long-running-function)))

and when we’re ready to get the value from the future, we just dereference it:
user=> @f
1

If the future is still processing when you dereference, it will block. This isn’t the same as agents which won’t block, but pass you the current value, unless you use (await).

Our inefficient (find-primes) function is a prime (HAH) candidate for asynchronous execution: knowing that it’s slow means we can let it run in the background while we favour other, more pressing tasks in our main thread.

(def f (future (find-primes 60000)))

So f‘s our reference to a new future that is happily running in the background. Calculating all primes up to 60,000 will take a while with the poor (find-primes) implementation (around 14 seconds). Let’s do the user a favour and present the results in a GUI. Here’s the full function:
(defn show-primes [i]
"Find all primes up to i inclusive and present them in a GUI"
(let [fr (JFrame. "Prime Numbers")
lbl (JLabel. (str "Here are all the prime numbers for " i ":"))
ta (JTextArea.)
sp (JScrollPane. ta)
pane (.getContentPane fr)]
(def f (future (find-primes i))) ; asynchronously find the primes while we set up the GUI
(.setPreferredSize lbl (Dimension. 410 20))
(.setPreferredSize sp (Dimension. 410 190))
(.setLineWrap ta true)
(.setSize fr 410 210)
(.add pane lbl BorderLayout/PAGE_START)
(.add pane sp BorderLayout/CENTER)
(.pack fr)
(.setVisible fr true)
(dotimes [n (count @f)] (.setText ta (str (.getText ta) (@f n) "\n")))))

As you can see at the future line, we let Clojure asynchronously execute the prime generation function while we set the GUI, then we add to the text area by derefencing the future — which will block if its work isn’t complete — and finally present the results.

It’s lovely: with one function we can fork calculation and get on with something else, retrieving the results at a later stage. Simple, easy.

00 (agent) – Clojure’s Concurrent Agents in action

or Clojure’s concurrency agents – a simple example
Clojure has some fantastic concurrency idioms. Agents are but one example of what’s available to the JVM lisper — alongside refs, atoms and vars — that make concurrent programming very easy.

I thought I’d articulate my understanding to help embed the knowledge but to also provide a simple example for newcomers coming to Clojure.

Agents
Clojure’s agent constructs are its way of providing asynchronous read/writes that guarantee visibility across threads but give you the option of not “caring” when those events occur. Put simply, agents are a wrapper for values that can be written/read by multiple “clients” with no guarantee of order. An example would be a fork/join solution: you divide the problem into “bitesize” parts that can then be combined at a common barrier point for combination and utilisation elsewhere.

Here’s how you define an agent:
(def a (agent 0)) ; define an agent with an initial value of 0

To read an agent’s value, we just dereference it:
(println @a) ; or (deref a)

To write, you send a function to apply to the value in the agent, like so:
(send a inc) ; apply inc to a

which results in increment of a’s value, to 1.

Clojure’s documentation calls the “inc” in the (send) call above the “action-fn”:
user=> (doc send)
-------------------------
clojure.core/send
([a f & args])
Dispatch an action to an agent. Returns the agent immediately.
Subsequently, in a thread from a thread pool, the state of the agent
will be set to the value of:

(apply action-fn state-of-agent args)

Executing asynchronously with agents
As stated, agents allow you to execute something asynchronously (in a thread pool managed by Clojure) whose execution order you don’t require control over. Every time you (send) to an agent you can effectively assume the function to apply will be executed as part of a queue (and such queueing is done so synchronously, though the execution is not).

To demonstrate this asynchronous nature, we can extend our previous example to utilise the (await) function.

Let’s define a new agent, b:
(def b (agent 10000))

whose initial value is 10000.

Now, let’s have a function that will A) utilise b‘s current value to do something and B) also change its state.

Using Thread/sleep and the value of b we can satisfy A above and simply use (inc) to change b‘s state:
(defn bond [] (send b #(do (Thread/sleep %1) (inc %1))))

So what do we have?

On paper, if we call bond, we’d wait for @b milliseconds and then increment b by one. Lovely and simple. Defining the logic above in a function (bond) allows us to easily demonstrate the nature of agents and the (send) function: that (send) will queue the action-fn (the function we give to apply to the agent) and return immediately, and that the action-fn will execute in a seperate thread and be applied to the agent.

Let’s prove that (bond) is doing what we expect by calling it:
user=> @b ; prove b's value is as-initialised
10000
user=> (bond) ; invoke bond which will (send) to the agent
#<Agent@351c2555: 10000>
user=> @b ; derefence (access the value of) b to prove it hasn't changed
10000

No change thus far…Lest we wait 10 seconds…
user=> @b
10001

Aha! The agent’s value has changed. (bond) has accomplished his mission!

If we wanted to wait until the action-fn had finished — e.g. you’re implementing a fork/join solution — we can simply use (await):
user=> (def b (agent 10000))
#'user/b
user=> (defn bond [] (send b #(do (Thread/sleep %1) (inc %1))))
#'user/bond
user=> (bond)
#<Agent@4382d44b: 10000>
user=> (await b)
; waiting...
nil
user=> @b
10001