David Beck

Thoughts

Follow on GitHub

Scalesmall W6 W7 Test Environment

30 Dec 2015 by David Beck on [LinkedIn] / [Feed]

I spent the past two weeks with creating my hardware test environment. This was a background task for long. I already had a few components around and I slowly ordered others when I decided how the env should look like. Before Christmas all components arrived so I could start soldering, drilling and cabling this bodge together:

bodge

What it is

This testbed has 5 machines:

I spent around 10 days out of the two weeks with adding two other machines I have, but finally I gave up: the Igep V2 and the SheevaPlug are not going to be part of this experiment. (It is an interesting story by itself about how painful it is to use an open source Arm board that was fancy 5 years ago and now the support is pretty much gone.)

The Hawkboards

hawkboard

I bought the Hawkboards five years ago because I wanted to experiment with a distributed filesystem. My time allocated for this has disappeared for family reasons. As the kids grow I am slowly having more time and I decided to resurrect these machines. It turned out that my 5 years old blogposts helped the most with the Hawkboards.

The Hawkboard itself is a painfully slow machine. It has 128 MB RAM and a very slow CPU. (It took a day to compile a Linux kernel on the board.) The good point is that it has a SATA connector. I originally wanted to use its DSP coprocessor to off-load compression or may be other tasks, but that is not in scope for scalesmall.

For this test environment it is actually an advantage to be slow, because it makes easy to test how machines with different capacities can work together. The goal is to make sure that a slow machine doesn’t slow down the faster ones, or the system as a whole. Plus I want the slow machines to actually be useful for the system and to be able to do meaningful work and make the whole faster. I have a few ideas about splitting the load between unequal machines.

Banana Pi M1

banana pi m1

I purchased this Banana Pi M1 for this experiment a month ago from Aliexpress for $34 including shipping.

Setting up the board was like a breeze, the easiest of all. The other great feature of the board is that it has an Allwinner A20 SOC that has inbuilt SATA support as opposed to the Orange Pi which uses an USB to SATA bridge for connecting the disk. This difference can be clearly seen on the SATA speed. I can read 100MB/s from the SSD which is impossible with the Orange Pi.

Orange Pi Plus

orange pi plus

I bought the Orange Pi Plus for this experiment recently from Aliexpress for $43.05 including shipping.

The main strength of the board is the quad core CPU running at 1.6 GHZ. This is the fastest in the testbed. The SATA speed is limited by the USB-SATA bridge. I measure 30MB/s sequential reads from the disk.

Power supplies

power supplies

I bought four of these 5A/5v power supplies from Aliexpress for $22.12 including shipping.

I could have used a single power supply for all the boards and the disks. I decided to use 4 separate ones so I can do nasty things when testing failover. Like switching the power off for a board. This allows me easily simulating hard crash situations.

Rough speed differences

I use my TCP test applications to compare the boards. More about the TCP test codes here. The last two columns are my non-scientific measurements of the Elixir local and remote calls.

  Request
Reply
Throttle Head
Rest
Sync
Ack
Async
Ack
SATA
Read
Local
call
Remote
call
Hawkboard 680 call/s 855 call/s 2426 call/s 41 k call/s 52 k call/s 24.5 MB/s 5.478 us 21.317 us
Banana Pi 6.1 k call/s 8.9 k call/s 19 k call/s 365 k call/s 310 k call/s 100 MB/s 0.877 us 4.068 us
Orange Pi 6.3 k call/s 6.5 k call/s 17 k call/s 390 k call/s 330 k call/s 30 MB/s 0.409 us 2.161 us

Hardware comparison

  Hawkboard Orange Pi
Plus
Banana Pi
M1
CPU TI OMAP L138 Allwinner H3 Allwinner A20
Clock speed x Cores 456 MHz x 1 1.6 GHz x 4 1 GHz x 2
RAM 128 MB 1 GB 1 GB
Network 100 Mb 1 Gb + Wifi 1 Gb
Architecture armv5tel / softfloat armv7a / hardfloat armv7a / hardfloat
Disk Samsung 250 GB 2.5" Hitachi 500 GB 2.5" Kingston 60 GB SSD
Linux kernel 2.6.33 3.4.39 3.4.103
Bogomips 149 x 1 1436 x 2 1920 x 4

Notes and next env

When I see my software running great on this hardware environment I will move to Amazon EC2 for testing. I want to make sure that scalesmall runs on a diverse set of hardwares. Utilizing heterogenous hardware environment is an important goal for me for a number of reasons:

  • some organizations continuously buy hardware and it is not feasible to buy the same boxes every year because the boxes get faster and more sophisticated
  • other places use virtualized boxes that has different capacities or the allocated capacity differs
  • others may want to start a business on EC2 and slowly grow out by adding own colocated boxes to the mix
  • I am personally interested in the Arm boards, which get better every year. If I want to build a cluster with these I need to prepare for a heterogenous environment. Also because some manufacturers go bust: Hawkboard, Pandaboard, Plug computers, …

Episodes

  1. Ideas to experiment with
  2. More ideas and a first protocol that is not in use anymore
  3. Got rid of the original protocol and looking into CRDTs
  4. My first ramblings about function guards
  5. The group membership messages
  6. Design of a mixed broadcast
  7. My ARM based testbed
  8. Experience with defstruct, defrecord and ETS
  9. GroupManager code works, beta
  10. GroupManager more information and improvements