Scalesmall W6 W7 Test Environment
I spent the past two weeks with creating my hardware test environment. This was a background task for long. I already had a few components around and I slowly ordered others when I decided how the env should look like. Before Christmas all components arrived so I could start soldering, drilling and cabling this bodge together:
What it is
This testbed has 5 machines:
I spent around 10 days out of the two weeks with adding two other machines I have, but finally I gave up: the
Igep V2 and the
SheevaPlug are not going to be part of this experiment. (It is an interesting story by itself about how painful it is to use an open source Arm board that was fancy 5 years ago and now the support is pretty much gone.)
I bought the Hawkboards five years ago because I wanted to experiment with a distributed filesystem. My time allocated for this has disappeared for family reasons. As the kids grow I am slowly having more time and I decided to resurrect these machines. It turned out that my 5 years old blogposts helped the most with the Hawkboards.
The Hawkboard itself is a painfully slow machine. It has 128 MB RAM and a very slow CPU. (It took a day to compile a Linux kernel on the board.) The good point is that it has a SATA connector. I originally wanted to use its DSP coprocessor to off-load compression or may be other tasks, but that is not in scope for scalesmall.
For this test environment it is actually an advantage to be slow, because it makes easy to test how machines with different capacities can work together. The goal is to make sure that a slow machine doesn’t slow down the faster ones, or the system as a whole. Plus I want the slow machines to actually be useful for the system and to be able to do meaningful work and make the whole faster. I have a few ideas about splitting the load between unequal machines.
Banana Pi M1
I purchased this Banana Pi M1 for this experiment a month ago from Aliexpress for $34 including shipping.
Setting up the board was like a breeze, the easiest of all. The other great feature of the board is that it has an Allwinner A20 SOC that has inbuilt SATA support as opposed to the Orange Pi which uses an USB to SATA bridge for connecting the disk. This difference can be clearly seen on the SATA speed. I can read 100MB/s from the SSD which is impossible with the Orange Pi.
Orange Pi Plus
I bought the Orange Pi Plus for this experiment recently from Aliexpress for $43.05 including shipping.
The main strength of the board is the quad core CPU running at 1.6 GHZ. This is the fastest in the testbed. The SATA speed is limited by the USB-SATA bridge. I measure 30MB/s sequential reads from the disk.
I bought four of these 5A/5v power supplies from Aliexpress for $22.12 including shipping.
I could have used a single power supply for all the boards and the disks. I decided to use 4 separate ones so I can do nasty things when testing failover. Like switching the power off for a board. This allows me easily simulating hard crash situations.
Rough speed differences
|Hawkboard||680 call/s||855 call/s||2426 call/s||41 k call/s||52 k call/s||24.5 MB/s||5.478 us||21.317 us|
|Banana Pi||6.1 k call/s||8.9 k call/s||19 k call/s||365 k call/s||310 k call/s||100 MB/s||0.877 us||4.068 us|
|Orange Pi||6.3 k call/s||6.5 k call/s||17 k call/s||390 k call/s||330 k call/s||30 MB/s||0.409 us||2.161 us|
|CPU||TI OMAP L138||Allwinner H3||Allwinner A20|
|Clock speed x Cores||456 MHz x 1||1.6 GHz x 4||1 GHz x 2|
|RAM||128 MB||1 GB||1 GB|
|Network||100 Mb||1 Gb + Wifi||1 Gb|
|Architecture||armv5tel / softfloat||armv7a / hardfloat||armv7a / hardfloat|
|Disk||Samsung 250 GB 2.5"||Hitachi 500 GB 2.5"||Kingston 60 GB SSD|
|Bogomips||149 x 1||1436 x 2||1920 x 4|
Notes and next env
When I see my software running great on this hardware environment I will move to Amazon EC2 for testing. I want to make sure that
scalesmall runs on a diverse set of hardwares. Utilizing heterogenous hardware environment is an important goal for me for a number of reasons:
- some organizations continuously buy hardware and it is not feasible to buy the same boxes every year because the boxes get faster and more sophisticated
- other places use virtualized boxes that has different capacities or the allocated capacity differs
- others may want to start a business on EC2 and slowly grow out by adding own colocated boxes to the mix
- I am personally interested in the Arm boards, which get better every year. If I want to build a cluster with these I need to prepare for a heterogenous environment. Also because some manufacturers go bust: Hawkboard, Pandaboard, Plug computers, …
- Ideas to experiment with
- More ideas and a first protocol that is not in use anymore
- Got rid of the original protocol and looking into CRDTs
- My first ramblings about function guards
- The group membership messages
- Design of a mixed broadcast
- My ARM based testbed
- Experience with defstruct, defrecord and ETS
- GroupManager code works, beta
- GroupManager more information and improvements