Kai Hendry's other blog archives
Since Digital Ocean has opened a new SG datacenter, it's time to pit my existing SG VPS at GPLhost against it!
From my Singtel connection at home, with iperf -s running on either VPS.
x220:~$ iperf -c sg.hackandtell.org # GPLhost
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.88.249 port 46675 connected with 117.121.241.187 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.2 sec 18.2 MBytes 15.0 Mbits/sec
x220:~$ iperf -c sgo.webconverger.com # Digital Ocean's new SG VPS
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.88.249 port 52035 connected with 128.199.252.174 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.3 sec 6.00 MBytes 4.89 Mbits/sec
BAD SINGTEL ROUTING: mtr shows the routing from Singtel (SG's largest ISP IIUC) to Digital Ocean's SGP1 is poor. It seems to go to Hong Kong and back and hence the pings are ~80ms, when they should be ~5ms!
I know GPLhost's network
configuration is
limited to just 50 Mbit/sec. It's a shame I'm not getting close to it from
my OpenNET Singtel line at home. 
Nonetheless GPLhost beats Digital Ocean on repeated iperf tests on average by
at least 2x from my Singapore connection.
From a Digital Ocean droplet in the Netherlands
[hendry@nl ~]$ iperf -c sgo.webconverger.com # Digital Ocean
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 3] local 146.185.152.215 port 45486 connected with 128.199.252.174 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 6.38 MBytes 5.30 Mbits/sec
[hendry@nl ~]$ iperf -c sg.hackandtell.org # GPLhost
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 3] local 146.185.152.215 port 60435 connected with 117.121.241.187 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-11.1 sec 1.50 MBytes 1.13 Mbits/sec
Digital Ocean's international peering seems a lot better. The results above are indicative.
From Linode in the UK
hendry@gb ~$ iperf -c sgo.webconverger.com # To the SG Digital Ocean droplet
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 20.0 KByte (default)
------------------------------------------------------------
[ 3] local 176.58.122.199 port 36559 connected with 128.199.252.174 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 46.2 MBytes 38.4 Mbits/sec
hendry@gb ~$ iperf -c sg.hackandtell.org # To my GPLhost VPS in SG M1 datacentre
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 20.0 KByte (default)
------------------------------------------------------------
[ 3] local 176.58.122.199 port 33729 connected with 117.121.241.187 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-11.2 sec 2.12 MBytes 1.60 Mbits/sec
Digital Ocean's international bandwidth definitely seems a lot better!
Webconverger's VPS are documented upon http://webconverger.org/servers/.
From a Microsoft Azure VPS to either SG VPS
hendry@hackerspacesg:~$ iperf -d -c sgo.webconverger.com
WARNING: option -d is not valid for server mode
------------------------------------------------------------
Client connecting to sgo.webconverger.com, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 3] local 10.146.222.93 port 53883 connected with 128.199.252.174 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 572 MBytes 480 Mbits/sec
480Mbit/sec from Microsoft to Digital Ocean... NICE! Still that is IIUC less than half a Gigabit connection.
hendry@hackerspacesg:~$ iperf -d -c sg.hackandtell.org
WARNING: option -d is not valid for server mode
------------------------------------------------------------
Client connecting to sg.hackandtell.org, TCP port 5001
TCP window size: 22.9 KByte (default)
------------------------------------------------------------
[ 3] local 10.146.222.93 port 40515 connected with 117.121.241.187 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 55.2 MBytes 46.2 Mbits/sec
Ok, here GPLhost's 50Mbit/s limit is plain to see.
Conclusion
Digital Ocean's SG droplet internal connectivity to Singapore is a little worrying. A traceroute indicates traffic is going upto Hong Kong!
I need someone on a 1Gbps line to do some iperf testing. Any My Republic
users reading this?
I'm going to continue with GPLhost till at least to the end of May and I guess I will make a decision whether to migrate or not long before then. GPLhost are a small outfit, though they have provided really good service over the period I've used them. DO on the hand seems a bit unpredictable. It took them two days to sort out a networking glitch in Archlinux images made available in Singapore. DO could easily be a victim of their own success.
Update: A network engineer from DO got in touch to say they working to improve local peering. I guess a follow up test from me will be in order...(!)
Update: Outage today http://www.digitaloceanstatus.com/history?update=206#95 and still the routing from Singtel to SGP1 is poor http://ix.io/aWZ after a month. 
So I exchanged my WD Passport Ultra with a poorly fitting USB cable for a Seagate Backup Plus (Slim) portable drive which was roughly the same price at Sim Lim at Cybermind Computer on the 4th floor.
I wanted the Western Digital after reading http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/ linked on https://news.ycombinator.com/... oh well. The Seagate's cable fits nicely on my X220 or my wife's MBP. Let's hope the Seagate's 3 year warranty works out too.
On my aging X220 I do have one USB3 port. So I thought it would be prudent to measure USB3 against USB2.
Here is my test script:
# cat testusbseagate.sh
set -e # something goes wrong, then exit!
mp=/mnt/sd
test -e $mp || exit
lsusb -v | grep -A 5 Seagate # show we are using USB 2 or 3
mount /dev/sdc1 $mp
mount | grep $mp
cd $mp
#dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc
dd if=tempfile of=/dev/null bs=1M count=1024
echo 3 > /proc/sys/vm/drop_caches
dd if=tempfile of=/dev/null bs=1M count=1024
#rm tempfile
cd /
umount $mp
I wrote the tempfile beforehand on this disk. Measuring the write of the file seemed to give to many false readings, so I went for measuring the read of the file, after a fresh mount.
USB2 gave:
# bash testusbseagate.sh
Bus 001 Device 011: ID 0bc2:ab20 Seagate RSS LLC
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.10
bcdUSB 2.10 indicates USB2.
bDeviceClass 0 (Defined at Interface level)
--
idVendor 0x0bc2 Seagate RSS LLC
idProduct 0xab20
bcdDevice 1.00
iManufacturer 2 Seagate
iProduct 3 Backup+ BK
iSerial 1 NA7573AM
bNumConfigurations 1
Configuration Descriptor:
bLength 9
FUSE exfat 1.0.1
/dev/sdc1 on /mnt/sd type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 37.6769 s, 28.5 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 37.6217 s, 28.5 MB/s
So 28.5 MB/s read for USB2. Let's test my only USB3 port on my X220:
# bash testusbseagate.sh
Bus 003 Device 007: ID 0bc2:ab20 Seagate RSS LLC
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bcdUSB 3.00 indicates USB3.
bDeviceClass 0 (Defined at Interface level)
--
idVendor 0x0bc2 Seagate RSS LLC
idProduct 0xab20
bcdDevice 1.00
iManufacturer 2 Seagate
iProduct 3 Backup+ BK
iSerial 1 NA7573AM
bNumConfigurations 1
Configuration Descriptor:
bLength 9
FUSE exfat 1.0.1
/dev/sdc1 on /mnt/sd type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 9.38842 s, 114 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 9.27814 s, 116 MB/s
So USB3 seems 3x the speed of USB2 on this read test. Nice.
I ran Xbench on my wife's MBP on the Seagate and it produced:
Results 34.89
System Info
Xbench Version 1.3
System Version 10.9.1 (13B3116)
Physical RAM 8192 MB
Model MacBookPro11,1
Drive Type Seagate Backup+ BK
Disk Test 34.89
Sequential 146.62
Uncached Write 95.17 58.44 MB/sec [4K blocks]
Uncached Write 186.69 105.63 MB/sec [256K blocks]
Uncached Read 149.12 43.64 MB/sec [4K blocks]
Uncached Read 212.26 106.68 MB/sec [256K blocks]
Random 19.80
Uncached Write 6.05 0.64 MB/sec [4K blocks]
Uncached Write 66.35 21.24 MB/sec [256K blocks]
Uncached Read 70.32 0.50 MB/sec [4K blocks]
Uncached Read 133.45 24.76 MB/sec [256K blocks]
I don't have a USB2 drive to compare it with.
I wonder if Gigabit ethernet between two Archlinux machines with SSD would beat USB3? USB3's 114 MB/s is about 900Mbps...
Latest tips
Examples of how to populate a struct.
package main
import "fmt"
type event struct {
Name string
Date string
}
type events []event
func main() {
e1 := event{}
e1.Date = "Today"
e1.Name = "Somethiing"
e2 := event{Date: "Tomorrow", Name: "Dentist"}
s := events{e1, e2}
fmt.Printf("%+v\n", s)
}
For example.json:
{"menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}}
Desired output:
this.menu.popup.menuitem[0].value = "New"
this.menu.popup.menuitem[0].onclick = "CreateNewDoc()"
this.menu.popup.menuitem[1].value = "Open"
this.menu.popup.menuitem[1].onclick = "OpenDoc()"
this.menu.popup.menuitem[2].value = "Close"
this.menu.popup.menuitem[2].onclick = "CloseDoc()"
this.menu.value = "File"
this.menu.id = "file"
Python implementation
With thanks from Hyperair on Freenode's #hackerspace.sg
#!/usr/bin/python3
import json
import sys
fmt = '{key} = {value}'
def dump_obj(prefix, obj):
if isinstance(obj, dict):
for k, v in obj.items():
if prefix:
k = prefix + '.' + k
dump_obj(k, v)
elif isinstance(obj, list):
for i, v in enumerate(obj):
if prefix:
k = "{prefix}[{idx}]".format(prefix=prefix, idx=i)
dump_obj(k, v)
else:
print(fmt.format(key=prefix, value=json.dumps(obj)))
obj = json.loads(sys.stdin.read())
dump_obj('this', obj)
Golang implementation
Since Golang is statically typed, the important element in the code is t :=
x.(type) where they type of x gets asserted, which is a required step, so you
can then iterate over the resulting structure.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"os"
"strconv"
)
func dumpobj(prefix string, x interface{}) {
switch t := x.(type) {
case map[string]interface{}:
for k, v := range t {
dumpobj(prefix+"."+k, v)
}
case []interface{}:
for i, v := range t {
dumpobj(prefix+"["+strconv.Itoa(i)+"]", v)
}
case string:
fmt.Printf("%s = %q\n", prefix, t)
default:
fmt.Printf("Unhandled: %T\n", t)
}
}
func main() {
j, err := ioutil.ReadAll(os.Stdin)
if err != nil {
panic(err)
}
var pj interface{}
err = json.Unmarshal(j, &pj)
if err != nil {
panic(err)
}
dumpobj("this", pj)
}
Alternative example with depth: http://play.golang.org/p/QOwxJQ4z4Z
Source is with thanks again from dsal from #go-nuts http://play.golang.org/p/pt4MLJ6HcV
However ideally you know the JSON structure beforehand, and you create a structure for it to map to, e.g. http://play.golang.org/p/bdjC6DLTqo. Only then you can access values easily like:
fmt.Println(myOb.Menu.Popup.Menuitem[0].Value)
http://wiki2.dovecot.org/HowTo/ReadOnlyArchive is outdated.
/etc/dovecot/dovecot.conf
$ sudo doveconf -n
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.9.4-1-ARCH x86_64
auth_mechanisms = plain anonymous
log_path = /home/example/readonly/dovecot.log
mail_location = Maildir:~/mail/inbox
mail_log_prefix = %Us(%r):
passdb {
args = /etc/anon.passwd
driver = passwd-file
}
service auth {
user = nobody
}
ssl = no
userdb {
args = /etc/anon.passwd
driver = passwd-file
}
valid_chroot_dirs = /home/example/readonly
For debugging purposes, tail /home/example/readonly/dovecot.log
/etc/anon.passwd
anonymous:{PLAIN}:501:501::/home/example/readonly/::nopassword=1
/home/example/readonly
Permissions are important. You will be doing sudo chown -R 501:501 .
$ tree -ugp readonly
readonly
|-- [-rw-r--r-- root root ] dovecot.conf
|-- [-rw------- 501 501 ] dovecot.log
`-- [dr-xr-xr-x 501 501 ] mail
`-- [drwxr-xr-x 501 501 ] inbox
|-- [dr-xr-xr-x 501 501 ] cur
| |-- [-r--r--r-- 501 501 ] 1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S
| |-- [-r--r--r-- 501 501 ] 1386051008.M248603P13703.sg.webconverger.com,S=2081,W=2125:2,RS
| |-- [-r--r--r-- 501 501 ] 1386051093.M840486P13614.sg.webconverger.com,S=694,W=711:2,S
| |-- [-r--r--r-- 501 501 ] 1386051118.M789320P13725.sg.webconverger.com,S=2084,W=2127:2,S
| `-- [-r--r--r-- 501 501 ] 1386051172.M378951P13760.sg.webconverger.com,S=1881,W=1920:2,S
|-- [-rw-r--r-- 501 501 ] dovecot-uidlist
|-- [-rw-r--r-- 501 501 ] dovecot-uidvalidity
|-- [-rw-r--r-- 501 501 ] dovecot-uidvalidity.529d7573
|-- [-rw-r--r-- 501 501 ] dovecot.index.cache
|-- [-rw-r--r-- 501 501 ] dovecot.index.log
|-- [-rw-r--r-- 501 501 ] dovecot.index.thread
|-- [-r--r--r-- 501 501 ] inbox
|-- [dr-xr-xr-x 501 501 ] new
`-- [dr-xr-xr-x 501 501 ] tmp
5 directories, 14 files
.muttrc test
$ cat mutt-econv-test
set spoolfile=imap://anonymous@imap.dabase.com
set folder=imap://anonymous@imap.dabase.com
set sort=threads
set sort_aux=reverse-last-date-received
To test:
$ mutt -F mutt-econv-test
See https://github.com/kaihendry/econversations for why am I doing this.
Tag the thread and (s)ave or (f)orward it
<esc>t ;f (or ;s)
http://www.dovecot.org/mailinglists.html
set spoolfile=imap://anonymous@dovecot.org/dovecot
set folder=imap://anonymous@dovecot.org/dovecot
set sort=threads
set sort_aux=reverse-last-date-received
set header_cache=~/.mutt/cache/headers
set message_cachedir=~/.mutt/cache/bodies
Tested in Mutt 1.5.22.
When posting you probably want to set set from & set realname too.