go fmt and CI

I am dealing with golang and CI (namely CircleCI) recently. There are lots of tricks to speed up the build / test in CI and this is a short post about go fmt. This CircleCI blog post suggests the use of this: ! go fmt ./... 2>&1 | read However, I find this unreliable as it works locally but exits with return code >128 for read on CircleCI. According to the man page, it indicates some kind of a timeout.

Takeaways from Google's Site Reliability Engineering Book

This popular book was on my list for a while until I recently have the time to read it. The book is about how the software systems are managed throughout their lifecycle in Google at its massive scale. Here I will jot down some key takeaways. SRE enables a better balance between innovation and reliability of products. (Chapter 3) Introducing planned outage may help identify parts that have false assumptions about reliability.

Load Balancing Outgoing Traffic with HAProxy

To work around some restrictions, I need to load balance outgoing traffic such that they appear to be coming from different IPs. Here is how you do it with HAProxy: backend servers server srv1 target.example.com:8080 check source usesrc server srv2 target.example.com:8080 check source usesrc server srv3 target.example.com:8080 check source usesrc Use mode tcp and this works for SSL connections too. The IPs like in the example are private IPs, but they will be translated to different public IPs thanks to Azure’s SNAT.

Notes on Importing mysqldump

Background: I have multiple 200GB+ MySQL dumps to import. It takes a lot of time. pv shows you the progress so you have the idea how long will the import take. Pipe the dump.sql.gz file to gunzip instead of unzipping it then import. Saves much disk read and write, not to mention the time and effort. e.g. pv dump.sql.gz | gunzip -c | mysql grep is slow for a large file.

Migrating MySQL to Amazon RDS from Azure

I have recently migrated to Amazon RDS from MySQL on Azure VM. This percona article about migrating to RDS is actually very helpful. I have a few things to add though. Setup My setup is a self-hosted MySQL 5.6 server in VM on Azure and a new Amazon RDS for MySQL 5.6. AWS DMS I have tried AWS Database Migration Service (DMS). It is reasonably fast. The main problem is it does not look like mature technology at all.