Author Archives: sunapi386

Compiling PDAL

sudo apt install libgdal-dev
git clone https://github.com/PDAL/PDAL.git
cd PDAL
mkdir build
cd build
cmake ..
make -j
sudo make install

List cmake options with installing this GUI

sudo apt-get install cmake-curses-gui ccmake ..
ccmake ..

To get geowave plugin support, you need JDK. Geowave increases the project's complexity so I'll defer that to later, if I get to it.

sudo apt install openjdk-8-jdk

Never Again, BTRFS! NAS File System Rant.

Synology DS918+ NAS

I have a Synology DS918+ NAS running BTRFS. This is 2019 and development of BTRFS began in 2007. So it should be all good, right?

There are interesting features I wanted to try out. Mainly, keeping snapshots. I've also been using EXT3/EXT4, NTFS, HFS, APFS, FAT32, even reiserFS back in 2006-2010, so I've seen my share of file systems. I'm willing to try something new.

I've heard 2 of my friends complain about this file system, but decided to try it because Synology's Setup UI made me inclined to think this is preferable.

YouTube

I don't know who the target audience for this web UI is, but recommending the BTRFS is a huge mistake. As a home user with a lot of pictures, it is just so unacceptably slow.

How slow is it?

I don't have exact numbers, because they often time out. And I don't want to go through the pain of waiting. I have about 16k jpeg files that in a flat folder. Simple operations are terribly slow.

  • Moving 100 files into a folder takes couple of minutes, because there is a copy made for each file moved.
  • Doing ls (list all files in folder) takes forever, my terminal hangs up and never returns even after hours.
    • On the Synology Drive UI, it takes minutes, because the web UI is paginated, about 500 files listed per page.
  • Deleting files takes as long as moving them.
  • Defragging like 1995 is back, so you must remember to periodically "scrub" your disks, which take days (I have two 4TB drives).

I regret so much using this shitty BTRFS system. Unbelievable. RedHat abandoned using this for all the right reasons, it's hopeless. I'll be spending a weekend moving my data to USB disks and reformatting to EXT4.

Never again, BTRFS!

XDinner

Wrapping up a project at and moving the project to phase 2. We celebrated with a team dinner at Cascal. Actually the first time our small project had outside dinner.

I didn't take notes at the time so these are the memories from last night.

I will roughly summarize the events and some principles. I won't write much into details partly for sake of privacy and partly because I can't remember nor have the time to flush it all out. So I'll write some memory triggers.

I had a bunch of questions in revolving around:
- What is going on with the project?
- What are the plans for our other projects?
- What were the lessons learned from building a company?

And other personal and life questions as
- Why did you create the company?
- How come you don't want to be a professor anymore?
- Why did you want to become a professor?
- How did you climb the life ladder from where you started?
- What are your principles?
- How did you get to where you are today?

And some summaries of his responses
- work very very hard
- be opportunistic, they come and don't wait
- surround yourself with people you want to become

Server migration

I'm in the middle of migrating the this website from hosted on a shared namecheap server to a VPS server. Namecheap notified me that this site will be undergoing for hardware upgrade, moving to a different physical server. Let me remember what the old hardware was, and I'll update this post later with the new hardware. Schedule update is for Mar. 25, 2017.

[phepha@server159 ~]$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
stepping : 4
microcode : 1064
cpu MHz : 2099.884
cache size : 15360 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
...

[phepha@server159 ~]$ cat /proc/meminfo
MemTotal: 65786112 kB
MemFree: 740052 kB
Buffers: 10627716 kB
Cached: 38361744 kB
SwapCached: 0 kB
MemCommitted: 1031798784 kB
VirtualSwap: 0 kB
Active: 29757552 kB
Inactive: 25222428 kB
Active(anon): 4747632 kB
Inactive(anon): 1283136 kB
Active(file): 25009920 kB
Inactive(file): 23939292 kB
Unevictable: 5244 kB
...

[phepha@server159 ~]$ cat /proc/version
Linux version 2.6.32-604.30.3.lve1.3.63.el6.x86_64 (mockbuild@build.cloudlinux.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Sun Sep 27 06:34:10 EDT 2015

Generate https cert

```
sudo apt-get update
sudo apt-get install -y software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install -y python-certbot-nginx
```
Then get a certificate `sudo certbot --nginx certonly`

Now edit the nginx config to add the fullchain.pem and privkey.pem paths.

sudo vim /etc/nginx/sites-enabled/default

Add lines 22-25.

server {
20 # SSL configuration
21 #
22 listen 443 ssl default_server;
23 listen [::]:443 ssl default_server;
24 ssl_certificate_key /etc/letsencrypt/live/autox.network/privkey.pem;
25 ssl_certificate /etc/letsencrypt/live/autox.network/fullchain.pem;
...
}

Four coloring for large software

"In mathematics, the four colour theorem, or the four colour map theorem, states that, given any separation of a plane into contiguous regions, producing a figure called a map, no more than four colour are required to colour the regions of the map so that no two adjacent regions have the same colour." -- Wikipedia

"A codebase composed of small interconnected modules that communicate with each other in a well-defined way will have a lower complexity than one where the modules communicate with each other haphazardly." - http://alexkudlick.com/blog/what-the-four-color-theorem-can-teach-us-about-writing-software/

How to recover locked out AWS EC2 ssh machine

One time I accidentally messed with the `/etc/passwd` and locked myself out of being able to SSH into the machine. Since this is a remote machine in AWS I had no way of doing what I'd normally do. Which is attaching a keyboard and monitor and fixing this manually.

To fix, use the AWS EC2 Management page to:
- spin up a new instance of vanilla ubuntu EC2 (let's call it David)
- shutdown the locked machine (let's call it Goliath)
- unmount Goliath's volume
- attach the volume to David

Then follow this guide: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

Summary of what I did from this guide:
```
lsblk
sudo file -s /dev/xvdf # MBR (not data type)
sudo file -s /dev/xvdf1 # ext4
sudo mkdir mount_folder
sudo mount /dev/xvdf1 mount_folder # ext4 mounted
cd mount_folder
# undo crazy setting (see Note #1)
cd .. # to unmount
sudo umount /dev/xvdf1
# Note #2
```

Note #1: For me I tried to modify `/etc/ssh/sshd_config` to allow one more user to login. But this made me unable to login after. So I removed the offending line.

Note #2: now in the Volume webpage
- undo attach to David (Volumes tab)
- mount to Goliath (Volumes tab: attach as EBS path /dev/sda1)
- boot up Goliath (Instance tab)

Black Friday PC Building

 

Done/No,Description,Amount
TRUE,Cooler Master MasterLiquid Lite 240mm Liquid Cooling,$42.89
TRUE,Intel i7 7700 CPU,$269.36
TRUE,ASRock Z270 KILLER SLI/AC Z270 Motherboard,$128.69
FALSE,Mobo MIR,-$30.00
TRUE,"EVGA SuperNOVA 1000 G2, 80+ GOLD 1000W, Fully Modular",$118.51
TRUE,Thermaltake Core G21 Dual 4mm Tempered Two-Toned Glass Power Cover ATX Black Gaming Computer Case ,$64.34
FALSE,Case MIR,-$20.00
TRUE,Crucial Technology 32GB (2x 16GB),$259.99
TRUE,ZOTAC GeForce GTX 1060 6GB Mini,$267.89
TRUE,Toshiba Solid State Drive PCIe NVMe M.2 512GB,$247.90
TRUE,Windows 10 Home,$92.99
TRUE,Oculus Rift Headset,$679.99
TRUE,Oculus Touch Controller,$99.99
FALSE,,
,Total,"$2,272.54"

SLAM

SLAM is simultaneous localization and mapping; the goal is to build/update a map while simultaneously keeping track of location within.

In other words, SLAM takes sensory data as input (such as camera, lidar, ultrasound) and outputs a partial map and location within.

A popular open source framework is called ORB SLAM. This is fast, robust, and efficient.

 

I'll summarize my findings here.

There are three papers you can read that describes how it works. You should read them in order, because the ideas build on each other (1-2) and (3) describe improvements made.

1. Bag of Binary Words
2. ORB-SLAM paper
3. ORB-SLAM2 paper

I'm not much of an expert in this field, so I'll try to explain in layman's terms of how I understand it. Here are essence of what you need to know.

  • Sensory data are a stream, but it can be discretized, such as a video stream.
    • Discrete means to represent a real space using a discrete quantities.
  • Videos are sequences of images.
  • In an image, there are points that can be easily identified and tracked across images.
  • These points can be identified by an algorithm called Bag of Words (read 1).
    • A picture is like a paragraph, it can be described by visual words.
    • Visual words are small images that represent like noses, eyes, mouth.
    • With enough words, you can guess what objects you're looking at, and from what angle.
    • For example, if you had two eyes and a nose, you would guess a face.
    • And if the two eyes are much bigger than the nose, you're likely looking downward angle to the face.
  • Bag of Binary Words improve performances of the Bag of Words (BoW) model.
  • The ORB-SLAM system uses the BoW model to build a database.
    • Database stores BoW features that represent an image.
      • Meaning the image was converted into BoW features.
    • These features are used as an index into a location map.
  • From an image and a location, a Key Frame can be created.
    • Key Frames are an association container, which can represent a mapping of BoW features into a frame in location space, and vice versa.
  • Using these BoW features, and sensor calibration data, a location is triangulated from mathematical calculation.
  • Many of these Key Frames are linked together.
    • You can think of Key Frames being a vertex in a graph, and an edge represent co-visibility.
      • That is, the BoW feature is visible from the other Key Frame as well.
    • This graph is called a covisibility graph.
    • Which represents a map.
  • Having this graph aids in knowing when to insert a new Key Frame into the database.
    • For example, when two Key Frames have less than 75% common BoW features.
  • When given an image, the database is asked to return a list of plausible locations.
    • The image is converted into BoW feature.
    • This feature is used to search the covisibility graph to find similar Key Frames.
    • Returns a list of Key Frames with statistical likelihood to represent which Key Frames contains the most matched BoW features.
    • The covisibility graph reduces search complexity, therefore speeding up search, and also reduces storage needs.
  • So now a map is being built at the same time it is localized.
    • SLAM.
  • In the future, we can come back with more data to:
    • Improve the map.
    • Ask the database for a localization result.

That's all folks. I'll add some pictures later.

Here's a video.

 

Tools I'm loving now

Clion (actually you do not need to purchase a license if you use the EAP (Early Access Program) license. Downside is you have to download a new one each month. But it'll be free to use for 1 month, and you don't have to pay $199 USD per year.
https://www.jetbrains.com/clion/nextversion/?fromIDE=

HackerRank
Terminator
Sublime
OpenCV
Python
Tensorflow
Slam mono
Slack
Fish Shell
Git/GitLab
C++11