Author Archives: sunapi386

Mount locked partition with same volume group name

Background/Setup

  • I have two physical 1TB disks with identical setup.
  • Both are encrypted.
  • I unlocked and booted off one of them.
  • The other disk is still locked at this point.
  • I am using fish shell.

1. Identify which disk you want to unlock

root@computer ~# lsblk -f
sda                                                                                                    
├─sda1                                        vfat              7F3B-9703                              /boot/efi
├─sda2                                        ext2              b6220db2-916c-4322-b64b-c86769f6b18b   /boot
└─sda3                                        crypto_LUKS       3a07b8a9-3e75-41a9-88d4-3be937181613   
  └─luks-3a07b8a9-3e75-41a9-88d4-3be937181613 LVM2_member       uCvHaW-RlQc-PT2d-cBg2-SWyY-WS0A-ZCEvA6 
    ├─ubuntu--vg-root                         ext4              a78662e2-d582-4faa-88b6-b6db5e23aed2   /
    └─ubuntu--vg-swap_1                       swap              5c4ba6cc-4735-4417-9e87-74a76a7fc415   [SWAP]
sdb                                                                                                    
├─sdb1                                        vfat              1EC8-1F58                              
├─sdb2                                        ext2              7108fe1d-dc37-4213-a3bc-8070a8f84f31   /media/jsun/7108fe1d-dc37-4213-a3bc-8070a8f84f31
└─sdb3                                        crypto_LUKS       fc560468-588c-4455-af2c-295998c41c88   

We see that sdb3 is the unmounted target.

2. Unlock the partition

root@computer ~# udisksctl unlock -b /dev/sdb3 
Passphrase:
Unlocked /dev/sdb3 as /dev/dm-3.

See which one is the new unlocked

root@computer ~# ls -la /dev/mapper/ | grep dm-3 
lrwxrwxrwx 1 root root 7 Mar 22 17:32 luks-fc560468-588c-4455-af2c-295998c41c88 -> ../dm-3

luks-fc560468-588c-4455-af2c-295998c41c88 is our target. Let's remember that with a variable.

set target luks-fc560468-588c-4455-af2c-295998c41c88

3.

The VG Name of both drives is the same; this is problematic and will prevent you from being able to mount the drives both at the same time.

root@computer ~# pvdisplay
--- Physical volume ---
PV Name /dev/mapper/luks-3a07b8a9-3e75-41a9-88d4-3be937181613
VG Name ubuntu-vg PV Size 930.53 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 238216
Free PE 0
Allocated PE 238216
PV UUID uCvHaW-RlQc-PT2d-cBg2-SWyY-WS0A-ZCEvA6
--- Physical volume ---
PV Name /dev/mapper/luks-fc560468-588c-4455-af2c-295998c41c88
VG Name ubuntu-vg
PV Size 930.53 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 238216
Free PE 0
Allocated PE 238216
PV UUID lbvecI-E6w6-fpuj-P61G-5NCb-obOK-ooivpe

Get the uuid of the volume

root@computer ~# uuidgen
1ec80451-b05b-4d59-94c1-f1ad70b24255
root@computer ~# vgrename $uuid 1ec80451-b05b-4d59-94c1-f1ad70b24255
Processing VG ubuntu-vg because of matching UUID TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v
Volume group "TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v" successfully renamed to "1ec80451-b05b-4d59-94c1-f1ad70b24255"
root@computer ~# pvs -o +vg_uuid
PV VG Fmt Attr PSize PFree VG UUID
/dev/mapper/luks-3a07b8a9-3e75-41a9-88d4-3be937181613 ubuntu-vg lvm2 a-- 930.53g 0 TAva2M-zNnV-Wh5h-3YcY-Vc5U-W4se-TI27Du
/dev/mapper/luks-fc560468-588c-4455-af2c-295998c41c88 1ec80451-b05b-4d59-94c1-f1ad70b24255 lvm2 a-- 930.53g 0 TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v

The device UUID is TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v. Let's remember that with a variable.

set uuid TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v

Change the volume group

I'm going to generate a UUID, but you can name whatever you want.

root@computer ~# uuidgen
1ec80451-b05b-4d59-94c1-f1ad70b24255
root@computer ~# vgrename $uuid 1ec80451-b05b-4d59-94c1-f1ad70b24255
Processing VG ubuntu-vg because of matching UUID TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v
Volume group "TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v" successfully renamed to "1ec80451-b05b-4d59-94c1-f1ad70b24255"
root@computer ~# pvs -o +vg_uuid
PV VG Fmt Attr PSize PFree VG UUID
/dev/mapper/luks-3a07b8a9-3e75-41a9-88d4-3be937181613 ubuntu-vg lvm2 a-- 930.53g 0 TAva2M-zNnV-Wh5h-3YcY-Vc5U-W4se-TI27Du
/dev/mapper/luks-fc560468-588c-4455-af2c-295998c41c88 1ec80451-b05b-4d59-94c1-f1ad70b24255 lvm2 a-- 930.53g 0 TJgeFw-xDcf-TaJ2-07dL-RlUQ-yCsb-zGGp4v

Check/notice the new volume group name 1ec80451-b05b-4d59-94c1-f1ad70b24255.

Confirm the change

root@computer ~# vgchange -a y
2 logical volume(s) in volume group "ubuntu-vg" now active
2 logical volume(s) in volume group "1ec80451-b05b-4d59-94c1-f1ad70b24255" now active

Remember to rename your volume group back to ubuntu-vg if you want the volume to still be bootable.

Mount

root@computer ~# mkdir /media/badboy
root@computer ~# mount /dev/1ec80451-b05b-4d59-94c1-f1ad70b24255/root /media/badboy
root@computer ~# cd /media/badboy/
root@computer /m/badboy# ls
bin/ cdrom/ etc/ initrd.img@ lib/ lib64/ media/ opt/ root/ sbin/ srv/ tmp/ var/
boot/ dev/ home/ initrd.img.old@ lib32/ lost+found/ mnt/ proc/ run/ snap/ sys/ usr/ vmlinuz@

You can now access your data.

Merge EPT Entwine Point Tile Maps

EPT format is a storage method for point clouds, base on an octree structure. The encoding of the point cloud is up to the user, whether las/laz, binary, or whatever custom format. So let's say you store it in laz. What EPT does is to generate the octree to manage how those laz files are stored.

Merging EPT

Given two geographically separated point clouds, it is possible to merge them into the same EPT structure, given they use the same frame of reference. Because Potree doesn't handle latitude and longitude rendering (as they are angles and not euclidean), you have to use euclidean systems, such as UTM coordinates.

Merging two point clouds together

You must first specify a large bounds for the initial build, and add more files later like this:

entwine build -i ~/data/xyz -o ~/entwine/xyz -b "[100, 100, 0, 400, 400, 200]"
entwine build -i ~/data/abc -o ~/entwine/xyz

See Entwine issue 109

Caveats

A caveat is that when you're generating the first frame, you need to custom input the bounding cube bounds, because by EPT design the octree cannot be rebalanced (without recomputing for all points).

Another caveat is that UTM zone is not encoded in the laz file, using UTM cannot be extended over geographical regions. Thus we may have to store xyz points in lat/lng/alt format.

Yet another caveat is the source ID for these points are going to collide. E.g. in file1.laz you have frames ID 0-100 and file2.laz you have frames ID 0-10, then the 0-10 IDs would collide. A workaround would be to assign global unique IDs.

WD 8TB Hard Drive Shucking For NAS

The Western Digital (WD) Easystore 8TB is an external hard drive sold exclusively by Best Buy. It's highly sought after because it contains a WD Red or unlabelled WD White 8TB drive, which can be "shucked" and used in a home server.

Typically you can get 8TB for $140+tax when on sale, whereas a regular WD 8TB NAS drive costs around $260+tax.

There are more details in this reddit post about the drives and how to shuck them.

Start by inserting plastic cards into these places and pry them out.

Wiggle out the drive.

Two screws hold the PCB board on the drive.

Slide out the connector

Done, now to mount into the Synology drive mount.

Insert and then configure in Synology.
Disk 3 shows up in Synology Storage Manager.
Add the disk, and it'll take some time to wipe and initialize.

After a few hours the volume is expanded.

Install g++-8 compiler

The new C++17 language requires g++ >= 8.0. To install it in Ubuntu 18.04:

$ sudo apt install g++-8
...
$ which g++
/usr/bin/g++
$ which g++-8
/usr/bin/g++-8
$ sudo unlink g++
$ sudo link -s /usr/bin/g++-8 /usr/bin/g++
$ g++ --version
g++ (Ubuntu 8.2.0-1ubuntu2~18.04) 8.2.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Re-link After Make Install

The magical command sudo /sbin/ldconfig -v reconfigures/indexes the files you installed into the lib.

For example, if you see errors such as:

~/w/h/build (master) [127]> curse
curse: error while loading shared libraries: libhexer.so.1.1.0: cannot open shared object file: No such file or directory

Or similiarly

~/w/e/build (master)> entwine
entwine: error while loading shared libraries: libentwine.so.2: cannot open shared object file: No such file or directory

Then go and run the magical command.

Typically this is needed when you do sudo make install after compiling some C++ code. I suppose you don't need the -v verbose flag, because ldconfig - configure dynamic linker run-time bindings.

Compiling PDAL

sudo apt install libgdal-dev
git clone https://github.com/PDAL/PDAL.git
cd PDAL
mkdir build
cd build
cmake ..
make -j
sudo make install

List cmake options with installing this GUI

sudo apt-get install cmake-curses-gui ccmake ..
ccmake ..

To get geowave plugin support, you need JDK. Geowave increases the project's complexity so I'll defer that to later, if I get to it.

sudo apt install openjdk-8-jdk

Never Again, BTRFS! NAS File System Rant.

Synology DS918+ NAS

I have a Synology DS918+ NAS running BTRFS. This is 2019 and development of BTRFS began in 2007. So it should be all good, right?

There are interesting features I wanted to try out. Mainly, keeping snapshots. I've also been using EXT3/EXT4, NTFS, HFS, APFS, FAT32, even reiserFS back in 2006-2010, so I've seen my share of file systems. I'm willing to try something new.

I've heard 2 of my friends complain about this file system, but decided to try it because Synology's Setup UI made me inclined to think this is preferable.

YouTube

I don't know who the target audience for this web UI is, but recommending the BTRFS is a huge mistake. As a home user with a lot of pictures, it is just so unacceptably slow.

How slow is it?

I don't have exact numbers, because they often time out. And I don't want to go through the pain of waiting. I have about 16k jpeg files that in a flat folder. Simple operations are terribly slow.

  • Moving 100 files into a folder takes couple of minutes, because there is a copy made for each file moved.
  • Doing ls (list all files in folder) takes forever, my terminal hangs up and never returns even after hours.
    • On the Synology Drive UI, it takes minutes, because the web UI is paginated, about 500 files listed per page.
  • Deleting files takes as long as moving them.
  • Defragging like 1995 is back, so you must remember to periodically "scrub" your disks, which take days (I have two 4TB drives).

I regret so much using this shitty BTRFS system. Unbelievable. RedHat abandoned using this for all the right reasons, it's hopeless. I'll be spending a weekend moving my data to USB disks and reformatting to EXT4.

Never again, BTRFS!

XDinner

Wrapping up a project at and moving the project to phase 2. We celebrated with a team dinner at Cascal. Actually the first time our small project had outside dinner.

I didn't take notes at the time so these are the memories from last night.

I will roughly summarize the events and some principles. I won't write much into details partly for sake of privacy and partly because I can't remember nor have the time to flush it all out. So I'll write some memory triggers.

I had a bunch of questions in revolving around:
- What is going on with the project?
- What are the plans for our other projects?
- What were the lessons learned from building a company?

And other personal and life questions as
- Why did you create the company?
- How come you don't want to be a professor anymore?
- Why did you want to become a professor?
- How did you climb the life ladder from where you started?
- What are your principles?
- How did you get to where you are today?

And some summaries of his responses
- work very very hard
- be opportunistic, they come and don't wait
- surround yourself with people you want to become

Server migration

I'm in the middle of migrating the this website from hosted on a shared namecheap server to a VPS server. Namecheap notified me that this site will be undergoing for hardware upgrade, moving to a different physical server. Let me remember what the old hardware was, and I'll update this post later with the new hardware. Schedule update is for Mar. 25, 2017.

[phepha@server159 ~]$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
stepping : 4
microcode : 1064
cpu MHz : 2099.884
cache size : 15360 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
...

[phepha@server159 ~]$ cat /proc/meminfo
MemTotal: 65786112 kB
MemFree: 740052 kB
Buffers: 10627716 kB
Cached: 38361744 kB
SwapCached: 0 kB
MemCommitted: 1031798784 kB
VirtualSwap: 0 kB
Active: 29757552 kB
Inactive: 25222428 kB
Active(anon): 4747632 kB
Inactive(anon): 1283136 kB
Active(file): 25009920 kB
Inactive(file): 23939292 kB
Unevictable: 5244 kB
...

[phepha@server159 ~]$ cat /proc/version
Linux version 2.6.32-604.30.3.lve1.3.63.el6.x86_64 (mockbuild@build.cloudlinux.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Sun Sep 27 06:34:10 EDT 2015