Nginx hosting multiple domains with a single SSL certificate from Let's Encrypt

Leave a reply

Multiple domains can be hosted with nginx server blocks. Below I give an example of how to generate a shared SSL certificate from Let's Encrypt, and how to setup two server blocks to use a shared SSL certificate.

Assuming you use Let's Encrypt as your CA authority, generate a SSL certificate for the domains:

sudo letsencrypt certonly -a webroot --webroot-path=/var/www/default/html/ -d example.org -d example.com

Alternatively, if your not using Let's Encrypt as your CA, read up the docs http://nginx.org/en/docs/http/configuring_https_servers.html take a pause to note:

The SSL certificate needs to contain several names, in the SubjectAltName certificate field, for example, you might want to have example.com and example.org domains. Note the SubjectAltName field length is limited, to about 1000 characters IIRC.

Now I assume you have the SSL certificate generated.

Update the two server blocks /etc/nginx/sites-available/example.com and /etc/nginx/sites-available/example.org accordingly. A /etc/nginx/sites-available/default is not need. See the diff:

server {                                                        server {
  listen 80;                                                      listen 80;
  listen [::]:80;                                                 listen [::]:80;
  server_name example.com;                                    |   server_name example.org;
  return 301 https://$server_name$request_uri;                    return 301 https://$server_name$request_uri;
}                                                               }
server {                                                        server {
  server_name example.com;                                    |   server_name example.org;
  listen 443 ssl http2;                                           listen 443 ssl http2;
  listen [::]:443 ssl http2;                                      listen [::]:443 ssl http2;

  ssl_certificate /etc/letsencrypt/live/default/fullchain.pem     ssl_certificate /etc/letsencrypt/live/default/fullchain.pem
  ssl_certificate_key /etc/letsencrypt/live/default/privkey.p     ssl_certificate_key /etc/letsencrypt/live/default/privkey.p
  include snippets/ssl-params.conf;                               include snippets/ssl-params.conf;

  root /var/www/example.com/html;                             |   root /var/www/example.org/html;
  index index.php                                                 index index.php 
  location / {                                                    location / {
    try_files $uri $uri/ /index.html =404;                          try_files $uri $uri/ /index.html =404;
    autoindex on;                                                   autoindex on;
  }                                                               }
  location ~ /.well-known {                                       location ~ /.well-known {
    allow all;                                                      allow all;
  }                                                               }
  location ~ \.php$ {                                             location ~ \.php$ {
    include snippets/fastcgi-php.conf;                              include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/var/run/php5-fpm.sock;                       fastcgi_pass unix:/var/run/php5-fpm.sock;
  }                                                               }
}                                                               }

Note the folder that contains the SSL certificates /etc/letsencrypt/live/default/. You may need rename the folders that letsencrypt generated.

Additional references:

https://www.digitalocean.com/community/questions/letsencrypt-for-multiple-domains-on-nginx

Lessons from Real-Time Programming class

Leave a reply

Lessons from Real-Time Programming class

This post is about lessons I learnt in CS452 Real-Time Programming, a 4th year undergraduate class at the University of Waterloo. Better known as “trains” or “realtime” class.

“Easy stuff! You play with trains all day (literally, maybe even night). All you need to do is write one simple program to control multiple trains so they don’t crash!”

— chko on BirdCourses.com

The trains lab. Room 3018, 3rd floor Math & Computer building.

The class is a bit unconventional because it indirectly teaches other topics, like time management and teamwork, by providing a large amount of work — so that you have to learn these skills. I’ll talk about these lessons, hoping to address a more general audience.

History

This class has been around since at least the 80’s. Currently Bill Cowanteaches this class, and has been for over 20 years.

The equipment have evolved since then, but the underlying challenges have not.

For example, teamwork, setting priorities, and dealing with real (imperfect) systems.

Prof. Bill Cowan explains the why sensor pickup sometimes have trouble detecting slow moving trains. The gap between the contact points may cause the pickup switch to flip twice.

The early version of the QNX real-time operating system took off from a pair of students taking this class, Gordon Bell and Dan Dodge. They decided to expand the class project into a full-fledged commercial RTOS (real time operating system).

“You should try to find a partner for this class, it’ll be hard without one. Generally I’ve seen two outcomes:
(1) you become friends with your partner, or
(2) you stop being friends and hate each other.
I recommend not starting with a partner who is your friend, because you are already friends. Don’t risk it.”

— Prof. Bill Cowan

Curriculum

About 65% of the class involves building a small embedded kernel, with the other 35% working on the train management application.

You can choose to program in whatever language you want, because there aren’t any standard libraries provided. People generally program in C, because it’s known as a systems language, and the programming examples given in class. But there’s no reason why you can’t do it in Rust language, for example — although godspeed, you’re venturing into uncharted territory!

The kernel runs on a TS-7200 embedded system, and it talks to a Märklin Train System over RS232 type serial. The firmware is loaded on via RedBoot, a bootstrap environment for embedded systems.

The Märklin Train System. You can see the center console where the RS232 is attached. Off to the right is the grey box that houses the TS-7200.

The final exam is a take home 24 hour long exam, where you can use any resources you want, and you only need to do 3 of the 5 questions. The assignments and kernel make up 70% of the grade.

Optional Technical Info

The microkernel did very little; it

only kickstarts a couple of initial processes.
handled message passing between processes.
provided very basic interrupt handler and clock.
supports realtime operations; the scheduler is predictable (through priorities, and pre-emption).
avoided disk IO by loading kernel and programs into RAM directly.
needed to provide serial output to a PC terminal, to display graphics relating to the real-time actions being controlled, such as displaying position of trains on the track showing movement.

The realtime program is probably more work than the kernel. Because initially there were many constraints, but later on, assignments were of more guidelines than objectives.

Lessons

I’ll talk about some of the lessons I learnt, and provide some reasoning or backstory to them.

Don’t trust anything. Don’t make any assumptions.

One assumption that was made was while building a acceleration and deceleration profile for the train — that the train would stop where we expected to. Calibration was done by running the train at full speed around the tracks, and when it hits a sensor, tell it to stop.

That assumption turned out false: at max speed, the train varies its stopping distance quite a bit, up to ±3 cm. Not only that, the train accelerates and decelerates non-linearly!

Real systems are often imprecise. For example, look at the stopping distance of a train. In our experiments, it never stops in the same place — for a variety of reasons: such as buffering, interrupt handling, incorrect system timing.

This meant that over the course of several start and stop move commands, there can accumulate quite a difference between where you think the train is, and where it actually is.

Consequently, our model couldn’t be trusted. We’d assume it was safe to perform a certain operation, and it of course, wouldn’t be. And that causes accidents like multi-track drifting to occur.

Multi-track drifting. Happens when a track switch flips direction while the train has yet to clear both axles.

Multi-track drifting is when you thought it was safe to flip a switch, and then it wasn’t. The poor train comes to a sudden stop because the front axles and rear axles are going to different directions.

When your model of the world and the real world itself are different, you get problems. We couldn’t trust our model. We’d assume it was safe to perform a certain operation, thinking we knew where the train was, and it wasn’t safe.

This explains the comic that is pasted on the walls beside the train set. Now I get it!

A printed out version of this comic was taped to the wall beside the train tracks. The original multi-track drifting.

If this was a real train, this would result it some serious accidents. A lesson we learnt after trying to figure out how this happened.

Assumptions are going to be wrong! So it’s important to build in error correction and recovery.

Often assumptions are wrong, or become wrong during the course of real life usage. Real systems don’t always run in your ideal situations. You can’t always think of all the edge-cases. Errors are going to happen. Your best chance is to try to realize the error as early as possible and correct for it.

Instead of eagerly switching tracks for the next train, we switched tracks only if the train had ownership of the track.

Map of the train switches and segments. We had two tracks, A and B, and they have different configurations. We hard coded the map to build the train track reservation system.

To solve this, we ended up building in margin of error around where we think the train is, assigned the train to have ownership to tracks, and made sure when trains approach tracks that they didn’t own, it would stop.

You are as only good as your tools.

Building the path/route finding portion of the project, we were manually reading a map and trying to reason about what the shortest ways were. This took a long time, and felt complicated.

Route a train from point A to point B was not difficult, but factoring in what paths are currently, or will be, blocked by other trains made it harder. We often stared at the layout map for a long time, trying to figure out whether this was the quickest path through this or not.

This is the track layout map. It’s easy to find the shortest path visually, for one train. Now try imagine putting more trains on here. After two trains, becomes pretty difficult. Trains probably need to negotiate with each other, and it’s not enough to try to all take the shortest path.

After staring at this for a long time, and it was clear that reading a map like felt a lot of work. Time to make our life easier.

This map contains the same information as the track layout map, but represented as a tree. This way, we can easily figure out what were shortest paths by counting the least number of moves. This tree factored in backwards movements as well. Because point A and point B may be most easily reached by reversing direction.

This tree map didn’t take much more effort, it was generated by GraphViz in the Dot language.

Tools empower you to be more productive. As a programmer, you’re always thinking about how to make things automated, and easier. A commonly seen trait that programmers have is to be lazy, a desire to automate things and make their own lives easier. Determine how much time you spent doing some work, and see if you can build a tool to make it easier.

Invest in building better tools when you find yourself repeating a task, as a rule of thumb, more than three times.

It’s ok to take on technical debt. Keep it simple, stupid.™

For example, having fancy colorful terminals. Ours was initially very pretty, but we noticed a lot of things had to be changed. Towards the end of the projects, we had avoided building fancy graphics because they took a lot of work and are likely to be changed later.

If you wanted to have pretty looking terminals, lot of time had to be dedicated for it. If you’re really bored, and have time to kill, I suppose it’s fine.

Our colorful terminal. It was all pretty, initially. Later on, we couldn’t afford much time to make things look good.

Later on, we took on more technical debts with screen drawing. It no longer looked as nice, but it worked.

If there are technical debts with higher priority, they should be addressed before diving into a lesser problem.

This sounds intuitive, but sometimes people become irrational and just spend time on low priority issues. Like making things look pretty.

Pair programming is effective teamworking.

Initially we dividing up projects, and mostly worked separately. This was okay for well-defined problems, as it was the case for initial kernel implementation. But later on, the problems we needed to solve became so inter-dependent that it was no longer feasible to build something in the dark.

Many quick idea iterations were needed, and we needed each other to bounce ideas around. Having a comfy couch is important!

And towards the end, we were doing pair programming out of despair; we had to write working code, and we had to write it fast.

Pair programming produces quality code that is better thought out, and a rate that is comparable, if not faster, to separating out tasks and doing code reviews.

Summary

I think taking this class was very valuable, and was a very nice way to wrap up my Computer Science undergrad.

These lessons are ingrained in my memories, and helped shape my behavior.

For example, after pair programming, I subconsciously debate about solutions to problems and bounce ideas around. To an outsider, it may seem argumentative, but it’s really an effective way to solve problems.

Thank you for reading.

On planning for the future

Leave a reply

Growing up, I looked up to a lot of heroes for motivation, and Steve Jobs is one such inspirational hero for me. He was influential to a lot of the values I hold true today. I had read his autobiography – multiple times – and I think it is one of the best books I've read. I wish I had been able to be join Apple while he was still around, to work with him, and experience the reality distortion field™ in person.

Most of the conversation I have with my friends and family would generally contain topics of what my ideas are, and how I plan to achieve them. They are interest in what my values are, how I make decisions. Usually I end up paraphrasing things that I've before, to others.

I want to summarize some of the principles I hold, as best as I could. Hopefully I don't forget too much.

Principles

On What You Should Do

In the FB post, I talked about two principles (that help decide what you should do).

Do the things that inspire you, that you love, and
try to be really good at it, hopefully having commitment and courage to do it.

Let me elaborate on this, and then talk about some additional principles for how to get you to the goals you want.

Firstly, by doing the things that inspire you and that you love, this results in personal satisfaction. It's very important, because it keeps you motivated for when the problems get tough – and there are going to be a lot of tough problems.

Anything worth doing is going to be difficult.

Of course, the assumption is that these things that you do are going to be creating value; whether it be for yourself or others. Just don't use this as an excuse for binge playing video games, or binge watching Netflix.

Secondly, trying to be really good at it is important for you to create something of value. It's important to have both breadth and depth – and being good at something gives you depth, a competitive edge over others.

Sure, there may be others that are smarter than you. But they have only 24 hours in a day, just as you do. So that places a limit on what others can possibly do, during their spare time. You don't have to be the smartest to have a competitive edge – it's sufficient to excel at skill, albeit seemingly small.

Commitment and courage are other necessary conditions to developing the skill necessary. You should already have the motivation and desire to pursuit something you like doing; whether it be solving a problem or becoming really good in a skill.

The second point complements the first.

These two principles are just about sufficient to be a guide on how to decide what to do, what your next target is, but not how to get there.

On How To Get There

Now that the objectives have been set, what determines the success of it is how to get there.

Manage your free time. We all have 24 hours, how that time is spent that really matters.
- Identify things that you should invest your time in, in order to reap the fruits of your labor.
- But also balance life and work. Allocate your most valuable resource, your time, well – with significant others, working out, even video games. The important concept is moderation. For example, watching a season of Netflix is probably no use to getting to your goal.
Read more. Non-fiction. On topics that immediately relevant to you.
- The value of information depends on context. If I told you what the stock price of some company would be in the future, that would be very valuable; on the other hand, stock price in the past is less valuable.
- Knowledge is key. Sharing knowledge is also key. Don't forget about your peers.
- Reading can give you a head start, often what your peers cannot obtain. Compared to others, you are making more informed decisions, better strategies and tactics. Against competition, even the smallest leverage or advantage makes a difference.
Build projects. Develop your skills.
- But don't work in isolation. Get feedback to improve faster.
- Success and failure doesn't matter as much; there are things to be learnt even within failures.
Network with others, build your connections.
- The brain's neurons work with others to achieve a common goal. A single neuron can't achieve much on its own. As I said earlier, anything worth doing is going to be difficult. And difficult things need more resources to be tackled.
- In machine learning, a neural network is inspired by the neuron architecture. When you learn, you are making new connections; and when you practice, you are strengthening the connections.
- A strong network makes you better informed, helps you get relevant information quicker, creates more opportunities.
Start today, rather than tomorrow.
- Why is today such a good place to start? Starting today gives you a head start than starting tomorrow. And since you didn't start yesterday, then today is the optimal day.
- Take a small step, begin by just doing a little. Great accomplishments don't happen overnight. Great Wall of China wasn't built in a day.

That's all for now folks. Thanks for reading!

PS:

I'm writing this while at the Starbucks on El Camino Real near my house, and coffee's power is fading. I will try to contribute and share knowledge more often.

If you found this blog to be useful, let me know! My email is jason@sunapi386.ca and Facebook is https://www.facebook.com/sunapi386 – feel free to contact me.

Music: Driftmoon.

How I met Sophie

Leave a reply

This is a tale about how I met my Sophie. Happy Valentine, Sophie! 情人节快乐！

In February 2016, I spent the first two weeks travelling in China and celebrating Chinese New Year. Having family across the world felt great, and I could visit them any time. China is in an economic boom, and growing very fast. It felt good to be connected to other parts of the world. I had recently graduated university and moved to the Bay Area for just about 3 months. I still felt there was a lot of travelling I could do, to explore the world. But travel is costly, from the time and financial perspective. Is there is another way?

Becoming a hosting on couchsurfing.com

From December to end of March, I was renting the master bedroom in a two bedroom apartment. Since our apartment had two couches, I felt I could make use of them to use it to host some couchsurfers and make friends from all over the world. So I subscribed to the couchsurfing email newsletter.

One day while scanning through my email of couchsurfers looking for hosts, Sophie's profile photo caught my eye. She was very pretty. Her profile described herself as a high level headhunter, ambitious, eager, and interested in startups.

I offered to host her on the first day she lands in US. But she replied with moderate interest, and describing an overly ambitious schedule of visiting too many cities.

Unfortunately, she had found a host already, but she was willing to meeting up for coffee. We met on Sunday Feb. 21, 2016.

That Friday and Saturday, I was hosting my first couchsurfer, Satoshi, from Japan. His profile described him as an aspiring entrepreneur visiting SV. I was hosting him, my friend Shaon was looking to buy a house. While we drove around, I was messaging Sophie to see when and where she can meet up with us.

Meeting my 2nd couchsurfer

At the end, we settled for meeting at my home, and both Sophie and her host met my friend Shaon and Satoshi.

When I first saw her, I had two impressions: "oh my goodness she is beautiful, cuter than her profile photo" and followed by "oh she's shorter than I thought".

After getting acquainted, we all went on a spontaneous trip to see Half Moon Bay and try to catch sunset. I briefly chatted with Sophie in the backseat, before doing the driving myself.

Returning, we went to visit Facebook, which was a fun place and suitable for visitors. Apple had no such open door policy. Sophie and I chatted some more about who we were.

Taking silly photos at Instagram HQ

Towards the end of our visit at Facebook, I had made a good impression with Sophie. She had changed her mind about not staying over and asked her couchsurfer whether it was okay to stay with me. Of course it was.

First impressions

We chatted late into the night. And as I closed my eyes, I knew I liked her. A lot. I was so excited that I woke up at 5 am to blog. And then after I was done, I woke her up to chat.

We had talked and next thing I know, it was morning. I still had to work, but I decided to take half day off to get up later, show Sophie around my office, then drop her off at Stanford University. And I missed her immediately.

Date #1: Las Vegas

Immediately after dropping her off, I convinced and coordinated with her so I can come to Las Vegas and travel together. Below is a cute video taken at Hoover Dam. I had rented a yellow Camaro. I had always wanted to drive a yellow sports car!

After this trip, Sophie had returned to China. We decided to try long-distance dating, and we did video calling consistently, about 3 times a day, 30 minutes each time.

Date #2: Hawaii

After a while, we wanted to meet up again. My family wanted to do a big vacation, and we settled on going to Hawaii. I invited Sophie to meet with my family.

After the trip to Hawaii, we still continued to do long-distance, and we meet up again. This time I wanted to meet her parents. So I visited China.

Date #3: Shanghai

Being with Sophie in China is like being a tourist. Although I had gone to Shanghai before, having Sophie show me around is a completely different experience.

I had remembered to take some videos this time. (I didn't have videos from Hawaii trip. Shame.)

We are now progressing well.

Ember rails developer

Leave a reply

Background

Since April this year, I essentially transitioned from an application developer to a web developer. Full stack. It's quite exciting! There are some who perceive web development as an inferior choice of work, but I have to disagree. In an era where mom and pop shops have websites, not having a webpage front for your business essentially means you don't exist. Because having a webpage allows you to reach a greater audience. So, I see it as a very powerful leverage.

I have a little background developing for Ruby on Rails, from my Dotabuff scraper project. Now that I look at it, my understanding of rails is fairly incomplete. Rails, and Ember alike, are scaffolding heavy. The intuition is that as the creators of the web framework, they are more knowledgable about how you should go about designing the architecture than you are. Consequently, it means transitioning from one Ember/Rails project to another means very little time is required.

What is Rails/Ember

Rails is a complete web framework. It handles the entire web pipeline. From

routing a URI,
to finding the controller for the page,
who translates the URI to some requests on the data model and accessing the database,
to finally rendering the web page template.

Rails 5 (which should come out of beta this year) has an API-only mode. That means it doesn't template web pages (no step #4). Rather, it replies with the data directly in JSON format.

Ember is like Rails, but it's designed to handle the frontend, leaving the backend architecture up to the user. It means the focus of this framework is on rendering web page template. Any dynamic data that needs to be displayed gets translated to an API call to go fetch the data. Then it converts the response into an ember-data model, and finally fills in the template to display it.

Why make it more complicated by using yet another framework just for the frontend?

So at this point, you're probably wondering why the additional complexity? Why not just use Rails since it supports the entire process?

Well, you see, it's about speed and responsiveness. Every time you visit a different web page on Rails, an entire html file is rendered for you by the Rails server and sent to you.

Think about that. What does it mean? It means it's slow. Unresponsive.

Rendering pages cost time. Sending the rendered page across the network also cost time. The larger the rendered file size, the longer it takes. And this is not cheap, you notice the web page flashing when you visit different pages.

So that begs the question of how do we avoid sending out rendered pages from the server? Rendering could be done on the client, rather by the server. This is where frontend frameworks come in. Rather than requesting entire web pages all the time, just request the data the page needs as an API call, and render it on the client. In between visiting different pages, data that was available before is put into client's local storage, for faster access via the cache.

Frontend frameworks

Ember is one of three popular frameworks. It's also the most complete framework, as in the most to learn. In terms of scaffolding and complexity, they rank like this:

Ember. The framework for ambitious projects.
Angular. Somewhere in-between.
React. So lightweight, it can hardly be considered a framework.

Any of these would be fine. Indeed, as with any beginner, I did some digging. And discovered that the bigger I dig, the more mess I got myself into. As a rule of thumb, I certainly knew I had no clue what I'm doing with a frontend framework. So if I'm to learn a frontend framework, I might as well learn from what are the best practices, the most scaffolding frameworks. For the same reasons why I prefer Rails as well.

My project

Currently I am building an internal project for Apple, and I (obviously) can't disclose what it is. But it uses Ember and Rails 5 API-only mode.

I setup both Ember and Rails to use web-sockets, but not really hooking them up at the moment, because I'm working on getting out a functional demo website. When the time comes, this website will not only be responsive, but updates in real-time.

See you at my next blog post folks!

Migrating copying blog from QQ Qzone to WordPress

Leave a reply

Recently I made a website for Sophie (smoosophie.com) and scraped her blog from QQ's QZone.
QQ has an API that can be used for QZone, but it is hard to develop for. Mainly the API is poorly documented and hard to use for a native English speaker. Although I could read Chinese (slowly), reading professional terms is something I have to work on.

Workflow screenshots

Anyhow, I decided to practice javascript and manually scrape it. Here are the steps I took:

Screen Shot 2016-04-03 at 18.43.17

This is what the QZone page looks like.

This is running the fetching script.

After running the fetching script, it all of the blog links as individual iframes onto the page. Because these loads are network requests, I manually wait until all of them are loaded. The javascript console will stop scrolling and printing things if they've all completely loaded.

Screen Shot 2016-04-03 at 18.45.31

Here are all the frames after being loaded. Then I run the scraper javascript, and it scrapes the iframe for their blog content and post them to the current screen.

I manually copy paste them into csv files.

This is the first line of a csv entry.

Then I wrote a ruby script to add these csv to a wordpress blog.

Here's what running it looks like.

Source code

Run the fetcher.js on the blog page.

// fetcher.js
// Javascript to paste into chrome console to fetch the posts.

var ul = document.createElement("UL") // Create a <ul> node
ul.setAttribute("id", "myList")
document.body.appendChild(ul);
var s = "|^@^|"; // seperator
var iframesArray = [];
var messagesArray = [];
var posts = document.getElementById("tblog").contentWindow.document.getElementById("listArea").getElementsByTagName("li")
for (var i = 0; i < posts.length /**/; i++) {
 var postTitle = posts[i].getElementsByClassName("article")[0].textContent;
 var postLink = posts[i].getElementsByClassName("article")[0].getElementsByTagName("a")[0].href
 // http://user.qzone.qq.com/765591203/blog/1400775512
 var entryDate = posts[i].getElementsByClassName("list_op")[0].childNodes[0].textContent;
 var readCount = posts[i].getElementsByClassName("list_op")[0].childNodes[2].textContent;
 var message = i +s+ postTitle +s+ entryDate +s+ readCount +s+ postLink +s;
 messagesArray.push(message);
 var iframe = document.createElement('iframe')
 iframe.src = postLink;
 document.body.appendChild(iframe)
 iframesArray.push(iframe);
}

After you manually determine the time to wait, run this to scrape all the iframes and add to the current html. There you can copy paste into a csv file.

// scraper.js
// Execute this when everything seems to have laoded
for (var i = 0; i < iframesArray.length /**/; i++) {
 var contWin = iframesArray[i].contentWindow;
 var win = contWin.document;
 // Scraping
 var blogsection = win.getElementById("tblog").contentWindow.document;
 var postTexts = blogsection.getElementById('blogDetailDiv').children;
 var postString = "";

 // Stupid check because sometimes if there are no childrens.
 if (postTexts.length == 0) {
 postString = blogsection.getElementById('blogDetailDiv').innerHTML;
 }
 else if (postTexts[0].tagName == "BR") {
 var brString = blogsection.getElementById('blogDetailDiv').innerHTML;
 postString = brString; // .replace(/<br>/g, "\n")
 }
 else {
 for (var p = 0; p < postTexts.length; p++) {
 postString += "<p>" + postTexts[p].innerHTML + "\n&nbsp;</p>";
 }
 }

 postString = postString.trim();
 if (postString.length == 0) {
 postString = "ERROR parsing!";
 }

 var message = messagesArray[i] + postString;
 console.log(message)
 var node = document.createElement("LI") // Create a <li> node
 var textnode = document.createTextNode(message) // Create a text node
 node.appendChild(textnode) // Append the text to <li>
 document.getElementById("myList").appendChild(node) // Append <li> to <ul> with id="myList"
 // document.body.removeChild(iframe)
 // win.body.parentNode.removeChild(win.body)
}

Once the csv files are created, post it to the wordpress blog.

#poster.rb
require 'rubypress'
require "csv"
require 'sanitize'
wp = Rubypress::Client.new(:host => "smoosophie.com",
 :username => "sophie",
 :password => "1Smoosophie!",
 # :use_ssl => true,
 :retry_timeouts => true)
Dir["*.csv"].each do |name|
 puts "============= #{name} ============="
 csv = CSV.read(name, {:col_sep => "|^@^|", :quote_char => "" }) # whatahack 
 csv.each do |row|
 id, title, pubdate, readcount, link, text = row
 cleantext = Sanitize.clean(text, :elements => ['br','p', 'a'])
 puts("#{id}-#{title}")
 # puts("#id #{id}, title #{title}, pubdate #{pubdate}, readcount #{readcount}, link #{link}, text #{text}")
 begin
 retries ||= 0
 puts "try ##{ retries }"
 sleep(1)
 wp.newPost( :blog_id => "0", # 0 unless using WP Multi-Site, then use the blog id
 :content => {
 :post_status => "publish",
 :post_date => Time.parse(pubdate),
 :post_content => cleantext,
 :post_title => title,
 :post_author => 1, # 1 if there only is admin user, otherwise user's id
 :terms_names => {:category => ['QQZone'], :post_tag => ['QQZone'] }
 }
 )
 rescue
 retry if (retries += 1) < 3
 end
 end
end

Neural Style

Leave a reply

Inspiration comes from https://github.com/jcjohnson/neural-style.
Because installing all the required toolchains on OS X 10.11.3 is a bit challenging, I here are my installation steps.

 cd workspace/
 git clone git@github.com:jcjohnson/neural-style.git
 git clone https://github.com/torch/distro.git ~/torch --recursive
 cd ~/torch; bash install-deps;

This will fail a few times if you have already installed them (but different versions). I needed fiddle around unlinking things.

 brew unlink qt
 brew linkapps qt
 brew link --overwrite wget
 bash install-deps;
 brew unlink cmake
 bash install-deps;
 brew unlink imagemagick
 brew unlink brew-cask
 bash install-deps;

Anyhow, make sure the install-deps script doesn't error out, otherwise you'll be missing dependencies.

 ./install.sh

This succeeds. It tells you to activate, but I'm using non-standard shell (fish shell), so I mess with the fish config.

 . /Users/jason/torch/install/bin/torch-activate
 th #checking this exists in path, and it doesn't
 luarocks install image
 source ~/.profile
 . ~/.profile
 vim ~/.bashrc
 subl /Users/jason/torch/install/bin/torch-activate
 subl ~/.config/fish/config.fish
 th #now it exists

The change into fish.config that was necessary (for my user, my paths) was:

# . /Users/jason/torch/install/bin/torch-activate
set LUA_PATH '/Users/jason/.luarocks/share/lua/5.1/?.lua;/Users/jason/.luarocks/share/lua/5.1/?/init.lua;/Users/jason/torch/install/share/lua/5.1/?.lua;/Users/jason/torch/install/share/lua/5.1/?/init.lua;./?.lua;/Users/jason/torch/install/share/luajit-2.1.0-beta1/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua'
set LUA_CPATH '/Users/jason/.luarocks/lib/lua/5.1/?.so;/Users/jason/torch/install/lib/lua/5.1/?.so;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so'
set PATH /Users/jason/torch/install/bin $PATH
set LD_LIBRARY_PATH /Users/jason/torch/install/lib $LD_LIBRARY_PATH
set DYLD_LIBRARY_PATH /Users/jason/torch/install/lib $DYLD_LIBRARY_PATH
set LUA_CPATH '/Users/jason/torch/install/lib/?.dylib;'$LUA_CPATH

Convert the bash syntax to fish syntax by replacing "export" with "set" and ":" with " ".
I have a Nvidia graphics card, so I download and install CUDA.

Then we can continue installing dependencies.

 brew install protobuf
 luarocks install loadcaffe
 luarocks install torch
 luarocks install nn

I found that having Xcode 7 means the clang compiler is too new and not supported by cutorch and cunn. The error you would see is this:

nvcc fatal   : The version ('70002') of the host compiler ('Apple clang') is not supported

Sometimes the error messages are garbled. Concurrency build, I presume. I downloaded Xcode 6.4, and replaced my Xcode 7:

cd /Applications
sudo mv Xcode.app/ Xcode7.app
sudo mv Xcode\ 2.app/ Xcode.app # this is Xcode 6.4 when you install it
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
clang -v 
#Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
#Target: x86_64-apple-darwin15.3.0
#Thread model: posix

But now I get another fatal issue:

/usr/local/cuda/include/common_functions.h:65:10: fatal error: 'string.h' file not found
#include

Seems like this is an issue people have, cutorch issue 241. Can be resolved by doing

xcode-select --install

This gets me a little further, now the issue seems related to torch.

make[2]: *** No rule to make target `/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/Accelerate.framework', needed by `lib/THC/libTHC.dylib'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[ 64%] Building C object lib/THC/CMakeFiles/THC.dir/THCGeneral.c.o
[ 69%] Building C object lib/THC/CMakeFiles/THC.dir/THCAllocator.c.o
[ 71%] Building C object lib/THC/CMakeFiles/THC.dir/THCStorage.c.o
[ 76%] Building C object lib/THC/CMakeFiles/THC.dir/THCTensorCopy.c.o
[ 76%] Building C object lib/THC/CMakeFiles/THC.dir/THCStorageCopy.c.o
[ 76%] Building C object lib/THC/CMakeFiles/THC.dir/THCTensor.c.o
/tmp/luarocks_cutorch-scm-1-3748/cutorch/lib/THC/THCGeneral.c:633:7: warning: absolute value function 'abs' given an
      argument of type 'long' but has parameter of type 'int' which may cause truncation of value [-Wabsolute-value]
  if (abs(state->heapDelta) < heapMaxDelta) { ^ /tmp/luarocks_cutorch-scm-1-3748/cutorch/lib/THC/THCGeneral.c:633:7: note: use function 'labs' instead if (abs(state->heapDelta) < heapMaxDelta) {
      ^~~
      labs
1 warning generated.
make[1]: *** [lib/THC/CMakeFiles/THC.dir/all] Error 2
make: *** [all] Error 2

Error: Build error: Failed building.

Since I just updated to OS X 10.11, I presume frameworks in 10.10 should be ok. So this hack should be ok as well, to make Accelerate.framework appear.

ln -s "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk"

Finally I try to install cutorch, with success.

luarocks install cutorch
luarocks install cunn #this installs fine as well, it didn't before

Everything should be good to go. But nothing works smooth.

jason@jmbp15-nvidia ~/w/neural-style (master)> 
th neural_style.lua -style_image IMG_2663.JPG -content_image IMG_2911.JPG 
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 574671192
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-5715/cutorch/lib/THC/generic/THCStorage.cu line=40 error=2 : out of memory
/Users/jason/torch/install/bin/luajit: /Users/jason/torch/install/share/lua/5.1/nn/utils.lua:11: cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-5715/cutorch/lib/THC/generic/THCStorage.cu:40
stack traceback:
	[C]: in function 'resize'
	/Users/jason/torch/install/share/lua/5.1/nn/utils.lua:11: in function 'torch_Storage_type'
	/Users/jason/torch/install/share/lua/5.1/nn/utils.lua:57: in function 'recursiveType'
	/Users/jason/torch/install/share/lua/5.1/nn/Module.lua:123: in function 'type'
	/Users/jason/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
	/Users/jason/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
	/Users/jason/torch/install/share/lua/5.1/nn/Module.lua:123: in function 'cuda'
	neural_style.lua:76: in function 'main'
	neural_style.lua:500: in main chunk
	[C]: in function 'dofile'
	...ason/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
	[C]: at 0x0109a0dd50

I guess I ran out of GPU memory? Seems to be an issue here https://github.com/jcjohnson/neural-style/issues/150. My pictures aren't that small, I guess. I'll just resize them.

jason@jmbp15-nvidia ~/w/neural-style (master)> ls -lh
total 5480
-rw-r--r--@ 1 jason  staff   2.4M 11 Mar 21:39 IMG_2663.JPG
-rw-r--r--@ 1 jason  staff   235K 11 Mar 21:38 IMG_2911.JPG
-rw-r--r--  1 jason  staff   9.1K 11 Mar 20:34 INSTALL.md
-rw-r--r--  1 jason  staff   1.1K 11 Mar 20:34 LICENSE
-rw-r--r--  1 jason  staff    16K 11 Mar 20:34 README.md
drwxr-xr-x  4 jason  staff   136B 11 Mar 20:34 examples
drwxr-xr-x  8 jason  staff   272B 12 Mar 06:24 models
-rw-r--r--  1 jason  staff    16K 11 Mar 20:34 neural_style.lua
jason@jmbp15-nvidia ~/w/neural-style (master)> sips IMG_2663.JPG -Z 680
/Users/jason/workspace/neural-style/IMG_2663.JPG
 [ (kCGColorSpaceDeviceRGB)] ( 0 0 0 1 )
  /Users/jason/workspace/neural-style/IMG_2663.JPG
jason@jmbp15-nvidia ~/w/neural-style (master)> sips IMG_2911.JPG  -Z 680
/Users/jason/workspace/neural-style/IMG_2911.JPG
 [ (kCGColorSpaceDeviceRGB)] ( 0 0 0 1 )
  /Users/jason/workspace/neural-style/IMG_2911.JPG
jason@jmbp15-nvidia ~/w/neural-style (master)> ls -lh
total 2224
-rw-r--r--  1 jason  staff   105K 12 Mar 06:27 IMG_2663.JPG
-rw-r--r--  1 jason  staff   64K 12 Mar 06:28 IMG_2911.JPG
-rw-r--r--  1 jason  staff   9.1K 11 Mar 20:34 INSTALL.md
-rw-r--r--  1 jason  staff   1.1K 11 Mar 20:34 LICENSE
-rw-r--r--  1 jason  staff    16K 11 Mar 20:34 README.md
drwxr-xr-x  4 jason  staff   136B 11 Mar 20:34 examples
drwxr-xr-x  8 jason  staff   272B 12 Mar 06:24 models
-rw-r--r--  1 jason  staff    16K 11 Mar 20:34 neural_style.lua
jason@jmbp15-nvidia ~/w/neural-style (master)>

Well resizing didn't work, still runs out of memory. Default `th` uses nn, so I will try using cudnn, but not working.

jason@jmbp15-nvidia ~/w/neural-style (master)> 
th neural_style.lua -style_image IMG_2663.JPG -content_image IMG_2911.JPG  -backend cudnn
nil	
/Users/jason/torch/install/bin/luajit: /Users/jason/torch/install/share/lua/5.1/trepl/init.lua:384: /Users/jason/torch/install/share/lua/5.1/trepl/init.lua:384: /Users/jason/torch/install/share/lua/5.1/cudnn/ffi.lua:1279: 'libcudnn (R4) not found in library path.
Please install CuDNN from https://developer.nvidia.com/cuDNN
Then make sure files named as libcudnn.so.4 or libcudnn.4.dylib are placed in your library load path (for example /usr/local/lib , or manually add a path to LD_LIBRARY_PATH)

stack traceback:
	[C]: in function 'error'
	/Users/jason/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
	neural_style.lua:64: in function 'main'
	neural_style.lua:500: in main chunk
	[C]: in function 'dofile'
	...ason/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
	[C]: at 0x0106769d50

So I go to https://developer.nvidia.com/rdp/cudnn-download and download cudnn-7.0-osx-x64-v4.0-prod.tgz and follow their install guide:

PREREQUISITES

    CUDA 7.0 and a GPU of compute capability 3.0 or higher are required.

ALL PLATFORMS

    Extract the cuDNN archive to a directory of your choice, referred to below as .
    Then follow the platform-specific instructions as follows.

LINUX

    cd 
    export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH

    Add  to your build and link process by adding -I to your compile
    line and -L -lcudnn to your link line.

OS X

    cd 
    export DYLD_LIBRARY_PATH=`pwd`:$DYLD_LIBRARY_PATH

    Add  to your build and link process by adding -I to your compile
    line and -L -lcudnn to your link line.

WINDOWS

    Add  to the PATH environment variable.

    In your Visual Studio project properties, add  to the Include Directories 
    and Library Directories lists and add cudnn.lib to Linker->Input->Additional Dependencies.

Opening the tgz file gives me cuda folder. I just need to add this my path

jason@jmbp15-nvidia ~/workspace> cd cuda/
jason@jmbp15-nvidia ~/w/cuda> set DYLD_LIBRARY_PATH (pwd) $DYLD_LIBRARY_PATH
jason@jmbp15-nvidia ~/w/cuda> echo $DYLD_LIBRARY_PATH
/Users/jason/workspace/cuda /Users/jason/torch/install/lib
jason@jmbp15-nvidia ~/w/cuda [127]> tree 
.
├── cd
├── include
│   └── cudnn.h
└── lib
    ├── libcudnn.4.dylib
    ├── libcudnn.dylib -> libcudnn.4.dylib
    └── libcudnn_static.a

2 directories, 5 files
jason@jmbp15-nvidia ~/w/cuda> set LD_LIBRARY_PATH (pwd)/lib/ $LD_LIBRARY_PATH
jason@jmbp15-nvidia ~/w/cuda> echo $LD_LIBRARY_PATH
/Users/jason/workspace/cuda/lib/ /Users/jason/torch/install/lib

Couchsurfing

Leave a reply

Recently I hosted two couchsurfers. It has been a very rewarding experience and I think opened me to meeting new people. Couchsurfing is a tool that solves the problem of not meeting enough interesting people. In our day to day grind, it is difficult to meet new friends.

This summarizes my initial experience with hosting couchsurfing guests.

I had two guests. Satoshi and Sophie. Both of these guests I selected for their entrepreneurial drive and could be considered seasoned travellers. I've had a couchsurfing account since 2012 and never done it before. I feared that it would be unsafe. The reason being if as a travelling couchsurfer, not being in your native country and on a time schedule, there is little that you can do if something goes wrong. Every legal due process would take extremely long. I decided to start as a host because I have the option to turn down guests and being in my familiar territory. I thought this would make a good transition into becoming a full fledged couchsurfer. Anyways, I selected these guests based on their profile. I don't just host anyone, but rather only people whom I see have a high chance of being friends. This reduces the safety risk and is a more efficient use of time than hosting random strangers.

Satoshi comes from Tokyo and travels a lot. Currently taking time off from work and school, he has 6 more months left of his undergraduate in business. He's been across a large number of situations such as being homeless. I admire his courage. He's travelled good parts of the world. He's also tried and failed to create a a startup. This tells me he's a safe choice to host and there'll be many commonalities we share. I'd love to make a friend in Tokyo to visit in the future. I learned a lot about Japanese culture and can't wait to actually go experience it one day.

Sophie comes from Shanghai and is taking this trip as a vacation. She is normally working full time in Shanghai and is a headhunter. She has a wonderful cheerful personality that I could best describe as 小燕子 in 还珠格格. Both in looks and personality. I admire her bravery even more than Satoshi because she travels as a single girl, and at her height, she's probably 1.56m. Here I am worrying about whether if it'd safe for me to couchsurf. Silly. I found her to be very driven and goal oriented personality. She has such a lively personality that would be of a great asset to any company. There are a lot of traits to be admired.

This is just the start of the couchsurfing experience for me. I'm sure there are many adventures that await me ahead and can't wait to spend more time meeting interesting people around the world!

Projects I want to work on

Leave a reply

Lately, Feb 16 2016, I have had a few projects in mind I want to dig deeper into. I just returned from a trip to China, for Chinese New Year.

Family Tree – Document the best to my knowledge who I am from the perspective of who my families were.
Violin – A friend of my Father gifted me a violin when I went on trip to China. I intend to start practicing this. Perhaps look into going to local Meetups.
Shipping Info - Elementary school friend of my Father runs a shipping company in China. He intends on doing IPO (Initial Public Offering). In order to make the company worth more, he desires to build a ecommerce website so it would be more competitive. I need to explore similar web platforms and provide some references.
Make Friends – Since moving to California full time, I have mostly spent time settling down and getting used to the flow. It is as if I was in first year university, going into an environment I'm not comfortable with. It is as if I was in Switzerland, too. Difference here is that there are relatively less networking events.

Computer Science Topics

Leave a reply

Background

I've been keeping track a list of interesting articles to read. Right now they are scattered across Google Docs, Pocket, bookmarks, and pdf files I've downloaded. I'll combine them all into this list, without any explicit ordering – although the topics on top should be more interesting. I'll also explain why I have them on my list.

List of projects I want to build

Software Architecture

Different GUI architectures.
- http://martinfowler.com/eaaDev/uiArchs.html
- Unlike school assignments, anytime one builds a customer facing product, the user interface needs to be considered. Building the UI is no trivial work, so knowing good design patterns is important. This article provides some fast overview.

Machine Learning

Deep Learning for Computer Vision
- https://getpocket.com/a/read/1094687772
- Wouldn't it be cool to feed your computer a video and have it learn about things that appears in it? You can train the computer to do this. GPU technology has advanced quite a bit since its invention during the 90s. It just so happens that neural networks are coming back into popularity. This article blogs about leveraging the two of the best platforms we have today: Matlab and Nvidia cuDNN (cuda Deep Neural Network).
Question Answering on a computer
- http://googleresearch.blogspot.ca/2015/11/computer-respond-to-this-email.html
- This article came around as I was working on a project with my professor for artificial intelligence. Our goal was to try and recommend previously given answers based on a similar sounding question.

How this list came to be

My usual conversations with friends borders somewhat about computer science topics. They are just interesting problems discuss. In particular, I have a few friends who I always strive to learn from, and I admire how they keep talking about these things. I must do my part and keep up.

Why I need this list

Computer science is a wide field, I've only opened the door and had a glimpse into the world. I have a bunch of interesting things I want to build, and I've been keeping them on the back burner for far too long. This list will motivate me by serving as a reminder of just how much of exploration is possible.