Deftly.net

Unlocking SSH FIDO keys on device connect.   OpenSSH YubiKey OpenBSD

• The problem
  

  As a lazy type, I often find it trying to type ssh-add -K over and over. I even felt depleted typing it here!

  Fortunately for me, OpenBSD makes it trivial to resolve this issue. All we need is:

  • hotplugd
  • pkill
  • Some shell foo.
• The adder
  

  This script will run our ssh-add -K command:

  #!/bin/sh
  
  trap 'ssh-add -K' USR1
  
  while true; do
  	sleep 1;
  done
  

  Notice the trap line there? More on that later! This script should be called via /usr/local/bin/fido & from ~/.xsession or similar. The important thing is that it runs after you log in.

• The watcher
  

  hotplugd (in OpenBSD base) does things when stuff happens. That's just what we need!

  This script (/etc/hotplugd/attach) will be called every time we attach a device:

  #!/bin/sh
  
  DEVCLASS=$1
  DEVNAME=$2
  
  case "$DEVNAME" in
  	fido0)
  		pkill -USR1 -xf "/bin/sh /usr/local/bin/fido"
  	;;
  esac
  

  Notice that pkill command with USR1? That's the magic that hits our trap line in the adder script!

  Now enable / start up hotplugd:

  # rcctl enable hotplugd
  # rcctl start hotplugd
  

• That's it!
  

  If you have all these bits in place, you should see `ssh-askpass` pop up when you connect a FIDO key to your machine!

  Here is a video of it in action:

  Thanks to kn@ for the USR1 suggestion! It really helped me be more lazy!

OpenSSH - Configuring FIDO2 Resident Keys   OpenSSH YubiKey

• The Setup
  

  If you haven't heard, OpenSSH recently ([2020-02-14 Fri]) gained support for FIDO2/U2F hardware authenticators like the YubiKey 5!

  This allows one to log into remote hosts with the touch of a button and it makes me feel like I am living in the future!

  Some of these hardware tokens even support multiple slots, allowing one to have multiple keys!

  On top of all that, the tokens can do "resident" and "non-resident" keys. "Resident" means that the key is effectively retrievable from the token (it doesn't actually get the key - it's a handle that lets one use the hardware key on the device).

  This got me thinking about how I could use a single token (with two keys) to access the various machines I use.

  In my use case, I have two types of machines I want to connect to:

  greater security

  machines I want to grant access to from a very select number of devices.

  The greater key will require me to copy the "key handle" to the machines I want to use it from.

  lesser security

  machines I want to access from devices that may not be as secure.

  The lesser key will be "resident" to the YubiKey. This means it can be downloaded from the YubiKey itself. Because of this, it should be trusted a bit less.

• Creating keys
  

  When creating FIDO keys (really they are key handles) one needs to explicitly tell the tool being used that it needs to pick the next slot. Otherwise generating the second key will clobber the first!

  • Generating the non-resident handle
    

    greater will require me to send the ~/.ssh/ed25519_sk_greater handle to the various hosts I want to use it from.

    We will be using ssh-keygen to create our resident key.

    ssh-keygen -t ed25519-sk -Oapplication=ssh:greater -f ~/.ssh/ed25519_sk_greater
    

  • Generating the resident handle
    

    Because resident keys allow for the handle to be downloaded from the token, I have changed the PIN on my token. The PIN is the only defense against a stolen key. Note: the PIN can be a full passphrase!

    Again via ssh-keygen.

    ssh-keygen -t ed25519-sk -Oresident -Oapplication=ssh:lesser -f ~/.ssh/ed25519_sk_lesser
    

• Using the token
  
  • Resident
    

    The resident key can be used by adding it to ssh-agent or by downloading the handle / public key using ssh-keygen:

  • Transient usage with ssh-add
    

    ssh-add -K
    

    This will prompt for the PIN (which should be set as it's the only defense against a stolen key!)

    No handle files will be placed on the machine you run this on. Handy for machines you want to ssh from but don't fully trust.

  • Permanent usage with ssh-agent
    

    ssh-keygen -K
    

    This will also prompt for the PIN, however, it will create the private key handle and corresponding public key and place them in $CWD.

  • Non-resident
    

    The non-resident key will only work from hosts that have the handle (in our case ~/.ssh/ed25519_sk_greater). As such, the handle must be copied to the machines you want to allow access from.

    Once the handle is in place, you can specify it's usage in ~/.ssh/config:

    Host secretsauce
        IdentityFile ~/.ssh/ed25519_sk_greater
    

Websockets with OpenBSD's relayd   OpenBSD relayd

• The need
  

  I am in the process of replacing all my NGINX instances with [httpd](https://man.openbsd.org/httpd)/[relayd](https://man.openbsd.org/relayd). So far this has been going pretty smoothly.

  I did, however, run into an issue with websockets in Safari on iOS and macOS which made me think they weren't working at all! Further testing proved they were working fine in other browsers, so .. more digging needs to be done!

• The configs
  

  I tested this in a VM running on OpenBSD. It's 'external' IP is 10.10.10.15.

  This config also works with TLS but for simplicity, this example will be plain text.

  • relayd.conf
    

    ext_addr="10.10.10.15"
    
    log connection errors
    
    table <websocketd> { 127.0.0.1 }
    
    http protocol ws {
    	match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
    	match request header append "X-Forwarded-By" \
    		value "$SERVER_ADDR:$SERVER_PORT"
    	match request header "Host" value "10.10.10.15" forward to <websocketd>
    
    	http websockets
    }
    
    relay ws {
    	listen on $ext_addr port 8000
    	protocol ws
    	forward to <websocketd> port 9999
    }
    

    Here we are setting up a "websocket" listener on port `8000` and forwarding it to port `9999` on `127.0.0.1` where we will be running [websocketd](http://websocketd.com/).

    The key directive is `http websockets` in the `http` block. Without this the proper headers won't be set and the connection will not work.

  • httpd.conf
    

    # $OpenBSD: httpd.conf,v 1.20 2018/06/13 15:08:24 reyk Exp $
    
    server "10.10.10.15" {
    	listen on * port 80
    	location "/*" {
    		directory auto index
    	}
    }
    

    Pretty simple. We are just going to serve the html file below.

  • /var/www/htdocs/index.html
    

    This html blurb simply creates a websocket and pumps the data it receives into a div that we can see.

    <!doctype html>
    <html>
    <head>
    <meta charset="utf-8">
    <title>ws test</title>
    </head>
    <body>
      <div id="output"></div>
    </body>
    <script>
    let ws = new WebSocket("ws://10.10.10.15:8000/weechat");
    let d = document.getElementById('output');
    ws.onopen = function() {
      ws.send("hi");
    };
    
    ws.onmessage = function (e) {
      d.innerText = d.innerText + " " + e.data;
    };
    
    ws.onclose = function() {
      d.innerText += (' done.');
    };
    </script>
    </html>
    

  • websocketd
    

    Now we use `websocketd` to serve up some sweet sweet websocket action!

    #!/bin/sh
    
    echo 'hi'
    
    for i in $(jot 5); do
    	echo $i;
    	sleep 1;
    done
    

    Use websocketd to run the above script:

    websocketd --port 9999 --address 127.0.0.1 ./above_script.sh
    

    Now point your browser at [http://10.10.10.15/](http://10.10.10.15/)! You will see "hi" and every second for five seconds you will see a count appended to <div id="output"></div>!

• The issues
  

  The error I saw on Safari on iOS and macOS is:

  'Connection' header value is not 'Upgrade'
  

  Which is strange, bucaese I can see that it is infact set to 'Upgrade' in a tcpdump.

Test packages for Node on OpenBSD   OpenBSD Node js

• Test builds for node
  

  In an attempt to get a wider range of testing for `lang/node`, I have decided to make test builds available here on [deftly.net](https://deftly.net/pub/OpenBSD).

  If you want to test node, but don't want to build the package yourself, you can simply run the following to upgrade your current install:

  ``` env PKG<sub>PATH</sub>=https://deftly.net/pub/OpenBSD/snapshots/packages/$(machine) pkg<sub>add</sub> -u node ```

  All the packages pushed here will be signed via [signify(1)](https://man.openbsd.org/signify).

  To make use of the signing, you will have to copy the below pub key to `/etc/signify/abieber-pkg.pub`

  ``` untrusted comment: signify public key RWRaFou/737fpovRq3OP3lZnz/97lbq2wYXFFk90nYUaW8xbc2HTd2xj ```

OpenBSD on the Lenovo A485   OpenBSD Hardware

• Hardware
  

   

  In typical Lenovo fashion, the A485 has three point eight gajillion configuration options one can select from during "Customization". You can even get it with a built in SmartCard™ reader!

  You can select SATA SSD or NVMe SSD, anywhere from 4GB to 32GB of Memory, AMD Ryzen™ 3, 5 or 7!

  Since this is going to be my main machine for as many years as I can possibly squeeze out of it, I opted for "all the things!"!

  The display options leave a little to be desired. Max resolution is 1920x1080. Not bad but not great either.

• Installing OpenBSD
  
  • Put install media into laptop.
  • Politely ask Henry on stand on the "enter" key.
  • \o/

  ## Support Summary

  OpenBSD-current as of 2018-10-15

  Component	Works?	Notes
  ---------	:–—:	-----
  Accelerated graphics	No	The Radeon™ Vega 6 integrated GPU isn't supported currently. X11 is usable via the [efifb](https://man.openbsd.org/efifb) driver which does an impressive job and displays things at the native resolution (1920x1080)! I am able to watch youtube videos without issue.
  AC adapter	Yes	USB-C connector. Has a satisfying click when connecting.. flops about wildly while connected. Attaches as [acpiac](https://man.openbsd.org/acpiac).
  Audio	Yes-ish	Audio works as expected though there is a known bug with [azalia](https://man.openbsd.org/azalia) on Ryzen that causes it to stop working after a period of time.
  Battery status	Yes	Status is visible in `apm` and `hw.sensors`. Attaches as [acpibat](https://man.openbsd.org/acpibat).
  Bluetooth	No	No support in OpenBSD. BT can be disabled in the bios.
  Cameras	No	When attempting to use the camera via `/dev/video`, the "activity" light turns on, but it seems the video device isn't compatible. Awesome side note, there is a built-in slider to cover the camera lens!
  Ethernet	Yes	Weirdly, there are two ethernet devices, `re0` and `re1`. `re1` is the physical ethernet port on the laptop. I can only assume that `re0` is the would-be-dock-ethernet?
  Hibernation	Maybe	This could be because I didn't create a swap partition that was 32GB.
  MicroSD slot	Yes	Inserted cards show up fine.
  Suspend/Resume	No	Technically the machine suspends / resumes, but X11 never comes back.
  Volume buttons	Yes	Via [acpithinkpad](https://man.openbsd.org/acpithinkpad).
  Wireless	No	The machine comes with a Realtek RTL8822BE, which isn't supported currently. I was able to replace this with an Intel Dual Band Wireless-AC 8265.

  Here are some photos of the built-in camera cover I mention in the Summary above.  

   

• dmesg
  

  OpenBSD 6.4 (GENERIC.MP) #0: Thu Oct 11 14:54:13 MDT 2018
      qbit@ren.tapenet.org:/sys/arch/amd64/compile/GENERIC.MP
  real mem = 33139396608 (31604MB)
  avail mem = 32125767680 (30637MB)
  mpath0 at root
  scsibus0 at mpath0: 256 targets
  mainbus0 at root
  bios0 at mainbus0: SMBIOS rev. 3.1 @ 0x68607000 (62 entries)
  bios0: vendor LENOVO version "R0WET34W (1.02 )" date 07/05/2018
  bios0: LENOVO 20MUCTO1WW
  acpi0 at bios0: rev 2
  acpi0: BGRT checksum error
  acpi0: sleep states S0 S3 S4 S5
  acpi0: tables DSDT FACP SSDT SSDT CRAT CDIT SSDT TPM2 UEFI MSDM BATB HPET APIC MCFG SBST VFCT IVRS FPDT SSDT SSDT SSDT UEFI SSDT BGRT
  acpi0: wakeup devices GPP0(S3) GPP1(S3) GPP2(S3) GPP3(S3) GPP4(S3) L850(S3) GPP5(S3) GPP6(S3) GP17(S3) XHC0(S3) XHC1(S3) GP18(S3) LID_(S3) SLPB(S3)
  acpitimer0 at acpi0: 3579545 Hz, 32 bits
  acpihpet0 at acpi0: 14318180 Hz
  acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
  cpu0 at mainbus0: apid 0 (boot processor)
  cpu0: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2196.29 MHz, 17-11-00
  cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu0: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu0: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu0: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu0: smt 0, core 0, package 0
  mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
  cpu0: apic clock running at 24MHz
  cpu0: mwait min=64, max=64, C-substates=1.1, IBE
  cpu1 at mainbus0: apid 1 (application processor)
  cpu1: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2195.85 MHz, 17-11-00
  cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu1: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu1: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu1: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu1: smt 1, core 0, package 0
  cpu2 at mainbus0: apid 2 (application processor)
  cpu2: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2195.85 MHz, 17-11-00
  cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu2: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu2: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu2: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu2: smt 0, core 1, package 0
  cpu3 at mainbus0: apid 3 (application processor)
  cpu3: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2195.85 MHz, 17-11-00
  cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu3: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu3: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu3: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu3: smt 1, core 1, package 0
  cpu4 at mainbus0: apid 4 (application processor)
  cpu4: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2195.85 MHz, 17-11-00
  cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu4: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu4: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu4: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu4: smt 0, core 2, package 0
  cpu5 at mainbus0: apid 5 (application processor)
  cpu5: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2195.85 MHz, 17-11-00
  cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu5: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu5: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu5: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu5: smt 1, core 2, package 0
  cpu6 at mainbus0: apid 6 (application processor)
  cpu6: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2195.85 MHz, 17-11-00
  cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu6: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu6: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu6: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu6: smt 0, core 3, package 0
  cpu7 at mainbus0: apid 7 (application processor)
  cpu7: AMD Ryzen 7 PRO 2700U w/ Radeon Vega Mobile Gfx, 2195.85 MHz, 17-11-00
  cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
  cpu7: 64KB 64b/line 4-way I-cache, 32KB 64b/line 8-way D-cache, 512KB 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
  cpu7: ITLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu7: DTLB 64 4KB entries fully associative, 64 4MB entries fully associative
  cpu7: smt 1, core 3, package 0
  ioapic0 at mainbus0: apid 32 pa 0xfec00000, version 21, 24 pins, can't remap
  ioapic1 at mainbus0: apid 33 pa 0xfec01000, version 21, 32 pins, can't remap
  acpimcfg0 at acpi0
  acpimcfg0: addr 0xf8000000, bus 0-63
  acpiprt0 at acpi0: bus 0 (PCI0)
  acpiprt1 at acpi0: bus 1 (GPP0)
  acpiprt2 at acpi0: bus 2 (GPP1)
  acpiprt3 at acpi0: bus 3 (GPP2)
  acpiprt4 at acpi0: bus 4 (GPP3)
  acpiprt5 at acpi0: bus -1 (GPP4)
  acpiprt6 at acpi0: bus 5 (GPP5)
  acpiprt7 at acpi0: bus -1 (GPP6)
  acpiprt8 at acpi0: bus 6 (GP17)
  acpiprt9 at acpi0: bus 7 (GP18)
  acpiec0 at acpi0
  acpicpu0 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpicpu1 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpicpu2 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpicpu3 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpicpu4 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpicpu5 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpicpu6 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpicpu7 at acpi0: C2(0@400 io@0x414), C1(0@1 mwait), PSS
  acpipwrres0 at acpi0: PUBS, resource for XHC0
  acpipwrres1 at acpi0: P0ST, resource for SATA
  acpipwrres2 at acpi0: P3ST, resource for SATA
  acpibtn0 at acpi0: PWRB
  acpicmos0 at acpi0
  acpibat0 at acpi0: BAT0 model "01AV489" serial  3225 type LiP oem "LGC"
  acpibat1 at acpi0: BAT1 model "01AV452" serial  1401 type LiP oem "SMP"
  acpiac0 at acpi0: AC unit online
  acpithinkpad0 at acpi0
  "SMB0001" at acpi0 not configured
  acpibtn1 at acpi0: LID_
  acpibtn2 at acpi0: SLPB
  "PNP0C14" at acpi0 not configured
  "PNP0C14" at acpi0 not configured
  "PNP0C14" at acpi0 not configured
  "STM7304" at acpi0 not configured
  "USBC000" at acpi0 not configured
  acpivideo0 at acpi0: VGA_
  cpu0: 2196 MHz: speeds: 2200 1700 1600 MHz
  pci0 at mainbus0 bus 0
  pchb0 at pci0 dev 0 function 0 vendor "AMD", unknown product 0x15d0 rev 0x00
  vendor "AMD", unknown product 0x15d1 (class system subclass IOMMU, rev 0x00) at pci0 dev 0 function 2 not configured
  pchb1 at pci0 dev 1 function 0 "AMD AMD64 17h PCIE" rev 0x00
  ppb0 at pci0 dev 1 function 1 vendor "AMD", unknown product 0x15d3 rev 0x00: msi
  pci1 at ppb0 bus 1
  rtsx0 at pci1 dev 0 function 0 "Realtek RTS522A Card Reader" rev 0x01: msi
  sdmmc0 at rtsx0: 4-bit, dma
  ppb1 at pci0 dev 1 function 2 vendor "AMD", unknown product 0x15d3 rev 0x00: msi
  pci2 at ppb1 bus 2
  iwm0 at pci2 dev 0 function 0 "Intel Dual Band Wireless-AC 8265" rev 0x78, msi
  ppb2 at pci0 dev 1 function 3 vendor "AMD", unknown product 0x15d3 rev 0x00: msi
  pci3 at ppb2 bus 3
  nvme0 at pci3 dev 0 function 0 "Samsung SM981/PM981 NVMe" rev 0x00: msi, NVMe 1.2
  nvme0: SAMSUNG MZVLB512HAJQ-000L7, firmware 4L2QEXA7, serial S3TNNX0K825659
  scsibus1 at nvme0: 1 targets
  sd0 at scsibus1 targ 0 lun 0: <NVMe, SAMSUNG MZVLB512, 4L2Q> SCSI4 0/direct fixed
  sd0: 488386MB, 512 bytes/sector, 1000215216 sectors
  ppb3 at pci0 dev 1 function 4 vendor "AMD", unknown product 0x15d3 rev 0x00: msi
  pci4 at ppb3 bus 4
  re0 at pci4 dev 0 function 0 "Realtek 8168" rev 0x0e: RTL8168EP/8111EP (0x5000), msi, address 8c:16:45:c9:32:ae
  rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
  vendor "Realtek", unknown product 0x816a (class communications subclass serial, rev 0x0e) at pci4 dev 0 function 1 not configured
  vendor "Realtek", unknown product 0x816b (class communications subclass serial, rev 0x0e) at pci4 dev 0 function 2 not configured
  vendor "Realtek", unknown product 0x816c (class serial bus subclass IPMI, rev 0x0e) at pci4 dev 0 function 3 not configured
  ehci0 at pci4 dev 0 function 4 vendor "Realtek", unknown product 0x816d rev 0x0e: apic 33 int 15
  ehci0: pre-2.0 USB rev
  ppb4 at pci0 dev 1 function 6 vendor "AMD", unknown product 0x15d3 rev 0x00: msi
  pci5 at ppb4 bus 5
  re1 at pci5 dev 0 function 0 "Realtek 8168" rev 0x10: RTL8168GU/8111GU (0x5080), msi, address 8c:16:45:c9:32:ad
  rgephy1 at re1 phy 7: RTL8251 PHY, rev. 0
  pchb2 at pci0 dev 8 function 0 "AMD AMD64 17h PCIE" rev 0x00
  ppb5 at pci0 dev 8 function 1 vendor "AMD", unknown product 0x15db rev 0x00
  pci6 at ppb5 bus 6
  vendor "ATI", unknown product 0x15dd (class display subclass VGA, rev 0xd0) at pci6 dev 0 function 0 not configured
  azalia0 at pci6 dev 0 function 1 vendor "ATI", unknown product 0x15de rev 0x00: msi
  azalia0: no supported codecs
  vendor "AMD", unknown product 0x15df (class crypto subclass miscellaneous, rev 0x00) at pci6 dev 0 function 2 not configured
  xhci0 at pci6 dev 0 function 3 vendor "AMD", unknown product 0x15e0 rev 0x00: msi, xHCI 1.16
  usb0 at xhci0: USB revision 3.0
  uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
  xhci1 at pci6 dev 0 function 4 vendor "AMD", unknown product 0x15e1 rev 0x00: msi, xHCI 1.16
  usb1 at xhci1: USB revision 3.0
  uhub1 at usb1 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
  azalia1 at pci6 dev 0 function 6 "AMD Raven Ridge HD Audio" rev 0x00: apic 33 int 30
  azalia1: codecs: Realtek/0x0257
  audio0 at azalia1
  ppb6 at pci0 dev 8 function 2 vendor "AMD", unknown product 0x15dc rev 0x00
  pci7 at ppb6 bus 7
  ahci0 at pci7 dev 0 function 0 "AMD Carrizo AHCI" rev 0x61: msi, AHCI 1.3.1
  scsibus2 at ahci0: 32 targets
  "AMD Carrizo SMBus" rev 0x61 at pci0 dev 20 function 0 not configured
  pcib0 at pci0 dev 20 function 3 "AMD Carrizo LPC" rev 0x51
  pchb3 at pci0 dev 24 function 0 vendor "AMD", unknown product 0x15e8 rev 0x00
  pchb4 at pci0 dev 24 function 1 vendor "AMD", unknown product 0x15e9 rev 0x00
  pchb5 at pci0 dev 24 function 2 vendor "AMD", unknown product 0x15ea rev 0x00
  pchb6 at pci0 dev 24 function 3 vendor "AMD", unknown product 0x15eb rev 0x00
  pchb7 at pci0 dev 24 function 4 vendor "AMD", unknown product 0x15ec rev 0x00
  pchb8 at pci0 dev 24 function 5 vendor "AMD", unknown product 0x15ed rev 0x00
  pchb9 at pci0 dev 24 function 6 vendor "AMD", unknown product 0x15ee rev 0x00
  pchb10 at pci0 dev 24 function 7 vendor "AMD", unknown product 0x15ef rev 0x00
  isa0 at pcib0
  isadma0 at isa0
  pckbc0 at isa0 port 0x60/5 irq 1 irq 12
  pckbd0 at pckbc0 (kbd slot)
  wskbd0 at pckbd0: console keyboard
  pms0 at pckbc0 (aux slot)
  wsmouse0 at pms0 mux 0
  wsmouse1 at pms0 mux 0
  pms0: Synaptics clickpad, firmware 8.16, 0x1e2b1 0x940300
  pcppi0 at isa0 port 0x61
  spkr0 at pcppi0
  vmm0 at mainbus0: SVM/RVI
  efifb0 at mainbus0: 1920x1080, 32bpp
  wsdisplay0 at efifb0 mux 1: console (std, vt100 emulation), using wskbd0
  wsdisplay0: screen 1-5 added (std, vt100 emulation)
  uhub2 at uhub1 port 1 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/60.52 addr 2
  ugen0 at uhub2 port 1 "vendor 0x06cb product 0x009a" rev 2.00/1.64 addr 3
  ugen1 at uhub2 port 3 "Generic EMV Smartcard Reader" rev 2.01/1.20 addr 4
  uhub3 at uhub1 port 2 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/60.52 addr 5
  uvideo0 at uhub3 port 3 configuration 1 interface 0 "Chicony Electronics Co.,Ltd. Integrated Camera" rev 2.01/0.27 addr 6
  video0 at uvideo0
  vscsi0 at root
  scsibus3 at vscsi0: 256 targets
  softraid0 at root
  scsibus4 at softraid0: 256 targets
  sd1 at scsibus4 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
  sd1: 488385MB, 512 bytes/sector, 1000213601 sectors
  

Passing off the Complexity   OpenBSD Passwords

• The Password Dilemma
  

  For quite some time I was a user of KeePass. Life was great I had a client that worked on every platform (OpenBSD via `mono`, Android and iOS via 3rd party apps). Every now and then I would manually copy the file from my "main" machine to all the others.

  After a while the syncing process became tedious. I attempted to find things that would aid in the process.. Tools like `syncthing`, `unison`, dropbox.. etc. They all had issues. With `syncthing` the synchronizing was unreliable. Most of the time it couldn't find other hosts on my network.. and when it finally did I would end up needing to tell it how to resolve conflicts. Dropbox didn't have a native client for OpenBSD - not to mention putting your password database online is a terrifying proposition (my main issue with LastPass)!

  Out of all the management methods I tried - LastPass was the most "complete":

  Name	Syncing	Browser	OpenBSD	iOS/Android	My Trust
  ----	----	----	----	----	----
  LastPass	✓	✓	✓	✓	✗
  KeePass	✗	✗	✓	✓	✓
  Password-Gorilla	✗	✗	✓	✓	✓

• Enter pass
  

  `pass` is touted as "the standard Unix password manager". It works by keeping your passwords in individual PGP encrypted files. For example, the entry `Web/google.com` would hold my "google.com" password. An entry can contain more than just your passwords (I have `username: ….` set on the 2nd line.), the first line is always expected to be the password.

  Out of the box, pass supports syncing via git, multiple recipients (meaning it can encrypt a password file for multiple people), random password generation and copying to clipboard with auto-clear!

  It doesn't, however, work out of the box on iOS/Android or my browser. Fortunately for me - pass has a fantastic ecosystem with a multitude of 3rd party browser extensions, iOS/Android apps.. best all, they are open source! There are even full replacements for pass itself! One such replacements is [gopass](https://github.com/justwatchcom/gopass).

  `gopass` extends pass a bit further, it's written in Go, seamlessly supports multiple "stores" (allowing me to have an extra level of privledge separation for sensitive passwords), is pledge()'d, AND can run on every OS I use!

  For browsers, I have settled on [browserpass](https://github.com/dannyvankooten/browserpass). It's also written in Go, pledge()'d, works on all my systems* (OpenBSD, Windows, macos), has auto-fill and easy to grok source code!

  For iOS/Android there are [passforios](https://github.com/mssun/passforios) and [Android-Password-Store](https://github.com/zeapo/Android-Password-Store). All of which support syncing via git!

  Couple all this together and I get:

  Name	Syncing	Browser	OpenBSD	iOS/Android	My Trust
  gopass/browserpass/mobile app	✓	✓	✓	✓	✓

• A few hurdles
  
  • Key management
    

    PGP key management was an initial hurdle for me. Having to copy keys from one computer to another put me back in the same situation I had with KeePass.. syncing.

    My solution to this problem is to store my PGP keys on a SmartCard (Yubikey 4 in this case). This lets me "transfer" my PGP key between my main computers without issue. It also has the added advantage of giving me an ssh key I can use on less trusted machines!

    For me, the Yubikey is ideal because of the form factor, however, there are other alternatives such as NitroKey.

  • Remote access
    

    Because pass and friends sync by way of git, you must expose your git repo to be able to use the sync mechanism. I didn't want to expose my main repo to the world (even though it is only available via ssh), so it is only accessible via my home network.

    I have an ssh key per device, and use the `command=` option in `~/.ssh/authorized<sub>keys</sub>` to force git-only ssh access to less trusted devices.

    For example, my phone is less trusted than my OpenBSD laptop, so I have an entry like this:

    ``` command="git-shell -c \"$SSH<sub>ORIGINALCOMMAND</sub>\"",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa …….. dumbdevice ```

    This limits the device `dumbdevice` to only executing `git-shell`!

• Conclusion
  

  All things considered, this setup takes a little bit to configure initially (with keys and git repos.. etc), but its advantages out weigh that hurdle for me:

  1. It's very simple. No databases, no extra exposed services.. etc.
  2. It uses existing tools to manage / sync my passwords. I trust ssh way

  more than I trust a web server!

  1. The ecosystem is very active.
  2. If need be, I can operate entirely without the "pass" tools via

  something like: `gpg -d ~/.password-store/Web/google.com | head -n1 | xclip`

  \* Currently there is a bug in Firefox on OpenBSD that prevents native extensions from functioning.. I hope to get this fixed soon! Chromium works fine, however.

Using cabal on OpenBSD   OpenBSD Software Haskell

Measuring the weight of an electron   OpenBSD Electron Software

• Getting the tools
  

  At first glance Electron seems like a pretty solid app, it has decent [docs](https://github.com/electron/electron/tree/master/docs), it's consolidated in a single repository, has a lot of [visibility](https://github.com/electron/electron/stargazers), porting it shouldn't be a big deal, right?

  First things first, clone that repo!

  ``` git clone git@github.com:electron/electron.git ```

  If you want to follow along, we will be using the [build instructions for linux](https://github.com/electron/electron/blob/master/docs/development/build-instructions-linux.md) doc.

  Reading through the doc, right off the bat there are a few interesting things:

  At least 25GB disk space

  Huh, OK, some how this ~47M repository is going to blow up to 25G? I glance at Henry, he gives me the "what?" look. We carry on.

  Clang 3.4 or later.

  This one isn't odd until we have more context. More on this one later.

  Continuing along with the build, I know I have two versions of `clang` installed on OpenBSD, one from ports and one in base. Hopefully I will be able to tell the build to use one of these versions.

  Indeed Electron has that [ability](https://github.com/electron/electron/blob/master/docs/development/build-instructions-linux.md#using-system-clang-instead-of-downloaded-clang-binaries)! Their example is even using the same prefix OpenBSD's clang port!

  So, we run the bootstrap:

  ``` ./script/bootstrap.py -v –clang<sub>dir</sub> /usr/local Traceback (most recent call last): File "./script/bootstrap.py", line 10, in <module> from lib.config import BASE<sub>URL</sub>, PLATFORM, enable<sub>verbosemode</sub>, \ File "/home/qbit/dev/electron<sub>wut</sub>/script/lib/config.py", line 17, in <module> }[sys.platform] KeyError: 'openbsd6' ```

  Dang. Looks like we need to tell bootstrap about OpenBSD. Easy enough:

  ``` diff –git a/script/lib/config.py b/script/lib/config.py index 58f467b5b..646af08f7 100644 — a/script/lib/config.py + b/script/lib/config.py @@ -14,6 +14,7 @@ PLATFORM = { 'darwin': 'darwin', 'linux2': 'linux', 'win32': 'win32',

  • 'openbsd6': 'openbsd', }[sys.platform]

    verbose<sub>mode</sub> = False ```

  We re-run the bootstrap, things seem to be going well.. Then the Henry squeaks: "whoa!!":

  ``` Synchronizing submodule url for 'vendor/requests' git submodule update –init –recursive Cloning into '/home/qbit/dev/electron<sub>wut</sub>/vendor/boto'… error: object c1eddff4ee3f62b6039f1083651b9118883e7f07: badTimezone: invalid author/committer line - bad time zone fatal: Error in object fatal: index-pack failed fatal: clone of 'https://github.com/boto/boto.git' into submodule path '/home/qbit/dev/electron<sub>wut</sub>/vendor/boto' failed Failed to clone 'vendor/boto'. Retry scheduled Cloning into '/home/qbit/dev/electron<sub>wut</sub>/vendor/breakpad'…

  ```

  We just failed to clone the `boto` repo, but the build is still going.. does this mean it was an optional dependency and isn't needed for the build?

  Henry doesn't look happy, none the less, he assures me it's OK to go on. What a trooper!

  ``` Cloning into '/home/qbit/dev/electron<sub>wut</sub>/vendor/requests'… error: object 5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8: badTimezone: invalid author/committer line - bad time zone fatal: Error in object fatal: index-pack failed fatal: clone of 'https://github.com/kennethreitz/requests' into submodule path '/home/qbit/dev/electron<sub>wut</sub>/vendor/requests' failed Failed to clone 'vendor/requests'. Retry scheduled Cloning into '/home/qbit/dev/electron<sub>wut</sub>/vendor/boto'… error: object c1eddff4ee3f62b6039f1083651b9118883e7f07: badTimezone: invalid author/committer line - bad time zone fatal: Error in object fatal: index-pack failed fatal: clone of 'https://github.com/boto/boto.git' into submodule path '/home/qbit/dev/electron<sub>wut</sub>/vendor/boto' failed Failed to clone 'vendor/boto' a second time, aborting ``` Wait. Another repository failed to clone? At least this time the build failed after trying to clone `boto`.. again. I am guessing it tried twice because something might have changed between now and the last clone? Off in the distance we catch a familiar tune, it almost sounds like Gnarls Barkley's song Crazy, can't tell for sure.

  As it turns out, if you are using [git-fsck](https://git-scm.com/docs/git-fsck), you are unable to clone [boto](https://github.com/boto/boto/issues/3507) and [requests](https://github.com/requests/requests/issues/3805). Obviously the proper fix for his is to not care about the validity of the git objects!

  So we die a little inside and comment out `fsckobjects` in our `~/.gitconfig`.

  I look at Henry, he assures me it's safe to go on.

  We re-run bootstrap… thousands of lines fly past.. "`npm verb… something something`". I can only think npm is puking this info for it's on benefit. It definitely isn't for ours!

  Bah, another error:

  ``` subprocess.CalledProcessError: Command '['/usr/local/bin/python', '/home/qbit/dev/electron<sub>wut</sub>/vendor/libchromiumcontent/script/download', '-s', '-f', '-c', '94c58176db175d72d88621afe8223b4175eecba5', '–target<sub>arch</sub>', 'x64', 'https://s3.amazonaws.com/github-janky-artifacts/libchromiumcontent', '/home/qbit/dev/electron<sub>wut</sub>/vendor/download/libchromiumcontent']' returned non-zero exit status 1 ```

  Looks like it's trying to download a pre-built `libchromiumcontent`, we reference the doc again, finding the `–build<sub>libchromiumcontent</sub>` option. Re-run!

  This time we are faced with `.`'s, I have no idea what is happening:

  ``` /usr/local/bin/python /home/qbit/dev/electron<sub>wut</sub>/vendor/libchromiumcontent/script/update -t x64 –defines make<sub>clangdir</sub>=/usr/local clang<sub>usechromeplugins</sub>=0 ……. ```

  Looking in `top`, we can see a python process with a WAIT of `netio`, maybe it's downloading something? Looking in the `electron<sub>wut</sub>` directory reveals a growing file named `chromium-58.0.3029.110.tar.xz`. We let it finish downloading.

  Out of curiosity we look at `vendor/libchromiumcontent/script/update`, it seems its purpose is to download / extract chromium clang and node, good thing we already specified `–clang<sub>dir</sub>` or it might try to build clang again!

  544 dots and 45 minutes later, we have an error! The `chromium-58.0.3029.110.tar.xz` file is mysteriously not there anymore.. Interesting.

  Scrolling up in the terminal points us to something disheartening:

  ``` Extracting… Updating Clang to 296320-1… Creating directory /home/qbit/dev/electron<sub>wut</sub>/vendor/libchromiumcontent/src/third<sub>party</sub>/llvm-build Traceback (most recent call last): File "/home/qbit/dev/electron<sub>wut</sub>/vendor/libchromiumcontent/src/tools/clang/scripts/update.py", line 902, in <module> sys.exit(main()) File "/home/qbit/dev/electron<sub>wut</sub>/vendor/libchromiumcontent/src/tools/clang/scripts/update.py", line 898, in main return UpdateClang(args) File "/home/qbit/dev/electron<sub>wut</sub>/vendor/libchromiumcontent/src/tools/clang/scripts/update.py", line 420, in UpdateClangi assert sys.platform.startswith('linux') AssertionError None ```

  Wut. "Updating Clang…". Didn't I explicitly say not to build clang?

  At this point we have to shift projects, no longer are we working on Electron.. It's `libchromiumcontent` that needs our attention.

• Fixing sub-tools
  

  ``` git clone git@github.com:electron/libchromiumcontent.git ```

  Following the [instructions](https://github.com/electron/libchromiumcontent) on this repo, we run `script/bootstrap`.. it seems to complete without issue!

  On to the next steps:

  ``` script/update -t x64 ```

  Ahh, our old friends the dots! This is the second time waiting 45+ minutes for a 500+ MB file to download. We are fairly confident it will fail, delete the file out from under itself and hinder the process even further, so we add an explicit exit to the `update` script. This way we can copy the file somewhere safe!

  ``` diff –git a/script/update b/script/update index 234e4b3..b2639bf 100755 — a/script/update + b/script/update @@ -107,6 +107,7 @@ def download<sub>sourcetarball</sub>(version): sys.stderr.flush() t.write(chunk)

  • sys.exit() sys.stderr.write('\nExtracting…\n') sys.stderr.flush()

  ```

  544 dots and 43 minutes later…. `chromium-58.0.3029.110.tar.xz` is safe!

  Fool me once…

  ``` mkdir safe<sub>space</sub> cp chromium-58.0.3029.110.tar.xz safe<sub>space</sub>/ ```

  We remove the `sys.exit()` and re-run! Wut.. dots again!? Lets look deeper into this `update` script:

  ``` if not args.no<sub>download</sub>: version = chromium<sub>version</sub>() if not is<sub>sourcetarballupdated</sub>(version): download<sub>sourcetarball</sub>(version) else: print "Skipping Chromium Source Tarball Download"

  ``` Ok, lets try that.. We copy the tar.xz out of its safe<sub>space</sub>…

  ``` Skipping Chromium Source Tarball Download Traceback (most recent call last): File "/home/qbit/dev/libchromiumcontent/vendor/python-patch/patch.py", line 1136, in <module> patch.apply(options.strip, root=options.directory) or sys.exit(-1) File "/home/qbit/dev/libchromiumcontent/vendor/python-patch/patch.py", line 778, in apply os.chdir(root) OSError: [Errno 2] No such file or directory: 'home/qbit/dev/libchromiumcontent/src.' ```

  Sigh. The above (or similar) was repeated about 50 times… The trend here seems to be: Ignore errors! They are stupid and meaningless anyway!

  Since `def download<sub>sourcetarball</sub>` should actually be `def download<sub>sourcetarballthenextract</sub>`, we do that part for it… and pat ourselves on the back for having a `safe<sub>space</sub>`!

  As chromium extracts, Henry and I can't shake the feeling that everything until now was just the tip of the iceberg

  We remove the call to `update<sub>clang</sub>`, because.. well.. we have two copies of it already and the Electron doc said everything would be fine if we had >= clang 3.4!

  Re-run..

  ``` already patched third<sub>party</sub>/WebKit/Source/core/paint/ThemePainterMac.mm already patched third<sub>party</sub>/WebKit/Source/platform/mac/KillRingMac.mm qbit@slip[1]:libchromiumcontent[master *%=]λ ```

  That `[1]` in my PS1 means that the update script exited with the return code `1`… but there is no indication of why..

  Henry's lovely yellow plumage seems to be becoming a darker shade of yellow.. How much more of this can we take?!

  If we have learned anything so far, it has to be "errors don't matter!". This one, however, warrants further investigation!

  We dig deeper into script/update

  `update<sub>gn</sub>()`.. pulls down a binary `gn`.. which, interestingly, can be generated with the code we have right below our feet… but for some reason, they have this component already built. There is no pre-built version for OpenBSD.

  At this point, Henry and I are getting pretty irritated.. it's time to bring in some big guns! We are going to leverage the countless hours of work that have already been put into properly building these components! (novel, right?!)

  We quickly move to `/usr/ports/www/chromium`, low and behold, it's the exact version that `libchromiumcontent` is trying to build! We review the `Makefile` to find this gem: `2. bootstrap gn, the tool to generate ninja files`

  Running `make configure` quickly gets us a usable `gn` binary, we make the appropriate directories under src/buildtools, copy `gn` in, modify our `script/update` file:

  ``` diff –git a/script/update b/script/update index 234e4b3..b5c4afc 100755 — a/script/update + b/script/update @@ -52,9 +52,9 @@ def main():

  return (apply<sub>patches</sub>() or copy<sub>chromiumcontentfiles</sub>() or

  • update<sub>clang</sub>() or
  • update<sub>gn</sub>() or
  • update<sub>node</sub>() or
  • #update<sub>clang</sub>() or
  • #update<sub>gn</sub>() or
  • #update<sub>node</sub>() or run<sub>gn</sub>(target<sub>arch</sub>, args.defines))

  @@ -248,6 +248,8 @@ def run<sub>gn</sub>(target<sub>arch</sub>, defines): gn = os.path.join(SRC<sub>DIR</sub>, 'buildtools', 'linux64', 'gn') elif sys.platform == 'darwin': gn = os.path.join(SRC<sub>DIR</sub>, 'buildtools', 'mac', 'gn')

  • elif sys.platform == 'openbsd6':

  • gn = os.path.join(SRC<sub>DIR</sub>, 'buildtools', 'openbsd', 'gn')

    env = os.environ.copy() if sys.platform in ['win32', 'cygwin']:

  ```

  Re-run!

  ``` ERROR at //build/config/sysroot.gni:95:5: Assertion failed. assert( ^–— Missing sysroot (//build/linux/debian<sub>wheezyamd64</sub>-sysroot). To fix, run: build/linux/sysroot<sub>scripts</sub>/install-sysroot.py –arch=amd64 See //build/config/sysroot.gni:96:9: exec<sub>script</sub>("//build/dir<sub>exists.py</sub>", ^------------------------------–— ```

  Wheezy?! Where is that getting set?! ***We stop and ponder.. how the hell did we get here? What could have possibly warranted abandoning makefiles and shell scripts in favor of this monstrosity!?***

  Just for fun.. lets try to run the second step (after all, the first step only produced a `1`, right!?)

  ``` script/build -t x64 ```

  FML:

  ``` qbit@slip[1]:libchromiumcontent[master *%=]λ script/build -t x64 Unsupported OS OpenBSD No prebuilt ninja binary was found for this system. Try building your own binary by doing: cd ~ git clone https://github.com/martine/ninja.git -b v1.7.2 cd ninja && ./configure.py –bootstrap Then add ~/ninja/ to your PATH. Traceback (most recent call last): File "script/build", line 57, in <module> sys.exit(main()) File "script/build", line 43, in main subprocess.check<sub>call</sub>([NINJA, '-C', os.path.relpath(out<sub>dir</sub>), target], env=env) File "/usr/local/lib/python2.7/subprocess.py", line 186, in check<sub>call</sub> raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/home/qbit/dev/libchromiumcontent/vendor/depot<sub>tools</sub>/ninja', '-C', 'src/out-x64/static<sub>library</sub>', 'chromiumcontent:chromiumcontent']' returned non-zero exit status 1 qbit@slip[1]:libchromiumcontent[master *%=]λ which ninja /usr/local/bin/ninja qbit@slip[0]:libchromiumcontent[master *%=]λ ```

  Clearly we are dealing with a beast that is too smart for its own good.

• Fixing sub-sub-tools
  

  Since `depot<sub>tools</sub>` is a Google project, it's easier to edit the files in the `vendor` directory and pretend nothing ever happened.

  ``` diff –git a/ninja b/ninja index 282cc276..e22cbb9a 100755 — a/ninja + b/ninja @@ -37,7 +37,5 @@ case "\(OS" in Darwin) exec "\){THIS<sub>DIR</sub>}/ninja-mac" "$@";; CYGWIN*) exec cmd.exe /c $(cygpath -t windows \(0).exe "\)@";; MINGW*) cmd.exe //c \(0.exe "\)@";;

  • *) echo "Unsupported OS ${OS}"
  • print<sub>help</sub>
  • exit 1;;
  • *) exec "/usr/local/bin/ninja" "$@";; esac

  ```

  Sigh. So many assumptions, lets continue the trend!

  ``` cd ../../ ```

  Re-run `script/build -t x64`…

  No luck. At this point we are faced with a complex web of python scripts that execute `gn` on GN files to produce ninja files… which then build the various components and somewhere in that cluster, something doesn't know about OpenBSD…

  ***I look at Henry, he is looking a photo of his wife and kids. They are sitting on a telephone wire, the morning sun illuminating their beautiful faces. Henry looks back at me and says "It's not worth it."*

  We slam the laptop shut and go outside.

Tab completion in OpenBSD's ksh   OpenBSD ksh

• OpenBSD's ksh
  

  ..is the little shell that could. In ~20k lines of code, it has many of the same features as more popular shells like zsh and bash. Plus it has the added bonus of being [pledge(2)'d](http://man.openbsd.org/pledge)!

  One of the features OpenBSD's ksh shares with its more popular friends is user definable completions! Something that sets it apart, however, is the simplicity of these completions. From the man page:

  ``` Custom completions may be configured by creating an array named ‘complete<sub>command</sub>’, optionally suffixed with an argument number to complete only for a single argument. ``` For example, here is a completion for [vmctl(8)](http://man.openbsd.org/vmctl) with expansion of defined VM names:

  ```

  set -A complete<sub>vmctl</sub> – console load reload start stop reset status set -A complete<sub>vmctl2</sub> – $(vmctl status | awk '!/NAME/{print $NF}') ```

  Hats off to Nicholas Marriott nicm@ for implementing it and to brynet@ for pointing it out to me! I was close to switching to [fish](https://fishshell.org) in an attempt to save my wrists.. but the ~180k lines of extra (yeah, on top of the 20k!) code in fish was kinda scaring me!

  For anyone interested, here is a quick comparison of CLOC for a few popular shells:

  Shell	Version	Total Lines (cloc)
  :-----	:--------	----–—:
  ksh	Mon May 1 07:17:54 UTC 2017	19680
  zsh	5.3.1	127246
  fish	2.5.0	207597
  bash	4.4	351043

  I understand LOC isn't a good measure of quality, but it sure does mean you have a lot more reading to find that quality!

  A full list of my completions can be found [here](https://github.com/qbit/dotfiles/blob/master/common/dot_ksh_completions)!

SSH Fingerprint Verification via Tor   SSH Tor

• The Problem
  

  OpenSSH (really, are there any other implementations?) requires [Trust on First Use](https://en.wikipedia.org/wiki/Trust_on_first_use) for fingerprint verification.

  Verification can be especially problematic when using remote services like VPS or colocation.

  How can you trust that the initial connection isn't being [Man In The Middle'd](https://en.wikipedia.org/wiki/Man-in-the-middle_attack)?

• My Solution
  

  .. for remote hosts, is to use Tor as supplemental verification. Fortunately OpenSSH makes this very easy as connections can be proxied (`ProxyCommand`) via arbitrary commands (`socat` in this case).

  #!/bin/sh
  
  # To make use of this, you need:
  # - Tor installed / running
  # - socat installed
  # - Line 1 of your ~/.ssh/config should have: 'Include ~/.ssh/torify'
  
  if [ $# -lt 1 ];then
  	echo "Please specify hostname to check!"
  	exit 1;
  fi
  
  TFILE=~/.ssh/torify
  HOST=$1
  
  CONF=$(cat <<'EOF'
  Host *
  	ProxyCommand socat STDIO SOCKS4A:localhost:%h:%p,socksport=9050
  EOF
  );
  
  echo "$CONF" > "${TFILE}"
  IP=$(tor-resolve "${HOST}")
  for i in 1 2 3 4 5; do
  	ssh "${IP}" & sleep 3; kill $!
  done
  
  echo "" > "${TFILE}"
  ssh "$HOST" & sleep 3; kill $!
  

  Latest version of this script can be pulled from [here](https://github.com/qbit/dotfiles/blob/master/bin/verify_ssh_fp)

  The above script makes five cut-short ssh connections (waiting 3 seconds before cutting the connection by killing the ssh pid) to an IP address that is resolved using Tor. It then makes a single non-Tor'd cut-short connection to print the fingerprint as seen from your default outbound connection.

  If all six of the output fingerprints match, it's a bit more safe to assume that your connection to the remote host isn't being tampered with!

  Obviously, this solution isn't 100%. Your Tor connection could be compromised.. Snakes could be on planes… etc. So use it at your own risk.

Why I Run OpenBSD   OpenBSD Linux

• UPI
  

  One thing I learned during my travels between OSs: consistency is everything.

  Most operating systems seem to, at least, keep a consistent interface between themselves and binaries / applications. They do this by keeping consistent APIs (Application Programming Interfaces) and ABIs (Application Binary Interfaces). If you take a binary from a really old version of Linux and run or build it on a brand-spanking new install of Linux, it will likely Just Work™. This is great for applications and developers of applications. Vendors can build binaries for distribution and worry less about their product working when it gets out in the wild (sure this binary built in 2016 will run on RedHat AS2.1!!).

  With all this catering to applications and developers, one would think that a similar level of attention would be applied to the users of the applications and systems: User Program Interfaces (UPI) as I like to call them!

  A good example of a poor UPI is (was?) `ifconfig(8)` on Linux. From a user's perspective, `ifconfig`, a command to "configure network interface parameters" should work on… well, network interfaces! This includes wireless, hard-wired, cell based… etc.

  On Linux, however, this is no longer the case (at least for some devices).

  This inconsistency seems to have come to be when Linux started getting wireless support. For some reason someone (vendors, maybe?) decided that `ifconfig` wasn't a good place to let users interact with their wireless device. Maybe they felt their device was special? Maybe there were technical reasons? The bottom line is, someone decided to create a new utility to manage a wireless device… and then another one came along… pretty soon there was `iwconfig(8)`, `iw(8)`, `ifconfig(8)`, some funky thing that let windows drivers interface with Linux.. and one called `ip(8)` I am sure there are others I am forgetting, but I prefer to forget. I have moved onto greener pastures and the knowledge of these programs no longer serves me.

  Simply configuring a wireless network on Linux became a huge hassle:

  • User: Which tool do I use to configure my wireless? `ifconfig`

  doesn't seem to be able to help me.

  • Google[1]: Well it depends on what driver you are using.
  • User: Intel blablablbla.
  • Google[1] -> Intel Site: `iwconfig`! The command is going to look

  very similar to `ifconfig` but don't let that fool you! It's very different and only works with one very specific type of wireless device!

  [1] Note, the use of google here. This is very crucial, and the very first sign of poor UPI, if the user has to go somewhere outside the system to find information on the system, you have already lost. Your UPI sucks.

• The Double Drill Set
  

  All of this inconsistency with the wireless stuff left me with a dirty, uneasay feeling. Where else is this disregard for users manifesting? Is there no process that looks at tooling and says: "Wait, these tools do very similar things from a users perspective. Let's not clutter the environment with more tools that do almost the same thing."

  It's like having two drills, one that drives screws in, and another that take them out.

  Maybe none of this is Linux (the kernel)'s fault, maybe it's because userland is built over here by these people… and the kernel over there by those people. Perhaps it's the job of a distributor like Debian or RedHat to keep things consistent for the end user. I don't really know, but at this point, it didn't seem like it's a high priority for any of them.

• Back and Forth
  

  Throughout my OS travels, I have consistently gone back and forth between Linux and the BSDs. At least yearly, I would have FreeBSD, OpenBSD or NetBSD on my main system. Usually this would end after a few months because one tiny thing that worked in Linux was missing. Be it an application or a driver.

  But then something changed. I decided that I had enough of this dirty feeling. I was going to abandon Linux and use one of the BSDs full time. If something didn't work, I was going to fix it. If I didn't have an application I needed, I was going to port it. But I didn't know which BSD to choose. All of them had issues with at least one thing I needed, so all were candidates for adoption!

• The Showdown
  

  One month, one BSD. Starting with FreeBSD, then OpenBSD, then NetBSD.

  Since none of the BSDs completely worked for me, I knew it would be a tough journey. There would be sacrifices, there would be work that needed to be done.

  A few things were clear:

  1. If I found myself googling for information that the system should

  have provided, that system was not the one for me.

  1. If the system had glaring UPI violations, it wasn't the system for me.
  2. If system simplicity was created via overly complex mechanisms, the

  system was not for me.

• Long Story Short
  

  OpenBSD won the showdown. It was the most complete, simple, and coherent system. The documentation was thorough, the code was easy to follow and understand.

  It had one command to configure all of the network interfaces!

  I didn't have wireless, but I was able to find a cheap USB adapter that worked by simply running `man -k wireless` and reading about the USB entries.

  It didn't have some of the applications I use regularly, so I started reading about ports (intuitively, via `man ports`!).

• The Test
  

  Shortly after selecting OpenBSD, I switched my ISP from Comcast to Century Link. Early on I decided I would run my modem in "bridge" mode, and have OpenBSD doing all the PPPoE stuff. To test my metal (and OpenBSD's) I was going to configure everything without consulting the internet!

  Armed with ONLY OpenBSD and its excellent documentation, I was able to configure an OpenBSD router doing PPPoE, NAT, DNS and DHCP. All without installing a single thing outside of the base OS (which, I might add, was installed on a 2G CF card with room to spare!) and not a single search engine query (no internet, remember? :P)!

• FastForward to Now
  

  OpenBSD is my main OS and the only OS I truly enjoy running. With every release it gains new, awesome features that embody simplicity, security, and consistency.

On Shells and Static Paths   Shell OpenBSD

• Right!
  

  Lets talk about a specific set of standards, [The Single UNIX® Specification](http://pubs.opengroup.org/onlinepubs/7990989775/index.html) and [POSIX IEEE Std 1003.1](http://pubs.opengroup.org/onlinepubs/9699919799/)

  It's true that not all Unixie operating systems conform to these standards, but all of them implement enough to solve the problem of a given shell not being installed in `/bin`!

• The wisdom of The Grey Ones
  

  Long ago, the Great Grey Ones knew that not everyone would put things in `/bin`. They also knew that for a script/binary to be usefull.. it needed to be in your PATH environment variable! Because of this.. they made statements like:

  > Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in.

  Both POSIX and SUS define this.

  I know what you are thinking: "Surely this applies only to `sh`, `bash` is the new standard, `bash` is everywhere! Anything that doesn't have `bash` is old and therefore not used! My `bash` is in `/bin/bash` - yours MUST be as well!"

• No!
  

  What if I told you, some systems like [OpenBSD](http://openbsd.org) and [FreeBSD](http://freebsd.org) do things a little differently. They have a clear distinction between "base" applications and applications that are installed via their respective `ports` systems.

  For various reasons, things like `bash` and `zsh` are not included in "base". This means they will be installed outside of the typical "/bin, /usr/bin" directory structure. Which means when you put lines like: `#!/bin/bash` at the top of your script you are ensuring that your script will not run on OpenBSD or FreeBSD (or any other system that has `bash` installed somewhere else)!

• Back to The Grey Ones
  

  They planned for this! They gave us nifty tools that allow us to invoke these these shells without knowing the explicit path for said shell!

  Tools like `/usr/bin/env` which:

  > executes utility after modifying the environment as specified on the command line.

  Simply calling `env` will spit out a list of all your currently set variables! These vars get passed to `utility` when you execute something like `env bash` (where `bash` is the utility).

  This means if we have our scripts execute `#!/usr/bin/env bash` as the first line of our script, the PATH variable will be set to something that is more likely to contain `bash` than an explicit declaration like `/bin/bash` which, as we learned, doesn't exist on some systems!

• Conclusion
  

  Don't be that guy. If your script is meant to run on a variety of systems, follow the advice of The Grey Ones, use something like `env` to make your scripts portable. I have yet to see a *NIX OS that lacks `/usr/bin/env`. This includes things like True64.

pledge(2)'ing Xmonad   OpenBSD

• Background
  

  For those that don't know, [pledge(2)](http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/pledge.2?query=pledge&arch=i386) is an OpenBSD specific feature that allows applications to "promise" to use a subset of system calls. If an app has promised to only use, say, `rpath` (obviously this is a contrived example) it would only be allowed to run this subset of system calls: `chdir(2)`, `getcwd(3)`, `openat(2)`, `fstatat(2)`, `faccessat(2)`, `readlinkat(2)`, `lstat(2)`, `chmod(2)`, `fchmod(2)`, `fchmodat(2)`, `chflags(2)`, `chflagsat(2)`, `chown(2)`, `fchown(2)`, `fchownat(2)`, `fstat(2)`, `getfsstat(2)`. If the app tries to access a system call other than these, the kernel will kill the app with a SIGABRT.

  Since the addition of pledge(2) a handful of OpenBSD ports (including ghc) have added support for it! This means we can use pledge(2) in haskell apps like Xmonad!

  When you pledge an app, you need to read through and understand what it is doing so that you can properly set the promises it will use. In larger apps you can make successive calls to pledge(2) from various parts of the app, this allows you to conditionally ratchet down the required promises.

  For example, you could make an initial loose pledge of `stdio rpath wpath cpath proc exec unix`, then later remove a few calls like so: `stdio rpath wpath cpath unix`.

  The second set of promises listed above is a fairly bad example in this case, as window managers will need `proc` and `exec` to start new programs.

• Hello Pledge!
  

  Lets write an extremely limited app that can only make calls from the `stdio` set.

  import System.OpenBSD.Process ( pledge )
  
  main = do
      _ <- pledge "stdio" Nothing
      putStrLn "Hello, World!"
  

  As a test, you can run with `tty` specified instead of `stdio`. ghc will be able to build the binary, but upon execution you will see something like following in dmesg:

  ``` hello(21115): syscall 92 "stdio" ```

  ## Pledged Xmonad

  Finally on to Xmonad! Here is an extremely basic example of a `xmonad.hs` file you could use.

  import XMonad
  import System.IO
  import System.OpenBSD.Process ( pledge )
  
  main = do
      _ <- pledge "stdio rpath proc exec unix" Nothing
      xmonad $ defaultConfig
  

Experiments in Wood Carving   Carving Ceder

• Ceder Beard Comb!
  

  

  First attempt at a beard comb made of wood! Found it very easy to carve compared to bone!

  It's a little rough around the edges.. I need to figure out a good way to make the teeth a bit more even!

Setting up networking on OpenBSD hosted VMs   OpenBSD

doas echo "net.inet.ip.forwarding=1" >> /etc/sysctl.conf
# Only run the above if you want this all to start at boot
sysctl net.inet.ip.forwarding=1

Experiments in Bone Carving - Hei matau   Carving Bone

Experiments in Coffee Roasting   Coffee Roasting

• Etgiopia Yirga Cheffe Kore Kochoer & Brazil Pulp Natural Fazenda do Sertao
  

  Beans: 50/50 of each. (Batch of 100g)

  First Crack: Ooops Second Crack: Ooops Total Roast Time: Ooops 

  First Crack: 2:35 Second Crack: 4:00 Total Roast Time: 4:55 

  First Crack: 2:08 Second Crack: 3:54 Total Roast Time: 4:34 

Experiments in Bone Carving - part one   Carving Bone

• Beard Comb - first attempt!
  

  

  I have been interested in bone carving for some time now, but it hasn't been until recently that I have actually started doing it! It is proving to be a fun and challenging hobby!

  Today's project was a Beard Comb carved from a rib bone! In hind sight, rib might not have been the best choice for this particular project. The bone is very strong length wise, as that is the direction of the "grain". This makes for weak teeth.

  Some other valuable lessons I learned while undertaking this are:

  1. Use of a jig is very important.
  2. Remove all material to form the shape of your project prior to diving into aspects like the teeth of the comb.
  3. Pick the right bone for the job.
  4. Pencil rubs off fast.

  While I was able to "finish" the comb, it isn't very useful. I broke off three teeth while cutting them out (torqued the saw just a bit when taking it out of the slot). The teeth are also too thick which prevents the comb from actually being used.

Revisiting the PicoLCD 256x64   OpenBSD

Hey Kid, I'ma Interpreter!! Stop all the static interpreter referenci'n!   Scripting OpenBSD

• STOP IT!
  

  Not every system has binaries in the same location!

  #!/bin/bash
  

  The above might seem awesome, and it might be tempting to use it, but if you do, and your project is something that I am interested in, you will receive a pull request ([here](https://github.com/JuliaLang/julia/pull/5493) are [some examples](https://github.com/nitrogen/nitrogen/pull/67)) to change it to:

  #!/usr/bin/env bash
  

  On OpenBSD, bash is not in base, meaning it is not installed in `/bin` or even `/usr/bin`. It gets plopped right into `/usr/local/bin`. Almost all users will have `/usr/local/bin` set in their `PATH` variable, so use the above and do not be a turdburgler!

Using a picoLCD 256×64 on OpenBSD 4.7   OpenBSD

464     if (strncmp(di.udi_devnames[0], "ugen", 4) != 0)
465       /* best not to play with things we don't understand */
466       continue;

Using VIM to make erlang pretty   Erlang Vim

function! ErlPretty()
    silent !erl -noshell -eval 'erl_tidy:file("%",[verbose]).' -s erlang halt
endfunction
nmap ep :execute ErlPretty()

Concurrent Hello with Erlang   Erlang

-module (chello).
-export ([loop/0, rpc/2]).

rpc(Pid, Request) ->
    io:format("rpc[~p]  sending ~p to ~p~n", [self(), Request, Pid]),
    Pid ! {self(), Request},
    receive
	Response ->
	    io:format("rpc[~p]  responding with : ~p~n", [self(), Response]),
	    {Pid,Response}
    end.

loop() ->
receive
    {From, {hello}} ->
	io:format("loop[~p] received info from: ~p~n", [self(), From]),
	From ! {self(), "Hello"},
	loop();
    {From, {goodbye}} ->
	io:format("loop[~p] received info from: ~p~n", [self(), From]),
	From ! {self(),"Goodbye"},
	loop();
    {From, Other} ->
	io:format("loop[~p] received info from: ~p~n", [self, From]),
	From ! {self(),{error, Other}},
	loop()
    end.

1> Pid = spawn(fun chello:loop/0).

<0.38.0>

2> chello:rpc(Pid, {hello}).
rpc[<0.31.0>] sending {hello} to <0.38.0>

loop[<0.38.0>] received info from: <0.31.0>

rpc[<0.31.0>] responding with : {<0.38.0>,”Hello”}

{<0.38.0>,{<0.38.0>,”Hello”}}