Use Your iPad as a Second Monitor

Sometimes carrying around a laptop and an iPad seems like overkill. Yet, sometimes, it's purely fantastic. That's right, with a nifty app called Duet Display, one can turn an iPad in to a second display! 

There are a couple different apps that let you turn your iPad into a secondary display, each with their pros and cos. The two other main contenders are AirDisplay 3 and iDisplay which were reviewed along with Duet by LifeHacker. Long story short, Duet works well and I'm a happy camper who can now have dual displays even at a coffee shop. If you have an iPad, go ahead and get this app and enjoy the extra screen real estate! 

Oh, and if you are wondering about the wonderfully colorful iPad stand, you need not go further than these Lil' Engineer Construction Blocks. They may be rated for 3-5 year olds, but who says Dad can't have any fun? 

Programmatically Logging in to a site with the Auth0 Lock Widget

I had the joy of having to figure out how to programmatically log in to a website that uses the Auth0 Lock Widget for authentication. Since Auth0 provides authentication as s service, the login flow is slightly more complex than a simple POST to the site's login endpoint. Fortunately for you, I've spent the time figuring it out ;)

In a nutshell, there are 4 calls that have to be made for a successful login

1. GET to the site you are trying to log in to (to get a state variable)
2. POST to Auth0 with the username and password
3. POST to the callback handler on Auth0 with the results from the previous POST
4. GET to the redirect page that the previous POST indicates to redirect to

The following Ruby script does exactly what you'd expect it to do making use of the RestClient and Nokogiri gems. Note, you could make this script work without these two gems, but they do make life quite a bit easier.

	# The site you want to log in to
	SITE_URL = nil
	# The username/password you want to log in with
	USERNAME = nil
	PASSWORD = nil
	# All of these details are visible when you do a "login" and
	# inspect the parameters POSTed when you click "login" on
	# the Lock Widget. You'll want to do this in your favorite
	# web browser while having the developer console open and
	# inspecting the network traffic.
	AUTH0_CLIENT_ID = nil
	AUTH0_ACCOUNT_URL = nil
	AUTH0_CONNECTION = nil
	AUTH0_TENANT = nil
	AUTH0_APP_LOGIN_REDIRECT_URL = nil
	# Make an initial call to the page that has the
	# state value that is used when POSTing to login
	response = RestClient.get(SITE_URL)
	state = /state: '([^']+)'/.match(response.body)[1]
	# POST the username/password and attempt to log in
	response = RestClient.post(
	"#{AUTH0_ACCOUNT_URL}/usernamepassword/login",
	payload={
	client_id: AUTH0_CLIENT_ID,
	connection: AUTH0_CONNECTION,
	redirect_uri: AUTH0_APP_LOGIN_REDIRECT_URL,
	response_type: "code",
	scope: "openid email crud:all",
	sso: true,
	state: state,
	tenant: AUTH0_TENANT,
	username: USERNAME,
	password: PASSWORD
	},
	headers={
	cookies: response.cookies
	}
	)
	# Extract the hidden form elements that we need to
	# POST to the callback
	form = Nokogiri::HTML(response.body)
	payload = {}
	form.css('input').each do |input|
	if input.attributes['type'].value == 'hidden'
	payload[input.attributes['name'].value] = input.attributes['value'].value
	end
	end
	# Call the callback, and then redirect to where the
	# callback says to go.
	begin
	RestClient.post(
	"#{AUTH0_ACCOUNT_URL}/login/callback",
	payload=payload,
	headers={
	cookies: response.cookies
	}
	)
	rescue RestClient::MovedPermanently,
	RestClient::Found,
	RestClient::TemporaryRedirect => err
	logged_in_response = RestClient.get(err.response.headers[:location], headers={cookies: err.response.cookies})
	end
	# The user is now logged in and logged_in_response now has all the
	# appropriate cookies that can be then used in subsequent call the
	# the site

view raw auth0_login.rb hosted with ❤ by GitHub

Let's hope that Auth0 doesn't change it's login specs anytime soon!

Nested send and hash/array access for Ruby Objects

I had to do some work where I needed to be able to perform nested calls to an object mixing both send 'sending' and accessing data from a hash/array in a single call. In this gist I've put together a method that can be added to the base Ruby Object that lets you perform complex nested calls. For example

value = obj.send_nested("data.foo['bar'].id")

and under the hood this will do something akin to

obj.send(data).send(foo)['bar'].send(id)

This also works with symbols in the attribute string

value = obj.send_nested('data.foo[:bar][0].id')

which will do something akin to

obj.send(data).send(foo)[:bar][0].send(id)

In the event that you want to use indifferent access you can add that as a parameter as well. E.g.

value = obj.send_nested('data.foo[:bar][0].id', with_indifferent_access: true)

Since it's a bit more involved, here is the link to the gist that you can use to add that method to the base Ruby Object. The gist also includes the relevant tests for your peace of mind.

Converting an ADT HL7 message to JSON

If anyone has ever had the pleasure of working the HL7 ADT messages in to exchange information between healthcare systems, you'll know the frustration of trying to actually using that information in a meaningful way (i.e. trying to work with it without having to read 200 pages of documentation to understand what the different segments of an HL7 message are).

Although the HL7 standard makes it less verbose when sending the information along, I prefer working with JSON objects as opposed to pipe delimited strings. If you are in the same boat, then you can use the following two python functions to convert an HL7 message to an easier to digest/understand. (Note these two functsion depend on the hl7apy python library)

	"""Converting HL7 messages to dictionaries
	Example Usage:
	import pprint
	from hl7apy.parser import parse_message
	# Taken from http://hl7apy.org/tutorial/index.html#elements-manipulation
	s = """MSH|^~\&|GHH_ADT||||20080115153000||ADT^A01^ADT_A01|0123456789|P|2.5||||AL
	EVN||20080115153000||AAA|AAA|20080114003000
	PID|1||566-554-3423^^^GHH^MR||EVERYMAN^ADAM^A|||M|||2222 HOME STREET^^ANN ARBOR^MI^^USA||555-555-2004~444-333-222|||M
	NK1|1|NUCLEAR^NELDA^W|SPO|2222 HOME STREET^^ANN ARBOR^MI^^USA"""
	pprint.pprint(hl7_str_to_dict(s))
	Yup, it's as simple as that.
	"""
	from hl7apy.parser import parse_message
	def hl7_str_to_dict(s, use_long_name=True):
	"""Convert an HL7 string to a dictionary
	:param s: The input HL7 string
	:param use_long_name: Whether or not to user the long names
	(e.g. "patient_name" instead of "pid_5")
	:returns: A dictionary representation of the HL7 message
	"""
	s = s.replace("\n", "\r")
	m = parse_message(s)
	return hl7_message_to_dict(m, use_long_name=use_long_name)
	def hl7_message_to_dict(m, use_long_name=True):
	"""Convert an HL7 message to a dictionary
	:param m: The HL7 message as returned by :func:`hl7apy.parser.parse_message`
	:param use_long_name: Whether or not to user the long names
	(e.g. "patient_name" instead of "pid_5")
	:returns: A dictionary representation of the HL7 message
	"""
	if m.children:
	d = {}
	for c in m.children:
	name = str(c.name).lower()
	if use_long_name:
	name = str(c.long_name).lower() if c.long_name else name
	dictified = hl7_message_to_dict(c, use_long_name=use_long_name)
	if name in d:
	if not isinstance(d[name], list):
	d[name] = [d[name]]
	d[name].append(dictified)
	else:
	d[name] = dictified
	return d
	else:
	return m.to_er7()

view raw hl7_to_dict.py hosted with ❤ by GitHub

Yes, these functions return a python dictionary and not a JSON object, but you can trivially convert a dictionary to a JSON string.

import json

from hl7apy.parser import parse_message

# Taken from http://hl7apy.org/tutorial/index.html#elements-manipulation
s = """MSH|^~\&|GHH_ADT||||20080115153000||ADT^A01^ADT_A01|0123456789|P|2.5||||AL
EVN||20080115153000||AAA|AAA|20080114003000
PID|1||566-554-3423^^^GHH^MR||EVERYMAN^ADAM^A|||M|||2222 HOME STREET^^ANN ARBOR^MI^^USA||555-555-2004~444-333-222|||M
NK1|1|NUCLEAR^NELDA^W|SPO|2222 HOME STREET^^ANN ARBOR^MI^^USA"""

# Convert it
d = hl7_str_to_dict(s)

# Dump it as a JSON string
print json.dumps(d)

Hope this helps someone who appreciates new data representations more than old data representations ;)

Simple open_sftp() context manager for sftp read and writing of files

I had to do some reading/writing of files from an SFTP source/destination and ended up putting together a simple context manager for being able to do this so that it follows the same general interface as open() for local files. Usage is as simple as

from open_sftp import open_sftp

path = "sftp://user:p@ssw0rd@test.com/path/to/file.txt"

# Read a file
with open_sftp(path) as f:
    s = f.read() 
print s

# Write to a file
with open_sftp(path, mode='w') as f:
    f.write("Some content.") 

It's as simple as that. The full code can be found as a gist on GitHub. Note: This assumes that the directory already exists, but one could modify this trivially to create the path automatically by adding the details from this StackOverflow thread.

Configuring the Python Elasticsearch Client to use TLSv1.1

We spent an hour trying to configure the python Elasticsearch client to work over SSL. In the end, it is a very easy solution (and it's even partially documented!), but in case any one else runs in to the issue... here is what the symptom and the solution was.

Basic Setup

First, here was snippet of code that we were using to connect to our Elasticsearch instance. (Note,obviously that IP address isn't the one we are actually using)

from elasticsearch import ElasticSearch

es = ElasticSearch(
    "hosts": [
        {
            "host": "123.45.67.890",
            "use_ssl": true
        }
    ]
)

print es.info()

If you were to look at the docs, you'd thing that this is all that you would have to do. Unfortunately, this (most likely) won't work. And if you are reading this, then it probably didn't work for you either.

Symptom & Diagnosis

Running the above snippet yielded the following error message:

ConnectionError: ConnectionError(HTTPSConnectionPool(host=u'123.45.67.890', port=9200): 
Max retries exceeded with url: / (Caused by : )) 
caused by: MaxRetryError(HTTPSConnectionPool(host=u'123.45.67.890', port=9200):
Max retries exceeded with url: / (Caused by : ))

We then checked in a regular browser to make sure that we can actually reach the Elasticsearch server (i.e. visited https://123.45.67.890:9200) and we indeed were able to connect and we received a nice response with some basic config details.

Following this we did a tcpdump to make sure that we actually were able to connect the the Elasticsearch server, and, as you might expect, according to the dump, a TCP connection was being made. More specifically, we did:

sudo tcpdump -n host 123.45.67.890

With a result that included valid connections and responses from the server:

...
16:50:49.060077 IP 10.1.248.172.49322 > 123.45.67.890.9200: Flags [S], seq 4274669687, 
    win 65535, options [mss 1366,nop,wscale 5,nop,nop,TS val 1362130305 ecr 0,
    sackOK,eol], length 0
16:50:49.125589 IP 10.1.248.172.49322 > 123.45.67.890.9200: Flags [.], ack 1, 
    win 8192, length 0
16:50:49.127457 IP 10.1.248.172.49322 > 123.45.67.890.9200: Flags [P.], seq 1:96, 
    ack 1, win 8192, length 95
...

So, by the looks of it, we were able to connect to the server with a browser, AND our python snippet was correctly sending data to our server, but things were not working. After some head scratching we looked at the logs on the Elasticsearch server (in our case that was in /var/log/messages)and discovered the following interesting error:

javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 
    not enabled or not supported
  at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)
  at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
  at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
  at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
  at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
  at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
  at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
  at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(
    FrameDecoder.java:425)
  at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
  at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(
    SimpleChannelUpstreamHandler.java:70)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(
    DefaultChannelPipeline.java:564)
  at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(
    DefaultChannelPipeline.java:559)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
  at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(
    AbstractNioWorker.java:108)
  at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(
    AbstractNioSelector.java:337)
  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

It looks like by default the Elasticsearch client uses TLSv1. Now, most machines (correctly) have TLSv1 disabled due to known vulnerabilities. But don't worry, before getting upset about having to downgrade to an insecure TLSv1, there is a very easy solution to this problem.

Solution

The only thing that you have to change when you setup the client it to make it use the RequestsHttpConnection. It's really as simple as that.

from elasticSearch import ElasticSearch, RequestsHttpConnection 

es = ElasticSearch(
    "hosts": [
        {
            "host": "123.45.67.890",
            "use_ssl": true
        }
    ],
    connection_class=RequestsHttpConnection
)

print es.info()

Note this will require you to install the requests library.

In the documentation this functionality is described when using it to connect to AWS with IAM, but not as how one should set it up to use TLSv1.1. Well, I guess now we know.

Hopefully this saves someone some pain and frustration!

Start of day in UTC timezone

Since all of our times are stored in UTC in our database (hopefully yours are as well!), getting all items created "yesterday" is not as straightforward as one would like. Since I have to look up how to get the correct UTC time, I figured I'd simply write it down.

from datetime import datetime, date, time, timedelta

utc_offset = datetime.utcnow() - datetime.now()

today_start = datetime.combine(date.today(), time())
today_start += utc_offset
today_end = today_start + timedelta(hours=24)

Hopefully this saves someone a little time.