ElasticSearch 1.0.0 is out, hooray! Great stuff, congrats to everyone involved. Not that I had any complaints about v0.9, but still, this is a great achievement. One of the changes is some major rework of the Percolation Query API, making it much more powerful than before. Unfortunately, the update broke the percolation query mechanism in the BirdWatch application. But the fix wasn’t very hard. So in today’s article, I will revisit the topic of Percolation Queries by explaining what they are, how the new version has become more powerful and what was needed to fix my application. Please refer to this earlier article if you want to know more about the overall architecture of the BirdWatch application.
In this article I will present a simple reactive web application using Scala.js and ReactJS on the client side. It is based on sse-chat, an application I initially wrote for demonstrating the use of AngularJS with Play Framework. I then rewrote the client for an article about using ReactJS on the client side. In the latest version now, there is an additional client that connects to the same server and utilizes Scala.js to build the web client. I recently gave a talk about this at Ping Conference in Budapest, check it out if you’re interested. I discovered ReactJS through David Nolen’s blog and his excellent OM library which combines ReactJS with ClojureScript. His second article on Om also inspired me to try out an undo functionality with the immutable data structures that Scala.js has to offer. For learning more about ReactJS, I recommend going through the tutorial and also reading my last blog post.
Over the holidays I discovered Facebook’s React, an interesting library for generating reactive user interfaces. I wanted to try it out in a real-world application, and I thought of one such application I still had running as a demo: sse-chat, a little chat application I wrote last summer to learn how to make Play Framework and AngularJS cooperate in a very basic way. So I thought, why not rewrite the client side using React, offering the exact same functionality as the AngularJS version. Both are also available in the new version with no changes to the backend code except for the added route, as both versions can be accessed in parallel.
I never really thought much of New Year’s resolutions because I could never live up to them. 2013 was different though. One year ago my resolution was to start writing a blog and to create one or more open source projects, just for the fun of it. And this time I actually succeeded. So I thought I might continue with this resolution thing.
A few weeks ago I started working on the follow-up to my BirdWatch project. This new project is another single page application based on AngularJS, but that is not part of the story I am going to tell you today at all. Instead, today I will talk about speed. As in, how long does it take for a web page to load, on a mobile device? I was doing some research by opening different websites under suboptimal conditions, such as 3G with only two to three bars, or even worse the dreaded E with four to five bars. Not terribly difficult to simulate, I only need to disable the Wifi and walk into different corners of my apartment for that. Opening my own blog made me sad: with the bad 3G connection it took like 10 seconds for the index page to show anything at all. No way I would ever wait that long for any page to load. And I would quite likely not even try again. So I went on a quest to make this better. The result is of course on GitHub.
So far I have found my BirdWatch application nice to look at but not terribly useful as an original way of finding information. Let’s face it - the vast majority of tweets are not terribly useful. But there are some in there that are highly relevant. What are their characteristics? At the most basic level, they come from people with huge numbers of followers and / or have been re-tweeted a lot. It’s these tweets that have a large audience, not the ones from users with low follower counts. The latter make up the majority of the chatter, though. How do we find these more relevant tweets within an observation period?
BirdWatch is an open-source reactive web application that consumes the Twitter Streaming API for a selection of terms. It processes those matching tweets in a server side application that is based on Play Framework. The tweets are then stored inside ElasticSearch, where they are available for complex full-text searches. On the client side, a Single Page Application based on AngularJS allows the user to perform a live search for tweets with certain keywords and to do some analysis on them, such as word count statistics, activity over time and sorting results by followers and retweet counts.
I am happy to get a huge update of the BirdWatch application out of the way. The changes are a lot more than what I would normally have wanted to work in for a single article, but then again there is enough interesting stuff going on in this new version that calls for multiple blog articles. Initially this application was only meant to be an exercise in streaming information to web clients. But along the way I noticed that this application can be useful and interesting beyond being a mere learning exercise. Let me explain what it has evolved to:
Last week I was dealing with an odd behavior of the chat application demo I was running for this article. The issue was timing-related and there were no actual exceptions that would have helped in identifying the problem. How are you going to even notice spikes and pauses in potentially thousands of lines in a logfile? I was upset, mostly with myself for not finding the issue earlier, and I promised myself to find a better tool. I needed a way to transform the raw logging data into useful information so I could first understand and then tackle the problem. In this article I will show what I have put together over the weekend. Part I describes the general approach and applies to any application out there, no matter what language or framework you are using. Part II describes one possible implementation of this approach using Play Framework.
This is the follow-up on last week’s article about AngularJS and Play Framework. I want to share with you the problems that I encountered on the server side while running the demo as well as my ideas of how to deal with comparable problems more efficiently in the future. I have not encountered any AngularJS-related problems with the chat application so we won’t deal with it today. I’ll have more on AngularJS next week.