Project Plugh: Open Source Log Analysis May 21, 2013
A hollow voice said "plugh"
I'm building an open source log manage/analysis tool that will offer folks what Splunk offers folks, except it's open. https://github.com/projectplugh
I'm using a combination of "right for the task" open source things to build a product that is long overdue. Specifically
- Clojure for the web front end/user interaction piece. Why? I want to learn Clojure. And core.logic gives us Datalog out of the box which means I don't have to write a query language. Clojure plus AngularJS plus some bits of Lift-style server-push means flowing results to the browser as they become available.
I signed up to give a presentation at Strange Loop on pushing data to the browser via Scala/Lift and Clojure... problem is I haven't done much of any Clojure work, so I needed a project that would let me build some nifty server-push technology in Clojure (yeah... stuff exists, but it wasn't invented here...) I advertised a 75% discount in my rate for a project that would let me learn Clojure and do server-push stuff, but nobody took me up on the offer.
I was lamenting to @meangrape (Jay Edwards) and he said, "let's build something together," so we noodled a bunch of ideas and came up with the idea of an open source project that would out-do Splunk. We looked around at technologies (I had wanted to use Datomic, but it's not open and its got crappy write performance.) Jay recommended Riak. I looked at it and said, "holy crap, this solves all my problems and it'll pour me a beer."
Then I pinged Jordan West... we chatted and I got even more excited.
We're developing in the open (no mailing list yet… but soon.) We've got a Minimally Viable Product and I'm currently slogging through hooking Netty up to Clojure (yeah… I could use a library, but learning Clojure is what I want to do.)
The name comes from Colossal Caves Adventure... "A hollow voice says, 'Plugh'". It's a magic word. My current tag line is "Plugh: open source magic to spelunk your log data."