I like foursquare‘s libs. If you’re using liftweb a lot (and there is no reason you shouldn’t), then you’ve probably discovered the trove of treasures foursquare is managing in their repositories on GitHub.

If you haven’t used Rogue, then you’re missing out. It offers the neatest DSL for querying/modifying/deleting MongoDB with accordance to your Lift-Record Models.

The absolute awesome pendant for your Apache SOLR Backend is Slashem. It provides nearly the same Query DSL to SOLR as Rogue does, with the exception that it does not handle writes by default. There is no workable .save() or something that persists your record to SOLR. You have to do it yourself, and i agree that this can be quite cumbersome unless you are using my Trait. :)

Only extend your SolrSchema class with this Trait, your Meta will get it through the class!

What it does

It uses the default SOLR-client created by each SolrMeta object and creates POST-requests out of your model’s .asJValue. I added the .asJValueForSolr with a default-value of .asJValue (can be overridden in the model).

It offers 2 ways of saving to SOLR. You can either use myRecordObj.saveToSolr() to save a single record or MyModel.saveToSolr(listOfMyRecordObjs) in order to save a bunch of objects in one request to SOLR.

Last but not least you should run a MyModel.commitToSolr() to trigger a commit.

Caveats

• MyModel.saveToSolr() does not work, since it’s not an instanced record.
• myRecordObj.saveToSolr(listOfMyRecordObjs) will not append myRecordObj to the listOfMyRecordObjs, you have to do it.

Merry Christmas everyone.

So i’ve been super busy for the past few months. You might have noticed since i didn’t blog. So i’ve been doing a lot of Netty 4 related work. One project amongst those is the Liftweb Netty Integration where David Whittaker and me are filling the gaps from Netty to Servlet Containers. But there is a whole Story including Podcast upcoming on September 2nd, once i held my talk The Liftweb Netty Session at SkillsMatter London.

The Problem

As we all know, when we’re programming we need every bit of concentration. Every disturbance is possibly detrimental to your flow of thought. If you are like me and exclusively work from Home and live in an urban/city area, you might have children in your neighborhood screaming, you might encounter people drilling holes in their walls 3 houses down and you still hear the vibrations through the walls. There are shops outside, you might have a doctor above your appartment who gets visits every 15 minutes or a Waxing Studio. Apart from those “bad” disturbances you might have “good” ones as well like your girlfriend coming home or DHL bringing a package. Good varies from person to person, but you have to make due with those distractions.

My solution

Since working during the day is somewhat impossible for me (i live in the inner city of Nuremberg, Germany), i had to find a way to get more productive at night while still getting all my chores and tiny todos done. It took me a while to realize that in order to be as good as a coder as i want to be, i had to deal with those distractions rather than force a solution onto me.

Time Management

It does not matter if you’re a DevOp or if you’re just a Dev or an Op, you will always have small and big tasks. My advice after a lot of years, split your work into those 2 groups.

Solution 1

I am currently looking for a better appartment for me and my girlfriend, something with a yard and less noisy so i can work distraction-free during the day. But since we’ve been looking for a few months and did not find anything affordable/worthy of its price, i had to adjust.

Solution 2

Customers mail me frequently for a variety of reasons during the day (as most of them are from Germany), only a few mail me at night which is rather a thank you response than a real job to do. After analysing this, i started doing e-Mails and small ToDos which boggle my mind during the day, were i rarely have more than 15 minutes without a disturbance. I don’t mind being disturbed doing a 5-minute task, i do mind getting disturbed while mapping software in my mind. I also started doing all long and heavy coding/admin-tasks at nights, so i can spend 4-6 hours a night with a few clear goals i want to achieve.

Pros

Since i don’t like keeping a list of lots of small todos, do them right away during the day or if they come at night, leave the email unread in your inbox. If it’s a bigger thing, put it in your ToDo List and set a due-date.

Cons

Solution 2 is only a backup-plan since Solution 1 is not available right now. It is also only feasable if you don’t work in a straight company environment requiring you to be reachable at hours x-y, since you’ll be spending 4am-11am sleeping.

Conclusion

I’ve been able to reach goals a lot better, since i didn’t have to look at a long and demotivating todo but only a list of 6-7 items i need to do over the next weeks until my vacation. This list might grow, some might be finished early, but it’s still better than staring down a list of 30-60 things.

Don’t forget

to sign up to one of David Pollak’s Fast Track to Lift Courses i teach at SkillsMatter, London.

Today i just want to share a quick Scala goodie. We all know that i love Netty and Scala, so i wanted to write a little Goodie with Scala’s implicits.

With Implicits we can wrap transparently around an Object (such as io.netty.channel.Channel) and execute our own custom methods onto it.

Below we use the Netty 4 AttributeMap together with a few nice getters and setters. This makes it typesafe and comfortable to use. :)

Also notice the “answer” at the end, this way we can have central “answering” or general connection handling methods which can be easily called.

Have fun with it

After reading this excellent Blogpost about the new Scala Try, i was thinking of how this would come in handy for “flow-control”, but without the StackTrace overhead.

The Problem

1

val myE = new FlowException("foo")

Since throw does not generate the StackTrace, but new does, we simply extend it with NoStackTrace!

1

Now we have an Exception-class which does not contain an expensive StackTrace since we don’t need to know where, but if an error has happened.

Why do we need it?

In our netflow.io project, we need to be able to know “upstream” when there is something wrong, but using Either is imho very ugly.

Returning the case class Failure with a new StackTrace-less exception is very cool, especially since now your “real” exceptions thrown by Java libraries will be handled as usual and you can filter the Exception for being a subclass of your own (e.g. FlowException).

This way we know that a Failure(e: FlowException) is a “known” or expected exception and thus does not need any further investigation.

Upstream you can do whatever you want with that information, in our case, we simply let the akka Actor Logger.warn/error that exception without running .printStackTrace.

In my Opinion

this is a really good thing, i might be wrong due to a lack of in-depth JVM knowledge, but i think the reasoning is sound. ;)

As promised on my last Talk in London, i’ve finally outdone myself with our newest piece of code called:

The project aims to provide an extensible flow collector written in Scala, using Netty as IO Framework as well as Akka as Messaging Framework.

Supported flow-types

• NetFlow v5 - fully
• NetFlow v9 (RFC3954) - almost fully
• NetFlow IPFIX/v10 (RFC3917, RFC3955) - untested due to the lack of an exporter

Database support

Currently only the Redis Key-Value-Store is supported, but it is not much code (~100 lines) to adapt it to any other Key-Value-Store.

The goodies

At the end of the day, you’ll need our utility library wasted-util installed in your local sbt repository, since it’s not mirrored on any Ivy repository yet. :)

Also, you’ll find my very own NetFlow Parsers in the package licensed under the Apache License! So feel free to do amazing stuff with it :)

What’s coming next?

• NetFlow v1
• NetFlow v6
• NetFlow v7
• NetFlow v9 and v10/IPFIX Sampling and Option flows
• SFlow support
• Cassandra storage support

The code

can be found at https://github.com/wasted/netflow! Enjoy!

Thanks to wasted.io for sponsoring this!

Good evening everyone, so you thought i stopped blogging, but i didn’t. What i did do? Well, i wrote some amazing Scala stuff.

Apart from my Liftweb, Netty and Akka Talk at SkillsMatter, London, i was busy coding some awesome stuff for one of the new projects i am involved in. Maybe you’ve already seen me tweeting about them, but in case you haven’t, you may Follow @wastedio on Twitter.

Why am i blogging? Well you can guess it’s awesome, otherwise i wouldn’t write about it ;)

wasted.io Scala-Util

Since Twitter and everyone has their own “utils” Library, we also had to have our own. But since we don’t simply clone other people’s work, we did some nice IP Subnet Matching for my high performance audience.

It all started when i repeatedly googled for a Scala IPv4 and IPv6 Subnet Matching Library. Since i write a lot of Liftweb Code, i sometimes need to match for a certain IP-Range, which is not easy when you have to use different libraries (v6 Java, v4 Scala) in order to find out if the given client comes from Range $x.

I found a 1-year old question on StackOverflow and tried out the code, to my dismay this happened:

But if you calculate the network 1.2.176.0/20 then you get a Range up to 1.2.191.255 (which should return true on res2):

So i felt obligated to contribute my own code to the Scala Community which can be found on the wasted.io Github Repository for their Scala-Utils. It is licensed under Apache-License so feel free to use it however you feel like!

The Results

Now we are able to do real Subnet matching:

To use it, you currently have to assemble and publish it local, we’re working on getting it on a publicly known Ivy/Maven-Repository. Also check the README.md if you’re having troubles.

I can only encourage everyone to follow the stuff wasted.io (@wastedio on Twitter) has coming out in the future. We are preparing the release of our Open Source Netflow Collector called netflow.io which is based on netty and akka which is compatible with RFC3954, RFC5101, RFC5102, so make sure you stay tuned!

So last night i got really bored, so i tried to integrate VLANs in my existing Xen4 setup.

While googling for solutions, i only found things where i’d have to statically allocate VLANs for every customer on every host through xend-config.sxp network-script.

This also meant, that if i had a new customer vlan which i want to add to my host, i’d have to either restart xend or run

/etc/xen/scripts/network-bridge start netdev=eth1 vlan=5

Since this is absolutely not my style, i started investigating. Turns out there is a much better way to do so!

My current dom0 setup is as follows:

• ArchLinux minimal with gentoo-xen kernel patches (nothing special)
• eth0: mgmt link (domU migrations, ssh)
• eth1: data link (domU traffic/bridge)

First off, we disable the network-script directive in xend-config.sxp.
Second, we change vif-script directive to “vif-vlan-bridge”.

Now place the following shellscript into /etc/xen/scripts/vif-vlan-bridge.

What does this do?

Well, now you can specify in your domU configuration file things like this:

vif = [ “mac=xyz,vlan=5,netdev=eth1” ]

This will create the desired vlan out of the given parameters whenever the domU starts up. It also checks if the vif is up, and therefor only tries to create it when it is not in “ip link sh”.

It is working flawlessly so far, i’ll monitor it over the next few days, but so far it looks very promising. So please have a crack at it! :)

cheers

A few days ago i had to use Cassandra for my special project, so i went on to search for usable clients.

First i tried Cascal, which felt a bit clunky, so i decided to try Cassie (Twitter) which couldn’t create Keyspaces in code.

So i was left with Hector which is a Java client. Since i don’t like Java’s bureaucracy, i decided to write a little Wrapper in Scala.

If you look at Hector’s examples, you’ll find that you have to handle things like Keyspace and an Entity Manager (if you’re going for the ORM), which didn’t feel very comfortable.

Since i am leaving on sunday for a vacation, plus i have to code shitloads of lines until then, i will make this one short. ;)

It basically wraps everything we need in implicits, so the only thing you need in your current context when handling Entities is a Keyspace (it can also be a virtual!)

Example is in the main method. You will need to adjust a few lines of code to get it working in your setup!

Disclaimer

• I haven’t slept that much
• This was just a quick hack to see if i could do it :)

The code

Here is the Connection and Entity Manager:

To make things nice and easy for our traits, here is the implicit in the Model-package:

To top it off, i’ve prepared two sample classes with referencing (ModelB entity references to a ModelA entity) :

Here is a link to the full gist: https://gist.github.com/3317399

best regards

First off, i’ve never thought of myself as a presenter. My dad always taught me things but i didn’t really listen, to my dismay i must add. Anyway, i think i’ve done quite okay in the past (since i got good feedback).

But this fall, you have not two, not three, but four chances of seeing (and hearing) me blab on about Scala and Liftweb stuff.

September 5th, 2012

First occasion is in Nuremberg, Germany (my home town) where i was invited to speak at this year’s Herbstcampus which is roughly “Fallcampus” in german. The whole event is german, so my Liftweb talk will be german as well.

October 15th, 2012 - UPDATE: cancelled

I will be holding one of the sacred David Pollak’s Lift Basics Workshop at Skills Matter, London!

Afterwards there will be an “In The Brain” Session with me titled Scala & LiftWeb for Web Development where i’ll be presenting my experiences with Liftweb and Netty, as well as Web-Development with Scala in general. In-depth session on Long-Polling, Comet and WebSocket requests!

December 3rd, 2012

This will be my second tour to London this Fall, where i will also be teaching a David Pollak’s Lift Basics Workshop at Skills Matter, London.

If you are a Liftweb newcomer and want to learn how to get started with Liftweb, make sure to drop your reservation early! First come, first served! ;)

Cheers!

For a few months now i’ve been doing some CSV Imports in my Liftweb projects. This mostly is because the old databases i am migrating to something normal (PostgreSQL or MongoDB) are badly designed (mostly MySQL) and therefore i don’t even want to bother with their API. So XML seemed a bit heavy for most MySQL-Admins and -Users to work with, so CSV was the next “best” thing.

I have written many CSV Parsers in A LOT of languages, but writing it in Scala felt right for the first time. By the way, i’ve also tried the typical split()-variant, but i wasn’t too happy with it after some time, so i started googling and found this on StackOverflow. But i didn’t like it just yet.

What didn’t fit

• It is limited to Comma (sometimes you get a Semicolon)
• It only gives you a List[List[String]], i wanted a List[Map[String, String]]

My version

This will use your first line in the CSV to create Headers, regardless if it has a # in front of it, at the same time it will allow you to access Fields by iterating over the rows and accessing the Map’s index with a Fieldname.

All Fieldnames will be stripped of Spaces!!! Column “First Name” would become “FirstName”.

new CSV(";").parse(scala.io.Source.fromFile("foo.csv", "UTF-8")) foreach { row =>
	println(row("FirstName"))
}

Feedback much appreciated!