After reading this excellent Blogpost about the new Scala Try, i was thinking of how this would come in handy for “flow-control”, but without the StackTrace overhead.

The Problem

1

val myE = new FlowException("foo")

Since throw does not generate the StackTrace, but new does, we simply extend it with NoStackTrace!

1

Now we have an Exception-class which does not contain an expensive StackTrace since we don’t need to know where, but if an error has happened.

Why do we need it?

In our netflow.io project, we need to be able to know “upstream” when there is something wrong, but using Either is imho very ugly.

Returning the case class Failure with a new StackTrace-less exception is very cool, especially since now your “real” exceptions thrown by Java libraries will be handled as usual and you can filter the Exception for being a subclass of your own (e.g. FlowException).

This way we know that a Failure(e: FlowException) is a “known” or expected exception and thus does not need any further investigation.

Upstream you can do whatever you want with that information, in our case, we simply let the akka Actor Logger.warn/error that exception without running .printStackTrace.

In my Opinion

this is a really good thing, i might be wrong due to a lack of in-depth JVM knowledge, but i think the reasoning is sound. ;)

As promised on my last Talk in London, i’ve finally outdone myself with our newest piece of code called:

The project aims to provide an extensible flow collector written in Scala, using Netty as IO Framework as well as Akka as Messaging Framework.

Supported flow-types

• NetFlow v5 - fully
• NetFlow v9 (RFC3954) - almost fully
• NetFlow IPFIX/v10 (RFC3917, RFC3955) - untested due to the lack of an exporter

Database support

Currently only the Redis Key-Value-Store is supported, but it is not much code (~100 lines) to adapt it to any other Key-Value-Store.

The goodies

At the end of the day, you’ll need our utility library wasted-util installed in your local sbt repository, since it’s not mirrored on any Ivy repository yet. :)

Also, you’ll find my very own NetFlow Parsers in the package licensed under the Apache License! So feel free to do amazing stuff with it :)

What’s coming next?

• NetFlow v1
• NetFlow v6
• NetFlow v7
• NetFlow v9 and v10/IPFIX Sampling and Option flows
• SFlow support
• Cassandra storage support

The code

can be found at https://github.com/wasted/netflow! Enjoy!

Thanks to wasted.io for sponsoring this!

Good evening everyone, so you thought i stopped blogging, but i didn’t. What i did do? Well, i wrote some amazing Scala stuff.

Apart from my Liftweb, Netty and Akka Talk at SkillsMatter, London, i was busy coding some awesome stuff for one of the new projects i am involved in. Maybe you’ve already seen me tweeting about them, but in case you haven’t, you may Follow @wastedio on Twitter.

Why am i blogging? Well you can guess it’s awesome, otherwise i wouldn’t write about it ;)

wasted.io Scala-Util

Since Twitter and everyone has their own “utils” Library, we also had to have our own. But since we don’t simply clone other people’s work, we did some nice IP Subnet Matching for my high performance audience.

It all started when i repeatedly googled for a Scala IPv4 and IPv6 Subnet Matching Library. Since i write a lot of Liftweb Code, i sometimes need to match for a certain IP-Range, which is not easy when you have to use different libraries (v6 Java, v4 Scala) in order to find out if the given client comes from Range $x.

I found a 1-year old question on StackOverflow and tried out the code, to my dismay this happened:

But if you calculate the network 1.2.176.0/20 then you get a Range up to 1.2.191.255 (which should return true on res2):

So i felt obligated to contribute my own code to the Scala Community which can be found on the wasted.io Github Repository for their Scala-Utils. It is licensed under Apache-License so feel free to use it however you feel like!

The Results

Now we are able to do real Subnet matching:

To use it, you currently have to assemble and publish it local, we’re working on getting it on a publicly known Ivy/Maven-Repository. Also check the README.md if you’re having troubles.

I can only encourage everyone to follow the stuff wasted.io (@wastedio on Twitter) has coming out in the future. We are preparing the release of our Open Source Netflow Collector called netflow.io which is based on netty and akka which is compatible with RFC3954, RFC5101, RFC5102, so make sure you stay tuned!

So last night i got really bored, so i tried to integrate VLANs in my existing Xen4 setup.

While googling for solutions, i only found things where i’d have to statically allocate VLANs for every customer on every host through xend-config.sxp network-script.

This also meant, that if i had a new customer vlan which i want to add to my host, i’d have to either restart xend or run

/etc/xen/scripts/network-bridge start netdev=eth1 vlan=5

Since this is absolutely not my style, i started investigating. Turns out there is a much better way to do so!

My current dom0 setup is as follows:

• ArchLinux minimal with gentoo-xen kernel patches (nothing special)
• eth0: mgmt link (domU migrations, ssh)
• eth1: data link (domU traffic/bridge)

First off, we disable the network-script directive in xend-config.sxp.
Second, we change vif-script directive to “vif-vlan-bridge”.

Now place the following shellscript into /etc/xen/scripts/vif-vlan-bridge.

What does this do?

Well, now you can specify in your domU configuration file things like this:

vif = [ “mac=xyz,vlan=5,netdev=eth1” ]

This will create the desired vlan out of the given parameters whenever the domU starts up. It also checks if the vif is up, and therefor only tries to create it when it is not in “ip link sh”.

It is working flawlessly so far, i’ll monitor it over the next few days, but so far it looks very promising. So please have a crack at it! :)

cheers

A few days ago i had to use Cassandra for my special project, so i went on to search for usable clients.

First i tried Cascal, which felt a bit clunky, so i decided to try Cassie (Twitter) which couldn’t create Keyspaces in code.

So i was left with Hector which is a Java client. Since i don’t like Java’s bureaucracy, i decided to write a little Wrapper in Scala.

If you look at Hector’s examples, you’ll find that you have to handle things like Keyspace and an Entity Manager (if you’re going for the ORM), which didn’t feel very comfortable.

Since i am leaving on sunday for a vacation, plus i have to code shitloads of lines until then, i will make this one short. ;)

It basically wraps everything we need in implicits, so the only thing you need in your current context when handling Entities is a Keyspace (it can also be a virtual!)

Example is in the main method. You will need to adjust a few lines of code to get it working in your setup!

Disclaimer

• I haven’t slept that much
• This was just a quick hack to see if i could do it :)

The code

Here is the Connection and Entity Manager:

To make things nice and easy for our traits, here is the implicit in the Model-package:

To top it off, i’ve prepared two sample classes with referencing (ModelB entity references to a ModelA entity) :

Here is a link to the full gist: https://gist.github.com/3317399

best regards

First off, i’ve never thought of myself as a presenter. My dad always taught me things but i didn’t really listen, to my dismay i must add. Anyway, i think i’ve done quite okay in the past (since i got good feedback).

But this fall, you have not two, not three, but four chances of seeing (and hearing) me blab on about Scala and Liftweb stuff.

September 5th, 2012

First occasion is in Nuremberg, Germany (my home town) where i was invited to speak at this year’s Herbstcampus which is roughly “Fallcampus” in german. The whole event is german, so my Liftweb talk will be german as well.

October 15th, 2012 - UPDATE: cancelled

I will be holding one of the sacred David Pollak’s Lift Basics Workshop at Skills Matter, London!

Afterwards there will be an “In The Brain” Session with me titled Scala & LiftWeb for Web Development where i’ll be presenting my experiences with Liftweb and Netty, as well as Web-Development with Scala in general. In-depth session on Long-Polling, Comet and WebSocket requests!

December 3rd, 2012

This will be my second tour to London this Fall, where i will also be teaching a David Pollak’s Lift Basics Workshop at Skills Matter, London.

If you are a Liftweb newcomer and want to learn how to get started with Liftweb, make sure to drop your reservation early! First come, first served! ;)

Cheers!

For a few months now i’ve been doing some CSV Imports in my Liftweb projects. This mostly is because the old databases i am migrating to something normal (PostgreSQL or MongoDB) are badly designed (mostly MySQL) and therefore i don’t even want to bother with their API. So XML seemed a bit heavy for most MySQL-Admins and -Users to work with, so CSV was the next “best” thing.

I have written many CSV Parsers in A LOT of languages, but writing it in Scala felt right for the first time. By the way, i’ve also tried the typical split()-variant, but i wasn’t too happy with it after some time, so i started googling and found this on StackOverflow. But i didn’t like it just yet.

What didn’t fit

• It is limited to Comma (sometimes you get a Semicolon)
• It only gives you a List[List[String]], i wanted a List[Map[String, String]]

My version

This will use your first line in the CSV to create Headers, regardless if it has a # in front of it, at the same time it will allow you to access Fields by iterating over the rows and accessing the Map’s index with a Fieldname.

All Fieldnames will be stripped of Spaces!!! Column “First Name” would become “FirstName”.

new CSV(";").parse(scala.io.Source.fromFile("foo.csv", "UTF-8")) foreach { row =>
	println(row("FirstName"))
}

Feedback much appreciated!

I’ve been talking with a lot of folks lately about Configuration Management/Automation Systems. Mainly about puppet vs cfengine.

The first question i usually ask people recommending interpreter solutions (ruby, perl, python..) is “Why?”. Why would anyone want an interpreted solution?

I get the point that the scripted languages are usually easier and therefore easy to maintain. This is true if you compare cfengine to almost any other solution. Yes, cfengine is a bit quirky.

So here are the main issues i’ve encountered with interpreted solutions over the past 4 years.

#1 Libraries (Gems and Eggs)

If your config management needs libraries like Ruby’s Gems or Python’s eggs on runtime, then it is clearly a bad idea. Imagine you having to keep libraries up to date, just so your config management runs.

I want my management system to be able to run anywhere, without having to worry about Gem-versions and Egg-versions.

#2 Interpreters (Ruby, Perl, Python…)

Having said the version-part about Gems and Eggs, this applies also to the language it’s written in.

Having the requirement of an actual interpreter for your config management installed is more than unsettling to me, why? Because you will run into issues regarding different OS/Architecture stuff that the language might handle differently/ineffeciently or make it segfault or deadlocking. Also having a common interpreter for config management (like Ruby) gives a whole new front of issues when you’re for example a Rails- or in general a Ruby- coder.

Since you most certainly need a specific version of (e.g Ruby 1.8) to run certain code in your interpreter, plus you might need to keep your libraries/gems/eggs up to date.

So you think my points are all bogus?

Why don’t i just tell you my 1.5y-experience with Puppet back in ‘09 (i think)?

I had puppet running on roughly 40 VMs. The first thing that i found good at the time was, that it is Ruby! I know ruby, awesome! But the feeling that the decision was right, didn’t hold up for long. Having to update multiple libraries AND your config mangement, then you have a severe upgrade-impact.

After having those servers run for a couple of days, i noticed that some of the VM’s puppetd weren’t running anymore. That got me curious, so i started digging. Turned out that puppet somehow crashed quite frequently. One day i decided to let puppet put itself into every VM’s crontab, so it would do a /etc/init.d/puppetd restart every hour.

All was great, until one day my IRC VM was lagging like hell. I did BW checks at home and in the DC, nothing there. Then i logged into the VM Host just to find a couple of VMs running at 100% CPU. Great, now what?

Turns out that ruby/puppetd deadlocked itself while eating 100% CPU - yaaay! I restarted those puppets and all was good for another few months when it hit again. Several puppetd’s dead, another few are just doodling at 100% CPU..

During that time

i had to mingle with different Gem- and Ruby-Versions (as 1.9 was just hitting the scene) just so i could get my management and my development back on track.

Clearly, this is NOT what you want.

So what do i use?

Cfengine. Cfengine. Cfengine.

Why? Because it’s C.

When i’m done with compiling my static cfengine package for archlinux:
-rw-r–r– 1 fbettag wheel 447K Jul 22 2011 cfengine-3.2.0b1-1-x86_64.pkg.tar.xz

Yes, you’ve read right. Statically compiled it’s 447K in a .tar.xz, no library dependencies, no nothing. Also, i’ve never seen cfengine eat more than 10MB RAM during Runtime and i never had a box where cfengine crashed or segfaulted.

Admittedly, it is some kind of PITA to get to know Cfengine, i at least spent 24 Hours in migrating my puppet setup to Cfengine, but it was worth it!

Wir hören ja immer wieder, dass die Piratenpartei kein Programm hat, dass es an Inhalten fehlt. Ganz abgesehen davon, dass dies nicht stimmt, gibt es bestimmt viele Piraten-Wähler, die sich trotzdem nicht mit dem Programm auskennen oder es nichtmal gelesen haben (wie ich).

Warum? Ganz einfach eigentlich.. Herr Lasse Becker meinte bei ZDF-Login, dass Transparenz kein Inhalt ist. Dieser Aussage stimme ich zu, jedoch kenne ich die Grundeinstellung der Partei: Basisdemokratie.

Basisdemokratie?

Wie wir in den letzten 10 Jahren gesehen haben, funktioniert es nicht, alle vier Jahre ein Kreuzchen zu machen, da die Gründe für die wir Kreuze vergeben, in der Regel sowieso nicht eingehalten werden oder am Ende einfach nur leere Versprechen waren. -heftig zu CDU/SPD schiel-

Es ist auch unmöglich, dass eine Partei Probleme vorhersagt (Finanzkrise, Eurokrise, EU-Richtlinie zur Vorratsdatenspeicherung), demnach ist es (für mich zumindest) als Bürger wichtig, dass wenn diese Probleme auftreten, ich dann aktiv mich einbringen kann, und nicht darauf hoffen muss, dass die Partei die ich gewählt habe, schon das richtige tun wird. Da wurde ich als ehemaliger CDU-Wähler (nicht Mitglied!) schon öfters enttäuscht.

Demnach ist es für mich wichtig, mich einbringen zu können, wenn ein Problem auftritt.

Antwort auf $alles

Zugegeben, mir geht der Satz “dazu haben wir noch keine…” auch auf den Senkel, aber das erfrischende daran ist, dass diese Partei eben nicht versucht, auf Teufelkommraus für Alles eine Antwort zu haben. Bei etablierten gilt da oft eher schlecht als Recht. (Recht ist bewusst groß)

Ja es wird noch entschieden, ja es gibt Leute die diese Partei wählen, aber wenn man realistisch ist, warum brauchen die Piraten Antworten auf alles?

Es gibt doch genug etablierte Parteien, die den Wirtschafts-, Finanz- oder Agrar-Sektor betreuen, denen fehlt es an Kompetenzen um auf IT-Fragen wie die VDS ordentlich zu reagieren. Oder wie man in der Debatte in Berlin um diesen Berlin-Trojaner gesehen hat, dass die Abgeordeneten die darüber bei CDU und SPD entscheiden, einfach nicht das nötige Verständnis für IT-Angelegenheiten haben..

Warum also sollte die Piratenpartei eine Partei für alle Themen sein? Mir wäre es lieber, wenn wir mehr Kompetenz-Parteien hätten, die sich zusammen in Bündnissen/Koalitionen ergänzen anstatt sich gegenseitig zu behindern.

Reicht es nicht, dass die Piraten gute Antworten auf Fragen aus der IT-Welt, der Bildungspolitik und der Sozialpolitik haben?

Wenn man realistisch ist, dann werden die Piraten irgendwann bei 20-30% sein. Wenn die Piraten irgendwann mal bei 100% angekommen sein sollten, ja gut, dann braucht man wirklich ein Vollprogramm… Aber ich glaube, da haben wir noch gute 5 Jahre, vorausgesetzt die etablierten machen so weiter ;)

Die Lüge der Inhaltsleere

Wie gesagt, diese Lüge oder auch Unwissen ist weit verbreitet. Ich kenn das Programm nicht, nur die Teile die mich interessieren, dass bedeutet aber nicht, dass die Partei deswegen kein Programm hat.

Das Problem an diesem Stigma ist, dass man zum Beispiel auf Gruppen treffen kann, die nicht dazu passen, sich dennoch irgendwie verbunden sehen.

Ein Beispiel wäre hier die Nazi-Problematik auf dem Piraten Parteitag oder das wiederholte Auftauchen der Anti-EU Partei bei den ACTA Demos (in Wien).

Vielleicht muss man hier vermehrt etwas mehr Programm-Marketing betreiben, da möchte ich aber niemandem unterstellen, dass da bis jetzt zu wenig passiert ist. Desinformation gehört wohl heutzutage immer dazu.

Hier würde es vielleicht auch helfen, anstatt “Wir haben noch nicht..” einfach “Dies wurde von unserer Basis noch nicht eingebracht. Wenn dieses Thema für Sie (Zuschauer) wichtig ist, dann können Sie sich entweder bei uns einbringen, oder eine Partei wählen, die Ihre Interessen in dieser Hinsicht bereits vertritt.”

Vielleicht erkennt dann der ein oder andere renitente Politiker anderer Fraktionen auch, dass man nicht immer eine Antwort braucht.

Politische Idee / Vision

Herr Thorsten Denkler behauptete im ZDF-Login Interview, dass man “normalerweise” eine Politische Vision oder Idee hat, bevor man eine Partei gründet, dies bei den Piraten allerdings fehlt, und genau dieses Statement entzieht sich meinem Verstand.

Diese Basisdemokratische Partei möchte mehr Transparenz, mehr Bürgerbeteiligung und ein freies Internet.

Ich zähle in diesem Satz schonmal vier Visionen und Ideen, die in unserer derzeitigen Politik nicht vorkommen, vielleicht geht ja unsere Vorstellung von Ideen und Visionen etwas auseinander, ich meine jedenfalls nicht die Visionen, die man im Drogenrausch hat.

Transparenz

Herr Denkler betonte auch, dass es keine Hinterzimmer-Diskussionen gab, die nicht am nächsten Tag in der Zeitung standen. Na super! Dann kann man es auch gleich öffentlich machen!

Andersherum, wenn man sich anschaut wie die Rede von Herrn Uhl im Nachhinein verändert wurde, dann kann man sich auf den “transparenten” Prozess Presse auch nicht 100%ig verlassen.

Ebenfalls haben Herr Denkler und Herr Becker auch wiederholt angebracht, dass Transparenz kein Inhalt ist. Allerdings ist es ein Praxis, die in unserer Repräsentativen Politik abhanden gekommen ist.

Warum also ist es nicht legitim, dass ich eine Partei deswegen wähle, weil Sie mehr Licht ins Dunkel rücken will? Frau Weisband betont es immer wieder, und ich finde es eine tolle Aussage, “Warum kann ich etwas, das ich im Namen der Bürger beschließe, nicht auch vor diesen Verhandeln/Besprechen?”

Es ist mir klar, dass wir unsere Militär-Strategien oder Whatna nicht frei offenlegen sollten, aber bei Dingen wie ACTA, SOPA, PIPA und Weissgottnoch-Alles, da wundert man sich als Politiker (vor Allem Herr Becker), dass Leute eine Partei wählen, die für Transparenz und gegen Hinterzimmer-Politik steht?

Fazit

Auch wenn man die Piraten nicht wegen Inhalte wählt, es ist in jedem Fall eine Investition in die Zukunft, denn man ganz sicher davon ausgehen, dass basisdemokratische Entscheidungen getroffen werden, egal ob die mir nun passen oder nicht.

Wer allerdings lieber seine Interessen über die Interessen der Mehrheit (Demokratie), oder auch Tyrannei der Masse bei der FDP genannt, treffen möchte, der ist in einer Diktatur besser aufgehoben.

When using Ruby, i found myself frequently making things like formatting methods to display a variety of things.

def formatDouble(dbl)
     "%.2f" % dbl
end

Often these output formatters are render-specific, so defining them on the model seems kinda unflexible to me. Of course if you only have one Double to format, this is not the best/quickest way to go, but imagine you have a Snippet which has 3-4 BigDecimals to format.

class MySnippet {
     def formatDouble(a: Box[BigDecimal]) = a match {
           case Full(dbl) => "%.2f".format(dbl)
           case _ => ""
     }

     def render = ".field1 *" #> formatDouble(myRecord.field1.is) &
                  ".field2 *" #> formatDouble(myRecord.field2.is)
}

Putting a format around everything like this would be too much code for me, even for 2 lines this looks like crap to me.

What to do about it?

There is a very simple way of dealing with things, this is due to the fact that #> will only take String and NodeSeq (and a few other’s you usually don’t come to contact with), so we can work with Scala’s implicit.

class MySnippet {
     implicit def formatDouble(a: Box[BigDecimal]) = a match {
           case Full(dbl) => "%.2f".format(dbl)
           case _ => ""
     }

     def render = ".field1 *" #> myRecord.field1.is &
                  ".field2 *" #> myRecord.field2.is
}

In my (humble LOL) opinion, this looks a lot better. You can also apply this on different needs like with Tuples:

class MySnippet {
     implicit def formatPrice(a: (Box[BigDecimal], String)) = a._1 match {
           case Full(dbl) => "%.2f %s".format(dbl, a._2)
           case _ => ""
     }

     def render = ".field1 *" #> (myRecord.price.is, myRecord.currency.is) &
                  ".field2 *" #> (myRecord.shipping.is, myRecord.currency.is)
}

I hope this helps some of you folks to make your code more readable!

And yes, i admit that these example are prolly not the most advanced use cases, but you should get the idea :)

Cheers