Enabling a full XEN domU login console

So I got rid of vserver and I'm rebuilding my server with xen. I'm building a XEN 4.1 with Gentoo XEN kernels for domain 0 and unprivileged domains. There were a number of issues with the process but I managed to get a stable fully functional dom0 kernel going. Unprivileged domains will have to be built from scratch as the current file systems were tweaked for the vserver environment.

The base file system is a Gentoo amd64 stage3 mounted in loopback mode. I also have a functional domU kernel so it was time to create a sample configuration file and fire up a virtual host with

xm create <config_file> -c

It seemed to bootup properly but console output ceased immediately after the kernel booted - the point at which process 1 is called: init. Some theory as to why this happens can be found here: http://www.xen.org/files/xensummit_4/xensummit_linux_console_slides.pdf

So to enable a fully functional xen login console the following is required (as always, there are other methods for similar or different purposes):

• Make sure your domU kernel has all serial ports disabled. This may not be required but it will save some potential hassle because of how xen handles domU kernels;
  
  
• Make sure your domU file system is populated with a bare base of device files in /dev (console, null, etc.). Gentoo's stage {1,2,3} base filesystems have all the necessary files;
  
  
• Configure the kernel's virtual terminal driver to use xen's subsystem by adding the following command line parameter

  xencons=xvc

  As far as I understand, this is the default for current XEN kernels, so this parameter may not be required (it wasn't in my case but it's here for the sake of completeness);
  
  
• Configure the kernel's console to output to a xvc type terminal. This is done by adding

  console=xvc0

  to the domU's kernel command line;
  
  
• Adding kernel command line parameters can be done by editing the configuration file and adding (or adding to) a 'extra=' entry with whatever command line parameters as required. Specifically for this case, that would be

  extra = 'xencons=xvc console=xvc0'

  If 'extra=' already exists and contains something, just add the console parameter at the end:

  extra = 'parameter=value param2=value2 xencons=xvc console=xvc0'

  
  
• Observe the kernel bootup messages looking for lines with 'console'. There should be one similar to:

  Xen virtual console successfully installed as xvc0

  
  

At this point, there should be a working console past the init process, service startup output (rc*) will be visible. However, it is likely that a login prompt won't appear. If that's the case and you want one, read on.

• /etc/inittab can be setup to fire respawning login terminals at character devices, such as serial ports or the xen console (xvc0). One or more terminal lines are probably already on /etc/inittab with getty processes such as

  c1:12345:respawn:/sbin/agetty 38400 tty1 linux

  I modified one of those to point at /dev/xvc0 rather than at /dev/tty1:

  c1:12345:respawn:/sbin/agetty 38400 xvc0 linux

  (in case you're wondering, the first parameter c1 is only a label). In addition, for xen domU virtual hosts, there is little point in having any other login terminals, so the remaining (at tty2, tty3 and so on) can safely be commented out;
  
  
• Remember to setup a root password...;
  
  
• The final step is to get your system to allow root logins on the xen console. /etc/securetty contains a list of terminal devices over which root logins are allowed, to which 'xvc0' needs to be added (no /dev/);
  
  

Done!

A few more things I learnt while setting up this template file system:

• When creating sparse loopback file systems, make sure the host file system can accommodate the entire file, or else the loopback file system will become corrupt;
• Linux does strange things when it runs out of space on /;

kthxby!

code snippet and a quote

Code snippet unrelated to quote and quote unrelated to code snippet.

Code snippet [needed this for a while]:

#!/bin/bash

function fsuffix() {
       local LEN=2
       # ${1} full path
       FILE=$(basename ${1})
       DIR=$(dirname ${1})

       # get last file of the specified type
       LS=$(ls -1 "${1}"-* 2>/dev/null| egrep "${1}-[0-9]+$" | sort -nr | head -n 1)

       if [ -z "${LS}" ]; then
               FSTSUF=1
               LSTSUF=0
               LST=""
               LSTSFX_OUT=""
       else
               LSTSUF=$(echo ${LS} | egrep -o "[0-9]*$")
               FSTSUF=$((LSTSUF+1))
               printf -v LST "%s-%0${LEN}d" "${FILE}" "${LSTSUF}"
               LSTSFX_OUT="${DIR}/${LST}"
       fi

       printf -v FST "%s-%0${LEN}d" "${FILE}" "${FSTSUF}"

       FSTSFX_OUT="${DIR}/${FST}"
       return ${FSTSUF}
}

Quote:

I'm going to record this in your file, under commendations-oh, there's a lot of space here. "Did well...enough"

Ducati Desmosedici RR

One of the past weekend's highlights was spotting a rare bike... Alice.

Alice is a Ducati Desmosedici RR: a £40k road legal MotoGP motorcycle replica. This is a rare find, particularly due to its price, but also because of scarce availability.

There's a lot special about this bike from an engineering point of view.

• Desmosedici RR is the road legal Racing Replica of the MotoGP Ducati Desmosedici GPn;
• Desmosedici GPn is the n'th season motogp racing bike developed by Ducati. Currently in the MotoGP season of 2011, Ducati is running the Desmosedici GP11;
• The term "desmosedici" is a contraption of 'desmodromic' and 'sedici' (sixteen in Italian) that would translate to something like '16 valve Desmodromic valvetrain';
• Desmodromic distribution is a solution devised to overcome problems related to spring actuated valves at high engine rpm first implemented around 1910;
• (All current Ducati engines employ the desmodromic valvetrain system: desmoquattro, testastretta, testastretta evoluzione);

Here's a detailed technical description and comparison between a Desmosedici RR and a GPn: http://robotpig.net/__automotive/ducati_desmosedici.php?page=1
And this is a very good succinct 3d model of a desmodromic valve: http://www.seastarsuperbikes.co.uk/ducatiengines.html

From a rider's standpoint this is probably as close as one can get to a road legal motogp bike. And that's ~200bhp at the rear wheel, slipper clutch, 171Kg (== over 1000 bhp per 1000Kg), a beautiful growling Ducati V-4 engine attached to a glorious - barely legal - exhaust and handling manners of a race bike.
This is _not_ my type of bike, but it's as exciting as a GT1 racing car. Like such, I would not pass the opportunity to try it out. :-]

This week's reading list

A covert distributed file system implemented on top of hacked printers.
http://www.remote-exploit.org/wp-content/uploads/2011/03/Printers-Gone-Wild.pdf
Video here: http://www.remote-exploit.org/?page_id=764

A more generic, yet much longer and deeper printer hacking presentation. Included in the discussion are the issues of firmware infection and remote attacks to printers with malicious physical consequences.
http://archive.hack.lu/2010/Costin-HackingPrintersForFunAndProfit-slides.pdf

Cisco's guide of IPv6 for dummies. This is a long PDF presentation that is well worth the time to go through.
http://ipv6forum.se/wordpress/wp-content/uploads/2009/01/ipv6-for-dummies-se-090120.pdf

TCP hijacking state of the art (in the context of proxy services)
http://www.squid-cache.org/~adrian/talks/20080510%20BSDCan%20TCP%20Hijacking%202.pdf
Complements well with this tool: http://intrepidusgroup.com/insight/mallory/

fun!

eroded compact disc

Metallica: the black album, an album of a very rare breed of musical work. I bought this CD a long time ago - around 1994 (about 17 years ago). Shortly after I bought it, maybe a year or so, I noticed these tiny cracks appearing around the edges. Those tiny cracks have been growing as if erosion or corrosion has been taking place in the reflective material.

eroded compact disc?

No other CD I own has ever had this kind of issue. I'm guessing this was a manufacturing defect, perhaps a one-off or an entire batch, who knows? The last track doesn't play any more - there's no reflective material left to cover the entire surface that contains it.

Isn't it ironic that this kind of physical wear would develop in a compact disc - one of the most robust digital support ever made, originally developed and aimed at the consumer market as reliable media to record and playback music? The fact that this was a one-off in my collection and that a (quick) google search revealed nothing of this kind probably means that this is indeed rare.

But a curious one, nonetheless...

long traceroutes

Physical distance and network distance have an interesting relationship. Even though intra and inter-city links can be horribly slow and therefore not a good measure of physical distance, the same is not as true for international and inter-continental links.

From the UK (a.a.a.a) to Japan (z.z.z.z) via the Internet:

traceroute to z.z.z.z (z.z.z.z), 30 hops max, 40 byte packets
 1  b.b.b.b (b.b.b.b)  0.710 ms  0.606 ms  0.636 ms
 2  c.c.c.c (c.c.c.c)  3.065 ms  2.447 ms  2.817 ms
 3  d.ukcore.bt.net (d.d.d.d)  2.810 ms  2.793 ms  2.483 ms
 4  e.e.e.e (e.e.e.e)  7.235 ms  6.432 ms  7.068 ms
 5  f.ukcore.bt.net (f.f.f.f)  6.449 ms  6.360 ms  6.221 ms
 6  * g.eu.bt.net (g.g.g.g)  6.151 ms  5.906 ms
 7  h.eu.bt.net (h.h.h.h)  88.322 ms  88.575 ms  88.227 ms
 8  i.eu.bt.net (i.i.i.i)  88.652 ms  88.257 ms  88.850 ms
 9  * * *
10  j.j.j.j (j.j.j.j)  164.092 ms  163.958 ms  163.862 ms
11  k.kddnet.ad.jp (k.k.k.k)  151.157 ms  150.945 ms  150.999 ms
12  l.kddnet.ad.jp (l.l.l.l)  262.545 ms  264.215 ms  262.769 ms
13  m.kddnet.ad.jp (m.m.m.m)  284.810 ms  278.666 ms  276.577 ms
14  n.kddi.ne.jp (n.n.n.n)  270.713 ms  270.666 ms  282.805 ms
15  o.kddi.ne.jp (o.o.o.o)  278.036 ms  278.105 ms  278.458 ms
16  p.kddi.ne.jp (p.p.p.p)  269.161 ms  266.220 ms  270.104 ms
17  q.kddi.ne.jp (q.q.q.q)  279.407 ms  283.477 ms  279.755 ms
18  r.r.r.r (r.r.r.r)  295.864 ms  266.645 ms  267.897 ms
19  s.s.s.s (s.s.s.s)  268.964 ms  268.345 ms  267.745 ms
20  z.jp (z.z.z.z)  265.074 ms  275.107 ms  263.744 ms

The key hops are 6 to 7 (~82ms) [UK-Europe], 8 to 10 (~76ms) [Europe-Japan] and 11 to 12 (~110ms). These represent the respective network distances of the links between UK and somewhere in Europe and between somewhere in Europe and Japan.

From the UK (a.a.a.a) to Australia (z.z.z.z) via a private link:

traceroute to z.z.z.z (z.z.z.z), 30 hops max, 40 byte packets
 1  b.b.b.b (b.b.b.b)  0.522 ms  0.491 ms  0.827 ms
 2  c.c.c.c (c.c.c.c)  2.811 ms  2.788 ms  2.753 ms
 3  d.d.d.d (d.d.d.d)  0.567 ms  0.543 ms  0.515 ms
 4  e.e.e.e (e.e.e.e)  0.467 ms  0.771 ms  0.556 ms
 5  f.f.f.f (f.f.f.f)  6.864 ms  8.488 ms  11.544 ms
 6  g.g.g.g (g.g.g.g)  305.581 ms  306.083 ms  306.598 ms
 7  z.z (z.z.z.z)  305.507 ms  305.506 ms  305.377 ms

In this case, the important hops are 4 to 5 (~8ms) [likely intercity] and 5 to 6 (~295ms) [obviously intercontinental].

An interesting trivia to take away from the latter case is packets travelled half the world and back at a difficult-to-conceive 408 216 000 km/h or 253 653 663 mph. Or in a scientific perspective 0.378*c [where c is approximately the speed of light in vaccum].

My trusty UDP packets travelled half the world and back at an amazing ~37.8% of the speed of light! That's including routing and other network type processing, variable speeds of propagation trough different mediums, etc.

robust internet connection 1/2

Until recently, my overkill home network had two routers with DSL cards in them. An 877W and a 2621XM with a WIC-1ADSL.

The 870 served as the outside router, providing an Internet connection via aDSL to my ISP. I have OSPF running between the two routers, including default route distribution (default-information originate) from the outside router, informing other routers about an Internet access path.

Typical setup. Cisco 877W is Internet gateway and responsible for NAT. Traffic is routed via 2621XM.

As I was fiddling with routing protocols and had multiple DSL cards, one question came up: could I have such a setup that if I swap my DSL cable from one router to the other, the user network can keep accessing the Internet without having to configure/reconfigure anything?

I thought so, and I can! What's required?

1. A routing protocol to update the routing tables of both routers if such a change would happen. done: OSPF. But now the core router would also have a default route to distribute should it become the Internet uplink (so default-information originate on it too);
2. Redundant NAT configurations, so both routers perform address translation in the same way: can do;

Alternative setup. Cisco 2621XM becomes Internet gateway, also responsible for NAT.

So after a little while, I got it going. Sadly, the WIC-1ADSL is far far from being as good as the 877W's DSL chipset, so Internet performance suffers a good bit. However, the exercise here is technological and it is about redundancy and/or robustness (albeit partially manual, as I have to switch the cable between routers).

... to be continued.

Power delivery

Power delivery, originally uploaded by pjvenda.

The final link between drive train and rear wheel.

St Michael's mount

St Michael's mount, originally uploaded by pjvenda.

The narrow twisty footpath into the castle.

Spotify maturing as a business?

Spotify is a great product, I use it every day. It's an online music streaming service available for a number of countries in Europe (sorry, Portugal is not in the list yet... shame on you Spotify).

Warning: This is my own opinion... a long one.

I have been using it since its early days, when signing up was open and free and their ads seemed "amateur" at most. They even had a "Spotify voice mail" into which users left their messages of spotify-glory eventually ending up in self-advertising ads.

Discussions with business aware people led me to believe that there is something not quite right with their business model. To users it was music on tap for free, high quality, high availability, nagging ads but nothing we couldn't live with. I know that the key of success in online services is the ability to create a critical mass of users. Hence it made sense to give away music on tap for free.

Starting a business is difficult and incurs a lot of risk. Most technology companies have significant losses during the first years of operation, but often after braking even, the gains far outweigh the losses. In other words, losses in this case == investment.

But this is not sustainable forever and something was bound to happen. They would have to change the revenue model or increase the density of ads (a lot!) in the free accounts to make people switch to the paid version or go bust or be backed up by a company that made money some other way...

Adding to this, they implemented some serious anti-tampering stuff into their windows client since the very initial versions, including fancy anti-debugging and obfuscation code. Clearly they either over-engineered it or just prepared for a longer-term reality... Not that it is a bad thing, implementing decent security measures since day one, but it was also a visible sign that they did not want people cracking it.

The way I see it, there's a thin line between giving away enough to attract interest and new users and charging enough (from whatever multiple sources) to keep the business afloat, or at least following to the almighty business plan. Giving away stuff keeps people happy and attracts new users, etc, but does not pay the bills. Shutting the service to paid customers would kill the expansion of their user base. I think they never got this balance exactly right (assuming of course it was possible in the first place).

Over the past year or so they introduced various changes that affect the afore-mentioned balance. Their initial model was something like:

• Paid ads;
• Free accounts to everyone: Lots of music available, ads that could not be skipped;
• Premium accounts for £9.99 per month: No ads, higher quality streaming;

Shortly after the following features also appeared:

• Paid ads: Arbitrary companies were able to advertise in Spotify, spamming free accounts forever. Artists were also advertising their work via Spotify ads;
• Premium accounts: Offline mode, mobile access for some smartphone platforms and unrestricted international access, even higher quality streaming, invites were provided;
• Free accounts I: access out of allowed countries was limited to 14 days, [the mac client paused ads if sound was muted - sneaky, eh?];
• Mobile access: Various applications were created and deployed for the most widely used smartphone platforms (iPhone, android, any other?). This is, of course, restricted to premium accounts;
• Free accounts II: All existing free accounts kept working normally. However, free accounts could no longer be created without an invite. Invites were being handed to paying users and could be used as tokens to create new free accounts. So free accounts stopped being free, essentially;

Now the free user base could only expand with the help of the paying user base. Clever! The number of invites given to premium account holders was now controlling expansion of the free user base. So if you didn't have an account, you'd be left with 2 choices: either find someone with a premium account and get an invite from them or buy a premium account yourself.

Also there was little choice to the user base. Either you have a free account with nagging ads or you pay £9.99 per month... or the ridiculous £9.99 for one day. There weren't too many commercial ads, most were artists promoting their music/albums/singles/whatever and the rest were self-advertising. Targeted ads seem to work so-so. Some ads my wife gets I never heard and vice-versa. However, I still get spammed by Rhianna's promotions... something's very wrong there... My playlists are little more than  Iron Maiden, Bruce Dickinson, Megadeth, Manowar... you get the picture...

Fast forward to 2010 Q1/Q2 (IIRC) and here's some more changes. I think these prove what I described above. Spotify decided to press on and attempt to expand its paying user base now. This was expected, and as far as I'm concerned, overdue by now. What we have now is:

• Paid ads: Arbitrary companies were able to advertise in Spotify, spamming free accounts forever. Artists were also advertising their work via Spotify ads;
• Free accounts: Left untouched. All free accounts are kept working, but can only be created with an invite coming from a premium account;
• Premium accounts: Mobile access for some smartphone platforms and unrestricted international access, even higher quality streaming;
• Integration with locally stored music files: The Spotify player can now play locally stored files and even share them with mobile clients. I don't know exactly how and when this works, but it sounds 'local'. Nonetheless, it makes sense to me;
• Open accounts: Free version of Spotify, no invite necessary, but limited to 20h per month. Like a demo, really;
• Unlimited accounts: Poor man's premium account. Half the price of a premium account but no mobile access, no offline mode and no streaming abroad. Pretty much all the good features gone except for the music, of course, and with no ads;

There it is, another shift towards expanding their paying user base. This starts to open up the choice range of non-paying customers and the game just might start changing. I'm not yet willing to invest £60 per year on a Spotify Unlimited account (the premium is still too much), particularly because most a lot of the music I enjoy is not there (which I reckon is mostly my own problem). However, it begins to become more and more reasonable to a wider and wider range of users and potential users.

I must also say that my ramblings about expensive and cheap are, of course, relative. What's expensive for me may be very cheap for you and vice-versa.

Good stuff, Spotify! Now get those label contracts going (get Rammstein back on the lists please, btw), as well as Metallica, AC/DC and every other song in the world :] A bit like Google! Hmm.....