Search folders on Thunderbird

Hi people,

Thought I should share something I found today. Mozilla Thunderbird's "Saved Searches".

Well, I found it because I googled for it because I needed to start using it. And I was fairly sure that Thunderbird would have some support for search folders.

Saved searches are virtual folders that contain the result of a permanent search filter through a pool of searchable data. These virtual folders are just a way to represent search results in a well known and user friendly way.
Since we are talking about Mozilla Thunderbird and it deals with email, Saved Searches in Thunderbird are folder representations of search filters performed on sets of email messages. Currently, the search filters for saved searches include the following criteria: Subject, From, Body, Date, Priority, Status, To, Cc, To or Cc, Age in days, Size, Tags, Attachment status, Junk status and a customisable arbitrary set of headers.

Ok, great, but how is it done? I mean, if it was intuitive, I would have found it by myself - no googling required... But it is not that intuitive, really...

Here's a quick step-by-step guide:

• Step 1: Select a mailbox or an Inbox folder. Your saved search will be placed under this initial selection:
  

• Step 2: Select menu "File" - "New" - "Saved Search...":
  

• Step 3: Fill in the name of the folder and configure your search filter. I called it "Important messages" and I chose to search all messages tagged as "Important":
  

• Step 4: Select the real mail folder(s) from which messages are to be be searched. I chose my Inbox. Multiple folders may be selected:
  

• Step 5: Find your Search Folder under selection made in Step 1 with the respective search results updated whenever it is opened. See my important messages?:
  

Not as straight forward as being able to create a sub folder of the type "Saved Search" and configuring it from there...

Interface issues aside, this is still a great feature that is probably found on all decent mail clients.

Credit where credit is due. The following was the blog post that answered the question I googled for:

• http://www.bartbusschots.ie/blog/?p=385

Cheers, PJ.

Reading queue

Hey everyone,

Thought I should advertise what nice books I got this week and what is my current reading queue. I am not a book guy, I don't read books frequently, but recent events (my birthday) have built a reasonable queue (for my standards):

• Richard Hammond: On the edge: My story (ISBN 0297853279);

My wife offered this book to me because she knows I admire Hammond and I too am a petrol head. She always gives me the best gifts... How does she do it? Thanks Mrs V.

Richard Hammond is one of the presenters of BBC's Top Gear - a program for petrol heads. Last year he had a major crash on the "Vampire" jet powered car just after breaking the British land speed record previously set by the same car. The crash inflicted serious brain damage and recovery would take a long time. He got better quickly and returned to his job in Top Gear, exactly as before. This book is his view of his (risky) job, the story of the accident and recovery, his comeback to Top Gear, his reasons for risking his life, very nearly loosing it, go back and keep doing it.

• John Freeman: Collins Digital SLR handbook (ISBN 0007242107);

My friends gave me this one. I am also looking to learn more and more about photography and this will make a valuable addition to my knowledge. Thanks guys! Brilliant stuff!

• Lucian Gheorghe: Designing and Implementing Linux Firewalls with QoS using netfilter, iproute2, NAT and L7-filter (ISBN 1904811655);

Beware! Highly geeky stuff inside! I bought it because Linux x Firewalls x QoS is one of my fields of interest. One which I haven't spent enough time investigating, hence the book. This is also a reference book, but I will give it a patient end-to-end read before getting my hands dirty. This is not my typical approach to learn on these kinds of fields... A gift from me.

Cheers, PJ.

Curious warning message: password too secure?

This happened just a couple of weeks ago. I was trying to log in to some remote host to which I haven't connected for some time. So my password had expired and I was asked to change it. Fine. But after changed it, I got this warning saying that my password may be too complex for older windows systems.

"The password is longer than older versions of Windows, such as Windows 98 or Windows 95, can use. Press Cancel to enter a new password, or OK to proceed with this password." - Click on the screenshot to enlarge.

This is a warning message, so I am explicitly being made aware that my password is too strong. Knowing that the host is (a) running Windows Server 2003 and (b) not in a domain, let's consider this message for a moment - let's think about the possible meanings of it:

• Should I choose a simpler password because I may access some service on one of these "older" systems with my current credentials?
• Should I choose a simpler password because I may access this service from one of these "older" systems with my current credentials?

I understand that this is a result of a forever-standing compatibility policy. But let's be a bit realistic... Windows 98 or Windows 95? Who, in their right mind, would use Windows 95/98/Me in whatever corporate environment that absolutely requires inter-(windows)-system compatibility even in very controlled and very internal networks??

Sorry for bashing on Microsoft again, but this is just plain silly.
I know that I am given the choice of keeping compatibility or not, but that fact itself tells me that Windows 2003 Server retains compatibility with such older systems.

Security systems evolve because older ones are found insecure and thus made obsolete. This kind of warning message just leaks that Microsoft prioritises retro compatibility over breaking older APIs, even if important security enhancements are at stake. I guess it is just better for business.

Cheers, PJ.

My new laptop: Apple MacBook Pro

Yes, I have (finally) surrendered to the mighty Apple. They have assimilated me.

For that latest 4 months or so I have been stuck with my older Acer Travelmate 4001Wlmi of September 2004. It is still a decent laptop, but the absence of a working USB controller, the tired screen (low and non-uniform brightness - "stained") and the inability to run Flight Simulator 2004 decently with some addons made me rush the decision to buy a new laptop.

So I did, and in style. I bought a shiny new 15" MacBook Pro [@Apple website / @wikipedia:Macbook_pro]. I really dig Apple hardware's look and feel, their approach to styling, the kind of materials they employ, even the packaging is a small work of art.

I chose to buy a MacBook Pro (even though I still think they are a slightly over priced for their specs) due to the following reasons (in no particular order):

• It's a light weight and well built 15" wide-screen laptop;
• The Intel Core 2 Duo [@wikipedia:Intel_Core_2] with 2GB of RAM looked really appealing as a Gentoo Linux desktop platform;
• nVidia GeForce 8600M GT [@wikipedia:Geforce_8] with 256MB of VRAM. Important for Half Life 2 and Flight Simulator X (with addons);
• Mac OS X [Tiger@Apple website / @wikipedia:Mac_os_x];
• High resolution, LED backlit screen;
• Apple store offers great deals for students.
  

I ordered it from Apple Store UK along with an iPod classic for my wife and it took 17 calendar days to arrive. I must admit that the wait was a little hard to accept, but all is well now.

First impressions were great. Intelligent and stylish packaging (as is common with all Apple products), battery charged at about 75% and Mac OS X 10.4 Tiger pre-installed. Apart from laptop and charger there was a Quick Start guide, a DVI-VGA cable adapter, infrared remote control and CD/DVD pack with operating system and software extras.

I first saw Mac OS X when I worked at RNL [@my website] where some of my fellow sysadmins owned iBook laptops with OS X. We also had to handle a couple of older iMacs running Mac OS 9, which gave me an idea of how much better OS X was when compared to OS 9. My wife also has a macbook for some months now, so OS X is not new to me, although I never used it on an daily basis.
Being a UNIX-head geek, I tend to see Mac OS X as if it is on top of a pedestal. It is by far the best achieved OS ever made and it runs on top of UNIX. Does it even get any better? Yes, with every release! Now that finally I have a system where I can legally run Mac OS X, all I can do is agree on myself. Now I think that I took too long to make this switch. Man... is it worth it!!


While I believe that Mac OS X is brilliant, it does not mean that I like to use all the applications Mr Steve Jobs would like me to use. My every day system is still Linux and I still play my heavy graphics games... sooo... I had to infect my MacBook Pro with a (legal) copy of MS Windows XP (exclusively for games) and I also had to install a Gentoo Linux distribution (mainly because of amarok, digikam and because my work flow for producing panoramas involves a set of [free] programs that I still don't know if I can use on OS X).

I will have some detailed notes on installing the two extra operating systems on my website on a stand alone article - "Yet another guide for setting up Triple Booting on a MacBook Pro".
Just some notes:

• rEFIt is incredible. It's trivial to install and use (no configuration); It's clever (finds out about bootable partitions and CDs by itself); It's pretty (see screenshot below);
• The Windows XP installer is an incredibly limited, unbelievably useless piece of software. It's utter crap. Right after I got the installer to do its job (install windows) I rebooted a Linux Live CD and used ntfsclone to save the freshly installed windows partition (filesystem). When I need to reinstall the system, I'll just rewrite my windows partition with that file system;
• After the pain of installation and after installing the corresponding hardware drivers, windows XP runs very well. Can't wait to test HL2 Ep1 on it;
• Gentoo Live CD 2007.0 does not have a driver for the ethernet network card or the wireless card, so before compiling a recent kernel (>=2.6.21) I had to do stuff without internet connection;
• Linux on this laptop is frighteningly fast! Probably because I am used to run Gentoo on my previous laptop laptop or on VMWare hosts... Now that I think of it, my Acer Travelmate 4001WLMi has a 1.5GHz Intel Pentium-M (banias) processor with an FSB of 400MHz, while this MacBook Pro has 2x 2.4GHz Intel Core 2 processors with an FSB of 800MHz. That alone is a big performance boost.
  

My website contains a page with Linux compatibility notes and tips about every one of my laptops (Linux on Laptops). The MacBook Pro will not be an exception. However, the page isn't up yet. This section will be part of a huge update long long overdue. When? Don't know yet... Soon?

References:

• Mac OS X 10.4 Tiger:
  • http://www.apple.com/macosx/tiger/
  • http://en.wikipedia.org/wiki/Mac_OS_X_v10.4


• (Coming very soon now) Mac OS X 10.5 Leopard:
  • http://www.apple.com/macosx/leopard/
  • http://en.wikipedia.org/wiki/Mac_OS_X_v10.5


• Website of Amit Singh - the most notorious OS X hacker of all times. This is also a bottomless pit of OS X information.
  • http://www.kernelthread.com


• Mac OS X Internals: A systems approach, by Amit Singh.
  • http://www.osxbook.com/


• "How to Switch to the Mac", blog post by a fellow Portuguese blogger:
  • http://the.taoofmac.com/space/HOWTO/Switch%20To%20The%20Mac

Cheers, PJ.

Priority to Windows Media Player in Vista, over Network and I/O operations

Hardly recent news, but this one of those issues where marketing decisions seem to win over good architecture design. Trouble often awaits at the end of a shortcut...

People found out first through occasional events and then through more focused and scientific experiments that the Windows Vista scheduler deliberately (and grossly) gives CPU priority to its Media Player, against all other processes - even network and I/O operations!

But let's go for a little walk around the scheduler architecture theory park before going back to windows vista...
All of today's operating systems are multi tasking, simulating an ideal environment where each process has its own CPU. Inside every operating system's kernel there is a scheduler which job is to allocate bits of CPU time to all processes. Schedulers employ very complex algorithms to distribute an average CPU time as evenly and/or fairly as possible while keeping maximum response times for sake of interactivity and avoiding resource starvation issues (where a small group of processes keeps an unfair amount of CPU and I/O resources for themselves, leaving other processes waiting forever or, in OS jargon "starving".

I am not a scheduler guru, but I know enough to understand that they are damn hard to get right. Tuning a scheduler is an exhausting iterative task that requires great technical knowledge and creativity to come up with clever optimisations. The unsolved problem with schedulers is that nobody ever found a perfect algorithm for all kinds of usage scenarios and load patterns. Probably because such scheduler does not exist!
Playing games, browsing the web, editing audio and/or video and accessing compressed media (generic media players) are perfect examples of very different usage scenarios that happen often on the same hosts. Each has a unique set of requirements and needs a different amount of system resources to itself.
Multiple mechanisms and techniques were invented to handle this broad range of patterns as fairly as possible. In some cases schedulers categorise and tune their settings individually to each process by learning its patterns and inferring its specific category. Some other cases though are affected by users where given processes are explicitly marked as having higher priority or otherwise. With these techniques, the scheduler tries to tune itself globally to the actual usage scenario that it needs to handle. Computer scientists are working on it and slowly coming closer to the goal of the perfect scheduler that handles well every usage scenario thrown at it.

This, in general and qualitative terms, is what is involved with every scheduler in every multi tasking operating system. The source code for some schedulers is open and can be accessed, changed, tuned, made better and contributed back to the project. It is not an easy task though! As I mentioned before, schedulers are made of several elaborated algorithms that take time to locate, contain, read, understand and finally understand (this is not a typo).

Other operating system vendors, however, CHEAT!
Microsoft decided to give a higher priority specifically to its media player software, to improve interactivity and responsiveness, hopefully regardless system load. This is obviously the wrong kind of scheduler tuning probably derived from marketing and "strategy" personnel, who clearly don't know much about schedulers.
How is this wrong? Recalling that the scheduler tries to make a single CPU host behave like it runs multiple processes at the same time, increasing priority on one process is only made at the cost of performance on all other processes. This is wrong because Microsoft is giving an unfair advantage to their media player software, perpetuating their locking techniques.

The technique seems simple to me: hard code priority boosts to their own software to make all others run worse. People obviously prefer those that run well and will eventually stick with Microsoft technology. Today it's their media player... What about tomorrow? Internet Explorer?? Imagine Firefox taking 2 minutes to start or 30 seconds to load a flash based page? I would surely not use it in such conditions.

This case is serious because one of the processes that suffers a priority decrease and a notable performance decrease is related to the system's network I/O handlers. Network traffic is seriously slowed when using Microsoft media player on Windows Vista.
... but believe me, music and videos play ultra-smoothly!

References:

• First public report/discussion of this problem (2007-06-25):
  
  • http://forums.2cpu.com/showthread.php?t=83112


• Adrian Kingsley-Hughes' takes on the subject. These articles demonstrate with some simple graphics just how much the Vista's scheduler hurts network performance:
  
  • http://blogs.zdnet.com/hardware/?p=702
  • http://blogs.zdnet.com/hardware/?p=709
  • http://blogs.zdnet.com/hardware/?p=717
    


• Mark Russinovich's excellent technical explanation:
  
  • http://blogs.technet.com/markrussinovich/archive/2007/08/27/1833290.aspx


• Robert Love's comments:
  
  • http://blog.rlove.org/2007_08_01_archive.html


• Microsoft's official press reply (2007-08-25), reported by Adrian Kingsley-Hughes:
  
  • http://blogs.zdnet.com/hardware/?p=724
  • http://www.betanews.com/article/Microsoft_Responds_to_Rediscovery_of_Vista_Network_Slowdowns/1188244273

Now let me explain that I am yet to install windows Vista on any of my hosts due to several different reasons:

1. As happens with anyone else, my time is limited. I use it to research my own fields of interest and to enjoy my life as much as I can. The vast majority of Microsoft's products do not fit into any of the mentioned categories (a notable exception is Flight Simulator);
2. My every day operating system is Linux, so testing windows Vista would be for experimental purposes only. However, there are other operating systems in queue for me to experiment with before Vista;
3. I refuse to pay directly for a copy of windows Vista, therefore, unless there is some other legal way for me to get a licensed copy I am keeping XP. Maybe when I buy a laptop or a workstation that is only sold with a Vista pre-installation... maybe I'll give it a good long-run test;
4. Flight Simulator X seems to run decently on XP;
5. Oh, haven't you noticed? Vista's scheduler is broken.
   

Cheers, PJ.

My new toy: Globalsat DG-100 GPS Data Logger

Being a geek, I need my geek toys. My latest is a GPS Data Logger, that I bought to assist me in geotagging my photographs.

A GPS Data Logger is a portable device that periodically records geographical coordinates of its location, from a received GPS signal. The list of logged data points is meant to be dumped later to a computer.

These devices are used to track hiking or cycling paths, to help with wardriving, to track touristic walks or to help with geotagging photographies. The latter is my main purpose for the data logger I bought.

Globalsat DG-100:

The Globalsat DG-100 is a very nice small and affordable (low end) GPS data logger. According to the reviews and comparisons from GpsPassion, PocketGPSworld and from Richard Ackerman's blog, I chose this logger among others because of the following key features:

• Time-based or distance-based periodic logging or GPS locations. The basic function of a GPS data logger;
  
• On-demand recording of current location. In addition to periodic logging;
• SiRFStar III chipset: sensitive enough to give me decent GPS signals in non-ideal conditions (cloudy weather, in-house, etc). This is a fundamental feature because without GPS signal, a data logger is no different from paper weight... but that's obvious;
• Long battery life - reportedly up to 36 hours or continous usage;
• USB interface;
• Interface software allows extraction of the logged data into a number of file formats. This is another fundamental feature because although "Globalsat Data Logger PC Utility" is a windows-only tool, I intend to use the logged data on some other non-windows programs (like open source ones). This is actually irrelevant because I already found an open source and portable program that interfaces with the device and is able to extract data from it. Whoohoo!!;
  

Added value features:

• Plug for an external GPS antenna;
• 2 rechargeable AA batteries included. The USB connector charges them. It would be a shame if it did not...;

Workflow:

1. Make sure that my camera's clock is correctly set;
2. Walk around with my data logger while on photographic raids. Try to log the specific spots where I shoot pictures;
3. Dump data into a GPX file;
4. Dump my pictures into some folder;
5. Use digikam with the gpssync plugin to match photos with GPS locations. Matching is made from timing information. The picture IMG_NNNN.JPG, shot at the time "hh:mm:ss" has the location obtained from the GPX log at the time equal or closest to "hh:mm:ss";
   

Linux Support:

By looking at the supplied software, it becomes apparent that the device communicates through a serial to USB line (Prolific PL-2303X USB-serial adapter). There is a driver in the Linux kernel tree for that USB-serial adapter called pl2303, so no problems expected regarding chatting with the device. The final piece of the puzzle is some program that knows the protocol to communicate with the DG-100.

After a quick research, I found that Linux offers quick and easy support for a large number of GPS devices and that there are many tools that easily interface with them to provide different functionalities, obviously related to GPS positioning.

gpsd does not work

The clever way to achieve this is by using a consistent TCP to GPS interface called gpsd. One one end, gpsd contains a number of drivers and parsers for the many different GPS devices it supports. On the other end, it serves requests to client applications from a TCP socket, using the help of an interface independent of the GPS device in use.
Clever, but does not support the DG-100 yet... The list of GPS devices supported by gpsd is big but as all such tools, not complete. Being an open source tool, gpsd is developed by contributors that give their own free time to improve it further. Hopefully support for the DG-100 will be included by a friendly contributor.

bug gpsbabel does... or will, soon:

gpsbabel aims to be a multi-platform universal translator between many GPS data formats. It is a different tool than gpsd, but incidentally, it also contains drivers and/or parsers for a number of devices. gpsbabel is used by several other tools to perform GPS data translations.
The current version of gpsbabel is 1.3.4, which does not include support for the DG-100 as well. But good news is that Robert Lipe commited code with initial support for Globalsat DG-100 on 1st September 2007. The code is copyrighted to Mirko Parthey and Robert Lipe and released under the GPL license. The driver is now in its revision 1.17 (2007-09-13) and is likely to be included in one of the next gpsbabel releases.

So, to use gpsbabel with support for my DG-100, I checked out the HEAD code and compiled it on my host. It compiled fine and I was able to pull the data records from the device into a file. Woohoo!

Just a quick guide. I already had the module pl2303 so as soon as I plugged in and turned on the device, the kernel reported its presence:

usb 6-1: new full speed USB device using ohci_hcd and address 2
usb 6-1: configuration #1 chosen from 1 choice
usbcore: registered new interface driver usbserial
drivers/usb/serial/usb-serial.c: USB Serial support registered for generic
usbcore: registered new interface driver usbserial_generic
drivers/usb/serial/usb-serial.c: USB Serial Driver core
drivers/usb/serial/usb-serial.c: USB Serial support registered for pl2303
pl2303 6-1:1.0: pl2303 converter detected
usb 6-1: pl2303 converter now attached to ttyUSB0
usbcore: registered new interface driver pl2303
drivers/usb/serial/pl2303.c: Prolific PL2303 USB to serial adaptor driver

Udev politely created /dev/ttyUSB0 (serial port device file) automatically for me. Then all I need to do is:

• gpsbabel -i dg-100,erase=0 -f /dev/ttyUSB0 -o gpx -F latest-dump.gpx

Pulls all the records from the data logger (reads from USB-serial device: -f /dev/ttyUSB0 in dg-100 format: -i dg-100) to a file called latest-dump.gpx, in GPX format.

Prefer KML instead of GPX? Fine:

• gpsbabel -i dg-100,erase=0 -f /dev/ttyUSB0 -o kml -F latest-dump.kml
  

Or alternatively convert the file between GPX and KML:

• gpsbabel -i gpx -f latest-dump.gpx -o kml -F latest-dump.kml
  

Dead simple!
(the used gpsbabel binary was, of course, result of a CVS HEAD checkout compilation done in 2007-09-22, because as I referred, the dg-100 driver has not yet been included in an official release)

Final words:

Although I have not tried any other data logger, I would seriously recommend the Globalsat DG-100 unit for everyone that would make use of a GPS data logger. It is in fact small, battery efficient, it has good signal levels even inside supermarkets, I can use it on Linux and it fits my purpose perfectly.

As one of my work mates usually says: "Brilliant stuff!"
Cheers, PJ.

proactive security - software inspector tool

Recently I found a really nice software inspector published by the respected Secunia free for personal use.

Secunia Personal Software Inspector (PSI)

The software inspector is a tool that lists all your installed programs and extracts version numbers for each one. After that it verifies the list against secunia's database of vulnerabilities.
A score is then calculated to indicate the up-to-date percentage of your system, along with a summary of all the programs that should be updated due to expected security flaws. For many programs, PSI will produce a direct link to download the related update file. A low percentage means a high security risk because software versions are old and known to have vulnerabilities or other security issues. On the other hand, a high percentage means that the system's software in general is up to date making the host less prone to be successfully attacked.


Additionally, an agent remains running in the background maintaining the software list updated and reacting when you install, upgrade or downgrade programs, informing or advising accordingly.

http://psi.secunia.com/

I would seriously recommend this inspector for every windows workstation connected to the internet.

Cheers, PJ.

- I am in no way associated with secunia nor I benefit in any way from advertising their product.
- Updated in 2007-09-20 to put a differente screenshot. this does not show a 100% score and suggests an update to a program.

the worst blogger ever

... must be me!

This is the ninth month of the year 2007. So far this year I posted 11 times on this blog (including this post). That yields an average of 1.22 posts per month (standard deviation of 1.91... sorry about that, never mind).

Personally, I consider this unacceptable.
It is not that I consider that every blogger that does less than N posts per day/week/month/year is a bad one - It is just that I am sometimes too busy and other too lazy to post my thoughts as I find them relevant for the blog.

Anyway, I have about 5 or 6 posts in preparation, so let's see if I can publish them all in September to improve my statistics :)

Cheers, PJ.

My list of Firefox extensions

Hello everyone!

It has been a long time since I have written anything on my blog, as unfortunately it is becoming a habit... But believe me, I have about 10 half written/in research posts to put here, so by all means, it is not over.

Although it may have plenty of flaws, Mozilla's Firefox is clearly one of the best, most ubiquitous browsers of today's offer. [This can be seen as an almost religious, unjustified sentence, but while I respect other people's opinion, I also expect mine to be respected.] It is open source, multi platform, actively maintained, does what it was designed to do very well and widely used - just to name a few of the key features that make it such a good browser. Its API allows companies and the community to design extensions that change/add/remove any functionality you might dream.

My post today may seem like "deja-vu"... it is... sort of. There are maybe thousands of ".* firefox extensions" posts around the internet, mostly in blogs. However I could not care less, I tend to forget things pretty easily, so I decided to list the firefox extensions that I use, divided by categories (for your convenience, and mine). Of course such topic is not at all static. Some extensions dissapear (stop being developed by whatever reasons), some change names and new ones eventually replace other older extensions due to improved design and funcionality (a perfect example of this is FireBug vs Web developer). This list was gathered on Firefox 2.0.0.4 and is/was up to date in July 2007.

Generic / essential:

Generic/essential extensions are the ones that I fetch and install before browsing any website, just after installing the browser on a host that I will be using.

• British English Dictionary: The British English Dictionary.
• DownThemAll!: A download manager for Firefox.
• Tab Mix Plus: Extra features for tabbed browsing.

Not essential:

Not essential extensions make the browser more visually pleasant, maybe less cluttered and perhaps with some extra (useless for common users) information.

• Personal Menu: Replaces some menus and/or sub menus by easily accessible buttons on the address bar.
• Server Spy: Indicates what brand of HTTP server runs on the visited site.
• Fission: Progress bar in the address bar (Safari style).
• Shazou: Tracks the geo location of the current page, displays it on a separate window with google maps and supplies some extra information about the domain and ip block owner.
• ShowIP: Shows the IP address of the current page on the status bar.

Security / privacy:

Security and/or privacy extensions help keep your privacy, to get around simple and elaborated data mining techniques and to help avoid XSS and phishing attacks.

• Passive Cache: Retrieves a cached Google page of a link, as well as an Archive.org page, with no images and no connections to the remote site.
• Redirect Remover: Removes Redirects from Links and Images.
• RefControl: Control what gets sent as the HTTP Referer on a per-site basis.
• SafeCache: Defends against cache-based web privacy attacks.
• SafeHistory: Defends against visited-link-based web privacy attacks.
• Firekeeper: Intrusion Detection and Prevention System for Firefox.
• FireGPG: An extension that acts as an assistant for using GPG. The main purpose of this tool is to sign and/or encrypt emails with GPG on the gmail interface directly.

Web auditing:

Web auditing extensions provide means to access and/or modify cookies and requests/replies made to/received from web servers.

• Add N Edit Cookies: Cookie Editor that allows you add and edit session and saved cookies.
• Tamper Data: View and modify HTTP/HTTPS headers etc. Track and time requests.
• HackBar: A toolbar that helps you find and test SQL injections. (not good for generic browsing - takes too much space).
• FoxyProxy: A proxy server manager. Can route requests to different proxies according to specified URI patterns. Allows quick selection/change/edit of different proxy servers and/or corresponding settings.

Web development:

Web auditing extensions are essential if you happen to do any (X)HTML/CSS "coding", javascript, or any other form of web development. These allow you to analyse different parts of the in-development or deployed static or dynamic websites. Firebug is absolutely unavoidable.

• Firebug: An incredibly good HTTP profiler, Javascript debugger, (X)HTML/CSS designer aid. Web developer was good, but does not compare to this.
• View Source Chart: Creates a Colorful Chart of a Webpage's Rendered Source.
• User Agent Switcher: Adds a menu and a toolbar button to switch the user agent of the browser.
• Total Validator: Validates web pages in numerous ways. Supports uploading the displayed page instead of passing the link to the online validators. Particularly useful to test internal/in development websites.

For blog posts:

To help me build this list automatically, I used the following extension, temporarily.

• Extension List Dumper: Dumps a list of the installed extensions.

This set of extensions enables a far more comfortable, private and secure browsing experience. Additionally, it supplies some advanced tools for my auditing work. And in case you are wondering, yes, I fully admit that I use far too many extensions.

Cheers, PJ.

physical security: places to stash money

All fields of Security are of great interest to me. I am still not sure why... Today I stumbled upon these really interesting eye-openers about stashing valuables at home.

• A good insight on the theory of tactics and targets
• Some good tips (given by a former burglar) on bad places to hide money at home
• A nice hack on the human brain
• Some more advice from a former burglar

Cheers, PJ.

I have done it! Now everything is going to change.

This is one of those events in my life certainly classified as "changes everything".

I am moving into the UK tomorrow (Saturday) and starting a new job on information security. I will be living with my better half in Oxford and working nearby. This means that everything in my life changes starting tomorrow. I am sure that most of the changes will be for the better!

The worse part of this is leaving my family, my friends and Portugal...

Cheers, PJ.

Using Linux without X

Luke, the author of "Terminally Incoherent" took some notes on his blog about some cool console applications. This post caught my attention (from osnews.com, I believe) and I decided to promote it further.

He believes that it is possible to survive on Linux without X and I agree, but that idea is now being avoided as much as possible by users and distributions. The idea of avoiding the console is not necessarily bad but, for people that are used to unices for a long time, it is still a powerful tool to do anything and an incredible loss if somehow it gets eliminated.

http://www.terminally-incoherent.com/blog/2007/05/21/a-day-without-x/

The short version of his post follows:

• elinks: web browsing (works well with javascript-less gmail) - I use sometimes.
• mutt: email client
• pine: email client - I used frequently.
• freetalk: jabber IM client
• naim: IM client
• irssi: IRC client
• raggle: RSS feed reader
• midnight commander: file manager
• zgv: image viewer
• mplayer: media player (supports ASCII output!!) - I use frequently.
• vim/emacs: text editing/coding/whatever - I use vim frequently.
• cplay: music player
• rtorrent: P2P (bittorrent) - I use frequently.
• twin: console window manager (YES!)
• screen: multiplex pseudo terminals onto one shell - I use frequently.

Update [2007-05-31]: links and a few extra apps posted.

Cheers, PJ.

scanning on linux

Hello Everyone,

Yesterday I needed to scan some documents and I borrowed an HP PSC 1110 from my sister. Knowing that the official HP drivers for that printer/scanner are distributed only in a 160MB bundle with all sorts of unnecessary software, I thought about it (for just a second) and decided to try to do my scans on linux. I do know that the HP PSC 1110 printer is damn hard to get working but would the scanner be tricky as well?

Let's get things started: scanning on linux == SANE (Scanner Access Now Easy)
So I fired up Gentoo's emerge utils and found two interesting packages: sane-frontends and sane-backends. Looks simple...

# emerge sane-frontends

brought along sane-backends as a dependency, as expected.

Next step: Detect my USB scanner
Ok, so sane-find-scanner seems to understand that there is some kind of a scanner attached to an USB port, but does not know anything else about it. scanimage -L complains about "no scanner found".

Perhaps a specific backend is necessary to talk to that particular scanner. So I went to take a look at the hplip (HP Linux Imaging and Printing) project. Indeed they have a backend for the PSC 1110 (and others) and all I had to do was to re-emerge the hplip package with the scanner use flag activated;

# USE="scanner" emerge hplip

This time hplip installed libsane-hpaio.so and changed /etc/sane.d/dll.conf to include the following line: "hpaio" - the backend for an entire class of HP scanners including mine.

Finally scanimage -L detects my scanner as something like:
device 'hpaio:/usb/psc_1100_series?serial=XXXXXXXXXXXX' is a hp psc_1100_series multi-function peripheral. Whoohoo!!

Time to scan! Multiple frontends may be used with libsane (list of frontends) but I used what is probably the simplest available: scanimage.

$ scanimage --batch-mode=yes > file.pnm

How cool is that??

If you do not like the terminal interface, you have xsane (with gimp plugin), kooka (kde's sane frontend), openoffice, scanlite (java), sanetwain (sane to twain bridge - for windows apps), ...

Cheers, PJ.

I am not alone!

"A very thourough and insightful analysis of the benefits of Linux vs. Windows without a lot of opinionated points. Great job!" - first comment to the blog post below, by Tavis

Things I can do in Linux that I can't do on Windows

Cheers, PJ.

media players, collections and corrupt digital music

Hello everyone,

Sometimes things go wrong and we (me in this case) do not have a clue of where to start looking for the problem to fix it. Sometimes because of lack of time, some other times because of lack of imagination and even some other times because of lack of resources.

About one/two years ago, I had a problem with AmaroK (media player, KDE) which caused it to quickly consume all available resources when rebuilding the music library. Exactly what I mean is that the application ate about all available physical memory within a minute or so and then it kept going until all swap space was also used. At that point, the kernel's OOM (Out Of Memory) procedure kicked in and killed amarok. When amarok would restart, it had the library marked as "unclean" and would itself reinitiate the library rebuild process... over and over again. That happened because of an amarok's bug - a known memory leak when handling .mp4 files. It got fixed and I carried on with my life listening to Iron Maiden over and over again with amarok.

Until recently (maybe four months ago) the issue came back. This time a little different: amarok was not being killed by the OOM but it still ate about 85% of total memory which means the computer became unusable due to continuous swapping. The symptoms were very similar which would indicate a similar cause. I looked at the KDE's buzilla but did not find anything relevant... Eventually, due to practical reasons, I stopped using amarok and stopped listening to music while working and playing, etc until I would find some free time to really dig the problem out.

Today I decided: No more! I will listen to my music!
Strategy: If amarok cannot, surely others can!
After verifying that JuK (another KDE media player) handled .ogg media, I fired it up and started to build my music library, only to see it mimic the amarok problem.
The most trivial test yielded the most important result: *the problem was not on the media player*!

For my next test, I started one of the media players and setup a very small collection. All went well, so either there was something in my collection that consistently crashed the indexing process or it was too big. It could not be too big - I see people with collections of 100GB+ of digital music frequently, so my 20GB could not be problematic!

Then I took my collection and split it into two roughly equal parts and put JuK rebuilding both parts, in turn - one of them crashed JuK and the other did not. *There was some problem somewhere in my digital music library!*
I kept using the bissection algorithm while carefully monitoring the media player's behaviour (to foresee the leak before the computer needed 5 minutes to open a terminal window) and after about six iterations I narrowed the problem in the library to one (well known) band, where I had four albums. By applying the same method again I found the album and the specific music file that was crashing the indexing done by both media players. I re-encoded the whole album again and copied the corrupted (?) file for later analysis.

In conclusion, imagination and creativity are just as important as technical resources and determination to solve even the simplest problems. The crucial test here was probably the simplest I could do - try another media player.

Now go do something else while I listen to "Louder than Hell" once more. After that it is time for "Brave New World". :)

Cheers, PJ.

NTP receives dynamic interface tracking

This is to all time keepers that happen to have the bad idea of reading my blog. I must admit I am a terrible blogger - I have about 5 partially written posts for about two months now and my posting average is dangerously close to one per month...

To the point:
The world famous NTP daemon gained an important feature for all internet users with dynamic IP assignments. "Dynamic interface tracking" is a simple feature (from the user's point of view) that makes the NTP daemon aware of IP changes on network interfaces to avoid loosing synchronisation.

Before this feature existed, upon startup, the NTP daemon would store all the IP addresses to which it would bind. While doing its job, the daemon used those stored IPs as source IPs to contact servers and peers. If one IP would change on a given network interface, all servers and peers contacted through that interface would become unreachable because the stored source IP no longer matched the new interface's IP. In the common case of a server with a DSL internet connection, all external servers and peers are reached through the PPP interface and if that IP changes (as it does frequently in my case) no more servers can be contacted resulting in loosing synchronisation.

Stable version 4.2.4p0 of the NTP implementation (http://www.ntp.org/) includes dynamic interface tracking and an important bugfix (#765) which allows using this feature while running the daemon with an unprivileged user. The first official version including dynamic interface tracking was 4.2.4.

This feature is activated automatically and the scanning intervals can be controlled with the daemon option "-U". In my case, instead of using "ntpd -u ntp:ntp -g" I now use "ntpd -u ntp:ntp -g -U 600" and I do not have to restart ntpd regularly because of frequent IP changes (every 17h55m).

[EDIT: forgot to mention the Gentoo overlay ebuild method]

There is not yet a Gentoo Linux ebuild for ntp 4.2.4p0 (although I offered one on Gentoo bugzilla), but it is easy to overlay one and try out the updated 4.2.4:

1. add the following line into /etc/make.conf if there isn't one already
   PORTDIR_OVERLAY="/usr/local/portage"


2. create the new overlay for ntp 4.2.4p0
   # mkdir -p /usr/local/portage/net-misc/ntp
   # cp /usr/portage/net-misc/ntp/ntp-4.2.4-r1.ebuild /usr/local/portage/net-misc/ntp/ntp-4.2.4_p0.ebuild
   # cp -r /usr/portage/net-misc/ntp/files /usr/local/portage/net-misc/ntp/


3. edit /usr/local/portage/net-misc/ntp/ntp-4.2.4_p0.ebuild and change:
   mirror://gentoo/ntp-${MY_P}-manpages.tar.bz2
   into
   mirror://gentoo/ntp-${PV:0:5}-manpages.tar.bz2


4. create digests for the new overlayed ebuild
   # ebuild /usr/local/portage/net-misc/ntp/ntp-4.2.4_p0.ebuild digest


5. emerge the new ntp
   # emerge =net-misc/ntp-4.2.4_p0


6. be aware that if you had ntp-4.2.4-r1 installed, the portage system will think you are downgrading ntp, so you need to pay attention when upgrading the system to prevent portage from "updating" ntp back to 4.2.4-r1

Cheers everyone,
PJ.

Beryl accelerated window manager for linux

For the last three weeks, I've been using Beryl [wikipedia], the hardware accelerated window manager for Linux.

Let me say it is brilliant when seen from several different perspectives:

• Features beautiful window manager effects: zoom, fade, transparencies, wobbly windows, lots of effects for different events, etc;
• Places the typical 4 desktops in 4 faces of a cube, so to switch between them, the cube rotates (something similar to what Mac OS X does when switching users);
• Did I mention the cube? It rotates freely, can be translucid when rotating, windows can be "bent" between two sides of the cube and still work well;
• New task switcher; Fades everything else, windows that show non-static content continue to do so when appearing in the task switcher (like Mac OS X or windows vista);
• Method to view all active windows much like Mac OS X's exposé;
• Highly flexible look; bitmapped borders, transparencies, shadows, etc;
• Lots of GPL'd themes and a usable theme fetching interface;

All these features sum up what I would call a 'brilliant prototype of the next Linux desktop'. It is waaaaaay better than that vista thing, it feels beautifully smooth, it is very very stable (for its age, plus when it crashes, beryl-manager reverts automatically to the previous manager like kwin, metacity or xfce). The effects are great, the cube is awesome and it has the ability to impress friends and people at work (even microsoft addicts).

Wait, did I say "prototype"? Yes, I'm afraid so. You see, all these pretty smooth effects tend to tire the user and eventually waste his/her time. In my case and for all ATi owners, it can also be a resource hog, but that's another story (one for the next paragraph).

Beryl started by being a fork of compiz [wikipedia] - the first compositing window manager that takes advantage of openGL graphics acceleration, written by Novell to work with a new version of David Reveman's open source Xgl [wikipedia] X server architecture.

Currently there are two (or three) different ways to get hardware accelerated X servers with composition window managers: Xgl, AIGLX (and a proprietary nvidia extension of which I don't know anything about).
Xgl itself is a X server that runs below Xorg and uses it as its only display window. However, it consumes much more resources than what I find acceptable. AIGLX is the elegant and open source way - it is a native Xorg extension.

ATi owners have a choice of three drivers: radeon (no 3d acceleration, beryl cannot be used), fglrx (ATi proprietary driver with 3d acceleration) and r300 (open source reverse engineered driver with 3d acceleration). Up until version 8.33.6, fglrx lacks the GLX_EXT_texture_from_pixmap extension necessary for AIGLX. This means that AIGLX will not work with the fglrx driver, and people that want to use it due to superior performance have to stick with Xgl and less resources. r300, on the other hand, does supporte that particular extension and some users have reported working configurations of r300+AIGLX.

It is not very useful to post screenshots as the beauty of Beryl lies in its animations and fluidity, so a quick search for 'beryl' in youtube reveals many good demonstrations. Below I've selected 3 of those demnostrations, but there are many other and as Beryl is being developed with a fast paced, these are quickly outdated.

• http://www.youtube.com/watch?v=CBRba0jsxGQ (0:51)
• http://www.youtube.com/watch?v=_cQTEZPTZL4 (1:44)
• http://www.youtube.com/watch?v=RwlutJTZj9w (0:59)

So, there it is - Linux, Xorg and Beryl are by far the best window desktop in the world!

2006 is left behind.

Who would believe that 2006 would ever end?? Quickly a new year came to take 2006's place. 2007 they call it.

Sorry for my bad jokes. Now that the dust is settled and I have a little more time, I've decided to write a couple posts on different subjects. Today I'm writing about the New Year's Eve.

This year I spent the New Year's Eve with a group of friends in a very traditional Portuguese village called "Covilhã". Covilhã is located just on the second tallest Portuguese mountain chain: "Serra da Estrela". It's a dead beautiful place to visit either on the winter or on the summer.

Samples of some of the available landscapes:







With everything summed, it was a great weekend of 3 days with a new year's eve included. Already looking forward for the next new year's eve or long weekend, whichever comes first!

Cheers, PJ.