my BSD Blog Planet

May 20, 2012

cperciva

Speeding up Portsnap via geolocation

Ever since I wrote Portsnap in 2004, I've been hearing the same question from FreeBSD users outside of North America: "Can we have some nearby mirrors? The US mirrors are too slow!" I've helped many companies set up Portsnap mirrors within their networks, but most of the public Portsnap mirrors have always been in the US, for three simple reasons: The existing mirrors were more than enough to handle the load (even when it peaks following a "megacommit" which touches many ports); administering multiple similar-but-not-quite-identically-configured mirrors is a headache (and when it comes to system administration, I optimize for lack of headaches); and despite my attempt to improve mirror selection by providing "us.portsnap.freebsd.org", "eu.portsnap.freebsd.org", and other similarly "local" names with customized mirror lists, most people still used the default "portsnap.freebsd.org" server pool. Thanks to Amazon EC2 and Amazon Route 53, this has now improved.

May 20, 2012 05:10 AM

May 17, 2012

dds

Git

Even by our field’s dizzying rate of progress I wouldn’t expect to revisit the subject of version control just six years after I first wrote about it in this column ( Version Control Systems . Software , 22(5):108–109, September/October 2005). Yet here we are. The new kid on the block is git, a distributed revision control system available on all mainstream development platforms through a Free Software license. Git, a brainchild of Linus Torvalds, began its life in 2005 as the revision management system used for coordinating the development of the Linux kernel. Over the years its functionality, portability, efficiency, and third-party adoption have evolved by leaps and bounds to make it its category’s leader. (Two other systems with similar characteristics are Mercurial and Bazaar.)

by Diomidis D. Spinellis at May 17, 2012 12:41 PM

Package Management Systems

DLL hell was a condition that often afflicted unfortunate users of old Microsoft Windows versions. Under it, the installation of one program would render others unusable due to incompatibilities between dynamically linked libraries. Suffering users would have to carefully juggle their conflicting DLLs to find a stable configuration. Similar problems distress any administrator manually installing software that depends on incompatible versions of other helper modules.

by Diomidis D. Spinellis at May 17, 2012 12:35 PM

May 14, 2012

ivoras

BSDCan 2012 - Day 2

The second, and unfortunately the last day of BSDCan was filled with interesting talks, again with much overlap. There are simply so many interesting things going on in FreeBSD that all of them simply don't fit in just two days of conferencing! From all of those, I'd recommend (even though I wasn't able to attend some of them) the talks on netmap, ZFS, AWS, pkgng and IPv6 security - don't miss them when the videos go online!

Read more...

May 14, 2012 02:16 PM

May 12, 2012

ivoras

DevSummit 2012 day 2 and BSDCan 2012 day 1

The second day of the DevSummit continued with interesting technical discussions in the Virtualization track, which was paralleled with the Teaching OS Courses track and the Administration and Toolchain tracks. The BSDCan day began with an epic bagpipe performance followed by full four tracks of highly interesting topics - unfortunate as there is much overlap. I gave my talk on Bullet Cache which describes some of the more interesting technical aspects and presents new performance measurements.

Read more...

May 12, 2012 03:49 PM

May 10, 2012

tobez

Why does not it meow?

Today I've spent quite some time chasing a bug in a legacy code at work. In retrospect, the problem is trivially simple.

It can be illustrated by the following snippet.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body>
<script src="irrelevant.js" type="text/javascript"/>
<script type="text/javascript">
function meow() { alert("meow"); }
</script>
<a href="http://dailyotter.org/"
 onclick="meow(); return false;">
click for meow</a>
</body>

So, why does it show you otters instead of meowing, and how long did it take you to spot the bug?

by tobez at May 10, 2012 01:33 PM

semicomplete - Jordan Sissel

Installing Windows 8 Consumer Preview

I have a fresh workstation and am running through the windows 8 installer on USB. When choosing the drive to install to, I get an error:
We couldn't create a new partition or locate an existing one
Lots of googling and I didn't find any hints for windows 8, but windows 7 has a similar error and folks pointed at diskpart nonsense to fix it. So let's do that -
  • At the installer, choose "Repair your computer"
  • Choose "troubleshoot"
  • Choose "advanced options"
  • Choose "command prompt"
  • Run diskpart.
In diskpart, you'll want to make sure your target drive is formatted and active.
list disk

# now pick your disk
select disk 0
clear
create partition
format fs=ntfs compress quick
active
Now reboot and try the installer again, it worked for me.

May 10, 2012 06:50 AM

May 09, 2012

ivoras

BSDCan 2012 - DevSummit

Another year - another BSDCan! It's very nice and even comforting to see such a large number of familiar faces again, and even more as the ranks are filled in by fresh new developers. The conference and the Developers' Summit before it promise a great program and a great time for the BSDers.

Read more...

May 09, 2012 09:37 PM

May 06, 2012

anholt

Backyard slackline limits reached

Two weeks ago we built a slackline setup in our back yard. The issue we had was that we don't have any trees back there to tie up to. Common solutions in this case involve building an A frame and using whatever sort of anchor you can come up with, with plenty of options available.

We wanted better. The yard could only go to about 40 ft of line, and we didn't want to sacrifice precious length between our anchors and the A frame.

The first plan we were working with was to put a pipe in some cement, then slide a smaller pipe into it, and use that as our fake tree to anchor to: Now there's a solid anchor, but it's removable if I decide to sell the house or something some day. I found some numbers for guidelines for building railings, though, that indicated that you'd need massive steel pipe to support the loads we're talking about.

What we went with in the end was a wooden 4x4. We'd heard that slackliners were successfully using those in home setups. But we were a little wary of trusting a wood 4x4 more than a steel pipe. So what we buried in the cement was a post sleeve so that we could just slide our 4x4 into the cement hole after it was set. The cement was 3 feet deep and just over 1 foot across (if you decide to go this route: post hole diggers are *awesome*). This let us put an 8 foot 4x4 in each and be able to set a line at heights up to around 4 feet off the ground. But just in case, we also dropped some heavy chains into the cement as well in case we want anchors for A frames if this posts thing doesn't work out.

We first used the system last Sunday with great success. It's a typical 4-carabiner primitive system but we used a double pulley system behind that to get enough tension from a single person tightening that you'd stay off the ground in the middle. There was a disturbing amount of bending and some creaking in the 4x4s, but they held.

Today Scott was setting up the line again, and said "I got it nice and tight, look at that!", and I hopped on. I made it about 1/3 of the way, when there was a snapping sound and suddenly I was on the ground. Luckily failure wasn't as catastrophic as we feared. The post had just bent over, and not detached and gone flying.

Our next plan was to use steel I-beams: the backup plan that justified the 4x4 sleeves. I'm still concerned though -- a beam stress calculator program says that for what we're thinking is like up to 1600lbs of force at 4 feet from the support point, we end up with a maximum bending stress at the support point of 164 ksi on a S3x7.5 I-beam (the biggest that will fit in our sleeves as far as I can see). If I'm supposed to compare this number to the yield stress of the steel the beam would be made of, that number is only 22 ksi.

The plan for the moment is to throw together some A frames (actually, X frames -- Scott built and used some of those successfully this week, and it sounds easy enough) and use that unless we can figure out that I was wrong and steel will hold.

by eric@anholt.net at May 06, 2012 01:42 AM

May 01, 2012

erwin

ivoras

Pirates ahoy!

I'm blogging a little bit less frequently then previously simply because I have less time to do it - but I'm happy to say it's for a good cause which also makes me personally happy: I've been involved in bootstrapping the Croatian Pirate Party.

 

Read more...

May 01, 2012 10:52 AM

April 22, 2012

remko

DSPAM

Since recent (with the very great help of Ion-Mihai Tetu, a fellow FreeBSD committer and developer for dspam) we (JR-Hosting) are running our anti-spam infrastructure on DSPAM. We stopped using SpamAssassin after some testing and resolving problems. The interesting fact is that we share most directories through nullfs so that both the webjail and the mailjail share data and our users are able to modify settings, see their stats etc. Very great and after overcoming our issues (local delivery was not OK in the beginning and the webjail was not able to properly use the MySQL database backend at first, which was odd because the main system WAS looking into it and the webjail wasn’t), it works just fine. Ofcourse it is still learning but it seems that it finds spam efficiently and quick, and it’s footprint is much much lower then SpamAssassin was. I might want to figure out how to run the daemonized version as per advise of Ion-Mihai, till then it works as a deliveryagent.

I am writing a ‘hosting environment howto’ (or something that will largely look like that) in which I will write about the setup as well.

by Remko Lodder at April 22, 2012 02:03 PM

April 15, 2012

erwin

April 14, 2012

erwin

Iceland, day 2 and 3

ms 226 ms 233 ms 239 ms 246 ms 247 ms 253 ms 257

Related posts:

  1. Iceland, day 1 part 2 ...
  2. Iceland, day 1 ...
  3. Moving to Copenhagen Finally, last weekend the rest of my things were moved...

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at April 14, 2012 11:39 AM

Iceland, day 1 part 2

IMG_4129 IMG_4135 IMG_4159 IMG_4162 IMG_4168 IMG_4169 IMG_4187 IMG_4191 IMG_4203 IMG_4221

Related posts:

  1. Iceland, day 1 ...
  2. Moving to Copenhagen Finally, last weekend the rest of my things were moved...

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at April 14, 2012 11:17 AM

April 13, 2012

erwin

Iceland, day 1

ms 217a ms 213a ms 206a ms 205a ms 184

Related posts:

  1. Moving to Copenhagen Finally, last weekend the rest of my things were moved...
  2. Crete, Greece Last month, we went on a delayed honeymoon to Crete,...

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at April 13, 2012 11:21 PM

April 05, 2012

des

Sherlock

Having watched season 1 and the first episode of season 2 of Sherlock, I absolutely love the show, although 2x01 was very weird; very slow, very drawn-out, you think they've wrapped it up then discover that it was just the beginning. A bit like those games I shall not name where you play for 20 or 30 hours before you realize that was just the tutorial.

As a long-time Sherlock Holmes fan who has read all the stories and all four novels at least twice, I was very impressed by how faithful they managed to remain to the original despite moving it 120 years forward in time. All the episodes I've seen so far stick fairly close to the plot of one or two of the original stories, with a load of (often tongue-in-cheek) references to others.

The greatest departures I've seen so far are in the secondary characters: Mycroft is less sedentary than in the original stories, Moriarty is younger and crazier, and Lestrade is far more likeable and far more respectful of Holmes's intellect and methods (the original character is described in very unflattering terms and is often resentful of Holmes's success and not above taking the credit for Holmes's work).

Many self-proclaimed Sherlock Holmes fan are very critical of this series and of the Guy Ritchie movies. I disagree; I suspect that their dissatisfaction stems from a skewed view of Holmes, from reading the original text through the lens of an interminable series of movies and TV series where Holmes is depicted as an elderly, pot-bellied, gray-haired upper-class gentleman. In that respect, they're not unlike those Hunger Games fans who were shocked to see so many dark-skinned actors in the movie, despite the fact that the characters they portray are described as such in the novels.

The Guy Ritchie movies are obviously a reimagining rather than an adaptation, bearing little relationship to the plot of any of the original stories, but I think they're quite faithful to the spirit of Sherlock Holmes. Their pace is obviously much faster, but that is Ritchie's signature—cf. Tom Lehrer's brilliant take on Oh My Darling, Clementine.

by Dag-Erling Smørgrav (noreply@blogger.com) at April 05, 2012 09:36 PM

ivoras

Google Summer of Code

Time flies! The last year's barely passed and here we are again - another Google Summer of Code. Even though the deadline for student's submissions is less than 24 hours ahead, I'd still take the opportunity to call anyone interested, anywhere in the world, to submit a proposal - it's fun to participate!

Read more...

April 05, 2012 07:43 PM

des

You are always the product

I just discovered that Amazon allows anyone to “follow” a Kindle user, and there is no way to turn that feature off or to block followers. They say your followers can only see notes that you explicitly make public, but they don't say anything about whether they can see your book list, which is the union of books you've bought, books you've reviewed and books on your wish lists—and not just eBooks, but paper books as well. Wish lists can be made private, but the FAQ doesn't say whether that means they don't appear in my book list on kindle.amazon.com.

This makes me extremely uncomfortable. I didn't realize until now that Amazon has absolutely no privacy settings whatsoever, beyond the option of marking a wish list private (which should be the default, but isn't). It's also interesting that they describe public lists as “lists your friends can see” and not as what they really are—lists anyone can see. Arguably, that's what wish lists are for, so I don't really mind, although I have now marked all my wish lists private until I can look through them and think about what they say about me, and whether I really want to share that with the world.

(I am tempted to create a single public wishlist with nothing but My Little Pony merchandise)

In hindsight, I shouldn't be surprised that this feature exists. When I found out about kindle.amazon.com a few months ago, I was slightly baffled: I could not see the use of it, as the only functionality it offers that “Manage my Kindle” on the main site doesn't is the ability to view my highlights so I can more easily copy-paste them to my blog, in here, or wherever else I may want to quote something I've been reading on my Kindle. Now I understand: the purpose of kindle.amazon.com is to sell me.

There is an oft-repeated aphorism that goes “if you're not paying for it, you are the product”. What people may not realize is that even if you are paying for it, you may still be the product. Amazon may not sell information about their customers to marketeers, but they use it to make their products and services more attractive to other customers, viz. your friends, family and cyberstalkers.

O brave new world that has such websites in it.

by Dag-Erling Smørgrav (noreply@blogger.com) at April 05, 2012 08:38 PM

April 02, 2012

des

On testing

Last fall, I wrote a completely new configuration parser for OpenPAM Lycopsida. Although the new parser was far more robust than the one it replaced, it was large, unwieldy, and suffered from a number of issues relating to whitespace handling, which stemmed from reusing some old code which unfortunately was thoroughly documented and therefore could not be easily modified. So I decided to rewrite it again, from scratch this time.

Then I did what I should have done last fall but didn't: I wrote some unit tests. And of the first dozen or so tests I came up with, three failed, revealing two different bugs—one of them fairly serious.

There's a lesson in here somewhere...

by Dag-Erling Smørgrav (noreply@blogger.com) at April 02, 2012 12:29 AM

March 27, 2012

cperciva

Adding durability to the kivaloo data store

365 days ago, I announced my kivaloo data store here. Architected to maximize performance for the particular workload my online backup service has, it provides a much better cost:performance ratio than Amazon's DynamoDB; but as a single-machine data store it had some limitations:
  • It wrote data to a single disk, so its durability was limited to the durability of "local (fsynced) disk";
  • It ran on a single machine, so its availability was limited to the availability of a single machine and its network connection; and
  • It was mostly single-threaded (I/O had separate threads, but all the "real" work was done in a single thread) so its performance was limited to about 100k 80-byte key-value pairs per second.
As of my latest set of SVN commits — available on the Google Code site but not yet released on the main kivaloo website — the first of these problems has been fixed.

March 27, 2012 11:20 AM

March 22, 2012

dds

How do Big US Firms Use Open Source Software?

We hear a lot about the adoption of open source software, but when I was asked to provide hard evidence there was little I could find. In an article I recently published in the Journal of Systems and Software together with my colleague Vaggelis Giannikas we tried to fill this gap by examining the type of software the US Fortune 1000 companies use in their web-facing operations. The results were not what I was expecting.

by Diomidis D. Spinellis at March 22, 2012 06:47 PM

March 20, 2012

remko

FIXED: FreeBSD Jails PHP dirname WordPress

Dear Reader,

I had fixed the issue. Instead of using nullfs to get access to the /usr/home directories, I am using unionfs, which basically does the same for my goals (unless someone corrects me in misunderstanding things) and this does not seem to generate the same issues. Various sites are now running happily behind the WWW Jail. Time to finish my document on how I did setup the entire beast.

Thanks all for listening, helping, and giving tips (Alexander and Miroslav!)

by Remko Lodder at March 20, 2012 10:27 PM

HELP: FreeBSD Jails PHP dirname WordPress

So, I am still building up my jail structure and the last few evenings I was testing the FreeBSD jail wrt. PHP, Apache22-mpm-itk and wordpress.

Things started to break when I redirected external traffic to the jail. It seemed that require_once(dirname(dirname(__FILE__))) . ‘/wp-load.php’; does not work from within the jail.

I decided to do a little test and testing reveals that in a stand alone configuration the dirnames behave exactly the same, in both the host and the jail. Printing the directive within WordPress (when loading the admin pages f.ex.) reveals a ‘.’ instead of the ‘/path’ . It is resolvable by adding a ‘.’ to the directive so that wp-admin/admin.php loads the ../wp-load.php file instead of ‘/path/to/wordpress/wp-load.php’. Though this sounds very sily todo.

Did someone else encounter this? I Do not want to change enforcement of the statfs to some other value since the defaults should be good enough (given the testsript).

Relevant details: the /usr/home where the public_html files live, are nullfs rw mounted from the host and are available in the jail. The jail does username/group lookups through Ldap, and can see the various users. Apache had been build with the ITK patches so that every host runs under his/her own user. I do not see obvious differences between the regular host and the jail, the only real difference is the internal/external addresses used in the vhost configuration, but that is kinda obvious to me.

Let me know :-)

by Remko Lodder at March 20, 2012 12:00 PM

March 19, 2012

des

Downtime

I haven't been able to read email sent to des@des.no or des@freebsd.org for five days, due to a series of unfortunate incidents involving dodgy power supplies and the fragility of ZFS boot in FreeBSD. Work and other duties prevented me from addressing the issue in a more timely manner, but I am now regaining control. Luckily, neither my ~30 GB IMAP spool nor any other data was lost, nor did my backup MX bounce any mail. My IMAP server is now back up with a small UFS SU+J boot / root partition instead of ZFS. I am still unable to read email, but that should be fixed within 24 hours.

I also uncovered an annoying but luckily not fatal bug in the Cyrus IMAP server. When TLS is configured, the IMAP daemon stores state for each TLS session in a DB file. If that file is corrupted, the server will start, but it will refuse any incoming IMAP or LMTP connections, and will instead spit out a stream of completely unhelpful error messages. The only recourse is to delete the TLS session state database; I set up an rc script to do that at boot time, so hopefully this won't bite me again.

by Dag-Erling Smørgrav (noreply@blogger.com) at March 19, 2012 12:15 AM

March 11, 2012

stas

HostObzor Autumn 2008

Recently I have been attending Hostobzor 12th, the Russian conference of hosting providers, beeing held at Raivola hotel near St. Petersburg. The event was great as always thanks to organizers. There was a number of intersting talks given, a lot of interesting discussions held, and, what I appreciate better, a lot of new people with great ideas met.

I gave a talk on using the FreeBSD Ports system to mange a large-scale virtual hosting installations based on Hosting Telesystems experience. I tried to describe in detail how we use the ports collection to deploy a large number of servers diverced by architecture and OS versions, how we build packages and distribute them among servers, talked about how we use Mercurial VCS to incrementally merge upstream changes into our modified ports collection and FreeBSD src trees. Hopefully, I’ve not screwed it much… At least, some people was interested a lot and asked interesting questions.

If you’re interested you can grab the talk draft and slides here (in russian): [paper] [slides].

Video should be available later from HostObzor website.

by stas at March 11, 2012 05:12 AM

Russian electronics. We don't need microelectronic technology, we were doing pico- for decades already.

This suspicious device was found recently at MEPhI microelectronics departent catacombs. After breaking the deivice's case to our great disappointment we weren't able to discover any signs of microelectronic footprint inside. The pins weren't even connected (at least it seems so). It looks like the soviet microelectonics went beyond the current nano-sized structures several decades ago. Picoelectronic device, indeed. :-)

PS: I would really like to know which microprocessor architecture this device implements. Getting FreeBSD runned there will be really great!

by stas at March 11, 2012 05:12 AM

Enlightenment FreeBSD port updated

The Enlightenment E17 FreeBSD port has been updated to the recent snapshot today. Among bugfixes and new APIs, it also brings a lot of new libraries and modules that you will enjoy. Thanks a lot to Grzegorz Blach for preparing the original patch and for his hard work on bringing all the new e17 components to FreeBSD!

by stas at March 11, 2012 05:10 AM

March 08, 2012

matteo

Generating all combinations of n elements taken t at a time

I don’t usually share the code I write for my research projects, unless someone asks for it. Anyway, today I had fun writing some C code, and I think it may turn out to be useful for other people. It’s written almost well enough to be acceptable, so here you are.

I implemented Chase’s sequence algorithm to generate the bit-strings corresponding to each combination of n elements taken t at a time. The algorithm is described in Donald E. Knuth’s “The Art Of Computer Programming”, Vol. 4, Fasc. 3, page 13. I implemented it in C and added very few comments, so refer to TAOCP if you have any doubts.

The code is distributed under the BSD 2-clauses License.

Download chaseseqcombgen.c

Posted from Providence, Rhode Island, United States.

by Matteo at March 08, 2012 06:38 PM

March 07, 2012

remko

FreeBSD Activity++

Lately I have become more active then the last year in total for FreeBSD. I committed several enhancements that were in my queue already for a long time, but finally came into the FreeBSD tree. Some too late for 8.3 which is upcoming and some not.

I also understand merging much better then before, taking the time for a commit and making a few mistakes really helps, and ofcourse the community is not too shy to mention my faults :-)

One of the things that got committed is the force setting of carp, whether it’s a master or backup node (make sure pre-empt is disabled if you do this, else it will just rollback within seconds :-) ), which eases maintenance for example. One of the other things is that I am using an extensive network of local jails now that service my needs, most seperated items. I also started writing an Howto (or bsdmag article if there is interest) to demonstrate and tell how we setup most items. One of the things that is still causing me headaches is that we have nullfs rw mounts of /home to the jails because mailservers need to write, ftp servers need to write etc. But if we were sharing these information sources via NFS we would have had the same challenges :-)

I feel good in taking the time for FreeBSD again, and I would like to hear recommendations on what targets I can persue in FreeBSD (low hanging fruit is good enough for the time being) and additional things, also please comment on the nullfs mounts (rw,nosuid) to enhance my security level and which makes my article even better :)

by Remko Lodder at March 07, 2012 04:27 PM

tobez

Make xpdf remember position in files

Here's a tiny ugly patch to make xpdf remember positions in files and restore them. You open a PDF document, read it a bit, then quit xpdf. Next time you open the same file, the same page will be shown as it was when you quit.

The positions are stored in ~/.xpdf.bookmarks.

If you are using xpdf from FreeBSD ports, just put the patches into /usr/ports/graphics/xpdf/files directory and rebuild.

Patch 1
Patch 2

by tobez at March 07, 2012 04:03 PM

March 03, 2012

ivoras

Kindle text editor bounty

I love my Kindle 3. It's a great little gadget which has increased the amount of books I read - which is by itself an outstanding accomplishment. I feel I don't need to describe its many benefits here because the interested reader either already has it or knows how to use Google well enough to find out all about it.

But, there is one very significant feature from Kindle which I very much miss - a simple text editor. I'm not a passive content consumer and I write a lot, and so since I haven't found (to my great surprise) any such thing already, available, I'd like to offer a bounty of $60 to anyone who can create a working text editor for Kindle 3, with the required features describe in the rest of this blog post.

Read more...

March 03, 2012 11:33 PM

erwin

February 29, 2012

semicomplete - Jordan Sissel

When all you have is a hammer, make your own tools?

Clarifying my position from this post:
The "ops folks need coding skills" groupthink is lame. Software requires extra coding because it is shitty, not because people are unskilled

I will lead with this: I want more people who use technology to grow and learn better skills for bending that technology to their needs. An ops guy with programming skills is, to me, more valuable than one who cannot - programming in any language or platform lets you extend an otherwise static system.

Anyway, back at the post in question, I'm not trying to say people (ops or otherwise) shouldn't want stronger programming skills. I'm saying the equipment we use is pretty shitty.

I am part of the generation raised near devices ever blinking "12:00". Devices which have no business caring what time it is, nor any sane reason to make the state of "I don't know what time it is" a high priority alert worth blinking forever.

It's 2012, and this problem persists - my microwave refuses to cook food unless it has the time *and* date from user input. Now I have to program it every time it has a power disruption (which is has frequently due to some bug in the hardware causing it to power off randomly with certain dishes at home).

Now I have to learn to program or configure these devices before they'll stop irritating me. And, damn it I hate that. If, instead, this were enterprise software, I could report these irritations to the vendor who would kindly offer me training and consulting for extortionate piles of money.

I love coding. It's fun, and many times lets me solve problems I couldn't otherwise. Allowing me to abuse an analogy, "When all you have is a hammer, you can sit down and build whatever tool you need to repair the delusion that everything is a nail."

But despite being able to solve my own problems in software, I don't think this is a great pattern of work. I write code, most of the time, because the solutions available are terrible or don't meet my requirements. With a new software popping up every day, I see a strong correlation between software availability and people asking for more programmers.

So, the more software we have, the more programmers we need to work around limitations in the available body of software. I think that's pretty lame :(

And regarding my microwave problems, I want some confidence that the problems being solved are meaningful problems, not programming learned for the sake of working around bugs and misfeatures in software we're suffering with.

February 29, 2012 06:01 AM

February 27, 2012

des

Wires and switches

ESR is at it again:
I can best introduce you to our concerns by quoting another of our philosopher / elders, John Gilmore. He said: “The Internet interprets censorship as damage and routes around it.” To understand that, you have to grasp that “the Internet” isn’t just a network of wires and switches, it’s also a sort of reactive social organism composed of the people who keep those wires humming and those switches clicking. John Gilmore is one of them. I’m another. And there are some things we will not stand having done to our network.
Dear ESR,

I'm pretty sure John Gilmore knows that wires aren't supposed to hum and switches aren't supposed to click. It seems you don't. Perhaps it would be best if you stepped away from the Internet before you break something expensive.

by Dag-Erling Smørgrav (noreply@blogger.com) at February 27, 2012 04:59 PM

February 26, 2012

erwin

To Blu-ray or not to Blu-ray

How did Blu-ray[tm] ever win over HD-DVD during the so-called format wars? How could HD-DVD possibly be even worse or might it have had nothing to do with technical implementation or what consumers want? Take the following characteristics of Blu-ray:

  • Slow loading times
  • Unique menu system, confusing finding your way around on each new disk
  • No resume possible on most disks
  • Firmware updates, download, but on USB stick, lots of clicking on the remote and a reboot later
  • Incompatible on-disk software and player firmware

The best part is that all of the above are design features, not bugs. But they do bug me. Admittedly, they look and sounds great, but after yet another weekend of anticipation to watching a rented (for real hard-earned cash) movie, which couldn’t be played, even after the latest firmware upgrade, I wonder why I even bother. I’m sure there must be alternatives.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at February 26, 2012 07:46 PM

February 17, 2012

remko

Family news…

Dear all,

It is with very great pleasure that I would like to tell you, that we (Denise, Luca and myself) are expecting our second child. Currently we are around 12 weeks and everything is looking good.

We saw the first images of our soon kid and new FreeBSD Hacker? ;-) He or she is looking beautiful already. We do not yet know the gender, but we are expecting the kid around the beginning of September.

Luca is also happy with these developments, so everything is in the works (actively as we speak) to move his room to the upperfloor, and we are going to prepare his old room for the baby.

Leave a message so that I can read them later on (I would like that): do realise that it’s moderated and that it might take a little before I can acknowledge your message :-)

by Remko Lodder at February 17, 2012 04:26 PM

February 15, 2012

remko

FreeBSD: jails, ezjail, pfSense

During the last couple of days I am intensively using ezjail to administer several jails on my machines. They are currently IPv6 only (internet-facing) and are used to build pfsense images to test locally (still setting this up, need to cross compile to i386 from amd64), offer a testjail to a collegue to work together on a Opsview implementation on FreeBSD, whether or not we are going to succeed in that, and I just installed a test environment for my webservices. They are all contained in their own little box, having IPv4 connectivity outgoing through NAT, and native IPv6 connectivity from my “Vendor” on an extra subnet that I obtained.

I like this, so I am probably going to setup some more services here and there to perform some magic for me that might need external access. I will also tie them together with LDAP and the like so that it’s an uniform base. At the moment I do not have additional ideas about moving production services towards jails as well though.

Thanks to FreeBSD this all is damned easy. You should try it, or poke me in case you want to know more! :)

by Remko Lodder at February 15, 2012 10:12 PM

February 12, 2012

des

Dag-Erling collating papers

As soon as I realized that I had forgotten to tick the “collate” box when I printed out five copies of The Eye of Argon and that I would therefore have to collate them by hand, I knew I had to have this photo taken.



I hereby challenge the Internet to collate the hell out of some papers. Remember to link to the original.

(backstory)

by Dag-Erling Smørgrav (noreply@blogger.com) at February 12, 2012 01:15 AM

February 11, 2012

ivoras

Dealing with Trac performance problems

Trac is a great little web tool for both developers and random collaborators - it's a wiki integrated with a Subversion browser with simple project management tools and extensible with a lot of different plugins. Unfortunately it's also slow. I've had to deal with an Internet-facing Trac wiki and came up with some solutions which lessen this problem.

Read more...

February 11, 2012 11:15 PM

February 08, 2012

dds

How to Decrypt "Secrets for Android" Files

Secrets for Android is a nifty Android application that allows you to securely store passwords and other sensitive data on your Android phone. Your data are encoded with your supplied password using strong cryptography and are therefore protected if your phone gets stolen. Although the application offers a backup and an export facility, I found both wanting in terms of the availability and confidentiality associated with their use.

by Diomidis D. Spinellis at February 08, 2012 11:19 PM

January 31, 2012

erwin

FOSDEM 2012

In a few days, I’ll be heading to the FOSDEM conference in Brussels again this year. On Saturday, you’ll most likely find me around the FreeBSD booth representing the FreeBSD Foundation, so if you’re there drop by to say hi, discuss the Foundation’s work, pick up a Foundation flyer, check out the swag, or make a donation. On Sunday, I’ll be in the BSD DevRoom where there will be some interesting presentations and discussions. Remember, FOSDEM is free to attend. Hope to see you there!

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at January 31, 2012 04:03 PM

January 23, 2012

cperciva

Why Tarsnap won't use DynamoDB

When I heard last Wednesday that Amazon was launching DynamoDB I was immediately excited. The "hard" server-side work for my Tarsnap online backup service consists mostly of two really big key-value maps, and I've spent most of the past two years trying to make those faster and more scalable. Having a key-value datastore service would make my life much simpler, I thought. Unfortunately, upon reading into the details, I've decided that DynamoDB — at least in its present form — isn't something I want to use for Tarsnap.

January 23, 2012 04:20 PM

January 19, 2012

cperciva

Playing chicken with cat.jpg

In a game of chicken, which is the better strategy: Writing a lengthy and detailed "persistence policy" guaranteeing that you'll persist in your course and will not, under any circumstances, swerve to avoid your opponent; or ostentatiously removing your steering wheel and throwing it out the window? As noted by innumerable game theorists over the past fifty years, the latter strategy is the only one which is useful: Humans can't be — and aren't — trusted to follow their stated intentions.

January 19, 2012 01:00 PM

des

Dear Members of Congress

The Internet's resilience and ability to heal itself and work around intentional or unintentional damage are the stuff of legends. As with most legends, however, it is only partly true. Key components of the Internet's technical infrastructure are still, to a large degree, and despite repeated efforts to decentralize them, concentrated in and controlled by the United States.

As a non-US resident who makes his living largely from the Internet, I am extremely pleased to see that the United States House of Representatives has recognized this weakness and is considering decisive measures to remedy the situation.

I am, of course, referring to SOPA. To summarize, SOPA is an incentive program designed for two purposes: first, to encourage organizations outside the US to replicate key parts of Internet infrastructure so that the Internet will be able to operate entirely outside US control; and second, to encourage US organizations to move their Internet operations overseas. If SOPA is passed into law, the Internet will soon become a truly global, independent, distributed and decentralized entity, able to operate completely outside the control of any single government or organization and organically work around any form of censorship or sabotage. The downside is that the United States will become a sort of Internet ghetto, fenced off from the rest of the world and strictly policed, much like Iran and China are today; but that is a small price to pay. The Internet, with a few regrettable exceptions, will finally be truly free.

So please, dear Members of Congress, I implore you: vote for SOPA. Not for yourself, not for your country, but for the world.

by Dag-Erling Smørgrav (noreply@blogger.com) at January 19, 2012 07:38 AM

January 16, 2012

cperciva

FreeBSD now on all EC2 instance types

Six months ago I announced here that I had managed to get FreeBSD running on 64-bit Amazon EC2 instances by defenestrating Windows AMIs. That took the set of EC2 instance types FreeBSD could run on from three (t1.micro and c[cg]1.4xlarge) up to nine by adding all of the large and extra-large instance types; but FreeBSD still couldn't boot on "high-CPU medium" instances or on the "standard small" instance type — the one which got EC2 started, and which I suspect is still the most popular of all the options. Today I am pleased to announce that FreeBSD 9.0-RELEASE AMIs are now available for all EC2 instance types.

January 16, 2012 10:50 PM

Automatically populating .ssh/known_hosts

One of the more irritating things about working with virtual machines is SSH host keys. Launch a new virtual machine. Get a new host key generated. Try to SSH in. Get a pesky warning message telling you that the authenticity of the host can't be established. Find the host key fingerprint in the virtual machine's console logs. Eyeball the two 32-character hexadecimal strings. Type "yes" and hope that they really were the same and not just mostly the same. Of course, if you don't care about security you could arrange for all your virtual machines to use the same host key, or use the -o StrictHostKeyChecking=no option; but as the FreeBSD Security Officer and the author of a secure online backup service neither of those are acceptable as far as I'm concerned.

My work on FreeBSD AMIs for EC2 has made me even more sensitive to the irritation of host key checking, since building a set of AMIs for the 7 EC2 regions involves launching and SSHing into no less than 20 virtual machines. A couple of weeks ago I asked twitter for advice about this; ten people replied, and two people — Daniel Shahaf and Markus Friedl — made the critical observation that I wanted to use two tools: ssh-keyscan, to get a host key in a form suitable for the known_hosts file; and ssh-keygen -lf to take the host key from that form and convert it into a fingerprint I could compare against a known good value.

January 16, 2012 11:30 AM

January 11, 2012

dds

Refactoring on the Cheap

The refactorings that a good integrated development environment can perform are impressive. Yet, there are many reasons to master some cheap-and-cheerful alternative approaches. First, there will always be refactorings that your IDE won’t support. Also, although your IDE might offer excellent refactoring support for some programming languages, it could fall short on others. Modern projects increasingly mix and match implementation languages, and switching to a specialized IDE for each language is burdensome and inefficient. Finally, IDE-provided refactorings resemble an intellectual straightjacket. If you only know how to use the ready-made refactorings, you’ll miss out on opportunities for other code improvements.

by Diomidis D. Spinellis at January 11, 2012 04:23 PM

January 08, 2012

dds

Mind Mapping

In a recent NPR interview the journalist described how I used a mind map to organize my work while I served as Secretary General for Information Systems at the Greek Ministry of Finance. A number of people asked me for more details; if you're interested read on.

by Diomidis D. Spinellis at January 08, 2012 05:42 PM

January 07, 2012

ivoras

Bullet Cache use cases, Part 2: Data sharing

In addition to simple data caching, there are some interesting advanced features made possible by Bullet Cache's unique features. Data sharing between applications (or between application instances) is a very important one, especially for the PHP environment (and other CGI-like environments). This post is a part of series on Bullet Cache use cases.

Read more...

January 07, 2012 02:49 PM

Bullet Cache use cases, Part 1: Data caching with record tags

Use cases for Bullet Cache are numerous - on the one hand it is a very convenient (and fast) memory cache server, but on the other it implements some advanced features which make it applicable in surprisingly many different scenarios. This post starts a series of descriptions on some of the real-world use cases which fit Bullet Cache suprisingly nicely, and some of these are already described in the Bullet Cache User's manual.

Read more...

January 07, 2012 01:55 PM

January 01, 2012

semicomplete - Jordan Sissel

Goodbye, 2011.

This year's been pretty good, but the last two months were pretty lame.

In the last six weeks, I found out Caramel has lymphoma, got unemployed, and had emergency surgery to remove my appendix on Christmas Day. The unemployment caused me to lose an in-progress mortgage refinance.

I'll pick up the mortgage thing once I remedy the employment problem, but I'm staying quite happily unemployed until after my kid is born - should be any day now!

Most of my career-growing moves were outside of work: at meetups, in open source efforts, or in networking with folks on IRC or twitter. Lots of awesome folks out there, so go introduce yourself. Don't be a dick. :)

I didn't write much on this site, but mainly, that was due to an increase in my activities on IRC and twitter. Most of what I published this year was code and was less writing about said code. I'd like to fix that, though.

This years successes were topped by two new major projects, fpm and logstash. I also released some major improvements to xdotool and other tools.

The current implementation of logstash isn't very old, but prototypes, hacks, and other incarnations of pretty much the same thing date back to at least 2005 and probably earlier. This project has been a long-time-coming, and Pete Fritchman and I have been talking about logstash for years, so it's nice to finally have some code shipped and a community building around it.

FPM had a crazy positive response. I wrote it as a hack, and it's used all over the place now. Bonus that people are contributing patches and other improvements as well.

Sysadvent was another excellent success, the end of which marked the 4th year and 100th article posted to the project. It is awesome seeing such community involvement from so many different authors.

This year also cemented my move to git from svn. Why? Github, mostly, and not really the features of git itself. Sharing code and patches is so much easier on github than it is with other services.

I went to CarolinaCon and OSCON to talk about logstash. I also went to DevOps Days Mountain View and gave a lightning talk on logstash.

My OSCON talk was overflowing with people standing at the back of the room, etc; it went awesomely. I've also been able to do lunchtime logstash presentations at places like Square and others. I also gave talks at BayLISA meetings. It was a good year for getting out of the house and talking about code.

I tried to get a count of how much code I'd written this year, but I had lots of web-based projects that included third-party stuff like jquery, and I'm too lazy to pick through the results and trim that stuff out. I'm up to about 70 different projects on github now, some useful; some not; all fun!

Looking forward to 2012 :)

January 01, 2012 12:29 AM

December 30, 2011

ivoras

Donate to the FreeBSD foundation!

I'm on a trip so I'll be brief: Donate to the FreeBSD Foundation! It supports and funds vital FreeBSD development! See more at http://www.freebsdfoundation.org/donate/ .

Read more...

December 30, 2011 01:29 AM

December 28, 2011

dds

Pretend Invitations

Choosing between people you want to invite to a function and people you have to invite is sometimes difficult. Say Alice wants to invite Tom, Dick, and Harry to a party, but she'd actually prefer if Dick didn't show up. Here's how Alice can send invitations by email from an email-capable Unix system to achieve the desired result, while covering her scheming with plausible deniability.

by Diomidis D. Spinellis at December 28, 2011 11:29 AM

December 25, 2011

ivoras

Lessons learned with Bullet Cache

I'm preparing for the 1.0 release of Bullet Cache and have squashed the last (known) bug which plagued it, so I'm cautiously optimistic that it deserves the "1.0" label. It's been very fun working on it and though none of this is terribly exiting news, I'd like to share a few things I've encountered while making it...

Read more...

December 25, 2011 01:58 AM

December 23, 2011

des

True Love

Around one in the morning...

“This is blasphemous, you know.”

“Hmmm?”

“This colostrum-based cold medicine. They have a promotion where they've hidden images of golden calves in some of the boxes and you win a prize if you find one. It's blasphemous.”

“Huh?”

“You know, the golden calf? The false idol the Israelites made while Moses was up on the mountain and he got so angry when he saw it that he broke the tablets and had to go back and get a new set?”

“Yes?”

“It's blasphemous. I'm probably going to hell for taking this cold medicine.”

“...”

“If I go to hell, will you come with me?”

“Yes. Now go to sleep.”

by Dag-Erling Smørgrav (noreply@blogger.com) at December 23, 2011 09:23 PM

December 21, 2011

cperciva

'Tis the season for giving

Whether you celebrate Christmas, Hanukkah, Yule, Pancha Ganapati, Hogmanay, Newtonmas, or simply the end of the Gregorian year, odds are that you're giving gifts some time around now. We give gifts to family; we give gifts to friends; we donate to charities; and many people also offer up tithes to religious institutions. Gifts to individuals are a social bonding ritual — the voluntary transfer of wealth signals a lower bound on the value we place on a relationship, and the giving of non-monetary gifts in particular can be a way to communicate our level of personal understanding — but these do not apply to charitable and religious donations. For those, I think an entirely different explanation is required: We pay voluntary taxes in order to help create the world we want to live in.

This also applies to companies. I run an online backup service, and for the past two years I've donated all of the profits made during the month of December to the FreeBSD Foundation; I'm going to be doing the same thing this year too. I'm not doing this just because I'm a FreeBSD developer, because I use FreeBSD personally, or because I would never have launched Tarsnap if I hadn't been able to build on the open source code in FreeBSD: I'm doing it because I think supporting FreeBSD development will make the world a better place for both Tarsnap and many other startup companies.

December 21, 2011 01:40 PM

December 18, 2011

des

Because you told me to

I'm tired of Windows 7 complaining every time it boots about the Bluetooth radio not being discoverable. The irony is that the Bluetooth settings dialog recommends that you turn off discoverability (for obvious reasons), but when you do, it will bitch and bitch and bitch until you turn it back on. Does anyone know how to make it stop nagging?

by Dag-Erling Smørgrav (noreply@blogger.com) at December 18, 2011 04:49 AM

December 17, 2011

dds

Advice from Successful Greek IT Startups

Members of the Hellenic Association of Mobile Application Companies and the Hellenic Semiconductor Industry Association , assorted biotechnology companies, and representatives from Greek and US-based venture capital funds gathered on Friday December 17, 2011 in a meeting to exchange advice, tips, and war stories on venturing abroad. It was one of the most inspiring meetings I've attended for some time. These are my notes from the meeting.

by Diomidis D. Spinellis at December 17, 2011 12:31 PM

cperciva

POSIX close(2) is broken

In the world of POSIX, everything is a file. Well, sort of. There's sockets and pipes, which behave rather like files except that you can't seek on them and they have some extra metadata. And there's devices, where sometimes you can only read and write appropriately-sized blocks, not individual bytes. And then there's terminals, which are all sorts of weird. But in all these cases, you've got a file descriptor, and when you're finished you release the resource by calling the close(2) system call.

There's just one small problem: The way POSIX has defined close(2) is completely and utterly broken.

December 17, 2011 08:30 AM

December 14, 2011

dds

Apps are the New Users

Some facilities provided by mature multi-user operating systems appear arcane today. Administrators of computers running Mac OS X or Linux can see users logged-in from remote terminals, they can specify limits on the disk space one can use, and they can run accounting statistics to see how much CPU time or disk I/O a user has consumed over a month. These operating systems also offer facilities to group users together, to specify various protection levels for each user's files, and to prescribe which commands a user can run.

by Diomidis D. Spinellis at December 14, 2011 04:24 PM

December 06, 2011

erwin

Rome

I’ll be back

20111206-083113.jpg

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at December 06, 2011 07:31 AM

November 27, 2011

dds

Using the HP 4470c Scanner Under Windows 7

Neither Hewlett Packard nor Microsoft Windows 7 offer native support for my HP 4470c scanner. Throwing a working scanner away to buy a new one only because some software was missing seemed like a waste, so I looked for an alternative solution. This is how I made it work using SANE , an open source framework for scanners.

by Diomidis D. Spinellis at November 27, 2011 05:54 PM

November 22, 2011

des

ZFS-to-ZFS backups

ZFS has a couple of very useful functions, zfs send and zfs receive, which allow you to serialize a complete ZFS dataset and recreate it in a different location. They can also be used to serialize a delta between two snapshots and apply that delta to a previously created copy of the dataset. You see where I'm going with this... That's right, incremental backups of a ZFS dataset or even an entire pool to a different ZFS dataset or pool.

Why would you want to perform incremental ZFS-to-ZFS backups instead of just adding redundancy to the pool, or cloning a snapshot? Because—provided the ZFS pool and filesystem versions match—it allows you to duplicate your dataset or pool on removable media (which you can store off-site), or even on a different machine across the network. This technique is far more efficient than rsync, because there is no need to compare the source and destination: ZFS already knows exactly what has changed. It also preserves the filesystem hierarchy and dataset properties.

In my case, I need to duplicate a pool onto removable media because I am replacing a server that only takes PATA disks with another that only takes SATA disks, which precludes just moving the disks over and progressively replacing them with new ones. Using this technique, when the time comes, I can slide the new server into the rack, hook up the backup disk, and restore just the parts I want to keep.

Of course, like a good little hacker, I wrote a script, which you can find here, to automate this.

The script takes two arguments: the source dataset and the destination dataset. Either of these can be the root of a ZFS pool or a dataset within a pool; they can even be datasets within the same pool, provided they do not overlap. The script selects the latest snapshot of the destination dataset (it uses a naming scheme which ensures that lexical order corresponds to chronological order), verifies that the source dataset has a snapshot with the same name, takes a new snapshot of the source dataset, and streams the difference between the old and new snapshots from the source dataset to the destination dataset. Finally, it deletes the old snapshot to allow ZFS to reclaim the space occupied by old data.

You can use this script with multiple backup disks, since it will only delete the snapshot that was actually used for the current disk. If you have one disk for each day of the week, for instance, it will delete last Monday's snapshot once it has completed this Monday's backup, but leave the other six in place. Likewise, if you decide to keep Sunday's disk for a month instead of reusing it next Sunday, the script will leave the snapshot in place until you run it again with the same disk.

The script does not currently support over-the-network backups, but it should be fairly easy to implement.

by Dag-Erling Smørgrav (noreply@blogger.com) at November 22, 2011 02:31 PM

November 18, 2011

ivoras

Encrypted PostgreSQL data types

The last few nights I have been working on something very interesting: encrypted data types for PostgreSQL. The goal here is to introduce transparent data encryption for applications which need to protect "data at rest", i.e. while the data is stored in the database within the file system. I didn't find any such mechanism (pgcrypto only offers security primitives which can be used on the application side) so I wrote pgenctypes.

Read more...

November 18, 2011 10:35 PM

November 12, 2011

ivoras

Call for testing the BSD Hypervisor (BHyVe)

Those who were present at BSDCan 2011 have probably seen or heard about the unveling of BHyVe, the native hypervisor for FreeBSD. FreeBSD is very much lacking virtualization features (not counting jails) and this is in any case excellent news for the project! Interested users are now invited to test it!

Read more...

November 12, 2011 11:30 PM

November 10, 2011

ivoras

Bullet Cache - RC1 and main features

I've just uploaded the first Release Candidate version of Bullet Cache! It is basically feature complete and done, and I'm happy to say that it looks like I have a small number of users and also some feedback on the project - so keep it up :) At this point I'd like to shortly talk about what made me write Bullet Cache - which also leads to why it was done the way it is and what are its main strengths. For the impatient, these are flexibility in cached data expiry and performance, but read on for the details...

Read more...

November 10, 2011 01:15 AM

October 30, 2011

dds

Lessons from Space

By Diomidis Spinellis and Henry Spencer We can lick gravity, but sometimes the paperwork is overwhelming. — Wernher von Braun Twilight saw the landing of Atlantis at the Kennedy Space Center on 21 July 2011, marking the end of the 30-year Space Shuttle program and leaving the Soyuz series of spacecraft as the only remaining major option for sending humans into space. With a history of 1,700 flights over an almost half-century lifetime, the Soyuz rocket and spacecraft are arguably a tremendously successful spaceflight design. Given the parallels between the complexity of human spaceflight and large software systems, what can we developers learn from the Soyuz program?

by Diomidis D. Spinellis at October 30, 2011 01:05 PM

October 25, 2011

ivoras

My digital life

I went to the customs office to pick up a package, ended up told that I need more papers than I brought in, specifically a PayPal bill printout which I didn't have. Now, the usual thing to do would be to return home, print it out, then return back to the customs office and proceed from there. What I did instead was amazing - from the point of view that almost nothing of that infrastructure even existed a few years (and especially decades) ago.

Read more...

October 25, 2011 12:17 PM

October 20, 2011

ivoras

FreeBSD's /rescue directory and system recovery

One of the many neat things FreeBSD uses to increase the operating system's resilience is the /rescue directory which is present by default and contains basic utilities and commands which can be used in case one or more of the default binaries gets corrupted or lost. In fact, binaries in /rescue are also statically linked so they can be used even if a large part of the system fails. Basically, if you have init and a shell (any shell, possibly the one from /rescue; even init is in /rescue) working, you can get your system back.

Read more...

October 20, 2011 12:59 PM

FreeBSD's /rescue directory and system recovery

One of the many neat things FreeBSD uses to increase the operating system's resilience is the /rescue directory which is present by default and contains basic utilities and commands which can be used in case one or more of the default binaries gets corrupted or lost. In fact, binaries in /rescue are also statically linked so they can be used even if a large part of the system fails. Basically, if you have init and a shell (any shell, possibly the one from /rescue; even init is in /rescue) working, you can get your system back.

Read more...

October 20, 2011 12:59 PM

October 19, 2011

ivoras

Bullet Cache - The C API tutorial

This post in the Bullet Cache series introduces the primary API to the cache server, implemented in the for of a C library. This is important as it allows for maximum portability across different application environments while at the same time retaining maximum efficiency and performance. All of the standard benchmarks were done using this exact library without any tweaking. The standard distribution also includes the PHP API which is an almost exact wrapper around the C API. Both of these APIs are also comprehensively described in the Bullet Cache User Guide.

Read more...

October 19, 2011 10:59 PM

Bullet Cache - The C API tutorial

This post in the Bullet Cache series introduces the primary API to the cache server, implemented in the for of a C library. This is important as it allows for maximum portability across different application environments while at the same time retaining maximum efficiency and performance. All of the standard benchmarks were done using this exact library without any tweaking. The standard distribution also includes the PHP API which is an almost exact wrapper around the C API. Both of these APIs are also comprehensively described in the Bullet Cache User Guide.

Read more...

October 19, 2011 10:59 PM

October 16, 2011

erwin

Furesø

20111016-193639.jpg

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at October 16, 2011 05:37 PM

October 11, 2011

ivoras

Bullet Cache - Installation and testing

I've written previously about my Bullet Cache server, and now I'd like to continue with basic installation, startup and testing. The server itself is written in a combination of C and C++, and should be usable on most modern Unix-like systems.

Read more...

October 11, 2011 04:59 PM

October 09, 2011

semicomplete - Jordan Sissel

Insist on better asserts

I never really liked C's assert() feature. If an assertion is violated, it'll tell you what assertion failed but completely lacks any context:
example: example.c:9: main: Assertion `i == 3' failed.
This is better:
Assertion failed insist.c:7 in main(), insist(i == 3): Something went wrong, wanted i == 3, got 4
The main difference here is that there's context about what failed. A message for humans looking to debug this. This is especially important on Linux these days because every distro I've used recently hates sysadmins and hates debugging - all libraries are stripped of debug symbols and coredumps are disabled by default.

What's the usage look like?

#include 
#include 
#include "insist.h"

int main() {
  int i = 4;
  //assert(i == 3);
  insist(i == 3, "Something went wrong, wanted i == 3, got %d", i);
  return 0;
} 
I also added a special 'return' version of this, 'insist_return' that lets you do error checking and early aborting like this:
insist_return(fd >= 0, START_FAILURE,
              "socket() returned %d, an error: %s", fd, strerror(errno));
Works just like insist() except returns START_FAILURE if 'fd > 0' is false and additionally logs the error formatted above.

Code here: insist.h

October 09, 2011 05:25 AM

October 01, 2011

erwin

September 30, 2011

des

More Advanced Format drives: Samsung SpinPoint F4 EcoGreen and Seagate Barracuda Green

I've acquired a couple more 2 TB Advanced Format drives: a Seagate Barracuda Green (ST2000DL003) and a Samsung SpinPoint F4 EcoGreen (HD204UI, no data sheet available online).

I was extremely impressed with the Samsung HD204UI. It's the first AF drive I've seen with decent performance. In fact, it's the fastest disk I've tested so far—its unaligned writes are faster than the non-AF Hitachi I used as a reference last time, and its aligned writes are twice as fast.

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096       43984    2979    2979
  131072    1024     512    4096      127047    1031    1031

   65536    2048       0    8192       14764    4438    8877
   65536    2048     512    8192       12453    5262   10524
   65536    2048    1024    8192       12460    5259   10518

   32768    4096       0   16384        4609    7109   28436
   32768    4096     512   16384        7829    4185   16740
   32768    4096    1024   16384        8413    3894   15579
   32768    4096    2048   16384        8211    3990   15961

   16384    8192       0   32768        3952    4145   33165
   16384    8192     512   32768        9050    1810   14481
   16384    8192    1024   32768        9317    1758   14067
   16384    8192    2048   32768        9315    1758   14069
   16384    8192    4096   32768        3996    4099   32793

The Seagate ST2000DL003, on the other hand, is so slow it's not even funny. It's actually the slowest of all the drives I've tested: its performance on aligned random writes is half that of the Western Digital WD20EARS. It's three times as fast on unaligned writes, but three times nothing (100 kBps) is still nothing (300 kBps) compared to the Samsung HD204UI (15 MBps). Here are the numbers:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096     2419280      54      54
  131072    1024     512    4096     2199286      59      59

   65536    2048       0    8192     1283667      51     102
   65536    2048     512    8192      985184      66     133
   65536    2048    1024    8192      995423      65     131

   32768    4096       0   16384       45980     712    2850
   32768    4096     512   16384      345291      94     379
   32768    4096    1024   16384      432533      75     303
   32768    4096    2048   16384      429781      76     304

   16384    8192       0   32768       34192     479    3833
   16384    8192     512   32768      166440      98     787
   16384    8192    1024   32768      210147      77     623
   16384    8192    2048   32768      207356      79     632
   16384    8192    4096   32768       34221     478    3830

This time, I also ran sequential write tests—basically, dding eight gigabytes' worth of zeroes to the disk in 128 kB blocks, which is the optimal I/O size for FreeBSD. This time, the results are pretty close: the Samsung HD204UI gets slightly less than 90 MBps, and the Seagate ST2000DL003 gets slightly less than 80 MBps.

by Dag-Erling Smørgrav (noreply@blogger.com) at September 30, 2011 02:27 AM

September 24, 2011

erwin

Wave!

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at September 24, 2011 04:16 PM

September 20, 2011

ivoras

Bullet Cache - Introduction

I am going to take the opportunity in the next few blog post to advertise my open source project - Bullet Cache, which is almost finished. Think of Bullet Cache as [*] "Memcached, only better" :) It offers some unique data query options and is blazingly fast! And good news for everyone who's asked: as promised, I've licensed it under the BSD license!

I'm announcing the availability of the 1.0-beta1 version of Bullet Cache! It is feature-complete and I consider it ready for public testing. C and PHP client libraries / "drivers" are currently implemented.

[*] for some values of "better"

Read more...

September 20, 2011 11:53 PM

erwin

It’s oh so quiet


Lukas sleeping snugly with his personal pacifier.

Related posts:

  1. It’s a boy! On August 23, 2011, we welcomed into this world Lukas...

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at September 20, 2011 06:57 PM

September 18, 2011

dds

Faking it

This column is about a tool we no longer have: the continuous rise of the CPU clock frequency. We were enjoying this trend for decades, but in the past few years, progress stalled. CPUs are no longer getting faster because their makers can’t handle the heat of faster-switching transistors. Furthermore, increasing the CPU’s sophistication to execute our instructions more cleverly has hit the law of diminishing returns. Consequently, CPU manufacturers now package the constantly increasing number of transistors they can fit onto a chip into multiple cores—processing elements—and then ask us developers to put the cores to good use.

by Diomidis D. Spinellis at September 18, 2011 02:43 PM

September 14, 2011

ivoras

"Nobody got fired for choosing IBM" - what is COTS?

I've just been to a meeting about what should have been a mundane and easy topic: there is a need for the update of a technologically old Web system of a government agency and through word-of-mouth it was thought that my group could do something good about it. As it turned out, that's a minor part of the work (and doable) and the rest of the talk was eerily Dilbert-like, practically the real-world incarnation of the phrase "Nobody got fired for choosing IBM."

Read more...

September 14, 2011 08:49 PM

September 09, 2011

erwin

It’s a boy!

Lukas Lansing

On August 23, 2011, we welcomed into this world Lukas Lansing, 53cm (21″), 3,4kg (7.5 lbs).

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at September 09, 2011 03:08 PM

September 07, 2011

edwin

iPhone dock bar

The iPhone home screen has four icons stuck at the bottom of the screen. For the last year I had there Safari, Mail, Phone and Facebook.

Yesterday I swapped Facebook for Podcaster, which I have been using more frequent in the last four months.

Facebook is for people who are bored.
Podcasts are for people who want to learn!

September 07, 2011 10:00 PM

September 05, 2011

cperciva

Lessons learned from bountying bugs

A bit over a week ago, I wrote here about the $1265 of Tarsnap bugs fixed as a result of the Tarsnap bug bounty which I've been running since April. I was impressed by the amount of traffic that post received — over 82000 hits so far — as well as the number of people who said they were considering following my example. For them and anyone else interested in "crowdsourcing" bug hunting, here's some of the lessons I learned over the past few months.

September 05, 2011 02:45 PM

September 01, 2011

cperciva

Iran forged the wrong SSL certificate

There has been a lot of talk recently about how someone — whom everyone presumes is the Iranian government — obtained a fake SSL certificate for *.google.com from DigiNotar; this is the second such case this year, as in March someone (again, presumed to be the Iranian government) obtained fraudulent certificates from Comodo for Firefox extensions, Google, Gmail, Skype, Windows Live, and Yahoo. (Interestingly, while everybody is removing DigiNotar's certificate authority key from their trusted lists, Comodo — which has issued far more certificates — is still widely trusted. I wonder if they got a free ride because nobody wants to ship "the web browser which doesn't work with my bank".)

If you want to be really evil, however, *.google.com is the wrong SSL certificate to forge. The right one? ssl.google-analytics.com.

September 01, 2011 07:30 AM

August 26, 2011

cperciva

What does $1265 of bugs look like?

Four months ago, I announced Tarsnap bug bounties ranging from $1 to $2000; and yesterday I released version 1.0.30 of the Tarsnap client code — which I'm calling the "bug bounty edition". Over four months I awarded 211 bounties totalling $1265; running the bug bounties has been a very interesting experience, and I'll be writing later about some of the lessons I've learned from it, but first I'd like to answer a simpler question: What does $1265 of bugs look like?

August 26, 2011 09:30 AM

August 04, 2011

cperciva

What I meant to say...

Yesterday morning I was interviewed on Floss Weekly about my Tarsnap backup service and my work bringing FreeBSD to Amazon EC2. This was the first non-print news interview I've done since high school and the first live interview I've ever done, and it was a very interesting experience. I have considerable experience speaking at conferences, including two talks about FreeBSD on EC2, but I was surprised at how different being interviewed was: Rather than covering topics methodically, we jumped around a lot, making it hard for me to keep track of what I had said and what still needed saying.

Watching the video this morning I was struck by the number of times that I forgot to mention something or started to say something but lost track of where the sentence was headed before I got to the end of it; so without further ado, here's the "what I meant to say" errata for the interview:

August 04, 2011 02:20 PM

edwin

Morrison Gedicht - Twee

ik wou dat ik een oma had
die ik soms zomaar op mocht bellen
en die 's avonds bij m'n bedje zat
om mij een sprookje te vertellen

maar oma's hebben allemaal al iemand
voor wie ze oma kunnen zijn
ze zitten dan wel in tehuizen
maar elke zondag is het kamertje te klein

dan komen ze allemaal op visite
en vragen of ze een zwaantje vouwt
en als ze 's avonds moe gaat slapen
weet ze dat er iemand is die van haar houdt

hoe zou het dan toch komen
dat heel veel oma's eenzaam zijn
en van hun kleine kinderen dromen
die nu veel groter en verhinderd zijn

al die oma's die truien breien
waarvan niemand zegt: wat fijn!
die hoeven me dat truitje niet te geven
maar willen ze alstjeblieft mijn oma zijn?

Morrison

August 04, 2011 06:00 AM

Morrison Gedicht - Een

het is al bijna avond
wat gaat zo'n dag toch gauw
ik klim zo in mijn bedje
en dan
denk ik weer aan jou

dan lig ik stil te luisteren
naar de geluiden om me heen
dan hoor ik
zoemen tikken fluisteren
want ik lig hier niet alleen!

soms vertel ik mijn avonturen
aan Tiberius
da's een bromvlieg
en die woont op het kozijn
dan snort ie heel tevreden
want als er iemand tegen 'm praat
dat vindt Tiberius hartstikke fijn

vandaag ook Ricky nog gesproken
die woont bij de kersenboom
het is een soort van rups
maar hij wil later vlinder worden
net als zijn vader en moeder
en zijn tante en z'n oom

zelf wil ik
als ik later groot word
proberen klein te blijven
omdat Tiberius en Ricky anders
bang voor me zijn

dan blijf ik ook dichter bij
de bloemen
en zal ik altijd
gelukkig zijn

Morrison

August 04, 2011 06:00 AM

July 31, 2011

dds

Impact Factor of Computer Science Journals 2010

The Thomson Reuters Web of Knowledge has published the 2010 Journal Citation Reports . Following similar studies I performed in 2007 , 2008 , 2009 , and 2010 , here is my analysis of the current status and trends for the impact factor of computer science journals.

by Diomidis D. Spinellis at July 31, 2011 04:15 PM

July 29, 2011

ivoras

JSP Performance - I have stumbled upon an oxymoron

As one of the commenters figured out, I made a fatal error in not running Tomcat as a standalone server, which had a horrible impact on its performance. After fixing this, the performance results between PHP and JSP are very near, probably within measurement error.

So I'm retracting this too-hastily written post and apologize for any grief it caused to Java users :)

It is the year 2011 and the "Java is slow" notoriety still isn't dead - and rightly so. In choosing the technology for a project I vas sort of leaning to use Java (or is everything which combines Java and the Web automatically called J2EE?) instead of PHP which I normally do (for better or for worse). I was expecting Java to be more optimized and together with JIT compiling, the faster solution. But I tend not to assume and do my own benchmarks so imagine my surprise...

Read more...

July 29, 2011 12:18 AM

July 27, 2011

ivoras

Some simple Xen performance measurements

When it doesn't do IO, virtualization is pretty good performance-wise these days. Here's how Xen looks like  on FreeBSD 8-stable AMD64 XENHVM (pysysbench).

Read more...

July 27, 2011 09:10 PM

des

Stalkers

I don't get Google+.

Much of it looks like a better facebook, with e.g. far better photo handling and a more nuanced relationship system—where facebook tried to patch the black-and-white “friend” concept by adding “pages” and “groups”, Google+ simply generalized it, eliminating the need for such hacks.

However, I don't understand why Google+ allows people I have never even heard of—including people hiding behind pseudonyms and whose identity even a Google search (hah!) can't reveal, so much for their controversial “real name” policy—to push updates into my “Incoming” stream. I thought the idea was to let me decide who I wanted to follow, not the other way around.

I almost feel like I'm being stalked, except it's a weird sort of reverse stalking where, instead of following me around and poking their noses into my private life, the stalkers follow me around and grind my nose into their private lives.

Who the hell are you people?

by Dag-Erling Smørgrav (noreply@blogger.com) at July 27, 2011 03:13 AM

July 26, 2011

ivoras

Isaac3 - yet another release

It looks like I'm releasing a new version of my ultra-compact single-file image gallery approximately once every two years :) It is extremely simple and definitely lacks modern features (it was started in 2005!) but if you use it, enjoy!

Read more...

July 26, 2011 08:27 PM

des

Unanimiter et constanter

Today, there are no roses to be had in Oslo, for love nor money.

* * *

I am standing by the fountain outside the City Hall. With me around the fountain are hundreds of others. Around them are thousands more, and beyond those, tens of thousands. I do not know precisely how many, and to be honest, I do not think anyone does. A hundred thousand? Two hundred thousand? Three hundred thousand? Most of them with a rose in their hand, many with two or more.

This is our finest hour.

We are gathered to mourn, but more importantly, we are gathered to show Anders Behring Breivik, and the world, that we are not scared.

I expect that many outside Norway will not understand, especially in countries which, when faced with similar actions, have answered hate with hate and violence with violence. But that is not the Norwegian way. Prime Minister Jens Stoltenberg said on Friday that “we are a small nation, we are a proud nation”. He should have added “we are a stubborn nation”. Anders Behring Breivik wants us to be scared. Therefore, we will not be scared.

As I look around from my spot by the fountain, waiting for the speeches to start, I note that there are no snipers on the roofs of the surrounding buildings. No armored cars. No police in riot gear. A police helicopter buzzes high above me. Crowd control has been delegated to security guards who are working pro bono.

While we wait, the crowd sings, alternating between the national anthem and Nordahl Grieg's “Til Ungdommen”.

Their Royal Highnesses Crown Prince Haakon and Crown Princess Mette-Marit; Princess Märtha Louise; Prime Minister Jens Stoltenberg and his wife Ingrid Schulerud; around a third of the cabinet; Mayor Fabian Stang; leader of the Labor party's youth organization and Utøya survivor Eskil Pedersen; former Prime Minister and “mother of the people” Gro Harlem Brundtland: these are some of the dignitaries attending this rose rally. Their Royal Highnesses and the Prime Minister are accompanied, I presume, by the usual half-dozen smartly dressed and highly trained young men and women. The others have no security detail whatsoever. What they have instead is, to paraphrase the late King Olav V, a hundred thousand bodyguards.

* * *

A few hours later, the crowd is dispersing quietly through the streets of Oslo, depositing their flowers here and there, covering fountains, statues, trees, fences... Many gather outside the cathedral, where several hundred square meters of pavement are already covered in flowers and candles.

Spontaneous applause breaks out wherever police, paramedics or firefighters appear. Mounted police ride slowly up Karl Johan, their harnesses festooned with roses given them by passers-by. On Egertorget, a young woman in a Red Cross uniform is sitting on a bench, crying, a rose in her hand. I work my way to the central fire station, where I offer mine to the firefighters who were the first to respond to the explosion.

I have never been more proud of my country and my city.

by Dag-Erling Smørgrav (noreply@blogger.com) at July 26, 2011 07:40 PM

July 23, 2011

dds

How I Dealt with Student Plagiarism

Panos Ipeirotis , a colleague at the NYU Stern School of Business , received considerable media attention when, in a blog post he subsequently removed , he discussed how his aggressive use of plagiarism detection software on student assignments poisoned the classroom atmosphere and tanked his teaching evaluations. As detailed in a story posted on the Chronicle of Higher Education blog , Mr. Ipeirotis proposes instead that professors should design assignments that cannot be plagiarized. Along these lines here are two methods I've used in the past.

by Diomidis D. Spinellis at July 23, 2011 01:35 PM

July 13, 2011

ivoras

Testing the (new) FBFS scheduler

As some of the regular readers may know, I'm mentoring a Google Summer of Code project that aims to bring an experimental light-weight scheduler to FreeBSD. There are many differences between Linux which is the original scheduler's initial implementation platform and FreeBSD, so the port will be more like an reimplementation of some of the ideas. This project is important because it brings a new perspective to the problem of scheduling - namely how would a simplified scheduler (without expensive tracking of process performance) behave for modern workloads.

Today, the FBFS scheduler is offered for testing!

Read more...

July 13, 2011 05:33 PM

Testing the (new) FBFS scheduler

As some of the regular readers may know, I'm mentoring a Google Summer of Code project that aims to bring an experimental light-weight scheduler to FreeBSD. There are many differences between Linux which is the original scheduler's initial implementation platform and FreeBSD, so the port will be more like an reimplementation of some of the ideas. This project is important because it brings a new perspective to the problem of scheduling - namely how would a simplified scheduler (without expensive tracking of process performance) behave for modern workloads.

Today, the FBFS scheduler is offered for testing!

Read more...

July 13, 2011 05:33 PM

July 12, 2011

ivoras

Google+ and scalability

In a curious twist, it looks like Google+ currently causes more trouble for Google than it's worth it. I have no doubt that the problems will be solved soon (if nothing else succeeds - they can always throw more hardware at it), but it's instructive to say the least.

As I'm writing this, all non-search services of Google I've tried are broken to a greater or smaller extent, starting with Google+.

Read more...

July 12, 2011 01:49 PM

Google+ and scalability

In a curious twist, it looks like Google+ currently causes more trouble for Google than it's worth it. I have no doubt that the problems will be solved soon (if nothing else succeeds - they can always throw more hardware at it), but it's instructive to say the least.

As I'm writing this, all non-search services of Google I've tried are broken to a greater or smaller extent, starting with Google+.

Read more...

July 12, 2011 01:49 PM

July 08, 2011

cperciva

FreeBSD on EC2 via defenestration

On December 13th I announced the availability of FreeBSD on EC2 t1.micro instances, and on March 22nd I announced the availability of FreeBSD on EC2 cc1.4xlarge "cluster compute" instances; but for most people, these options were either too small or much too large. Today I am happy to announce the availability of FreeBSD on EC2 m1.large and m1.xlarge "standard" instances; m2.xlarge, m2.2xlarge, and m2.4xlarge "high-memory" instances; and c1.xlarge "high-CPU" instances.

July 08, 2011 02:00 AM

July 04, 2011

cperciva

The spiped secure pipe daemon

Three weeks ago, Bump Technologies released their new Scalable TLS Unwrapping Daemon. As someone who has written in the past about using stunnel to unwrap HTTPS connections while keeping OpenSSL away from other sensitive code, I found this quite exciting: STUD is just a few hundred lines of code, compared to almost ten thousand lines for stunnel. Since complexity is highly correlated with insecurity, I trust a simple daemon which only unwraps SSL/TLS far more than I trust a multifunctional monolith.

A few days later, I was looking at my kivaloo data store, contemplating the next steps I should take with it, and I decided that it was time to add some security. After all, I'm going to be using it in Tarsnap, and the Tarsnap web server needs to access user accounting data so that people can see their account balances and recent usage; since I want the web front-end kept separate from the core backup service code, traffic will need to go over the internet. My first thought was to build this into kivaloo directly, but then I reconsidered: Why not write a generic secure pipe daemon, and keep the complexity of encryption and authentication out of kivaloo? Enter spiped.

July 04, 2011 03:45 PM

July 03, 2011

erwin

Crete, Greece

Last month, we went on a delayed honeymoon to Crete, Greece. The weather couldn’t have been better and neither could the food, so needless to say, we spent quite a lot of time near the pool. We did manage to save some time for a road trip or two and I just don’t want to keep the pictures from you.
1024x640-IMG_1450 1024x682-IMG_1712
More pictures here.

Related posts:

  1. Moving to Copenhagen Finally, last weekend the rest of my things were moved...
  2. Frederiksberg Have Last week, we saw one of the first days with...

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at July 03, 2011 06:26 PM

dds

Agility Drivers

When the facts change, I change my mind. What do you do, sir? — John Maynard Keynes A management practice is mature when even government bureaucracies decide to adopt it. The March 2011 publication of UK’s ICT strategy marks this moment by advocating that “the application of agile ICT delivery methods [...] will improve government’s capability to deliver projects successfully and realise benefits faster.”. This begs the question: were we misguided during the decades we were advocating stringent control of requirements and a tightly milestone-driven development process? Interestingly, this was not the case. We were right then, and we’re right now. Things have changed, and this is why we can nowadays smugly apply agile practices reaping impressive dividends. Numerous new factors are driving agility by increasing our productivity. Our growing ability to swiftly put together sophisticated software affords us the luxury to listen to our customers, to try out new things, to collaborate across formal boundaries, to make mistakes, to redesign as we move along—in short to be agile. Knowing these factors helps us realize when we can afford to be agile and when not. (Hint: agile development of a plane’s flight control software from the ground up is still not a good idea.)

by Diomidis D. Spinellis at July 03, 2011 05:00 PM

June 20, 2011

erwin

Note to self: IPv6 on Hetzner

Hetzner has a strange IPv6 routing setup where the default gateway is not in the same subnet as the host. Clearly, this is to avoid wasting space on glue nets, but it does not look pretty. Obviously, Linux just ignores the fact that it isn’t supposed to work, but no such luck on FreeBSD. As Bitmand wrote some time ago, it can be solved with a static route but the FreeBSD startup scripts, apply the default route before static routes, leaving it no other option but to reject the default route as it has no route to its target at that time. Instead of hacking the startup scripts, this can be solved easier with two static routes with fixed order, which is still a hack though not as ugly and less error prone when upgrading the system files. For reference, especially for myself next time I need this, here’s my full ipv6 startup configuration:

rc.conf:
ipv6_enable="YES"
ipv6_default_interface="re0"
ipv6_static_routes="defgw def"
ipv6_route_defgw="2a01:XXXX:XXXX:XXXX::1/59 -prefixlen 59 -iface re0"
ipv6_route_def="default 2a01:XXXX:XXXX:XXXX::1"
ipv6_ifconfig_re0="2a01:YYYY:YYYY:YYYY::2/64"

systctl.conf
net.inet6.ip6.accept_rtadv=1

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at June 20, 2011 12:16 PM

ivoras

My little niece

My "little" sister gave birth to a brand new baby girl about two months ago, and is now going through the usual baby-raising process, invloving a lot of crying, pooping and not much sleeping. The baby is - cute, like all babies are. I'm much drawn to the process where the baby is "Awakening" (yes, with a capital A - you know what I mean). It still has a long way to go but it already seems to have some pattern matching going on, though usually nonsensical. I usually visit my sister every few months so I am very interested in observing this process.

Read more...

June 20, 2011 09:47 AM

June 15, 2011

edwin

My mallet finger - Wednesday 15 June 2011

Last week it was decided that my mallet finger had to be fixed via an operation during which I will get two wires inserted into my finger: One below the nail which will push the broken piece back against the bone, and one through the upper bone and halfway the middle bone so that bone doesn't move anymore.

Today was the big day. We had to be in the hospital at 08:30, which is about half an hour before Dirkie and Hanorah go to school. So they slept with their grandparents and were very excited about the fact that they got breakfast in a plastic bag in the car on their way to school. I wish everybody was so easily pleaseble.

When you made the appointment to go to the hospital, you know from the moment you enter the building you have lost control over your life until you leave you are on somebody elses schedule. It will involve a lot of waiting, and there is nothing you can do about it:

  • Waiting at reception to go to the waiting area for day surgery.
  • Waiting at the waiting area for day surgery to go to the beds at day surgery.
  • Waiting at the beds at day surgery to go to the pre-operation room.
  • Waiting at the pre-operation room to go to the operation waiting room.
  • Waiting at the operation waiting room to go into the operation room.
  • No waiting here, because this operation room is expensive!
  • Waiting at the operation room to go into the recovery room.
  • Waiting at the recovery room to go to the day surgery.
  • Waiting at the day surgery to get out.

Before you get in the operation room, you will be asked the same question every time: Your name, date of birth, name of the doctor and what they are going to do on you. Just to make sure they have the right guy in front of them.

The procedure done on my finger was over in 20 minutes. The anastetic I got was a finger block, two needles in my hand which neutralized all feeling in the ring finger, and some drowsiness stuff which I think didn't really work at all. During the operation I could hear the drill, but not feel the things they did on my finger.

At 14:30, after the operation and when I was out of bed, I was given an arm sling to keep my hand up and a prescription for painkillers. Six hours real-time for a 20 minutes procedure, it's very low duty-cycle.

The finger itself now has a splint at the top of the hand and a lot of bandage around it. You can see the wire sticking out at the top of my finger, which is right now not scary yet...

I was told to take the painkillers when my fingers started to tingle, which was a couple of hours later. Since the pain didn't come back after that, I didn't take anymore and slept through the night.

So is there pain? Yes and no.

There is irritated skin (for lack of a better description) around where the wires are sticking out. But there is no pain because of the drilling, which can be either because there is no pain or because the nerves in my pink and ring finger there are numb: For the last four years I haven't had any feeling in them. I have seen a specialist for it who has done the famous frog tests which will pull your muscles when an electrical current is going through them and they didn't find anything wrong with the nerves there.

Maybe that has gotten me through the night without painkillers, maybe there was no pain to start with...

On Monday I have my first physiotherapy at 08:00.

June 15, 2011 06:00 AM

June 10, 2011

erwin

Moving to Copenhagen

Finally, last weekend the rest of my things were moved from Tranbjerg near Aarhus to Amager near Copenhagen. The trip (pictures) included both a quiet and beautiful ferry trip across the Kattegat and a trip to western Jutland with the tiny truck. The end result: A lot of boxes everywhere…
1024x682-IMG_1444 1024x682-IMG_1445

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at June 10, 2011 04:07 PM

June 09, 2011

edwin

My mallet finger - Thursday 9 June 2011

We went to see the hand-surgeon. From the X-rays he saw that the break was at a nasty location: it was broken of in the contact area of the joint. First a proper cast instead if the splint and then new X-rays in case the piece of bone was put back in place by the cast: it wasn't.

So the next options were: leave it like this and it will be half-fine or have an operation and it will be fully fine. There are two kind of operations which he could do: a screw with which the broken piece get puts back or a two-wire approach with which the broken piece gets pushed against the bone and regrows that way. Because of the size of the piece broken off we chose for the wire.

After the operation I will have two pieces of wire sticking out of my finger for four to six weeks, but they are luckily under a dressing, and have a cask for that period too. After that everything should be back in volleyball-playing-condition again!

Next update: Coming Wednesday most likely.

June 09, 2011 06:00 AM

June 08, 2011

ivoras

Bullet cache

I have written before on mdcached - my not-so-pet-project I've been doing over the years, which I'm looking forward to bring to a really usable state in the coming weeks. I've cleaned it up a lot since March and I'm confident that the core of the system is correct and reasonably bug-free. I am renaming it to "Bullet Cache" but I am still calling it "alpha" code as it is not yet feature complete (some commands are missing). Anyway, it looks like due to the combination of advances in hardware and some algorithmical improvements, its performance is really good, close to 1M transactions per second.

Read more...

June 08, 2011 11:32 PM

June 07, 2011

ivoras

World IPv6 Day - How to run 6to4 on FreeBSD

As you know, or should know, today (or tomorrow, depending on the timezone) is World IPv6 Day. What this means in practice is that a lot of top-tier IT businesses will make their web sites available via both IPv6 and IPv4, at least for a day. If you want to join in testing but your ISP is following the rest of the herd and isn't doing anything with IPv6, you can use various methods to tunnel your IPv6 traffic over IPv4.

Read more...

June 07, 2011 11:00 PM

edwin

My mallet finger - Tuesday 7 June 2011

A normal Tuesday: wake up, train, work, train, dinner, volleyball. And then hospital, because my lefthand ring-finger got damaged during the volleyball game.

When the finger got hurt, by a bad catch of a badly placed ball, it didn't seem too much. About half a second later I was off the field to get some ice for it.

The people from the Menai Sports center have a great solution for providing cold-packs for injured players: just use a bag of frozen peas, it works as well and doesn't cost as much if never returned.

The "let's see if thus goes over after ten minutes" approach worked fine, except that I nearly fainted after the ten minutes when the finger got bent. And there was a strange hump of the top of the finger. Plan B, off to the hospital.

In the Emergency department you get first investigated by the triage nurse (m/f for the Dutch people who have a different word for a male and female nurse). He wisely didn't touch the finger and wanted to make an X-ray first.

The X-ray departments these days don't work with photographic paper anymore, the receiver is a kind of flat CCD device which wirelessly transfers the image to the image library. Welcome to the 21st century, goodbye to the "clunk clunk" sound of the replacement of the film.

It showed that there was a little triangle of bone broken off and I was told that the tendon was still attached to the piece. That was the reason why it a. Hurt so much when trying to stretch the finger and b. The finger didn't get stretched anymore.

The doctor could tell me that it was called a "mallet finger" and that it was a common breakage for people who play ball-sports. I still don't know the dutch name for it, maybe it only happens in English speaking countries! The doctor knew so much about it because he had had the same injury from the same sport earlier in life. And his finger was close to back to normal and he played volleyball again, so there is hope!

Later in the hospital I got a mega-splint on it and the message for the hand-surgeon, including a CDROM with a JPEG and an uncompressed picture of the X-ray, and I went home.

June 07, 2011 01:59 PM

June 03, 2011

cperciva

Insecurity in the Jungle (disk)

A few weeks ago, in the wake of stories about Dropbox's poor security, a user of my Tarsnap online backup service mentioned that he had heard Jungle Disk recommended as a secure alternative. This surprised me, since I remembered from the early days on the Amazon Web Services developers forums that JungleDave — as the author called himself — was always far more concerned with ease of use than with security. Had things improved? I decided to investigate, and I wasn't impressed with what I found.

June 03, 2011 03:10 PM

June 01, 2011

ivoras

Progress in CPU architectures

I got access to a Sandy Bridge Xeon server for a while and decided to try some benchmarks on it. Usually, I'd run unixbench from the FreeBSD ports, but - it's old. It is very, very old and the only reason I still use it is convenience. So, having some time, I started my own multiprocessor-friendly benchmark suite. Yup, I could have named it "yet another benchmark suite" since there are so many of them. Unfortunately, most are bad, and I will try to make this one into something reasonable. Anyway, I did a comparison between some Xeons I have access to, out of curiosity.

Read more...

June 01, 2011 10:42 PM

May 30, 2011

semicomplete - Jordan Sissel

xdotool 2.20110530release

It's been about 8 months since the last xdotool release, and I think it's long overdue! This release has a ton of new feature and fixes.

Download: xdotool-2.20110530.1.tar.gz

As usual, if you find problems or have feature requests, please file bugs or send an email to the list.

Changelist since previous announcement:

2.20110530.*
  - New set_window feature: --urgency. This lets you set the urgency flag on a
    window Window managers will interpret this as something about your window
    needing attention. It might flash in the taskbar, pop up, or other.
    Original patch and suggestion by ervandew.
    Issue: http://code.google.com/p/semicomplete/issues/detail?id=39
  - New function: xdo_window_seturgency (see above)
  - Hack in OS X support as it is missing a proper clock_gettime.
    Should fix http://code.google.com/p/semicomplete/issues/detail?id=37
    Reported by ryandesign.com
  - Add support for typing UTF-8 characters. Patch from Joseph Krahn.
  - Make all output call fflush to send data immediately (for pipes). Reported
    by Andreas Wagner on the mailing list.
  - Make 'get_desktop_viewport' output usable with 'set_desktop_viewport'
    http://code.google.com/p/semicomplete/issues/detail?id=47
  - You can now make 'libxdo.a' for embedding libxdo into your binary
    (Requested by psc on the mailing list).
  - Fixed a typing bug where the keymap changes unnecessarily 
  - Should now build cleanly in C++ environments (Reported by psc on the
    mailing list)
  - bugfix: xdotool should use command names first before trying file scripts. 
    See https://bbs.archlinux.org/viewtopic.php?pid=938309 for original report.
  - Add a 'sleep' command. (Requested by Joseph Krahn via mailing list)
  - Add --relative flag to windowmve. (Requested by Anthony Thyssen via mailing
  - Add --desktop flag to the search command. This lets you search for windows
    on specific desktop. Requires a window manager that supports multiple
    desktops in a way that EWMH supports.
    Fixes http://code.google.com/p/semicomplete/issues/detail?id=38
  - Add --limit flag to search. This allows you to break the search early after
    a certain number of matches. (Requested by Anthony Thyssen)
  - New command 'getwindowgeometry' for fetching window position and size
    (Requested by Anthony Thyssen via mailing list)
  - Add --sync flag to search command; blocks until results are found.
    xdotool will search every 0.5 seconds for results.
    http://code.google.com/p/semicomplete/issues/detail?id=42
  - windowmove can now move windows along an axis. Give literal 'x' or 'y'
    instead of a coordinate and it uses the current position. (Requested by
    etnlIcarus via mailing list)
  - Add '--args N' and '--terminator TERMINATOR' to the 'exec' command.
    Default terminator unless specified (or --args is) is ':' (Requested by
    Joseph Krahn and Henning Bekel via mailing list)
  - set_desktop now supports --relative flag (+N or -N to move relative)
    (Requested by Anthony Thyssen)
  - The mouse cursor now changes during 'xdotool selectwindow' (Requested by
    Anthony Thyssen via mailing list)
  - Added '--args N' and '--terminator TERMINATOR' to the 'type' command.
  - Add 'getdisplaygeometry' command for querying the size of your screen.
    (Requested by @rrwo via twitter)
  - Add xdo_get_viewport_dimensions function.

May 30, 2011 08:34 AM

May 29, 2011

imp

Canon Rebel Xt and 32GB CF

Just thought I'd mention that I've been using a 32GB CF card with no special formatting in the Canon Rebel Xt. It has been working great for years.

by Warner Losh (noreply@blogger.com) at May 29, 2011 11:58 PM

dds

Ten Lessons I Learned from Fixing my Laptop's Motherboard

A month ago I managed to break my laptop, by reversing the polarity of a universal power supply. The repair shop diagnosed the problem as a failed motherboard, and asked for €659 to replace it. I found the price preposterous and the notion of throwing away a motherboard for a single failed component ecologically unsound. Here is how I fixed the laptop on my own, and what I learned in the process.

by Diomidis D. Spinellis at May 29, 2011 08:14 PM

May 24, 2011

ivoras

Two FreeBSD Summer of Code projects

FreeBSD has 17 Google Summer of Code projects this year and most of them look very interesting! As I am involved as a mentor or co-mentor in two of those, I would like to describe them here.

Read more...

May 24, 2011 01:18 PM

May 21, 2011

dds

Code Verification Scripts

Which of my classes contain instance variables? Which classes call the method userGet , but don't call the method userRegister ? These and similar questions often come up when you want to verify that your code is free from some errors. For example, instance variable can be a problem in servlet classes. Or you may have found a bug related to the userGet and userRegister methods, and you want to look for other places where this occurs. Your IDE is unlikely to answer such questions, and this is where a few lines in the Unix shell can save you hours of frustration.

by Diomidis D. Spinellis at May 21, 2011 05:40 PM

May 14, 2011

ivoras

BSDCan 2011 and DevSummit

Once a year, Ottawa (in Canada) becomes the hottest place to be to discuss BSD-related projects, at the event unsurprisingly named BSDCan. Hidden within this event is the FreeBSD developers' summit which is itself a great opportunity to see all the people I've communicated over the year, talk and exchange ideas. This year's BSDCan / DevSummit was one of the better ones, measured in terms of ideas and projects presented; it was definitely a success!

Read more...

May 14, 2011 07:42 PM

erwin

Summary of the FreeBSD Ports and Packages Summit at BSDCan 2011

Just a quick note to point to my slides that summarize the Ports and Packages Summit at the FreeBSD DevSummit during BSDCan 2011, which can be found here. Also, we looking forward to feedback on the PKGNG project that was announced earlier and will replace the current pkg_* tools to handle ports installation and package handling and which will be a focus for portmgr over the next few months.

Related posts:

  1. FOSDEM 2011 In a few weeks, I’ll be heading to the FOSDEM...

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at May 14, 2011 03:44 AM

May 10, 2011

jeff

Asynchronous partial truncation

I have spent a month of my life on partial truncation. Softupdates asynchronously handles the case where you were completely truncating a file, such as is the case when you delete a file. The operation would be scheduled, the in memory inode updated, and the whole thing would proceed in the background. However, when you truncated to a non zero length it would do many blocking operations while synchronously truncating. When I wrapped this synchronous operation for SUJ, I did not do it quite correctly, and as a result SUJ could leak blocks if you crashed during a partial truncation. This could actually lead to filesystem corruption if the checker confused a regular disk block for an indirect block and started freeing random pointers.

To resolve this, I modified the truncation machinery to handle partial truncation. This is hard because you may have many indirect blocks involved with many children blocks in different states. An indirect is a filesystem block that does nothing but point to other blocks, like a page table does for memory. It also had to handle ffs's somewhat complex fragment rules as well as zeroing partially empty blocks. It handles all of this now and supports an arbitrary number of partial truncations to the same file without any blocking operations. It always keeps the on disk copy safe while the in memory copy is free to grow again and indeed be truncated again after that. New pointers are not recorded in an indirect until prior truncation completes so there is no ambiguity about what revision of the file the blocks are from. This brings more complexity to fsync() which must now flush all pending truncations to disk before it can return.

The truncation code is a kind of asynchronous state machine that operates on leaf blocks first and then walks backwards up the tree until it reaches the root. This ensures that we always have a valid path to a block in case we crash. Indirects are only freed when all of their children are freed. For partial truncate, the block is zeroed only once those child pointers that need be are freed. Finally when all blocks have been freed the journal space can be reclaimed.

This post can not convey how complex this work was. It may not sound very dramatic or impressive but it truly has been one of the most complex projects I have ever undertaken.

May 10, 2011 03:41 AM

May 08, 2011

ivoras

LibreOffice 3.4? Why?

Apparently, the LibreOffice is project set to release version 3.4 ... with these changes. I find it very WTF-worthy that these changes should amount to a whole new version instead of just 3.3.3 or something like that.

Read more...

May 08, 2011 06:30 PM

erwin

Skype for Mac 5.x security vulnerability

As the Leave Comment feature on Skype’s blog does not seem to work in Safari, I’ll leave some comment on this here. Some serious information is incorrect or missing from Skype’s security information for Skype for Mac 5.x (2.x is rerported not to be affected).

Skype released a very terse message on their Mac blot today pointing to an earlier post on their Security blog. The information in this post is either incorrect or there is another vulnerability out there that they haven’t informed their users about yet. The vulnerability described in the blog post explicitly states that a specially crafted, malicious “message would have to come from someone already in your Skype Contact List”. I have seen many crashes over the last few weeks with version 5.1.0.914, none of which caused by messages from people in my contact list, but by contact requests, which for obvious reasons can be sent by people not (yet) in ones contact list.

If these crashes are related to this vulnerability, it would contradict both Skype’s statement that the malicious message has to be sent by an approved contact and that the vulnerability is not exploited in the wild. Hopefully, Skype is right and they are unrelated, but to err on the safe side I would recommend anyone using Skype for Mac 5.x to not wait for Skype to release a new version next week, but to upgrade immediately to the latest release 5.1.0.922 here. This version will not show up via the Check for Updates menu as Skype deemed the hotfix non-critical, exactly because it can only be caused by approved contacts and is not seen in the wild, so their users have to find out and install the update manually themselves.

Update: Skype has released Skype for Mac 5.1.0.935 which includes unspecified Security updates with no further explanation as they wish to “wait for the majority of our users to update before detailing / discussing any of the specific issues that have been fixed”.

Update 2:Legitimate contact requests also crash Skype.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at May 08, 2011 09:07 AM

May 07, 2011

semicomplete - Jordan Sissel

logstash's first major release - 1.0.0

Ready for log and event management that doesn't suck or drain your budget? It's time to logstash.

After lots of refactoring and improvements to logstash since the first minor release last November, logstash is ready for wider usage now.

Read my announcement here.

The logstash site is also online and has docs, intros, slides, and videos.

http://logstash.net

Happy logstashing!

May 07, 2011 08:00 AM

May 02, 2011

cperciva

Election 2011 watching guide

A quick apology to my regular readers: This blog post is non-technical and entirely about Canadian politics. My normal style and selection of topics will resume shortly.

The 41st Canadian general election is taking place today — polls are already open in Atlantic Canada — and like most Canadians, I hope to know the outcome as quickly as possible. Being a resident of British Columbia, I am disadvantaged in this respect by section 329 of the Canada Elections Act, which forbids the transmission of results from East to West until polls have closed here; so given my penchant for statistics, I decided to put together an "election results watching guide" to assist me in analyzing the results when they finally arrive.

For the benefit of other eager Canadian election-watchers, here it is:


Conservative Majority

The big question all through the election campaign has been whether Canadians will hand the Conservative party a majority government. Based on the latest polls alone, I would say that this is unlikely; but in the past two elections the incumbent party has significantly outperformed the polling consensus, and the Conservative party is known for being good at making sure that their supporters turn up to vote; so I wouldn't absolutely rule this out.

May 02, 2011 12:15 PM

May 01, 2011

dds

Choosing and Using Open Source Components

The developers of the SQLite open source database engine estimate that it’s deployed in roughly half a billion systems around the world (users include Airbus, Google, and Skype). Think of the hundreds of thousands of open source components, just one click away from you. If you know how to choose and use them effectively , your project can benefit mightily.

by Diomidis D. Spinellis at May 01, 2011 06:05 PM

April 26, 2011

erwin

Vestamager

With the lovely spring weather during the Easter holidays, it was time to explore so-called nature around a major city like Copenhagen. Not far from here is the nature reserver Vestamager. As can be seen in the picture below, it’s actually so close to downtown Copenhagen that you can see Town Hall. Even with Copenhagen Airport, modern highrises, and a highway constantly visible and audible in the background, it’s quite a nice outting, especially only with a 15 minute metro ride.

Copenhagen Town Hall and cows

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at April 26, 2011 03:33 PM

April 25, 2011

andrew

FreeBSD on BeagleBoard

I’ve managed to get FreeBSD booting to Single user mode on a BeagleBoard. I’ve uploaded a patch of the changes made. My plan now is to clean it up and submit it in stages for feedback.

This wouldn’t have been possible without the work of Ben Gray for the initial work, Mark Murry for porting the patch to a newer version of FreeBSD and Andrew Thompson for donating the hardware.

by Andrew at April 25, 2011 05:29 AM

April 24, 2011

erwin

Spring is here

Also on the inside of the window

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at April 24, 2011 10:33 AM

April 21, 2011

cperciva

Tarsnap bug bounties

When Taylor R Campbell wrote to me three months ago to point out a critical security bug in Tarsnap, he also convinced me to do something I had been considering for a long time: Instituting a bug bounty. While I awarded him the first Tarsnap bug bounty — $500 — it took me a while to iron out the details of how the program would work; but I'm happy now to officially announce the availability of Tarsnap bug bounties ranging from $1 to $2000.

April 21, 2011 06:15 AM

April 16, 2011

ivoras

My brief experience with SSDs on the desktop

This is the first time I've tried to used an SSD on a desktop, and it was primarily to try to speed up Windows booting and application start. In short: don't bother.

Read more...

April 16, 2011 09:36 PM

My brief experience with SSDs on the desktop

This is the first time I've tried to used an SSD on a desktop, and it was primarily to try to speed up Windows booting and application start. In short: don't bother.

Read more...

April 16, 2011 09:36 PM

April 05, 2011

semicomplete - Jordan Sissel

EC2 reserved vs on-demand costs (and R graphs!)

I'm sure this is covered well elsewhere online, but that's never the point of these things ;)

I was helping with some capacity planning and run-rate math today at work and found that ec2 reserved instances are much cheaper compared to on-demand - If this is obvious to you, chill out, I have historically never really used EC2 nor have I ever been close to budgeting. ;)

I proved this conclusion with some math, but frankly I like visualizations better, so I decided to learn R. I wrote an R script that will graph an on-demand vs reserved pricing for one m1.large instance (code at end of the post).

The result is this graph:

The graph says it all, and definitely tells me that we need to be reserving all of our instances at Loggly - and it gives me a rule-of-thumb:

  • If we're going to use one instance unit for at least 9 months, reserve for 3 years.
  • If we're going to use one instance unit for at least 6 months, reserve for 1 year.
  • Otherwise, stick with on-demand.
The "reserved instances" pay structure is you pay a one-time fee for access to a reduced hourly rate.

This also means that our random "debug something" deployments that are shutdown much of the time are probably best off being reserved instances as well- at least for a 1-year thing - since we are likely to use those deployments for more than half of a year.

A 3-year on-demand price for m1.large is just shy of $9000, which is twice as expensive as the 3-year reserve. Capaticy plan and maybe start buying reserved instances. Make your CFO happy.

And in case you were going to ask, I ran the same plot with data from EC2 "quaduple extra large" instances and the savings and break-even points were the same. I bet the rest of the prices flow similarly.

The R script is follows, run it with 'R --save yourscript.r':

# Values taken from http://aws.amazon.com/ec2/pricing/
# for an m1.large ("Large") instance
on_demand_hourly = 0.34
reserve_hourly = 0.12
reserve_1year = 910       
reserve_3year = 1400

# quadruple extra large instances
#on_demand_hourly = 1.60
#reserve_hourly = 0.56
#reserve_1year = 4290
#reserve_3year = 6590

on_demand_daily = on_demand_hourly * 24
reserve_daily = reserve_hourly * 24
x - c(0, 365)
y - on_demand_daily * x

# Calculate day of break-even point reserve vs on-demand rates
break_1year_x = reserve_1year / (on_demand_daily - reserve_daily)
break_3year_x = reserve_3year / (on_demand_daily - reserve_daily)

png(filename = "ec2_m1large_cost.png", width = 500, height=375)
plot(x,y, type="l", col='red', xlab="", ylab="cost ($USD)")
title("EC2 cost analysis for m1.large", 
      sprintf("(days)\n1-year is cheaper than on-demand after %.0f days of usage,\n 3-year is cheaper after %.0f days", break_1year_x, break_3year_x))
text(60, 0, sprintf("on-demand=$%.2f/hour", on_demand_hourly), pos=3)

abline(reserve_1year, reserve_daily, col='green')
text(60, reserve_1year, sprintf("1-year=$%.0f+$%.2f/hour", reserve_1year, reserve_hourly), pos=3)

abline(reserve_3year, reserve_daily, col='blue')
text(60, reserve_3year, sprintf("3-year=$%.0f+$%.2f/hour", reserve_3year, reserve_hourly), pos=3)

point_y = reserve_1year + reserve_daily * break_1year_x
points(break_1year_x, point_y)
text(break_1year_x, point_y, labels = sprintf("%.0f days", break_1year_x), pos=1)

point_y = reserve_3year + reserve_daily * break_3year_x
points(break_3year_x, point_y)
text(break_3year_x, point_y, labels = sprintf("%.0f days", break_3year_x), pos=1)

dev.off()
quit()

April 05, 2011 06:33 AM

jeff

Performance problems in SUJ

SUJ has been around for a year now and 9.0 will release with it this summer. In preparation I am working on the few known performance problems. The problems are sufficiently general to softupdates that they may be of interest to those who study different filesystem consistency mechanisms.

The new code and dependencies add some extra CPU overhead to each filesystem operation but in practice this has been negligible. However once disks reach ops per second rates similar to that of network interface cards we will have to re-evaluate filesystems entirely. Back on topic, the two classes of problems we have encountered relate to synchronous journal writes and excessive rollbacks.

You may recall that softupdates uses rollbacks to revert metadata operations that are not yet safe when a buffer is written to disk. When the write completes the change is rolled forward in memory and the buffer is marked dirty again. This allows us to separate potentially circular dependencies, rolling back some while writing others, allowing the filesystem state to move forward. This eliminates the types of journaling problems that can occur when many operations are allowed to aggregate for efficiency reasons which may lead to waiting on unrelated IO when fsync() is called. Our notion of a transaction is less simplistic.

The journaling code adds new dependencies and new rollbacks to the filesystem. Most importantly, the allocation bitmaps are now rolled back. In some cases we may discover that one filesystem operation undoes another and softupdates handles this by canceling all of the dependencies after reverting the metadata changes. It turned out there were some cases where the time between canceling the dependencies and the actual reversion of the changes could be longer than I expected. This would leave a dependency that was unsatisfied which would hold a cylinder-group dirty for several seconds. The solution was to simply allow the journal record to proceed even when we decide to cancel the operation. If the operation is undone before the write is issued we will still eliminate it, however, there is no harm in journaling an operation that does not happen. The checker will discover the true state of all the metadata and take no action.

The second problem has to do with blocking journal writes. There are some cases where rollbacks would be impractical so instead we detect them and force a synchronous journal write. There are very few instances of this in the filesystem but one that remains is particularly egregious. The checker requires that a new block allocation is journaled before the block is actually written. The filesystem assumes that it can write to datablocks in any order and indeed it does so before the allocation bits hit the cylinder group. These are not compatible so a new block which is immediately written to disk after allocation will wait first for the journal write and second for the block write, doubling the latency. This is tricky because we only need to block in the case that the previous identity of the block was as an indirect block for a file whose truncation still exists in the journal. The new record must first be written so the checker doesn't attempt to interpret the block as a table of indirect block pointers.

I haven't yet solved this second problem. My intent is to cache the list of recently freed indirect blocks in some fashion but I need to do it with the least memory and cpu overhead I can. My hope is to solve this soon. Experimental kernels where this restriction is relaxed perform as well as softupdates without journaling in all of the tests I've tried.

April 05, 2011 03:43 AM

March 30, 2011

jeff

Interactivity score in ULE

I sometimes speak with Con Kolvis who is known for several Linux schedulers. Con is an interesting fellow because his background is not CS and he is very pragmatic about desktop performance. He doesn't care for the interactivity boost that ULE and previous Linux schedulers use in various forms. He periodically challenges me to consider the interactivity algorithm and whether it is ultimately necessary and effective. Below I present some analysis done when constructing the algorithm in use in ULE and why I believe it is effective and necessary while not suffering many of the pitfalls of earlier approaches.

Firstly, let me define the properties of what I believe is a good interactivity algorithm. These were my guiding principles in creating the ULE algorithm.

1) Any interactivity boost is gained slowly and lost quickly.
2) Interactivity should be harder to achieve the greater the system load.
3) The algorithm should not be exploitable to achieve an unfair share of the CPU.
4) The algorithm should be cheap to maintain and compute.
5) There should be sufficient history to permit bursty applications like web browsers.

The ULE algorithm uses a decaying history of voluntary sleep time and run time. Similar to %cpu, however, involuntary sleep time is not considered. That is to say, threads that are waiting due to contention for CPU resources are not given an interactivity boost for their time waiting. That allows the algorithm to work properly regardless of CPU load where if you only consider %cpu eventually all threads on a busy system will look interactive.

The algorithm scales the ratio of run time to sleep time to a value between 1 and 100. This is quite awkward in the kernel where we can't use floating point math. It decides the divisor depending on which value is larger giving a sort of bimodal distribution.

Here is a graph of what we theoretically would like the score to produce before we switch the divisor around:



And here is a graph generated by running the algorithm with a matrix of inputs:



The second graph uses larger numbers as we do in the kernel to reduce rounding effects. You can see an irregularity at 45 degrees where we switch divisors when the run time exceeds the sleep time. In practice these are never computed as we define a threshold of 20 above which tasks are not considered interactive so there is no point in computing the score when run time exceeds sleep time unless this threshold is moved.

Going from left to right runtime is increasing. From background to foreground sleep time is increasing. A thread would trace a path forward and to the right depending on its behavior. When they increase equally the score quickly reaches an equilibrium well above the threshold for interactive scheduling. A thread looking to abuse the system couldn't use much more than 20% of the cpu in a steady state. This can be adjusted by reducing the interactive threshold. On a busy system this 20% dwindles depending on load, ultimately providing no advantage to a would be exploiter. A thread running right out of the gate raises its score super-linearly to 50 within milliseconds, while a recently awoken thread climbs linearly as it accumulates cpu time.

The algorithm requires a lot of sleep time to be accumulated before a thread can be considered interactive. This remembered sleep time is capped at a few seconds so it only takes a few hundred milliseconds before we discover that a thread is no-longer interactive. It does permit interactive UI applications to wake up with the lowest possible latency since they have a very high priority. If they then abuse this benefit for very long they are scheduled round-robin based on cpu utilization like other bulk tasks. In practice we have picked values that keep desktop user applications interactive as well as is possible.

March 30, 2011 02:04 AM

March 29, 2011

edwin

A proper WAN Optimization analogy

Even after working for Riverbed Technology for two and a half years now, I still have to come up with a bullet-proof analogy of how WAN optimization works.

Consider a network from one side of this planet to the other side: A round trip time of 300 milliseconds for 20 million meters. Of that 300 milliseconds, you have two factors: The speed of light to get from A to B, and serialization delay which happens at every hop and is related to the bandwidth and the size of the packet. One is constant, the other one is variable.

To move data in a stream, the throughput is limited to the smallest bandwidth in the path. Although you can move data faster on other parts, it all has to go through this one.

As an example, say you have a stream of 300 Mb and a smallest bandwidth of 2 Mbps. Without any protocol overhead, this will take 20 minutes to go through there. With a fourty byte protocol overhead and an MTU size of 1500, this will take 20 minutes and 33 seconds.

Now with WAN optimization. It consists of three parts: Optimization on TCP level, which has been ignored for now. Latency optimization on application specific protocols, which has been ignored for now. And data optimization, where the data is either compressed or only referred to. If the same or similar data gets transfered via two WAN optimizers twice, the first time you would get a relative small reduction factor, depending on the compressability of the data, while the second time you would get a large reduction factor because the data patterns is alreayd known on both devices.

If that 300 Mb is split into segments of 1024 bytes, making it 300 000 segments, and each segment has a 64 byte label, you end up with only about 19 Mb worth of labels.

Transfering that 19 Mb through a 2 Mbps link will be take 80 seconds, about 15 times faster.

Now back to the topic: A good analogy for WAN optimization.

Is it faster than light? It feels like it, but the Round Trip Time of the WAN is still the same. And the speed the packets go via is still the same.

Is it a "wormhole"? Wormhole-based paths which are shorter than a non-wormhole-based paths. The path travelled travelled for optimized traffic still has the same distance.

Is it comparable with ships, where goods are stored in containers (labels) and then transported in large bulk carriers? If the speed limit of other ships was limited to the speed limit of the bulk carriers, then it would be a good start.

Is it a train analogy, where passengers are cramped into carriages and efficiently transported across the rail network? It could be, except that on the railroad network everything is put into train carriages and transported efficiently on it. Comparing it with the French TGV and the Japanese bullet trains does not work neither, because the speed of the packets is still the same while these trains are way fast.

So, the analogy needs to use the same speed limits on the transport mechanism, and needs to give the impression that the delivery gets faster without changing the distance.

The best thing I come up with is transport of goods via large trucks instead of via small delivery vans: Goods are shipped via small delivery vans to a distribution point, stored into a single large truck which then uses the same transport infrastructure as small fast vans would have used if they would have transported their payload. Instead of a long convoy of small vans, you get one truck towards the distribution point which there gets reloaded into numerous small vans. The only thing which does not make sense yet is that small delivery vans are often 2 x 4 x 1.5 meters and big trucks are 3 x rather long x rather high, which gives the impression that size still matters while this isn't the case on WAN optimized traffic...

That is the problem if you work with magic :-)

March 29, 2011 08:00 AM

des

Pi

More-or-less aimless surfing brought me to an old blog post about math in movies, which mentions Pi and Contact. This mostly coincidental juxtaposition reminded me of the conclusion of the novel behind the latter.

When I first read the book, a zillion years ago, I thought the ending was pretty clever. Then I realized that it was incredibly dumb. It later occurred to me that it might actually be intentionally dumb, and therefore incredibly clever, because Carl Sagan really ought to have known better, but... I don't know. I think he either genuinely goofed or assumed (probably correctly, in most cases) that his readers wouldn't notice.

A short aside: if you haven't read the novel or seen the film, you haven't really missed anything. The novel is basically a not-bad-but-not-brilliant ripoff of Stanisław Lem's Głos Pana and Solaris. The film is... well, a film. It's OK, I guess, and stars several excellent actors, and is reasonably but not entirely true to the novel.

If you haven't read the novel but intend to, I should warn you that the rest of this post is a HONKIN' HUGE SPOILER.

Here goes: at the end of the film, the perceived failure of the project is more or less covered up and Ellie returns to her former job as head of the SETI program. In the novel, however, she is disgraced and (from my recollection—remember, it's been years since I read it) ends up as a glorified tour guide. She somehow manages to wrangle sufficient computer time to search for something that was hinted at earlier in the novel: a proof of a Universal Creator, embedded somewhere in the digits of π. And guess what... The computer discovers that if you print the digits of π with a specific number of digits per line, after billions and billions of digits you come across a pattern of zeroes that forms a circle on the page. There is a God. QED.

Here's the problem: π is transcendental. If you search long enough, no matter what you're looking for, you'll eventually find it.

Monkeys and typewriters, Ellie. Monkeys and typewriters.

by Dag-Erling Smørgrav (noreply@blogger.com) at March 29, 2011 02:57 AM

March 28, 2011

cperciva

The kivaloo data store

Just over a year ago, I sat down to a late breakfast with Patrick Collison to discuss his latest startup. At some point over the next couple of hours, we started talking about my online backup service, Tarsnap, and I mentioned that I was keeping my eye on some server-side scalability issues. "I'm OK for the next year at the current growth rate, but then I'll need to get a more sophisticated data store in place to handle block metadata; right now I'm using a very simple, obviously correct, but rather slow data structure."

"I'm impressed with what the rethinks are doing, but it feels like they're doing too much — my data store needs are very minimal," I continued. "Maybe I should just write my own data store; it can't take more than a few months."

I'm very pleased to finally announce the availability of version 1.0.0 of the kivaloo data store as BSD-licensed open source software.

March 28, 2011 12:40 PM

March 26, 2011

erwin

Frederiksberg Have

Last week, we saw one of the first days with warm and sunny spring weather. We decided to go for a trip to Frederiksberg Have for ice cream. Of course, we did bring the camera and shoot some of the local city wildlife. The ducks were clearly more interested in the food, but we did manage to get some actions shots. All pictures here.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at March 26, 2011 10:53 AM

March 23, 2011

des

Banana banana banana

I stumbled upon the banana argument today. I don't know whether to laugh or cry. A bit of both, I guess.

Laugh: Pardon me for resorting to such crude humor, but nearly every argument Ray Comfort presents as “proof” that the banana was designed by God for human consumption can also be used to “prove” that the human penis was designed by God for fellatio. Even better, the penis and its wrapper are reusable!

Cry: Comfort quotes Darwin to bolster his argument that the eye cannot possibly have evolved. However, the quote he uses is merely the first sentence of an entire chapter devoted to explaining how Darwin thinks it evolved. Here is the complete first paragraph:

To suppose that the eye with all its inimitable contrivances for adjusting the focus to different distances, for admitting different amounts of light, and for the correction of spherical and chromatic aberration, could have been formed by natural selection, seems, I freely confess, absurd in the highest degree. When it was first said that the sun stood still and the world turned round, the common sense of mankind declared the doctrine false; but the old saying of Vox populi, vox Dei, as every philosopher knows, cannot be trusted in science. Reason tells me, that if numerous gradations from a simple and imperfect eye to one complex and perfect can be shown to exist, each grade being useful to its possessor, as is certainly the case; if further, the eye ever varies and the variations be inherited, as is likewise certainly the case; and if such variations should be useful to any animal under changing conditions of life, then the difficulty of believing that a perfect and complex eye could be formed by natural selection, though insuperable by our imagination, should not be considered as subversive of the theory. How a nerve comes to be sensitive to light, hardly concerns us more than how life itself originated; but I may remark that, as some of the lowest organisms in which nerves cannot be detected, are capable of perceiving light, it does not seem impossible that certain sensitive elements in their sarcode should become aggregated and developed into nerves, endowed with this special sensibility.

The rest is here (chapter 6, “Organs of Extreme Perfection and Complication”)

Comfort also quotes Einstein as a further appeal to authority. The problem is that Einstein, notwithstanding his quip that “God does not play dice”, was an avowed atheist, although he used the term “agnostic” to distance himself from those he called “professional atheists”. In his own words:

I do not believe in a personal God and I have never denied this but have expressed it clearly. If something is in me which can be called religious then it is the unbounded admiration for the structure of the world so far as our science can reveal it.

by Dag-Erling Smørgrav (noreply@blogger.com) at March 23, 2011 11:02 PM

ivoras

Firefox 4 in FreeBSD ports!

Without any delay at all, Firefox 4 is available in FreeBSD ports! Big thanks to everyone involved! So far I tried Acid3 and WebM movies and they all work fine...

Read more...

March 23, 2011 01:25 PM

Firefox 4 in FreeBSD ports!

Without any delay at all, Firefox 4 is available in FreeBSD ports! Big thanks to everyone involved! So far I tried Acid3 and WebM movies and they all work fine...

Read more...

March 23, 2011 01:25 PM

flz

Quiet FAIL

Here’s me thinking “quiet” means “just tell me what I want to know”:


dev1578 ~ % yumdownloader -q—urls—source tmux
Loading “fastestmirror” plugin
Loading mirror speeds from cached hostfile

  • epel: yum.any
  • common: yum.any
  • base: yum.any
  • site-packages: yum.any
  • updates: yum.any
  • addons: yum.any
  • extras: yum.any
    Enabling site-packages-source repository
    No source RPM found for tmux – 1.2-1.x86_64
    http://yum.any/yum/centos/5_1/site-packages/SRPMS/tmux-1.2-2.src.rpm

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

by Florent Thoumie at March 23, 2011 12:11 PM

jeff

OFED, 10gigE, and SUJ

I have merged the OFED 1.5.3 Infiniband stack into FreeBSD CURRENT. We have achieved feature and performance parity with the Linux stack using a combination of wrappers and re-implementation of sensitive pieces. lwn wrote an article about the wrapper work here. Some FreeBSD developers are understandably concerned about growing a Linux kernel compat layer and how that could lower the quality of FreeBSD drivers. I don't foresee this as a real complication but only time will tell.

I'm working on bringing in support for Mellanox's 10gigE adapters now. It's always interesting for me to explore different directions operating systems take to accomplish the same features. Network buffering is one of those areas that is starkly different across operating systems. Maybe that deserves its own post.

I am now looking for bug reports for SUJ for another round of bug-fixing before 9.0 ships. There are a couple of areas where performance isn't great due to latency involved in blocking journal writes. I know how to eliminate these but it will take some time to implement. We are hoping to ship SUJ as the default in 9.0 and then I may provide an official backport to 8.x.

March 23, 2011 01:37 AM

March 22, 2011

cperciva

FreeBSD/EC2 cluster compute

A few months ago, I announced experimental FreeBSD/EC2 support, and for the past four weeks FreeBSD 8.2-RELEASE AMIs have been available on Amazon EC2; but unfortunately these have been limited to "t1.micro" instances. It's impressive how much can be done with a fraction of a CPU and 600 MB of RAM; but sometimes you really need something a bit more powerful. I'm pleased to announce that, thanks to support from SegPub and vtalk, FreeBSD is now available on cc1.4xlarge instances.

March 22, 2011 05:00 AM

March 17, 2011

edwin

All about MAC addresses

The "FreeBSD laptop as a Wireless Access Point for an iPhone" project I wrote earlier about has made me some followers, mostly they have no idea where the free internet connection comes from. But, based on the amount of download measured on it, they are enjoying it. One of the methods to determine how many people are on it is to use the output of the "arp -na" command: Every MAC address you see there is a mobile device which is associated with the wireless access-point you created.

One thing which you can do with that data is to match it against manufacturers. Very boring for non-networking techies... Don't read the rest :-)

MAC addresses consist of 12 hex-digits (48 bits) which are split in two parts: A six hex-digit (24 bits) prefix and a six hex-digit sequence number.

The MAC (or OUI as the IEEE calls it) prefix database can be found on the website of the IEEE at http://standards.ieee.org/regauth/oui/oui.txt. It contains at the moment of writing 14765 prefixes. The manuf(acturers) file from the Wireshark project can be found at http://anonsvn.wireshark.org/wireshark/trunk/manuf and contains 18321 prefixes plus a handful of shared prefixes.

Why is the one from the Wireshark project larger? Not really sure, but if you look at the registration costs it (US$ 1750 for a public registrered prefix OR US$ 1750 plus US$ 2100 per year for a private registered prefix) must be part of it. So it could be that the list from the Wireshark project has determined a bunch of the private ones. And unlike IP space which you can register in advance, you can't get a new prefix until you have certificated that you have used 95% of the sequence numbers.

Some statistics based on grep and cut and wc:

 Number of prefixes	Company
		503	Cisco
		122	Shenzhen
		112	Motorola
		109	Nokia Danmark A/S
		84	Samsung Electronics
		84	Apple
		78	Intel Corp
		59	Advanced (??)
		61	Hewlett Packard
		51	Private

"Advanced" could be a mistake in here, since it matches Advanced This and Advanced That. "Private" means a company who pays the US$ 2100 per year. Shenzhen has the same issue as "Advanced", it is a large bunch of companies in the Shenzhen city in China (near Hongkong). Apple was the company I didn't expect in the Top 10, but considering their iPhone / iPad success, it shouldn't surprise much.

Every prefix has 2 ** 24 entries in it, or 16 777 216 (about 16 million if you are conservate, or 17 million if you are optimistic), making there 1.4 billion Apple MAC addresses in the world. That number is not the number of Apple devices, since you need one per network interface: Ethernet, wireless or Bluetooth.

But the other number of Cisco is much more impressive: 8 438 939 648 MAC addresses. More than the next five in the list together.

Unfortunately the list of prefixes does not contain any assignment dates, it would have been interested to see what happened when LANs based on switching instead of hubs became the norm (and thus Cisco when Cisco started to sell their switches) and when mobile devices like the iPhone became popular, it would have boosted the allocation rate by Apple for sure.

MAC prefix exhaustion?

Unlike other technologies, and IPv4 comes in mind here, the MAC address prefix pool is pretty much unlimited but also only slowly being touched: There are 2 ** 22 or 4 194 304 prefixes. The number is 22, not 24 because two bits in the first byte of the prefix are used to determine if the MAC address is globally unique one or a special one. And right now, a good 35 years after the invention of Ethernet and Tokenring there are not even 19 thousand used.

The other causes are of the more strict rules the IEEE handles: You get a single prefix and don't get more until you have informed us officially that you have used 95% of them, and of course that you actually need to produce (and sell) something which uses a MAC address.

March 17, 2011 08:00 AM

March 16, 2011

ivoras

Google Summer of Code 2011

As probably everyone knows by now, Google Summer of Code 2011 is announced! As in previous years, FreeBSD is expected to have a presence and this is a great opportunity for every student interested in FreeBSD to get involved and get payed doing so! FreeBSD has a pile of ideas page and I'd like to promote some of those I think are interesting.

Read more...

March 16, 2011 01:42 AM

March 08, 2011

ivoras

mdcached

For some years (many more than really necessary) I have been workin on and off on a pet project of mine, mdcached - a cache server similar to memcached. I've figured it's time to finish it and bring it to usable state now, so (I hope) I'll be writing up the progress here. For now, I have some really nice benchmark results which show off its strengths compared to memcached.

Read more...

March 08, 2011 01:44 PM

des

Objectively subjective

What is truth?

This question is being asked with increasingly frequency these days, and I&aposm sure some of you winced when you read it. However, it was very much on my mind when I stepped out of the shower this morning, owing to the fact that my 50 l water heater does not allow me to shower long enough to get warm after a night in a chilly room.

What does that have to do with epistemology? Well, it led me to reflect on the semantics of the following statements:

  1. I am cold.
  2. I feel cold.

Setting aside their truthfulness, which of these statements is objective, and which is subjective?

The conclusion I arrived at is the opposite of the obvious one, which is that the first statement is an objective statement, while the second is a subjective statement.

The first statement, “I am cold”, is based on the speaker&aposs perception of his environment and of his body. It is entirely subjective. Although a thermometer may show that the speaker&aposs core temperature is within the normal range, the speaker is still telling the truth as he or she perceives it. Therefore, both the speaker and the thermometer are correct, even though they disagree.

The second statement, “I feel cold”, is actually a meta-statement: it is akin to saying “although this may not actually be the case, I am currently experiencing sensations consistent with having a low core temperature”. It is an objective description of the speaker's subjective sensation (unless the speaker is lying) and may be corroborated by symptoms such as shivering, goosebumps, etc.

In fact, a person with a rising fever will feel cold despite having an abnormally high core temperature, and usually also a high skin temperature.

That said, I can easily see arguments in support of the opposite thesis.

Discuss.

by Dag-Erling Smørgrav (noreply@blogger.com) at March 08, 2011 12:03 PM

March 07, 2011

ivoras

New micro-servers

As previously noted, I have an interest in running large-scale services on equipment which would be called underpowered by many, so I find that this 256-CPU-cluster-in-a-rack is in many ways a dream machine. Of course, I don't think of them as underpowered but as efficient for the task, scalable and finally, green.

Read more...

March 07, 2011 06:00 PM

March 05, 2011

ivoras

Bad experience with DealExtreme

Like probably everyone who is reading this, I occasionally buy gadgets online, usually via eBay. I've heard of DealExtreme before and some people I know had ok experiences with it so I decided to try it - that was last year in December. Due to some bad luck and apparently horrible organization of DX, I still haven't received the things I bought. At this time, effectively I've been cheated for around $400 by DealExtreme.

Read more...

March 05, 2011 03:25 PM

March 02, 2011

ivoras

Direct file versioning

Another thing I've wanted for a long time is integrated file versioning (if possible coupled with pervasive auto-save). It looks like the new Mac OS X versions will have it. Apple apparently has the knack for getting things right.

Read more...

March 02, 2011 04:09 PM

Computers in your hand

Something I've wanted for a long time is finally happening, though slowly and with shy experimental feel to it. "Smartphones" nowadays are essentially computers with more computing power than "real" desktop workstations 5 years ago: remember that it was only in late 2005 that dual-core Intel CPUs were coming fresh out from foundries and with initial clocks lower than 2 GHz. Now, all we need is good (and if possible wireless) peripheral connectivity options for smartphones to become the information centres of individual persons.

Read more...

March 02, 2011 01:02 PM

edwin

Online newspapers, waiting on the iPhone to load them

One of the uses of my iPhone is as a replacement for the newspapers and other news sources. Yes, I can do this on the computer too, so far nothing new. The thing which fascinates me is how much time I spend on waiting for the data to be retrieved.

Take the ABC app. On it, I only read two sections: The Just In section and The Drum. I open the app, have to wait for the Latest News section (which is not the Just In section) has been loaded, then I tap for the overview of the sections (which is instantly) and then tap on the Just In section, for which the overview gets loaded. To read an article, I tap on that article which then gets loaded. So I have to wait three times before I can read a single article, and every other article I need to wait again. Even after having read the ABC news this way for about a good year now, it still doesn't remember which news sections I'm interested in and doesn't pre-load them.

Take the SMH app. It is a little bit nicer, you can tell it which sections you are interested in and only shows those. It opens the one you opened last time up first. Could be close to a winner! But then it has the same cons as the ABC app: It loads the index and when you tap on the article you want to read it loads that one. Waiting twice again.

Take the Volkskrant app. It has two major sections, the Just In section and the Opinion section. When you start the app, the index of the Just In section gets loaded together with the text of all the articles. When you tap on an article, it comes up immediately and then loads any images: You can read it immediately. So you only wait once.

Take the BBC app. Just like the Volkskrant app, it loads the index and contents of the main sections at startup, which also has a lot of images on it. When the article gets loaded, it will load the image on the article.

The method of loading of content done by the BBC app and the Volkskrant app is by far the most ideal way of using an online news app. I am not yet sure if I prefer the sober layout of the Volkskrant or the smooth horizontal scrolling layout of the BBC. But overall, I am looking forward to the day the Australian news apps have caught up!

March 02, 2011 08:00 AM

semicomplete - Jordan Sissel

Introducing FPM - Effing Package Management

Having become fed up with dealing with rpmbuild, spec files, debian control files, dh_make, debuild, and the whole lot, I automated my way back to sanity.

The result is a tool I call "fpm" which aims to help you make and mangle packages however you choose, all (ideally) without having to care about the internals of your particular native package format.

The goal of this project is not to undermine upstream packaging but to grant everyone the ability to trivially build and edit packages. Why? Not all software is packaged. Not all software of the version you want is packaged. And further, not all users are willing or able to take the time to learn all the ins and outs of their package build tools.

For example, you can package up your /etc/init.d directory as an RPM by doing simply this:

% fpm -s dir -t rpm -n myinitfiles -v 1.0 /etc/init.d
...
Created /home/jls/rpm/myinitfiles-1.0.x86_64.rpm
fpm will create a simple package for you and put it in your current directory. The result:
% rpm -qp myinitfiles-1.0.x86_64.rpm -l
/etc/init.d
/etc/init.d/.legacy-bootordering
/etc/init.d/NetworkManager.dpkg-backup
...

% rpm -qp myinitfiles-1.0.x86_64.rpm --provides
myinitfiles = 1.0-1
% rpm -qp myinitfiles-1.0.x86_64.rpm --requires
rpmlib(PayloadFilesHavePrefix) = 4.0-1
rpmlib(CompressedFileNames) = 3.0.4-1
You can package up any directory. But there's more.

Above, I didn't specify a package summary, so how about fixing the rpm to include the description? You can use RPMs as the source (-s flag) in fpm. There's also a helpful '-e' (--edit) flag that'll let you edit the rpm spec (or debian control) file before building.

% rpm -qp myinitfiles-1.0.x86_64.rpm --info | grep Summary
Summary     : no summary given

% fpm -s rpm -t rpm -e myinitfiles-1.0.x86_64.rpm
... this opens up $EDITOR so you can edit the spec file it generated ...
... make some changes to the spec, including adding a proper 'Summary' ...
Created /home/jls/rpm/myinitfiles-1.0-1.x86_64.rpm

% rpm -qp myinitfiles-1.0-1.x86_64.rpm --info | grep Summary
Summary     : my /etc/init.d directory
The '-s dir' flag says the source of the package is a directory. There's also support for other package sources like rubygems, other rpms, debs, and more on the way.

With FPM, you can specify dependencies, architecture, maintainer, etc. All from a simple command line, and never forcing you to learn the pain and suffering that can come with rpm spec files or debian package building.

You can install fpm with: gem install fpm

The project page is here: https://github.com/jordansissel/fpm

The wiki is here (has more examples): https://github.com/jordansissel/fpm/wiki

March 02, 2011 01:41 AM

March 01, 2011

ivoras

February 28, 2011

edwin

Volleyball, summer competition 2010/2011

Despite the serious losses our team had in the previous Winter 2010 competition, for example the zero won games and thus being ranked last, I decided to go continue in the Summer 2010/2011 competition. Minor detail: Nobody from the old group kept playing. Luckely we got a new team together.

This season wasn't as bad as the previous one despite that we still ended up second last: We won two games.

Our new team has a good players, we just don't work together or take the basic rules into account. Too many times the ball just gets slammed into our side of the net, too many times we don't setup properly, and too many times we lose the ball because we don't take a step to the left or right to properly catch and play the ball.

For the next competition, how to can explain these issues to the members of my team without upsetting him or her and everybody else? Maybe I should just do it and see what happens...

February 28, 2011 08:00 AM

February 27, 2011

ivoras

ZFS v28 in FreeBSD 9-CURRENT!

As has been expected, previously announced and tested, ZFS v28 has been committed to FreeBSD HEAD!

New features include:

  • RAID-Z3 (triple parity - one more parity drive than RAID-6)
  • Deduplication
  • Better recovery support during import (forced log rewind, read-only import)
  • Snapshot-level diff (like regular diff but working on file systems)
  • zpool split (split a RAID-1 / mirrored set of drives into separate / independant zpools)

As always, testers are welcome!

Read more...

February 27, 2011 10:28 PM

ZFS v28 in FreeBSD 9-CURRENT!

As has been expected, previously announced and tested, ZFS v28 has been committed to FreeBSD HEAD!

New features include:

  • RAID-Z3 (triple parity - one more parity drive than RAID-6)
  • Deduplication
  • Better recovery support during import (forced log rewind, read-only import)
  • Snapshot-level diff (like regular diff but working on file systems)
  • zpool split (split a RAID-1 / mirrored set of drives into separate / independant zpools)

As always, testers are welcome!

Read more...

February 27, 2011 10:28 PM

dds

elytS edoC

Sure, you can write English right to left. You can also write software code to look like a disc or even a train (see www.ioccc.org/1988/westley.c and 1986/marshall.c ). However, you can’t then complain when you have to fight with your magazine’s editor or production staff about accepting your column’s title for publication, or if your colleagues refuse to touch your code with a 10-foot pole. Writing code in a readable and consistent style is difficult, uninteresting, tedious, underappreciated, and, extremely important.

by Diomidis D. Spinellis at February 27, 2011 05:49 PM

February 22, 2011

cperciva

FreeBSD Update build howto

One of the questions I am asked most often about FreeBSD Update is "how can I build my own updates?". Usually I've pointed people at the FreeBSD Update server source code and wished them luck; in most cases I've heard back a while later that after spending a few days trying they gave up. I'm happy to say that thanks to Jason Helfman and Experts Exchange I can now point people at a far more useful resource.

February 22, 2011 10:00 AM

February 16, 2011

ivoras

OpenOffice, LibreOffice: why do they suck so much?

LibreOffice is a recent fork of OpenOffice, made to counter Oracle's mistreatment of the neat stuff they bought with Sun. Whatever Sun did or didn't do, its policy of openess is something so good and unique for a large company that I doubt we will see it again any time soon.

Anyway, LibreOffice has big plans. They are trying to set up a foundation to ensure resources and governance for future software development - which is a good thing, even great thing, and I hope they won't end up as some other foundations.

Read more...

February 16, 2011 07:38 PM

February 15, 2011

edwin

Project New House - introduction

Project New House

Earlier this year my family and I started project New House. After renting for about 10 years here in Australia, we close the door on that issue and open the door of our own house.

Buying a house is not cheap, but thanks to proper saving in the last ten years and some luck (Getting shares from the company you work for at the lowest possible price at the beginning of the GFC for example :-) we now own the lower part of the house while the bank owns the upper half.

The town house itself is relatively new, 15 years old, one old-lady-owner who didn't do too much damage to it. So far I haven't found any skeletons in the cupboards.

The stats are: Three bedrooms, nice kitchen, nice bathroom, a grand total of three toilets (WTF? Why does a house like this three toilets?), two car garage (so the car is allowed inside next to the bicycles).

We already met four of the four neighbours, nice people. The fifth neighbour, not met despite various communication attempts) is described as a woman who does not like children and complains about everything. Oh well, five friendly families out of six houses is not bad. And if she wants to make her life miserable, go for it.

Some things need to be fixed, adjusted or just peed on to make it ours:

Because we moved in the hottest days in summer this year (not worlds greatest idea), we know exactly where the hottest parts and least ventilated parts in the house are and have invested in a couple of ceiling-fans for these rooms.

The light in the ceiling of the bathroom might be at the right position for small Australia old ladies to look in the mirror, for tall Dutchmen it just gives me a nice halo around the head, so we will get an extra light above the mirror.

While swapping the lightbulbs for energy-saving ones, I found out that the light above the frontdoor has a screw-fitting instead of a banjonet-fitting, which gives me the opportunity to use an old present from my brother: An 1.5 Watt LED colour-changing-lightbulb. Dank broertje!

Over time I hope to give more updates on the sub-projects of Project New House, like fixing the garden (Currently covered with "Buffalo grass" or "St Augustine Grass" or "Stenotaphrum secundatum" which makes it impossible to mow with a handmower and getting a solar-thingie on the roof.

February 15, 2011 10:00 PM

February 10, 2011

des

Three things

you need to watch now if you haven't seen them already. None of them are new, they just happen to speak strongly to me right now.

1. Easy A. The actual movie is at least as funny as and far more serious than the trailer.

After you've watched it, go rent (or buy) all the classics Olive references: Say Anything, Can't Buy Me Love, Sixteen Candles, The Breakfast Club, Ferris Bueller's Day Off (sorry, can't be bothered to add links). Did I miss any? Then put this on repeat on the stereo:

2. P!nk: Fuckin' perfect. The original version, not the “family-friendly” shit they show on VH1.

3. It Gets Better. There are hundreds of clips, but you can start with Barack Obama:

Compare and contrast with whoever that guy was that P!nk sings about here:

Don't read too much into this, I'm not coming out or anything.

by Dag-Erling Smørgrav (noreply@blogger.com) at February 10, 2011 10:03 PM

February 02, 2011

imp

How to run something other than login on a tty

Sometimes you don't want to run login on the console or other terminal. The reasons for this vary. Some appliances want to boot to a shell prompt on the console for debugging purposes (this console typically isn't exposed, so no security problems there). Sometimes you want to use init's restart feature to keep critical daemons alive. I've developed several control and measurement applications that used this feature. Sometimes your want to offer a menu driven interface to your users instead of a cli-based one.

Recently, I had to add support for a menu console to FreeNAS. I couldn't find any immediately available documentation on how-to do this, I had to dive into some little-configured recesses of FreeBSD to make this happen. Thankfully, no code changes to FreeBSD were required to make this work. I thought I'd do a quick how-to here to cover the basics.

We'll start with /etc/ttys. This is the file that has entries like the following:

ttyv0 "/usr/libexec/getty Pc" cons25 on secure

which tells init to run getty with the parameter Pc on /dev/ttyv0 with TERM set to cons25. Getty is a program, for those that don't know, that sets up the tty device for interactive use so that normal interaction works as you'd expect. While one can run programs without getty, especially on a 'device' that doesn't exist, I'll ignore that path for this post. This is about creating an interactive program that runs on a tty device.

The 'Pc' here is the key to understanding what getty is doing. Pc refers to the gettytab entry 'Pc' which looks something like:
P|Pc|Pc console:\
:ht:np:sp#115200:

which, according to the gettytab(5) man page means "Use 115200 baud, no parity, hard tabs". This is great if you want to run login to get an account and password, but what if you want to run a program other than login instead? Maybe one that doesn't know about tty sessions, stdin/stdout redirection etc?

The answer turns out to be fairly straight forward. You just tell tty to use a different entry. This scales well for a small number of programs, but not so well if you have dozens since you can't pass parameters to the final program. For my case, I just needed a menu for FreeNAS. I added the following entry:
#
# FreeNAS menu system entry
#
FreeNAS|freenas|FreeNAS Menu:\
:ht:np:sp#15200:lo=/etc/netcli.sh:al=root:

to /etc/gettytab. This tells getty to run /etc/netcli.sh as root. Since netcli.sh was mucking with the network it had to run as root, but there's no reason it couldn't run as a different user for safety. Once I had this entry in gettytab, I changed the above /etc/ttys line to look like:
ttyv0 "/usr/libexec/getty freenas" cons25 on secure

and sent init a hup with a "kill -1 1" command (the -1 is very important, otherwise you reboot your system). Once init reparsed /etc/ttys, netcli.sh started running.

Normally, that would be the end of it. However, in this case netcli really is a pyhton script. Why did I have to wrap it in a shell script that looks like:
#!/bin/sh
# Helper script to set the path for netcli menu
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
exec /etc/netcli

When you run out of getty, you have a very restrictive shell. The python script called a bunch of other programs, so it needed to have a good path. Also, since the shell didn't need to stick around, I tossed the typical 'exec' at the end of it to save a little bit of memory on this embedded system.

And there you have it. All the steps to create a program that runs automatically at boot. If you have any other cool tricks, please feel free to comment here...

by Warner Losh (noreply@blogger.com) at February 02, 2011 01:11 PM

January 29, 2011

ivoras

Who uses FreeBSD?

One reason why I maintain this blog is to do some advocacy for FreeBSD - attempt to spread the word about new developments and generally promote it.

It's not an easy job as the system constantly struggles for popularity aside the much more popular Linux (and on the other side - older Unixes and Windows). I've accidentally come across this page on www.freebsd.org listing some popular FreeBSD users but I think it badly needs updating.

Are you a big FreeBSD user? Working in an organization which uses FreeBSD on a larger scale? Write about it here (in comments)! No need to mention details if you think they are confidential!

Read more...

January 29, 2011 06:52 PM

January 28, 2011

jeff

wherein I replicate my feet

I sometimes do things unrelated to computers that are probably still considered geeky. I have made reference to being a cyclist before. As you know cyclists are obsessed with all things carbon fiber and I am no different. With the help of a boat builder friend of mine, I finally had an opportunity to make my own composite parts in pursuit of the perfectly fitting shoe. Behind the cut are some pictures and a description of the process.



First, to identify the goal; My feet have very high arches and long toes. This presents a number of challenges which are best solved with orthotic insoles. Unfortunately I couldn't find any I really liked. Cycling is also unusual in that a perfectly stiff insole is not undesirable. The average force on your feet is much lower than most sports and impacts are relatively non-existent.

The solution was to make a plaster of paris mold of my feet. From here I shaped the plaster and then built a plug, or positive mold inside of it. The plug was made of polyester resin and fiberglass. After curing the plaster was removed from the plug and then the plug was shaped further and sanded smooth. I essentially eliminated the toes beyond the metatarsal joints and kept everything else very close to my actual foot shape.

Once I had the mold I made a negative mold from it. Once this was also cured and shaped I sandwiched 8 layers of carbon and epoxy between the two and let them cure over night. After a lot of sanding and shaping this was the result:



Ultimately I had to completely remove the toe area so that this would fit in the shoe. I have since re-added the toe area using 2mm EVA foam. I further used a double-sided adhesive roll to adhere some suede to the insole to add a degree of comfort and traction along with a better looking finish.



Using a 3d motion capture technology I have actually shown that these insoles straightened out my feet and removed lateral motion from my knees. So not only are they exceptionally comfortable but they also improve performance by eliminating some forces applied tangential to the motion of the pedal. The total cost of goods was around $100 and it probably took 15hrs of working time.

January 28, 2011 12:38 AM

January 25, 2011

des

OpenBSD IPSec backdoor allegations: update

I'm sure I don't need to remind anyone what this is about...

The latest news: Theo now says that it is probable that NetSec was indeed contracted to insert backdoor code into OpenBSD, but after a month of review and changelog archeology, there is still no sign that they succeeded or even attempted to push tainted code into the tree.

The audit (which is still ongoing) did uncover one serious bug, but there is no reason to believe that it was planted deliberately. This relates to CBC mode, an encryption protocol in which each block of plaintext is combined with the ciphertext of the previous block before encryption to make it harder to attack ciphertext blocks individually.

If I understand Theo's message correctly,

  • It used to be common practice to use the last ciphertext block from one message as IV for the next message. This seemed like a good idea at the time, because the alternative is to generate a random IV for each new message, which requires a strong, fast PRNG, and strong, fast PRNGs didn't grow on trees back when this scheme was devised. By reusing the last ciphertext block from the previous message, a costly random IV was only required for the very first message.
  • This practice was discovered to be a bad idea because in n - 1 out of n cases (where n is the block size in bytes), the last plaintext block of any message encrypted with a block cipher contains somewhat predictable padding.
  • The flawed IV logic was replicated in several parts of the OpenBSD source tree, and the fix was implemented in some of them, but not all.
  • The person who implemented this flawed logic was at that time a NetSec employee, but he had been involved in the development of OpenBSD's IPSec stack for years before he was hired, and, as previously mentioned, he was only following common practice.
  • The same person implemented the obvious fix (generating a new, random IV for every message) once the attack was discovered.
  • The person responsible for those parts of the tree in which the fix was not implemented is one of the people fingered by Perry, but his tenure started after Perry had left and ended before the attack was discovered.
  • Anyone with any amount of experience in a large F/OSS project, or any large software development effort for that matter, can tell you that this kind of oversight is the rule rather than the exception. Although there is no evidence that he did not intentionally “forget” to fix his code, it is far more likely that he simply did not realize that the fix that had already been committed did not extend to his own code, or that he wasn't paying attention, and nobody else noticed.

My bounty still stands, and I will even relax the requirements a bit: you are not required to show that OpenBSD is still exploitable, only that it was exploitable on December 11, 2010 (the date of Perry's email to Theo).

by Dag-Erling Smørgrav (noreply@blogger.com) at January 25, 2011 05:52 PM

ivoras

FreeBSD Status Report - Q4 2010

It's time for another FreeBSD status report! Among the many news I'd like to highlight:

  1. Another effort towards a new installer - BSDInstall. Good luck :)
  2. Non-executable stack support
  3. ZFS v28
  4. TRIM Support for UFS
  5. Xen domU work enabling use of FreeBSD on EC2

Read the whole report for details!

Read more...

January 25, 2011 02:31 PM

FreeBSD Status Report - Q4 2010

It's time for another FreeBSD status report! Among the many news I'd like to highlight:

  1. Another effort towards a new installer - BSDInstall. Good luck :)
  2. Non-executable stack support
  3. ZFS v28
  4. TRIM Support for UFS
  5. Xen domU work enabling use of FreeBSD on EC2

Read the whole report for details!

Read more...

January 25, 2011 02:31 PM

January 19, 2011

des

Attack Magic Summon Item Run

That's what Final Fantasy used to be about. And leveling. And more leveling. And chocobos. Been there, done that, got the T-shirt (and a couple of others for when I'm in a Konami or Nintendo mood).

Final Fantasy XIII, on the other hand, is mostly about pressing .

FFXIII's combat system is very different from that of the previous games. The most obvious difference is that it is no longer turn-based, which was a bit of a shock to me, and which has probably angered many fans. Unless you enjoy standing still while your adversary pummels you, you often have little choice other than to select “auto-battle” and let the game make all the decisions for you. When you face an adversary you know well (or one you've Libra'ed), this usually works out OK for Commandos and Ravagers, and to a certain degree also Medics, but not for Synergists and Saboteurs. Synergists, especially, seem to stick to a predefined sequence of buffs instead of prioritizing those which are appropriate for the current adversary. For instance, there is no point in casting Protect when fighting a monster that uses only magic attacks.

At the end of every battle, you get a rating and a score which are based mainly on how long the battle lasted compared to how long the game thinks it should have. This rewards players who use Haste or are good at sneaking up on enemies (which is not always possible). However... neither the rating nor the score have any effect on the game whatsoever except for how fast your TP bar replenishes. Getting a high rating or score, which can sometimes only be achieved by consuming hard-to-get Fortisol or Deceptisol potions, does not give you better drops or help you level faster. You level by spending Crystarium points, and the amount of CP you get for each battle seems to be based exclusively on either the number of enemies you fought or their total HP (I haven't quite figured it out), and you get enough CP from the fights you can't avoid to max out your Crystarium. Thus, there is no incentive to go looking for trouble and no incentive to fight well, and no character development. All you have to do is fight the enemies that show up in the storyline, make it out alive (and if you don't, the game just takes you back to a point shortly before the fight), and spend your CP.

So far, I've played to chapter 7 (about twenty hours), and only in the last chapter or so have the fights been sufficiently difficult to require some thought and to be enjoyable, and I suspect that that's only because I haven't leveled any of my weapons or accessories. That's right, I'm still fighting with level 1 everything, and I'm doing just fine.

The storyline, by the way, is just that: a storyline. You follow a narrow path, fight the monsters along it, and watch the many cutscenes. There are a few places where you can circle around a difficult enemy or take a small detour to pick up a treasure, but you always end up back where you would have been if you'd continued straight on.

Now, I've heard that the game opens up in chapter 10, which should be about ten hours from where I am. It is not uncommon for games to have some sort of “cold open”, to borrow from TV terminology, where your choices are restricted and the outcome has little or no effect on the rest of your game, as a sort of tutorial, but I've never played a game that had a thirty hour long tutorial, and I won't blame people who give up long before then. To be honest, I'm not sure why I haven't; I guess it's partly because the graphics are gorgeous (the game itself is in 720p, but the many, many cutscenes are in 1080p), partly because I'm starting to enjoy the combat system, or at least consider it a worthy challenge, and partly because I'm hoping it'll get better.

There is plenty more to dislike about FFXIII. The story is frankly uninteresting and does not really progress; most of the cutscenes so far in the games are either about things that happened before the game started, or about the characters being emo and acting completely irrationally. Everything that happens in the cutscenes in summarized in your Datalog, which is an interesting read, and not in a good way. Not only is the prose as purple as the back cover of a pulp fantasy novel, but the emotions and inner dialog described in the text rarely bear any resemblance to what the characters actually say in the cutscenes. Let me give you an example: according to the Datalog, Hope is seething with rage and working up the strength, skills and courage to confront and kill Snow, whom he blames for his mother's death. If you only watched the cutscenes, though, you'd think he was actually working up the courage to declare his undying love to Snow—until chapter 5, where he discusses the issue openly with Lightning. However, while he does voice these thoughts and feelings in chapter 5 (and 7), they are never reflected in his intonation or body language.

Now Vanille... Vanille really gets on my nerves. I can't decide whether she is an overendowed young girl in very inappropriate dress, or an emotionally retarded young woman in very inappropriate dress. She has the body shape of a grown woman but the body language of a ten-year-old girl. She also giggles and moans a lot. In fact, she giggles and moans so much that I sometimes worry the neighbors might think I'm watching porn. Vanille is voiced by Australian actress Georgia van Cuylenburg, who stated in an interview that “[i]t was important for me to keep my Aussie accent to show that Vanille is from somewhere different”, but she doesn't sound Australian at all; to my ears, she sounds like an American voice actress who occasionally remembers that she was asked to speak with a British accent. Fang, who is from the same “somewhere different” as Vanille, is voiced by American actress Rachel Robinson; ironically, her Australian accent, while obviously fake, is far more convincing than Vanille's.

So, we've covered Hope, Vanille and (very briefly) Fang. Who's left? Snow is a conceited ass. Sazh is emo as hell in cutscenes, and jittery as hell in combat. Lightning... Lightning's OK, I guess, except for the way she treats Snow (and Serah, for that matter). She is also the most proficient fighter; Sazh would be a close second if his combat animations weren't so bloody annoying. There isn't really that much more to say about the characters; they're nicely rendered but completely unconvincing. The NPCs aren't much better; I liked Sazh's son and Hope's mother, but Snow's buddies are as annoying as he is, or worse.

BTW, where are the chocobos? I want chocobos! No, the chick is Sazh's 'fro doesn't count.

Looking forward to chapter 10...

by Dag-Erling Smørgrav (noreply@blogger.com) at January 19, 2011 07:18 PM

January 18, 2011

cperciva

Tarsnap critical security bug

Tarsnap versions 1.0.22 through 1.0.27 have a critical security bug. It may be possible for me, Amazon, or US government agencies with access to Amazon's datacenters to decrypt data stored with those versions of Tarsnap. This is an absolutely unacceptable compromise of Tarsnap's security principles, and I sincerely apologize to everyone affected.

January 18, 2011 09:05 PM

imp

FreeBSD/mips for Cavium Octeon

I've recently improved the ABI selection in FreeBSD/mips. Now it is all handled via MACHINE_ARCH or TARGET_ARCH. mipsel and mipseb are the o32 32-bit versions. mips64eb and mips64el are for n64 64-bit versions.

I've knocked together a script for building an image for the Cavium Octeon eval boards that have CF that can boot using uboot from SDK 1.9.0 and 2.0.0 (and likely earlier versions, but I've not tested them). It creates two partitions: a FAT partition for the kernel and BSD partition for the rest of FreeBSD. It also takes care of building a big endian ufs system on a little endian system. You can find the script here. I've also create an image as well you can grab here. The script contains instructions for how to create the CF image.

Enjoy

by Warner Losh (noreply@blogger.com) at January 18, 2011 09:55 AM

January 14, 2011

erwin

FOSDEM 2011

In a few weeks, I’ll be heading to the FOSDEM conference in Brussels again this year. I’ll spend most of my time at the FreeBSD booth for the FreeBSD Foundation, so if you’re there drop by to say hi, discuss the Foundation’s work, pick up a Foundation flyer, check out the swag, or make a donation. There will also be a BSD DevRoom where there will be some interesting presentations and discussions that I might attend. Remember, FOSDEM is free to attend. Hope to see you there!

Related posts:

  1. EuroBSDCon In a few days, I’ll be heading off for another...
  2. FreeBSD Foundation End-of-Year Fundraising Campaign While the snow falls outside and the holidays approaching fast,...
  3. NLLGG BSD community day, Utrecht (NL) This weekend I had the pleasure of attending the third...

Related posts brought to you by Yet Another Related Posts Plugin.


by erwin at January 14, 2011 03:49 PM

January 12, 2011

mavetju

FreeBSD laptop as a Wireless Access Point for an iPhone

Recently I was on a holiday where the provider of my iPhone had no signal, but where the provider of my 3G modem for the laptop did have a signal. At least my glass was half-full!

In the past I have tried to setup Bluetooth between my laptop and my iPhone, and that resulted in a night of hard work and no effort. This time I tried a different approach: Instead of using Bluetooth for communication, I transformed the FreeBSD laptop into a wireless access point.

The command to change the wireless card from a normal client to a wireless access point are:

[~] edwin@lappie>cat wlan-iphone
#!/bin/sh ifconfig wlan0 destroy
ifconfig wlan0 create \ wlandev ath0 \ wlanmode hostap \ bssid \ authmode open \ ssid "My iPhone WiFi" ifconfig wlan0 up
ifconfig wlan0 inet 10.0.0.1 netmask 255.255.255.0
sleep 1 sysctl -a net.inet.ip.forwarding=1 service isc-dhcpd restart

Notes:

  • Experiment with the "open" authmode before you set it to a more secure alternative. Get things working first, then secure then properly.
  • The interface "ath0" is the Atheros wireless interface of this laptop.

The 3G connection is setup via ppp(8) and to enable NAT on the outgoing packets, you need to enter the following command or add it to the right label in your ppp.conf:

ppp ON lappie> dial
Ppp ON lappie>
PPp ON lappie>
PPp ON lappie> Warning: 0.0.0.0/0: Change route failed: errno: No such process
PPP ON lappie> nat enable yes
PPP ON lappie> 

And to make sure that the connected clients get their IP address, you should run the ISC DHCP server with for example the following configuration:

option domain-name "";
option domain-name-servers 8.8.8.8; default-lease-time 150;
max-lease-time 300; ddns-update-style none; authoritative; log-facility local7; subnet 10.0.0.0 netmask 255.255.255.0 { range 10.0.0.10 10.0.0.99; option routers 10.0.0.1;
}

Notes:

  • 8.8.8.8 is the Google DNS server.

Everything is working now, your glass is full again! :-)

January 12, 2011 06:00 PM

des

4k drive update

Just to let you know what the current status is wrt. 4k drives:

It looks like the consensus in the industry (meaning everyone except Western Digital) is to announce dual sector sizes, i.e. 512-byte logical sectors on top of 4096-byte physical sectors.

Ivan Voras has taken the initiative to organize a 4k BoF at BSDCan, although judging from the (private) email exchange on the subject, it's quite possible that a decision will be made before then. Currently, it looks like we're moving towards having the low-level driver report a 512-byte sector size and 4096-byte stripe width (and, if necessary, an appropriate offset) to GEOM. This preserves backward compatibility, but announces to GEOM consumers that it is a good idea to do I/O in 4096-byte blocks and align data structures on 4096-byte boundaries. All that remains is then to make sure that those GEOM consumers we care about (particularly ZFS) take advantage of this information.

The situation for WD “Advanced Format” drives is a bit more complex, because they announce 512-byte logical sectors. The only solution I can see is to add a quirk system to the ada driver (and possibly to ata as well, if we still care about it) similar to the ones we have for SCSI and USB devices, and match the model number. I believe /WD\d+[A-Z]+RS/ should match all existing Advanced Format drives with no false positives.

by Dag-Erling Smørgrav (noreply@blogger.com) at January 12, 2011 05:52 PM

edwin

FreeBSD laptop as a Wireless Access Point for an iPhone

Recently I was on a holiday where the provider of my iPhone had no signal, but where the provider of my 3G modem for the laptop did have a signal. At least my glass was half-full!

In the past I have tried to setup Bluetooth between my laptop and my iPhone, and that resulted in a night of hard work and no effort. This time I tried a different approach: Instead of using Bluetooth for communication, I transformed the FreeBSD laptop into a wireless access point.

The command to change the wireless card from a normal client to a wireless access point are:

[~] edwin@lappie>cat wlan-iphone
#!/bin/sh

ifconfig wlan0 destroy
ifconfig wlan0 create \
	wlandev ath0 \
	wlanmode hostap \
	bssid \
	authmode open \
	ssid "My iPhone WiFi"

ifconfig wlan0 up
ifconfig wlan0 inet 10.0.0.1 netmask 255.255.255.0
sleep 1

sysctl -a net.inet.ip.forwarding=1

service isc-dhcpd restart

Notes:

  • Experiment with the "open" authmode before you set it to a more secure alternative. Get things working first, then secure then properly.
  • The interface "ath0" is the Atheros wireless interface of this laptop.

The 3G connection is setup via ppp(8) and to enable NAT on the outgoing packets, you need to enter the following command or add it to the right label in your ppp.conf:

ppp ON lappie> dial
Ppp ON lappie>
PPp ON lappie>
PPp ON lappie> Warning: 0.0.0.0/0: Change route failed: errno: No such process
PPP ON lappie> nat enable yes
PPP ON lappie> 

And to make sure that the connected clients get their IP address, you should run the ISC DHCP server with for example the following configuration:

option domain-name "";
option domain-name-servers 8.8.8.8;

default-lease-time 150;
max-lease-time 300;

ddns-update-style none;

authoritative;

log-facility local7;

subnet 10.0.0.0 netmask 255.255.255.0 {
        range 10.0.0.10 10.0.0.99;
        option routers 10.0.0.1;
}

Notes:

  • 8.8.8.8 is the Google DNS server.

Everything is working now, your glass is full again! :-)

January 12, 2011 08:00 AM

January 10, 2011

edwin

Buying Riverbed things on eBay

Just for the fun of it, I checked out some Riverbed appliances and hardware on eBay.

  • DELL RIVERBED SERVER 3GHZ 2GB RAM - Nice piece of hardware for 188 dollars. You can't use it as a Steelhead appliance anymore: "Operating System Not Installed". Just put FreeBSD or Linux or it and you have a great server.
  • Riverbed Steelhead Central Management Console 8000 - For 500 dollars you have a Dell PowerEdge 850 only, because: "unit is unlicensed" which means that the CMC software won't function. And before you can get the license keys from Riverbed, you need to get a support contract for it.
  • Riverbed Dual Copper GIG-E Bypass PCI-X CMP-00028 - For 25 dollars you get a nice bypass card which only works on the xx10 and xx20 series hardware, not on the xx50 hardware. Make sure you check the Bypass Card Installation Guide before you buy these things!
  • Riverbed Steelhead-2000 P4SCE 2U Server w/P4 2.2GHZ/2GB - For 500 dollars you have a chassis without any capabilities, because "hard disk drives are not included.". That is not a Steelhead appliance, that is a normal computer.
  • Used Riverbed Steelhead 2010 WAN Acceleration Appliance - Nothing spectular here, except for the picture on the side. With a picture like this (despite the bad photoshopping), who can resist such a deal?
  • RIVERBED SHA-06020 STEELHEAD MODEL 6020 - Imagine paying 4000 dollars for a 120 thousand dollars box. Oh, but it doesn't come with any harddisks. Which means you have to buy new harddisks from Riverbed. But a support contract first. Plus you still have no way to get the operating system on the machine. So you buy a computer, not a Steelhead appliance. Again.

So far no luck, still haven't been able to find a full working Steelhead appliance on eBay. Time to go to bed!

January 10, 2011 01:00 PM

cperciva

Inequality in Equalland

Life in the nation of Equalland (population 80 million) is idyllic. Boring, but idyllic. By all measures, it is a wonderful place to live: Zero infant mortality; 100% high school graduation; 100% college graduation; zero unemployment; zero income inequality; a steadily rising stock market; no poverty; etc. There is one measure which raises some eyebrows, however: The wealthiest 20% of households own well over 50% of the nation's wealth.

January 10, 2011 03:00 AM

January 06, 2011

dds

Sophisticated Targeted Link Spam

What appeared to be an intelligent comment in one of my blog postings turned out to be targeted link spam. This is a worrying trend, because, although we can defend ourselves against mass attacks, we're very vulnerable to targeted strikes.

by Diomidis D. Spinellis at January 06, 2011 10:32 AM

January 03, 2011

jeff

Year in review

I have not posted in a very long time. I have been busy though and I'll try to summarize the last year here.

Firstly, I collaborated with my good friends at fairwheel bikes to work on a modification to Shimano's new electronic shifting group. You can read about that at cyclingnews.com. I replaced the stock computer with my own micro-controller that enables some advanced shifting features. I'm trying to turn this into a commercial enterprise with a friend. There is a chance that a pro team will be using it next year.

The majority of my year has been occupied with a port of the OpenFabrics Enterprise Distribution infiniband stack from Linux to FreeBSD. This is dual BSD/GPL licensed which permits the port. In pursuit of this I have created a 10,000 line Linux kernel api compatibility layer which allows us to run the vast majority of the infiniband code unmodified. As I mentioned on arch@freebsd the following pieces are emulated:

> atomics, types, bitops, byte order conversion, character devices, pci
> devices, dma, non-device files, idr tables, interrupts, ioremap, hashes,
> kobjects, radix trees, lists, modules, notifier blocks, rbtrees, rwlock,
> rwsem, semaphore, schedule, spinlocks, kalloc, wait queues, workqueues,
> timers, etc.

Additionally I have worked more on SUJ, mostly bug fixing. Kirk and Kostik have been most helpful in that and really did most of the work. There were some nasty bugs but we've whittled them down and now there are only a few performance regressions (and improvements) to concern ourselves with.

I wish I hadn't let this journal go for so long. If anyone has any specific interests let me know and I will try to post more frequently.

January 03, 2011 11:02 PM

January 01, 2011

ivoras

FreeBSD on 4K sector drives

All major FreeBSD filesystems support 4K sectors (UFS, ZFS, ext2), and so does the lower level - GEOM - but currently there's an issue of communicating this configuration between all the layers. A part of the problem is that the current drives (and the situation will probably not change during this new decade) advertise two sector sizes: both 512 byte and 4K, and the system needs to correctly interpret them. All this will be resolved when a consensus on the topic gets achieved, but until that happens (hopefully soon), there is a set of easy workarounds, which I'll describe here.

Read more...

January 01, 2011 03:37 PM

FreeBSD on 4K sector drives

All major FreeBSD filesystems support 4K sectors (UFS, ZFS, ext2), and so does the lower level - GEOM - but currently there's an issue of communicating this configuration between all the layers. A part of the problem is that the current drives (and the situation will probably not change during this new decade) advertise two sector sizes: both 512 byte and 4K, and the system needs to correctly interpret them. All this will be resolved when a consensus on the topic gets achieved, but until that happens (hopefully soon), there is a set of easy workarounds, which I'll describe here.

Read more...

January 01, 2011 03:37 PM

December 31, 2010

remko

Happy New Year – 2011

In just one minute it will be 2011 (hey, scheduling things is fun, this gives you something to read while I am jumping around, celebrating with Luca, Denise, Rik and Larissa the coming of the new year, and perhaps drink a beer, or more but enough about that.

It’s my tradition to have a new years post, and this year I decided to schedule it for the first year. I wont be able to write a post before tommorrow or perhaps even later so “Sad but true”.

The last year saw a lot of sad things, sad changes and sad news, deaths and so on. Please take a minute to remember the persons you lost this year, think about the bad and the good things you shared. Cheerish those good moments, you can be upset about the negative things, but it will only make you more grumpy, which isn’t worth it. Life is too short!

OK So we considered the negative things of the last year, but ending the year with a negative thing is not right, right? So also take a minute to remember the positive things, positive changes and positive news, the birth’s you saw this year, the news that people are pregnant and are expecting a child, the new job, consider it and remember it.

From my position I would like to offer you my very best wishes for the upcoming year, I hope that you will see the positive things of life, respect eachother, and that you are healthy and can remain healthy (and your relatives).

Ofcourse my new years post wouldn’t be the same without mentioning my beloved FreeBSD. The last year we saw a few new releases, saw a lot of hard work, had to deal with the economic crises and loads of more things. This year we will get generous donations from you… right? So that we can build even more funky stuff, and keep the best operating system!

Welcome.. 2011!

by & at December 31, 2010 09:59 PM

December 29, 2010

ivoras

December 22, 2010

philip

FOSDEM 2011 keysigning

About a year ago I complained loudly that republishing material from previous years on a website suffering from content damagement system was inordinately difficult. A number of people helpfully pointed out some less-than-obvious things to me since, and I'm happy to report that getting things going was less painful this year.

There is still a lot of clicking involved and it's obviously nowhere near as intuitive as just fishing up a revision from source control. But at least my blood pressure stayed reasonably constant throughout. What can I say. I do operating systems, not websites.

Anyway - the keysigning bits for FOSDEM 2011 are online. Submit your keys.

December 22, 2010 10:42 AM

December 21, 2010

tobez

In praise of ikiwiki

I've been using ikiwiki as my private wiki for several months now, and have been very happy with it.

It's the ultimate geek wiki. You get:

  1. A simple, yet well-known default input format, markdown.
  2. A real version control system of your choosing, as opposed to some ugly bolted on custom thing (I chose git).
  3. An ability to edit the content using your favourite text editor, which is, frankly, a huge improvement over HTML text areas.
  4. It is written in Perl, and the code is reasonably clean and well thought-out. Since Perl is the language I've been using the most, it is a pretty big bonus for me.
  5. The actual wiki content is a collection of generated static HTML pages, so you don't actually have to think about CPU resources spent by the server.

Those are the big points. You get much more than that, of course, but those were the sellers for me personally.

Ikiwiki can also work as a blog, and since today, it powers this blog as well. I'll do a comprehensive writeup on what it took to convert this blog from Movable Type to ikiwiki once I am reasonably sure everything works to my satisfaction.

by tobez at December 21, 2010 02:06 PM

Do not use synergy, use synergy+

For months, I've been plagued by intermittent mouse freezes on one of my boxes.

It started after a regular Xorg upgrade. According to various mailing lists, that particular upgrade caused similar problems to a lot of people, so I tried different suggested fixes. No luck.

A bit later, Xorg on FreeBSD was modified to fix the reported problems. But the upgrade did not fix my problem.

Eventually I came to a realization that it is likely that the problem is not with the mouse driver or with any other part of Xorg. Rather, it was a problem with synergy client interaction with the new xcb. I even found a problem report with a supposed fix to the problem. By the time I've found it, the fix was committed to the synergy port, and was subsequently rolled back because it lead to other problems. I tried the patch in the PR anyway. Still did not help me.

Not wanting to spend too much time on this, I was coping with the delays and only occasionally, when annoyed more than usual, was trying to find another fix. Unsuccessfully, I must add, until this morning, when I discovered synergy+, a maintenance fork of the original synergy. I was not aware that synergy+ is basically a drop-in replacement to synergy, the binaries having the same names as in the original. Better still, synergy+ client works just fine with the original synergy server. So I've decided to give it a shot, removed the synergy package, and installed the synergy+ port. Voila, the freezes are gone. I am a happy camper now.

by tobez at December 21, 2010 12:01 PM

New smartmontools and munin

With the recent (2009-12-23) update to FreeBSD's sysutils/smartmontools port smartctl stopped working if run as non-root. I did not investigate whether it is because of the change in the way smartctl operates, or whether it just stopped to be setuid root.

Normally I don't mind going root to run smartctl by hand, but it presents a bit of a problem for the hddtemp_smartctl Munin plugin.

One possible solution is to add the munin user to the operator group, add the following two lines to /etc/devfs.conf:

perm ata 0660
perm xpt0 0660

And finally, run sh /etc/rc.d/devfs restart.

Being the dummy that I am, I only thought about a simpler solution when composing this post: just add user root into the [hddtemp_smartctl] section of your munin/plugin-conf.d/plugins.conf file. Besides being simpler, this method has an added advantage: an updated version of the sysutils/munin-node port can easily incorporate this change. Dag-Erling: hint, hint. :-)

by tobez at December 21, 2010 12:01 PM

Scraping ASP.NET sites with Perl

Today at work I needed to locate and extract, automatically, some information from a website.

There was no direct URL to the information I needed, some fields had to be filled and some POST forms had to be submitted.

Normally I would use WWW::Mechanize for such a task, but in this particular instance the situation was made somewhat less managable because the site in question was implemented with ASP.NET.

The problem with this is that every link has an associated JavaScript event handler which does some housekeeping, assigns things to funnily named hidden input fields like __EVENTTARGET and __EVENTARGUMENT and then POSTs a form.

My first thought was to try and find a CPAN module which handles those complications. Not surprizingly, there is one, aptly named HTML::TreeBuilderX::ASP_NET.

According to its documentation, the module works in combination with the standard LWP::UserAgent and HTML::TreeBuilder, and converts ASP.NET JavaScript posting redirects into an HTTP::Request object which can be fed to LWP::UserAgent's request() method. Just what the doctor ordered.

However, it turned out that my joy was a bit premature:

  • it requires Perl 5.10, which we do not yet have on our production systems;
  • documentation is incomplete and inaccurate at times - it insists naming its httpRequest() method as httpResponse();
  • it fails its own tests, not only on two machines I have tried to run them, but also on a lot of other systems according to CPAN Testers.

After a bit of pondering I decided that spending time on trying to fix the HTML::TreeBuilderX::ASP_NET module is a bit counter-productive - I needed the working code soon.

So what to do?

One thing we should keep in mind is that those JavaScript postbacks do not do anything fancy. The hidden fields that are filled in depend on what was clicked on the page, nothing else. After they are filled, a normal POST occurs.

So if we know what to POST, we could just use WWW::Mechanize and get the job done easily and quickly.

So the solution naturally splits into two parts - finding out what fields to set, and automating the process.

The first part is to launch a browser, do clicking and entering by hand, and capture what gets POSTed at each step. This capturing could be done by a variety of methods:

  • tcpdump/wireshark - listen to 'em on the wire!
  • having a proxy which outputs the POSTed parameters;
  • using a browser extension that shows POSTed parameters.

I have chosen the second option, since I had a script similar to what I need already, and since it is easy to filter out any parameters which I did not want to see, like __VIEWSTATE, which can easily be several kilobytes long.

Enter spyproxy.pl:

#! /usr/bin/perl
use strict;
use warnings;
use HTTP::Proxy;
use CGI;

my $proxy = HTTP::Proxy->new(host => "localhost");
$proxy->logmask(32); # 32 - FILTERS
$proxy->push_filter(
        request => Spy::BodyFilter->new(),
);
$proxy->start;

package Spy::BodyFilter;
use base qw(HTTP::Proxy::BodyFilter);

sub will_modify { 0 }

sub filter
{
    my ($me, undef, $req) = @_;
    print $req->method, " ", $req->uri, "\n";
    return unless $req->method eq "POST";
    my $body = $req->content;
    my $q = new CGI($body);
    for my $p ($q->param) {
        next if $p eq "__VIEWSTATE";
        print "$p\n\t", $q->param($p), "\n";
    }
}

Launch it locally in a terminal, set your browser's proxy settings to localhost:8080, and watch the output in the terminal.

The second part of the puzzle is to use the wonderful WWW::Mechanize::Shell. It provides an interactive shell, in which we can issue GET requests, see the content of the responses, view links, forms, and form fields with their values, follow the links, set the value of the fields, click on buttons and submit the forms. Best of all, after getting what we are after we can issue a script command and get a piece of Perl code that will perform all the tasks we've just done.

So the final solution looks like this:

  1. Load the start page in your browser (through the spyproxy).
  2. Load the same page in WWW::Mechanize::Shell.
  3. In the browser, fill in any fields that need filling, and click where you want.
  4. Observe the spyproxy output, note any fields that need setting. In a typical ASP.NET application, you will want to ignore the vast majority of the fields at any given moment. Don't worry, humans are good at this sort of pattern recognition. :-) Pay special attention to __EVENTTARGET and __EVENTARGUMENT fields.
  5. Set the same fields to the same values in the shell (use value fieldname fieldvalue).
  6. If __EVENTTARGET was set, type submit in the shell; otherwise, find the name of the button that was pressed (see step 4), and type click buttonname in the shell;
  7. Examine the content of the response (content in the shell) to make sure that what you've got in the shell makes sense.
  8. If more clicking and entering is to be done, go to step 3.
  9. Type script script-name.pl in the shell.
  10. Go edit script-name.pl - remove any prints you do not need, change constants you entered in the fields with variables where needed.
  11. Your custom scraping script is ready to use.
  12. ...
  13. Profit!

I hope this trick will be of use to somebody. Enjoy!

by tobez at December 21, 2010 12:01 PM

Books giveaway

For reasons which I am not going to delve into here (this is a topic for another post), we are going to get rid of about half of our books.

There are some (low) hundreds of books for the taking, slightly more than half in English, the rest being mostly Russian with a sprinkling of Danish here and there.

Fiction, non-fiction, textbooks, science fiction, you name it.

So, if you are in Copenhagen area and are interested, write me a note and consider coming over to have a look, maybe you'll find something you'd like to keep. All books are to be had for free, although we would not mind selling them if you will insist.

by tobez at December 21, 2010 12:01 PM

How to time command execution in zsh

Often I want to know how long it took for a particular command to finish.

An obvious solution to use the time(1) command does not work without a degree of anticipation on my part that I do not normally posess.

At some point I became sufficiently annoyed to actually add some hooks to my .zshrc. All commands executed in an iteractive shell are timed, but the reporting is done only for those that took longer than 10 seconds to execute.

This ugly code does the job:

note_remind=0
note_ignore="yes"
note_command="?"

note_report()
{
    echo ""
    echo "note_report: $note_command completed in $1 seconds"
}

preexec()
{
    if [ "x$TTY" != "x" ]; then
        note_remind="$SECONDS"
        note_ignore=""
        note_command="$2"
    fi
}

precmd()
{
    local xx
    if [ "x$TTY" != "x" ]; then
        if [ "x$note_ignore" = "x" ]; then
            note_ignore="yes"
            xx=$(($SECONDS-$note_remind))
            if [ $xx -gt 10 ]; then
                if [ $TTYIDLE -gt 10 ]; then
                    note_report $xx
                fi
            fi
        fi
    fi
}

Enjoy.

by tobez at December 21, 2010 12:01 PM