The second, and unfortunately the last day of BSDCan was filled with interesting talks, again with much overlap. There are simply so many interesting things going on in FreeBSD that all of them simply don't fit in just two days of conferencing! From all of those, I'd recommend (even though I wasn't able to attend some of them) the talks on netmap, ZFS, AWS, pkgng and IPv6 security - don't miss them when the videos go online!
The second day of the DevSummit continued with interesting technical discussions in the Virtualization track, which was paralleled with the Teaching OS Courses track and the Administration and Toolchain tracks. The BSDCan day began with an epic bagpipe performance followed by full four tracks of highly interesting topics - unfortunate as there is much overlap. I gave my talk on Bullet Cache which describes some of the more interesting technical aspects and presents new performance measurements.
Today I've spent quite some time chasing a bug in a legacy code at work. In retrospect, the problem is trivially simple.
It can be illustrated by the following snippet.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body>
<script src="irrelevant.js" type="text/javascript"/>
<script type="text/javascript">
function meow() { alert("meow"); }
</script>
<a href="http://dailyotter.org/"
onclick="meow(); return false;">
click for meow</a>
</body>
So, why does it show you otters instead of meowing, and how long did it take you to spot the bug?
We couldn't create a new partition or locate an existing oneLots of googling and I didn't find any hints for windows 8, but windows 7 has a similar error and folks pointed at diskpart nonsense to fix it. So let's do that -
list disk # now pick your disk select disk 0 clear create partition format fs=ntfs compress quick activeNow reboot and try the installer again, it worked for me.
Another year - another BSDCan! It's very nice and even comforting to see such a large number of familiar faces again, and even more as the ranks are filled in by fresh new developers. The conference and the Developers' Summit before it promise a great program and a great time for the BSDers.
Two weeks ago we built a slackline setup in our back yard. The issue we had was that we don't have any trees back there to tie up to. Common solutions in this case involve building an A frame and using whatever sort of anchor you can come up with, with plenty of options available.
We first used the system last Sunday with great success. It's a typical 4-carabiner primitive system but we used a double pulley system behind that to get enough tension from a single person tightening that you'd stay off the ground in the middle. There was a disturbing amount of bending and some creaking in the 4x4s, but they held.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
I'm blogging a little bit less frequently then previously simply because I have less time to do it - but I'm happy to say it's for a good cause which also makes me personally happy: I've been involved in bootstrapping the Croatian Pirate Party.
![]()
Since recent (with the very great help of Ion-Mihai Tetu, a fellow FreeBSD committer and developer for dspam) we (JR-Hosting) are running our anti-spam infrastructure on DSPAM. We stopped using SpamAssassin after some testing and resolving problems. The interesting fact is that we share most directories through nullfs so that both the webjail and the mailjail share data and our users are able to modify settings, see their stats etc. Very great and after overcoming our issues (local delivery was not OK in the beginning and the webjail was not able to properly use the MySQL database backend at first, which was odd because the main system WAS looking into it and the webjail wasn’t), it works just fine. Ofcourse it is still learning but it seems that it finds spam efficiently and quick, and it’s footprint is much much lower then SpamAssassin was. I might want to figure out how to run the daemonized version as per advise of Ion-Mihai, till then it works as a deliveryagent.
I am writing a ‘hosting environment howto’ (or something that will largely look like that) in which I will write about the setup as well.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
http://www.youtube.com/watch?v=piHLmJaytCc
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Having watched season 1 and the first episode of season 2 of Sherlock, I absolutely love the show, although 2x01 was very weird; very slow, very drawn-out, you think they've wrapped it up then discover that it was just the beginning. A bit like those games I shall not name where you play for 20 or 30 hours before you realize that was just the tutorial.
As a long-time Sherlock Holmes fan who has read all the stories and all four novels at least twice, I was very impressed by how faithful they managed to remain to the original despite moving it 120 years forward in time. All the episodes I've seen so far stick fairly close to the plot of one or two of the original stories, with a load of (often tongue-in-cheek) references to others.
The greatest departures I've seen so far are in the secondary characters: Mycroft is less sedentary than in the original stories, Moriarty is younger and crazier, and Lestrade is far more likeable and far more respectful of Holmes's intellect and methods (the original character is described in very unflattering terms and is often resentful of Holmes's success and not above taking the credit for Holmes's work).
Many self-proclaimed Sherlock Holmes fan are very critical of this series and of the Guy Ritchie movies. I disagree; I suspect that their dissatisfaction stems from a skewed view of Holmes, from reading the original text through the lens of an interminable series of movies and TV series where Holmes is depicted as an elderly, pot-bellied, gray-haired upper-class gentleman. In that respect, they're not unlike those Hunger Games fans who were shocked to see so many dark-skinned actors in the movie, despite the fact that the characters they portray are described as such in the novels.
The Guy Ritchie movies are obviously a reimagining rather than an adaptation, bearing little relationship to the plot of any of the original stories, but I think they're quite faithful to the spirit of Sherlock Holmes. Their pace is obviously much faster, but that is Ritchie's signature—cf. Tom Lehrer's brilliant take on Oh My Darling, Clementine.
by Dag-Erling Smørgrav (noreply@blogger.com) at April 05, 2012 09:36 PM
Time flies! The last year's barely passed and here we are again - another Google Summer of Code. Even though the deadline for student's submissions is less than 24 hours ahead, I'd still take the opportunity to call anyone interested, anywhere in the world, to submit a proposal - it's fun to participate!
I just discovered that Amazon allows anyone to “follow” a Kindle user, and there is no way to turn that feature off or to block followers. They say your followers can only see notes that you explicitly make public, but they don't say anything about whether they can see your book list, which is the union of books you've bought, books you've reviewed and books on your wish lists—and not just eBooks, but paper books as well. Wish lists can be made private, but the FAQ doesn't say whether that means they don't appear in my book list on kindle.amazon.com.
This makes me extremely uncomfortable. I didn't realize until now that Amazon has absolutely no privacy settings whatsoever, beyond the option of marking a wish list private (which should be the default, but isn't). It's also interesting that they describe public lists as “lists your friends can see” and not as what they really are—lists anyone can see. Arguably, that's what wish lists are for, so I don't really mind, although I have now marked all my wish lists private until I can look through them and think about what they say about me, and whether I really want to share that with the world.
(I am tempted to create a single public wishlist with nothing but My Little Pony merchandise)
In hindsight, I shouldn't be surprised that this feature exists. When I found out about kindle.amazon.com a few months ago, I was slightly baffled: I could not see the use of it, as the only functionality it offers that “Manage my Kindle” on the main site doesn't is the ability to view my highlights so I can more easily copy-paste them to my blog, in here, or wherever else I may want to quote something I've been reading on my Kindle. Now I understand: the purpose of kindle.amazon.com is to sell me.
There is an oft-repeated aphorism that goes “if you're not paying for it, you are the product”. What people may not realize is that even if you are paying for it, you may still be the product. Amazon may not sell information about their customers to marketeers, but they use it to make their products and services more attractive to other customers, viz. your friends, family and cyberstalkers.
O brave new world that has such websites in it.
by Dag-Erling Smørgrav (noreply@blogger.com) at April 05, 2012 08:38 PM
Last fall, I wrote a completely new configuration parser for OpenPAM Lycopsida. Although the new parser was far more robust than the one it replaced, it was large, unwieldy, and suffered from a number of issues relating to whitespace handling, which stemmed from reusing some old code which unfortunately was thoroughly documented and therefore could not be easily modified. So I decided to rewrite it again, from scratch this time.
Then I did what I should have done last fall but didn't: I wrote some unit tests. And of the first dozen or so tests I came up with, three failed, revealing two different bugs—one of them fairly serious.
There's a lesson in here somewhere...
by Dag-Erling Smørgrav (noreply@blogger.com) at April 02, 2012 12:29 AM
Dear Reader,
I had fixed the issue. Instead of using nullfs to get access to the /usr/home directories, I am using unionfs, which basically does the same for my goals (unless someone corrects me in misunderstanding things) and this does not seem to generate the same issues. Various sites are now running happily behind the WWW Jail. Time to finish my document on how I did setup the entire beast.
Thanks all for listening, helping, and giving tips (Alexander and Miroslav!)
So, I am still building up my jail structure and the last few evenings I was testing the FreeBSD jail wrt. PHP, Apache22-mpm-itk and wordpress.
Things started to break when I redirected external traffic to the jail. It seemed that require_once(dirname(dirname(__FILE__))) . ‘/wp-load.php’; does not work from within the jail.
I decided to do a little test and testing reveals that in a stand alone configuration the dirnames behave exactly the same, in both the host and the jail. Printing the directive within WordPress (when loading the admin pages f.ex.) reveals a ‘.’ instead of the ‘/path’ . It is resolvable by adding a ‘.’ to the directive so that wp-admin/admin.php loads the ../wp-load.php file instead of ‘/path/to/wordpress/wp-load.php’. Though this sounds very sily todo.
Did someone else encounter this? I Do not want to change enforcement of the statfs to some other value since the defaults should be good enough (given the testsript).
Relevant details: the /usr/home where the public_html files live, are nullfs rw mounted from the host and are available in the jail. The jail does username/group lookups through Ldap, and can see the various users. Apache had been build with the ITK patches so that every host runs under his/her own user. I do not see obvious differences between the regular host and the jail, the only real difference is the internal/external addresses used in the vhost configuration, but that is kinda obvious to me.
Let me know
I haven't been able to read email sent to des@des.no or des@freebsd.org for five days, due to a series of unfortunate incidents involving dodgy power supplies and the fragility of ZFS boot in FreeBSD. Work and other duties prevented me from addressing the issue in a more timely manner, but I am now regaining control. Luckily, neither my ~30 GB IMAP spool nor any other data was lost, nor did my backup MX bounce any mail. My IMAP server is now back up with a small UFS SU+J boot / root partition instead of ZFS. I am still unable to read email, but that should be fixed within 24 hours.
I also uncovered an annoying but luckily not fatal bug in the Cyrus IMAP server. When TLS is configured, the IMAP daemon stores state for each TLS session in a DB file. If that file is corrupted, the server will start, but it will refuse any incoming IMAP or LMTP connections, and will instead spit out a stream of completely unhelpful error messages. The only recourse is to delete the TLS session state database; I set up an rc script to do that at boot time, so hopefully this won't bite me again.
by Dag-Erling Smørgrav (noreply@blogger.com) at March 19, 2012 12:15 AM
Recently I have been attending Hostobzor 12th, the Russian conference of hosting providers, beeing held at Raivola hotel near St. Petersburg. The event was great as always thanks to organizers. There was a number of intersting talks given, a lot of interesting discussions held, and, what I appreciate better, a lot of new people with great ideas met.
I gave a talk on using the FreeBSD Ports system to mange a large-scale virtual hosting installations based on Hosting Telesystems experience. I tried to describe in detail how we use the ports collection to deploy a large number of servers diverced by architecture and OS versions, how we build packages and distribute them among servers, talked about how we use Mercurial VCS to incrementally merge upstream changes into our modified ports collection and FreeBSD src trees. Hopefully, I’ve not screwed it much… At least, some people was interested a lot and asked interesting questions.
If you’re interested you can grab the talk draft and slides here (in russian): [paper] [slides].
Video should be available later from HostObzor website.
This suspicious device was found recently at MEPhI microelectronics departent catacombs. After breaking the deivice's case to our great disappointment we weren't able to discover any signs of microelectronic footprint inside. The pins weren't even connected (at least it seems so). It looks like the soviet microelectonics went beyond the current nano-sized structures several decades ago. Picoelectronic device, indeed. :-)
PS: I would really like to know which microprocessor architecture this device implements. Getting FreeBSD runned there will be really great!
The Enlightenment E17 FreeBSD port has been updated to the recent snapshot today. Among bugfixes and new APIs, it also brings a lot of new libraries and modules that you will enjoy. Thanks a lot to Grzegorz Blach for preparing the original patch and for his hard work on bringing all the new e17 components to FreeBSD!
I don’t usually share the code I write for my research projects, unless someone asks for it. Anyway, today I had fun writing some C code, and I think it may turn out to be useful for other people. It’s written almost well enough to be acceptable, so here you are.
I implemented Chase’s sequence algorithm to generate the bit-strings corresponding to each combination of n elements taken t at a time. The algorithm is described in Donald E. Knuth’s “The Art Of Computer Programming”, Vol. 4, Fasc. 3, page 13. I implemented it in C and added very few comments, so refer to TAOCP if you have any doubts.
The code is distributed under the BSD 2-clauses License.
Download chaseseqcombgen.c
Posted from Providence, Rhode Island, United States.
Lately I have become more active then the last year in total for FreeBSD. I committed several enhancements that were in my queue already for a long time, but finally came into the FreeBSD tree. Some too late for 8.3 which is upcoming and some not.
I also understand merging much better then before, taking the time for a commit and making a few mistakes really helps, and ofcourse the community is not too shy to mention my faults
One of the things that got committed is the force setting of carp, whether it’s a master or backup node (make sure pre-empt is disabled if you do this, else it will just rollback within seconds
), which eases maintenance for example. One of the other things is that I am using an extensive network of local jails now that service my needs, most seperated items. I also started writing an Howto (or bsdmag article if there is interest) to demonstrate and tell how we setup most items. One of the things that is still causing me headaches is that we have nullfs rw mounts of /home to the jails because mailservers need to write, ftp servers need to write etc. But if we were sharing these information sources via NFS we would have had the same challenges
I feel good in taking the time for FreeBSD again, and I would like to hear recommendations on what targets I can persue in FreeBSD (low hanging fruit is good enough for the time being) and additional things, also please comment on the nullfs mounts (rw,nosuid) to enhance my security level and which makes my article even better

Here's a tiny ugly patch to make xpdf remember positions in files and restore them. You open a PDF document, read it a bit, then quit xpdf. Next time you open the same file, the same page will be shown as it was when you quit.
The positions are stored in ~/.xpdf.bookmarks.
If you are using xpdf from FreeBSD
ports, just put the patches into /usr/ports/graphics/xpdf/files
directory and rebuild.
I love my Kindle 3. It's a great little gadget which has increased the amount of books I read - which is by itself an outstanding accomplishment. I feel I don't need to describe its many benefits here because the interested reader either already has it or knows how to use Google well enough to find out all about it.
But, there is one very significant feature from Kindle which I very much miss - a simple text editor. I'm not a passive content consumer and I write a lot, and so since I haven't found (to my great surprise) any such thing already, available, I'd like to offer a bounty of $60 to anyone who can create a working text editor for Kindle 3, with the required features describe in the rest of this blog post.
The "ops folks need coding skills" groupthink is lame. Software requires extra coding because it is shitty, not because people are unskilled
I will lead with this: I want more people who use technology to grow and learn better skills for bending that technology to their needs. An ops guy with programming skills is, to me, more valuable than one who cannot - programming in any language or platform lets you extend an otherwise static system.
Anyway, back at the post in question, I'm not trying to say people (ops or otherwise) shouldn't want stronger programming skills. I'm saying the equipment we use is pretty shitty.
I am part of the generation raised near devices ever blinking "12:00". Devices which have no business caring what time it is, nor any sane reason to make the state of "I don't know what time it is" a high priority alert worth blinking forever.
It's 2012, and this problem persists - my microwave refuses to cook food unless it has the time *and* date from user input. Now I have to program it every time it has a power disruption (which is has frequently due to some bug in the hardware causing it to power off randomly with certain dishes at home).
Now I have to learn to program or configure these devices before they'll stop irritating me. And, damn it I hate that. If, instead, this were enterprise software, I could report these irritations to the vendor who would kindly offer me training and consulting for extortionate piles of money.
I love coding. It's fun, and many times lets me solve problems I couldn't otherwise. Allowing me to abuse an analogy, "When all you have is a hammer, you can sit down and build whatever tool you need to repair the delusion that everything is a nail."
But despite being able to solve my own problems in software, I don't think this is a great pattern of work. I write code, most of the time, because the solutions available are terrible or don't meet my requirements. With a new software popping up every day, I see a strong correlation between software availability and people asking for more programmers.
So, the more software we have, the more programmers we need to work around limitations in the available body of software. I think that's pretty lame :(
And regarding my microwave problems, I want some confidence that the problems
being solved are meaningful problems, not programming learned for the sake of
working around bugs and misfeatures in software we're suffering with.
![]()
I can best introduce you to our concerns by quoting another of our philosopher / elders, John Gilmore. He said: “The Internet interprets censorship as damage and routes around it.” To understand that, you have to grasp that “the Internet” isn’t just a network of wires and switches, it’s also a sort of reactive social organism composed of the people who keep those wires humming and those switches clicking. John Gilmore is one of them. I’m another. And there are some things we will not stand having done to our network.Dear ESR,
by Dag-Erling Smørgrav (noreply@blogger.com) at February 27, 2012 04:59 PM
How did Blu-ray[tm] ever win over HD-DVD during the so-called format wars? How could HD-DVD possibly be even worse or might it have had nothing to do with technical implementation or what consumers want? Take the following characteristics of Blu-ray:
The best part is that all of the above are design features, not bugs. But they do bug me. Admittedly, they look and sounds great, but after yet another weekend of anticipation to watching a rented (for real hard-earned cash) movie, which couldn’t be played, even after the latest firmware upgrade, I wonder why I even bother. I’m sure there must be alternatives.
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
Dear all,
It is with very great pleasure that I would like to tell you, that we (Denise, Luca and myself) are expecting our second child. Currently we are around 12 weeks and everything is looking good.
We saw the first images of our soon kid and new FreeBSD Hacker?
He or she is looking beautiful already. We do not yet know the gender, but we are expecting the kid around the beginning of September.
Luca is also happy with these developments, so everything is in the works (actively as we speak) to move his room to the upperfloor, and we are going to prepare his old room for the baby.
Leave a message so that I can read them later on (I would like that): do realise that it’s moderated and that it might take a little before I can acknowledge your message
During the last couple of days I am intensively using ezjail to administer several jails on my machines. They are currently IPv6 only (internet-facing) and are used to build pfsense images to test locally (still setting this up, need to cross compile to i386 from amd64), offer a testjail to a collegue to work together on a Opsview implementation on FreeBSD, whether or not we are going to succeed in that, and I just installed a test environment for my webservices. They are all contained in their own little box, having IPv4 connectivity outgoing through NAT, and native IPv6 connectivity from my “Vendor” on an extra subnet that I obtained.
I like this, so I am probably going to setup some more services here and there to perform some magic for me that might need external access. I will also tie them together with LDAP and the like so that it’s an uniform base. At the moment I do not have additional ideas about moving production services towards jails as well though.
Thanks to FreeBSD this all is damned easy. You should try it, or poke me in case you want to know more!
by Dag-Erling Smørgrav (noreply@blogger.com) at February 12, 2012 01:15 AM
Trac is a great little web tool for both developers and random collaborators - it's a wiki integrated with a Subversion browser with simple project management tools and extensible with a lot of different plugins. Unfortunately it's also slow. I've had to deal with an Internet-facing Trac wiki and came up with some solutions which lessen this problem.
In a few days, I’ll be heading to the FOSDEM conference in Brussels again this year. On Saturday, you’ll most likely find me around the FreeBSD booth representing the FreeBSD Foundation, so if you’re there drop by to say hi, discuss the Foundation’s work, pick up a Foundation flyer, check out the swag, or make a donation. On Sunday, I’ll be in the BSD DevRoom where there will be some interesting presentations and discussions. Remember, FOSDEM is free to attend. Hope to see you there!
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
by Dag-Erling Smørgrav (noreply@blogger.com) at January 19, 2012 07:38 AM
My work on FreeBSD AMIs for EC2 has made me even more sensitive to the irritation of host key checking, since building a set of AMIs for the 7 EC2 regions involves launching and SSHing into no less than 20 virtual machines. A couple of weeks ago I asked twitter for advice about this; ten people replied, and two people — Daniel Shahaf and Markus Friedl — made the critical observation that I wanted to use two tools: ssh-keyscan, to get a host key in a form suitable for the known_hosts file; and ssh-keygen -lf to take the host key from that form and convert it into a fingerprint I could compare against a known good value.
In addition to simple data caching, there are some interesting advanced features made possible by Bullet Cache's unique features. Data sharing between applications (or between application instances) is a very important one, especially for the PHP environment (and other CGI-like environments). This post is a part of series on Bullet Cache use cases.
Use cases for Bullet Cache are numerous - on the one hand it is a very convenient (and fast) memory cache server, but on the other it implements some advanced features which make it applicable in surprisingly many different scenarios. This post starts a series of descriptions on some of the real-world use cases which fit Bullet Cache suprisingly nicely, and some of these are already described in the Bullet Cache User's manual.
In the last six weeks, I found out Caramel has lymphoma, got unemployed, and had emergency surgery to remove my appendix on Christmas Day. The unemployment caused me to lose an in-progress mortgage refinance.
I'll pick up the mortgage thing once I remedy the employment problem, but I'm staying quite happily unemployed until after my kid is born - should be any day now!
Most of my career-growing moves were outside of work: at meetups, in open source efforts, or in networking with folks on IRC or twitter. Lots of awesome folks out there, so go introduce yourself. Don't be a dick. :)
I didn't write much on this site, but mainly, that was due to an increase in my activities on IRC and twitter. Most of what I published this year was code and was less writing about said code. I'd like to fix that, though.
This years successes were topped by two new major projects, fpm and logstash. I also released some major improvements to xdotool and other tools.
The current implementation of logstash isn't very old, but prototypes, hacks, and other incarnations of pretty much the same thing date back to at least 2005 and probably earlier. This project has been a long-time-coming, and Pete Fritchman and I have been talking about logstash for years, so it's nice to finally have some code shipped and a community building around it.
FPM had a crazy positive response. I wrote it as a hack, and it's used all over the place now. Bonus that people are contributing patches and other improvements as well.
Sysadvent was another excellent success, the end of which marked the 4th year and 100th article posted to the project. It is awesome seeing such community involvement from so many different authors.
This year also cemented my move to git from svn. Why? Github, mostly, and not really the features of git itself. Sharing code and patches is so much easier on github than it is with other services.
I went to CarolinaCon and OSCON to talk about logstash. I also went to DevOps Days Mountain View and gave a lightning talk on logstash.
My OSCON talk was overflowing with people standing at the back of the room, etc; it went awesomely. I've also been able to do lunchtime logstash presentations at places like Square and others. I also gave talks at BayLISA meetings. It was a good year for getting out of the house and talking about code.
I tried to get a count of how much code I'd written this year, but I had lots of web-based projects that included third-party stuff like jquery, and I'm too lazy to pick through the results and trim that stuff out. I'm up to about 70 different projects on github now, some useful; some not; all fun!
Looking forward to 2012 :)
![]()
I'm preparing for the 1.0 release of Bullet Cache and have squashed the last (known) bug which plagued it, so I'm cautiously optimistic that it deserves the "1.0" label. It's been very fun working on it and though none of this is terribly exiting news, I'd like to share a few things I've encountered while making it...
Around one in the morning...
“This is blasphemous, you know.”
“Hmmm?”
“This colostrum-based cold medicine. They have a promotion where they've hidden images of golden calves in some of the boxes and you win a prize if you find one. It's blasphemous.”
“Huh?”
“You know, the golden calf? The false idol the Israelites made while Moses was up on the mountain and he got so angry when he saw it that he broke the tablets and had to go back and get a new set?”
“Yes?”
“It's blasphemous. I'm probably going to hell for taking this cold medicine.”
“...”
“If I go to hell, will you come with me?”
“Yes. Now go to sleep.”
by Dag-Erling Smørgrav (noreply@blogger.com) at December 23, 2011 09:23 PM
This also applies to companies. I run an online backup service, and for the past two years I've donated all of the profits made during the month of December to the FreeBSD Foundation; I'm going to be doing the same thing this year too. I'm not doing this just because I'm a FreeBSD developer, because I use FreeBSD personally, or because I would never have launched Tarsnap if I hadn't been able to build on the open source code in FreeBSD: I'm doing it because I think supporting FreeBSD development will make the world a better place for both Tarsnap and many other startup companies.
I'm tired of Windows 7 complaining every time it boots about the Bluetooth radio not being discoverable. The irony is that the Bluetooth settings dialog recommends that you turn off discoverability (for obvious reasons), but when you do, it will bitch and bitch and bitch until you turn it back on. Does anyone know how to make it stop nagging?
by Dag-Erling Smørgrav (noreply@blogger.com) at December 18, 2011 04:49 AM
There's just one small problem: The way POSIX has defined close(2) is completely and utterly broken.
ZFS has a couple of very useful functions, zfs send and zfs receive, which allow you to serialize a complete ZFS dataset and recreate it in a different location. They can also be used to serialize a delta between two snapshots and apply that delta to a previously created copy of the dataset. You see where I'm going with this... That's right, incremental backups of a ZFS dataset or even an entire pool to a different ZFS dataset or pool.
Why would you want to perform incremental ZFS-to-ZFS backups instead of just adding redundancy to the pool, or cloning a snapshot? Because—provided the ZFS pool and filesystem versions match—it allows you to duplicate your dataset or pool on removable media (which you can store off-site), or even on a different machine across the network. This technique is far more efficient than rsync, because there is no need to compare the source and destination: ZFS already knows exactly what has changed. It also preserves the filesystem hierarchy and dataset properties.
In my case, I need to duplicate a pool onto removable media because I am replacing a server that only takes PATA disks with another that only takes SATA disks, which precludes just moving the disks over and progressively replacing them with new ones. Using this technique, when the time comes, I can slide the new server into the rack, hook up the backup disk, and restore just the parts I want to keep.
Of course, like a good little hacker, I wrote a script, which you can find here, to automate this.
The script takes two arguments: the source dataset and the destination dataset. Either of these can be the root of a ZFS pool or a dataset within a pool; they can even be datasets within the same pool, provided they do not overlap. The script selects the latest snapshot of the destination dataset (it uses a naming scheme which ensures that lexical order corresponds to chronological order), verifies that the source dataset has a snapshot with the same name, takes a new snapshot of the source dataset, and streams the difference between the old and new snapshots from the source dataset to the destination dataset. Finally, it deletes the old snapshot to allow ZFS to reclaim the space occupied by old data.
You can use this script with multiple backup disks, since it will only delete the snapshot that was actually used for the current disk. If you have one disk for each day of the week, for instance, it will delete last Monday's snapshot once it has completed this Monday's backup, but leave the other six in place. Likewise, if you decide to keep Sunday's disk for a month instead of reusing it next Sunday, the script will leave the snapshot in place until you run it again with the same disk.
The script does not currently support over-the-network backups, but it should be fairly easy to implement.
by Dag-Erling Smørgrav (noreply@blogger.com) at November 22, 2011 02:31 PM
The last few nights I have been working on something very interesting: encrypted data types for PostgreSQL. The goal here is to introduce transparent data encryption for applications which need to protect "data at rest", i.e. while the data is stored in the database within the file system. I didn't find any such mechanism (pgcrypto only offers security primitives which can be used on the application side) so I wrote pgenctypes.
Those who were present at BSDCan 2011 have probably seen or heard about the unveling of BHyVe, the native hypervisor for FreeBSD. FreeBSD is very much lacking virtualization features (not counting jails) and this is in any case excellent news for the project! Interested users are now invited to test it!
I've just uploaded the first Release Candidate version of Bullet Cache! It is basically feature complete and done, and I'm happy to say that it looks like I have a small number of users and also some feedback on the project - so keep it up :) At this point I'd like to shortly talk about what made me write Bullet Cache - which also leads to why it was done the way it is and what are its main strengths. For the impatient, these are flexibility in cached data expiry and performance, but read on for the details...
I went to the customs office to pick up a package, ended up told that I need more papers than I brought in, specifically a PayPal bill printout which I didn't have. Now, the usual thing to do would be to return home, print it out, then return back to the customs office and proceed from there. What I did instead was amazing - from the point of view that almost nothing of that infrastructure even existed a few years (and especially decades) ago.
One of the many neat things FreeBSD uses to increase the operating system's resilience is the /rescue directory which is present by default and contains basic utilities and commands which can be used in case one or more of the default binaries gets corrupted or lost. In fact, binaries in /rescue are also statically linked so they can be used even if a large part of the system fails. Basically, if you have init and a shell (any shell, possibly the one from /rescue; even init is in /rescue) working, you can get your system back.
One of the many neat things FreeBSD uses to increase the operating system's resilience is the /rescue directory which is present by default and contains basic utilities and commands which can be used in case one or more of the default binaries gets corrupted or lost. In fact, binaries in /rescue are also statically linked so they can be used even if a large part of the system fails. Basically, if you have init and a shell (any shell, possibly the one from /rescue; even init is in /rescue) working, you can get your system back.
This post in the Bullet Cache series introduces the primary API to the cache server, implemented in the for of a C library. This is important as it allows for maximum portability across different application environments while at the same time retaining maximum efficiency and performance. All of the standard benchmarks were done using this exact library without any tweaking. The standard distribution also includes the PHP API which is an almost exact wrapper around the C API. Both of these APIs are also comprehensively described in the Bullet Cache User Guide.
This post in the Bullet Cache series introduces the primary API to the cache server, implemented in the for of a C library. This is important as it allows for maximum portability across different application environments while at the same time retaining maximum efficiency and performance. All of the standard benchmarks were done using this exact library without any tweaking. The standard distribution also includes the PHP API which is an almost exact wrapper around the C API. Both of these APIs are also comprehensively described in the Bullet Cache User Guide.
I've written previously about my Bullet Cache server, and now I'd like to continue with basic installation, startup and testing. The server itself is written in a combination of C and C++, and should be usable on most modern Unix-like systems.
example: example.c:9: main: Assertion `i == 3' failed.This is better:
Assertion failed insist.c:7 in main(), insist(i == 3): Something went wrong, wanted i == 3, got 4The main difference here is that there's context about what failed. A message for humans looking to debug this. This is especially important on Linux these days because every distro I've used recently hates sysadmins and hates debugging - all libraries are stripped of debug symbols and coredumps are disabled by default.
What's the usage look like?
#include
#include
#include "insist.h"
int main() {
int i = 4;
//assert(i == 3);
insist(i == 3, "Something went wrong, wanted i == 3, got %d", i);
return 0;
}
I also added a special 'return' version of this, 'insist_return' that lets you
do error checking and early aborting like this:
insist_return(fd >= 0, START_FAILURE,
"socket() returned %d, an error: %s", fd, strerror(errno));
Works just like insist() except returns START_FAILURE if 'fd > 0' is false and
additionally logs the error formatted above.
Code here: insist.h
![]()
I've acquired a couple more 2 TB Advanced Format drives: a Seagate Barracuda Green (ST2000DL003) and a Samsung SpinPoint F4 EcoGreen (HD204UI, no data sheet available online).
I was extremely impressed with the Samsung HD204UI. It's the first AF drive I've seen with decent performance. In fact, it's the fastest disk I've tested so far—its unaligned writes are faster than the non-AF Hitachi I used as a reference last time, and its aligned writes are twice as fast.
count size offset step msec tps kBps 131072 1024 0 4096 43984 2979 2979 131072 1024 512 4096 127047 1031 1031 65536 2048 0 8192 14764 4438 8877 65536 2048 512 8192 12453 5262 10524 65536 2048 1024 8192 12460 5259 10518 32768 4096 0 16384 4609 7109 28436 32768 4096 512 16384 7829 4185 16740 32768 4096 1024 16384 8413 3894 15579 32768 4096 2048 16384 8211 3990 15961 16384 8192 0 32768 3952 4145 33165 16384 8192 512 32768 9050 1810 14481 16384 8192 1024 32768 9317 1758 14067 16384 8192 2048 32768 9315 1758 14069 16384 8192 4096 32768 3996 4099 32793
The Seagate ST2000DL003, on the other hand, is so slow it's not even funny. It's actually the slowest of all the drives I've tested: its performance on aligned random writes is half that of the Western Digital WD20EARS. It's three times as fast on unaligned writes, but three times nothing (100 kBps) is still nothing (300 kBps) compared to the Samsung HD204UI (15 MBps). Here are the numbers:
count size offset step msec tps kBps 131072 1024 0 4096 2419280 54 54 131072 1024 512 4096 2199286 59 59 65536 2048 0 8192 1283667 51 102 65536 2048 512 8192 985184 66 133 65536 2048 1024 8192 995423 65 131 32768 4096 0 16384 45980 712 2850 32768 4096 512 16384 345291 94 379 32768 4096 1024 16384 432533 75 303 32768 4096 2048 16384 429781 76 304 16384 8192 0 32768 34192 479 3833 16384 8192 512 32768 166440 98 787 16384 8192 1024 32768 210147 77 623 16384 8192 2048 32768 207356 79 632 16384 8192 4096 32768 34221 478 3830
This time, I also ran sequential write tests—basically, dding eight gigabytes' worth of zeroes to the disk in 128 kB blocks, which is the optimal I/O size for FreeBSD. This time, the results are pretty close: the Samsung HD204UI gets slightly less than 90 MBps, and the Seagate ST2000DL003 gets slightly less than 80 MBps.
by Dag-Erling Smørgrav (noreply@blogger.com) at September 30, 2011 02:27 AM
I am going to take the opportunity in the next few blog post to advertise my open source project - Bullet Cache, which is almost finished. Think of Bullet Cache as [*] "Memcached, only better" :) It offers some unique data query options and is blazingly fast! And good news for everyone who's asked: as promised, I've licensed it under the BSD license!
I'm announcing the availability of the 1.0-beta1 version of Bullet Cache! It is feature-complete and I consider it ready for public testing. C and PHP client libraries / "drivers" are currently implemented.
[*] for some values of "better"

Lukas sleeping snugly with his personal pacifier.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
I've just been to a meeting about what should have been a mundane and easy topic: there is a need for the update of a technologically old Web system of a government agency and through word-of-mouth it was thought that my group could do something good about it. As it turned out, that's a minor part of the work (and doable) and the rest of the talk was eerily Dilbert-like, practically the real-world incarnation of the phrase "Nobody got fired for choosing IBM."
On August 23, 2011, we welcomed into this world Lukas Lansing, 53cm (21″), 3,4kg (7.5 lbs).
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
The iPhone home screen has four icons stuck at the bottom of the screen. For the last year I had there Safari, Mail, Phone and Facebook.
Yesterday I swapped Facebook for Podcaster, which I have been using more frequent in the last four months.
Facebook is for people who are bored.
Podcasts are for people who want to learn!
If you want to be really evil, however, *.google.com is the wrong SSL certificate to forge. The right one? ssl.google-analytics.com.
Watching the video this morning I was struck by the number of times that I forgot to mention something or started to say something but lost track of where the sentence was headed before I got to the end of it; so without further ado, here's the "what I meant to say" errata for the interview:
ik wou dat ik een oma had die ik soms zomaar op mocht bellen en die 's avonds bij m'n bedje zat om mij een sprookje te vertellen maar oma's hebben allemaal al iemand voor wie ze oma kunnen zijn ze zitten dan wel in tehuizen maar elke zondag is het kamertje te klein dan komen ze allemaal op visite en vragen of ze een zwaantje vouwt en als ze 's avonds moe gaat slapen weet ze dat er iemand is die van haar houdt hoe zou het dan toch komen dat heel veel oma's eenzaam zijn en van hun kleine kinderen dromen die nu veel groter en verhinderd zijn al die oma's die truien breien waarvan niemand zegt: wat fijn! die hoeven me dat truitje niet te geven maar willen ze alstjeblieft mijn oma zijn? Morrison
het is al bijna avond wat gaat zo'n dag toch gauw ik klim zo in mijn bedje en dan denk ik weer aan jou dan lig ik stil te luisteren naar de geluiden om me heen dan hoor ik zoemen tikken fluisteren want ik lig hier niet alleen! soms vertel ik mijn avonturen aan Tiberius da's een bromvlieg en die woont op het kozijn dan snort ie heel tevreden want als er iemand tegen 'm praat dat vindt Tiberius hartstikke fijn vandaag ook Ricky nog gesproken die woont bij de kersenboom het is een soort van rups maar hij wil later vlinder worden net als zijn vader en moeder en zijn tante en z'n oom zelf wil ik als ik later groot word proberen klein te blijven omdat Tiberius en Ricky anders bang voor me zijn dan blijf ik ook dichter bij de bloemen en zal ik altijd gelukkig zijn Morrison
As one of the commenters figured out, I made a fatal error in not running Tomcat as a standalone server, which had a horrible impact on its performance. After fixing this, the performance results between PHP and JSP are very near, probably within measurement error.
So I'm retracting this too-hastily written post and apologize for any grief it caused to Java users :)
It is the year 2011 and the "Java is slow" notoriety still isn't dead - and rightly so. In choosing the technology for a project I vas sort of leaning to use Java (or is everything which combines Java and the Web automatically called J2EE?) instead of PHP which I normally do (for better or for worse). I was expecting Java to be more optimized and together with JIT compiling, the faster solution. But I tend not to assume and do my own benchmarks so imagine my surprise...
When it doesn't do IO, virtualization is pretty good performance-wise these days. Here's how Xen looks like on FreeBSD 8-stable AMD64 XENHVM (pysysbench).
by Dag-Erling Smørgrav (noreply@blogger.com) at July 27, 2011 03:13 AM
It looks like I'm releasing a new version of my ultra-compact single-file image gallery approximately once every two years :) It is extremely simple and definitely lacks modern features (it was started in 2005!) but if you use it, enjoy!
Today, there are no roses to be had in Oslo, for love nor money.by Dag-Erling Smørgrav (noreply@blogger.com) at July 26, 2011 07:40 PM
As some of the regular readers may know, I'm mentoring a Google Summer of Code project that aims to bring an experimental light-weight scheduler to FreeBSD. There are many differences between Linux which is the original scheduler's initial implementation platform and FreeBSD, so the port will be more like an reimplementation of some of the ideas. This project is important because it brings a new perspective to the problem of scheduling - namely how would a simplified scheduler (without expensive tracking of process performance) behave for modern workloads.
As some of the regular readers may know, I'm mentoring a Google Summer of Code project that aims to bring an experimental light-weight scheduler to FreeBSD. There are many differences between Linux which is the original scheduler's initial implementation platform and FreeBSD, so the port will be more like an reimplementation of some of the ideas. This project is important because it brings a new perspective to the problem of scheduling - namely how would a simplified scheduler (without expensive tracking of process performance) behave for modern workloads.
In a curious twist, it looks like Google+ currently causes more trouble for Google than it's worth it. I have no doubt that the problems will be solved soon (if nothing else succeeds - they can always throw more hardware at it), but it's instructive to say the least.
As I'm writing this, all non-search services of Google I've tried are broken to a greater or smaller extent, starting with Google+.
In a curious twist, it looks like Google+ currently causes more trouble for Google than it's worth it. I have no doubt that the problems will be solved soon (if nothing else succeeds - they can always throw more hardware at it), but it's instructive to say the least.
As I'm writing this, all non-search services of Google I've tried are broken to a greater or smaller extent, starting with Google+.
A few days later, I was looking at my kivaloo data store, contemplating the next steps I should take with it, and I decided that it was time to add some security. After all, I'm going to be using it in Tarsnap, and the Tarsnap web server needs to access user accounting data so that people can see their account balances and recent usage; since I want the web front-end kept separate from the core backup service code, traffic will need to go over the internet. My first thought was to build this into kivaloo directly, but then I reconsidered: Why not write a generic secure pipe daemon, and keep the complexity of encryption and authentication out of kivaloo? Enter spiped.
Last month, we went on a delayed honeymoon to Crete, Greece. The weather couldn’t have been better and neither could the food, so needless to say, we spent quite a lot of time near the pool. We did manage to save some time for a road trip or two and I just don’t want to keep the pictures from you.
More pictures here.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Hetzner has a strange IPv6 routing setup where the default gateway is not in the same subnet as the host. Clearly, this is to avoid wasting space on glue nets, but it does not look pretty. Obviously, Linux just ignores the fact that it isn’t supposed to work, but no such luck on FreeBSD. As Bitmand wrote some time ago, it can be solved with a static route but the FreeBSD startup scripts, apply the default route before static routes, leaving it no other option but to reject the default route as it has no route to its target at that time. Instead of hacking the startup scripts, this can be solved easier with two static routes with fixed order, which is still a hack though not as ugly and less error prone when upgrading the system files. For reference, especially for myself next time I need this, here’s my full ipv6 startup configuration:
rc.conf:
ipv6_enable="YES"
ipv6_default_interface="re0"
ipv6_static_routes="defgw def"
ipv6_route_defgw="2a01:XXXX:XXXX:XXXX::1/59 -prefixlen 59 -iface re0"
ipv6_route_def="default 2a01:XXXX:XXXX:XXXX::1"
ipv6_ifconfig_re0="2a01:YYYY:YYYY:YYYY::2/64"
systctl.conf
net.inet6.ip6.accept_rtadv=1
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
My "little" sister gave birth to a brand new baby girl about two months ago, and is now going through the usual baby-raising process, invloving a lot of crying, pooping and not much sleeping. The baby is - cute, like all babies are. I'm much drawn to the process where the baby is "Awakening" (yes, with a capital A - you know what I mean). It still has a long way to go but it already seems to have some pattern matching going on, though usually nonsensical. I usually visit my sister every few months so I am very interested in observing this process.
Last week it was decided that my mallet finger had to be fixed via an operation during which I will get two wires inserted into my finger: One below the nail which will push the broken piece back against the bone, and one through the upper bone and halfway the middle bone so that bone doesn't move anymore.
Today was the big day. We had to be in the hospital at 08:30, which is about half an hour before Dirkie and Hanorah go to school. So they slept with their grandparents and were very excited about the fact that they got breakfast in a plastic bag in the car on their way to school. I wish everybody was so easily pleaseble.
When you made the appointment to go to the hospital, you know from the moment you enter the building you have lost control over your life until you leave you are on somebody elses schedule. It will involve a lot of waiting, and there is nothing you can do about it:
Before you get in the operation room, you will be asked the same question every time: Your name, date of birth, name of the doctor and what they are going to do on you. Just to make sure they have the right guy in front of them.
The procedure done on my finger was over in 20 minutes. The anastetic I got was a finger block, two needles in my hand which neutralized all feeling in the ring finger, and some drowsiness stuff which I think didn't really work at all. During the operation I could hear the drill, but not feel the things they did on my finger.
At 14:30, after the operation and when I was out of bed, I was given an arm sling to keep my hand up and a prescription for painkillers. Six hours real-time for a 20 minutes procedure, it's very low duty-cycle.
The finger itself now has a splint at the top of the hand and a lot
of bandage around it. You can see the wire sticking out at the top
of my finger, which is right now not scary yet...
I was told to take the painkillers when my fingers started to tingle, which was a couple of hours later. Since the pain didn't come back after that, I didn't take anymore and slept through the night.
So is there pain? Yes and no.
There is irritated skin (for lack of a better description) around where the wires are sticking out. But there is no pain because of the drilling, which can be either because there is no pain or because the nerves in my pink and ring finger there are numb: For the last four years I haven't had any feeling in them. I have seen a specialist for it who has done the famous frog tests which will pull your muscles when an electrical current is going through them and they didn't find anything wrong with the nerves there.
Maybe that has gotten me through the night without painkillers, maybe there was no pain to start with...
On Monday I have my first physiotherapy at 08:00.
Finally, last weekend the rest of my things were moved from Tranbjerg near Aarhus to Amager near Copenhagen. The trip (pictures) included both a quiet and beautiful ferry trip across the Kattegat and a trip to western Jutland with the tiny truck. The end result: A lot of boxes everywhere…
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
We went to see the hand-surgeon. From the X-rays he saw that the break was at a nasty location: it was broken of in the contact area of the joint. First a proper cast instead if the splint and then new X-rays in case the piece of bone was put back in place by the cast: it wasn't.
So the next options were: leave it like this and it will be half-fine or have an operation and it will be fully fine. There are two kind of operations which he could do: a screw with which the broken piece get puts back or a two-wire approach with which the broken piece gets pushed against the bone and regrows that way. Because of the size of the piece broken off we chose for the wire.
After the operation I will have two pieces of wire sticking out of
my finger for four to six weeks, but they are luckily under a
dressing, and have a cask for that period too. After that everything
should be back in volleyball-playing-condition again!
Next update: Coming Wednesday most likely.
I have written before on mdcached - my not-so-pet-project I've been doing over the years, which I'm looking forward to bring to a really usable state in the coming weeks. I've cleaned it up a lot since March and I'm confident that the core of the system is correct and reasonably bug-free. I am renaming it to "Bullet Cache" but I am still calling it "alpha" code as it is not yet feature complete (some commands are missing). Anyway, it looks like due to the combination of advances in hardware and some algorithmical improvements, its performance is really good, close to 1M transactions per second.
As you know, or should know, today (or tomorrow, depending on the timezone) is World IPv6 Day. What this means in practice is that a lot of top-tier IT businesses will make their web sites available via both IPv6 and IPv4, at least for a day. If you want to join in testing but your ISP is following the rest of the herd and isn't doing anything with IPv6, you can use various methods to tunnel your IPv6 traffic over IPv4.
A normal Tuesday: wake up, train, work, train, dinner, volleyball. And then hospital, because my lefthand ring-finger got damaged during the volleyball game.
When the finger got hurt, by a bad catch of a badly placed ball, it didn't seem too much. About half a second later I was off the field to get some ice for it.
The people from the Menai Sports center have a great solution for providing cold-packs for injured players: just use a bag of frozen peas, it works as well and doesn't cost as much if never returned.
The "let's see if thus goes over after ten minutes" approach worked fine, except that I nearly fainted after the ten minutes when the finger got bent. And there was a strange hump of the top of the finger. Plan B, off to the hospital.
In the Emergency department you get first investigated by the triage nurse (m/f for the Dutch people who have a different word for a male and female nurse). He wisely didn't touch the finger and wanted to make an X-ray first.
The X-ray departments these days don't work with photographic paper anymore, the receiver is a kind of flat CCD device which wirelessly transfers the image to the image library. Welcome to the 21st century, goodbye to the "clunk clunk" sound of the replacement of the film.
It showed that there was a little triangle of bone broken off and I was told that the tendon was still attached to the piece. That was the reason why it a. Hurt so much when trying to stretch the finger and b. The finger didn't get stretched anymore.
The doctor could tell me that it was called a "mallet finger" and that it was a common breakage for people who play ball-sports. I still don't know the dutch name for it, maybe it only happens in English speaking countries! The doctor knew so much about it because he had had the same injury from the same sport earlier in life. And his finger was close to back to normal and he played volleyball again, so there is hope!
Later in the hospital I got a mega-splint on it and the message for
the hand-surgeon, including a CDROM with a JPEG and an uncompressed
picture of the X-ray, and I went home.
I got access to a Sandy Bridge Xeon server for a while and decided to try some benchmarks on it. Usually, I'd run unixbench from the FreeBSD ports, but - it's old. It is very, very old and the only reason I still use it is convenience. So, having some time, I started my own multiprocessor-friendly benchmark suite. Yup, I could have named it "yet another benchmark suite" since there are so many of them. Unfortunately, most are bad, and I will try to make this one into something reasonable. Anyway, I did a comparison between some Xeons I have access to, out of curiosity.
Download: xdotool-2.20110530.1.tar.gz
As usual, if you find problems or have feature requests, please file bugs or send an email to the list.
Changelist since previous announcement:
2.20110530.*
- New set_window feature: --urgency. This lets you set the urgency flag on a
window Window managers will interpret this as something about your window
needing attention. It might flash in the taskbar, pop up, or other.
Original patch and suggestion by ervandew.
Issue: http://code.google.com/p/semicomplete/issues/detail?id=39
- New function: xdo_window_seturgency (see above)
- Hack in OS X support as it is missing a proper clock_gettime.
Should fix http://code.google.com/p/semicomplete/issues/detail?id=37
Reported by ryandesign.com
- Add support for typing UTF-8 characters. Patch from Joseph Krahn.
- Make all output call fflush to send data immediately (for pipes). Reported
by Andreas Wagner on the mailing list.
- Make 'get_desktop_viewport' output usable with 'set_desktop_viewport'
http://code.google.com/p/semicomplete/issues/detail?id=47
- You can now make 'libxdo.a' for embedding libxdo into your binary
(Requested by psc on the mailing list).
- Fixed a typing bug where the keymap changes unnecessarily
- Should now build cleanly in C++ environments (Reported by psc on the
mailing list)
- bugfix: xdotool should use command names first before trying file scripts.
See https://bbs.archlinux.org/viewtopic.php?pid=938309 for original report.
- Add a 'sleep' command. (Requested by Joseph Krahn via mailing list)
- Add --relative flag to windowmve. (Requested by Anthony Thyssen via mailing
- Add --desktop flag to the search command. This lets you search for windows
on specific desktop. Requires a window manager that supports multiple
desktops in a way that EWMH supports.
Fixes http://code.google.com/p/semicomplete/issues/detail?id=38
- Add --limit flag to search. This allows you to break the search early after
a certain number of matches. (Requested by Anthony Thyssen)
- New command 'getwindowgeometry' for fetching window position and size
(Requested by Anthony Thyssen via mailing list)
- Add --sync flag to search command; blocks until results are found.
xdotool will search every 0.5 seconds for results.
http://code.google.com/p/semicomplete/issues/detail?id=42
- windowmove can now move windows along an axis. Give literal 'x' or 'y'
instead of a coordinate and it uses the current position. (Requested by
etnlIcarus via mailing list)
- Add '--args N' and '--terminator TERMINATOR' to the 'exec' command.
Default terminator unless specified (or --args is) is ':' (Requested by
Joseph Krahn and Henning Bekel via mailing list)
- set_desktop now supports --relative flag (+N or -N to move relative)
(Requested by Anthony Thyssen)
- The mouse cursor now changes during 'xdotool selectwindow' (Requested by
Anthony Thyssen via mailing list)
- Added '--args N' and '--terminator TERMINATOR' to the 'type' command.
- Add 'getdisplaygeometry' command for querying the size of your screen.
(Requested by @rrwo via twitter)
- Add xdo_get_viewport_dimensions function.
by Warner Losh (noreply@blogger.com) at May 29, 2011 11:58 PM
FreeBSD has 17 Google Summer of Code projects this year and most of them look very interesting! As I am involved as a mentor or co-mentor in two of those, I would like to describe them here.
Once a year, Ottawa (in Canada) becomes the hottest place to be to discuss BSD-related projects, at the event unsurprisingly named BSDCan. Hidden within this event is the FreeBSD developers' summit which is itself a great opportunity to see all the people I've communicated over the year, talk and exchange ideas. This year's BSDCan / DevSummit was one of the better ones, measured in terms of ideas and projects presented; it was definitely a success!
Just a quick note to point to my slides that summarize the Ports and Packages Summit at the FreeBSD DevSummit during BSDCan 2011, which can be found here. Also, we looking forward to feedback on the PKGNG project that was announced earlier and will replace the current pkg_* tools to handle ports installation and package handling and which will be a focus for portmgr over the next few months.
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Apparently, the LibreOffice is project set to release version 3.4 ... with these changes. I find it very WTF-worthy that these changes should amount to a whole new version instead of just 3.3.3 or something like that.
As the Leave Comment feature on Skype’s blog does not seem to work in Safari, I’ll leave some comment on this here. Some serious information is incorrect or missing from Skype’s security information for Skype for Mac 5.x (2.x is rerported not to be affected).
Skype released a very terse message on their Mac blot today pointing to an earlier post on their Security blog. The information in this post is either incorrect or there is another vulnerability out there that they haven’t informed their users about yet. The vulnerability described in the blog post explicitly states that a specially crafted, malicious “message would have to come from someone already in your Skype Contact List”. I have seen many crashes over the last few weeks with version 5.1.0.914, none of which caused by messages from people in my contact list, but by contact requests, which for obvious reasons can be sent by people not (yet) in ones contact list.
If these crashes are related to this vulnerability, it would contradict both Skype’s statement that the malicious message has to be sent by an approved contact and that the vulnerability is not exploited in the wild. Hopefully, Skype is right and they are unrelated, but to err on the safe side I would recommend anyone using Skype for Mac 5.x to not wait for Skype to release a new version next week, but to upgrade immediately to the latest release 5.1.0.922 here. This version will not show up via the Check for Updates menu as Skype deemed the hotfix non-critical, exactly because it can only be caused by approved contacts and is not seen in the wild, so their users have to find out and install the update manually themselves.
Update: Skype has released Skype for Mac 5.1.0.935 which includes unspecified Security updates with no further explanation as they wish to “wait for the majority of our users to update before detailing / discussing any of the specific issues that have been fixed”.
Update 2:Legitimate contact requests also crash Skype.
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
After lots of refactoring and improvements to logstash since the first minor release last November, logstash is ready for wider usage now.
Read my announcement here.
The logstash site is also online and has docs, intros, slides, and videos.
Happy logstashing!
![]()
The 41st Canadian general election is taking place today — polls are already open in Atlantic Canada — and like most Canadians, I hope to know the outcome as quickly as possible. Being a resident of British Columbia, I am disadvantaged in this respect by section 329 of the Canada Elections Act, which forbids the transmission of results from East to West until polls have closed here; so given my penchant for statistics, I decided to put together an "election results watching guide" to assist me in analyzing the results when they finally arrive.
For the benefit of other eager Canadian election-watchers, here it is:
With the lovely spring weather during the Easter holidays, it was time to explore so-called nature around a major city like Copenhagen. Not far from here is the nature reserver Vestamager. As can be seen in the picture below, it’s actually so close to downtown Copenhagen that you can see Town Hall. Even with Copenhagen Airport, modern highrises, and a highway constantly visible and audible in the background, it’s quite a nice outting, especially only with a 15 minute metro ride.
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
I’ve managed to get FreeBSD booting to Single user mode on a BeagleBoard. I’ve uploaded a patch of the changes made. My plan now is to clean it up and submit it in stages for feedback.
This wouldn’t have been possible without the work of Ben Gray for the initial work, Mark Murry for porting the patch to a newer version of FreeBSD and Andrew Thompson for donating the hardware.
This is the first time I've tried to used an SSD on a desktop, and it was primarily to try to speed up Windows booting and application start. In short: don't bother.
This is the first time I've tried to used an SSD on a desktop, and it was primarily to try to speed up Windows booting and application start. In short: don't bother.
I was helping with some capacity planning and run-rate math today at work and found that ec2 reserved instances are much cheaper compared to on-demand - If this is obvious to you, chill out, I have historically never really used EC2 nor have I ever been close to budgeting. ;)
I proved this conclusion with some math, but frankly I like visualizations better, so I decided to learn R. I wrote an R script that will graph an on-demand vs reserved pricing for one m1.large instance (code at end of the post).
The result is this graph:
The graph says it all, and definitely tells me that we need to be reserving all of our instances at Loggly - and it gives me a rule-of-thumb:
This also means that our random "debug something" deployments that are shutdown much of the time are probably best off being reserved instances as well- at least for a 1-year thing - since we are likely to use those deployments for more than half of a year.
A 3-year on-demand price for m1.large is just shy of $9000, which is twice as expensive as the 3-year reserve. Capaticy plan and maybe start buying reserved instances. Make your CFO happy.
And in case you were going to ask, I ran the same plot with data from EC2 "quaduple extra large" instances and the savings and break-even points were the same. I bet the rest of the prices flow similarly.
The R script is follows, run it with 'R --save yourscript.r':
# Values taken from http://aws.amazon.com/ec2/pricing/
# for an m1.large ("Large") instance
on_demand_hourly = 0.34
reserve_hourly = 0.12
reserve_1year = 910
reserve_3year = 1400
# quadruple extra large instances
#on_demand_hourly = 1.60
#reserve_hourly = 0.56
#reserve_1year = 4290
#reserve_3year = 6590
on_demand_daily = on_demand_hourly * 24
reserve_daily = reserve_hourly * 24
x - c(0, 365)
y - on_demand_daily * x
# Calculate day of break-even point reserve vs on-demand rates
break_1year_x = reserve_1year / (on_demand_daily - reserve_daily)
break_3year_x = reserve_3year / (on_demand_daily - reserve_daily)
png(filename = "ec2_m1large_cost.png", width = 500, height=375)
plot(x,y, type="l", col='red', xlab="", ylab="cost ($USD)")
title("EC2 cost analysis for m1.large",
sprintf("(days)\n1-year is cheaper than on-demand after %.0f days of usage,\n 3-year is cheaper after %.0f days", break_1year_x, break_3year_x))
text(60, 0, sprintf("on-demand=$%.2f/hour", on_demand_hourly), pos=3)
abline(reserve_1year, reserve_daily, col='green')
text(60, reserve_1year, sprintf("1-year=$%.0f+$%.2f/hour", reserve_1year, reserve_hourly), pos=3)
abline(reserve_3year, reserve_daily, col='blue')
text(60, reserve_3year, sprintf("3-year=$%.0f+$%.2f/hour", reserve_3year, reserve_hourly), pos=3)
point_y = reserve_1year + reserve_daily * break_1year_x
points(break_1year_x, point_y)
text(break_1year_x, point_y, labels = sprintf("%.0f days", break_1year_x), pos=1)
point_y = reserve_3year + reserve_daily * break_3year_x
points(break_3year_x, point_y)
text(break_3year_x, point_y, labels = sprintf("%.0f days", break_3year_x), pos=1)
dev.off()
quit()


Even after working for Riverbed Technology for two and a half years now, I still have to come up with a bullet-proof analogy of how WAN optimization works.
Consider a network from one side of this planet to the other side: A round trip time of 300 milliseconds for 20 million meters. Of that 300 milliseconds, you have two factors: The speed of light to get from A to B, and serialization delay which happens at every hop and is related to the bandwidth and the size of the packet. One is constant, the other one is variable.
To move data in a stream, the throughput is limited to the smallest bandwidth in the path. Although you can move data faster on other parts, it all has to go through this one.
As an example, say you have a stream of 300 Mb and a smallest bandwidth of 2 Mbps. Without any protocol overhead, this will take 20 minutes to go through there. With a fourty byte protocol overhead and an MTU size of 1500, this will take 20 minutes and 33 seconds.
Now with WAN optimization. It consists of three parts: Optimization on TCP level, which has been ignored for now. Latency optimization on application specific protocols, which has been ignored for now. And data optimization, where the data is either compressed or only referred to. If the same or similar data gets transfered via two WAN optimizers twice, the first time you would get a relative small reduction factor, depending on the compressability of the data, while the second time you would get a large reduction factor because the data patterns is alreayd known on both devices.
If that 300 Mb is split into segments of 1024 bytes, making it 300 000 segments, and each segment has a 64 byte label, you end up with only about 19 Mb worth of labels.
Transfering that 19 Mb through a 2 Mbps link will be take 80 seconds, about 15 times faster.
Now back to the topic: A good analogy for WAN optimization.
Is it faster than light? It feels like it, but the Round Trip Time of the WAN is still the same. And the speed the packets go via is still the same.
Is it a "wormhole"? Wormhole-based paths which are shorter than a non-wormhole-based paths. The path travelled travelled for optimized traffic still has the same distance.
Is it comparable with ships, where goods are stored in containers (labels) and then transported in large bulk carriers? If the speed limit of other ships was limited to the speed limit of the bulk carriers, then it would be a good start.
Is it a train analogy, where passengers are cramped into carriages and efficiently transported across the rail network? It could be, except that on the railroad network everything is put into train carriages and transported efficiently on it. Comparing it with the French TGV and the Japanese bullet trains does not work neither, because the speed of the packets is still the same while these trains are way fast.
So, the analogy needs to use the same speed limits on the transport mechanism, and needs to give the impression that the delivery gets faster without changing the distance.
The best thing I come up with is transport of goods via large trucks instead of via small delivery vans: Goods are shipped via small delivery vans to a distribution point, stored into a single large truck which then uses the same transport infrastructure as small fast vans would have used if they would have transported their payload. Instead of a long convoy of small vans, you get one truck towards the distribution point which there gets reloaded into numerous small vans. The only thing which does not make sense yet is that small delivery vans are often 2 x 4 x 1.5 meters and big trucks are 3 x rather long x rather high, which gives the impression that size still matters while this isn't the case on WAN optimized traffic...
That is the problem if you work with magic :-)
More-or-less aimless surfing brought me to an old blog post about math in movies, which mentions Pi and Contact. This mostly coincidental juxtaposition reminded me of the conclusion of the novel behind the latter.
When I first read the book, a zillion years ago, I thought the ending was pretty clever. Then I realized that it was incredibly dumb. It later occurred to me that it might actually be intentionally dumb, and therefore incredibly clever, because Carl Sagan really ought to have known better, but... I don't know. I think he either genuinely goofed or assumed (probably correctly, in most cases) that his readers wouldn't notice.
A short aside: if you haven't read the novel or seen the film, you haven't really missed anything. The novel is basically a not-bad-but-not-brilliant ripoff of Stanisław Lem's Głos Pana and Solaris. The film is... well, a film. It's OK, I guess, and stars several excellent actors, and is reasonably but not entirely true to the novel.
If you haven't read the novel but intend to, I should warn you that the rest of this post is a HONKIN' HUGE SPOILER.
Here goes: at the end of the film, the perceived failure of the project is more or less covered up and Ellie returns to her former job as head of the SETI program. In the novel, however, she is disgraced and (from my recollection—remember, it's been years since I read it) ends up as a glorified tour guide. She somehow manages to wrangle sufficient computer time to search for something that was hinted at earlier in the novel: a proof of a Universal Creator, embedded somewhere in the digits of π. And guess what... The computer discovers that if you print the digits of π with a specific number of digits per line, after billions and billions of digits you come across a pattern of zeroes that forms a circle on the page. There is a God. QED.
Here's the problem: π is transcendental. If you search long enough, no matter what you're looking for, you'll eventually find it.
Monkeys and typewriters, Ellie. Monkeys and typewriters.
by Dag-Erling Smørgrav (noreply@blogger.com) at March 29, 2011 02:57 AM
"I'm impressed with what the rethinks are doing, but it feels like they're doing too much — my data store needs are very minimal," I continued. "Maybe I should just write my own data store; it can't take more than a few months."
I'm very pleased to finally announce the availability of version 1.0.0 of the kivaloo data store as BSD-licensed open source software.
Last week, we saw one of the first days with warm and sunny spring weather. We decided to go for a trip to Frederiksberg Have for ice cream. Of course, we did bring the camera and shoot some of the local city wildlife. The ducks were clearly more interested in the food, but we did manage to get some actions shots. All pictures here.
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
I stumbled upon the banana argument today. I don't know whether to laugh or cry. A bit of both, I guess.
Laugh: Pardon me for resorting to such crude humor, but nearly every argument Ray Comfort presents as “proof” that the banana was designed by God for human consumption can also be used to “prove” that the human penis was designed by God for fellatio. Even better, the penis and its wrapper are reusable!
Cry: Comfort quotes Darwin to bolster his argument that the eye cannot possibly have evolved. However, the quote he uses is merely the first sentence of an entire chapter devoted to explaining how Darwin thinks it evolved. Here is the complete first paragraph:
To suppose that the eye with all its inimitable contrivances for adjusting the focus to different distances, for admitting different amounts of light, and for the correction of spherical and chromatic aberration, could have been formed by natural selection, seems, I freely confess, absurd in the highest degree. When it was first said that the sun stood still and the world turned round, the common sense of mankind declared the doctrine false; but the old saying of Vox populi, vox Dei, as every philosopher knows, cannot be trusted in science. Reason tells me, that if numerous gradations from a simple and imperfect eye to one complex and perfect can be shown to exist, each grade being useful to its possessor, as is certainly the case; if further, the eye ever varies and the variations be inherited, as is likewise certainly the case; and if such variations should be useful to any animal under changing conditions of life, then the difficulty of believing that a perfect and complex eye could be formed by natural selection, though insuperable by our imagination, should not be considered as subversive of the theory. How a nerve comes to be sensitive to light, hardly concerns us more than how life itself originated; but I may remark that, as some of the lowest organisms in which nerves cannot be detected, are capable of perceiving light, it does not seem impossible that certain sensitive elements in their sarcode should become aggregated and developed into nerves, endowed with this special sensibility.
The rest is here (chapter 6, “Organs of Extreme Perfection and Complication”)
Comfort also quotes Einstein as a further appeal to authority. The problem is that Einstein, notwithstanding his quip that “God does not play dice”, was an avowed atheist, although he used the term “agnostic” to distance himself from those he called “professional atheists”. In his own words:
I do not believe in a personal God and I have never denied this but have expressed it clearly. If something is in me which can be called religious then it is the unbounded admiration for the structure of the world so far as our science can reveal it.
by Dag-Erling Smørgrav (noreply@blogger.com) at March 23, 2011 11:02 PM
Without any delay at all, Firefox 4 is available in FreeBSD ports! Big thanks to everyone involved! So far I tried Acid3 and WebM movies and they all work fine...
Without any delay at all, Firefox 4 is available in FreeBSD ports! Big thanks to everyone involved! So far I tried Acid3 and WebM movies and they all work fine...
Here’s me thinking “quiet” means “just tell me what I want to know”:
dev1578 ~ % yumdownloader -q—urls—source tmux
Loading “fastestmirror” plugin
Loading mirror speeds from cached hostfile
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
The "FreeBSD laptop as a Wireless Access Point for an iPhone" project I wrote earlier about has made me some followers, mostly they have no idea where the free internet connection comes from. But, based on the amount of download measured on it, they are enjoying it. One of the methods to determine how many people are on it is to use the output of the "arp -na" command: Every MAC address you see there is a mobile device which is associated with the wireless access-point you created.
One thing which you can do with that data is to match it against manufacturers. Very boring for non-networking techies... Don't read the rest :-)
MAC addresses consist of 12 hex-digits (48 bits) which are split in two parts: A six hex-digit (24 bits) prefix and a six hex-digit sequence number.
The MAC (or OUI as the IEEE calls it) prefix database can be found on the website of the IEEE at http://standards.ieee.org/regauth/oui/oui.txt. It contains at the moment of writing 14765 prefixes. The manuf(acturers) file from the Wireshark project can be found at http://anonsvn.wireshark.org/wireshark/trunk/manuf and contains 18321 prefixes plus a handful of shared prefixes.
Why is the one from the Wireshark project larger? Not really sure, but if you look at the registration costs it (US$ 1750 for a public registrered prefix OR US$ 1750 plus US$ 2100 per year for a private registered prefix) must be part of it. So it could be that the list from the Wireshark project has determined a bunch of the private ones. And unlike IP space which you can register in advance, you can't get a new prefix until you have certificated that you have used 95% of the sequence numbers.
Some statistics based on grep and cut and wc:
Number of prefixes Company 503 Cisco 122 Shenzhen 112 Motorola 109 Nokia Danmark A/S 84 Samsung Electronics 84 Apple 78 Intel Corp 59 Advanced (??) 61 Hewlett Packard 51 Private
"Advanced" could be a mistake in here, since it matches Advanced This and Advanced That. "Private" means a company who pays the US$ 2100 per year. Shenzhen has the same issue as "Advanced", it is a large bunch of companies in the Shenzhen city in China (near Hongkong). Apple was the company I didn't expect in the Top 10, but considering their iPhone / iPad success, it shouldn't surprise much.
Every prefix has 2 ** 24 entries in it, or 16 777 216 (about 16 million if you are conservate, or 17 million if you are optimistic), making there 1.4 billion Apple MAC addresses in the world. That number is not the number of Apple devices, since you need one per network interface: Ethernet, wireless or Bluetooth.
But the other number of Cisco is much more impressive: 8 438 939 648 MAC addresses. More than the next five in the list together.
Unfortunately the list of prefixes does not contain any assignment dates, it would have been interested to see what happened when LANs based on switching instead of hubs became the norm (and thus Cisco when Cisco started to sell their switches) and when mobile devices like the iPhone became popular, it would have boosted the allocation rate by Apple for sure.
MAC prefix exhaustion?
Unlike other technologies, and IPv4 comes in mind here, the MAC address prefix pool is pretty much unlimited but also only slowly being touched: There are 2 ** 22 or 4 194 304 prefixes. The number is 22, not 24 because two bits in the first byte of the prefix are used to determine if the MAC address is globally unique one or a special one. And right now, a good 35 years after the invention of Ethernet and Tokenring there are not even 19 thousand used.
The other causes are of the more strict rules the IEEE handles: You get a single prefix and don't get more until you have informed us officially that you have used 95% of them, and of course that you actually need to produce (and sell) something which uses a MAC address.
As probably everyone knows by now, Google Summer of Code 2011 is announced! As in previous years, FreeBSD is expected to have a presence and this is a great opportunity for every student interested in FreeBSD to get involved and get payed doing so! FreeBSD has a pile of ideas page and I'd like to promote some of those I think are interesting.
For some years (many more than really necessary) I have been workin on and off on a pet project of mine, mdcached - a cache server similar to memcached. I've figured it's time to finish it and bring it to usable state now, so (I hope) I'll be writing up the progress here. For now, I have some really nice benchmark results which show off its strengths compared to memcached.
What is truth?
This question is being asked with increasingly frequency these days, and I&aposm sure some of you winced when you read it. However, it was very much on my mind when I stepped out of the shower this morning, owing to the fact that my 50 l water heater does not allow me to shower long enough to get warm after a night in a chilly room.
What does that have to do with epistemology? Well, it led me to reflect on the semantics of the following statements:
Setting aside their truthfulness, which of these statements is objective, and which is subjective?
The conclusion I arrived at is the opposite of the obvious one, which is that the first statement is an objective statement, while the second is a subjective statement.
The first statement, “I am cold”, is based on the speaker&aposs perception of his environment and of his body. It is entirely subjective. Although a thermometer may show that the speaker&aposs core temperature is within the normal range, the speaker is still telling the truth as he or she perceives it. Therefore, both the speaker and the thermometer are correct, even though they disagree.
The second statement, “I feel cold”, is actually a meta-statement: it is akin to saying “although this may not actually be the case, I am currently experiencing sensations consistent with having a low core temperature”. It is an objective description of the speaker's subjective sensation (unless the speaker is lying) and may be corroborated by symptoms such as shivering, goosebumps, etc.
In fact, a person with a rising fever will feel cold despite having an abnormally high core temperature, and usually also a high skin temperature.
That said, I can easily see arguments in support of the opposite thesis.
Discuss.
by Dag-Erling Smørgrav (noreply@blogger.com) at March 08, 2011 12:03 PM
As previously noted, I have an interest in running large-scale services on equipment which would be called underpowered by many, so I find that this 256-CPU-cluster-in-a-rack is in many ways a dream machine. Of course, I don't think of them as underpowered but as efficient for the task, scalable and finally, green.
Like probably everyone who is reading this, I occasionally buy gadgets online, usually via eBay. I've heard of DealExtreme before and some people I know had ok experiences with it so I decided to try it - that was last year in December. Due to some bad luck and apparently horrible organization of DX, I still haven't received the things I bought. At this time, effectively I've been cheated for around $400 by DealExtreme.
Another thing I've wanted for a long time is integrated file versioning (if possible coupled with pervasive auto-save). It looks like the new Mac OS X versions will have it. Apple apparently has the knack for getting things right.
Something I've wanted for a long time is finally happening, though slowly and with shy experimental feel to it. "Smartphones" nowadays are essentially computers with more computing power than "real" desktop workstations 5 years ago: remember that it was only in late 2005 that dual-core Intel CPUs were coming fresh out from foundries and with initial clocks lower than 2 GHz. Now, all we need is good (and if possible wireless) peripheral connectivity options for smartphones to become the information centres of individual persons.
One of the uses of my iPhone is as a replacement for the newspapers and other news sources. Yes, I can do this on the computer too, so far nothing new. The thing which fascinates me is how much time I spend on waiting for the data to be retrieved.
Take the ABC app. On it, I only read two sections: The Just In section and The Drum. I open the app, have to wait for the Latest News section (which is not the Just In section) has been loaded, then I tap for the overview of the sections (which is instantly) and then tap on the Just In section, for which the overview gets loaded. To read an article, I tap on that article which then gets loaded. So I have to wait three times before I can read a single article, and every other article I need to wait again. Even after having read the ABC news this way for about a good year now, it still doesn't remember which news sections I'm interested in and doesn't pre-load them.
Take the SMH app. It is a little bit nicer, you can tell it which sections you are interested in and only shows those. It opens the one you opened last time up first. Could be close to a winner! But then it has the same cons as the ABC app: It loads the index and when you tap on the article you want to read it loads that one. Waiting twice again.
Take the Volkskrant app. It has two major sections, the Just In section and the Opinion section. When you start the app, the index of the Just In section gets loaded together with the text of all the articles. When you tap on an article, it comes up immediately and then loads any images: You can read it immediately. So you only wait once.
Take the BBC app. Just like the Volkskrant app, it loads the index and contents of the main sections at startup, which also has a lot of images on it. When the article gets loaded, it will load the image on the article.
The method of loading of content done by the BBC app and the Volkskrant app is by far the most ideal way of using an online news app. I am not yet sure if I prefer the sober layout of the Volkskrant or the smooth horizontal scrolling layout of the BBC. But overall, I am looking forward to the day the Australian news apps have caught up!
The result is a tool I call "fpm" which aims to help you make and mangle packages however you choose, all (ideally) without having to care about the internals of your particular native package format.
The goal of this project is not to undermine upstream packaging but to grant everyone the ability to trivially build and edit packages. Why? Not all software is packaged. Not all software of the version you want is packaged. And further, not all users are willing or able to take the time to learn all the ins and outs of their package build tools.
For example, you can package up your /etc/init.d directory as an RPM by doing simply this:
% fpm -s dir -t rpm -n myinitfiles -v 1.0 /etc/init.d ... Created /home/jls/rpm/myinitfiles-1.0.x86_64.rpmfpm will create a simple package for you and put it in your current directory. The result:
% rpm -qp myinitfiles-1.0.x86_64.rpm -l /etc/init.d /etc/init.d/.legacy-bootordering /etc/init.d/NetworkManager.dpkg-backup ... % rpm -qp myinitfiles-1.0.x86_64.rpm --provides myinitfiles = 1.0-1 % rpm -qp myinitfiles-1.0.x86_64.rpm --requires rpmlib(PayloadFilesHavePrefix) = 4.0-1 rpmlib(CompressedFileNames) = 3.0.4-1You can package up any directory. But there's more.
Above, I didn't specify a package summary, so how about fixing the rpm to include the description? You can use RPMs as the source (-s flag) in fpm. There's also a helpful '-e' (--edit) flag that'll let you edit the rpm spec (or debian control) file before building.
% rpm -qp myinitfiles-1.0.x86_64.rpm --info | grep Summary Summary : no summary given % fpm -s rpm -t rpm -e myinitfiles-1.0.x86_64.rpm ... this opens up $EDITOR so you can edit the spec file it generated ... ... make some changes to the spec, including adding a proper 'Summary' ... Created /home/jls/rpm/myinitfiles-1.0-1.x86_64.rpm % rpm -qp myinitfiles-1.0-1.x86_64.rpm --info | grep Summary Summary : my /etc/init.d directoryThe '-s dir' flag says the source of the package is a directory. There's also support for other package sources like rubygems, other rpms, debs, and more on the way.
With FPM, you can specify dependencies, architecture, maintainer, etc. All from a simple command line, and never forcing you to learn the pain and suffering that can come with rpm spec files or debian package building.
You can install fpm with: gem install fpm
The project page is here: https://github.com/jordansissel/fpm
The wiki is here (has more examples): https://github.com/jordansissel/fpm/wiki
![]()
Here is a couple of examples where "clouds" are a bad idea (basically a "RISKS" take on clouds):
Despite the serious losses our team had in the previous Winter 2010 competition, for example the zero won games and thus being ranked last, I decided to go continue in the Summer 2010/2011 competition. Minor detail: Nobody from the old group kept playing. Luckely we got a new team together.
This season wasn't as bad as the previous one despite that we still ended up second last: We won two games.
Our new team has a good players, we just don't work together or take the basic rules into account. Too many times the ball just gets slammed into our side of the net, too many times we don't setup properly, and too many times we lose the ball because we don't take a step to the left or right to properly catch and play the ball.
For the next competition, how to can explain these issues to the members of my team without upsetting him or her and everybody else? Maybe I should just do it and see what happens...
As has been expected, previously announced and tested, ZFS v28 has been committed to FreeBSD HEAD!
New features include:
As always, testers are welcome!
As has been expected, previously announced and tested, ZFS v28 has been committed to FreeBSD HEAD!
New features include:
As always, testers are welcome!
LibreOffice is a recent fork of OpenOffice, made to counter Oracle's mistreatment of the neat stuff they bought with Sun. Whatever Sun did or didn't do, its policy of openess is something so good and unique for a large company that I doubt we will see it again any time soon.
Anyway, LibreOffice has big plans. They are trying to set up a foundation to ensure resources and governance for future software development - which is a good thing, even great thing, and I hope they won't end up as some other foundations.
Project New House
Earlier this year my family and I started project New House. After renting for about 10 years here in Australia, we close the door on that issue and open the door of our own house.
Buying a house is not cheap, but thanks to proper saving in the last ten years and some luck (Getting shares from the company you work for at the lowest possible price at the beginning of the GFC for example :-) we now own the lower part of the house while the bank owns the upper half.
The town house itself is relatively new, 15 years old, one old-lady-owner who didn't do too much damage to it. So far I haven't found any skeletons in the cupboards.
The stats are: Three bedrooms, nice kitchen, nice bathroom, a grand total of three toilets (WTF? Why does a house like this three toilets?), two car garage (so the car is allowed inside next to the bicycles).
We already met four of the four neighbours, nice people. The fifth neighbour, not met despite various communication attempts) is described as a woman who does not like children and complains about everything. Oh well, five friendly families out of six houses is not bad. And if she wants to make her life miserable, go for it.
Some things need to be fixed, adjusted or just peed on to make it ours:
Because we moved in the hottest days in summer this year (not worlds greatest idea), we know exactly where the hottest parts and least ventilated parts in the house are and have invested in a couple of ceiling-fans for these rooms.
The light in the ceiling of the bathroom might be at the right position for small Australia old ladies to look in the mirror, for tall Dutchmen it just gives me a nice halo around the head, so we will get an extra light above the mirror.
While swapping the lightbulbs for energy-saving ones, I found out that the light above the frontdoor has a screw-fitting instead of a banjonet-fitting, which gives me the opportunity to use an old present from my brother: An 1.5 Watt LED colour-changing-lightbulb. Dank broertje!
Over time I hope to give more updates on the sub-projects of Project New House, like fixing the garden (Currently covered with "Buffalo grass" or "St Augustine Grass" or "Stenotaphrum secundatum" which makes it impossible to mow with a handmower and getting a solar-thingie on the roof.
you need to watch now if you haven't seen them already. None of them are new, they just happen to speak strongly to me right now.
1. Easy A. The actual movie is at least as funny as and far more serious than the trailer.
After you've watched it, go rent (or buy) all the classics Olive references: Say Anything, Can't Buy Me Love, Sixteen Candles, The Breakfast Club, Ferris Bueller's Day Off (sorry, can't be bothered to add links). Did I miss any? Then put this on repeat on the stereo:
2. P!nk: Fuckin' perfect. The original version, not the “family-friendly” shit they show on VH1.
3. It Gets Better. There are hundreds of clips, but you can start with Barack Obama:
Compare and contrast with whoever that guy was that P!nk sings about here:
Don't read too much into this, I'm not coming out or anything.
by Dag-Erling Smørgrav (noreply@blogger.com) at February 10, 2011 10:03 PM
ttyv0 "/usr/libexec/getty Pc" cons25 on secure
P|Pc|Pc console:\
:ht:np:sp#115200:
#
# FreeNAS menu system entry
#
FreeNAS|freenas|FreeNAS Menu:\
:ht:np:sp#15200:lo=/etc/netcli.sh:al=root:
ttyv0 "/usr/libexec/getty freenas" cons25 on secure
#!/bin/sh
# Helper script to set the path for netcli menu
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
exec /etc/netcli
by Warner Losh (noreply@blogger.com) at February 02, 2011 01:11 PM
One reason why I maintain this blog is to do some advocacy for FreeBSD - attempt to spread the word about new developments and generally promote it.
It's not an easy job as the system constantly struggles for popularity aside the much more popular Linux (and on the other side - older Unixes and Windows). I've accidentally come across this page on www.freebsd.org listing some popular FreeBSD users but I think it badly needs updating.
Are you a big FreeBSD user? Working in an organization which uses FreeBSD on a larger scale? Write about it here (in comments)! No need to mention details if you think they are confidential!


I'm sure I don't need to remind anyone what this is about...
The latest news: Theo now says that it is probable that NetSec was indeed contracted to insert backdoor code into OpenBSD, but after a month of review and changelog archeology, there is still no sign that they succeeded or even attempted to push tainted code into the tree.
The audit (which is still ongoing) did uncover one serious bug, but there is no reason to believe that it was planted deliberately. This relates to CBC mode, an encryption protocol in which each block of plaintext is combined with the ciphertext of the previous block before encryption to make it harder to attack ciphertext blocks individually.
If I understand Theo's message correctly,
My bounty still stands, and I will even relax the requirements a bit: you are not required to show that OpenBSD is still exploitable, only that it was exploitable on December 11, 2010 (the date of Perry's email to Theo).
by Dag-Erling Smørgrav (noreply@blogger.com) at January 25, 2011 05:52 PM
It's time for another FreeBSD status report! Among the many news I'd like to highlight:
Read the whole report for details!
It's time for another FreeBSD status report! Among the many news I'd like to highlight:
Read the whole report for details!
That's what Final Fantasy used to be about. And leveling. And more leveling. And chocobos. Been there, done that, got the T-shirt (and a couple of others for when I'm in a Konami or Nintendo mood).
Final Fantasy XIII, on the other hand, is mostly about pressing ⨉.
FFXIII's combat system is very different from that of the previous games. The most obvious difference is that it is no longer turn-based, which was a bit of a shock to me, and which has probably angered many fans. Unless you enjoy standing still while your adversary pummels you, you often have little choice other than to select “auto-battle” and let the game make all the decisions for you. When you face an adversary you know well (or one you've Libra'ed), this usually works out OK for Commandos and Ravagers, and to a certain degree also Medics, but not for Synergists and Saboteurs. Synergists, especially, seem to stick to a predefined sequence of buffs instead of prioritizing those which are appropriate for the current adversary. For instance, there is no point in casting Protect when fighting a monster that uses only magic attacks.
At the end of every battle, you get a rating and a score which are based mainly on how long the battle lasted compared to how long the game thinks it should have. This rewards players who use Haste or are good at sneaking up on enemies (which is not always possible). However... neither the rating nor the score have any effect on the game whatsoever except for how fast your TP bar replenishes. Getting a high rating or score, which can sometimes only be achieved by consuming hard-to-get Fortisol or Deceptisol potions, does not give you better drops or help you level faster. You level by spending Crystarium points, and the amount of CP you get for each battle seems to be based exclusively on either the number of enemies you fought or their total HP (I haven't quite figured it out), and you get enough CP from the fights you can't avoid to max out your Crystarium. Thus, there is no incentive to go looking for trouble and no incentive to fight well, and no character development. All you have to do is fight the enemies that show up in the storyline, make it out alive (and if you don't, the game just takes you back to a point shortly before the fight), and spend your CP.
So far, I've played to chapter 7 (about twenty hours), and only in the last chapter or so have the fights been sufficiently difficult to require some thought and to be enjoyable, and I suspect that that's only because I haven't leveled any of my weapons or accessories. That's right, I'm still fighting with level 1 everything, and I'm doing just fine.
The storyline, by the way, is just that: a storyline. You follow a narrow path, fight the monsters along it, and watch the many cutscenes. There are a few places where you can circle around a difficult enemy or take a small detour to pick up a treasure, but you always end up back where you would have been if you'd continued straight on.
Now, I've heard that the game opens up in chapter 10, which should be about ten hours from where I am. It is not uncommon for games to have some sort of “cold open”, to borrow from TV terminology, where your choices are restricted and the outcome has little or no effect on the rest of your game, as a sort of tutorial, but I've never played a game that had a thirty hour long tutorial, and I won't blame people who give up long before then. To be honest, I'm not sure why I haven't; I guess it's partly because the graphics are gorgeous (the game itself is in 720p, but the many, many cutscenes are in 1080p), partly because I'm starting to enjoy the combat system, or at least consider it a worthy challenge, and partly because I'm hoping it'll get better.
There is plenty more to dislike about FFXIII. The story is frankly uninteresting and does not really progress; most of the cutscenes so far in the games are either about things that happened before the game started, or about the characters being emo and acting completely irrationally. Everything that happens in the cutscenes in summarized in your Datalog, which is an interesting read, and not in a good way. Not only is the prose as purple as the back cover of a pulp fantasy novel, but the emotions and inner dialog described in the text rarely bear any resemblance to what the characters actually say in the cutscenes. Let me give you an example: according to the Datalog, Hope is seething with rage and working up the strength, skills and courage to confront and kill Snow, whom he blames for his mother's death. If you only watched the cutscenes, though, you'd think he was actually working up the courage to declare his undying love to Snow—until chapter 5, where he discusses the issue openly with Lightning. However, while he does voice these thoughts and feelings in chapter 5 (and 7), they are never reflected in his intonation or body language.
Now Vanille... Vanille really gets on my nerves. I can't decide whether she is an overendowed young girl in very inappropriate dress, or an emotionally retarded young woman in very inappropriate dress. She has the body shape of a grown woman but the body language of a ten-year-old girl. She also giggles and moans a lot. In fact, she giggles and moans so much that I sometimes worry the neighbors might think I'm watching porn. Vanille is voiced by Australian actress Georgia van Cuylenburg, who stated in an interview that “[i]t was important for me to keep my Aussie accent to show that Vanille is from somewhere different”, but she doesn't sound Australian at all; to my ears, she sounds like an American voice actress who occasionally remembers that she was asked to speak with a British accent. Fang, who is from the same “somewhere different” as Vanille, is voiced by American actress Rachel Robinson; ironically, her Australian accent, while obviously fake, is far more convincing than Vanille's.
So, we've covered Hope, Vanille and (very briefly) Fang. Who's left? Snow is a conceited ass. Sazh is emo as hell in cutscenes, and jittery as hell in combat. Lightning... Lightning's OK, I guess, except for the way she treats Snow (and Serah, for that matter). She is also the most proficient fighter; Sazh would be a close second if his combat animations weren't so bloody annoying. There isn't really that much more to say about the characters; they're nicely rendered but completely unconvincing. The NPCs aren't much better; I liked Sazh's son and Hope's mother, but Snow's buddies are as annoying as he is, or worse.
BTW, where are the chocobos? I want chocobos! No, the chick is Sazh's 'fro doesn't count.
Looking forward to chapter 10...
by Dag-Erling Smørgrav (noreply@blogger.com) at January 19, 2011 07:18 PM
by Warner Losh (noreply@blogger.com) at January 18, 2011 09:55 AM
In a few weeks, I’ll be heading to the FOSDEM conference in Brussels again this year. I’ll spend most of my time at the FreeBSD booth for the FreeBSD Foundation, so if you’re there drop by to say hi, discuss the Foundation’s work, pick up a Foundation flyer, check out the swag, or make a donation. There will also be a BSD DevRoom where there will be some interesting presentations and discussions that I might attend. Remember, FOSDEM is free to attend. Hope to see you there!
Related posts:
Related posts brought to you by Yet Another Related Posts Plugin.
Recently I was on a holiday where the provider of my iPhone had no signal, but where the provider of my 3G modem for the laptop did have a signal. At least my glass was half-full!
In the past I have tried to setup Bluetooth between my laptop and my iPhone, and that resulted in a night of hard work and no effort. This time I tried a different approach: Instead of using Bluetooth for communication, I transformed the FreeBSD laptop into a wireless access point.
The command to change the wireless card from a normal client to a wireless access point are:
[~] edwin@lappie>cat wlan-iphone #!/bin/sh ifconfig wlan0 destroy ifconfig wlan0 create \ wlandev ath0 \ wlanmode hostap \ bssid \ authmode open \ ssid "My iPhone WiFi" ifconfig wlan0 up ifconfig wlan0 inet 10.0.0.1 netmask 255.255.255.0 sleep 1 sysctl -a net.inet.ip.forwarding=1 service isc-dhcpd restart
Notes:
The 3G connection is setup via ppp(8) and to enable NAT on the outgoing packets, you need to enter the following command or add it to the right label in your ppp.conf:
ppp ON lappie> dial Ppp ON lappie> PPp ON lappie> PPp ON lappie> Warning: 0.0.0.0/0: Change route failed: errno: No such process PPP ON lappie> nat enable yes PPP ON lappie>
And to make sure that the connected clients get their IP address, you should run the ISC DHCP server with for example the following configuration:
option domain-name "";
option domain-name-servers 8.8.8.8; default-lease-time 150;
max-lease-time 300; ddns-update-style none; authoritative; log-facility local7; subnet 10.0.0.0 netmask 255.255.255.0 { range 10.0.0.10 10.0.0.99; option routers 10.0.0.1;
}
Notes:
Everything is working now, your glass is full again! :-)
Just to let you know what the current status is wrt. 4k drives:
It looks like the consensus in the industry (meaning everyone except Western Digital) is to announce dual sector sizes, i.e. 512-byte logical sectors on top of 4096-byte physical sectors.
Ivan Voras has taken the initiative to organize a 4k BoF at BSDCan, although judging from the (private) email exchange on the subject, it's quite possible that a decision will be made before then. Currently, it looks like we're moving towards having the low-level driver report a 512-byte sector size and 4096-byte stripe width (and, if necessary, an appropriate offset) to GEOM. This preserves backward compatibility, but announces to GEOM consumers that it is a good idea to do I/O in 4096-byte blocks and align data structures on 4096-byte boundaries. All that remains is then to make sure that those GEOM consumers we care about (particularly ZFS) take advantage of this information.
The situation for WD “Advanced Format” drives is a bit more complex, because they announce 512-byte logical sectors. The only solution I can see is to add a quirk system to the ada driver (and possibly to ata as well, if we still care about it) similar to the ones we have for SCSI and USB devices, and match the model number. I believe /WD\d+[A-Z]+RS/ should match all existing Advanced Format drives with no false positives.
by Dag-Erling Smørgrav (noreply@blogger.com) at January 12, 2011 05:52 PM
Recently I was on a holiday where the provider of my iPhone had no signal, but where the provider of my 3G modem for the laptop did have a signal. At least my glass was half-full!
In the past I have tried to setup Bluetooth between my laptop and my iPhone, and that resulted in a night of hard work and no effort. This time I tried a different approach: Instead of using Bluetooth for communication, I transformed the FreeBSD laptop into a wireless access point.
The command to change the wireless card from a normal client to a wireless access point are:
[~] edwin@lappie>cat wlan-iphone #!/bin/sh ifconfig wlan0 destroy ifconfig wlan0 create \ wlandev ath0 \ wlanmode hostap \ bssid \ authmode open \ ssid "My iPhone WiFi" ifconfig wlan0 up ifconfig wlan0 inet 10.0.0.1 netmask 255.255.255.0 sleep 1 sysctl -a net.inet.ip.forwarding=1 service isc-dhcpd restart
Notes:
The 3G connection is setup via ppp(8) and to enable NAT on the outgoing packets, you need to enter the following command or add it to the right label in your ppp.conf:
ppp ON lappie> dial Ppp ON lappie> PPp ON lappie> PPp ON lappie> Warning: 0.0.0.0/0: Change route failed: errno: No such process PPP ON lappie> nat enable yes PPP ON lappie>
And to make sure that the connected clients get their IP address, you should run the ISC DHCP server with for example the following configuration:
option domain-name "";
option domain-name-servers 8.8.8.8;
default-lease-time 150;
max-lease-time 300;
ddns-update-style none;
authoritative;
log-facility local7;
subnet 10.0.0.0 netmask 255.255.255.0 {
range 10.0.0.10 10.0.0.99;
option routers 10.0.0.1;
}
Notes:
Everything is working now, your glass is full again! :-)
Just for the fun of it, I checked out some Riverbed appliances and hardware on eBay.
So far no luck, still haven't been able to find a full working Steelhead appliance on eBay. Time to go to bed!
All major FreeBSD filesystems support 4K sectors (UFS, ZFS, ext2), and so does the lower level - GEOM - but currently there's an issue of communicating this configuration between all the layers. A part of the problem is that the current drives (and the situation will probably not change during this new decade) advertise two sector sizes: both 512 byte and 4K, and the system needs to correctly interpret them. All this will be resolved when a consensus on the topic gets achieved, but until that happens (hopefully soon), there is a set of easy workarounds, which I'll describe here.
All major FreeBSD filesystems support 4K sectors (UFS, ZFS, ext2), and so does the lower level - GEOM - but currently there's an issue of communicating this configuration between all the layers. A part of the problem is that the current drives (and the situation will probably not change during this new decade) advertise two sector sizes: both 512 byte and 4K, and the system needs to correctly interpret them. All this will be resolved when a consensus on the topic gets achieved, but until that happens (hopefully soon), there is a set of easy workarounds, which I'll describe here.
In just one minute it will be 2011 (hey, scheduling things is fun, this gives you something to read while I am jumping around, celebrating with Luca, Denise, Rik and Larissa the coming of the new year, and perhaps drink a beer, or more but enough about that.
It’s my tradition to have a new years post, and this year I decided to schedule it for the first year. I wont be able to write a post before tommorrow or perhaps even later so “Sad but true”.
The last year saw a lot of sad things, sad changes and sad news, deaths and so on. Please take a minute to remember the persons you lost this year, think about the bad and the good things you shared. Cheerish those good moments, you can be upset about the negative things, but it will only make you more grumpy, which isn’t worth it. Life is too short!
OK So we considered the negative things of the last year, but ending the year with a negative thing is not right, right? So also take a minute to remember the positive things, positive changes and positive news, the birth’s you saw this year, the news that people are pregnant and are expecting a child, the new job, consider it and remember it.
From my position I would like to offer you my very best wishes for the upcoming year, I hope that you will see the positive things of life, respect eachother, and that you are healthy and can remain healthy (and your relatives).
Ofcourse my new years post wouldn’t be the same without mentioning my beloved FreeBSD. The last year we saw a few new releases, saw a lot of hard work, had to deal with the economic crises and loads of more things. This year we will get generous donations from you… right? So that we can build even more funky stuff, and keep the best operating system!
Welcome.. 2011!
When money abounds, you buy office furniture and gamble with investments, when it's tight, you do something with it.
When money abounds, you buy office furniture and gamble with investments, when it's tight, you do something with it.
About a year ago I complained loudly that republishing material from previous years on a website suffering from content damagement system was inordinately difficult. A number of people helpfully pointed out some less-than-obvious things to me since, and I'm happy to report that getting things going was less painful this year.
There is still a lot of clicking involved and it's obviously nowhere near as intuitive as just fishing up a revision from source control. But at least my blood pressure stayed reasonably constant throughout. What can I say. I do operating systems, not websites.
Anyway - the keysigning bits for FOSDEM 2011 are online. Submit your keys.
I've been using ikiwiki
as my private wiki for several months
now, and have been very happy with it.
It's the ultimate geek wiki. You get:
markdown.git).Those are the big points. You get much more than that, of course, but those were the sellers for me personally.
Ikiwiki can also work as a blog, and since today, it powers this blog as well. I'll do a comprehensive writeup on what it took to convert this blog from Movable Type to ikiwiki once I am reasonably sure everything works to my satisfaction.
For months, I've been plagued by intermittent mouse freezes on one of my boxes.
It started after a regular Xorg upgrade. According to various mailing lists, that particular upgrade caused similar problems to a lot of people, so I tried different suggested fixes. No luck.
A bit later, Xorg on FreeBSD was modified to fix the reported problems. But the upgrade did not fix my problem.
Eventually I came to a realization that it is likely that the problem is not with the mouse driver or with any other part of Xorg. Rather, it was a problem with synergy client interaction with the new xcb. I even found a problem report with a supposed fix to the problem. By the time I've found it, the fix was committed to the synergy port, and was subsequently rolled back because it lead to other problems. I tried the patch in the PR anyway. Still did not help me.
Not wanting to spend too much time on this, I was coping with the delays and only occasionally, when annoyed more than usual, was trying to find another fix. Unsuccessfully, I must add, until this morning, when I discovered synergy+, a maintenance fork of the original synergy. I was not aware that synergy+ is basically a drop-in replacement to synergy, the binaries having the same names as in the original. Better still, synergy+ client works just fine with the original synergy server. So I've decided to give it a shot, removed the synergy package, and installed the synergy+ port. Voila, the freezes are gone. I am a happy camper now.
With the recent (2009-12-23) update to
FreeBSD's
sysutils/smartmontools port
smartctl
stopped working if run as non-root.
I did not investigate whether it is because of the change
in the way smartctl operates,
or whether it just stopped to be setuid root.
Normally I don't mind going root to run smartctl by hand,
but it presents a bit of a problem for the hddtemp_smartctl
Munin
plugin.
One possible solution is to add the munin user to the operator group,
add the following two lines to /etc/devfs.conf:
perm ata 0660
perm xpt0 0660
And finally, run sh /etc/rc.d/devfs restart.
Being the dummy that I am, I only thought about a simpler solution
when composing this post: just add user root into the [hddtemp_smartctl] section
of your munin/plugin-conf.d/plugins.conf file.
Besides being simpler, this method has an added advantage: an updated
version of the
sysutils/munin-node port
can easily incorporate this change.
Dag-Erling: hint, hint. 
Today at work I needed to locate and extract, automatically, some information from a website.
There was no direct URL to the information I needed, some fields had to be filled and some POST forms had to be submitted.
Normally I would use WWW::Mechanize for such a task, but in this particular instance the situation was made somewhat less managable because the site in question was implemented with ASP.NET.
The problem with this is that every link
has an associated JavaScript event handler
which does some housekeeping,
assigns things to funnily named
hidden input fields like __EVENTTARGET and __EVENTARGUMENT
and then POSTs a form.
My first thought was to try and find a CPAN module which handles those complications. Not surprizingly, there is one, aptly named HTML::TreeBuilderX::ASP_NET.
According to its documentation, the module works in combination with the standard LWP::UserAgent and HTML::TreeBuilder, and converts ASP.NET JavaScript posting redirects into an HTTP::Request object which can be fed to LWP::UserAgent's request() method. Just what the doctor ordered.
However, it turned out that my joy was a bit premature:
After a bit of pondering I decided that spending time on trying to fix the HTML::TreeBuilderX::ASP_NET module is a bit counter-productive - I needed the working code soon.
So what to do?
One thing we should keep in mind is that those JavaScript postbacks do not do anything fancy. The hidden fields that are filled in depend on what was clicked on the page, nothing else. After they are filled, a normal POST occurs.
So if we know what to POST, we could just use WWW::Mechanize and get the job done easily and quickly.
So the solution naturally splits into two parts - finding out what fields to set, and automating the process.
The first part is to launch a browser, do clicking and entering by hand, and capture what gets POSTed at each step. This capturing could be done by a variety of methods:
I have chosen the second option, since I had a script similar to what I need already, and since it is easy to filter out any parameters which I did not want to see, like __VIEWSTATE, which can easily be several kilobytes long.
Enter spyproxy.pl:
#! /usr/bin/perl
use strict;
use warnings;
use HTTP::Proxy;
use CGI;
my $proxy = HTTP::Proxy->new(host => "localhost");
$proxy->logmask(32); # 32 - FILTERS
$proxy->push_filter(
request => Spy::BodyFilter->new(),
);
$proxy->start;
package Spy::BodyFilter;
use base qw(HTTP::Proxy::BodyFilter);
sub will_modify { 0 }
sub filter
{
my ($me, undef, $req) = @_;
print $req->method, " ", $req->uri, "\n";
return unless $req->method eq "POST";
my $body = $req->content;
my $q = new CGI($body);
for my $p ($q->param) {
next if $p eq "__VIEWSTATE";
print "$p\n\t", $q->param($p), "\n";
}
}
Launch it locally in a terminal, set your browser's
proxy settings to localhost:8080, and watch the output
in the terminal.
The second part of the puzzle is to use the wonderful
WWW::Mechanize::Shell.
It provides an interactive shell, in which we
can issue GET requests, see the content of the responses,
view links, forms, and form fields with their values,
follow the links, set the value of the fields, click
on buttons and submit the forms.
Best of all, after getting what we are after
we can issue a script command and get a
piece of Perl code that will perform all
the tasks we've just done.
So the final solution looks like this:
spyproxy).ASP.NET application, you will want to ignore
the vast majority of the fields at any given moment. Don't
worry, humans are good at this sort of pattern recognition.
Pay special attention to __EVENTTARGET and __EVENTARGUMENT
fields.value fieldname fieldvalue).__EVENTTARGET was set, type submit in the shell;
otherwise, find the name of the button that was pressed
(see step 4), and type click buttonname in the shell;content in the shell)
to make sure that what you've got in the shell makes sense.script script-name.pl in the shell.I hope this trick will be of use to somebody. Enjoy!
For reasons which I am not going to delve into here (this is a topic for another post), we are going to get rid of about half of our books.
There are some (low) hundreds of books for the taking, slightly more than half in English, the rest being mostly Russian with a sprinkling of Danish here and there.
Fiction, non-fiction, textbooks, science fiction, you name it.
So, if you are in Copenhagen area and are interested, write me a note and consider coming over to have a look, maybe you'll find something you'd like to keep. All books are to be had for free, although we would not mind selling them if you will insist.
Often I want to know how long it took for a particular command to finish.
An obvious solution to use the time(1)
command does not work without a degree of
anticipation on my part that I do not normally
posess.
At some point I became sufficiently annoyed
to actually add some hooks to my .zshrc.
All commands executed in an iteractive shell
are timed, but the reporting is done
only for those that took longer than 10 seconds
to execute.
This ugly code does the job:
note_remind=0
note_ignore="yes"
note_command="?"
note_report()
{
echo ""
echo "note_report: $note_command completed in $1 seconds"
}
preexec()
{
if [ "x$TTY" != "x" ]; then
note_remind="$SECONDS"
note_ignore=""
note_command="$2"
fi
}
precmd()
{
local xx
if [ "x$TTY" != "x" ]; then
if [ "x$note_ignore" = "x" ]; then
note_ignore="yes"
xx=$(($SECONDS-$note_remind))
if [ $xx -gt 10 ]; then
if [ $TTYIDLE -gt 10 ]; then
note_report $xx
fi
fi
fi
fi
}
Enjoy.