Thursday, November 17, 2011

[LIFE] your guide to the 99%

Unless you've been living under a rock (like I do), you must have heard about the "Occupy" protests. You might consequently be wondering what the protests are about. You may even have googled the topic. Unsuccessfully, of course, because nobody knows what the protesters are protesting about. They're basically a group of self-proclaimed "average Joe" types, as evidenced by their refrain "we are the 99%".

Along with a long list of other demands, they want rich people (the 1%) to be taxed more. I am against this proposal, not because I'm in the 1% (I probably earn vastly less than the average protester), but for the simple reason that it is economically absurd. After all, the 1% can simply move to wherever they are taxed less. The U.S. is not the only place where business is being conducted. A country can efficiently tax only those individuals who are relatively inflexibly stationed in it. In other words, the 99%.

For the most part, the protests are simply about disgruntled individuals broadcasting the fact that they are unhappy. Now, I'm not saying it's OK for people to be unhappy, but they should consider that perhaps they are partly responsible for their situation. Sure, banks extended them adjustable-rate mortgages they could never pay after the "adjustment". But they signed off on it and borrowed the money, cognizant of the risks. Repeated attempts to create a "safe" economy where everyone would be happy have failed in the past -- they simply made everyone safe and miserable. Until they collapsed, making everyone very unsafe and consequently highly miserable.

My suggestion is simple: until we have a workable replacement for capitalism, let's not protest capitalism. Recent hardship may not be the fault of capitalism itself. It could very well be the combined effects of diminishing natural resources and reduced quality of human capital. No system can defeat the GIGO principle: garbage in, garbage out.

So, what are protesters shouting into society's collective ear these days? Do they have a plan for a replacing the current economic system with one that produces happiness from garbage? What kind of change are they trying to effect by "occupying" the system? That's right, "N/A"! They don't actually have a plan -- they just know that they're unhappy and the system is responsible for making them happy. Protesters seemingly expect the system to figure out what's wrong with itself and replace itself with something better. All on it's own. Unfortunately, few systems can diagnose themselves and nothing likes to be replaced.

Incidentally, while we don't fully understand what the protesters are trying to do, we know some of the things what the protesters are not trying to do -- they're not trying to better organize their lives, they're not trying to learn new skills, they're not trying to find jobs, etc. We know they're not trying to do any of these things because any of these things would easily take up all of the time that they spend protesting.

Saturday, October 22, 2011

[TECH] fixing the tab key in VNC

Setting up a simple VNC server such as TightVNC (tightvncserver) has gotten a lot harder in recent Ubuntu distributions, because the tab key seems not to work properly! I find that using the command line is nearly impossible when the tab key doesn't work in VNC. The problem seems to depend on the window manager in use (I use xfce4). After some hunting, it became apparent that the default window manager key bindings global binds the "<Super>"-Tab combination to a feature known as "switching windows within the same application." I don't know what a "Super" key is and apparently neither does the system, because it always treats the Tab key as "<Super>"-Tab, triggering this feature rather than passing the Tab key event through to the application. Clearing this key from xfce4-settings-manager => window manager => keyboard fixes the problem, as does the following command line equivalent:
xfconf-query -c xfce4-keyboard-shortcuts -p /xfwm4/custom/'<'Super'>'Tab -r
With the default settings getting less sane every day, it's not clear where Ubuntu is headed. The whole point of Ubuntu was to provide sensible defaults and clear away pitfalls like these. Most annoyingly, this used to work -- only a few versions back it was possible to simply "apt-get -y install tightvncserver" and run tightvncserver on the command line and get up and running in minutes rather than hours.

Wednesday, August 3, 2011

[LIFE] top 10 ways to avoid reading top 10 tip lists

  • when you encounter a top 10 tip list purporting to improve the way in which you do X, consider whether X is something you want to be able to do efficiently. if it isn't, simply stop reading.
  • consider that the effort you put into reading a top 10 tip list only pays off if you actually find yourself in situation X often enough that you need an efficient way to deal with it.
  • remember, you can stop reading top 10 tip lists after the first few tips if you think those tips pretty much cover all the bases. top 10 tip lists are usually arranged from the most helpful tip to the least helpful tip.
  • if you still find yourself reading too many top 10 tip lists, perhaps you are trying for too much efficiency in too many aspects of your life. most people get by fine without reading all that many top 10 tip lists.
  • sometimes it helps to give yourself a gentle reminder that there are more important tasks in your life that may be in more urgent need of your attention than reading top 10 tip lists.
  • keep in mind that you can always stop reading. reading is a voluntary activity and a choice.
  • your (clearly compulsive) reading of top 10 tip lists may be stemming from an anxiety or dormant fear of being placed in a situation where you must rapidly master a complex, unfamiliar task. unless you are James Bond, you may find reassurance in contemplating the rarity of such situations.
  • it may help to imagine yourself in a scenario in which reading a top 10 tip list causes an adverse outcome. for example, imagine yourself being fired after reading a top 10 tip list on how to demand a higher wage.
  • if all else fails, seek professional help for your top 10 tip list reading addiction.
  • to be honest, I'd much rather you hadn't read this far. clearly your compulsion is out of control. it may help to read each tip over again from the beginning and ask yourself whether you have sincerely tried to apply it with an open mind.

Wednesday, May 11, 2011

[TECH] nonlinear video editing with ruby

I recently had to do some video editing, and I ran into a spot of difficulty: I couldn't find a simple script-based system. Everything seems to be bundled with a complex, cumbersome graphical user interface. So I wrote a ruby script. It's meant to be included via "require" from a custom script containing configuration data and EDLs. It works in two stages: in the first stage, source material is expanded to uncompressed PCM and 4:2:0 YUV (typically generating huge video files). Title page text can also serve as source material (it is rendered using ImageMagick's "convert" tool). In the second stage, audio and video encoders are launched with their input files connected to FIFOs and the ruby script feeds these FIFOs with portions of the raw files according to an edit decision list. At the end of the second stage, the encoded audio and video files are muxed using mkvmerge. The whole thing isn't very user-friendly, but it does what little I needed done and it's a good starting point for anyone who needs to add special-purpose functionality.

The recommended way to use it is to edit one section at a time. That is, to "test" a section of video X...Y use an EDL like Y-3...Y , X...X+3. This allows you to adjust X and Y to get the cut points just right for the sequence X...Y and only requires you to encode 6 seconds of video in one go (with --preset ultrafast, it takes on the order of 9 seconds to see the result on a setup where the raw files are stored on very slow external storage). After testing each section, you can put them together into one large EDL and encode them with more aggressive compression settings. The script prints a progress line after each second of video encoded.

Hints:
  • Source video that has been split into multiple files by your video camera can usually be concatenated with "cat" and the result will work fine for importing.
  • 30fps is rarely 30fps. it is usually 30000/1001fps = 29.(970029)fps.
  • The frame size for 4:2:0 YUV is always 3 * width * height / 2.
  • The frame size for audio depends on the number of channels and the bit depth. 2 * 2 is a very good guess.
  • Video codec junkies will tell you that tweaking the video codec is essential for good quality. They are right. What they typically don't tell you is that the tweaking has already been done by the codec author. Use the --preset option.

Here follows the script (it's called vided.rb), and an example usage script (typically called go.rb).

#!/bin/false
# vided.rb
# copyright (c) 2011 by andrei borac

# please define the following (values are examples)
#  $vided_video_wid = 1920;
#  $vided_video_hei = 1080;
#  $vided_video_fsz = 3 * $vided_video_wid * $vided_video_hei / 2;
#  $vided_video_fps = 30000.0 / 1001.0;
#  $vided_audio_fsz = 2 * 2;
#  $vided_audio_fps = 48000.0;

raise("\$video_vided_wid undefined") if (!defined?($vided_video_wid));
raise("\$video_vided_hei undefined") if (!defined?($vided_video_hei));
raise("\$video_video_fsz undefined") if (!defined?($vided_video_fsz));
raise("\$vided_video_fps undefined") if (!defined?($vided_video_fps));
raise("\$video_audio_fsz undefined") if (!defined?($vided_audio_fsz));
raise("\$vided_audio_fps undefined") if (!defined?($vided_audio_fps));

def centconv(f)
  f = (10000.0 * f).round;
  f = 0 if (f < 0);
  f = 9999 if (f > 9999);
  
  n = f / 100;
  d = f % 100;
  
  n = n.to_s; n = "0" + n while (n.length < 2);
  d = d.to_s; d = "0" + d while (d.length < 2);
  
  return n.to_s + "." + d + "%";
end

def timeconv(s)
  s = s.round; # forget about fractional seconds
  s = 0 if (s < 0);
  
  m = ((s + 0.0) / 60).to_i; s -= 60 * m;
  h = ((m + 0.0) / 60).to_i; m -= 60 * h;
  d = ((h + 0.0) / 24).to_i; h -= 24 * d;
  
  s = s.to_s; s = "0" + s while (s.length < 2);
  m = m.to_s; m = "0" + m while (m.length < 2);
  h = h.to_s; h = "0" + h while (h.length < 2);
  d = d.to_s; d = "0" + d while (d.length < 3);
  
  return d.to_s + "d" + h.to_s + "h" + m.to_s + "m" + s.to_s + "s";
end

def run(command)
  $stderr.puts("^" + command + "$");
  
  if (!system("bash", "-c", "set -o errexit; set -o nounset; set -o pipefail; " + command))
    $stderr.puts("failed command ^" + command + "\$");
    exit(1);
  end
end

def import_generic(name, from, type)
  if (!File.exists?(type + "-" + name + ".raw.wok"))
    run("echo 0 5 0 > '.chomp." + name + ".edl'");
    run("mencoder -demuxer lavf -ignore-start '" + from + "' -of raw" + type + " -o '" + type + "-" + name + ".raw' -hr-edl-seek -edl '.chomp." + name + ".edl' -oac pcm -ovc raw -vf format=i420");
    run("touch '" + type + "-" + name + ".raw.wok'");
  end
end

def import_audio(name, from)
  import_generic(name, from, "audio");
end

def import_video(name, from)
  import_generic(name, from, "video");
end

def import_streams(name, from)
  import_audio(name, from);
  import_video(name, from);
end

def import_title_screen(name, from)
  if (!File.exists?("audio-" + name + ".raw.wok"))
    run("dd if=/dev/zero of='audio-" + name + ".raw' bs=1M count=1");
    run("touch 'audio-" + name + ".raw.wok'");
  end
  
  if (!File.exists?("video-" + name + ".raw.wok"))
    run("mencoder -of rawvideo -ovc raw -vf format=i420 -o 'video-" + name + ".420' mf://'" + from + "'");
    run("( for i in `seq 1 100`; do cat 'video-" + name + ".420'; done ) > 'video-" + name + ".raw'");
    run("touch 'video-" + name + ".raw.wok'");
  end
end

# w, h - size of title image to write text on
# offL - the left crop amount (image to be expanded this much leftward)
# offT - the top crop amount (image to be expanded this much upward)
# image is automatically expended rightward and downward to fill global widxhei
def import_title_screen_generate(name, w, h, offL, offT, d, text)
  if (!File.exists?("video-title-" + name + ".raw.wok"))
    run("convert " +
        "-size " + w.to_s + "x" + h.to_s + " " +
        "-background black " +
        "-fill white " +
        "-font Palatino-Roman " +
        "-density " + d.to_s + " " +
        "-pointsize 16 " +
        "-interline-spacing 13 " +
        "-gravity center " +
        "label:'" + text + "' " +
        "-background white " +
        "-gravity southeast " +
        "-extent " + (w + offL).to_s + "x" + (h + offT).to_s + " " +
        "-gravity northwest " +
        "-extent " + $vided_video_wid.to_s + "x" + $vided_video_hei.to_s + " " +
        "title-'" + name + "'.tga");
    import_title_screen("title-" + name, "title-" + name + ".tga");
  end
end

def above_zero(x)
  if (x > 0)
    return x;
  else
    return 0;
  end
end

def calculate_frame_offset(fps, tc_enter)
  fps += 0.0; tc_enter += 0.0;
  return above_zero((tc_enter * fps).round);
end

def calculate_frame_amount(fps, tc_ideal, tc_track, tc_enter, tc_leave)
  fps += 0.0; tc_ideal += 0.0; tc_track += 0.0; tc_enter += 0.0; tc_leave += 0.0;
  return above_zero(((tc_track - tc_ideal + tc_leave - tc_enter) * fps).round);
end

def reset_fifo(name)
  run("rm -f '" + name + "'; mkfifo '" + name + "'");
end

def export_chomp(dst_name, mencopts, local_video_wid, local_video_hei, lameopts, x264opts, mkvmopts, sections)
  if (!File.exists?(dst_name + ".mkv"))
    [ ".audio-", ".video-", ".final-" ].each { |prefix|
      [ ".raw", ".mp3", ".mp3.wok", ".264", ".264.wok", ".mkv" ].each { |suffix|
        run("rm -f '" + prefix + dst_name + suffix + "'");
      }
    }
    
    reset_fifo(".video-" + dst_name + ".raw");
    #reset_fifo(".audio-" + dst_name + ".mp3");
    #reset_fifo(".video-" + dst_name + ".264");
    
    audio_fifo_name = ".audio-" + dst_name + ".raw";
    video_fifo_name = ".video-" + dst_name + ".pre";
    
    reset_fifo(audio_fifo_name);
    reset_fifo(video_fifo_name);
    
    #run("( mkvmerge " + mkvmopts + " -o '.final-" + dst_name + ".mkv' '.audio-" + dst_name + ".mp3' --default-duration 0:" + $vided_video_fps.to_s + "fps '.video-" + dst_name + ".264' ; touch '.final-" + dst_name + ".mkv.wok' ) &> '.log-mkvm-" + dst_name + "' &");
    run("( lame -r " + lameopts + " '.audio-" + dst_name + ".raw' '.audio-" + dst_name + ".mp3' ; touch '.audio-" + dst_name + ".mp3.wok' ) &> '.log-lame-" + dst_name + "' &");
    run("( x264 " + x264opts + " -o '.video-" + dst_name + ".264' '.video-" + dst_name + ".raw' " + local_video_wid.to_s + "x" + local_video_hei.to_s + " ; touch '.video-" + dst_name + ".264.wok' ) &> '.log-x264-" + dst_name + "' &");
    run("( mencoder -demuxer rawvideo -rawvideo i420:w=" + $vided_video_wid.to_s + ":h=" + $vided_video_hei.to_s + " '.video-" + dst_name + ".pre' -of rawvideo -o '.video-" + dst_name + ".raw' -nosound -ovc raw -vf " + ((mencopts.length() > 0) ? (mencopts + ",") : ("")) + "format=i420 ) &> '.log-menc-" + dst_name + "' &");
    
    io_audio_fifo = IO.new(IO.sysopen(audio_fifo_name, "wb"), "wb");
    io_video_fifo = IO.new(IO.sysopen(video_fifo_name, "wb"), "wb");
    
    schedule = [];
    
    sections.each { |src_name, tc_enter, tc_leave|
      schedule << \
      [
       IO.new(IO.sysopen("audio-" + src_name + ".raw", "rb"), "rb"),
       IO.new(IO.sysopen("video-" + src_name + ".raw", "rb"), "rb"),
       tc_enter,
       tc_leave
      ]
    }
    
    tc_final = 0.0;
    
    sections.each { |src_name, tc_enter, tc_leave|
      raise("backward section") if (tc_leave <= tc_enter);
      tc_final += (tc_leave - tc_enter);
    }
    
    tc_ideal = 0;
    tc_audio = 0;
    tc_video = 0;
    tc_wrote = 0;
    
    operation_began = Time.now.to_f;
    
    schedule.each { |io_audio_file, io_video_file, tc_enter, tc_leave|
      tc_enter += 0.0; tc_leave += 0.0;
      
      $stderr.puts("tc_enter=" + tc_enter.to_s);
      $stderr.puts("tc_leave=" + tc_leave.to_s);
      
      audio_frame_offset = calculate_frame_offset($vided_audio_fps, tc_enter);
      audio_frame_amount = calculate_frame_amount($vided_audio_fps, tc_ideal, tc_audio, tc_enter, tc_leave);
      
      video_frame_offset = calculate_frame_offset($vided_video_fps, tc_enter);
      video_frame_amount = calculate_frame_amount($vided_video_fps, tc_ideal, tc_video, tc_enter, tc_leave);
      
      $stderr.puts("audio_frame_offset=" + audio_frame_offset.to_s);
      $stderr.puts("audio_frame_amount=" + audio_frame_amount.to_s);
      $stderr.puts("video_frame_offset=" + video_frame_offset.to_s);
      $stderr.puts("video_frame_amount=" + video_frame_amount.to_s);
      
      tc_ideal += (tc_leave - tc_enter);
      tc_audio += (audio_frame_amount + 0.0) / ($vided_audio_fps + 0.0);
      tc_video += (video_frame_amount + 0.0) / ($vided_video_fps + 0.0);
      
      $stderr.puts("tc_ideal=" + tc_ideal.to_s);
      $stderr.puts("tc_audio=" + tc_audio.to_s);
      $stderr.puts("tc_video=" + tc_video.to_s);
      
      while ((audio_frame_amount > 0) || (video_frame_amount > 0))
        audio_frame_atonce = [ audio_frame_amount, [ 1, $vided_audio_fps.round ].max ].min;
        IO.copy_stream(io_audio_file, io_audio_fifo, audio_frame_atonce * $vided_audio_fsz, audio_frame_offset * $vided_audio_fsz);
        audio_frame_offset += audio_frame_atonce;
        audio_frame_amount -= audio_frame_atonce;
        tc_wrote += (audio_frame_atonce + 0.0) / ($vided_audio_fps + 0.0);
        
        video_frame_atonce = [ video_frame_amount, [ 1, $vided_video_fps.round ].max ].min;
        IO.copy_stream(io_video_file, io_video_fifo, video_frame_atonce * $vided_video_fsz, video_frame_offset * $vided_video_fsz);
        video_frame_offset += video_frame_atonce;
        video_frame_amount -= video_frame_atonce;
        tc_wrote += (video_frame_atonce + 0.0) / ($vided_video_fps + 0.0);
        
        elapsed = Time.now.to_f - operation_began;
        completion = ((tc_wrote + 0.0) / (2 * (tc_final + 0.0)));
        $stderr.puts("ETA " + timeconv((elapsed / completion) - elapsed) + " " + centconv(completion));
      end
      
      io_audio_file.close();
      io_video_file.close();
    }
    
    io_audio_fifo.close();
    io_video_fifo.close();
    
    while (!((File.exists?(".audio-" + dst_name + ".mp3.wok")) && (File.exists?(".video-" + dst_name + ".264.wok"))))
      sleep(1);
    end
    
    run("mkvmerge " + mkvmopts + " -o '.final-" + dst_name + ".mkv' '.audio-" + dst_name + ".mp3' --default-duration 0:" + $vided_video_fps.to_s + "fps '.video-" + dst_name + ".264' &> '.log-mkvm-" + dst_name + "'");
    run("mv '.final-" + dst_name + ".mkv' '" + dst_name + ".mkv'");
  end
  
  # here lie outdated notes on how to use mencoder:
  # mencoder -demuxer rawvideo -rawvideo fps=30000/1001:w=1920:h=1080:yv12 video.raw -audio-demuxer rawaudio -rawaudio channels=2:rate=48000:samplesize=2 -audiofile audio.raw -o out.avi -oac mp3lame -ovc x264 -vf scale=512:288
end

#!/usr/bin/ruby

$vided_video_wid = 1920;
$vided_video_hei = 1080;
$vided_video_fsz = 3 * $vided_video_wid * $vided_video_hei / 2; # yuv
$vided_video_fps = 30000.0 / 1001.0; # framerate is not exactly 30fps
$vided_audio_fsz = 2 * 2;   # 16-bit stereo
$vided_audio_fps = 48000.0; # 48kHz

require("/path/to/vided.rb");

import_streams("s", "your-camera-output-video-file.avi");
import_title_screen_generate("main", 1440, 1080, 96, 0, 275,
                             [
                              "The Main Thing",
                              "... etc, etc, etc ..."
                             ].join("\n"));

def mkopts_menc(w, h)
  return "crop=1440:1080:96:0,scale=" + w.to_s + ":" + h.to_s;
end

def mkopts_lame(quality)
  return "-s 48 -m j --bitwidth 16 --signed --little-endian -q 0 --lowpass -1 --highpass -1 --vbr-new -V " + quality.to_s;
end

def mkopts_x264_fast(quality)
  return "--crf " + quality.to_s + " --preset ultrafast --threads 5 --b-pyramid strict";
end

def mkopts_x264_best(quality)
  return "--crf " + quality.to_s + " --preset veryslow --threads 1 --b-pyramid strict";
end

chomp =
   [ [ "title-main", 0, 1 ] ] * 5 +
   [
    [ "s", 1021, 1043 ],
    [ "s", 1165, 1187 ]
   ];
export_chomp("output-main", mkopts_menc(720, 540), 720, 540, mkopts_lame(0), mkopts_x264_fast(15), "", chomp);

Sunday, January 9, 2011

[TECH] git encrypted backup script

Another free script! This one does encrypted space-efficient encrypted git backups from multiple repositories and working copies hosted on multiple machines to multiple destination directories on a single machine and/or an Amazon S3 bucket. It's called cmgeneral.sh, for "commit-master general." The details of operation are best understood by studying the script, but basically:
  • you need a modes/{modename} directory for each backup configuration. a backup configuration specifies what to back up from where and how to save the resulting archive. this directory must contain:
    • a blank file called .ddd1531eafda97524d852328d208469e, to guard against typos
    • a file called codename, that contains the name of the mode (usually the same as the directory name). this will be incorporated in the name of the final archive for identification.
    • a file called list, containing three words per line: a user name, a host name, and an absolute path leading to a git repository
    • a file called dest, containing, one-per-line, directories to save backup archives in. each such directory must contain a file called .ddd1531eafda97524d852328d208469e, to guard against typos
    • a soft link called j3cmd leading to the synchronize.sh script from an installation of jets3t (required for s3 backups only)
    • a file called j3env containing a bash expression to export JETS3T_HOME to point to the jets3t installation directory (required for s3 backups only)
    • a file called j3bin containing the name of an s3 bucket to save backup archives in (required for s3 backups only)
  • when invoking the script, specify the modename as a first argument, and optionally "s3" as a second argument to enable saving backup archives to s3.
  • the first time the script is invoked, it will walk you through generating an RSA keypair for encryption using gpg. since public-key cryptography is used, a passphrase is not needed for backup, only for recovery.
  • each archive contains the encryption keys as well as an embedded recovery script. the passphrase is required for recovery.
  • the backup does not save the full state of each repository. in particular, tags are not saved. it saves all commits, though, which is all I am personally interested in recovering. it is possible that I will improve the script to save tags as well; this seems to be somewhat nontrivial.
  • any questions? send me an email zerosum42 [AT] gmail [DOT] com
#!/bin/bash
# copyright (c) 2011 by andrei borac

ARG1="$1"
ARG2="$2"

sudo umount /tmp/cmg-ddd1531eafda97524d852328d208469e-*/.gnupg &> /dev/null
sudo umount /tmp/cmg-ddd1531eafda97524d852328d208469e-* &> /dev/null
sudo rmdir /tmp/cmg-ddd1531eafda97524d852328d208469e-* &> /dev/null

set -o errexit
set -o nounset
set -o pipefail

if [ ! -f modes/"$ARG1"/.ddd1531eafda97524d852328d208469e ]
then
  echo "sorry, invalid mode"
  exit 1
fi

XMODE="`readlink -f modes/"$ARG1"`"

STAMP="`date +%Yy%mm%dd`"
TMPWD=/tmp/cmg-ddd1531eafda97524d852328d208469e-"$STAMP"
mkdir -p "$TMPWD"
sudo mount -t tmpfs tmpfs "$TMPWD"
sudo chown "$USER":"$USER" "$TMPWD"
mkdir -p "$TMPWD"/.gnupg
sudo mount -t ramfs ramfs "$TMPWD"/.gnupg
sudo chown "$USER":"$USER" "$TMPWD"/.gnupg
chmod a-rwx,u+rwx "$TMPWD"/.gnupg
cd "$TMPWD"

if [ ! -f "$XMODE"/public.key ] || [ ! -f "$XMODE"/secret.key ]
then
  echo "when creating a keypair, use \"8, q, 4096, 1000y, y, cmgeneral,,, o, {pwd}\""
  echo -n "initialize keypair for this mode? (y/N) "
  read -s -n 1 YCHAR
  echo
  if [ "$YCHAR" == "y" ]
  then
    GPGOPT="--homedir .gnupg --keyring .gnupg/public.keyring --secret-keyring .gnupg/secret.keyring --no-default-keyring"
    gpg $GPGOPT --expert --gen-key
    gpg $GPGOPT --export            -a cmgeneral > "$XMODE"/public.key
    gpg $GPGOPT --export-secret-key -a cmgeneral > "$XMODE"/secret.key
  else
    exit 1
  fi
fi

if [ ! -f "$XMODE"/codename ]
then
  echo "echo codename > modes/$ARG1/codename"
  echo "the codename will suffix cmgeneral archives for easy identification"
  exit 1
fi

XCODE="`cat \"$XMODE\"/codename`"

XLIST="`cat \"$XMODE\"/list | tr '\t' ' ' | sed -e 's/^[ ]*//' -e 's/[ ]*$//' -e 's/[ ][ ]*/:/g' | tr '\n' '@' | sed -e 's/@@*/@/g' -e 's/^@*//g' -e 's/@*$//g'`""@"
echo "XLIST='$XLIST'"
XDEST="`cat \"$XMODE\"/dest | tr '\n' ' '`"
echo "XDEST='$XDEST'"

git init
XIDID=1000

while [ 1 ]
do
  XIDID=$((XIDID+1))
  
  XLINE="${XLIST%%@*}"
  XLIST="${XLIST#*@}"
  
  if [ "$XLINE" == "" ]
  then
    break
  fi
  
  XUSER="${XLINE%%:*}"
  XLINE="${XLINE#*:}"
  
  XHOST="${XLINE%%:*}"
  XLINE="${XLINE#*:}"
  
  XPATH="${XLINE%%:*}"
  XLINE="${XLINE#*:}"
  
  echo "XIDID='$XIDID'"
  echo "XUSER='$XUSER'"
  echo "XHOST='$XHOST'"
  echo "XPATH='$XPATH'"
  
  echo "$XIDID $XUSER $XHOST $XPATH" >> toc
  git remote add recover_"$XIDID" "$XUSER"@"$XHOST":"$XPATH"
  git fetch --no-tags recover_"$XIDID"
done

git fsck --full --strict
git repack -a -d -f --window=100 --depth=100 --window-memory=64m
git fsck --full --strict
git bundle create bundle.git --remotes
fakeroot tar -cf contents.tar toc bundle.git

cp "$XMODE"/public.key "$XMODE"/secret.key .
GPGOPT="--homedir .gnupg --keyring .gnupg/public.keyring --secret-keyring .gnupg/secret.keyring --no-default-keyring"
gpg $GPGOPT --import public.key
gpg $GPGOPT --import secret.key
gpg $GPGOPT --list-keys --with-colons --with-fingerprint | egrep fpr | sed -e 's/^fpr[:]*//g' | sed -e 's/$/6:/' | gpg $GPGOPT --import-ownertrust
gpg $GPGOPT --recipient cmgeneral --encrypt -z 0 < contents.tar > contents.tar.gpg
md5sum    contents.tar.gpg | egrep -o '^[0-9a-fA-F]{32}'  > contents.tar.gpg.md5sum
sha256sum contents.tar.gpg | egrep -o '^[0-9a-fA-F]{64}'  > contents.tar.gpg.sha256sum
sha512sum contents.tar.gpg | egrep -o '^[0-9a-fA-F]{128}' > contents.tar.gpg.sha512sum

cat > recover.sh << 'EOF'
#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

NONCE="`date +%Yy%mm%dd-%Hh%Mm%Ss-%ss-%Nn`"
mkdir recover-"$NONCE"
cd recover-"$NONCE"

mkdir .gnupg
sudo mount -t ramfs ramfs .gnupg
sudo chown "$USER":"$USER" .gnupg
chmod a-rwx,u+rwx .gnupg
GPGOPT="--homedir .gnupg --keyring .gnupg/public.keyring --secret-keyring .gnupg/secret.keyring --no-default-keyring"
gpg $GPGOPT --import ../public.key
gpg $GPGOPT --import ../secret.key
gpg $GPGOPT --list-keys --with-colons --with-fingerprint | egrep fpr | sed -e 's/^fpr[:]*//g' | sed -e 's/$/6:/' | gpg $GPGOPT --import-ownertrust
gpg $GPGOPT --decrypt < ../contents.tar.gpg > ../contents.tar
tar -C .. -xmf ../contents.tar

git init
git bundle unbundle ../bundle.git |\
while read SHA REF
do
  END="${REF#refs/remotes/}"
  git tag "${END//\//_}" "$SHA"
done

echo "use 'cd recover-*; git tag -l' to see what was recovered"
echo "remember, there is a table of contents in the 'toc' file"
echo "you won't see any files until you 'git merge' one of the recovered tags"
echo "good luck!"

sudo umount .gnupg
rmdir .gnupg
echo "+OK"
EOF
chmod a+x recover.sh

fakeroot tar -c recover.sh public.key secret.key contents.tar.gpg* > archive.tar

NONCE="`date +%Yy%mm%dd-%Hh%Mm%Ss-%ss-%Nn`"

for IDEST in $XDEST
do
  if [ ! -f "$IDEST"/.ddd1531eafda97524d852328d208469e ]
  then
    echo "sorry, magic file not found for destination '$IDEST', skipping"
  else
    cp archive.tar "$IDEST"/cmgeneral-"$NONCE"-"$XCODE".tar
  fi
done

if [ "$ARG2" == "s3" ] && [ -x "$XMODE"/s3cmd ] && [ -f "$XMODE"/s3cfg ] && [ -f "$XMODE"/s3bin ]
then
  mv archive.tar cmgeneral-"$NONCE"-"$XCODE".tar
  . "$XMODE"/j3env
  "$XMODE"/j3cmd --properties "$XMODE"/j3cfg UP --nodelete "`cat \"$XMODE\"/j3bin`" cmgeneral-*
fi

cd /
sudo umount "$TMPWD"/.gnupg
sudo umount "$TMPWD"
echo "+OK"

Wednesday, January 5, 2011

[TECH] java synchronization with junctions

The recommended synchronization mechanism in Java is, of course, the synchronized keyword. I have nothing against it, except that involves a lot of typing: first you have to create a new class to encapsulate the state that needs to be protected, then you have to write a bunch of accessor methods. Used correctly, a few volatile variables here and there offer a lightweight solution. The only problem is you can't easily block on a condition that is based on volatile variables. To remedy the situation, I've just defined and implemented the "junction" abstraction.

Here's the idea: A junction protects one or more volatile variables (usually just one). Where the variables reside is not known to the junction object (the junction object never accesses the variables). A thread that modifies any of the protected volatile variables must follow the modifications with a call to the junction's changed() method. Whereupon all threads waiting for a condition are woken up to test the condition. An example should make things clearer:
/***
 * Example.java
 * copyright (c) 2011 by andrei borac
 ***/

import java.io.*;

import zs42.junction.*;

public class Example
{
  static final int MAX = 1024;
  
  static Junction     stringj = new BlockingJunction();
  static volatile int stringc = 0;
  static String[]     strings = new String[MAX];
  
  public static void main(String[] args)
  {
    (new Thread()
      {
        public void run()
        {
          Junction.Waiter waiter = stringj.waiter();
          
          int stringi = 0;
          
          while (true) {
            // wait for a new string
            while (waiter.waitfor(stringc > stringi));
            String current = strings[stringi++];
            
            // null string indicates done
            if (current == null) break;
            
            // print the string
            System.out.println(current);
          }
        }
      }).start();
    
    (new Thread()
      {
        void pause()
        {
          try {
            Thread.currentThread().sleep(500);
          } catch (InterruptedException e) {
            // ignored
          }
        }
        
        void append(String[] currents)
        {
          // append strings, making sure to update the volatile just once, at the end
          {
            int stringi = stringc;
            
            for (String current : currents) {
              strings[stringi++] = current;
            }
            
            stringc = stringi;
          }
          
          // notify the junction that volatiles have been changed
          stringj.changed();
        }
        
        public void run()
        {
          append(new String[] { "hello", "world" });
          pause();
          append(new String[] { "how", "are", "you" });
          pause();
          append(new String[] { "this", "is", "the", "last", "installment", null });
        }
      }).start();
  }
}

Notice the simplicity of the "blocking" loop:
while (waiter.waitfor(stringc > stringi));
While it appears to be a spinning, it does actually block in waitfor. Also the waitfor call returns true each time it needs to test the condition. That way it doesn't need to know which volatile variables are involved in the condition and there is no need to express the condition to be tested in a way in which it could be evaluated in the body of waitfor.

Besides BlockingJunction, there are also SpinningJunction (keep testing the condition in a loop) and YieldingJunction (yield the processor but test once each time the current thread is scheduled). These might be useful in cases where latency is critical.

The code for the junction primitives is available on the Downloads page of my main website (www.zerosum42.com), and may be redistributed under the terms of the "zlib license."

One last thing: It is possible one or more readers will go completely ape at the idea of synchronizing using volatile. It really does work and is really guaranteed to work according to the new Java memory model. Of course, it is somewhat easier to get things wrong with volatile, so do study the memory model in detail before attempting this.