Wednesday, April 11, 2012

[TECH] circumventing EULA reverse engineering clauses

Background: Many EULAs (End-User License Agreements) of commercial software products attempt to prohibit reverse engineering of the software.

Question: Apart from complaining, what can be done about this?

First of all, a clause prohibiting "reverse engineering" is inherently vague, because reverse engineering means different things to different people. I believe that the term "reverse engineering", in its most usual sense, refers to the process of figuring out how a device or a piece software works by examining a specimen. Note that this is distinct from the process of "cloning" a design. One might produce an exact copy of a design without understanding how it works (cloning without reverse engineering), or one might attempt to understand how it works without producing an exact or even an approximate copy of a design (reverse engineering without cloning).

Normally, reverse-engineering is legal under U.S. law and under the law of most civilized countries. In its broad sense of attempting to understand a mechanism made by someone else, reverse engineering is not even unethical. A gray area is reached only when reverse engineering efforts are specifically directed at producing an exact or approximate replica (which may be protected under copyright laws). Of course, ideas that have been learned through reverse engineering may be subject to the protection of in-force patents, so there is no guarantee that ideas learned through reverse engineering may be used without encumbrance. Nevertheless, the act of reverse engineering itself is usually legitimate.

Of course, big software doesn't like this state of affairs. While big software does file patents, it's an expensive procedure since they have to pay a patent attorney to prepare the application. Frequently, the patent examiner is able to find prior art that invalidates the application. The risks are high since the chances of originality are low. Instead, lawyers sold them the idea that an anti-reverse-engineering clause in their software license would give them a "virtual patent" -- nobody would be allowed to grab hold of their ideas, and they would not have to prepare a patent application or subject their work to legal tests of originality.

But how does big software get the world at large to accept the terms of the software license? Usually, two parties can't enter into a contract without an exchange of value from both sides. This is known as "consideration". If you walk into a software shop and buy a boxed software product for cash, a basis for a contract is formed because you have provided the vendor with money and the vendor has provided you with an installation CD. This is a contract for the sale of the software. Under my personal interpretation of contract law, the subsequent "clickwrap" agreement (EULA) that appears when you attempt to install the software is not a legally binding contract because there is no further "consideration" -- you do not give additional money for the privilege of -using- the copy of the software you now own the installation CD for and the software vendor is not giving you anything you are not already entitled to.

While judges have considered interpretations of contract law similar to mine, they have ultimately decided on a different interpretation -- the clickwrap agreement "amends" or "supplements" or "clarifies" the original contract for the sale of the software (which met the legal condition of consideration, because you paid and they delivered). I consider this to be an example of judicial error. In the long run, I believe this mistake will be corrected, and clickwrap agreements formed after the sale of the software will be found to lack consideration. Until then, I can see a very simple workaround -- have a third party purchase and install the software onto a laptop computer (including pressing "I Accept"), and then purchase the laptop computer from the third party. In this arrangement, no contract between yourself and the vendor can possibly exist, since there is no consideration from either party to the other. You may then tinker and reverse engineer to your heart's content.

In a business setting, the third party must be kept at arm's length, in a contract that does not allow him/her to obligate the business to the EULA or any other legal agreement. For such a task, an employee would be a poor choice and a business partner would be a disastrous choice.

Courts frequently prioritize objective factors (the "I Agree" button was pressed) over subjective factors (the user didn't genuinely wish to enter the agreement in the EULA since it served purely to limit their options without any benefit to them). Therefore, however silly it may seem, you are in much better shape if you do not push the "I Agree" button yourself.

This workaround may seem to contradict the premise of the software license, which usually states that "by using this software you agree ...". Legally, however, it's a non-sequitur that you must agree to the vendor's terms simply because you use the software, even if the vendor has stated such a requirement. By continuing to read this page beyond this sentence, you agree to pay the author $10,000 USD. Will that be cheque or cash?

Obviously, in a free country, you have the right to do anything not explicitly disallowed by law. The law does not disallow you from using a vendor's software -without- accepting the vendor's terms, in the same way that the law does not disallow you from reading a sentence without obeying an imperative. It's just a matter of making a clear case for the stance that you did not ever accept the EULA. The vendor will try to push for the interpretation that you initially accepted the EULA but then decided to act against its provisions, but this is not an insurmountable obstacle. It should be noted that the state of Louisiana has a Louisiana Software License Enforcement Act that specifically forces users, through mere use of software, to bind to a subset of specifically enumerated clauses of a vendor's software license terms. For a time, this included anti-reverse-engineering clauses (until part of the act was invalidated because federal law stipulates that states may not create additional copyright protections).

In this vein, it should be noted that the concept of a software license is rather bogus itself. In most states, you don't need a license from an author to merely -use- a piece of software he/she wrote. You only need a license to -copy- the software, and only because the software is protected by copyright law, not because the author says so. In most states, the only case in which you need a license merely to -use- a piece of software is when the underlying technology is protected by a patent. In this case, you need a license from the owner of the patent, not the author of the software embodying it (though the vendor may hold both the copyright and the patent). In any case, such a license is not a "software" license, but a patent license.

Even so, you are not forced to enter an agreement with the vendor simply because use of the software would otherwise infringe a patent owned by the vendor. You have the sensible option of using the software without a patent license and waiting for the vendor to sue you for patent infringement. If you do not redistribute the software, or a clone of it, this will virtually never happen. Besides, the damages would not amount to much, as you are not commercially exploiting the invention, you are only employing it to the extent of studying a single instance of it. It would be difficult to justify a "reasonable royalty" greater than the price of the software and there would be no provable loss of revenue to the vendor. In fact, if your copy of the software was originally bought from the vendor, should find a strong defense from patent infringement claims from the vendor under the Exhaustion Doctrine, though not necessarily from patent claims brought by parties other than the vendor.

A noteworthy snag to the above reasoning that "mere use of software does not require a license" is that intermediate copies may be generated when a computer program is run. Lawyers have split hairs over the extent to which the generation of these intermediate copies may be a form of copyright infringement. Fortunately, courts have found intermediate copies produced during a reverse-engineering effort to be "fair use" when there are no other ways to obtain otherwise unprotected technical information. However, the DMCA introduces a ban on reverse engineering for the purpose of defeating a software protection mechanism. Again, a basic premise of legitimate reverse engineering is that the engineer is merely trying learn the techniques and technologies utilized by the software. Defeating a protection mechanism is a highly dubious motivation; it is not surprising that courts have held against it.

Summary: Reverse engineering is still legitimate even if the software ships with an EULA. It is a matter of jumping through hoops to avoid obligating yourself under the EULA. The pioneers of reverse engineering EULA-protected software have generally been burned -- partly through their own ignorance and partly through (in my view) unreasonable judicial interpretation. While legal scholars are content to document the cases that arise, few suggest workarounds. EULAs are an advance in the field of legal engineering, but they are still gimmicks and cannot possibly be equivalent to the "real" intellectual property protection that copyrights and patents afford. Accordingly, it is plausible that a simple workaround, such as purchasing pre-installed software from a third party, can defeat an EULA.

Warnings: I am not a lawyer. If you are contemplating reverse engineering, you should definitely consult a real lawyer and give him/her the full details of your reverse engineering project. In complete contradiction of common sense, anti-reverse-engineering clauses present in EULAs have been held legally binding by courts, notwithstanding the fact that such clauses harm the public interest. Definitely consult with a lawyer as early as possible in the process. Nothing in this document should be construed as being legal advice; it is written exclusively for the purposes of entertainment.

Tuesday, January 17, 2012

[LIFE] Thoughts on Wikipedia Global Blackout

Imagine life without Wikipedia. Imagine you're wondering how a combination lock works and because there's no Wikipedia, you can't find out. Knowledge is completely blocked. You have no way to get this information. Absolutely none. Imagine a college student writing a term paper that requires citing sources, but he/she can't find any. Knowledge is completely blocked. He/she has no way to find sources to cite. Absolutely none. Imagine you're idly wondering how many different types of cows there are and what they are all called. And you wouldn't be able to find out. Knowledge would be completely blocked. You'd have to find something else to do.

I'm not against Wikipedia or anything; I think it's a great project. I'm just a little concerned that while it can be a helpful tool to find information quickly, it can also become a crutch if people become accustomed that -all- the information they obtain about anything is going to come from Wikipedia. It's also remarkably easy to waste hours wading through information you don't really need in order to avoid upcoming tasks -- there's little doubt that Wikipedia fosters procrastination. So I think the blackout is a good idea, and we should use this opportunity to remind ourselves that knowledge in pill form is not automatically power.

Thursday, November 17, 2011

[LIFE] your guide to the 99%

Unless you've been living under a rock (like I do), you must have heard about the "Occupy" protests. You might consequently be wondering what the protests are about. You may even have googled the topic. Unsuccessfully, of course, because nobody knows what the protesters are protesting about. They're basically a group of self-proclaimed "average Joe" types, as evidenced by their refrain "we are the 99%".

Along with a long list of other demands, they want rich people (the 1%) to be taxed more. I am against this proposal, not because I'm in the 1% (I probably earn vastly less than the average protester), but for the simple reason that it is economically absurd. After all, the 1% can simply move to wherever they are taxed less. The U.S. is not the only place where business is being conducted. A country can efficiently tax only those individuals who are relatively inflexibly stationed in it. In other words, the 99%.

For the most part, the protests are simply about disgruntled individuals broadcasting the fact that they are unhappy. Now, I'm not saying it's OK for people to be unhappy, but they should consider that perhaps they are partly responsible for their situation. Sure, banks extended them adjustable-rate mortgages they could never pay after the "adjustment". But they signed off on it and borrowed the money, cognizant of the risks. Repeated attempts to create a "safe" economy where everyone would be happy have failed in the past -- they simply made everyone safe and miserable. Until they collapsed, making everyone very unsafe and consequently highly miserable.

My suggestion is simple: until we have a workable replacement for capitalism, let's not protest capitalism. Recent hardship may not be the fault of capitalism itself. It could very well be the combined effects of diminishing natural resources and reduced quality of human capital. No system can defeat the GIGO principle: garbage in, garbage out.

So, what are protesters shouting into society's collective ear these days? Do they have a plan for a replacing the current economic system with one that produces happiness from garbage? What kind of change are they trying to effect by "occupying" the system? That's right, "N/A"! They don't actually have a plan -- they just know that they're unhappy and the system is responsible for making them happy. Protesters seemingly expect the system to figure out what's wrong with itself and replace itself with something better. All on it's own. Unfortunately, few systems can diagnose themselves and nothing likes to be replaced.

Incidentally, while we don't fully understand what the protesters are trying to do, we know some of the things what the protesters are not trying to do -- they're not trying to better organize their lives, they're not trying to learn new skills, they're not trying to find jobs, etc. We know they're not trying to do any of these things because any of these things would easily take up all of the time that they spend protesting.

Saturday, October 22, 2011

[TECH] fixing the tab key in VNC

Setting up a simple VNC server such as TightVNC (tightvncserver) has gotten a lot harder in recent Ubuntu distributions, because the tab key seems not to work properly! I find that using the command line is nearly impossible when the tab key doesn't work in VNC. The problem seems to depend on the window manager in use (I use xfce4). After some hunting, it became apparent that the default window manager key bindings global binds the "<Super>"-Tab combination to a feature known as "switching windows within the same application." I don't know what a "Super" key is and apparently neither does the system, because it always treats the Tab key as "<Super>"-Tab, triggering this feature rather than passing the Tab key event through to the application. Clearing this key from xfce4-settings-manager => window manager => keyboard fixes the problem, as does the following command line equivalent:
xfconf-query -c xfce4-keyboard-shortcuts -p /xfwm4/custom/'<'Super'>'Tab -r
With the default settings getting less sane every day, it's not clear where Ubuntu is headed. The whole point of Ubuntu was to provide sensible defaults and clear away pitfalls like these. Most annoyingly, this used to work -- only a few versions back it was possible to simply "apt-get -y install tightvncserver" and run tightvncserver on the command line and get up and running in minutes rather than hours.

Wednesday, August 3, 2011

[LIFE] top 10 ways to avoid reading top 10 tip lists

  • when you encounter a top 10 tip list purporting to improve the way in which you do X, consider whether X is something you want to be able to do efficiently. if it isn't, simply stop reading.
  • consider that the effort you put into reading a top 10 tip list only pays off if you actually find yourself in situation X often enough that you need an efficient way to deal with it.
  • remember, you can stop reading top 10 tip lists after the first few tips if you think those tips pretty much cover all the bases. top 10 tip lists are usually arranged from the most helpful tip to the least helpful tip.
  • if you still find yourself reading too many top 10 tip lists, perhaps you are trying for too much efficiency in too many aspects of your life. most people get by fine without reading all that many top 10 tip lists.
  • sometimes it helps to give yourself a gentle reminder that there are more important tasks in your life that may be in more urgent need of your attention than reading top 10 tip lists.
  • keep in mind that you can always stop reading. reading is a voluntary activity and a choice.
  • your (clearly compulsive) reading of top 10 tip lists may be stemming from an anxiety or dormant fear of being placed in a situation where you must rapidly master a complex, unfamiliar task. unless you are James Bond, you may find reassurance in contemplating the rarity of such situations.
  • it may help to imagine yourself in a scenario in which reading a top 10 tip list causes an adverse outcome. for example, imagine yourself being fired after reading a top 10 tip list on how to demand a higher wage.
  • if all else fails, seek professional help for your top 10 tip list reading addiction.
  • to be honest, I'd much rather you hadn't read this far. clearly your compulsion is out of control. it may help to read each tip over again from the beginning and ask yourself whether you have sincerely tried to apply it with an open mind.

Wednesday, May 11, 2011

[TECH] nonlinear video editing with ruby

I recently had to do some video editing, and I ran into a spot of difficulty: I couldn't find a simple script-based system. Everything seems to be bundled with a complex, cumbersome graphical user interface. So I wrote a ruby script. It's meant to be included via "require" from a custom script containing configuration data and EDLs. It works in two stages: in the first stage, source material is expanded to uncompressed PCM and 4:2:0 YUV (typically generating huge video files). Title page text can also serve as source material (it is rendered using ImageMagick's "convert" tool). In the second stage, audio and video encoders are launched with their input files connected to FIFOs and the ruby script feeds these FIFOs with portions of the raw files according to an edit decision list. At the end of the second stage, the encoded audio and video files are muxed using mkvmerge. The whole thing isn't very user-friendly, but it does what little I needed done and it's a good starting point for anyone who needs to add special-purpose functionality.

The recommended way to use it is to edit one section at a time. That is, to "test" a section of video X...Y use an EDL like Y-3...Y , X...X+3. This allows you to adjust X and Y to get the cut points just right for the sequence X...Y and only requires you to encode 6 seconds of video in one go (with --preset ultrafast, it takes on the order of 9 seconds to see the result on a setup where the raw files are stored on very slow external storage). After testing each section, you can put them together into one large EDL and encode them with more aggressive compression settings. The script prints a progress line after each second of video encoded.

Hints:
  • Source video that has been split into multiple files by your video camera can usually be concatenated with "cat" and the result will work fine for importing.
  • 30fps is rarely 30fps. it is usually 30000/1001fps = 29.(970029)fps.
  • The frame size for 4:2:0 YUV is always 3 * width * height / 2.
  • The frame size for audio depends on the number of channels and the bit depth. 2 * 2 is a very good guess.
  • Video codec junkies will tell you that tweaking the video codec is essential for good quality. They are right. What they typically don't tell you is that the tweaking has already been done by the codec author. Use the --preset option.

Here follows the script (it's called vided.rb), and an example usage script (typically called go.rb).

#!/bin/false
# vided.rb
# copyright (c) 2011 by andrei borac

# please define the following (values are examples)
#  $vided_video_wid = 1920;
#  $vided_video_hei = 1080;
#  $vided_video_fsz = 3 * $vided_video_wid * $vided_video_hei / 2;
#  $vided_video_fps = 30000.0 / 1001.0;
#  $vided_audio_fsz = 2 * 2;
#  $vided_audio_fps = 48000.0;

raise("\$video_vided_wid undefined") if (!defined?($vided_video_wid));
raise("\$video_vided_hei undefined") if (!defined?($vided_video_hei));
raise("\$video_video_fsz undefined") if (!defined?($vided_video_fsz));
raise("\$vided_video_fps undefined") if (!defined?($vided_video_fps));
raise("\$video_audio_fsz undefined") if (!defined?($vided_audio_fsz));
raise("\$vided_audio_fps undefined") if (!defined?($vided_audio_fps));

def centconv(f)
  f = (10000.0 * f).round;
  f = 0 if (f < 0);
  f = 9999 if (f > 9999);
  
  n = f / 100;
  d = f % 100;
  
  n = n.to_s; n = "0" + n while (n.length < 2);
  d = d.to_s; d = "0" + d while (d.length < 2);
  
  return n.to_s + "." + d + "%";
end

def timeconv(s)
  s = s.round; # forget about fractional seconds
  s = 0 if (s < 0);
  
  m = ((s + 0.0) / 60).to_i; s -= 60 * m;
  h = ((m + 0.0) / 60).to_i; m -= 60 * h;
  d = ((h + 0.0) / 24).to_i; h -= 24 * d;
  
  s = s.to_s; s = "0" + s while (s.length < 2);
  m = m.to_s; m = "0" + m while (m.length < 2);
  h = h.to_s; h = "0" + h while (h.length < 2);
  d = d.to_s; d = "0" + d while (d.length < 3);
  
  return d.to_s + "d" + h.to_s + "h" + m.to_s + "m" + s.to_s + "s";
end

def run(command)
  $stderr.puts("^" + command + "$");
  
  if (!system("bash", "-c", "set -o errexit; set -o nounset; set -o pipefail; " + command))
    $stderr.puts("failed command ^" + command + "\$");
    exit(1);
  end
end

def import_generic(name, from, type)
  if (!File.exists?(type + "-" + name + ".raw.wok"))
    run("echo 0 5 0 > '.chomp." + name + ".edl'");
    run("mencoder -demuxer lavf -ignore-start '" + from + "' -of raw" + type + " -o '" + type + "-" + name + ".raw' -hr-edl-seek -edl '.chomp." + name + ".edl' -oac pcm -ovc raw -vf format=i420");
    run("touch '" + type + "-" + name + ".raw.wok'");
  end
end

def import_audio(name, from)
  import_generic(name, from, "audio");
end

def import_video(name, from)
  import_generic(name, from, "video");
end

def import_streams(name, from)
  import_audio(name, from);
  import_video(name, from);
end

def import_title_screen(name, from)
  if (!File.exists?("audio-" + name + ".raw.wok"))
    run("dd if=/dev/zero of='audio-" + name + ".raw' bs=1M count=1");
    run("touch 'audio-" + name + ".raw.wok'");
  end
  
  if (!File.exists?("video-" + name + ".raw.wok"))
    run("mencoder -of rawvideo -ovc raw -vf format=i420 -o 'video-" + name + ".420' mf://'" + from + "'");
    run("( for i in `seq 1 100`; do cat 'video-" + name + ".420'; done ) > 'video-" + name + ".raw'");
    run("touch 'video-" + name + ".raw.wok'");
  end
end

# w, h - size of title image to write text on
# offL - the left crop amount (image to be expanded this much leftward)
# offT - the top crop amount (image to be expanded this much upward)
# image is automatically expended rightward and downward to fill global widxhei
def import_title_screen_generate(name, w, h, offL, offT, d, text)
  if (!File.exists?("video-title-" + name + ".raw.wok"))
    run("convert " +
        "-size " + w.to_s + "x" + h.to_s + " " +
        "-background black " +
        "-fill white " +
        "-font Palatino-Roman " +
        "-density " + d.to_s + " " +
        "-pointsize 16 " +
        "-interline-spacing 13 " +
        "-gravity center " +
        "label:'" + text + "' " +
        "-background white " +
        "-gravity southeast " +
        "-extent " + (w + offL).to_s + "x" + (h + offT).to_s + " " +
        "-gravity northwest " +
        "-extent " + $vided_video_wid.to_s + "x" + $vided_video_hei.to_s + " " +
        "title-'" + name + "'.tga");
    import_title_screen("title-" + name, "title-" + name + ".tga");
  end
end

def above_zero(x)
  if (x > 0)
    return x;
  else
    return 0;
  end
end

def calculate_frame_offset(fps, tc_enter)
  fps += 0.0; tc_enter += 0.0;
  return above_zero((tc_enter * fps).round);
end

def calculate_frame_amount(fps, tc_ideal, tc_track, tc_enter, tc_leave)
  fps += 0.0; tc_ideal += 0.0; tc_track += 0.0; tc_enter += 0.0; tc_leave += 0.0;
  return above_zero(((tc_track - tc_ideal + tc_leave - tc_enter) * fps).round);
end

def reset_fifo(name)
  run("rm -f '" + name + "'; mkfifo '" + name + "'");
end

def export_chomp(dst_name, mencopts, local_video_wid, local_video_hei, lameopts, x264opts, mkvmopts, sections)
  if (!File.exists?(dst_name + ".mkv"))
    [ ".audio-", ".video-", ".final-" ].each { |prefix|
      [ ".raw", ".mp3", ".mp3.wok", ".264", ".264.wok", ".mkv" ].each { |suffix|
        run("rm -f '" + prefix + dst_name + suffix + "'");
      }
    }
    
    reset_fifo(".video-" + dst_name + ".raw");
    #reset_fifo(".audio-" + dst_name + ".mp3");
    #reset_fifo(".video-" + dst_name + ".264");
    
    audio_fifo_name = ".audio-" + dst_name + ".raw";
    video_fifo_name = ".video-" + dst_name + ".pre";
    
    reset_fifo(audio_fifo_name);
    reset_fifo(video_fifo_name);
    
    #run("( mkvmerge " + mkvmopts + " -o '.final-" + dst_name + ".mkv' '.audio-" + dst_name + ".mp3' --default-duration 0:" + $vided_video_fps.to_s + "fps '.video-" + dst_name + ".264' ; touch '.final-" + dst_name + ".mkv.wok' ) &> '.log-mkvm-" + dst_name + "' &");
    run("( lame -r " + lameopts + " '.audio-" + dst_name + ".raw' '.audio-" + dst_name + ".mp3' ; touch '.audio-" + dst_name + ".mp3.wok' ) &> '.log-lame-" + dst_name + "' &");
    run("( x264 " + x264opts + " -o '.video-" + dst_name + ".264' '.video-" + dst_name + ".raw' " + local_video_wid.to_s + "x" + local_video_hei.to_s + " ; touch '.video-" + dst_name + ".264.wok' ) &> '.log-x264-" + dst_name + "' &");
    run("( mencoder -demuxer rawvideo -rawvideo i420:w=" + $vided_video_wid.to_s + ":h=" + $vided_video_hei.to_s + " '.video-" + dst_name + ".pre' -of rawvideo -o '.video-" + dst_name + ".raw' -nosound -ovc raw -vf " + ((mencopts.length() > 0) ? (mencopts + ",") : ("")) + "format=i420 ) &> '.log-menc-" + dst_name + "' &");
    
    io_audio_fifo = IO.new(IO.sysopen(audio_fifo_name, "wb"), "wb");
    io_video_fifo = IO.new(IO.sysopen(video_fifo_name, "wb"), "wb");
    
    schedule = [];
    
    sections.each { |src_name, tc_enter, tc_leave|
      schedule << \
      [
       IO.new(IO.sysopen("audio-" + src_name + ".raw", "rb"), "rb"),
       IO.new(IO.sysopen("video-" + src_name + ".raw", "rb"), "rb"),
       tc_enter,
       tc_leave
      ]
    }
    
    tc_final = 0.0;
    
    sections.each { |src_name, tc_enter, tc_leave|
      raise("backward section") if (tc_leave <= tc_enter);
      tc_final += (tc_leave - tc_enter);
    }
    
    tc_ideal = 0;
    tc_audio = 0;
    tc_video = 0;
    tc_wrote = 0;
    
    operation_began = Time.now.to_f;
    
    schedule.each { |io_audio_file, io_video_file, tc_enter, tc_leave|
      tc_enter += 0.0; tc_leave += 0.0;
      
      $stderr.puts("tc_enter=" + tc_enter.to_s);
      $stderr.puts("tc_leave=" + tc_leave.to_s);
      
      audio_frame_offset = calculate_frame_offset($vided_audio_fps, tc_enter);
      audio_frame_amount = calculate_frame_amount($vided_audio_fps, tc_ideal, tc_audio, tc_enter, tc_leave);
      
      video_frame_offset = calculate_frame_offset($vided_video_fps, tc_enter);
      video_frame_amount = calculate_frame_amount($vided_video_fps, tc_ideal, tc_video, tc_enter, tc_leave);
      
      $stderr.puts("audio_frame_offset=" + audio_frame_offset.to_s);
      $stderr.puts("audio_frame_amount=" + audio_frame_amount.to_s);
      $stderr.puts("video_frame_offset=" + video_frame_offset.to_s);
      $stderr.puts("video_frame_amount=" + video_frame_amount.to_s);
      
      tc_ideal += (tc_leave - tc_enter);
      tc_audio += (audio_frame_amount + 0.0) / ($vided_audio_fps + 0.0);
      tc_video += (video_frame_amount + 0.0) / ($vided_video_fps + 0.0);
      
      $stderr.puts("tc_ideal=" + tc_ideal.to_s);
      $stderr.puts("tc_audio=" + tc_audio.to_s);
      $stderr.puts("tc_video=" + tc_video.to_s);
      
      while ((audio_frame_amount > 0) || (video_frame_amount > 0))
        audio_frame_atonce = [ audio_frame_amount, [ 1, $vided_audio_fps.round ].max ].min;
        IO.copy_stream(io_audio_file, io_audio_fifo, audio_frame_atonce * $vided_audio_fsz, audio_frame_offset * $vided_audio_fsz);
        audio_frame_offset += audio_frame_atonce;
        audio_frame_amount -= audio_frame_atonce;
        tc_wrote += (audio_frame_atonce + 0.0) / ($vided_audio_fps + 0.0);
        
        video_frame_atonce = [ video_frame_amount, [ 1, $vided_video_fps.round ].max ].min;
        IO.copy_stream(io_video_file, io_video_fifo, video_frame_atonce * $vided_video_fsz, video_frame_offset * $vided_video_fsz);
        video_frame_offset += video_frame_atonce;
        video_frame_amount -= video_frame_atonce;
        tc_wrote += (video_frame_atonce + 0.0) / ($vided_video_fps + 0.0);
        
        elapsed = Time.now.to_f - operation_began;
        completion = ((tc_wrote + 0.0) / (2 * (tc_final + 0.0)));
        $stderr.puts("ETA " + timeconv((elapsed / completion) - elapsed) + " " + centconv(completion));
      end
      
      io_audio_file.close();
      io_video_file.close();
    }
    
    io_audio_fifo.close();
    io_video_fifo.close();
    
    while (!((File.exists?(".audio-" + dst_name + ".mp3.wok")) && (File.exists?(".video-" + dst_name + ".264.wok"))))
      sleep(1);
    end
    
    run("mkvmerge " + mkvmopts + " -o '.final-" + dst_name + ".mkv' '.audio-" + dst_name + ".mp3' --default-duration 0:" + $vided_video_fps.to_s + "fps '.video-" + dst_name + ".264' &> '.log-mkvm-" + dst_name + "'");
    run("mv '.final-" + dst_name + ".mkv' '" + dst_name + ".mkv'");
  end
  
  # here lie outdated notes on how to use mencoder:
  # mencoder -demuxer rawvideo -rawvideo fps=30000/1001:w=1920:h=1080:yv12 video.raw -audio-demuxer rawaudio -rawaudio channels=2:rate=48000:samplesize=2 -audiofile audio.raw -o out.avi -oac mp3lame -ovc x264 -vf scale=512:288
end

#!/usr/bin/ruby

$vided_video_wid = 1920;
$vided_video_hei = 1080;
$vided_video_fsz = 3 * $vided_video_wid * $vided_video_hei / 2; # yuv
$vided_video_fps = 30000.0 / 1001.0; # framerate is not exactly 30fps
$vided_audio_fsz = 2 * 2;   # 16-bit stereo
$vided_audio_fps = 48000.0; # 48kHz

require("/path/to/vided.rb");

import_streams("s", "your-camera-output-video-file.avi");
import_title_screen_generate("main", 1440, 1080, 96, 0, 275,
                             [
                              "The Main Thing",
                              "... etc, etc, etc ..."
                             ].join("\n"));

def mkopts_menc(w, h)
  return "crop=1440:1080:96:0,scale=" + w.to_s + ":" + h.to_s;
end

def mkopts_lame(quality)
  return "-s 48 -m j --bitwidth 16 --signed --little-endian -q 0 --lowpass -1 --highpass -1 --vbr-new -V " + quality.to_s;
end

def mkopts_x264_fast(quality)
  return "--crf " + quality.to_s + " --preset ultrafast --threads 5 --b-pyramid strict";
end

def mkopts_x264_best(quality)
  return "--crf " + quality.to_s + " --preset veryslow --threads 1 --b-pyramid strict";
end

chomp =
   [ [ "title-main", 0, 1 ] ] * 5 +
   [
    [ "s", 1021, 1043 ],
    [ "s", 1165, 1187 ]
   ];
export_chomp("output-main", mkopts_menc(720, 540), 720, 540, mkopts_lame(0), mkopts_x264_fast(15), "", chomp);

Sunday, January 9, 2011

[TECH] git encrypted backup script

Another free script! This one does encrypted space-efficient encrypted git backups from multiple repositories and working copies hosted on multiple machines to multiple destination directories on a single machine and/or an Amazon S3 bucket. It's called cmgeneral.sh, for "commit-master general." The details of operation are best understood by studying the script, but basically:
  • you need a modes/{modename} directory for each backup configuration. a backup configuration specifies what to back up from where and how to save the resulting archive. this directory must contain:
    • a blank file called .ddd1531eafda97524d852328d208469e, to guard against typos
    • a file called codename, that contains the name of the mode (usually the same as the directory name). this will be incorporated in the name of the final archive for identification.
    • a file called list, containing three words per line: a user name, a host name, and an absolute path leading to a git repository
    • a file called dest, containing, one-per-line, directories to save backup archives in. each such directory must contain a file called .ddd1531eafda97524d852328d208469e, to guard against typos
    • a soft link called j3cmd leading to the synchronize.sh script from an installation of jets3t (required for s3 backups only)
    • a file called j3env containing a bash expression to export JETS3T_HOME to point to the jets3t installation directory (required for s3 backups only)
    • a file called j3bin containing the name of an s3 bucket to save backup archives in (required for s3 backups only)
  • when invoking the script, specify the modename as a first argument, and optionally "s3" as a second argument to enable saving backup archives to s3.
  • the first time the script is invoked, it will walk you through generating an RSA keypair for encryption using gpg. since public-key cryptography is used, a passphrase is not needed for backup, only for recovery.
  • each archive contains the encryption keys as well as an embedded recovery script. the passphrase is required for recovery.
  • the backup does not save the full state of each repository. in particular, tags are not saved. it saves all commits, though, which is all I am personally interested in recovering. it is possible that I will improve the script to save tags as well; this seems to be somewhat nontrivial.
  • any questions? send me an email zerosum42 [AT] gmail [DOT] com
#!/bin/bash
# copyright (c) 2011 by andrei borac

ARG1="$1"
ARG2="$2"

sudo umount /tmp/cmg-ddd1531eafda97524d852328d208469e-*/.gnupg &> /dev/null
sudo umount /tmp/cmg-ddd1531eafda97524d852328d208469e-* &> /dev/null
sudo rmdir /tmp/cmg-ddd1531eafda97524d852328d208469e-* &> /dev/null

set -o errexit
set -o nounset
set -o pipefail

if [ ! -f modes/"$ARG1"/.ddd1531eafda97524d852328d208469e ]
then
  echo "sorry, invalid mode"
  exit 1
fi

XMODE="`readlink -f modes/"$ARG1"`"

STAMP="`date +%Yy%mm%dd`"
TMPWD=/tmp/cmg-ddd1531eafda97524d852328d208469e-"$STAMP"
mkdir -p "$TMPWD"
sudo mount -t tmpfs tmpfs "$TMPWD"
sudo chown "$USER":"$USER" "$TMPWD"
mkdir -p "$TMPWD"/.gnupg
sudo mount -t ramfs ramfs "$TMPWD"/.gnupg
sudo chown "$USER":"$USER" "$TMPWD"/.gnupg
chmod a-rwx,u+rwx "$TMPWD"/.gnupg
cd "$TMPWD"

if [ ! -f "$XMODE"/public.key ] || [ ! -f "$XMODE"/secret.key ]
then
  echo "when creating a keypair, use \"8, q, 4096, 1000y, y, cmgeneral,,, o, {pwd}\""
  echo -n "initialize keypair for this mode? (y/N) "
  read -s -n 1 YCHAR
  echo
  if [ "$YCHAR" == "y" ]
  then
    GPGOPT="--homedir .gnupg --keyring .gnupg/public.keyring --secret-keyring .gnupg/secret.keyring --no-default-keyring"
    gpg $GPGOPT --expert --gen-key
    gpg $GPGOPT --export            -a cmgeneral > "$XMODE"/public.key
    gpg $GPGOPT --export-secret-key -a cmgeneral > "$XMODE"/secret.key
  else
    exit 1
  fi
fi

if [ ! -f "$XMODE"/codename ]
then
  echo "echo codename > modes/$ARG1/codename"
  echo "the codename will suffix cmgeneral archives for easy identification"
  exit 1
fi

XCODE="`cat \"$XMODE\"/codename`"

XLIST="`cat \"$XMODE\"/list | tr '\t' ' ' | sed -e 's/^[ ]*//' -e 's/[ ]*$//' -e 's/[ ][ ]*/:/g' | tr '\n' '@' | sed -e 's/@@*/@/g' -e 's/^@*//g' -e 's/@*$//g'`""@"
echo "XLIST='$XLIST'"
XDEST="`cat \"$XMODE\"/dest | tr '\n' ' '`"
echo "XDEST='$XDEST'"

git init
XIDID=1000

while [ 1 ]
do
  XIDID=$((XIDID+1))
  
  XLINE="${XLIST%%@*}"
  XLIST="${XLIST#*@}"
  
  if [ "$XLINE" == "" ]
  then
    break
  fi
  
  XUSER="${XLINE%%:*}"
  XLINE="${XLINE#*:}"
  
  XHOST="${XLINE%%:*}"
  XLINE="${XLINE#*:}"
  
  XPATH="${XLINE%%:*}"
  XLINE="${XLINE#*:}"
  
  echo "XIDID='$XIDID'"
  echo "XUSER='$XUSER'"
  echo "XHOST='$XHOST'"
  echo "XPATH='$XPATH'"
  
  echo "$XIDID $XUSER $XHOST $XPATH" >> toc
  git remote add recover_"$XIDID" "$XUSER"@"$XHOST":"$XPATH"
  git fetch --no-tags recover_"$XIDID"
done

git fsck --full --strict
git repack -a -d -f --window=100 --depth=100 --window-memory=64m
git fsck --full --strict
git bundle create bundle.git --remotes
fakeroot tar -cf contents.tar toc bundle.git

cp "$XMODE"/public.key "$XMODE"/secret.key .
GPGOPT="--homedir .gnupg --keyring .gnupg/public.keyring --secret-keyring .gnupg/secret.keyring --no-default-keyring"
gpg $GPGOPT --import public.key
gpg $GPGOPT --import secret.key
gpg $GPGOPT --list-keys --with-colons --with-fingerprint | egrep fpr | sed -e 's/^fpr[:]*//g' | sed -e 's/$/6:/' | gpg $GPGOPT --import-ownertrust
gpg $GPGOPT --recipient cmgeneral --encrypt -z 0 < contents.tar > contents.tar.gpg
md5sum    contents.tar.gpg | egrep -o '^[0-9a-fA-F]{32}'  > contents.tar.gpg.md5sum
sha256sum contents.tar.gpg | egrep -o '^[0-9a-fA-F]{64}'  > contents.tar.gpg.sha256sum
sha512sum contents.tar.gpg | egrep -o '^[0-9a-fA-F]{128}' > contents.tar.gpg.sha512sum

cat > recover.sh << 'EOF'
#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

NONCE="`date +%Yy%mm%dd-%Hh%Mm%Ss-%ss-%Nn`"
mkdir recover-"$NONCE"
cd recover-"$NONCE"

mkdir .gnupg
sudo mount -t ramfs ramfs .gnupg
sudo chown "$USER":"$USER" .gnupg
chmod a-rwx,u+rwx .gnupg
GPGOPT="--homedir .gnupg --keyring .gnupg/public.keyring --secret-keyring .gnupg/secret.keyring --no-default-keyring"
gpg $GPGOPT --import ../public.key
gpg $GPGOPT --import ../secret.key
gpg $GPGOPT --list-keys --with-colons --with-fingerprint | egrep fpr | sed -e 's/^fpr[:]*//g' | sed -e 's/$/6:/' | gpg $GPGOPT --import-ownertrust
gpg $GPGOPT --decrypt < ../contents.tar.gpg > ../contents.tar
tar -C .. -xmf ../contents.tar

git init
git bundle unbundle ../bundle.git |\
while read SHA REF
do
  END="${REF#refs/remotes/}"
  git tag "${END//\//_}" "$SHA"
done

echo "use 'cd recover-*; git tag -l' to see what was recovered"
echo "remember, there is a table of contents in the 'toc' file"
echo "you won't see any files until you 'git merge' one of the recovered tags"
echo "good luck!"

sudo umount .gnupg
rmdir .gnupg
echo "+OK"
EOF
chmod a+x recover.sh

fakeroot tar -c recover.sh public.key secret.key contents.tar.gpg* > archive.tar

NONCE="`date +%Yy%mm%dd-%Hh%Mm%Ss-%ss-%Nn`"

for IDEST in $XDEST
do
  if [ ! -f "$IDEST"/.ddd1531eafda97524d852328d208469e ]
  then
    echo "sorry, magic file not found for destination '$IDEST', skipping"
  else
    cp archive.tar "$IDEST"/cmgeneral-"$NONCE"-"$XCODE".tar
  fi
done

if [ "$ARG2" == "s3" ] && [ -x "$XMODE"/s3cmd ] && [ -f "$XMODE"/s3cfg ] && [ -f "$XMODE"/s3bin ]
then
  mv archive.tar cmgeneral-"$NONCE"-"$XCODE".tar
  . "$XMODE"/j3env
  "$XMODE"/j3cmd --properties "$XMODE"/j3cfg UP --nodelete "`cat \"$XMODE\"/j3bin`" cmgeneral-*
fi

cd /
sudo umount "$TMPWD"/.gnupg
sudo umount "$TMPWD"
echo "+OK"