explogExplore.
http://yati-sagade.github.io/
Why I believe P ≠ NP<h2 id="the-hamcycle-problem">The HAMCYCLE problem</h2>
<p>If a graph $G\left(V, E\right)$ contains a <a href="http://mathworld.wolfram.com/HamiltonianCycle.html">Hamiltonian cycle</a>, we can pick
a subset $E’ ⊆ E$ of edges in the graph such that:</p>
<ol>
<li>All vertices in $G$ appear in the resulting subgraph and,</li>
<li>Starting at any vertex, we are able to visit each vertex exactly once, except
for the starting vertex, which is also the last vertex in the cycle.</li>
</ol>
<p>The Hamiltonian cycle problem asks if a graph $G$ contains a Hamiltonian cycle
– i.e., a cycle on which each vertex appears exactly once, except for the first
(and the last) vertex of the cycle.</p>
<p>The problem is <a href="https://en.wikipedia.org/wiki/NP-completeness">NP-Complete</a>. It is in <a href="https://en.wikipedia.org/wiki/NP_(complexity)">NP</a> because while no polynomial
time algorithm is known to solve it, given a graph and a “certificate” which
consists of a set of vertices claimed to constitute a Hamiltonian cycle, one can
decide in polynomial time if the certificate is bogus. Additionally, <em>any</em>
problem in NP can be reduced in polynomial time to an instance of the
Hamiltonian cycle problem, hence making it NP-Complete. <a href="https://www.geeksforgeeks.org/proof-hamiltonian-path-np-complete/">Here is such an
example reduction</a>, which works on the related Hamiltonian Path problem,
which in turn can be reduced in polynomial time to the Hamiltonian Cycle
problem.</p>
<h2 id="the-formal-language-described-by-hamcycle">The formal language described by HAMCYCLE</h2>
<p>This problem describes a language <script type="math/tex">% <![CDATA[
L = \{<G>: \text{G is Hamiltonian}\} %]]></script> which
is in NP. Here, the notation <script type="math/tex">% <![CDATA[
<G> %]]></script> represents a suitable binary encoding of the
graph G (i.e., a string over <script type="math/tex">\{0, 1\}^*</script>).</p>
<h2 id="co-hamcycle">Co-HAMCYCLE</h2>
<p>Consider the complement of problem: Finding if a graph is <em>not</em> Hamiltonian.
This problem is at least as hard as the original problem, since if there was
a polynomial time algorithm to tell us whether a given graph G is
non-Hamiltonian, we could just use this to solve the original problem of
deciding whether G is Hamiltonian.</p>
<p>This complement is in the co-NP class by definition, and it is an open problem
whether co-NP = NP. Since P ⊆ NP, and P is closed under the complement,
P ⊆ NP ∩ co-NP. And if P = NP, co-NP = NP (but the inverse does not hold), but
if co-NP ≠ NP, then P ≠ NP.</p>
<p><img src="/assets/pconp-1.png" /></p>
<p>The figure above shows that if NP and co-NP are different, P has to be different
than NP, since it must be a subset of both NP and co-NP.</p>
<h2 id="what-if-co-np--np">What if co-NP = NP?</h2>
<p>Again, proving that co-NP = NP does not show that P = NP (although it does
address an important standing problem in its own right, and leaves the
possibility open that P = NP).</p>
<p>To show that co-NP = NP, we’d have to show that all problems in co-NP are
verifiable in polynomial time. Let’s see what that means in the context of the
co-HAMCYLE problem.</p>
<p>To provide a certificate for the co-HAMCYCLE problem would require encoding the
<em>impossibility</em> of the set of edges in G to be form a Hamiltonian graph. In
other words, the certificate would have to provide a badge for a claim that the
graph G is not Hamiltonian. An example certificate would be the enumeration of
all possible cycles in the graph. The verification algorithm would then have to
check each cycle for being non-Hamiltonian, and also check that the set of
cycles in the certificate is exhaustive (because there might be a Hamiltonian
cycle that the certificate simply does not contain).</p>
<p>However, for this certificate to be verified in polynomial time, its length must
be polynomial in the size of the graph, and the check for exhaustiveness of the
certificate also needs to be carried out in polynomial time. I think this is
pretty hard, if not impossible to do. I don’t know how to prove this of course,
but if we prove that there can exist no certificate of length polynomial in the
size of the input graph, we’d have proven that co-HAMCYCLE is <em>not</em> in NP, i.e.,
NP ≠ co-NP, and by extension, since P must be contained in NP ∩ co-NP, P ≠ NP.</p>
Sun, 02 Sep 2018 00:00:00 +0000
http://yati-sagade.github.io/2018/09/02/co-hamiltonian/
http://yati-sagade.github.io/2018/09/02/co-hamiltonian/Creating lock-files in Unix<p>Lockfiles are commonly used for process level mutal exclusion. For example, a
cronjob processing hourly logs can hold a lock so in the event it ends up taking
more time than an hour, the next hourly job does not clobber the working
directory. Databases like Postgres also use lockfiles in their data directories
to ensure at most one serving process is handling the data.</p>
<p>On Unix, a very simple way of doing this is to open a file with the desired path
with <code class="highlighter-rouge">O_RDWR</code> and <code class="highlighter-rouge">O_EXCL</code> specified:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>int fd = open("/path/to/.lockfile", O_CREAT | O_RDWR | O_EXCL, 0600);
</code></pre></div></div>
<ul>
<li><code class="highlighter-rouge">O_CREAT</code> asks <code class="highlighter-rouge">open()</code> to create the file if it does not exist.</li>
<li><code class="highlighter-rouge">O_RDWR</code> requests a read/write handle.</li>
<li>
<p><code class="highlighter-rouge">O_EXCL</code> ensures that <em>this</em> call creates the file. From the Linux <code class="highlighter-rouge">open(2)</code>
manpage,</p>
<p>…if this flag is specified in conjunction with O_CREAT, and pathname
already exists, then open() will fail. When these two flags are specified,
symbolic links are not followed: if pathname is a symbolic link, then open()
fails regardless of where the symbolic link points to.</p>
</li>
</ul>
<p>Once such an <code class="highlighter-rouge">open()</code> succeeds, it is common practice to put the pid of the
process that holds this “lock” into the opened lockfile before closing the
handle.</p>
<p>In Rust, this can be achieved using the <code class="highlighter-rouge">std::fs::OpenOptions::create_new()</code>
function. We can wrap that in this small struct:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>use std::fs::OpenOptions;
use std::path::{Path, PathBuf};
use std::process;
use std::io::{self, Write};
use std::fs;
#[derive(Debug)]
pub struct LockFile {
path: PathBuf,
}
impl LockFile {
pub fn new<P>(path: P, lockfile_contents: Option<&[u8]>) -> io::Result<LockFile>
where
P: AsRef<Path>,
{
{
// create_new() translates to O_EXCL|O_CREAT being specified to the
// underlying open() syscall on *nix (and CREATE_NEW to the
// CreateFileW Windows API), which means that the call is successful
// only if it is the one which created the file.
let mut file = OpenOptions::new().write(true).create_new(true).open(
path.as_ref(),
)?;
if let Some(contents) = lockfile_contents {
file.write_all(contents)?;
}
}
debug!(
"Successfully wrote lockfile {:?}, pid: {}",
path.as_ref(),
process::id()
);
// By this time the file we created is closed, and we are sure that
// we are the one who created it.
Ok(LockFile { path: path.as_ref().to_path_buf() })
}
}
impl Drop for LockFile {
fn drop(&mut self) {
debug!("Removing lockfile {:?}, pid: {}", &self.path, process::id());
fs::remove_file(&self.path).unwrap();
}
}
</code></pre></div></div>
<p>Note that the lockfile will automatically be removed when the <code class="highlighter-rouge">LockFile</code> value
associated with it is dropped.</p>
<p>For a much more interesting implementation, see the <a href="https://github.com/postgres/postgres/blob/c37b3d08ca6873f9d4eaf24c72a90a550970cbb8/src/backend/utils/init/miscinit.c#L864">CreateLockFile</a> function
from Postgres. It is of course way more involved because of failure recovery
requirements.</p>
Sun, 24 Jun 2018 00:00:00 +0000
http://yati-sagade.github.io/2018/06/24/creating-lockfiles/
http://yati-sagade.github.io/2018/06/24/creating-lockfiles/Summary of "ImageNet Classification with Deep Convolutional Neural Networks"<p>I decided to read interesting deep learning papers often and try to summarize
them to aid my own understanding of the topics.</p>
<p>This paper, <a href="http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">ImageNet Classification with Deep Convolutional Neural Networks</a>
demonstrates a record breaking result on the ImageNet LSVRC-2012 competition.
The authors participated in the competition under the name <a href="http://image-net.org/challenges/LSVRC/2012/results.html">SuperVision</a> (which is
extremently difficult to Google, especially in the “deep learning” context, which
makes Google surface supervised learning related results).</p>
<h2 id="data">Data</h2>
<p>They trained the model on the ImageNet dataset, which contains about 1.2 million
images of 1000 categories.</p>
<h3 id="image-preprocessing">Image preprocessing</h3>
<p>Since ImageNet images are variable resolution, and the model presented in this
paper requires fixed size images, they scaled every image to 256x256 pixels. The
scaling like so:</p>
<ol>
<li>Scale a possibly rectangular image so that the shorter side is 256 pixels.</li>
<li>Take the middle 256x256 patch as the input image.</li>
</ol>
<h3 id="data-augmentation">Data augmentation</h3>
<p>Actually, the network presented in the paper works with 224x224 images, which
are generated by randomly sampling patches of that size from each 256x256 image.
This is one of the ways they did data augmentation.</p>
<p>The other way they augmented the dataset involved perturbing the R,G,B values of
each input image by a scaled version of the principal components (in RGB space)
across the whole training set. If $\mathbf{p_1}$, $\mathbf{p_2}$ and
$\mathbf{p_3}$ be the eigenvectors and $\lambda_1$, $\lambda_2$, $\lambda_3$ the
corresponding eigenvalues, each pixel in a given image is perturbed such that:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
I' &= I + \begin{bmatrix}
\leftarrow \mathbf{p_1}^T \rightarrow \\
\leftarrow \mathbf{p_2}^T \rightarrow \\
\leftarrow \mathbf{p_3}^T \rightarrow \\
\end{bmatrix}
\begin{bmatrix}
\alpha_1 \lambda_1 \\
\alpha_2 \lambda_2 \\
\alpha_3 \lambda_3
\end{bmatrix}
\end{align} %]]></script>
<p>where each $\alpha_1$ is picked from a Gaussian with 0 mean and 0.1 std every
time an image is used for training (but the values are shared by all pixels in
the image once picked). They claim this captured an invariance of object
identity under changes in intensity and color of illumination, which seems to
have accounted for a ~ 1% decrease in top-1 error rate.</p>
<h2 id="model-architecture">Model architecture</h2>
<p>The model has 5 convolutional layers and 3 fully-connected layers.</p>
<p><img src="/assets/imagenet-cnn-model.png" alt="image showing the convolutional network architecture, picked from the paper" /></p>
<h3 id="choice-of-nonlinearity">Choice of nonlinearity</h3>
<p>Authors chose the ReLU function (<code class="highlighter-rouge">ReLU(x) = max(0,x)</code>) and were I think one of
the first to endorse it, and for good reasons. They saw that using the ReLU
activation function, as opposed to the conventional tanh or sigmoid, yielded
tremendous gains in training speed, owing to the non saturating nature of the
function. When pitted against the tanh activation with no other changes, they
were able to train their model to 25% error rate on the training set <em>6 times
faster</em> with the ReLU activation.</p>
<h3 id="pooling">Pooling</h3>
<p>They use max pooling but with overlapping windows (filter size of 3 and stride
of 2). Max pooling is applied after response renormalization, described next.</p>
<h3 id="local-response-renormalization">Local response renormalization</h3>
<p>Activations from a given filter/kernel for a given location $\left(x,y\right)$
are scaled by a term involving the activations on $n$ adjacent kernels for the
same location. $n$ and other terms in question are hyperparameters and described
very well in the corresponding section in the paper. The inspiration seems to be
biological in nature, and in effect causes filters in an $n$ group to “compete”
for having larger activations. Of course they probably wouldn’t have kept it had
it not given them a significant error rate reduction on their validation set
(which it did).</p>
<h3 id="dropout-momentum-and-weight-decay">Dropout, momentum and weight decay</h3>
<p>Dropout is used in the first two fully-connected layers (layers 6 and 7 in the
overall network). Essentially, a given neuron is switched-off in
a forward-backward cycle with probability <code class="highlighter-rouge">0.5</code>. This means that the model is
forced to learn redundant representations with shared weights. During testing,
they do not switch of neurons, but multiply their outputs by <code class="highlighter-rouge">0.5</code>. No one quite
knows exactly why dropout works well, but indeed it has worked extremely well
for the authors and for many other models that followed. It has become very
commonplace today to incorporate dropout in FC layers to reduce overfitting.
Dropout approximately doubled the training time.</p>
<p>Momentum is a technique to speed up training by pushing batch gradient descent
updates to move towards the optimum faster. This is especially important when
the objective function is like a long ravine leading to the optimum with steep
wells (or like a hammock with a heavy person inside, but in higher dimensions).</p>
<p><img src="/assets/ravine.png" alt="contours of an objective function which looks like a hammock" /></p>
<p>In such an objective, plain batch gradient descent, if starting on the left
side, will oscillate up and down, moving slowly towards the optimum. Momentum
uses exponentially weighted averages to take into account previous gradient
descent updates. This averaging has an effect of cancelling out the up and down
oscillations, thus speeding up the learning significantly. Authors use
a momentum parameter of $0.9$, which roughly keeps the past 10 values in the
moving average memory.</p>
<p>Weight decay is simply to scale down weights by a factor slightly less than 1 to
prevent them from becoming too big. This can be seen as a regularizing
technique. In all the gradient descent updates for the model weights look like
the following:</p>
<script type="math/tex; mode=display">v \leftarrow 0.9 v - 0.0005.\epsilon.w - \epsilon\frac{\partial L}{\partial w} \\
w \leftarrow w + v</script>
<h2 id="training">Training</h2>
<p>The training is done on two GPUs for parallelism, and the setup is quite
interesting. The GPUs used each have 3GB memory. The network is split into
halves, as can be seen in the model description figure, across the two GPUs. The
interesting bit is that while layer 4 (CONV) operates on input which comes from
the layer 3 activations from <em>both</em> GPUs, other conv layers in the network do
not have this cross GPU communication going on, and only work with activations
from the GPU local half of the network.</p>
<p>Selecting how many layers should have cross-GPU comms going on is a problem for
cross-validation, and apparently this scheme seems to have worked well for them.
In the following figure, we see the first conv layer weights visualized, the top
half being the first GPU, and the bottom half the other. As the authors mention
in the paper, the two halves of the network consistently specialized in
features, with one focusing on frequency/orientation, and the other in coloured
blob detection.</p>
<p><img src="/assets/gpu-specific-learning.png" alt="the first conv layer learned representations in both GPUs" /></p>
<p>Because of the sheer number of parameters (60 million), the training still takes
5-6 days. The learning rate had to be manually decayed when progress plateaued.</p>
<h2 id="conclusion">Conclusion</h2>
<p>This work was the first of its kind to have trained deep convolutional networks
on GPUs to achieve impressive results on the ImageNet dataset for object
detection. To conclude, here’s a Google Trends visualization for the term “deep
learning” over time:</p>
<script type="text/javascript" src="https://ssl.gstatic.com/trends_nrtr/1294_RC01/embed_loader.js"></script>
<script type="text/javascript">
trends.embed.renderExploreWidget("TIMESERIES", {"comparisonItem":[{"keyword":"deep learning","geo":"","time":"2004-01-01 2018-01-28"}],"category":0,"property":""}, {"exploreQuery":"date=all&q=deep%20learning","guestPath":"https://trends.google.com:443/trends/embed/"});
</script>
<p>(the paper was presented in December 2012 at NIPS).</p>
Sun, 28 Jan 2018 00:00:00 +0000
http://yati-sagade.github.io/2018/01/28/imagenet-deep-cnn-summary/
http://yati-sagade.github.io/2018/01/28/imagenet-deep-cnn-summary/Spawn, log, reap children with IPC::Run<p>In testing my implementation of a <a href="https://github.com/yati-sagade/phifd">distributed failure detector</a>, I needed to
be able to:</p>
<ol>
<li>Spawn <code class="highlighter-rouge">n</code> instances of the detector, each listening on a different port on localost, making a small local cluster.</li>
<li>Redirect each process’ standard output and error streams to separate logfiles, named such that they can be tied back to the generating processes later.</li>
<li>After running for some time, terminate all processes, and then analyse the generated logs.</li>
</ol>
<p>It turns out that getting step 1 right is nontrivial (but worth trying to
understand what facilities your language offers to this end).</p>
<p>I started out doing the regular <code class="highlighter-rouge">fork-exec</code> routine, except that the act of
redirecting the output stream of the target process in <code class="highlighter-rouge">exec</code> was causing the
<em>shell</em> to be involved:</p>
<div class="language-perl highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">my</span> <span class="nv">@cmd</span> <span class="o">=</span> <span class="p">(</span> <span class="s">'/home/ys/bin/phifd'</span><span class="p">,</span> <span class="o">...</span> <span class="p">);</span>
<span class="o">...</span>
<span class="k">my</span> <span class="nv">@children</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span> <span class="k">my</span> <span class="nv">$pid</span> <span class="o">=</span> <span class="nb">fork</span><span class="p">()</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">push</span> <span class="nv">@children</span><span class="p">,</span> <span class="nv">$pid</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c1"># presence of a shell metachar automatically causes a shell to be spawned.</span>
<span class="nb">exec</span> <span class="nv">@cmd</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="s">"$logfile"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The problem here is that when reaping the children later on, the SIGTERM that
we send to the child processes gets trapped by the shell, which dies, and then
<em>its</em> child, which is the actual process we wanted to kill, keeps running, now
parented by init.</p>
<p>One way to solve this is to avoid spawning a shell altogether, and relay
the child output streams to respective log files in the parent process, and in
Perl <code class="highlighter-rouge">IPC::Run</code> is a handy library for this. Here’s the code:</p>
<div class="language-perl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nv">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nv">warnings</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">IPC::</span><span class="nv">Run</span> <span class="sx">qw(run timeout start harness)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Path</span> <span class="sx">qw(make_path remove_tree)</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">File::</span><span class="nv">Slurp</span> <span class="sx">qw(read_dir read_file)</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$log_root_dir</span> <span class="o">=</span> <span class="nb">shift</span> <span class="o">||</span> <span class="s">'logs/'</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@cmds</span> <span class="o">=</span> <span class="p">(</span>
<span class="p">[</span>
<span class="s">'phifd'</span><span class="p">,</span> <span class="s">'-t'</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="s">'-a'</span><span class="p">,</span> <span class="s">'127.0.0.1:12345'</span>
<span class="p">],</span>
<span class="p">[</span>
<span class="s">'phifd'</span><span class="p">,</span> <span class="s">'-t'</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="s">'-a'</span><span class="p">,</span> <span class="s">'127.0.0.1:12346'</span><span class="p">,</span> <span class="s">'-i'</span><span class="p">,</span> <span class="s">'127.0.0.1:12345'</span>
<span class="p">],</span>
<span class="p">[</span>
<span class="s">'phifd'</span><span class="p">,</span> <span class="s">'-t'</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="s">'-a'</span><span class="p">,</span> <span class="s">'127.0.0.1:12347'</span><span class="p">,</span> <span class="s">'-i'</span><span class="p">,</span> <span class="s">'127.0.0.1:12346'</span>
<span class="p">],</span>
<span class="p">);</span>
<span class="k">my</span> <span class="nv">@handles</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@harnesses</span><span class="p">;</span>
<span class="k">for</span> <span class="k">my</span> <span class="nv">$procnum</span> <span class="p">(</span> <span class="mi">0</span> <span class="o">..</span> <span class="nv">$#cmds</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$logfile</span> <span class="o">=</span> <span class="nn">File::</span><span class="nv">Spec</span><span class="o">-></span><span class="nv">catfile</span><span class="p">(</span> <span class="nv">$log_root_dir</span><span class="p">,</span>
<span class="s">'proc_'</span> <span class="o">.</span> <span class="nv">$procnum</span> <span class="o">.</span> <span class="s">'.log'</span> <span class="p">);</span>
<span class="c1"># XXX: do we need an :encoding(utf8) discipline here?</span>
<span class="nb">open</span> <span class="k">my</span> <span class="nv">$fh</span><span class="p">,</span> <span class="s">'>'</span><span class="p">,</span> <span class="nv">$logfile</span>
<span class="ow">or</span> <span class="nb">die</span> <span class="s">"Cannot open $logfile for writing: $!"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$harness</span> <span class="o">=</span> <span class="nv">harness</span>
<span class="nv">$cmds</span><span class="p">[</span><span class="nv">$procnum</span><span class="p">],</span> <span class="c1"># command to run</span>
<span class="o">\</span><span class="nb">undef</span><span class="p">,</span> <span class="c1"># input (goes to child's stdin)</span>
<span class="nv">$fh</span><span class="p">,</span> <span class="c1"># output (from child's stdout)</span>
<span class="nv">$fh</span><span class="p">,</span> <span class="c1"># error (from child's stderr)</span>
<span class="nv">init</span> <span class="o">=></span> <span class="k">sub </span><span class="p">{</span>
<span class="c1"># Any environment setup goes here</span>
<span class="nv">$ENV</span><span class="p">{</span><span class="nv">RUST_BACKTRACE</span><span class="p">}</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">};</span>
<span class="nb">push</span> <span class="nv">@handles</span><span class="p">,</span> <span class="nv">$fh</span><span class="p">;</span>
<span class="nb">push</span> <span class="nv">@harnesses</span><span class="p">,</span> <span class="nv">$harness</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">my</span> <span class="nv">$start</span> <span class="o">=</span> <span class="nb">time</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$elapsed</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span> <span class="nv">$elapsed</span> <span class="o"><</span> <span class="nv">$runtime</span> <span class="p">)</span> <span class="p">{</span>
<span class="c1"># Nonblocking "pumping" -- in our case, we check if a child has output,</span>
<span class="c1"># and if so, write it to the respective log file handle. IPC::Run handles</span>
<span class="c1"># this for us.</span>
<span class="nv">$_</span><span class="o">-></span><span class="nv">pump_nb</span> <span class="k">for</span> <span class="nv">@harnesses</span><span class="p">;</span>
<span class="nv">$elapsed</span> <span class="o">=</span> <span class="nb">time</span> <span class="o">-</span> <span class="nv">$start</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1"># cleanup</span>
<span class="nv">$_</span><span class="o">-></span><span class="nv">kill_kill</span> <span class="k">for</span> <span class="nv">@harnesses</span><span class="p">;</span> <span class="c1"># kill with progressively stiffer signals.</span>
<span class="nv">@harnesses</span> <span class="o">=</span> <span class="p">();</span>
<span class="nb">close</span> <span class="nv">$_</span> <span class="k">for</span> <span class="nv">@handles</span><span class="p">;</span>
<span class="nv">@handles</span> <span class="o">=</span> <span class="p">();</span>
</code></pre></div></div>
<p>That’s it, <code class="highlighter-rouge">IPC::Run</code> is quite flexible, and the perldoc is quite well written.</p>
Tue, 16 Jan 2018 00:00:00 +0000
http://yati-sagade.github.io/2018/01/16/ipc-run/
http://yati-sagade.github.io/2018/01/16/ipc-run/The bakery algorithm for mutual exclusion<p>The <a href="http://lamport.azurewebsites.net/pubs/bakery.pdf">bakery algorithm</a> was proposed by Leslie Lamport as a solution to
Dijkstra’s concurrent programming problem. In the problem, Dijkstra had first
identified the need for mutual exclusion among a group of concurrently executing
processes.</p>
<p>We want to run <code class="highlighter-rouge">N</code> processes concurrently, with exclusive access to a shared
resource. Modern day languages solve this with help from hardware. But the
bakery algorithm allows for pure software mutual exclusion by making sure that
at most one process is executing its critical section, while other non-faulty
processes are either in their non-critical section, or spinning in place,
waiting to get to the critical section. In other words, mutual exclusion in the
critical section is achieved in software. Now this is inefficient, since it
involves processes “spinning” in a loop while waiting for a chance to execute
the critical section.</p>
<p>What is remarkable about this algorithm (as Lamport points out multiple times in
the paper) is that even though reads from memory may not be atomic (which is
important with overlapping reads and writes to the same location), the algorithm
works fine, since it does not care <em>what</em> exact value is read, and makes do with
a distinction between <em>zero</em> and <em>nonzero</em> values. Here’s an implementation
followed by a description (the code is on github: <code class="highlighter-rouge">go get
github.com/yati-sagade/bakery</code>):</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span><span class="x"> </span><span class="n">main</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"flag"</span><span class="x">
</span><span class="s">"fmt"</span><span class="x">
</span><span class="s">"time"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">max</span><span class="p">(</span><span class="n">xs</span><span class="x"> </span><span class="p">[]</span><span class="kt">int</span><span class="p">)</span><span class="x"> </span><span class="kt">int</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="nb">len</span><span class="p">(</span><span class="n">xs</span><span class="p">)</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="m">1</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="nb">panic</span><span class="p">(</span><span class="s">"max() on empty slice"</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">ret</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">xs</span><span class="p">[</span><span class="m">0</span><span class="p">]</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="m">1</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="nb">len</span><span class="p">(</span><span class="n">xs</span><span class="p">);</span><span class="x"> </span><span class="n">i</span><span class="o">++</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">xs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="x"> </span><span class="o">></span><span class="x"> </span><span class="n">ret</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">ret</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">xs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">ret</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">n</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">flag</span><span class="o">.</span><span class="n">Int</span><span class="p">(</span><span class="s">"nodes"</span><span class="p">,</span><span class="x"> </span><span class="m">5</span><span class="p">,</span><span class="x"> </span><span class="s">"number of participating nodes"</span><span class="p">)</span><span class="x">
</span><span class="n">iters</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">flag</span><span class="o">.</span><span class="n">Int</span><span class="p">(</span><span class="s">"iters"</span><span class="p">,</span><span class="x"> </span><span class="m">100000</span><span class="p">,</span><span class="x"> </span><span class="s">"number of iterations"</span><span class="p">)</span><span class="x">
</span><span class="n">debug</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">flag</span><span class="o">.</span><span class="n">Bool</span><span class="p">(</span><span class="s">"debug"</span><span class="p">,</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x"> </span><span class="s">"print debug trace"</span><span class="p">)</span><span class="x">
</span><span class="n">flag</span><span class="o">.</span><span class="n">Parse</span><span class="p">()</span><span class="x">
</span><span class="c">// 1. process i is in the doorway while choosing[i] == true</span><span class="x">
</span><span class="c">// 2. process i is in the bakery from when it resets choosing[i] to false till</span><span class="x">
</span><span class="c">// either it fails, or finishes its critical section.</span><span class="x">
</span><span class="n">choosing</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">([]</span><span class="kt">bool</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="m">1</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="m">1</span><span class="p">)</span><span class="x"> </span><span class="c">// one extra for the monitoring process</span><span class="x">
</span><span class="n">numbers</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">([]</span><span class="kt">int</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="m">1</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">n</span><span class="o">+</span><span class="m">1</span><span class="p">)</span><span class="x">
</span><span class="n">sharedMap</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">int</span><span class="p">{</span><span class="x">
</span><span class="s">"last_updated_by"</span><span class="o">:</span><span class="x"> </span><span class="o">-</span><span class="m">1</span><span class="p">,</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">sharedSlice</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">([]</span><span class="kt">int</span><span class="p">,</span><span class="x"> </span><span class="m">0</span><span class="p">)</span><span class="x">
</span><span class="n">lock</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">id</span><span class="x"> </span><span class="kt">int</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">choosing</span><span class="p">[</span><span class="n">id</span><span class="p">]</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="no">true</span><span class="x">
</span><span class="c">/* At the doorway */</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="o">*</span><span class="n">debug</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%d: choosing</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x"> </span><span class="n">id</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">numbers</span><span class="p">[</span><span class="n">id</span><span class="p">]</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="m">1</span><span class="x"> </span><span class="o">+</span><span class="x"> </span><span class="n">max</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="o">*</span><span class="n">debug</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%d: chose number %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x"> </span><span class="n">id</span><span class="p">,</span><span class="x"> </span><span class="n">numbers</span><span class="p">[</span><span class="n">id</span><span class="p">])</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">choosing</span><span class="p">[</span><span class="n">id</span><span class="p">]</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="no">false</span><span class="x">
</span><span class="c">/* Entered bakery */</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">k</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="m">0</span><span class="p">;</span><span class="x"> </span><span class="n">k</span><span class="x"> </span><span class="o"><=</span><span class="x"> </span><span class="o">*</span><span class="n">n</span><span class="p">;</span><span class="x"> </span><span class="n">k</span><span class="o">++</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">/* L1 */</span><span class="x">
</span><span class="n">printed</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="no">false</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">choosing</span><span class="p">[</span><span class="n">k</span><span class="p">]</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">/* L2 */</span><span class="x">
</span><span class="c">// spin</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="o">*</span><span class="n">debug</span><span class="x"> </span><span class="o">&&</span><span class="x"> </span><span class="o">!</span><span class="n">printed</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%d: spinning for %d (waiting for it to choose: %v)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x">
</span><span class="n">id</span><span class="p">,</span><span class="x"> </span><span class="n">k</span><span class="p">,</span><span class="x"> </span><span class="n">choosing</span><span class="p">)</span><span class="x">
</span><span class="n">printed</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="no">true</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="c">// this is important, as tight spinning here would cause only</span><span class="x">
</span><span class="c">// the first $n_core goroutines to be scheduled, starving</span><span class="x">
</span><span class="c">// all others. this would in turn cause the whole algorithm</span><span class="x">
</span><span class="c">// to stop making any progress.</span><span class="x">
</span><span class="n">time</span><span class="o">.</span><span class="n">Sleep</span><span class="p">(</span><span class="m">10</span><span class="x"> </span><span class="o">*</span><span class="x"> </span><span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">printed</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="no">false</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">numbers</span><span class="p">[</span><span class="n">k</span><span class="p">]</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="m">0</span><span class="x"> </span><span class="o">&&</span><span class="x">
</span><span class="p">((</span><span class="n">numbers</span><span class="p">[</span><span class="n">k</span><span class="p">]</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="n">numbers</span><span class="p">[</span><span class="n">id</span><span class="p">])</span><span class="x"> </span><span class="o">||</span><span class="x">
</span><span class="p">(</span><span class="n">numbers</span><span class="p">[</span><span class="n">k</span><span class="p">]</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="n">numbers</span><span class="p">[</span><span class="n">id</span><span class="p">]</span><span class="x"> </span><span class="o">&&</span><span class="x"> </span><span class="n">k</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="n">id</span><span class="p">))</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">/* L3 */</span><span class="x">
</span><span class="c">// spin</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="o">*</span><span class="n">debug</span><span class="x"> </span><span class="o">&&</span><span class="x"> </span><span class="o">!</span><span class="n">printed</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%d: spinning for %d (waiting for it to finish CS: %v)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x">
</span><span class="n">id</span><span class="p">,</span><span class="x"> </span><span class="n">k</span><span class="p">,</span><span class="x"> </span><span class="n">numbers</span><span class="p">)</span><span class="x">
</span><span class="n">printed</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="no">true</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">time</span><span class="o">.</span><span class="n">Sleep</span><span class="p">(</span><span class="m">10</span><span class="x"> </span><span class="o">*</span><span class="x"> </span><span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">unlock</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">id</span><span class="x"> </span><span class="kt">int</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="o">*</span><span class="n">debug</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%d: unlock: %v</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x"> </span><span class="n">id</span><span class="p">,</span><span class="x"> </span><span class="n">numbers</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">numbers</span><span class="p">[</span><span class="n">id</span><span class="p">]</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="m">0</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">proc</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">id</span><span class="x"> </span><span class="kt">int</span><span class="p">,</span><span class="x"> </span><span class="n">stop</span><span class="x"> </span><span class="k">chan</span><span class="x"> </span><span class="k">struct</span><span class="p">{},</span><span class="x"> </span><span class="n">ack</span><span class="x"> </span><span class="k">chan</span><span class="x"> </span><span class="k">struct</span><span class="p">{})</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">out</span><span class="o">:</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">select</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">case</span><span class="x"> </span><span class="o"><-</span><span class="n">stop</span><span class="o">:</span><span class="x">
</span><span class="k">break</span><span class="x"> </span><span class="n">out</span><span class="x">
</span><span class="k">default</span><span class="o">:</span><span class="x">
</span><span class="n">lock</span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%d: entering critical section</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x"> </span><span class="n">id</span><span class="p">)</span><span class="x">
</span><span class="c">// critical section</span><span class="x">
</span><span class="n">sharedMap</span><span class="p">[</span><span class="s">"last_updated_by"</span><span class="p">]</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">id</span><span class="x">
</span><span class="n">sharedSlice</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="nb">append</span><span class="p">(</span><span class="n">sharedSlice</span><span class="p">,</span><span class="x"> </span><span class="n">id</span><span class="p">)</span><span class="x">
</span><span class="n">sharedSlice</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">sharedSlice</span><span class="p">[</span><span class="m">1</span><span class="o">:</span><span class="p">]</span><span class="x">
</span><span class="c">// end critical section</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%d: done critical section</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x"> </span><span class="n">id</span><span class="p">)</span><span class="x">
</span><span class="n">unlock</span><span class="p">(</span><span class="n">id</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">ack</span><span class="x"> </span><span class="o"><-</span><span class="x"> </span><span class="k">struct</span><span class="p">{}{}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">stops</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">([]</span><span class="k">chan</span><span class="x"> </span><span class="k">struct</span><span class="p">{},</span><span class="x"> </span><span class="m">0</span><span class="p">)</span><span class="x">
</span><span class="n">acks</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">([]</span><span class="k">chan</span><span class="x"> </span><span class="k">struct</span><span class="p">{},</span><span class="x"> </span><span class="m">0</span><span class="p">)</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="m">0</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="o">*</span><span class="n">n</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="o">++</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">c</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">(</span><span class="k">chan</span><span class="x"> </span><span class="k">struct</span><span class="p">{})</span><span class="x">
</span><span class="n">d</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">(</span><span class="k">chan</span><span class="x"> </span><span class="k">struct</span><span class="p">{})</span><span class="x">
</span><span class="n">stops</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="nb">append</span><span class="p">(</span><span class="n">stops</span><span class="p">,</span><span class="x"> </span><span class="n">c</span><span class="p">)</span><span class="x">
</span><span class="n">acks</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="nb">append</span><span class="p">(</span><span class="n">acks</span><span class="p">,</span><span class="x"> </span><span class="n">d</span><span class="p">)</span><span class="x">
</span><span class="k">go</span><span class="x"> </span><span class="n">proc</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="x"> </span><span class="n">c</span><span class="p">,</span><span class="x"> </span><span class="n">d</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="c">// Monitor and quit after some time</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="m">1</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="x"> </span><span class="o"><</span><span class="x"> </span><span class="o">*</span><span class="n">iters</span><span class="p">;</span><span class="x"> </span><span class="n">i</span><span class="o">++</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">lock</span><span class="p">(</span><span class="o">*</span><span class="n">n</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="o">*</span><span class="n">debug</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"monitor lock acquired"</span><span class="p">,</span><span class="x"> </span><span class="n">sharedSlice</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="nb">len</span><span class="p">(</span><span class="n">sharedSlice</span><span class="p">)</span><span class="x"> </span><span class="o">></span><span class="x"> </span><span class="m">1</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="nb">panic</span><span class="p">(</span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"found concurrent access with %v"</span><span class="p">,</span><span class="x"> </span><span class="n">sharedSlice</span><span class="p">))</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="c">// Go will panic when sharedMap is written to by two goroutines</span><span class="x">
</span><span class="n">unlock</span><span class="p">(</span><span class="o">*</span><span class="n">n</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="o">*</span><span class="n">debug</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"monitor lock released"</span><span class="p">,</span><span class="x"> </span><span class="n">sharedSlice</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">stop</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">stops</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"stopping"</span><span class="p">)</span><span class="x">
</span><span class="n">stop</span><span class="x"> </span><span class="o"><-</span><span class="x"> </span><span class="k">struct</span><span class="p">{}{}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">ack</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">acks</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="o"><-</span><span class="n">ack</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<h3 id="description">Description</h3>
<p>So we have <code class="highlighter-rouge">n</code> goroutines, and each wants to mutate a shared map (by updating
the single value in it) and a shared slice (by appending the process id to it,
and then reslicing it to maintain a length of 1). There is also the main
monitoring goroutine that checks if the shared slice ever contains more than
two items. Simultaneous mutation of a map by more than one goroutines is
detected by the Go runtime, which panics in such a case.</p>
<p>The key idea of the algorithm is that just like in an old fashioned bakery (or
a bank), each participant gets a token, or a number, roughly in the order of
arrival. Participants then get serviced in order of their token numbers. The
thing is that in our situation, there is no single receptionist to hand these
tokens over to participant processes. So, our solution is to have processes pick
their own numbers, and such a tie is broken by letting the process with a lower
id enter the critical section first.</p>
<p>The <code class="highlighter-rouge">lock</code> routine is the meat of the bakery algorithm. There are two shared
slices, <code class="highlighter-rouge">choosing</code> and <code class="highlighter-rouge">numbers</code>, and process <code class="highlighter-rouge">i</code> only writes to <code class="highlighter-rouge">choosing[i]</code>
and <code class="highlighter-rouge">numbers[i]</code>, but can read from any other indices in <code class="highlighter-rouge">choosing</code> and
<code class="highlighter-rouge">numbers</code>.</p>
<h3 id="correctness">Correctness</h3>
<p>Our system with <code class="highlighter-rouge">N</code> processes (goroutines, here) functions such that at any
given time, at most one process is in the critical section. To prove this, let’s
see the execution from process <code class="highlighter-rouge">i</code>’s perspective, and first define some terms:</p>
<ul>
<li><script type="math/tex">t^{ik}_{L2}</script>: Time when process <code class="highlighter-rouge">i</code> completes the <em>last</em> iteration of loop
<code class="highlighter-rouge">L2</code> for process <code class="highlighter-rouge">k</code>. This is when it sees <code class="highlighter-rouge">choosing[k] == false</code>.</li>
<li><script type="math/tex">t^{ik}_{L3}</script>: Time when process <code class="highlighter-rouge">i</code> completes the <em>last</em> iteration of loop
<code class="highlighter-rouge">L3</code> for process <code class="highlighter-rouge">k</code>. This happens when process <code class="highlighter-rouge">i</code> sees a <code class="highlighter-rouge">0</code> in <code class="highlighter-rouge">numbers[k]</code>,
or if it thinks it should be serviced before process <code class="highlighter-rouge">k</code> owing to a lower token
number.</li>
<li><script type="math/tex">t^k_{d}</script>: Time at which process <code class="highlighter-rouge">k</code> completes <code class="highlighter-rouge">choosing[k] = true</code>.</li>
<li><script type="math/tex">t^k_{e}</script>: Time at which process <code class="highlighter-rouge">k</code> completes setting <code class="highlighter-rouge">choosing[k] = false</code>,
after choosing a number.</li>
</ul>
<p>We can now make some observations:</p>
<ul>
<li><script type="math/tex">t^k_d \lt t^k_e</script>.</li>
<li>At time $t^k_e$, <code class="highlighter-rouge">numbers[k]</code> is completely written, and a read at or after this
point of <code class="highlighter-rouge">numbers[k]</code> will not return a partial result.</li>
<li>
<p>At <script type="math/tex">t^{ik}_{L2}</script>, either process <code class="highlighter-rouge">k</code> is just out of the doorway (set
<code class="highlighter-rouge">choosing[k] = false</code>) and in the bakery (either waiting for other processes,
or executing its critical section); or is neither in the doorway or in the
bakery (maybe executing some non-critical section code). This corresponds to
either <script type="math/tex">t^{ik}_{L2} \gt t^k_e</script>, or <script type="math/tex">t^{ik}_{L2} \lt t^k_d</script>, respectively.</p>
<ul>
<li>
<p>When <script type="math/tex">t^{ik}_{L2} \gt t^k_e</script>, we also have <script type="math/tex">t^{ik}_{L3} \gt t^k_e</script>.
This means that process <code class="highlighter-rouge">i</code> found (at time <script type="math/tex">t^{ik}_{L3}</script>) either <code class="highlighter-rouge">numbers[i]
< numbers[k]</code>, or if both numbers are equal, that <code class="highlighter-rouge">i < k</code>. Note that <code class="highlighter-rouge">k</code>
<em>must have</em> picked a number at least as large as <code class="highlighter-rouge">numbers[i]</code> in this case.</p>
</li>
<li>
<p>When <script type="math/tex">t^{ik}_{L2} \lt t^k_d</script>, it must be that <code class="highlighter-rouge">numbers[i] < numbers[k]</code>,
since when <code class="highlighter-rouge">k</code> does enter the doorway (after the moment $t^k_d$), it will
pick a number that is at least one larger than <code class="highlighter-rouge">numbers[i]</code>, which has
already been written completely.</p>
</li>
</ul>
</li>
</ul>
<p>Hence, in both these cases, process <code class="highlighter-rouge">i</code> does not wait for process <code class="highlighter-rouge">k</code>.
Generalizing this to all other processes <code class="highlighter-rouge">k</code> that a given process <code class="highlighter-rouge">i</code> needs to
share resources with, it can be seen that at most one process will be able to
execute its critical section.</p>
<h3 id="failure">Failure</h3>
<p>If a process fails after setting <code class="highlighter-rouge">choosing[k] = true</code>, and then keeps failing
and recovering, never resetting <code class="highlighter-rouge">choosing[k]</code>, a deadlock can be reached as
other processes will keep spinning, waiting on the failed process.</p>
<h3 id="starvation-because-of-spinning">Starvation because of spinning</h3>
<p>If we do not yield to the runtime by calling <code class="highlighter-rouge">time.Sleep()</code> in every loop
iteration, only as many goroutines as the number of cores on our computer will
be able to run causing the algorithm to stop making any progress.</p>
Thu, 19 Oct 2017 00:00:00 +0000
http://yati-sagade.github.io/programming/concurrency/2017/10/19/bakery-algorithm/
http://yati-sagade.github.io/programming/concurrency/2017/10/19/bakery-algorithm/Optimal failure detector performance<p>This is a note to self on computing the lower bound on number of messages each
process in a distributed failure detector must send to guarantee adherence to
pre-specified values for:</p>
<ul>
<li>Time to first detection of a true failure.</li>
<li>False positive detection rate.</li>
<li>Message loss rate.</li>
</ul>
<p>In other words, the failure detector accepts as parameters the above three
values, and for those guarantees to be respected, each node in the failure
detector cluster must send at least $m$ number of messages, which is what we are
after.</p>
<p>The original paper from which I learnt most of this is <a href="http://www.cs.cornell.edu/projects/quicksilver/public_pdfs/On%20Scalable.pdf">On Scalable and
Efficient Distributed Failure Detectors</a>.</p>
<p>We shall talk about eventually complete (<em>every</em> failure is eventually detected)
failure detection, in a crash-stop failure model (actually it isn’t hard to
extend the analysis to a crash-recovery model, but we don’t assume Byzantine
failures).</p>
<h2 id="parameters">Parameters</h2>
<h3 id="1-time-to-first-detection-of-a-true-failure">1. Time to first detection of a true failure</h3>
<p>The failure detector in question accepts a parameter $T$, which is the maximum
time in seconds between a process failing, and <em>some</em> other process detecting
this. Note that this explicitly does <em>not</em> assume any dissemination mechanism,
and depending on what it is, <em>first</em> detection may or may not mean detection by
all non-faulty processes.</p>
<h3 id="2-false-positive-detection-rate">2. False positive detection rate</h3>
<p>The next parameter the failure detector accepts is $\alpha_{T}$, the acceptable
false failure detection rate till the first true failure detection. This is
a bit tricky to understand. Consider at time $t_0$, we have a bunch of
non-faulty processes $p_1, p_2, p_3.., p_M$ that have not been marked failed
yet. We want the probability that any <em>non faulty</em> process marks any of $p_i$ as
failed till $t_0 + T$ to be capped at $\alpha_{T}$. To look at this from another
perspective, consider 1000 random time periods, each of length $T$. For each
time period, we count the number of false failure detections, and sum them up
(good nodes marked as faulty by another good node). We want this number to be no
more than $1000\alpha_T$.</p>
<h3 id="3-message-loss-rate">3. Message loss rate</h3>
<p>The application also specifies the probability of any given message being lost.
This is applied independently to each message, and hence may not be a true
picture of what happens in the real world (correlated message losses around
congested network zones), but hey at least <a href="http://abstrusegoose.com/406">we are not talking about a spherical
cow in vaccum</a>. We call this parameter $p_{ml}$.</p>
<h2 id="result">Result</h2>
<p>With the parameters specified as above, in an $N$ node distributed failure
detector, the <em>minimum</em> number of messages each node has to send per second is
given by</p>
<script type="math/tex; mode=display">m_{min} = \frac{1}{T} \frac{\log\alpha_T}{\log{p_{ml}}}</script>
<p>This is neat, because it does not depend on the cluster size $N$!</p>
<h2 id="proof">Proof</h2>
<p>Suppose each node sends $m$ messages out (as heartbeats or pings to other nodes)
in a time period $T$. The only way a non-faulty process can appear as faulty to
all others is if <em>all</em> its outgoing messages are dropped in the time-period of
$T$ seconds. This happens with probability $\left(p_{ml}\right)^m$. We want this
probability to be capped at $\alpha_T$. So,</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\alpha_T & \ge \left(p_{ml}\right)^m \\
\implies \log{\alpha_T} & \ge m\log{p_{ml}} \\
\implies m & \ge \frac{\log\alpha_T}{\log{p_{ml}}} \\
& \left(\because 0 \lt p_{ml}
\lt 1, 0 \lt \alpha_T \lt 1 \right)
\end{align*} %]]></script>
<p>Since $m$ is the number of messages sent <em>every T seconds</em>, the number of
messages per second $m_r \ge \frac{1}{T} \frac{\log\alpha_T}{\log{p_{ml}}}$, and
hence, $m_{min} = \frac{1}{T} \frac{\log\alpha_T}{\log{p_{ml}}}$.</p>
Sat, 16 Sep 2017 00:00:00 +0000
http://yati-sagade.github.io/distributed-systems/2017/09/16/optimal-fd-load/
http://yati-sagade.github.io/distributed-systems/2017/09/16/optimal-fd-load/Analysis of gossip based dissemination<p>Gossip based protocols are widely used in distributed systems for robust
dissemination of information. The problem: spreading a message among a set of
processes. For example, in the Bitcoin P2P network, whenever a new transaction
happens, it needs to be broadcast to all peers in order for it to end up on the
blockchain. Typically, such information originates at one of the nodes in the
network, and needs to be communicated to the rest of the peers.</p>
<p>One elegant solution to this problem mimics how rumours spread in the society by
word of mouth, namely “gossip” based protocols. The gossip component of every
non-faulty process in the network maintains two main pieces of state: a list of
other known live peers, and a buffer of recent messages. Then every <code class="highlighter-rouge">T</code> seconds,
(called the <em>gossip period</em>), every node executes the following:</p>
<ol>
<li>Pick <code class="highlighter-rouge">f</code> peers from the membership list at random. Here, <code class="highlighter-rouge">f</code> is called the
gossip fanout.</li>
<li>Send recent messages from our messages buffer to each of these peers.</li>
</ol>
<p>Of course, in parallel, each node must listen for messages and update its
message buffer.</p>
<p>The details like the value of the gossip fanout, and what exact messages to
relay, and for how many rounds, etc. are what make different gossip protocols
different, but we aren’t going to talk much about it here.</p>
<h2 id="analysis">Analysis</h2>
<p>The analysis focuses on finding upper bounds for:</p>
<ul>
<li>The number of rounds needed to get an update to all participants.</li>
<li>The load on each participating peer.</li>
</ul>
<p>To begin with, we note that the inherent random nature of the algorithm means
that we can only comment about the <em>expected</em> behaviour of the protocol —
i.e., we are after the number of rounds that a cluster of <code class="highlighter-rouge">N</code> nodes needs to
have a message disseminated across the cluster <em>with high probability</em>.</p>
<h3 id="dissemination-time">Dissemination time</h3>
<p>This refers to the number of gossip rounds before we can be reasonably sure
about a message having been disseminated across the cluster.</p>
<p>This analysis is borrowed (indirectly) from work on epidemiology — our problem
is not so different from a situation where an infected organism comes in contact
with random uninfected individuals, thereby infecting them. These individuals
in turn go on to infect others and so on.</p>
<p>For our purpose, we shall say that a node in possession of a particular message
<code class="highlighter-rouge">M</code> is “infected”, while nodes which don’t know about <code class="highlighter-rouge">M</code> yet are uninfected.
Let’s further suppose that after round <code class="highlighter-rouge">T</code>, <code class="highlighter-rouge">x</code> nodes are still uninfected, and
<code class="highlighter-rouge">y</code> nodes are infected. Of course, at <code class="highlighter-rouge">T=0</code>, <code class="highlighter-rouge">x=N-1</code>, and <code class="highlighter-rouge">y=1</code> (i.e., we have
one “infected” node that knows of the message <code class="highlighter-rouge">M</code> and is now out to infect
others with this knowledge). Now because each node picks <code class="highlighter-rouge">f</code> others at random to
gossip with, and because the proportion of uninfected nodes in the cluster is
$\frac{x}{N}$, on average any given infected node will pick $\frac{x}{N}f$
uninfected nodes. Since there are $y$ such infected nodes, on average, we’ll
see $\frac{f}{N}xy$ infected-uninfected interactions in a round. Since an
uninfected node turns into an infected node after receiving the message <code class="highlighter-rouge">M</code>, on
average, each round results in a <em>decrease</em> in the number of uninfected nodes
($x$) by this quantity $\frac{f}{N}xy$. Therefore,</p>
<script type="math/tex; mode=display">\frac{dx}{dT} = -\frac{f}{N}xy</script>
<p>Let $ \beta = \frac{f}{N} $</p>
<p>Then,</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\frac{dx}{dT} &= -\beta xy \\
\implies \frac{dx}{dT} &= -\beta x\left(N-x\right) \\
\implies \frac{dx}{x\left(N-x\right)} &= -\beta dT \\
\implies \int \! \frac{dx}{x\left(N-x\right)} \, dx &= -\beta \int \!dT \tag{1} \label{eqn1}
\end{align*} %]]></script>
<p>Let $\frac{1}{x\left(N-x\right)} = \frac{A}{x} + \frac{B}{N-x} \implies A\left(N-x\right) + Bx = 1$.</p>
<p>Now setting $x=0 \implies A = \frac{1}{N}$, and setting $x=N \implies
B = \frac{1}{N}$. Using this in $\eqref{eqn}$, we have</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\frac{1}{N}\int \! \frac{dx}{x} + \frac{1}{N} \int \! \frac{dx}{N-x} &= -\beta \int \!dT \\
\implies \frac{1}{N} \left( \ln{x} - \ln{\left(N-x\right)} \right) + C &= -\beta T \tag{2} \label{eqn2}
\end{align*} %]]></script>
<p>Where $C$ is the constant of integration. To find it, we note that at $T=0,
x=N-1$. Using this in \eqref{eqn2}, we find that $C=-\frac{ln\left(N-1\right)}{N}$.
Substituting this value for $C$ in \eqref{eqn2} then yields:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\frac{1}{N} \left( \ln{x} - \ln{\left(N-x\right)} - \ln{\left(N-1\right)} \right) &= -\beta T \\
\implies \ln{\frac{x}{N-x}} - \ln{\left(N-1\right)} &= -N\beta T \\
\implies \ln{\frac{x}{N-x}} &= -N\beta T + \ln{\left(N-1\right)} \\
\implies \frac{x}{N-x} &= e^{-N\beta T + \ln{\left(N-1\right)} } = \left(N-1\right)e^{-N\beta T} \\
\implies \frac{N-x}{x} &= {e^{N\beta T}\over N-1} \\
\implies \frac{N}{x} &= 1 + {e^{N\beta T}\over N-1} \\
\implies \frac{x}{N} &= {1\over {1 + {e^{N\beta T}\over N-1}}} = \frac{N-1}{N-1+e^{N\beta T}} \\
\end{align*} %]]></script>
<p>So, as $T$ (number of gossip rounds) grows, the expected value of the fraction
$x\over N$ of uninfected nodes rapidly approaches zero. Since the number of
<em>infected</em> nodes $y=N-x$, we have the proportion of uninfected nodes</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
{y\over N} &= 1 - {x\over N} \\
&= 1 - \frac{N-1}{N-1+e^{N\beta T}} \\
&= \frac{e^{N\beta T}}{N-1+e^{N\beta T}}
\end{align*} %]]></script>
<p>As we grow $T\to\infty$,</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\lim_{T\to\infty}\frac{e^{N\beta T}}{N-1+e^{N\beta T}} &= \\
&= \lim_{T\to\infty}\frac{1}{\frac{N-1}{e^{N\beta T}}+1} \\
&= \frac{1}{\lim_{T\to\infty}\frac{N-1}{e^{N\beta T}}+1} \\
&= \frac{1}{0+1} \\
&= 1
\end{align*} %]]></script>
<p>Hence the expected proportion of infected nodes reaches 100% as the protocol
keeps running. But this begs the question, how long in practice do we have to
wait before the whole cluster gets the new message with high probability? Let’s
call $T_{1\over2}$ the number of rounds when <em>half</em> of the cluster gets the
update, i.e., at $T=T_{h}$, $\frac{x}{N}=\frac{y}{N}=\frac{1}{2}$.
Therefore, from the expression for $\frac{x}{N}$ above,</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\frac{1}{2} &= \frac{N-1}{N-1+e^{N\beta T_{h}}} \\
\implies e^{N\beta T_{h}} &= N-1 \\
\implies N\beta T_h &= \ln\left({N-1}\right) \\
\implies T_h &= \frac{1}{N\beta}\ln\left({N-1}\right)
\end{align*} %]]></script>
<p>Now using $\beta = {f\over N}$, we get $T_h = \frac{1}{f}\ln{\left(N-1\right)}
= O\left(\ln{N}\right)$. Hence, the half life of the process is logarithmic
in the number of participants $N$. Now, to find the number of rounds it takes
for $99\%$ of the uninfected nodes to befome infected can be computed as:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\frac{N-1}{N-1+e^{N\beta T_{0.99}}} = 0.99
\implies N\beta T_{0.99} &= \ln\left(\frac{N-1}{9}\right) \\
\implies T_{0.99} &= {\ln\left(\frac{N-1}{9}\right) \over N\beta } = O\left(\ln{N}\right)\\
\end{align*} %]]></script>
<p>So we see that within a logarithmic number of rounds, a <em>most</em> of the nodes
get infected with high probability.</p>
<h3 id="message-load-per-member">Message load per member</h3>
<p>A dissemination mechanism is no good if it puts unreasonable load on member
nodes, and/or fails to spread out the load evenly across the cluster. In gossip
based dissemination, each node sends out $f$ messages, and receives on
expectation $f\over N$ messages <em>per round</em>. Since $\Theta\left(\ln{N}\right)$
rounds are needed for a message to get to the whole cluster, the load on each
node is also logarithmic in $N$, and every node has similar load. This is quite
nice, since one can grow the cluster to very huge sizes, and even then the load
on each node remains reasonably low.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Gossip based protocols are amazingly simple and robust, which is why they form
crucial elements of failure detection and membership layers of many large scale
distributed systems. Note that I did not talk about nodes failing in this post,
and the focus was on the use of gossip for dissemination. However, gossip itself
can be used to implement failure detectors, like the <a href="https://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf">SWIM failure
detector</a>(very readable paper). Here, every node periodically gossips cluster
membership updates. The protocol is augmented with several features like
acknowledgements and indirect acknowledgements allows the failure detector to
scale with tunable false positive characteristics.</p>
Sun, 10 Sep 2017 00:00:00 +0000
http://yati-sagade.github.io/programming/distributed-systems/2017/09/10/analyzing-gossip/
http://yati-sagade.github.io/programming/distributed-systems/2017/09/10/analyzing-gossip/A simple backup setup using rsync<p>Suppose we have a bunch of files we’d like to have backed up on a remote host
periodically. In this post, I’ll describe my setup that uses <a href="https://rsync.samba.org/">rsync</a> with
cron.</p>
<h2 id="remote-host-setup">Remote host setup</h2>
<p>First we’ll need a place to back up to. I also have a Digital Ocean box, whose
IP address is known to me. Any remote box with sufficient disk space would do:
all we need is a way to access it. I have an entry for my remote box in the
<code class="highlighter-rouge">/etc/hosts</code> file (I call it “dome”):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="c"># /etc/hosts</span>
...
xx.xx.xx.xx dome
</code></pre></div></div>
<p>On this box, I have a <code class="highlighter-rouge">~/backup/</code> directory, which will where all the backed
up files from my local box will land.</p>
<p>For security, I an ssh key added to the remote box to let it identify
connections from my local box. To do this, first generate a keypair if you
haven’t:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>ssh-keygen
</code></pre></div></div>
<p>Now make note of the public key (found in e.g., <code class="highlighter-rouge">~/.ssh/id_rsa.pub</code>) and
ssh to the remote host (using, say, a password). Then, edit the file
<code class="highlighter-rouge">~/.ssh/authorized_keys</code> <strong>on the remote host</strong> and paste your full public
key in there. That’s it. To test if this worked, try sshing again to the remote
host. This time, it should not ask for a password (of course, if your usernames
on local and remote differ, you’ll have to <code class="highlighter-rouge">ssh remote_username@remote_host</code>).</p>
<h2 id="local-setup">Local setup</h2>
<p>I maintain the list of files to be backed up in a file, <code class="highlighter-rouge">~/backup.txt</code>, which
lists a path <em>relative to my home directory</em> per line. The rules are:</p>
<ul>
<li>Files are copied normally.</li>
<li>Symlinks are traversed and their referrents are copied.</li>
<li>Directories, when listed without a trailing slash, are copied recursively and
end up on the backup server with the toplevel directory intact.</li>
<li>Directories, when list <em>with</em> a trailing slash, have their <em>contents copied</em>
out into the remote backup directory.</li>
</ul>
<p>e.g.,</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>notes.org
Documents
Projects
Bak/
</code></pre></div></div>
<p>will backup <code class="highlighter-rouge">~/notes.org</code>, <code class="highlighter-rouge">~/Documents</code> and <code class="highlighter-rouge">~/Projects</code> (the latter two being
directories) to <code class="highlighter-rouge">~/backup/notes.org</code>, <code class="highlighter-rouge">~/backup/Documents</code>, <code class="highlighter-rouge">~/backup/Projects</code>,
respectively. However, the <em>contents of</em> <code class="highlighter-rouge">~/Bak</code> are copied directly to
<code class="highlighter-rouge">~/backup</code>.</p>
<h2 id="rsync">rsync</h2>
<p>The <code class="highlighter-rouge">rsync</code> command is pretty simple:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>rsync <span class="se">\</span>
<span class="nt">-avz</span> <span class="se">\ </span> <span class="c"># copy in [a]rchive mode, [v]erbose, compressed.</span>
<span class="nt">--files-from</span><span class="o">=</span>~/backup.txt <span class="se">\ </span> <span class="c"># our file list.</span>
<span class="nv">$HOME</span> <span class="se">\ </span> <span class="c"># this is prefixed to every path in the file.</span>
ys@dome:~/backup <span class="c"># remote path</span>
</code></pre></div></div>
<h2 id="scheduling-backups">Scheduling backups</h2>
<p>I have an hourly cronjob that runs the above command. To set it up, do</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nv">$ </span>crontab <span class="nt">-e</span>
</code></pre></div></div>
<p>and then add the following line:</p>
<pre><code class="language-cron">
0 * * * * rsync -az --files-from=/home/ys/backup.txt /home/ys ys@dome:~/backup
</code></pre>
<p>and exit. Whenever a new path is added to the backup list, the next run of
our cronjob will copy it over to the remote. For existing files, rsync only
sends the deltas.</p>
Fri, 08 Sep 2017 00:00:00 +0000
http://yati-sagade.github.io/notes/2017/09/08/simple-rsync-backup/
http://yati-sagade.github.io/notes/2017/09/08/simple-rsync-backup/Cryptographic hash functions<p>Hashing is commonly used when we want to reduce a message of an arbitrary size
to a fixed length “digest”, for purposes ranging from integrity checking of
files downloaded from the internet to building blocks for cryptocurrencies.
There are many functions one can use to map the input message to its hash. The
only requirement is that the output be of a known, fixed size. For example,
consider the function</p>
<script type="math/tex; mode=display">H_{mod3}\left(x\right) = x \mod 3</script>
<p>which maps an arbitrarily large value $x$ to a value in the set
$\left\{ 0,1,2 \right\}$. Such hash functions (i.e., based on simple modulo
arithmetic) are used (with other, more complicated hash functions) in
distributed systems (see <a href="https://en.wikipedia.org/wiki/Consistent_hashing">consistent hashing</a>). However, this simple function
has little utility when it comes to integrity checking. For example, we would
like to download a file and check if the file we downloaded was indeed the one
that we intended to download. This can be solved as follows:</p>
<ul>
<li>
<p>Website offering the file lists the file’s hash along with the hashing
function used.</p>
</li>
<li>
<p>We download the file and use the same hashing function on the contents of the
file to get its hash.</p>
</li>
<li>
<p>Next, we compare the hash published on the site with the answer we got, and
conclude that the integrity check passed if the hashes are equal.</p>
</li>
</ul>
<p>It is easy to see that the simple modulo function based hashing will not work at
all here, since given any message value $x$ (the contents of the file), every
third value, i.e., $x+3, x+6, …$ will have the <em>same</em> hash, and an attacker
could fool our integrity checking by adding appropriate content to the original
file.</p>
<p>So what do we need from a good hash function for integrity checking?</p>
<h2 id="1-collision-freedom">1. Collision freedom</h2>
<p>In the above example, we found <em>collisions</em> under the function $H_{mod3}$ –
i.e., we could easily craft a message $x’$ such that $H_{mod3}\left(x\right)
= H_{mod3}\left(x’\right)$ (just pick $x’ = x + 3k$ for any $k \gt 1$). We want
a hash function $H$ such that it is extremely hard to find collisions under it.
Note that since a hash function must transform an arbitrary message into
a <em>fixed size</em> output that is often much smaller than the input size, there will
necessarily be collisions – a large number of them indeed. What is important is
that it is extremely hard to craft a message that does not equal the original
message, but whose hash still matches that of the original message.</p>
<h2 id="2-hiding">2. Hiding</h2>
<p>It should be extremely hard to figure out the original message $x$ from its
hash $H\left(x\right)$. Our modulo function $H_{mod3}$ is actually pretty good
at this — given the remainder of a number of when divided by 3, you cannot
figure out the original number at all, even though you <em>can</em> reduce the number
of candidates by only looking at the numbers that produce the given remainder
when divided by 3. As you might notice, the degree of hiding offered by the
hashing function depends on the range of values that the message variable $x$
can take up: consider trying to hide the outcome of a coin toss. We’ll encode
<script type="math/tex">\text{HEAD}=0</script>, and <script type="math/tex">\text{TAIL}=1</script>.</p>
<script type="math/tex; mode=display">H_{toss}\left(\text{outcome}\right) = \left(1 + \text{outcome}\right) \mod 2</script>
<p>such that $H_{toss}\left(\text{HEAD}\right) = 1$ and $H_{toss}\left(\text{TAIL}\right) = 0$.</p>
<p>Now if the domain ($\left\{\text{HEAD},\text{TAIL}\right\}$) is known to the
attacker, given a hash value 1, they might simply try hashing each value in the
set to find out that the original message was HEAD. The problem here is that the
message domain is easily enumerable. Now if we pick a random integer of, say 256
bits, $r$ from a distribution with a high <a href="https://en.wikipedia.org/wiki/Min_entropy">min entropy</a>, and use its bits
appended to the original message’s bits, we suddenly have an input space which
is extremely hard to enumerate ($2^257$ possibilities). So, now, given a hash
value of 1, it is extremely hard for an attacker without the knowledge of the
random key $r$ to deduce the outcome of the toss. Min entropy of a distribution
is simply the negative logarithm of the probability of the <em>most probable</em> value
being taken up by the random variable distributed according to that
distribution. For example, if you have a loaded die that shows a 6 50% of the
times, with the other faces each having a probability of 10%, the min entropy of
this distribution would be $-\lg{\frac{1}{2}} = 1$, since the most probable
value, 6, comes up with a probability $\frac{1}{2}$. Intuitively, when all
values in a distribution are negligibly likely, and no particular value is
more likely than others, the distribution has a high min entropy, and it is
increasingly difficult to enumerate such a distribution’s values to break the
hiding property of a hash function.</p>
<h2 id="3-puzzle-friendliness">3 “Puzzle friendliness”</h2>
<p>For the sake of using hashes as challenges, as is done in cryptocurrency
implementations, we need an additional property from a good hash function. Given
a key $k$ chosen from a distribution with high min entropy, and a set of target
hash values $Y$, it should be hard to find a value $x$ such that the
concatenation of $k$ and $x$ hashes to a value in $Y$. More precisely, by “hard”
we mean that no strategy of picking $x$ values should be better than randomly
picking $x$ values and trying if $H(k|x) \in Y$.</p>
Sat, 12 Aug 2017 00:00:00 +0000
http://yati-sagade.github.io/notes/2017/08/12/cryptographic-hash-function/
http://yati-sagade.github.io/notes/2017/08/12/cryptographic-hash-function/Doing numbers without the numbers<p>Numbers are abstractions invented by humans to aid with various activities,
mainly counting, and sometimes recreation. While the “three” might be the number
of coins in my pocket right now, the number “three” is in itself an abstract
entity, worthy of study in its own right. It is defined as the successor to the
integer “two”. It is, in general, really hard to define what data is. According
to one definition, we define data in terms of operations possible on it, and
certain constraints on these operations, like an axiomatic system. Note that
under this scheme, the actual representation of the data object in question is
irrelevant: only the external operations on it and a set of properties obeyed by
those operations is enough.</p>
<p>For example, let’s consider natural numbers (0, 1, 2, 3..) as data objects. No
matter how we represent these numbers on a computer, we would like basic
arithmetic properties of numbers to hold, like addition and multiplication of
natural numbers behaving normally, the presence of a total ordering among the
numbers, etc.</p>
<p>On most computers, these numbers are simply represented in their binary form,
usually as 32 or 64 bits. However, like I mentioned above, as long as the
expected operations on natural numbers are available and obey certain
properties, the representation does not matter. One cool way of encoding numbers
is the <a href="https://en.wikipedia.org/wiki/Church_encoding">Church encoding</a>, due to Alonzo Church. In the simplest form, <strong>we
encode a number as the number of applications of a given function $f$</strong>. For
example, $0$ is encoded as <em>no</em> application, $1$ is encoded as just one
application of $f$, while an arbitrary natural number $n$ is encoded as the
function composition $f^n$. Let’s try this out in some Perl 6 code:</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6"># zero($f) returns a function that just returns its argument, i.e., the identity
# function.
sub zero ($f) {
-> $x { $x }
}
sub one ($f) {
-> $x { $f($x) }
}
sub two ($f) {
-> $x { $f($f($x)) }
}</code></pre></figure>
<p>So, for example, <code class="highlighter-rouge">two($foo)</code> is a function that, when called with some parameter
<code class="highlighter-rouge">$x</code>, will apply <code class="highlighter-rouge">$foo</code> twice, i.e., compute <code class="highlighter-rouge">$f($f($x))</code>.</p>
<p>Of course we don’t want to write out all numbers as functions, so we just define
<code class="highlighter-rouge">zero</code>, and a successor function, which transforms a Church encoded natural
number to the next higher natural number.</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6">sub successor ($n) {
-> $f {
-> $x {
$f($n($f)($x))
}
}
}</code></pre></figure>
<p>The key here is the bit <code class="highlighter-rouge">$f($n($f)($x))</code>, which is where the “increment” happens
by applying <code class="highlighter-rouge">$f</code> one more time:</p>
<ul>
<li>The expression <code class="highlighter-rouge">$n($f)</code> is a function that when called with <code class="highlighter-rouge">$x</code>, will apply
<code class="highlighter-rouge">$f</code> for <code class="highlighter-rouge">$n</code> times, to compute $f^n\left(x\right)$</li>
<li>i.e., <code class="highlighter-rouge">$n($f)($x)</code> <em>is</em> the result of applying the composition obtained by
repeating <code class="highlighter-rouge">$f</code> for <code class="highlighter-rouge">$n</code> times to <code class="highlighter-rouge">$x</code>.</li>
<li>Next, we simply add another <code class="highlighter-rouge">$f</code> call to the chain, thus ending up with
<code class="highlighter-rouge">$f($n($f)($x))</code>.</li>
</ul>
<p>Note that the returned value is still a function that takes a function <code class="highlighter-rouge">$f</code>, and
returns another function of the variable <code class="highlighter-rouge">$x</code>, which in turn applies <code class="highlighter-rouge">$f</code> for
$n+1$ times to <code class="highlighter-rouge">$x</code>. This is consistent with the interface we have for <code class="highlighter-rouge">zero</code>,
and more importantly, demonstrates the use of closures: the function of <code class="highlighter-rouge">$x</code>
<em>captures</em> its environment, which contains <code class="highlighter-rouge">$f</code>. Also, the function of <code class="highlighter-rouge">$f</code> (the
outer lambda returned by <code class="highlighter-rouge">successor</code>) captures the function <code class="highlighter-rouge">$n</code> as it was when
provided as an argument to <code class="highlighter-rouge">successor</code>.</p>
<p>To make it even clearer, let’s try to come up with a mechanism to convert from
the Church representation to normal Perl 6 numbers. We want to invent a function
<code class="highlighter-rouge">$g</code> and a value <code class="highlighter-rouge">$a</code>, that when used with any of our Church encoded numbers,
give us the corresponding integers. Concretely, we want:</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6">zero($g)($a) --> 0
one($g)($a) --> 1
successor(successor(one))($g)($a) --> 2
...</code></pre></figure>
<p>One choice could be to use <code class="highlighter-rouge">$g = { $_ + 1 }</code>, i.e., the increment function, with
<code class="highlighter-rouge">$a = 0</code>:</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6">sub increment ($x) { $x + 1 }
sub church-to-human ($n) {
$n(&increment)(0)
}
church-to-human(successor(successor(successor(zero)))).say; #: 3</code></pre></figure>
<p>Once we understand this function, the rest of the operations are easy to derive:</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6">sub zero ($f) {
-> $x { $x }
}
sub successor ($n) {
-> $f {
-> $x {
$f($n($f)($x))
}
}
}
sub add ($y, $z) {
-> $f {
-> $x {
$z($f)($y($f)($x))
}
}
}
sub multiply ($y, $z) {
-> $f {
-> $x {
$y($z($f))($x)
}
}
}</code></pre></figure>
<p>The predecessor function, however, is a bit more involved: given a numeral $n$,
that applies $f$ for $n$ times to an initial value $x$, we want to derive a
function that applies $f$ <em>one less time</em>. While there are multiple ways of
doing this, the easiest to understand for me was to apply $n$ to <em>transformed</em>
versions of $f$ and $x$. Let’s first define a function $g$ that takes a <em>pair</em>
$\left(a, b\right)$ and if $a=0$, returns $\left(0, f(a)\right)$, and if $a=1$,
returns $\left(0, a\right)$. In essence, the first element of the pair tells $g$
if it should <em>skip</em> applying $f$ to the second element. Here’s $g$:</p>
<script type="math/tex; mode=display">% <![CDATA[
g\left(p\right) =
\begin{cases}
\left(0, z\right), & \text{when } p = \left(1, z\right) \\
\left(0, f(z)\right), & \text{when } p = \left(0, z\right)
\end{cases} %]]></script>
<p>Note that if $g$ is composed for $n$ times and applied to a pair $\left(1,
x\right)$, we will in the end have the pair $\left(0,
f^{n-1}\left(x\right)\right)$!</p>
<p>e.g.,</p>
<script type="math/tex; mode=display">g\left(g\left(\left(1, x\right)\right)\right) \\
= g\left(\left(0, x\right)\right) \\
= \left(0, f\left(x\right)\right)</script>
<p>Now Perl 6 of course has lists which we could use to represent our pairs, but
let’s use an alternate representation in the spirit of this post. A pair is
a data object that stores two other objects, and provides the two functions
<code class="highlighter-rouge">first</code> and <code class="highlighter-rouge">second</code> for accessing those objects. This can be implemented like
so:</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6"># The constructor returns a function which takes a function and applies it
# to the contained objects.
sub pair ($x, $y) {
-> $f {
$f($x, $y)
}
}
# Apply the pair object to a function that returns its first argument
sub first ($pair) {
$pair(-> $x, $y { $x })
}
# Apply the pair object to a function that returns its second argument
sub second ($pair) {
$pair(-> $x, $y { $y })
}</code></pre></figure>
<p>To take things even further, let’s also encode the boolean type, and implement
a conditional function:</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6">my $true = -> $fst {
-> $snd {
$fst;
}
};
my $false = -> $fst {
-> $snd {
$snd;
}
};
sub If ($cond, $then, $else) {
$cond($then)($else)
}</code></pre></figure>
<p>Now let’s implement the predecessor function, finally:</p>
<figure class="highlight"><pre><code class="language-perl6" data-lang="perl6">sub pred ($n) {
-> $f {
-> $x {
my $g = -> $pair {
If(first($pair),
pair($false, second($pair)),
pair($false, $f(second($pair))))
};
my $p = pair($true, $x);
second($n($g)($p));
}
}
}
# Subtraction of $z from $y is the same as finding the $z'th predecessor of $y,
# which means we apply `pred` to $y for $z times, which is readily done by
# calling $z with pred().
sub subtract ($y, $z) {
-> $f {
-> $x {
$z(&pred)($y)($f)($x)
}
}
}</code></pre></figure>
Sun, 30 Jul 2017 00:00:00 +0000
http://yati-sagade.github.io/2017/07/30/church-numerals/
http://yati-sagade.github.io/2017/07/30/church-numerals/