Artifact Content
Not logged in

Artifact 67667043acfefb1aab2fd779d89e73fbe452cd0e:

Wiki page [Application Example: Silktorrent Package Manager] by martin_vahi on 2017-10-08 07:02:26.
D 2017-10-08T07:02:26.908
L Application\sExample:\sSilktorrent\sPackage\sManager
P fed1db21665dc53112c0c553f89f12f901242d0e
U martin_vahi
W 8652
<h1 style="text-align: left;">The Core Idea</h1>

<p>The Silktorrent packets are tar-files that are described by their secure
hash, not storage location. That allows the tar-files to be distributed from
multiple storage locations, "mirrors", without giving any party the opportunity
to change the files without making the files to fail their integrity checks.
The "mirrors" give the redundancy and part of the censorship countermeasures.&nbsp;</p>

<p><br>
</p>

<h1>Theory</h1>

<div>There is an assumption that <b>backwards compatibility of libraries does
not exist, even if advertised</b>. Whenever a software component dependency is
declared by stating that the version of the dependency is "newest" or "greater
than X", a flaw is introduced. That is the reason, why Linux/BSD/etc. package
collections are "unstable" and NEVER WILL BE "stable". For built/compiled
software components <b>each combination of build parameters, compiler,
environment set-up is a separate version of the software component</b>.&nbsp;</div>

<div><br>
</div>

<div><br>
</div>

<div><br>
</div>

<h1>Implementation Notes</h1>

<div><br>
</div>

<h2>API for the WWW and Data Storage Devices</h2>

<div>Silktorrent packets, the tar-files with the file extension of "stblob",
are served by an ordinary web server the same way downloadable files, for
example JPEG files, are generally served. The stblob-files are in folders that
have arbitrary names<i>(file system implementation limits like the maximum name
length, do apply)</i> and in addition to the stblob-files those folders can
also contain files that are not stblob-files. One of those <b>optionally
present files is a text file named find_stblob.txt</b>.&nbsp;</div>

<div><br>
</div>

<div>The format of the find_stblob.txt is:<br>
</div>

<div><br>
</div>
<pre>&nbsp; &nbsp; find . -name '*.stblob' &gt; ./find_stblob.txt</pre>

<div><br>
</div>

<div>except that the <b>find_stblob.txt is also allowed to contain blank lines
and the classical comments that start with the # and/or //</b> . A path from
the file system directory tree leaf to the file system directory tree root is
allowed to contain N_find_stblob_txt instances of the find_stblob.txt, where
the&nbsp;</div>

<div><br>
</div>
<pre>&nbsp; &nbsp; N_find_stblob_txt is in range \[0, &amp;lt;number of vertices on the path&amp;gt;\]</pre>

<div>The find_stblob.txt does not need to be static, for example, its content
may change multiple times per second.&nbsp;<b>stblob-files that are listed at
the find_stblob.txt are NOT guaranteed to be present and stblob-files are
allowed to go offline in the midst of their download sessions. </b>In the case
of data storage devices like USB sticks and DVDs the web server and internet
connection parts are &nbsp;omitted and the client is meant to read the
find_stblob.txt and the stblob-files directly from the data storage device,
"the disk".</div>

<div><br>
</div>

<div><br>
</div>

<div><br>
</div>

<h2>An Optional Storage Allocation Policy</h2>

<div>To make the Silktorrent network of package/packet hosting servers &nbsp;more
reliable, the package/packet hosting servers should use <b>multiple storage
allocation policies simultaneously.</b> There is one allocation agent per
policy. Each allocation agent has its own, fixed, size of storage space, disk
space, which will be allocated to Silktorrent packets according to the policy
that the agent implements. Some agent, bot, may sell paid subscriptions like
the Dropbox and alike use. Some agent might run a mirroring service in favor of
some public library or operating system packages repository. Some agent might
store Silktorrent packets according to popularity. Some agent may offer
personal storage service to the owner of the server. Some agent may service
some Silktorrent based messaging service. Some agent might service Silktorrent
based "web" (PDF-files, LibreOffice files, all-in-one-HTML-documents, etc.)</div>

<div><br>
</div>

<div>To avoid duplicate copies of the same Silktorrent packet at the same
storage server, the agents of a single storage server may use a single "storage
engine" that keeps track of the storage space quotas of the agents and
physically deletes the Silktorrent packet only, if no agent on this sotorage
server wants to store that Silktorrent packet.</div>

<div><br>
</div>

<div><br>
</div>

<div><br>
</div>

<h2>Storage Size Requirements</h2>

<div>The hashes within the Silktorrent packet names depend on the tar files,
silktorrent packets. Hash of a tar-file depends on the file attributes,
including the date-and-time attributes. That's why it is not possible to
recreate a Silktorrent packet from unmodified content of an unpacked
Silktorrent packet without some special hacking, meaning: copies of downloaded
Silktorrent packets must not be deleted, if they are going to be passed on
somewhere. To use the downloaded Silktorrent packets without un-tar-ing them
and waiting for the slow HDD operation to finish, an unmodified set of
un-tar-red Silktorrent packets must be stored. Given the huge amount of
Silktorrent packets, the packets must be distributed among a set of folders,
because otherwise the file systems are slow to use. The recursive folder names
may be derived from the first letters of Silktorrent packet names. The
Silktorrent packet names were intentionally designed to contain equally
distributed letters at the start of their packet names, because that allows
database indices that index the Silktorrent packet names to probabilistically
work faster.</div>

<div><br>
</div>

<div>Silktorrent packets, tar-files, that contain software/datacollection forks
or newer versions of the software/datacollection, might consist of many
bitstreams that match with the original version of the software/datacollection
Silktorrent packets. Storage space for storing a collection of tar-files that
contain relatively long, common, bitstreams, can be reduced by storing the
common, relatively long, bitstreams only once. The search for &nbsp;the common
bitstream tokens can be greatly mitigated by explicitly telling, which
Silktorrent packets, tar-files, are version wise or fork wise closely related.
The arising <a href="http://www.or.deis.unibo.it/knapsack.html">knapsack
problems</a> might be solved by using specialized
<a href="http://or.dei.unibo.it/library">open source libraries</a>. Compression
algorithm development and the search for common bitstream tokens might get some
inspiration from genetics software, which might be even re-used in some cases.
It is OK for the compression software to run as a not-that-well-optimized
background task, as long as the decompression is really fast.</div>

<div><br>
</div>

<div><br>
</div>

<div><br>
</div>

<h1>Example Use Cases</h1>

<div>

<p>A Silktorrent packet is a tar-file that contains folders&nbsp;</p>

<p><b>payload</b></p>

<p><b>header</b></p>

<p>In software projects and HTML pages &nbsp;the content of the folders&nbsp;<b>payload</b>&nbsp;and&nbsp;<b>header</b>&nbsp;can
be referenced by using the&nbsp;<a href="http://bitrary.softf1.com/index.php?title=Art_:_Source_Code#Tree_Encoding">tree
encoding</a><i>(<a href="https://archive.li/YDzvD">archival copy</a>)</i>:</p>

<p><b>&lt;tar-file name&gt;/payload/&lt;the file or folder relative to the
folder payload&gt;</b></p>

<p><b>&lt;tar-file name&gt;/header/&lt;the file or folder relative to the
folder header&gt;</b><br>
</p>

<p>The&nbsp;&lt;tar-file name&gt; can be the name of a local folder or some
folder at some publicly hosted web page. The include/require/src/uses
statements of various programming languages and configurations files, including
HTML, can be modified by changing the prefix of the &lt;tar-file name&gt;. That
allows an HTML-page to be switched from using JavaScript libraries from one
site to using the very same JavaScript libraries from another site.</p>

<p><b>http://www.first_site.com/</b>&lt;tar-file name&gt;/payload/the_JavaScript_library.js<br>
</p>

<p><b>http://www.second_site.com/</b>&lt;tar-file name&gt;/payload/the_JavaScript_library.js<br>
</p>

<p>Due to the possibility to use regular expressions, the proposed solution
does not require code generation. It might be usable with many existing IDE-s
without requiring any additional tools or plugins. The solution is also
programming language agnostic.</p></div>

<div><br>
</div>

<div><br>
</div>

<div><br>
</div>

<h1>Partial list of Similar Software</h1>

<div>
<ul>
<li><a href="http://www.toastball.net/toast/">toast</a> <i>(<a href="https://archive.is/uIRaz">archival
copy</a>, <a href="./ainfo/a99d7f95d0d8255e">source code</a>)</i></li>
</ul></div>

<div><br>
</div>

Z c38916ecbc1e556a852d3d7fbf03ae62