summaryrefslogtreecommitdiff
path: root/content/slide
diff options
context:
space:
mode:
authorAndreas Baumann <mail@andreasbaumann.cc>2017-02-05 10:48:36 +0100
committerAndreas Baumann <mail@andreasbaumann.cc>2017-02-05 10:48:36 +0100
commitb7b99ab942e2521553dfdb070c4b0abd88d12e71 (patch)
tree3d6c046b0184e5cbf56e634868064c74cd549838 /content/slide
parent40aab6c7b06b9c4cb4288d3e692b06d5d2e5a89b (diff)
downloadwww-andreasbaumann-cc-b7b99ab942e2521553dfdb070c4b0abd88d12e71.tar.gz
www-andreasbaumann-cc-b7b99ab942e2521553dfdb070c4b0abd88d12e71.tar.bz2
changed website and blog to use Hugo
Diffstat (limited to 'content/slide')
-rw-r--r--content/slide/_index.md13
-rw-r--r--content/slide/obs.md243
-rw-r--r--content/slide/struswebservice.md414
3 files changed, 670 insertions, 0 deletions
diff --git a/content/slide/_index.md b/content/slide/_index.md
new file mode 100644
index 0000000..c8a8771
--- /dev/null
+++ b/content/slide/_index.md
@@ -0,0 +1,13 @@
++++
+title = "Slides"
++++
+
+# Presentations
+
+## Build Systems
+
+[OBS Build System](/slide/obs) used for strus packaging.
+
+## Strus
+
+[Strus Web Service](/slide/struswebservice): what it is and roadmap.
diff --git a/content/slide/obs.md b/content/slide/obs.md
new file mode 100644
index 0000000..5c9d336
--- /dev/null
+++ b/content/slide/obs.md
@@ -0,0 +1,243 @@
++++
+title = "OpenBuild Service"
++++
+
+ <div class="reveal">
+
+ <!-- Any section element inside of this container is displayed as a slide -->
+ <div class="slides">
+ <section>
+ <h1>Open Build Service</h1>
+ <h3>Fighting the Linux binary distribution problem..</h3>
+ <p>
+ <small>Created by <a href="http://www.andreasbaumann.cc">Andreas Baumann</a></small>
+ </p>
+ <p>
+ <small><a href="http://www.andreasbaumann.cc/slide/obs">http://www.andreasbaumann.cc/slide/obs</a></small>
+ </p>
+ <p><small>(C)2015</small></p>
+ </section>
+
+ <section>
+ <h2>The Linux Binary Distribution Disaster</h2>
+ <p>
+ Linus Torvalds: Provides MacX and Windows binaries for his <i>divelog</i> application
+ but no Linux binaries, why?
+ </p>
+ <p>
+ There are no Linux binaries: there are Fedora binaries, Ubuntu
+ binaries, ArchLinux binaries, etc..
+ </p>
+ </section>
+
+ <section>
+ <h2>Linux binaries differ..</h2>
+ <ul>
+ <li>..in package formats (deb, rpm, tgz, ...)</li>
+ <li>..in build rules (compiler flags, fencing, optimization, ...)</li>
+ <li>Linux Standard Base (LSB): should actually guarantee ABI and API stability..</li>
+ <li>The problem is, there is still too much variation..</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Shared libraries</h2>
+ <ul>
+ <li>share common code on a platform, good idea</li>
+ <li>there is not a common view of what can be shared and should be shared</li>
+ <li>packet names and versions differ</li>
+ <li>sometimes compilation flags differ (e.g. no threading support in sqlite3 on RHEL5)</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Distribution types</h2>
+ <ul>
+ <li>Frozen releases, e.g. Debian 8, RHEL 7, Ubuntu LTS, ...,
+ long release cycles (4-8 years), they are stable,
+ but only if you use very old software</li>
+ <li>Rolling releases, e.g. Fedora, ArchLinux, Gentoo, ...
+ You have the newest and shiniest software, but
+ stability may suffer, backporting to other distros
+ gets almost impossible.
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Options for software developers</h2>
+ <p>Islands, part 1</p>
+ <ul>
+ <li>Create your own universe: examples Java, Go, Python, ...
+ </li>
+ <li>You deploy your software as a blob of static libraries,
+ there might be sharing inside your universe but not between
+ universes.
+ </li>
+ <li>In C++ we are "stuck" to the ELF format, shared libarires, binaries
+ in distribution packages. Build on one Linux (which one?) and hope
+ it runs everywhere
+ </li>
+ <li>Local libraries must be maintained (security patches), imagine
+ a bundled, unpatched, 5 years old libssl. :-)
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Package meta-systems</h2>
+ <p>Islands, part 2</p>
+ <ul>
+ <li><a href="http://www.openpkg.org/">http://www.openpkg.org/</a>: cross-platform RPMs</li>
+ <li>BSD-like port systems (gentoo, netsrc, etc.): but this means going back to distribution of source code only</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Package build systems</h2>
+ <p>Islands, part 3<p>
+ <ul>
+ <li><a href="http://build.opensuse.org">http://build.opensuse.org</a>: Open Build Service (for OpenSuSE)</li>
+ <li><a href="http://launchpad.net">http://launchpad.net</a>: Ubuntu only</li>
+ <li><a href="http://copr.fedoraproject.org">http://copr.fedoraproject.org</a>, koji: Fedora only</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Package build systems</h2>
+ <p>Islands, part 3</p>
+ <ul>
+ <li>Linux distros try to maintain their bleeding edge product with the help of the community.</li>
+ <li>Naturally communities form islands</li>
+ <li>Exception: OpenBuild system, they build SuSE, RHEL, Debian, Ubuntu, Mandriva, Arch,..</li>
+ <li>This "behaviour" should be honoured and supported as it is the really only way to go right now</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>OBS</h2>
+ <ul>
+ <li>Basically having one RPM SPEC-file and one Debian/Ubuntu tree covers the majority of Linux distributions</li>
+ <li>Choose your dependencies wisely and provide what's not available on old distros on your own.</li>
+ <li>Either bundle your stub packages with your software package or package them into a 3rdParty meta package</li>
+ <li>Make sure package names DON'T collide!</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Automatization</h2>
+ <ul>
+ <li>Combine it with github, build the source trees there (git tag)</li>
+ <li>Webhooks draw the tarballs and spec files to the OBS cluster</li>
+ <li>Publish releases on a web space</li>
+ <li>Keep the current git master as rolling last release of your software</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Current process</h2>
+ <ul>
+ <li>Update in CMakeLists.txt:
+ <pre><code class="hljs" data-trim contenteditable>
+set( STRUS_MAJOR_VERSION 0 )
+set( STRUS_MINOR_VERSION 2 )
+set( STRUS_PATCH_VERSION 1 )
+ </code></pre>
+ </li>
+ <li>Adapt %changelog and dist/debian/changelog. Keep STRICTLY to syntax! For tipps see also: <a href="http://keepachangelog.com/">http://keepachangelog.com/</a></li>
+ <li>Commit the master revision and tag it</li>
+ <pre><code class="hljs" data-trim contenteditable>
+git commit -a -m 'changed to release 0.2.1'
+git tag 0.2.1
+ </code></pre>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Current process</h2>
+ <ul>
+ <li>Deploy to OBS workspace:
+ <pre><code class="hljs" data-trim contenteditable>
+osc co home:PatrickFrey (into $HOME/home:PatrickFrey)</li>
+ </code></pre>
+ <pre><code class="hljs" data-trim contenteditable>
+cmake .
+dist/obs/deploy_to_obs.sh
+ </code></pre>
+ </li>
+
+ <li>Remove source tarballs of old release, add tarballs of new release:
+
+ <pre><code class="hljs" data-trim contenteditable>
+cd home:PatrickFrey/strus
+osc up
+osc rm strus-0.1.6*
+osc add strus-0.2.1*
+osc commit -m 'added version 0.2.1'
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Issues</h2>
+ <ul>
+ <li>dist/obs/deploy_to_obs.sh should be eliminated</li>
+ <li>Release versions need patching, the mirroring system plays mean tricks otherwise..</li>
+ <li>How to automatically update the OSC workspace, not having tarballs there, etc.</li>
+ <li>Have a central changelogs, create Debian and RHEL style from it</li>
+ <li>git commit is always needed, otherwise tarball lacks modified/added artifacts</li>
+ <li>Don't change git tags later on! Rather revoke broken releases.</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Local OSB builds</h2>
+ <ul>
+ <li>Build locally in a chrooted envorinment:
+ <pre><code class="hljs" data-trim contenteditable>
+cd home:PatrickFrey/strus
+osc build --noinit Debian_8.0 x86_64 strus.dsc
+osc build --noinit xUbuntu_12.04 x86_64 strus-xUbuntu_12.04.dsc
+osc build --noinit CentOS_5 i586 strus.spec
+ </code></pre>
+ </li>
+ <li>Access a shell in the chroot to investigate issues:</li>
+ <pre><code class="hljs" data-trim contenteditable>
+sudo chroot /var/tmp/build-root/xUbuntu_12.04-x86_64/ /bin/bash --login
+cat /etc/debian_version
+wheezy/sid
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Local OSB builds</h2>
+ <ul>
+ <li>The artifacts of the build system are somewhat hidden in the chroot:</li>
+ <pre><code class="hljs" data-trim contenteditable>
+/usr/src/packages/BUILD/
+/home/abuild/rpmbuild/BUILD/strus-0.2.1
+/usr/src/packages/BUILD/strus-0.2.1
+ </code></pre>
+ </li>
+ <li>The chrooted environment lacks things like an editor or a debugger or
+ even a package manager inside the chroot.
+ Either add a dependency for them 'BuildRequires:' or use
+ the package manager from outside the chroot (works nice on Archlinux
+ and for Ubuntu on Ubuntu, 'apt-get' on RHEL might be a problem).
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Links</h2>
+ <ul>
+ <li>OSC OpenBuild Service Command Line: <a href="http://openbuildservice.org/download/">http://openbuildservice.org/download/</a>
+ </li>
+ </ul>
+
+ </section>
+ </div>
+ </div>
diff --git a/content/slide/struswebservice.md b/content/slide/struswebservice.md
new file mode 100644
index 0000000..437b206
--- /dev/null
+++ b/content/slide/struswebservice.md
@@ -0,0 +1,414 @@
++++
+title = "Strus Web Service"
++++
+
+ <div class="reveal">
+
+ <!-- Any section element inside of this container is displayed as a slide -->
+ <div class="slides">
+ <section>
+ <h1>Strus - Web Service</h1>
+ <p>
+ <small>Created by <a href="http://www.andreasbaumann.cc">Andreas Baumann</a></small>
+ </p>
+ <p>
+ <small><a href="http://www.andreasbaumann.cc/slide/struswebservice">http://www.andreasbaumann.cc/slide/struswebservice</a></small>
+ </p>
+ <p><small>(C)2015/2016</small></p>
+ </section>
+
+ <section>
+ <h2>Outline</h2>
+ <ul>
+ <li>Have a web service, language independent, for search.</li>
+ <li>The software to the vision exists, it's called <b>Elasticsearch.</b></li>
+ <li>Based on Lucene, so we do the same with Strus</li>
+ <li>Currently there is a Java API which uses this service</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Strus web service</h2>
+ <ul>
+ <li>uses JSON and HTTP GET/POST</li>
+ <li>implemented using <a href="http://cppcms.com/wikipp/en/page/main">CppCMS</a></li>
+ <li>uses only the <a href="http://patrickfrey.github.io/strus/doc/doxygen/html/index.html">Strus core API</a> currently</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Design rationales</h2>
+ <ul>
+ <li>standard technology (HTTP, JSON)</li>
+ <li>CRUD (all basic functions exists for all classes)</lib>
+ <li>proxyable and securable</li>
+ <li>zero-configuration</li>
+ <li>scalable</li>
+ <li>C++</li>
+ <li>MPL v2</li>
+ <li>runs on Linux, OSX, FreeBSD</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Service functions</li>
+ <li>Index functions</li>
+ <li>Document functions</li>
+ <li>Query functions</li>
+ <li>Transactions</li>
+ <li>Statistics</li>
+ <li>Introspection and Configuration</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Manipulate indexes:
+ <pre><code class="hljs" data-trim contenteditable>
+curl -XPOST -H 'Content-Type: application/json'
+http://localhost:8080/strus/index/create/A
+'-d { "params" : { } }'
+
+{"result":"ok"}
+
+curl -XPOST http://localhost:8080/strus/index/delete/A
+{"result":"ok"}
+
+curl -XPOST http://localhost:8080/strus/index/exists/A
+{"exists":false,"result":"ok"}
+
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Insert document:
+ <pre><code class="hljs" data-trim contenteditable>
+curl -XPOST -H 'Content-Type: application/json'
+http://localhost:8080/strus/document/insert/A
+-d
+{ "doc" : { "docid" : "doc3",
+ "attributes" : [
+ { "key" : "title", "value" : "This is a Hello World Document" },
+ { "key" : "attr1", "value" : "val1" },
+ { "key" : "attr2", "value" : "val2" } ],
+ "metadata" : [
+ { "key" : "doclen", "value" : 23773 }, { "key" : "docweight", "value" : 3.1415 } ],
+ "forward" : [
+ { "type" : "word", "value" : "Hello", "pos": 1 },
+ { "type" : "word", "value" : "World", "pos" : 2 } ],
+ "search" : [
+ { "type" : "Word", "value" : "hello", "pos": 1 },
+ { "type" : "word", "value" : "world", "pos" : 2 } ] }
+}
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Query:
+ <pre><code class="hljs" data-trim contenteditable>
+curl -XPOST -H 'Content-Type: application/json'
+http://localhost:8080/strus/query/A
+-d
+{ "query": {
+ "first_rank": 0,
+ "nof_ranks": 20,
+ "weighting": {
+ "scheme": {
+ "name": "bm25",
+ "params": [
+ {
+ "key": "b",
+ "value": 0.75
+ },
+ {
+ "key": "k1",
+ "value": 1.0001
+ },
+ {
+ "key": "avgdoclen",
+ "value": 11943
+ }
+ ]
+ }
+ },
+ "summarizer": [
+ {
+ "attribute": "attribute",
+ "name": "attribute",
+ "params": [
+ {
+ "key": "name",
+ "value": "docid"
+ }
+ ]
+ }
+ ],
+ "features": [
+ {
+ "name": "feat",
+ "value": {
+ "term": {
+ "type": "word",
+ "value": "hello"
+ }
+ },
+ "weight": 1
+ },
+ {
+ "name": "sel",
+ "value": {
+ "expression": {
+ "operator": "union",
+ "range": 0,
+ "cardinality": 0,
+ "terms": [
+ {
+ "term": {
+ "type": "word",
+ "value": "hello"
+ }
+ }
+ ]
+ }
+ },
+ "weight": 1
+ }
+ ],
+ "select": [
+ "sel"
+ ]
+}
+
+{
+ "execution_time": 0.000983791,
+ "ranklist": {
+ "documents_ranked": 1,
+ "documents_visited": 1,
+ "passes_evaluated": 0,
+ "ranks": [
+ {
+ "attributes": [
+ {
+ "key": "docid",
+ "value": "doc3"
+ }
+ ],
+ "docno": 1,
+ "weight": 0
+ }
+ ]
+ },
+ "result": "ok"
+}
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Transactions:
+ <pre><code class="hljs" data-trim contenteditable>
+curl -XPOST -H 'Content-Type: application/json'
+http://localhost:8080/strus/transaction/begin/A/T1
+
+curl -XPOST -H 'Content-Type: application/json'
+http://localhost:8080/strus/document/insert/A
+-d
+{ "transaction" : { "id" : "T1" },
+ "doc" : { "docid" : "doc1",
+ ...
+}
+
+curl -XPOST -H 'Content-Type: application/json'
+http://localhost:8080/strus/transaction/commit/A/T1
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Introspection (document):
+ <pre><code class="hljs" data-trim contenteditable>
+curl -XPOST -H 'Content-Type: application/json'
+http://localhost:8080/strus/document/get/A/doc3
+
+{
+ "doc": {
+ "attributes": [
+ {
+ "key": "attr1",
+ "value": "val1"
+ },
+ {
+ "key": "attr2",
+ "value": "val2"
+ },
+ {
+ "key": "docid",
+ "value": "doc3"
+ },
+ {
+ "key": "title",
+ "value": "This is a Hello World Document"
+ }
+ ],
+ "docno": 1,
+ "forward": [
+ {
+ "pos": 1,
+ "type": "word",
+ "value": "Hello"
+ },
+ {
+ "pos": 2,
+ "type": "word",
+ "value": "World"
+ }
+ ],
+ "metadata": [
+ {
+ "key": "docweight",
+ "value": 3.141499996185303
+ },
+ {
+ "key": "doclen",
+ "value": 23773
+ },
+ {
+ "key": "date",
+ "value": 0
+ }
+ ],
+ "search": [
+ {
+ "pos": 1,
+ "type": "word",
+ "value": "hello"
+ },
+ {
+ "pos": 2,
+ "type": "word",
+ "value": "world"
+ }
+ ]
+ },
+ "execution_time": 0.00028157,
+ "result": "ok"
+}
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Statistics and introspection (index):
+ <pre><code class="hljs" data-trim contenteditable>
+curl http://localhost:8080/strus/index/stats/A
+
+{"result":"ok","stats":{"nof_docs":1}}
+
+curl http://localhost:8080/strus/index/config/A
+
+{"config":{
+ "attributes":["attr1","attr2","docid","title"],
+ "metadata":[{"name":"date","type":"UInt16"},{"name":"doclen","type":"UInt16"},{"name":"docweight","type":"Float32"}],
+ "types":["word"]},
+"result":"ok"}
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Protocol</h2>
+ <ul>
+ <li>Statistics and introspection (system):
+ <pre><code class="hljs" data-trim contenteditable>
+curl http://localhost:8080/strus/config
+{"config":{
+ "posting_join_operators":[
+ "chain","chain_struct","contains","diff","inrange","inrange_struct","intersect","pred","sequence","sequence_struct","succ","union","within","within_struct"],
+ "summarizer_functions":[
+ "accuvariable","attribute","matchphrase","matchpos","matchvariables","metadata"],
+ "weighting_funtions":[
+ "bm25","formula","metadata","td","tf"]},
+"result":"ok"}
+
+ ...
+ "weighting_functions": [
+ {
+ "description": "Calculate the document weight with the weighting scheme \"BM25\"",
+ "name": "bm25",
+ "parameter": [
+ {
+ "description": "defines the query features to weight",
+ "name": "match",
+ "type": "feature"
+ },
+ {
+ "description": "parameter of the BM25 weighting scheme",
+ "name": "k1",
+ "type": "numeric"
+ },
+ ...
+ </code></pre>
+ </li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Strus web service</h2>
+ <h3>Links</h3>
+ <ul>
+ <li><a href="http://github.com/Eurospider/strusWebService">http://github.com/Eurospider/strusWebService</a></li>
+ <li><a href="https://build.opensuse.org/package/show/home:andreas_baumann/struswebservice">packages and builds on OBS</a></li>
+ <li><a href="http://strustest.eurospider.com/strus/version">Test instance</a></li>
+ <li><a href="http://strustest.eurospider.com/strus/democlient/">Simple AJAX/JS demo interface</a></li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Java API</h2>
+ <ul>
+ <li>Uses latest Java 1.8 (aka requires Java 1.8)</li>
+ <li>Uses Jackson for POJO serialization from/to JSON</li>
+ <li>uses javax.ws.rs.client web service classes for HTTP</li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Java API</h2>
+ <h3>Links</h3>
+ <ul>
+ <li><a href="http://github.com/Eurospider/strusJavaAPI">http://github.com/Eurospider/strusJavaAPI</a></li>
+ <li><a href="http://eurospider.github.io/strusJavaApi/apidocs/index.html">JavaDoc</a></li>
+ </ul>
+ </section>
+
+ <section>
+ <h2>Todos and future</h2>
+ <ul>
+ <li>support all functions from strus API</li>
+ <li>integrate strusAnalyzer and strusStream APIs</li>
+ <li>start with simple cluster mode (distribution proxy service)</li>
+ </ul>
+ </section>
+ </div>
+ </div>