diff options
author | Andreas Baumann <mail@andreasbaumann.cc> | 2017-02-05 10:48:36 +0100 |
---|---|---|
committer | Andreas Baumann <mail@andreasbaumann.cc> | 2017-02-05 10:48:36 +0100 |
commit | b7b99ab942e2521553dfdb070c4b0abd88d12e71 (patch) | |
tree | 3d6c046b0184e5cbf56e634868064c74cd549838 /content/slide | |
parent | 40aab6c7b06b9c4cb4288d3e692b06d5d2e5a89b (diff) | |
download | www-andreasbaumann-cc-b7b99ab942e2521553dfdb070c4b0abd88d12e71.tar.gz www-andreasbaumann-cc-b7b99ab942e2521553dfdb070c4b0abd88d12e71.tar.bz2 |
changed website and blog to use Hugo
Diffstat (limited to 'content/slide')
-rw-r--r-- | content/slide/_index.md | 13 | ||||
-rw-r--r-- | content/slide/obs.md | 243 | ||||
-rw-r--r-- | content/slide/struswebservice.md | 414 |
3 files changed, 670 insertions, 0 deletions
diff --git a/content/slide/_index.md b/content/slide/_index.md new file mode 100644 index 0000000..c8a8771 --- /dev/null +++ b/content/slide/_index.md @@ -0,0 +1,13 @@ ++++ +title = "Slides" ++++ + +# Presentations + +## Build Systems + +[OBS Build System](/slide/obs) used for strus packaging. + +## Strus + +[Strus Web Service](/slide/struswebservice): what it is and roadmap. diff --git a/content/slide/obs.md b/content/slide/obs.md new file mode 100644 index 0000000..5c9d336 --- /dev/null +++ b/content/slide/obs.md @@ -0,0 +1,243 @@ ++++ +title = "OpenBuild Service" ++++ + + <div class="reveal"> + + <!-- Any section element inside of this container is displayed as a slide --> + <div class="slides"> + <section> + <h1>Open Build Service</h1> + <h3>Fighting the Linux binary distribution problem..</h3> + <p> + <small>Created by <a href="http://www.andreasbaumann.cc">Andreas Baumann</a></small> + </p> + <p> + <small><a href="http://www.andreasbaumann.cc/slide/obs">http://www.andreasbaumann.cc/slide/obs</a></small> + </p> + <p><small>(C)2015</small></p> + </section> + + <section> + <h2>The Linux Binary Distribution Disaster</h2> + <p> + Linus Torvalds: Provides MacX and Windows binaries for his <i>divelog</i> application + but no Linux binaries, why? + </p> + <p> + There are no Linux binaries: there are Fedora binaries, Ubuntu + binaries, ArchLinux binaries, etc.. + </p> + </section> + + <section> + <h2>Linux binaries differ..</h2> + <ul> + <li>..in package formats (deb, rpm, tgz, ...)</li> + <li>..in build rules (compiler flags, fencing, optimization, ...)</li> + <li>Linux Standard Base (LSB): should actually guarantee ABI and API stability..</li> + <li>The problem is, there is still too much variation..</li> + </ul> + </section> + + <section> + <h2>Shared libraries</h2> + <ul> + <li>share common code on a platform, good idea</li> + <li>there is not a common view of what can be shared and should be shared</li> + <li>packet names and versions differ</li> + <li>sometimes compilation flags differ (e.g. no threading support in sqlite3 on RHEL5)</li> + </ul> + </section> + + <section> + <h2>Distribution types</h2> + <ul> + <li>Frozen releases, e.g. Debian 8, RHEL 7, Ubuntu LTS, ..., + long release cycles (4-8 years), they are stable, + but only if you use very old software</li> + <li>Rolling releases, e.g. Fedora, ArchLinux, Gentoo, ... + You have the newest and shiniest software, but + stability may suffer, backporting to other distros + gets almost impossible. + </li> + </ul> + </section> + + <section> + <h2>Options for software developers</h2> + <p>Islands, part 1</p> + <ul> + <li>Create your own universe: examples Java, Go, Python, ... + </li> + <li>You deploy your software as a blob of static libraries, + there might be sharing inside your universe but not between + universes. + </li> + <li>In C++ we are "stuck" to the ELF format, shared libarires, binaries + in distribution packages. Build on one Linux (which one?) and hope + it runs everywhere + </li> + <li>Local libraries must be maintained (security patches), imagine + a bundled, unpatched, 5 years old libssl. :-) + </li> + </ul> + </section> + + <section> + <h2>Package meta-systems</h2> + <p>Islands, part 2</p> + <ul> + <li><a href="http://www.openpkg.org/">http://www.openpkg.org/</a>: cross-platform RPMs</li> + <li>BSD-like port systems (gentoo, netsrc, etc.): but this means going back to distribution of source code only</li> + </ul> + </section> + + <section> + <h2>Package build systems</h2> + <p>Islands, part 3<p> + <ul> + <li><a href="http://build.opensuse.org">http://build.opensuse.org</a>: Open Build Service (for OpenSuSE)</li> + <li><a href="http://launchpad.net">http://launchpad.net</a>: Ubuntu only</li> + <li><a href="http://copr.fedoraproject.org">http://copr.fedoraproject.org</a>, koji: Fedora only</li> + </ul> + </section> + + <section> + <h2>Package build systems</h2> + <p>Islands, part 3</p> + <ul> + <li>Linux distros try to maintain their bleeding edge product with the help of the community.</li> + <li>Naturally communities form islands</li> + <li>Exception: OpenBuild system, they build SuSE, RHEL, Debian, Ubuntu, Mandriva, Arch,..</li> + <li>This "behaviour" should be honoured and supported as it is the really only way to go right now</li> + </ul> + </section> + + <section> + <h2>OBS</h2> + <ul> + <li>Basically having one RPM SPEC-file and one Debian/Ubuntu tree covers the majority of Linux distributions</li> + <li>Choose your dependencies wisely and provide what's not available on old distros on your own.</li> + <li>Either bundle your stub packages with your software package or package them into a 3rdParty meta package</li> + <li>Make sure package names DON'T collide!</li> + </ul> + </section> + + <section> + <h2>Automatization</h2> + <ul> + <li>Combine it with github, build the source trees there (git tag)</li> + <li>Webhooks draw the tarballs and spec files to the OBS cluster</li> + <li>Publish releases on a web space</li> + <li>Keep the current git master as rolling last release of your software</li> + </ul> + </section> + + <section> + <h2>Current process</h2> + <ul> + <li>Update in CMakeLists.txt: + <pre><code class="hljs" data-trim contenteditable> +set( STRUS_MAJOR_VERSION 0 ) +set( STRUS_MINOR_VERSION 2 ) +set( STRUS_PATCH_VERSION 1 ) + </code></pre> + </li> + <li>Adapt %changelog and dist/debian/changelog. Keep STRICTLY to syntax! For tipps see also: <a href="http://keepachangelog.com/">http://keepachangelog.com/</a></li> + <li>Commit the master revision and tag it</li> + <pre><code class="hljs" data-trim contenteditable> +git commit -a -m 'changed to release 0.2.1' +git tag 0.2.1 + </code></pre> + </ul> + </section> + + <section> + <h2>Current process</h2> + <ul> + <li>Deploy to OBS workspace: + <pre><code class="hljs" data-trim contenteditable> +osc co home:PatrickFrey (into $HOME/home:PatrickFrey)</li> + </code></pre> + <pre><code class="hljs" data-trim contenteditable> +cmake . +dist/obs/deploy_to_obs.sh + </code></pre> + </li> + + <li>Remove source tarballs of old release, add tarballs of new release: + + <pre><code class="hljs" data-trim contenteditable> +cd home:PatrickFrey/strus +osc up +osc rm strus-0.1.6* +osc add strus-0.2.1* +osc commit -m 'added version 0.2.1' + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Issues</h2> + <ul> + <li>dist/obs/deploy_to_obs.sh should be eliminated</li> + <li>Release versions need patching, the mirroring system plays mean tricks otherwise..</li> + <li>How to automatically update the OSC workspace, not having tarballs there, etc.</li> + <li>Have a central changelogs, create Debian and RHEL style from it</li> + <li>git commit is always needed, otherwise tarball lacks modified/added artifacts</li> + <li>Don't change git tags later on! Rather revoke broken releases.</li> + </ul> + </section> + + <section> + <h2>Local OSB builds</h2> + <ul> + <li>Build locally in a chrooted envorinment: + <pre><code class="hljs" data-trim contenteditable> +cd home:PatrickFrey/strus +osc build --noinit Debian_8.0 x86_64 strus.dsc +osc build --noinit xUbuntu_12.04 x86_64 strus-xUbuntu_12.04.dsc +osc build --noinit CentOS_5 i586 strus.spec + </code></pre> + </li> + <li>Access a shell in the chroot to investigate issues:</li> + <pre><code class="hljs" data-trim contenteditable> +sudo chroot /var/tmp/build-root/xUbuntu_12.04-x86_64/ /bin/bash --login +cat /etc/debian_version +wheezy/sid + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Local OSB builds</h2> + <ul> + <li>The artifacts of the build system are somewhat hidden in the chroot:</li> + <pre><code class="hljs" data-trim contenteditable> +/usr/src/packages/BUILD/ +/home/abuild/rpmbuild/BUILD/strus-0.2.1 +/usr/src/packages/BUILD/strus-0.2.1 + </code></pre> + </li> + <li>The chrooted environment lacks things like an editor or a debugger or + even a package manager inside the chroot. + Either add a dependency for them 'BuildRequires:' or use + the package manager from outside the chroot (works nice on Archlinux + and for Ubuntu on Ubuntu, 'apt-get' on RHEL might be a problem). + </li> + </ul> + </section> + + <section> + <h2>Links</h2> + <ul> + <li>OSC OpenBuild Service Command Line: <a href="http://openbuildservice.org/download/">http://openbuildservice.org/download/</a> + </li> + </ul> + + </section> + </div> + </div> diff --git a/content/slide/struswebservice.md b/content/slide/struswebservice.md new file mode 100644 index 0000000..437b206 --- /dev/null +++ b/content/slide/struswebservice.md @@ -0,0 +1,414 @@ ++++ +title = "Strus Web Service" ++++ + + <div class="reveal"> + + <!-- Any section element inside of this container is displayed as a slide --> + <div class="slides"> + <section> + <h1>Strus - Web Service</h1> + <p> + <small>Created by <a href="http://www.andreasbaumann.cc">Andreas Baumann</a></small> + </p> + <p> + <small><a href="http://www.andreasbaumann.cc/slide/struswebservice">http://www.andreasbaumann.cc/slide/struswebservice</a></small> + </p> + <p><small>(C)2015/2016</small></p> + </section> + + <section> + <h2>Outline</h2> + <ul> + <li>Have a web service, language independent, for search.</li> + <li>The software to the vision exists, it's called <b>Elasticsearch.</b></li> + <li>Based on Lucene, so we do the same with Strus</li> + <li>Currently there is a Java API which uses this service</li> + </ul> + </section> + + <section> + <h2>Strus web service</h2> + <ul> + <li>uses JSON and HTTP GET/POST</li> + <li>implemented using <a href="http://cppcms.com/wikipp/en/page/main">CppCMS</a></li> + <li>uses only the <a href="http://patrickfrey.github.io/strus/doc/doxygen/html/index.html">Strus core API</a> currently</li> + </ul> + </section> + + <section> + <h2>Design rationales</h2> + <ul> + <li>standard technology (HTTP, JSON)</li> + <li>CRUD (all basic functions exists for all classes)</lib> + <li>proxyable and securable</li> + <li>zero-configuration</li> + <li>scalable</li> + <li>C++</li> + <li>MPL v2</li> + <li>runs on Linux, OSX, FreeBSD</li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Service functions</li> + <li>Index functions</li> + <li>Document functions</li> + <li>Query functions</li> + <li>Transactions</li> + <li>Statistics</li> + <li>Introspection and Configuration</li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Manipulate indexes: + <pre><code class="hljs" data-trim contenteditable> +curl -XPOST -H 'Content-Type: application/json' +http://localhost:8080/strus/index/create/A +'-d { "params" : { } }' + +{"result":"ok"} + +curl -XPOST http://localhost:8080/strus/index/delete/A +{"result":"ok"} + +curl -XPOST http://localhost:8080/strus/index/exists/A +{"exists":false,"result":"ok"} + + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Insert document: + <pre><code class="hljs" data-trim contenteditable> +curl -XPOST -H 'Content-Type: application/json' +http://localhost:8080/strus/document/insert/A +-d +{ "doc" : { "docid" : "doc3", + "attributes" : [ + { "key" : "title", "value" : "This is a Hello World Document" }, + { "key" : "attr1", "value" : "val1" }, + { "key" : "attr2", "value" : "val2" } ], + "metadata" : [ + { "key" : "doclen", "value" : 23773 }, { "key" : "docweight", "value" : 3.1415 } ], + "forward" : [ + { "type" : "word", "value" : "Hello", "pos": 1 }, + { "type" : "word", "value" : "World", "pos" : 2 } ], + "search" : [ + { "type" : "Word", "value" : "hello", "pos": 1 }, + { "type" : "word", "value" : "world", "pos" : 2 } ] } +} + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Query: + <pre><code class="hljs" data-trim contenteditable> +curl -XPOST -H 'Content-Type: application/json' +http://localhost:8080/strus/query/A +-d +{ "query": { + "first_rank": 0, + "nof_ranks": 20, + "weighting": { + "scheme": { + "name": "bm25", + "params": [ + { + "key": "b", + "value": 0.75 + }, + { + "key": "k1", + "value": 1.0001 + }, + { + "key": "avgdoclen", + "value": 11943 + } + ] + } + }, + "summarizer": [ + { + "attribute": "attribute", + "name": "attribute", + "params": [ + { + "key": "name", + "value": "docid" + } + ] + } + ], + "features": [ + { + "name": "feat", + "value": { + "term": { + "type": "word", + "value": "hello" + } + }, + "weight": 1 + }, + { + "name": "sel", + "value": { + "expression": { + "operator": "union", + "range": 0, + "cardinality": 0, + "terms": [ + { + "term": { + "type": "word", + "value": "hello" + } + } + ] + } + }, + "weight": 1 + } + ], + "select": [ + "sel" + ] +} + +{ + "execution_time": 0.000983791, + "ranklist": { + "documents_ranked": 1, + "documents_visited": 1, + "passes_evaluated": 0, + "ranks": [ + { + "attributes": [ + { + "key": "docid", + "value": "doc3" + } + ], + "docno": 1, + "weight": 0 + } + ] + }, + "result": "ok" +} + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Transactions: + <pre><code class="hljs" data-trim contenteditable> +curl -XPOST -H 'Content-Type: application/json' +http://localhost:8080/strus/transaction/begin/A/T1 + +curl -XPOST -H 'Content-Type: application/json' +http://localhost:8080/strus/document/insert/A +-d +{ "transaction" : { "id" : "T1" }, + "doc" : { "docid" : "doc1", + ... +} + +curl -XPOST -H 'Content-Type: application/json' +http://localhost:8080/strus/transaction/commit/A/T1 + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Introspection (document): + <pre><code class="hljs" data-trim contenteditable> +curl -XPOST -H 'Content-Type: application/json' +http://localhost:8080/strus/document/get/A/doc3 + +{ + "doc": { + "attributes": [ + { + "key": "attr1", + "value": "val1" + }, + { + "key": "attr2", + "value": "val2" + }, + { + "key": "docid", + "value": "doc3" + }, + { + "key": "title", + "value": "This is a Hello World Document" + } + ], + "docno": 1, + "forward": [ + { + "pos": 1, + "type": "word", + "value": "Hello" + }, + { + "pos": 2, + "type": "word", + "value": "World" + } + ], + "metadata": [ + { + "key": "docweight", + "value": 3.141499996185303 + }, + { + "key": "doclen", + "value": 23773 + }, + { + "key": "date", + "value": 0 + } + ], + "search": [ + { + "pos": 1, + "type": "word", + "value": "hello" + }, + { + "pos": 2, + "type": "word", + "value": "world" + } + ] + }, + "execution_time": 0.00028157, + "result": "ok" +} + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Statistics and introspection (index): + <pre><code class="hljs" data-trim contenteditable> +curl http://localhost:8080/strus/index/stats/A + +{"result":"ok","stats":{"nof_docs":1}} + +curl http://localhost:8080/strus/index/config/A + +{"config":{ + "attributes":["attr1","attr2","docid","title"], + "metadata":[{"name":"date","type":"UInt16"},{"name":"doclen","type":"UInt16"},{"name":"docweight","type":"Float32"}], + "types":["word"]}, +"result":"ok"} + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Protocol</h2> + <ul> + <li>Statistics and introspection (system): + <pre><code class="hljs" data-trim contenteditable> +curl http://localhost:8080/strus/config +{"config":{ + "posting_join_operators":[ + "chain","chain_struct","contains","diff","inrange","inrange_struct","intersect","pred","sequence","sequence_struct","succ","union","within","within_struct"], + "summarizer_functions":[ + "accuvariable","attribute","matchphrase","matchpos","matchvariables","metadata"], + "weighting_funtions":[ + "bm25","formula","metadata","td","tf"]}, +"result":"ok"} + + ... + "weighting_functions": [ + { + "description": "Calculate the document weight with the weighting scheme \"BM25\"", + "name": "bm25", + "parameter": [ + { + "description": "defines the query features to weight", + "name": "match", + "type": "feature" + }, + { + "description": "parameter of the BM25 weighting scheme", + "name": "k1", + "type": "numeric" + }, + ... + </code></pre> + </li> + </ul> + </section> + + <section> + <h2>Strus web service</h2> + <h3>Links</h3> + <ul> + <li><a href="http://github.com/Eurospider/strusWebService">http://github.com/Eurospider/strusWebService</a></li> + <li><a href="https://build.opensuse.org/package/show/home:andreas_baumann/struswebservice">packages and builds on OBS</a></li> + <li><a href="http://strustest.eurospider.com/strus/version">Test instance</a></li> + <li><a href="http://strustest.eurospider.com/strus/democlient/">Simple AJAX/JS demo interface</a></li> + </ul> + </section> + + <section> + <h2>Java API</h2> + <ul> + <li>Uses latest Java 1.8 (aka requires Java 1.8)</li> + <li>Uses Jackson for POJO serialization from/to JSON</li> + <li>uses javax.ws.rs.client web service classes for HTTP</li> + </ul> + </section> + + <section> + <h2>Java API</h2> + <h3>Links</h3> + <ul> + <li><a href="http://github.com/Eurospider/strusJavaAPI">http://github.com/Eurospider/strusJavaAPI</a></li> + <li><a href="http://eurospider.github.io/strusJavaApi/apidocs/index.html">JavaDoc</a></li> + </ul> + </section> + + <section> + <h2>Todos and future</h2> + <ul> + <li>support all functions from strus API</li> + <li>integrate strusAnalyzer and strusStream APIs</li> + <li>start with simple cluster mode (distribution proxy service)</li> + </ul> + </section> + </div> + </div> |