Posts on Sastibe's Data Science Blog

Everyman Proportion Test

Fri, 02 Jan 2026 21:33:28 +0100

TL;DR

Click a bookmark → paste four numbers → instantly know whether two conversion rates are meaningfully different.

No tools. No spreadsheets. No dashboards. Just your browser.

Why?

Have you ever been in a meeting, a Slack thread, or staring at an A/B test dashboard and thought:

“2.3% vs. 2.7% — is that actually meaningful, or just noise?”

Well then this bookmarklet is for you! Other recipients include but are not limited to…

Product managers sanity-checking A/B tests
Marketers comparing conversion rates
Analysts who want a quick second opinion
Anyone who does not want to open R, Python, Excel, or a stats website

You click a bookmark, paste four numbers, and immediately get:

the absolute difference in conversion rates
a p-value
a plain-English interpretation of significance

How?

Copy the code below → add it to your browser as a bookmark (as URL/Location) → click the bookmark whenever you need it.

No installation
No external website
Everything runs locally in your browser

javascript:(function(){function e(e){var t=e<0?-1:1;e=Math.abs(e);var a=.254829592,n=-.284496736,r=1.421413741,o=-1.453152027,i=1.061405429,l=.3275911,s=1/(1+l*e),u=1-((((i*s+o)*s+r)*s+n)*s+a)*s*Math.exp(-e*e);return t*u}function t(t){return.5*(1+e(t/Math.SQRT2))}function a(e){return(100*e).toFixed(2)+"%"}function n(e,t){return(Math.abs(e)>=1e3?e.toLocaleString():e.toFixed(t||4))}function r(e){return"string"==typeof e?parseFloat(e.trim().replace(",",".")):NaN}function pValueLabel(p){return p<.01?"Very significant":p<.05?"Significant":p<.1?"Barely significant":p<.15?"Probably different":"No clear difference"}try{var o=prompt("Please enter values in the format: Size A ; Rate A (in %) ; Size B ; Rate B (in %)\nExample: 10000 ; 2.5 ; 9500 ; 3.1","10000 ; 2 ; 10000 ; 3");if(!o)throw new Error("No input");var i=o.split(/;/);if(4!=i.length)throw new Error("Format error");var l=r(i[0]),s=r(i[1])/100,u=r(i[2]),c=r(i[3])/100;if(!isFinite(l)||!isFinite(u)||!isFinite(s)||!isFinite(c)||l<=0||u<=0||s<0||c<0)throw new Error("Invalid input");var f=Math.round(s*l),d=Math.round(c*u),m=f/l,h=d/u,v=(f+d)/(l+u),w=Math.sqrt(v*(1-v)*(1/l+1/u));if(!(w>0))throw new Error("Standard error is zero");var y=(h-m)/w,A=2*(1-t(Math.abs(y))),x=Math.sqrt(m*(1-m)/l+h*(1-h)/u),E=h-m-1.96*x,M=h-m+1.96*x,N=h-m,S=m>0?h/m-1:NaN,T=['📊 Two-sample proportion z-test (two-sided)','','--- Result ---','Difference (B − A): '+a(N)+'  (relative: '+(isFinite(S)?(100*S).toFixed(2)+"%":"n/a")+')','z = '+n(y,4),'p-value = '+n(A,6)+' → '+pValueLabel(A),'95% CI (B − A): ['+a(E)+', '+a(M)+']','','--- Input ---','Group A: n = '+Math.round(l)+', rate = '+a(m)+' (x = '+f+')','Group B: n = '+Math.round(u)+', rate = '+a(h)+' (x = '+d+')'].join("\n");try{navigator.clipboard.writeText(T)}catch(B){}alert(T+"\n\n(Result copied to clipboard)")}catch(B){alert("Error: "+B.message)}})();

How? (a bit more detailed)

Create a new browser bookmark
Name it something like Proportion Test
Paste the JavaScript code above into the URL / Location field
Open any webpage (content does not matter)
Click the bookmarklet
Enter values in the format:

Size A ; Rate A (in %) ; Size B ; Rate B (in %)

Press Enter to see the result.
The output is also copied to your clipboard.

What is actually happening here?

This bookmarklet performs a two-sample z-test for proportions (two-sided).

It answers the question:

If the true conversion rates were equal, how likely would we be to observe a difference at least this large?

That likelihood is the p-value.

Interpreting the p-value

p-value	Interpretation
< 0.01	Very significant
0.01–0.05	Significant
0.05–0.10	Barely significant
0.10–0.15	Probably different
≥ 0.15	No clear difference

Take Screenshots Straight into Org Files in Emacs on Win10

Tue, 04 Apr 2023 00:00:00 +0000

Update, 2023-04-05:

By popular demand, I updated the below solution to include a second function which stores the screenshots in a subfolder.

The Final Result

Let me start this article by showing you my usage of the functionality described. I work as a Data Scientist and use org-mode in Emacs for a large number of every day tasks. One of them is the documentation of new findings within datasets or other software's documentation or websites etc. In order to easily collect all these informations into a single reference, I like to use screenshots:

<div class="figure"> <img src="" width="600" height="338" alt="The story of this painting is really fascinating, please do check out the book 'A History of the World in 10 1/2 Chapters' by Julian Barnes (Ch. 5)"/> <p class="caption"> An example usage of the code presented here. </p> </div>

The only thing "hidden" from this clip is the fact that I hit the combination C-c s right after finishing my sentence. Everything else ran automatically.

The Step-by-Step Guide (there is only one step)

Assuming you run Emacs on Windows 10, the result is extremely simple to achieve. For Windows 7 and 8, I assume that this approach works perfectly fine, I couldn't put this assumption to the test, though. Here is the singe step needed: edit your .emacs or init.el to include the following block:

Copy it in, evaluate it, and you're ready to go: just hit C-c s whenever you are inside an org-mode-buffer and take screenshots at any whim. They will be saved inside your org-mode folder and displayed inline.

Alternative solution with subfolders

If you want to store your screenshots in a subfolder "images", relative to the location of the org-file, you simply need to replace the function with the second one in the code snippet above.

The (a little more) detailed description

So, what does the magic function do and how does it do it? There are three major parts:

Creating a Unique Name for the Image

This is the first 4 lines, in which the filename of the buffer, the current time stamp and a random combination of letters (via make-temp_name) is created.

Getting the Screenshot into the Clipboard

This is the function of the line

(shell-command "snippingtool /clip")

which calls the wonderful "SnippingTool" inside Windows and skips all preliminary steps with the command line argument /clip.

Saving the Contents of the Clipboard to Disk

This part took me the longest to figure out. The core idea is that powershell.exe, when called from CMD, has the functionality to save Clipboard contents to file. Hence, the command given here

Adds appropriate types for Powershell to understand
Checks if the Clipboard contains an image, and if so
Saves it to filename, given in step 3.

A toy example for Reinforcement Learning, CLI via fire, CI/CD on gitlab in Python

Mon, 25 Jul 2022 00:00:00 +0200

Eager to Play TicTacToe?

The package neckar contains a fully functional reinformcement learning model to train yourself and play against in the game TicTacToe. Here’s how you can simply download and install the result if you have a gitlab account (and can set up a personal access token…):

pip install neckar --extra-index-url https://__token__:personalaccesstoken@gitlab.com/api/v4/projects/37714906/packages/pypi/simple

… and if you don’t: Download the wheel from this link, cd to the wheel and run

pip install neckar-0.1.1-py3-none-any.whl

In both cases, you then start plaxing against the trained model by runnning

reilingen tictactoe play

The standard model was trained in 20000 rounds before shipping. If the computer is too good for you, you can dumb it down like so:

reilingen tictactoe train --rounds=500

and you should be able to win a game, even though the computer always starts.

How it’s done: Reinforcement Learning

This main part of the code, residing in scr/neckar/tictactoe.py is essentially copied directly from here. The only two major changes I did was in the data structure used: data frames instead of numpy arrays. This is to support the second change: the winning condition is evaluated with regexes rather than with sums. I plan on extending the package to include connect four, and wanted to streamline the detection of winning patterns.

The mathematical background to the reinforcement learning part is temporal difference learning as described on the wiki. The main formula consists in updating the state probabilities after each run using the regression In the code, this line reads

self.states_value[st] += self.lr * (self.decay_gamma * reward - self.states_value[st])

and its corresponding method takes up just five lines of code of the ~300 lines.

How it’s done: Package set up

As can be seen from the README, the package was set up with PyScaffold. In order to start with a fresh scaffold, I simply ran

pip install pyscaffold
putup neckar

and receieved a working a skeleton of a Python package, with the right structure and some dummy files. From there, I simple replaced the reinforcement learning code at the right places and added sensible unit tests in ./tests for my new winning detection.

How it’s done: CLI with fire

The skeleton provided by PyScaffold already comes with an entry point, see setup.cfg

[options.entry_points]
console_scripts =
     reilingen = neckar.main:run

I use fire to expose the functions play and train to the CLI. This allows the user to interact with the application and even train it, and is implemented in main.py and allows for the usage as described in Eager to Play TicTacToe?

How it’s done: CI/CD in gitlab

The step from a PyScaffold setup to a CI/CD pipeline in gitlab is simple but has many, many positive consequences. The pipeline is defined in .gitlab-ci.yml, and it is run any time there is a new commit or a new tag in the package in gitlab. The step run contains the line

python -m build
python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*

and therefore an entire whl is built any time a new commit is created. The final packages lie in the package repository and are publicly available as shown in Section Eager to Play TicTacToe?

Don't forget to sharpen your axe

Sun, 24 Jul 2022 00:00:00 +0200

A wise man once said…

Give me six hours to chop down a tree and I will spend the first four sharpening the axe.

Abraham Lincoln

In this statement there is a lot of truth. For instance, it is closely mimicked by on of one of the 7 habits of highly effective people by Stephen Covey. Obviously, it applies to writing software as well, as a sharpened “developer’s axe” means:

a suitable editor for writing and manipulating code, preferably with a good git integration,
good access to data if needed for analysis,
easy processes for deployment or publishing of the results with maximal automisation.

The rule of three

All of these aspects of the developer’s axe need to be sharpened regularly. There looms a problem of overdoing it, however: how much time should be spent on sharpening vs. actual hacking? If Lincoln had said that he’d spend 5 hours on sharpening, that would be a bit too much, no?

Similarly, when optimizing one’s own processes, one sometimes tends to overdo optimization and ends up getting nothing done. But at what point comes the sweet spot?

Though I cannot find the source for this any more, I recall reading a quote by Hadley Wickham somewhere that “Before I do something manually the fourth time, I write a function for it”. I’ve kept this rule in mind ever since, and I find it very well applicable in almost all situations: Three times are fine, but then you should consider functions/automization/….

My own ungrinded axe

Why am I writing all this? Because I was, until recently, working with a blunt axe myself and only now came to sharpening it:

I use Hugo to present my blog, but I write it in emacs using org mode. Hugo is able to ingest emacs org files directly, but there is also the possibility of exporting from org to Hugo via ox-hugo. Up until the penultimate post, I used the former solution, due to the following reasons:

it worked somehow, even though I needed to write an org file with some yaml at the beginning for the more fidgety parts of the header, I.e. thumbnail pic etc, and some html ALL IN THE SAME FILE…
there was no need for an export step from org to markdown.

In between these posts, I decided to sharpen my blog axe, and switch to a single, seperate org file that creates all of my future blog posts via ox-hugo. This lead to the following improvements:

much simpler writing of posts so far, because I only have to remember one set of Markdown Syntaxes and ox-hugo is incredibly well documented,
easy tracking of open tasks org-agenda, since the whole blog can be declared an org-agenda-files, and all open (i.e. “draft”) articles show up as TODOs,
I can write this post on my smartphone, as the central org file for the blog is now separate, and lives in my Dropbox, synced to orgzly on my phone (only the export files reside in my git repo, syncing to the productive environment).

What has sharpening the axe done for me? Previously, I could only reasonably work on my laptop, unless I opted for copy pasting text from email into org files or something. At the moment of writing this post I’m enjoying the sun at the public pool, so … an obvious plus.

The single org file used to built the last post and this one can be found here.

Writing DIN-Norm (German) Letters in Emacs

Sat, 16 Jul 2022 00:00:00 +0200

TL;DR

This post explains how to set up emacs so that the writing and sending of letters is always just a whim away. The resulting letters will adhere to the German DIN 5008 standard, and fit neatly into envelopes designed on that standard, as shown in the cover image on the top.

Prerequisites

All you for the implementation shown in this tutorial is

Emacs with org-mode and
a LaTeX installation with the dinbrief.cls installed. On Ubuntu and Debian all you need are the packages texlive-latex-base and texlive-latex-extra.

Setting up the capture template

I use the following set-up for capturing templates. It writes each new captured letter to a single “letter”-File in LETTERFILE.org. The process of starting a letter via capture, i.e. C-c c l then prompts you for filename/heading of the letter in the LETTERFILE, address (each line separately), topic and greeting formula. After that, you can simply write your plain text letter within the capture and finish with C-c C-c.

(setq org-capture-templates '(("l" "Letter" entry
			     (file+headline "LETTERFILE.org" "Letters")
			     "* %<%F> %^{Filename}
:PROPERTIES:
:EXPORT_FILE_NAME: %<%F>_%\\1
:END:
#+OPTIONS: toc:nil
#+LaTeX_CLASS: dinbrief
#+LaTeX: \\begin{letter}{
#+LaTeX: %^{Adresszeile 1 [Name]} \\\\
#+LaTex: %^{Adresszeile 2 [Street, Number]} \\\\
#+LaTex: %^{Adresszeile 3 [Place, ZIP]} }
#+LaTeX: \\subject{%^{Topic}}
#+LaTeX: \\opening{%^{Greeting}}
%?
#+LaTeX: \\closing{Kind regards,}
#+LaTeX: \\underline{\\hspace{12cm}}\\\\
#+LaTeX: Sebastian  Schweer\\\\[2cm]
#+LaTeX: \\end{letter}" :jump-to-captured t)))

Hints for further customization

Within the variable `org-latex-classes`, you can set up your backadress and favourite pacakges to use while writing, so you don’t have to include these manually each time you write a letter. Obviously though, these settings should only be used for stationary information. Everything that might change from letter to letter should be handled within the capture template.

(custom-set-variables
'(org-latex-classes
   '(("dinbrief" "\\documentclass{dinbrief}
\\address{Paul McCartney \\\\
Abbey Road 24 \\\\
6058 London}
\\backaddress{Paul McCartney, London}
\\place{London}"
      ("\\section{%s}" . "\\section*{%s}")
      ("\\subsection{%s}" . "\\subsection*{%s}")
      ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
      ("\\paragraph{%s}" . "\\paragraph*{%s}")
      ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
     ("article" "\\documentclass[11pt]{article}"
      ("\\section{%s}" . "\\section*{%s}")
      ("\\subsection{%s}" . "\\subsection*{%s}")
      ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
      ("\\paragraph{%s}" . "\\paragraph*{%s}")
      ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
     ("report" "\\documentclass[11pt]{report}"
      ("\\part{%s}" . "\\part*{%s}")
      ("\\chapter{%s}" . "\\chapter*{%s}")
      ("\\section{%s}" . "\\section*{%s}")
      ("\\subsection{%s}" . "\\subsection*{%s}")
      ("\\subsubsection{%s}" . "\\subsubsection*{%s}"))
     ("book" "\\documentclass[11pt]{book}"
      ("\\part{%s}" . "\\part*{%s}")
      ("\\chapter{%s}" . "\\chapter*{%s}")
      ("\\section{%s}" . "\\section*{%s}")
      ("\\subsection{%s}" . "\\subsection*{%s}")
      ("\\subsubsection{%s}" . "\\subsubsection*{%s}"))))
 '(org-latex-default-packages-alist
   '(("AUTO" "inputenc" t
      ("pdflatex"))
     ("T1" "fontenc" t
      ("pdflatex"))
     ("" "graphicx" t nil)
     ("" "grffile" t nil)
     ("" "longtable" nil nil)
     ("" "wrapfig" nil nil)
     ("" "rotating" nil nil)
     ("normalem" "ulem" t nil)
     ("" "amsmath" t nil)
     ("" "textcomp" t nil)
     ("" "amssymb" t nil)
     ("" "capt-of" nil nil)
     ("" "hyperref" nil nil)
     ("official" "eurosym" nil nil)
     ("" "ngerman" nil nil)))
 '(org-latex-title-command "")

Usage

The final step is simple … just use it :) Capture your letter, answer all the prompts, and write your text. Once your capture is finished, navigate to the corresponding subtree in LETTERFILE, use and the org-export to latex to create your pdf: C-c C-e C-s l o.

Umfragewerte oder Medienpräsenz: Was kommt zuerst?

Tue, 12 Oct 2021 00:00:00 +0000

Einleitung und Fragestellung

In den letzten Jahren² konnten in diversen Ländern der Aufstieg populistischer Phänomene beobachtet werden: der Aufstieg der AfD in Deutschland, die seit ihrer Gründung binnen 5 Jahren in sämtlichen deutschen Landtagen, dem Bundestag und dem europäischen Parlament vertreten ist; der Wahlsieg Donald J. Trumps in den USA; der Aufstieg der Lega in Italien und viele weitere Beispiele. Ein Verlaufsmuster, das bei jedem einzelnen dieser Aufstiege in der Öffentlichkeit auftritt ist das folgende Ping-Pong-Spiel zwischen Medien und Umfragewerten: Je größer die gesellschaftliche Bekanntheit, desto größer das Medienecho, desto größer die Bekanntheit, etc. In diesem Artikel werde ich diesen Zusammenhang empirisch untersuchen, und dabei folgenden Fragestellungen nachgehen:

Gibt es einen messbaren, zeitlichen Zusammenhang zwischen medialer Aufmerksamkeit für die AfD und den Umfragewerten der Partei?
Wenn ja, welche zeitliche Abfolge gilt: Erst Medienecho, dann Umfrageergebnis, oder umgekehrt?

In der Folge erkläre ich detailliert meine Vorgehensweise, nehme hier aber bereits die Antworten vorweg:

Ja, in der Phase zwischen "erster Spaltung" der AfD (Juli 2015) bis zur Bundestagswahl 2017 ist dieser Zusammenhang deutlich messbar
Laut Zahlenwerk am plausibelsten: Die Steigerung der Berichterstattung erfolgt knapp vorher oder genau zeitlich synchron; es lassen sich aber keine eindeutigen Aussagen treffen.

Datenquellen und erste Einblicke ¹

Als Datenquelle für die Analyse verwende ich zwei verschiedene Datenquellen:

für Umfrageergebnisse stütze ich mich auf die Ergebnisse der Umfrageinstitute Allensbach, Emnid, Forsa, FGW, GMS, Infratest und INSA, allesamt aufbereitet und bereitgestellt von www.wahlrecht.de³.
für Medienpräsenz habe ich auf die DeveloperAPI der Wochenzeitung Zeit zugegriffen, die sämtliche Artikel der gedruckten Zeit seit 1946 durchsuchbar macht⁴

Für einen ersten Überblick der verwendeten Daten hier zunächst der Blick auf die Anzahl der veröffentlichten Artikel in der Zeit, die den String "AfD":

<div class="figure"> <img src="" width="600" height="338" alt="Hmmm, ggplot2."/> <p class="caption"> Anzahl monatlicher Artikel auf zeit.de mit dem Stichwort "AfD" </p> </div>

Interessante Beobachtungen hierbei sind zum Einen die Spitzen, deren erste im Jahre 2017 sich einfach mit der Bundestagswahl in diesem Jahr erklären lässt. Der hohe Ausschlag im Jahr 2020 fällt auf den Februar, als die Bürgerschaftswahl in Hamburg anstand. Zum Anderen durchaus spannend ist das "Tal" zwischen 2018 und Mitte 2019, in welchem die Aufmerksamkeit für die Partei, zumindest in der Berichterstattung von zeit.de erstmal zu erlahmen schien. Erst die Bürgerschaftswahl und dann der beginnende Wahlkampf zur Bundestagswahl 2021 scheint das Interesse wieder geweckt zu haben.

Der zweite Datentopf besteht aus den aggregierten Wahlumfragen von wahlrecht.de ("Wenn nächsten Sonntag Bundestagswahl wäre…"), und ergibt folgendes Bild:

<div class="figure"> <img src="" width="600" height="338" alt="Hmmm2, ggplot2."/> <p class="caption"> Monatlich aggregierte Umfragewerte der AfD und Zeitpunkte der Bundestagswahlen </p> </div>

Je Monat wurden für die obige Darstellung für alle vorhandenen Umfrageergebnisse der Durchschnitt ermittelt. Der zeitliche Anker ist hierbei die Veröffentlichung der Umfrage, im Mittel liegt dieses Datum ca. 9 Tage nach Beginn der Umfrage.

Auf den langen Zeitraum betrachtet zeigt sich ein bewegter, insgesamt leicht steigender Verlauf mit bisherigem Höchststand Anfang 2018. Der Einfluss der Corona-Pandemie mit Begin um die Jahreswende 2019/20 scheint ein deutlich negativer zu sein, trotz (oder vielleicht gerade wegen) der Nähe der AfD zur pandemieskeptischen Kreisen.

Die Analysemethode

Zur Ermittelung eines Zusammenhangs zwischen den beiden Datentöpfen wird hier die Korrelation berechnet, je höher dieser Wert ausfällt, desto deutlicher ausgeprägt ist auch der Zusammenhang zwischen beiden Größen. Hierbei ist festzuhalten, dass Korrelation keine Kausation bedeutet, d.h. der festgestellte Zusammenhang muss nicht derart ausgeprägt sein, dass das Auftreten des einen Ereignisses direkt das andere Ereignis herbeiführt.

Um die zeitliche Komponenten der Untersuchung wiederzuspiegeln, wird die Korrelation in einem laufenden Fenster aus paarweise zueinander geordneten Beobachtungen berechnet. Hierbei werden verschiedene zeitliche Abstände probiert, um bspw. die Bewertung "tritt ein erhöhter Umfragewert zwei Monate nach Steigerung der Berichterstattung an?" beantworten zu können. Die Wahl der Länge des zeitlichen Fensters, das jeweils über den Zeithorizont geschoben wird, hat hierbei Auswirkungen in zwei Richtungen: Ein kurzes Fenster ermöglicht den Blick in sehr kurzfristige Veränderungen des Geschehens, hat aber eine statistisch viel größere Streuung als dies ein längeres Fenster hat. In den unten gezeigten Plots habe ich eine Fensterlänge von einem halben Jahr, d.h. 18 Messpunkten gewählt.

Das Ergebnis

In der folgenden Grafik sind die Verläufe von zwei verschiedenen Varianten der laufenden Korrelationsfenster angegeben:

in Rot: der Korrelationkoeffizient zu "synchronem" Verlauf,
in Hellblau: der Korrelationkoeffizient zu Berichterstattung läuft einen Monat voraus.

<div class="figure"> <img src="" width="600" height="338" alt="Hmmm3, ggplot2."/> <p class="caption"> Zeitlicher Verlauf der "laufenden Korrelationen" zu Zeitversatz 0 und 1 </p> </div>

Weiterhin wurde der Schwellwert 0.708 eingezeichnet, als der Schwellwert zur Signifikanz des Tests auf Unkorreliertheit zum Niveau 0,1% für n=18 ⁵. Die Tatsache, dass beide Kurven über einen Zeitraum von fast einem Jahr über diesem Schwellwert rangieren, deutet auf die Deutlichkeit des Zusammenhangs hin. Der Zeitraum bezieht sich hierbei auf die vorhergehenden 18 Monate je Messpunkt, die Grafik zeigt somit, dass die Aussage "Umfragewerte und Berichterstattung steigen gleichzeitig" für den Zeitraum zwischen September 2014 bis Februar 2017 statistisch sehr plausibel bewertet wird. Das gleiche gilt aber auch für die Aussage "Umfragewerte steigen einen Monat nach erhöhter Berichterstattung", und deutet damit auf die Unschärfen hin, die dieser Untersuchung zugrunde liegen. In der Tat steigen auch die Kurven zu anderen zeitlichen Abständen (+2 bis -2) in diesem Zeitraum auf statistisch signifikante Werte, jedoch nicht über einen derart langen Zeitraum.

Zusammengefasst: Die Fragestellung 1) kann eindeutig beantwortet werden, alle weiteren Fragestellungen entziehen sich einer eindeutigen Klärung. Dennoch halte ich das dargestellte Vorgehensweise für einen interessanten Weg, politische und mediale Phänomene messbar und bewertbar zu machen.

Den verwendeten Code gibt es <a href="/afd_zeit_korrelationsanalyse.R"> hier </a> zum Download.

Diesen Artikel schreibe ich, entgegen meiner üblichen Vorgehensweise, auf Deutsch, da das Thema sich dediziert mit deutscher Politik beschäftigt.

Mit freundlichen Grüßen und herzlichem Dank auch an die Entwickler des R packages <a href="https://cran.r-project.org/web/packages/coalitions/index.html"> coalitions</a>.

⁴

An dieser Stelle noch einmal meinen persönlichen Dank ans Entwicklerteam für die Genehmigung des Antrags an Silvester!

⁵

Siehe Lothar Sachs, Angewandte Statistik, 8. Ausgabe, Tabelle 193.

Setting up Emacs as Mail Client

Tue, 19 Jan 2021 00:00:00 +0000

The Final Result

Let me start this article by showing you the final result of this article. Below you see a screengrab of me

checking the new mails I received today, from across 3 separate mail accounts (including a gmail account)
deleting two mails on the fly, while skimming the remaining two
searching through all my mails for the keywords "thalia AND FIFA" and finding the result immediately

Everything is done within Emacs, no context switching or anything else needed. In this post, I'll describe my set-up as minimalistically as possible.

Setting up the framework

Mu

First, you need to install Mu from the sources, which works as a indexing and search engine on your offline email cache. On Ubuntu 20.04, this consists in extracting the sources, and running

./configure

within the extracted folder. For my setup, I additionally needed to install three dependencies for the configuration to cleanly run through:

sudo apt install libgtk2.0-dev libgmime-3.0-dev libxapian-dev

After this, a simple

make
make install

installed mu.

Encrypting passwords etc.

In what follows, this file will require numerous sensitive data entries, such as passwords, tokens etc. In order to avoid storing these entries in plain text, I follow this stackexchange answer but add a few tweaks. First, if not already existing, create a gpg key via

gpg --full-generate-key

Next, download and install pass and set it up by

pass init "Name of your GPG key"

With this setup, encrypting any type of information is as simple as

pass insert foo/bar

and the prompt will ask you for it, and store it in the given file destination.

Next, save the following python script as something like ~/.offlineimap.py :

import os
import subprocess

def mailpasswd(acct):
  acct = os.path.basename(acct)
  path = "/home/<username>/.passwd/%s.gpg" % acct
  args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
  try:
    return subprocess.check_output(args).strip()
  except subprocess.CalledProcessError:
    return ""
  
def prime_gpg_agent():
  ret = False
  i = 1
  while not ret:
    ret = (mailpasswd("prime") == "prime")
    if i > 2:
      from offlineimap.ui import getglobalui
      sys.stderr.write("Error reading in passwords. Terminating.\n")
      getglobalui().terminate()
    i += 1
  return ret

prime_gpg_agent()

change <username> accordingly.

Setting up offlineimap

Offlineimap is already pre-installed in Ubuntu, so the only "setting up" to do is the configuration of the aptly named file ~/.offlineimaprc.

GMail

For GMail, the necessary encryption standards have improved in recent times, and Gmail no longer likes the encryption provided by sending the user/password combination directly (via remoteuser and remotepasseval). Rather, it expects communication via XOAUTH2. Hence, following the official documentation, I created and received my oauth2_client_id and oauth2_client_secret from Google. After that, further following the official documentation, I cloned the gmail-oauth2-tools repo and created the oauth2_token as described, using

python python/oauth2.py --generate_oauth2_token \
            --client_id=YOUR_CLIENT_ID --client_secret=YOUR_CLIENT_SECRET

This generated both an access token and a refresh token, both of which I put into encrypted files, as described above. I finalized the offlineimaprc setup, it now looks like this, including one other account that uses basic password security:

[general]
accounts = gmail, sastibe
maxsyncaccounts = 2
pythonfile = ~/.offlineimap.py


[Account gmail]
localrepository = LocalGmail
remoterepository = RemoteGmail
quick = 10
postsynchook = mu index --maildir ~/Mails

[Repository LocalGmail]
type = Maildir
localfolders = ~/Mails/gmail

[Repository RemoteGmail]
type = Gmail
maxconnections = 2
remoteuser = sebastian.a.schweer@gmail.com
oauth2_request_url = https://accounts.google.com/o/oauth2/token

oauth2_client_id_eval = mailpasswd("gmail_oauth2_client_id")
oauth2_client_secret_eval = mailpasswd("gmail_oauth2_client_secret")
oauth2_refresh_token_eval = mailpasswd("gmail_oauth2_refresh_token")

folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail', '[Gmail]/Important']
sslcacertfile = /etc/ssl/certs/ca-certificates.crt


[Account sastibe]
localrepository = Localsastibe
remoterepository = Remotesastibe
quick = 10
postsynchook = mu index --maildir ~/Mails

[Repository Localsastibe]
type = Maildir
localfolders = ~/Mails/sastibe

[Repository Remotesastibe]
type = IMAP
remotehost = imap.1und1.de
ssl = yes
remoteuser = sastibear@sastibe.de
remotepasseval = mailpasswd("sastibe_password")

sslcacertfile = /etc/ssl/certs/ca-certificates.crt

Config for .emacs

Including mu4e

This is fortunately quite simple after the previous installation of mu, just add

(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e/")
(require 'mu4e)

to your .emacs and you are good to go.

Basic setup

These are the necessary settings for using mu4e with offlineimap and XOAUTH2 encryption:

(require 'smtpmail)
(require 'auth-source-xoauth2)

(setq mail-user-agent 'mu4e-user-agent)
(setq mu4e-sent-messages-behavior 'delete)
(setq mu4e-maildir "/home/sastibe/Mails")
(setq mu4e-update-interval 300)

(setq message-send-mail-function 'smtpmail-send-it)
(setq mu4e-get-mail-command "offlineimap")
(setq mu4e-headers-date-format "%d-%m-%Y %H:%M")
(setq mu4e-headers-fields '((:human-date . 20)
			    (:flags . 6)
			    (:from . 22)
			    (:maildir . 8)
			    (:subject)))

Encryption for XOAUTH2

For XOAUTH2 communication, GMail expects a specific set of parameters to be transmitted.

(defun my-xoauth2-get-secrets (host user port)
  (when (and (string= host "smtp.gmail.com")
             (string= user "sebastian.a.schweer@gmail.com"))
    (list
     :token-url "https://accounts.google.com/o/oauth2/token"
     :client-id     (auth-source-pass-get 'secret "Email/gmail_oauth2_client_id")
     :client-secret (auth-source-pass-get 'secret "Email/gmail_oauth2_client_secret")
     :refresh-token (auth-source-pass-get 'secret "Email/gmail_oauth2_refresh_token"))))
(setq auth-source-xoauth2-creds 'my-xoauth2-get-secrets)
(add-to-list 'smtpmail-auth-supported 'xoauth2)
(auth-source-xoauth2-enable)
(setq auth-sources (quote (xoauth2 password-store)))

Mu4e Contexts

In order to separate the accounts and keep the commands adjustable, I use mu4e-contexts like so:

(setq mu4e-contexts
 `( ,(make-mu4e-context
     :name "gmail"
     :match-func (lambda (msg) (when msg
       (string-prefix-p "/gmail" (mu4e-message-field msg :maildir))))
     :vars '(
	     (mu4e-refile-folder . "/Gmail/[Gmail].Archive")
	     (mu4e-drafts-folder . "/gmail/[Gmail].Drafts")
	     (mu4e-sent-folder   . "/gmail/[Gmail].Sent Mail")
	     (mu4e-trash-folder  . "/gmail/[Gmail].Trash")
	     (smtpmail-stream-type . ssl)
	     (smtpmail-default-smtp-server . "smtp.gmail.com")
	     (smtpmail-smtp-server  . "smtp.gmail.com")
	     (smtpmail-smtp-service . 465)
	     (smtpmail-smtp-user    . "sebastian.a.schweer@gmail.com")
	     (auth-source-xoauth2-creds . 'my-xoauth2-get-secrets)
       ))
   ,(make-mu4e-context
     :name "sastibe"
     :match-func (lambda (msg) (when msg
       (string-prefix-p "/sastibe" (mu4e-message-field msg :maildir))))
     :vars '(
	     (mu4e-sent-folder   . "/sastibe/Gesendete Objekte")
	     (mu4e-trash-folder  . "/sastibe/Gesendete Objekte")
	     (mu4e-refile-folder . "/sastibe/Spam")
	     (mu4e-drafts-folder . "/sastibe/Entw&APw-rfe")
	     (user-mail-address  . "sastibear@sastibe.de")
	     (smtpmail-default-smtp-server . "smtp.1und1.de")
	     (smtpmail-smtp-user . "sastibear@sastibe.de")
	     (smtpmail-smtp-server . "smtp.1und1.de")
	     (smtpmail-stream-type . starttls)
	     (smtpmail-smtp-service . 587)
     ))
   ))

Debugging

If things don't run as expected, the following settings allow for deeper troubleshooting:

(setq auth-source-debug t)
(setq smtpmail-debug-info t)

Sources & Acknowledgements

Update March 7, 2021: I modified the article to reflect in the code a much better way of keeping mu4e up to date. The previous version ran into conflicts with different processes trying to keep mu and mu4e up to date at the same time.

Essentially I follow this post from 2017, and adjust it according to my setup (Ubuntu 20.04) as well as new security policies as ordained by Google Mail. I am much indebted to two of the three discussions in the repo for auth-source-pass to figure out the proper way to apply this package.

De-Googling the Blog (and my life)

Mon, 23 Nov 2020 00:00:00 +0000

Today, just a short notification concerning the use of analytics on this blog: I have decided to get rid of the Google Analytic Tag on these pages. It was very joyful to know how many people surfed to my little blog from all over the world, yet it doesn't outweigh my reservation against Google obtaining entry into almost each and every single piece of software surrounding us.

Asides from on-site analytic tags I have recently decided to circumvent Google and its services as well as possible in my personal digital footprint as well. I search the web using DuckDuckGo and try to find my way around the world using OpenStreetMap Further suggestions can be found in other, well-researched articles such as this one.

Update, 2020-12-12: I have now found a suitable alternative to Google Analytics, Clicky. I already changed the Privacy Notice accordingly, and will keep you updated…

Supplement to 'Nonparametric estimation of the service time distribution in discrete-time queueing networks'

Thu, 30 Jan 2020 00:00:00 +0000

Great news: a scientific article I have co-authored has been accepted for publication and can now be found online here or via the DOI 10.1016/j.spa.2020.01.011. Yes, my list of publications has been amended ¹. This article has been through quite a lengthy review process, and was the main motivation for another one of my blog posts. This post dates to September 2018, yet I only started working on these simulations in the framework of the second round of peer review…

Anyways, the results presented in the paper are based on simulations I calculated using a specifically desigend R package, aptly named queueingnetworkR. This package was only ever designed for this singular purpose, for anyone who might be interested in repeating the experiments or using the code in any other fashion, the latest tar-ball can be found here, the GitLab page is here.

I am glad you asked.

Calculating the Required Sample Size for a Binomial Test in R

Tue, 21 Jan 2020 00:00:00 +0000

A Standard Problem: Determining Sample Size

Recently, I was tasked with a straightforward question: "In an A/B test setting, how many samples do I have to collect in order to obtain significant results?" As ususal in statistics, the answer is not quite as straightforward as the question, and it depends quite a bit on the framework. In this case, the A/B test was supposed to test whether the effect of a treatment on the success rate p had the assumed size e. The value of the success rate had to be estimated in both test and control group, i.e. p<sub>test</sub> and p<sub>control</sub>. In short, the test hypotheses were thus

H<sub>0</sub> : p<sub>test</sub> = p<sub>control</sub> vs.

H<sub>1</sub> : p<sub>test</sub> = p<sub>control</sub> + e

Now, for each statistical test, we aim at minimizing (or at least controlling for) the following types of error:

Type I: Even though H<sub>0</sub> is true, the test decides for H<sub>1</sub>
Type II: Even though H<sub>1</sub> is true, the test decides for H<sub>0</sub>.

Since I can never remember stuff like this, I immediately started looking for a simple mnemonic, and I found this one:

The null hypothesis is often represented as H<sub>0</sub>. Although mathematicians may disagree, where I live 0 is an even number, as evidenced by the fact that it is both preceded and followed by an odd number. Even numbers go together well. An even number and an odd number do not go together well. Hence the null hypothesis (even) is rejected by the type I error (odd), but accepted by the type II error (even).

The test statistic

In the given setup, the test now runs as follows: Calculate the contingency matrix of successes in both groups, and apply Fisher's exact test. If the test is negative, i.e. does not reject the null hypothesis, we have to repeat the experiment. However, we are quite sure that the null hypothesis is wrong and would like to prove that with as little effort as possible.

The basic question in this situation is "how many observations do I need to collect, in order to avoid both errors of Type I and II to an appropriate degree of certainty?". The "appropriate degree of certainty" is parametrized in the probability of errors of Type I (significance level) and Type II (power). The default choices for these values are 0.05 for the significance level, and 0.8 for power: In 5% of cases, we reject a "true" H<sub>0</sub>, and in 20% of cases we reject a "true" H<sub>1</sub>. Quite clearly, only the power of the test (and not the significance level) depends on the difference of the parameters p<sub>test</sub> and p<sub>control</sub>.

Existing functions in R

Are there already pre-defined functions to calculate minimal required sample sizes? A bit of digging around yields a match in the package Hmisc. There, the authors implement a method developed Fleiss, Tytun and Ury¹. However, according to the documentation, the function is written only for the two-sided test case and does not include the continuity correction. I disagree with both decisions:

the continuity correction term can grow quite large, and is always positive (see (5) in the cited paper). Thus, neglecting this term will always end in an underestimation of the necessary number of observations and may therefore lead to unsuccessful experiments.
the two-sided case is not the norm, but rather the exception. When testing p<sub>control</sub> vs. p<sub>test</sub>, the counterhypothesis will almost always read "p<sub>test</sub> > p<sub>control</sub>", since the measures taken assume to have, if any, a positive effect.

A new R function: calculate_binomial_samplesize

After these considerations, I decided to write my own function. Below is the code, the function allows for "switching the continuity correction off", and for differentiating between the one-sided and the two-sided case. In the two-sided case without continuity correction, it coincides with "Hmisc:bsamsize", as can be seen from the example provided.

#' Calculate the Required Sample Size for Testing Binomial Differences
#'
#' @description
#' Based on the method of Fleiss, Tytun and Ury, this function tests the null
#' hypothesis p0 against p1 > p_0 in a one-sided or two-sided test with significance level
#' alpha and power beta.
#'
#'
#' @usage
#' calculate_binomial_samplesize(ratio0, p0, p1)
#'
#'
#' @param ratio0 Numeric, proportion of sample of observations in group 0, the control
#' group
#' @param p1 Numeric, postulated binomial parameter in the treatment group.
#' @param p0 Numeric, postulated binomial parameter in the control group.
#' @param alpha Desired significance level for the test, defaults to 0.05
#' @param beta Desired power for the test, defaults to 0.8
#' @param one_sided Bool, whether the test is supposed to be one-sided or two-sided.
#' @param continuity_correction Bool, whether the sample size should be
#' Defaults to TRUE.
#'
#'
#' @return
#' A named numeric vector, containing the required sample size for the treatment group,
#' the control group, and the required total (the sum of both numbers).
#'
#'
#' @seealso [Hmisc::bsamsize()]
#'
#' @author Sebastian Schweer \email{sastibe_r@@sastibe.de}
#'
#' @references
#' Fleiss JL, Tytun A, Ury HK (1980): A simple approximation for calculating sample sizes
#' for comparing independent proportions. Biometrics 36:343-346.
#'
#' @examples
#'# Same result
#' alpha = 0.02; power = 0.9; fraction = 0.4; p_lower = 0.23; p_higher = 0.34
#'
#' Hmisc::bsamsize(p1= p_lower, p2 = p_higher, fraction = fraction,
#'                 alpha = alpha, power = power)
#'
#' calculate_binomial_samplesize(ratio0 = fraction, p1= p_higher, p0 = p_lower,
#'          alpha = alpha, beta = power, one_sided = FALSE, continuity_correction = FALSE)
#'
#'
#' @export


calculate_binomial_samplesize <- function(ratio0,
                                          p1,
                                          p0,
                                          alpha = 0.05,
                                          beta = 0.8,
                                          one_sided = TRUE,
                                          continuity_correction = TRUE){

  if(!is.numeric(ratio0) | !is.numeric(p1) | !is.numeric(p0))
    stop("Input parameters ratio0, p0 and p1 need to be numeric.")

  if(!is.numeric(alpha) | !is.numeric(beta))
    stop("Input parameters alpha and beta need to be numeric.")

  if(max(c(alpha, beta, ratio0, p0, p1)) >= 1 | min(c(alpha, beta, ratio0, p0, p1)) <= 0)
    stop("Input parameters ratio0, p0, p1, alpha, beta need to be in the interval (0,1)")

  delta = p1 - p0             # Nomenclature as in the paper

  r = 1 / ratio0 - 1          # Uniting the definitions

  if(one_sided == FALSE) {    # Last statement of the paper
    alpha = alpha / 2
    delta = abs(p1 - p0)
  }

  p_bar = (p0 + r*p1)/(r+1)

  m_dash_1 = qnorm(1 - alpha, mean = 0, sd = 1)*sqrt((r+1)*p_bar*(1 - p_bar))

  m_dash_2 = qnorm(beta, mean = 0, sd = 1)*sqrt(p1*(1-p1) + r*p0*(1-p0))

  m_dash = ( m_dash_1 + m_dash_2 )^2 / (r * delta^2)

  if(continuity_correction == TRUE){
    m_dash = m_dash + (r + 1) / (r*delta)
  }

  return(c(size_0 = m_dash,
           size_1 = r*m_dash,
           size_overall = m_dash + r*m_dash))
}

Fleiss JL, Tytun A, Ury HK (1980): A simple approximation for calculating sample sizes for comparing independent proportions. Biometrics 36:343-6.

New Feature on the Blog: Publication List

Sun, 08 Sep 2019 00:00:00 +0000

On this page I have collected the bibliographic details of all the articles I have written, mainly of scientific (i.e. statistical) nature. I have written almost all of the scientific papers while under contract at a public university, either in Darmstadt or in Heidelberg. I have submitted all papers to journals which are edited and refereed by other scientists in presumably public institutions. Yet, for some reason, the results of the publically funded, publically relevant¹ research is NOT officially publically available. This doesn't make sense.

Therefore I have included the DOIs in the bibliography, which make the search on Sci-Hub possible². I am a big fan of Sci-Hub, and I do hope that at some point in the future it will become obsolete, because all scientific research is just publically availabe legally. Until such time, Sci-Hub is an acceptable approximation; as of this writing, all my articles were available³.

I'm comfortable with the assessment that my own research isn't that publicaly relevant. My point is still valid.

But, you know, … don't.

… or so I have heard.

Chester (sorry, Liverpool) is the Most Popular City in the World (relative to use as password per inhabitant)

Fri, 08 Feb 2019 00:00:00 +0000

<b>Update 2018-02-17</b>: The title of this article has changed reflecting new information I have received since publishing. For mor information, I refer to the last paragraph.

A treasure trove of leaked passwords

The API of pwnedpasswords.com is quite remarkable. It not only allows you to fetch the results generally obtained by typing in your e-mail into the browser interface and finding out whether or not you've been pwned from the comfort of your shell. It further allows you to very simply check whether a certain password has ever been used in any of the dumps they have, and if so, how often. Since haveibeenpwned.com has collected over 550 millions of these in a multitude of data breaches, odds are your password might be amongst these.

Using the API is straightforward, but the way it is secured so that even pwnedpasswords.com itself doesn't know precisely which password or even password hash you are checking is ingenious, I highly recommend reading their tutorial. The following R snippet allows for obtaining the number of hits for a vector of passwords:

library(httr)
library(digest)
library(plyr)
library(dplyr)
library(stringr)

popularity <- function(password){
  passw_hash = digest(password, "sha1", serialize = FALSE)
  passw_front = toupper(substr(passw_hash, 1, 5))
  passw_back = toupper(substr(passw_hash, 6, 200))
  hashes = 
    read.table(text = content(GET(paste0("https://api.pwnedpasswords.com/range/", 
                                         passw_front)),
                              encoding = "UTF-8"),
               sep = ":") %>%
    rename(hashes = V1, count = V2) %>%
    filter(hashes == passw_back) %>%
    mutate(password = password) %>%
    select(password, count)
  if(nrow(hashes) == 0){
    hashes <- tibble("password" = password, "count" = 0)
  }
  return(hashes)
}

City names as passwords

Using this function allows us to search through various ranges of passwords. For instance, let's see how many people have chosen the names of cites in Baden-Württemberg as their passwords¹:

City Name	Number of Usages as Password
Freiburg	3077
Stuttgart	9496
Karlsruhe	1426
Heidelberg	4081
Mannheim	5040
Konstanz	924

It seems like the city name "stuttgart" appears most often in the password list, yet that is not incredibly surprising, as it is also the largest city. A plot of the number of password hits in relation to the number of inhabitants² looks like this:

<div class="figure"> <img src="" width="438" height="481" alt=""/> <p class="caption"> The red circle describes the number of inhabitants, the black circle the number of usages as password. The plot was created with ggmap. </p> </div>

Quite obviously, the ratio of "number of uses of city name as password" and "number of inhabitants", i.e. "Use of City Name as Password per Inhabitant" differs from city to city. It seems like this ratio is higher for the cities Heidelberg and Freiburg, each of which is known for a high quality of living and a very picturesque old town. So, let's look at some international (and especially British) competition:

City Name	Inhabitants	Number of Usages as Password	City Names as Password per 1000 Inhabitants
Liverpool	473073	280723	593.4
Manchester	520215	98831	190.0
Oxford	161291	23069	143.0
Cambridge	151832	12648	83.3
Heidelberg	160601	4081	25.4
London	8787892	196220	22.3
Mannheim	307997	5040	16.4
Paris	2190327	28699	13.1
Berlin	3613495	40952	11.3

In this longer list, the effect of having a famous football team (Liverpool, Manchester) as well as having a famous university in a small city (Oxford, Cambridge and Heidelberg) becomes obvious. In other words, it's not so much about how many people live in a certain city, but how many people feel a positive connection to that particular city, by loyality of a sports team or by time spent at the university.

Let me conclude this article by pointing out the obvious question: "Can any city beat Liverpool" in this contest? All my manual samples have so far yielded good results, but nothing close to Liverpools numbers, for instance:

City Name	Inhabitants	Number of Usages as Password	City Names as Password per 1000 Inhabitants
Liverpool	473073	280723	593.4
Green Bay	105139	23069	143.0
Barcelona	1620805	152196	129.9

Thus, until futher notice, I proclaim hereby that Liverpool is the most popular city in the world (relative to password use per inhabitant).

Update 2018-02-17: Chester has overtaken Liverpool

I published this post on Feburary 8th and have since received feedback concerning the core result of this post (i.e. which city is the most popular), in particular, the following tweet: <blockquote class="twitter-tweet" data-lang="de"><p lang="en" dir="ltr">Actually, the use of Chester as a password beats Liverpool on the same metric. 117,128 occurrences for a population of 118.200, i.e. 990.9 per 1000 population. cc <a href="https://twitter.com/CheshireLive?ref_src=twsrc%5Etfw">@CheshireLive</a></p>— John Murray (@MurrayData) <a href="https://twitter.com/MurrayData/status/1094957209400893440?ref_src=twsrc%5Etfw">11. Februar 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

@MurrayData is correct of course, "chester" has 117128 occurrences in the data base. "Chester" is not only a city name, however, but also a Christian name, and in particular that of the late lead singer of Linkin Park, Chester Bennington. It is plausible that many of these passwords are disconnected from the city itself. Yet the same argument hoilds for soccer teams as I had already pointed out in my original post, thus I opted to ignore these distortions, and I hereby present the new, updated table:

City Name	Inhabitants	Number of Usages as Password	City Names as Password per 1000 Inhabitants
Chester	118200	117128	990.9
Liverpool	473073	280723	593.4
Manchester	520215	98831	190.0
Oxford	161291	23069	143.0
Cambridge	151832	12648	83.3
Heidelberg	160601	4081	25.4
London	8787892	196220	22.3
Mannheim	307997	5040	16.4
Paris	2190327	28699	13.1
Berlin	3613495	40952	11.3

Rules are "only lowercase letters, spaces are eliminated". So for "New York" I looked for usage of "newyork", for instance.

Numbers of inhabitants as taken from wikipedia.org.

Tired of Waiting for your R Scripts to Finish? Let AWS do the Work, Get Notified by E-Mail

Thu, 27 Sep 2018 00:00:00 +0000

The Setting: Avoiding 4 Weeks of Runtime

Recently, I was faced with a problem: I had written a rather complex simulation of a discrete time queueing network, and I needed to let this simulation run

with some repetitions of the entire simulation,
for some varying different parameter values,
with many observations (i.e. ~ 2.000.000 observation).

The goal was to verify that a new estimating procedure for such queueing networks provides sensible results. For more details on the matter, I refer interested readers to my previous articles on this topic, available here and here, or under the DOIs 10.1080/15326349.2015.1060862 and 10.1016/j.spa.2014.09.003 for all you fans of SciHub¹ out there .

Anyways, the situation described above wouldn't be problematic as such, but the runtime for a single one of these simulations increased exponentially: for 1500 observations it ran ~ 0.3 secs, for 15000 ~ 9 secs, and for 30000 ~ 38 secs. My goal was to reach 1.500.000 observations for at least 3 times, so that I was facing (by very rough calculations, of course), 2430000 seconds, or a 4 week marathon, in which my laptop would be doing nothing else but crunching numbers.

Naturally, I wanted to avoid that and considered using Amazon AWS service for this task. I have described this in a recent blog post (and to make it clear: I don't get any money from AWS, I just like their usability). However, my requirements were a bit stricter than last time, I needed a workflow that

Starts up a decent sized AWS instance for my calculation,
Puts my entire simulation code on this instance,
Starts the simulation with a freely chosen parameter setting,
Notifies me by mail as soon as the simulation was done.

The last point was important since I was planning on using larger instance types, and AWS bills by the hour. Obviously, I didn't want to let precious money go to waste, so I needed a trigger to terminate the instance as soon as the simulation was over.

The Step-by-Step Guide:

Setting up the AWS instance

I have previouly posted a detailed description for how to set up an AWS instance, and that walkthrough guide is still viable. In a nutshell, the core ingredients are

an AWS account,
the RStudio AMI of Louis Aslett,
a security group that allows ssh and html access,
the correct specification of the instance.

For the latter, correct specification is obviously a non-trivial requirement. In my benchmark test for AWS instances I was aiming at a RAM intense task and ended up with a recommendation for the "r4" instance class. In my use case, however, the CPU performance was decisive. For this reason, I opted for a c5.large instance, but your experiences may vary.

Setting up E-Mail transmission from the instance

Once the AWS instance is up an running, I needed to enable it to send mails, for this I followed this post here. One first needs to install the necessary software by connecting to the instance via ssh and installing

sudo apt-get install ssmptp mpack

ssmtp allows for the configuration of the SMTP sending, mpack makes for an easy CLI for sending e-mails with attachments. Now, we need to tell the instance which E-Mail client to use for outgoing mail. In my case, I used my Google mail address for this purpose. Unfortunately, this path necessitates that the password to the gmail account needs to be saved unencrypted in the file /etc/ssmtp/ssmtp.conf, like so:

#
# Config file for sSMTP sendmail
#
# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=sastibe.aws@gmail.com

# The place where the mail goes. The actual machine name is required no
# MX records are consulted. Commonly mailhosts are named mail.domain.com
mailhub=smtp.gmail.com:587

# Where will the mail seem to come from?
#rewriteDomain=

# The full hostname
hostname=*****

# Are users allowed to set their own From: address?
# YES - Allow the user to specify their own From: address
# NO - Use the system generated From: address

FromLineOverride=YES
AuthUser=sastibe.aws@gmail.com
AuthPass=**************
UseSTARTTLS=YES
UseTLS=YES

In order for gmail to accept this unencrypted password, you need to checkmark the option for allowing 'less secure apps' to access your mail, in Google settings under 'Sign in and security': <div class="figure"> <img src="" alt=""/> <p class="caption"> Doesn't feel right to click it, does it? </p> </div> As you can probably gather from the e-mail adress in the code snippet, sastibe.aws@gmail.com is just a throw-away address for precisely this reason: I did not want to leave my real gmail account open to vulnerabilities.

Getting R Code on the Instance

This step is ostensibly simple, since you can just copy and paste the code to the RStudio window in your browser. But I did it differently: I wrote my simulation as an R package (a very nice and straightforward how-to can be found here). Writing a package comes with many advantages by forcing you to be clean on

documentation
testing
putting functionality in functions

Furthermore, since used Git with a remote repository on GitLab for version control, I am able to call my entire code with a simple

install.packages("devtools")
devtools::install_git(url = "https://gitlab.com/sastibe/spa_queueingnetwork",
                       branch = "dev")
library(queueingnetworkR)

Start the Simulation and the E-Mail notification

After all these preliminary steps this is the simplest of them all. I log on the RStudio instance by navigating to the public DNS, using the credentials "rstudio" and the (AWS) instance_id as password. Then, I simply choose an appropriate set of parameter values steering the simulation and call just one tailor-made function. Admittedly, this function was written with exactly this use case in mind, so your script lengths might vary. In my case, this was the entire run script I pasted into the RStudio on the AWS instance:

install.packages("devtools")
devtools::install_git(url = "https://gitlab.com/sastibe/spa_queueingnetwork",
                       branch = "master")
library(queueingnetworkR)

p_12 = 0.5
p_21 = 0.2
n_obs = 1000
burn_in = 5000
lambda_1 = 0.8
lambda_2 = 0.2
G_1 = 0.6
G_2 = 0.5

firstrun <- present_estimates(n_reps = 1, max_lag = 10, p_12 = p_12, p_21 = p_21, 
                              n_obs = n_obs, burn_in = burn_in, lambda_1 = lambda_1,
                              lambda_2 = lambda_2, G_1 = G_1, G_2 = G_2, progress = TRUE)

save(firstrun, file = "resultate.zip")
ggsave("result_G2_plot.png", plot = firstrun$plot_result_G2, device = "png")

system(paste0("mpack -s 'Skript durchgelaufen mit ", n_obs, 
              " Beobachtungen: Plot G2' result_G2_plot.png sastibear@gmx.de"))
system(paste0("mpack -s 'Skript durchgelaufen mit ", n_obs, 
              " Beobachtungen: Daten' resultate.zip sastibear@gmx.de"))

The last two lines are a simple system call of the function mpack, which uses the credentials provided in Step X to send a mail from my GMail account to "sastibear@gmx.de", and attaching the plotted file G2_result_plot.png as an attachment.

Receive the Glorious Results

I started the script above on a Friday afternoon, and received the following e-mail on Monday: <div class="figure"> <img src="" alt="" /> <p class="caption">Boom, Publishable!</p> </div>

Is it Worth the Effort?

In short: yes. Longer version: The answer to this question obviously depends on the metric. In terms of time elapsed, the calculation is simple: 4 weeks on my personal laptop vs. 3 days on AWS is a very one-sided competition. But let's try to look at the monetary aspect:

Calculation on…	Duration	Cost / hour	Type of Cost	Cost total
Laptop	4 weeks	0,87 Cents²	Electricity	5,84 €
AWS	3 days	9,7 US Cents	AWS fee	5,82 €³

Well, that's … surprising? To be quite honest, I started calculating this comparison under the strong preconception that AWS must surely be more expensive. Yet the accumulation of runtime over 4 weeks is enough to give the Amazon server farm the edge over my old-school laptop solution, thus cloud computing takes the cake once more. Hooray for technological progress!

I am one of them.

Calculated with an average consumption of 30 Watts an hour an an average electricity price of 0.29 € per kWh.

With an exchange rate of 0,85 $ to 1 €.

Registering my Blog at the VG Wort

Tue, 17 Jul 2018 00:00:00 +0000

Suddenly, Attention

So, it seems like my blog is getting some attention. Recently, I was even featured on the front page of Hacker News:

<div class="figure"> <img src="" alt="Of course I made a screenshot" /> <p class="caption">Number 17… only 16 more to go!</p> </div>

Obviously, I am delighted and flattered by the number of people reading and discussing my blog. But since we're living in a material world, and I am a material guy, I kept on wondering "is there any way to monetize this in an ethical way?". Obviously, advertisements are out of the question, but are there other ways?

VG Wort

The VG Wort, or Verwertungsgesellschaft Wort is a German association tasked with collecting and distributing profits made secondary exploitation rights of original textual contents. Essentially, it tries to reimburse authors and journalists for the money they "lose" by texts that are copied or digitally reproduced and thus "free of charge". The money received by the VG Wort stems mostly from so ca lled Kopierabgabegeräten ¹.

A Step-by-Step Guide for registering your Blog at VG Wort

In what follows, I recount the steps I needed to take to register my first blog post in VG Wort. Note that all of these steps are exclusively available to German citizens, which is unfortunate: An internanationally available solution like the one provided by VG Wort for "making money on the internet without resorting to ads" could have beneficial consequences for everyone. But I digress…

Registering an account

New authors nedd to first register an account, this can be done here. I already had my account set up for my scientific papers.

Registering a blog post

The first informational page on how to register a blog post is given here. It states the following preliminaries have to be met:

The blog post has to contain at least 1800 characters (including whitespaces)
There my be no copy-restriction on the resulting file (DRM)
The text was read by a certain number of people, counted by a VG Wort tracking pixel, or "Zählmarke"

Obtaining the Zählmarken is straightforward: In my VG Wort Account for the "registration of texts online", or Texte online melden, i.e. T.O.M. in German, I was able to order 100 Zählmarken as a somewhat weirdly formatted csv:

<div class="figure"> <img src="" alt="Ahh, LibreOfficeCalc in all its … beauty"/> <p class="caption">Nevermind the encoding problems… but repeating the header for each line? C'mon!</p> </div>

Keeping Track of the Zählmarken

Seeing the poorly formatted csv made me realize I needed to keep track of my Marken in a more sensible way. So I came up with the following (dummy) org-table:

Filename Zaehlmarke	No.	Public Key	Private Key	Post URL	No. Characters
vgwort_1.csv	1	"https://vg09.met.vgwort.de/na/foo"	bar	2018-01-27-setting-up-a-scalable-rstudio-instance-in-aws.html	12200

Counting the number of characters in all the blog posts using hugo

My blog is written in hugo, I'm using the Tranquilpeak theme to be precise. Even though I write most of my articles in org-mode nowadays, there's still quite a lot of HTML clutter in my source files. How to count the characters excluding such "meta-characters"?

My solution is a bit indirect, but it works quite well. For my RSS feed, I customized the template rss.xml in order to read

      ...
      {{ with .Site.Author.email }}<author>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</author>{{end}}
      <guid>{{ .Permalink }}</guid>
      <description>{{ .Content | html }}</description>
      ...

The small alteration of using .Content instead of the default .Summary is, that my feed contains "full content", i.e. the entire post. In order to count the characters in all of my posts, however, I alter the respective line to read

      ...
      {{ with .Site.Author.email }}<author>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</author>{{end}}
      <guid>{{ .Permalink }}</guid>
      <description>{{ .Plain | html }}</description>
      ...

hugo uses .Plain to wipe away all HTML parts of the content. Hence, I can build my blog locally, open the resulting index.xml in emacs (obviously) and run M-= on the post.

How much money is in it, though?

As for this question, I have no answer yet. As of the publishing of this post, I have implemented the Zählmarken in all of the posts, corrected the Datenschutzerklärung (again, obviously) and am now waiting for results. If I understand the process correctly, I will have news by July next year. So, let's see, eh?

Footnotes

I love the German language. It seems like no word can be considered official enough for law texts unless it has at least 5 syllables.

It's here: Org Agenda for the World Cup 2018

Tue, 05 Jun 2018 00:00:00 +0000

As <a href="2018/05/2018-05-11-emacs-org-mode-rest-apis-stocks">I recently pointed out</a>, I have grown rather fond of Emacs and org-mode especially in recent months. On an entirely unrelated note, the FIFA world cup is right around the corner. Wouldn't it be nice to combine my passion for the greatest sport in the world (even including all the inevitable diving, arguing with the referees etc…) with my new-found passion for clear and concise org-agendas?

A short Google research lead to these two GitHub repositories for the Euro 2012 and the World Cup 2014. As I couldn't find anything similar for the World Cup 2018, I decided to create it myself.

Here is the result

The content of the schedule is as accurate as I could manage: I read the contents of the raw data into the desired org format via regex (in Emacs, obviously) and manually checked the results against the entries on the wiki page. Writing the regex was actually … rather fun? A bit weird since it wasn't pretty, let me show you the first few symbols:

replace-regexp <RET>
|\s-+\([0-9]\{1,2\}\) |\([^|]+....

Anyways, it worked, and the result is anyone's to enjoy. For the record, let me state clearly that I Do Not Intend To Update The Scores In The Repository in any timely fashion. If I get to it, I will, but let's see.

Use Emacs Org Mode and REST APIs for an up-to-date Stock Portfolio

Fri, 11 May 2018 23:13:13 +0200

A couple of weeks ago, I started to work with Emacs, and I grow fonder of it every day. During a very short time period, it has become my go-to editor for nearly everything I do on my computer, including (but not limited to)

planning my Todos (in org-mode, to be precise),
setting up my agenda (org-mode again),
taking memos during meetings
writing my (longer) e-mails
play around with new stuff
write blog posts (this is the first of these…)

It is difficult to pin down exactly why Emacs is taking over so much. My main influences for starting with EMACS were blogposts, the first describing a general EMACS setup, the second detailing how to implement GTD, i.e. Getting Things Done in Emacs org-mode.

In this post, I will demonsrate the strengths of using Emacs in a very specific use case: Getting up-to-date financial data to use in a spread-sheet including all your financial data. Applications like these are usually provided by online banks themselves, so that I don't show you anything particularly new or shiny. However, the ability to customize every step of the way brings with a number of advantages.

Finding a REST API for Stock Quotes

First, we need to get up-to-date financial data from a REST API. I decided to use Alpha Vantage, a site I first stumbled upon after reading this blog post. The other APIs listed on that page had various issues, either being deprecated in the near future (google, yahoo, stooq) or not having a number of symbols (IEX). The API of Alphavantage is rather easy to understand, even though the naming convention is terrible. Try for instance this link using the demo API key, yielding the following result for the Microsoft stock:

{
    "Meta Data": {
        "1. Information": "Intraday (1min) prices and volumes",
        "2. Symbol": "MSFT",
        "3. Last Refreshed": "2018-05-11 16:00:00",
        "4. Interval": "1min",
        "5. Output Size": "Compact",
        "6. Time Zone": "US/Eastern"
    },
    "Time Series (1min)": {
        "2018-05-11 16:00:00": {
            "1. open": "97.5900",
            "2. high": "97.7300",
            "3. low": "97.5750",
            "4. close": "97.7000",
            "5. volume": "3776187"
        },
        "2018-05-11 15:59:00": {
            "1. open": "97.4800",
            "2. high": "97.5900",
            "3. low": "97.4700",
            "4. close": "97.5900",
            "5. volume": "257615"
        },...

Reading API Requests into Emacs Variables

Having located the data out in the internet was a good first step, but now we need to figure out a way how to use this information. Luckily, most of the work needed for this can be found in various places on the net, for instance in this blog post. I decided to follow their general setup, using the following packages:

org
org-babel
request
json The API used in their scenario gave different results with a much cleaner nomenclature. For the Alphavantage API, I had to become a little creative with the eLisp code.

(require 'request)
(require 'json)
(require 'cl)

(request
 "https://www.alphavantage.co/query"
 :params `(("function" . "TIME_SERIES_INTRADAY")
           ("symbol" . "SC0J")
           ("interval" . "1min")
           ("apikey" . "..."))
 :parser 'json-read
 :success (function*
           (lambda (&key data &allow-other-keys)
             (setq open_sc0j (string-to-number (cdr (elt (elt (elt data 1) 1) 1)))))))

The variable open_sc0j is evaluated as follows: From the received json, take the second entry of the second element of the second element. Not very nice, but it works…

I encountered a second difficulty in my portfolio: I have both European stocks (in EUR) and American stocks (traded in USD). In order to keep my balances comparable, I added yet another variable rate_usd_eur, which receives up-to-date exchange rates from USD to EUR from the appropriate query. All in all, my requests to Alphavantage look like this:

(request
 "https://www.alphavantage.co/query"
 :params `(("function" . "TIME_SERIES_INTRADAY")
           ("symbol" . "SC0J")
           ("interval" . "1min")
           ("apikey" . "..."))
 :parser 'json-read
 :success (function*
           (lambda (&key data &allow-other-keys)
             (setq open_sc0j (string-to-number (cdr (elt (elt (elt data 1) 1) 1)))))))


(request
 "https://www.alphavantage.co/query"
 :params `(("function" . "CURRENCY_EXCHANGE_RATE")
           ("from_currency" . "USD")
           ("to_currency" . "EUR")
           ("apikey" . "..."))
 :parser 'json-read
 :success (function*
           (lambda (&key data &allow-other-keys)
             (setq rate_usd_eur (string-to-number (cdr (elt (elt data 0) 5)))))))

(request
 "https://www.alphavantage.co/query"
 :params `(("function" . "TIME_SERIES_INTRADAY")
           ("symbol" . "PG")
           ("interval" . "1min")
           ("apikey" . "..."))
 :parser 'json-read
 :success (function*
           (lambda (&key data &allow-other-keys)
             (org-table-iterate-buffer-tables)
             (setq open_prg (* rate_usd_eur (string-to-number (cdr (elt (elt (elt data 1) 1) 1))))))))

Putting this code inside an org-mode file, bracketing it by code blocks #+BEGIN_SRC emacs-lisp :results none and #+END_SRC, and hitting "C-c C-c" inside it leads to the evaluation of the code block and thus the filling of the variables open_sc0j, rate_usd_eur and open_prg. Since we included the wonderful little function org-table-iterate-buffer-tables, the evaluation also repeats until all the columns in the table below are calculated correctly. This neat little trick I also copied from here.

Setting up a Custom Stock Portfolio Org Table

After these steps, we now set up an org-table to give us a customizable overview of how our stocks are doing. That means setting up an org-table with columns for historic data, such as the date of the buy. Additionally, we use the #+TBLFM function to calculate appropriate performance indicators. An example for such functions:

| Stock         | Symbol | Amt. |    Buy | Date Bought     |  Fees | Dividends | Close |   Gain | Gain Perc | Gain per Day |
|---------------+--------+------+--------+-----------------+-------+-----------+-------+--------+-----------+--------------|
| MSCI ETF      | SC0J   |   10 |  47.11 | [2018-04-16 Mo] | 12.35 |         0 |       |        |           |              |
| ProcterGamble | PG     |    5 | 65.014 | [2015-10-01 Do] | 10.61 |     72.03 |       |        |           |              |
#+TBLFM: $9=(-$4 + $8)*$3 - $6 + $7;%0.3f::$10=100*$9/($4*$3)::$11=$9/(now() - $5)::@2$8='(format "%f" open_sc0j)::@3$7=17.78 + 17.70 + 17.94 + 18.61::@3$8='(format "%f" open_prg)

This code leads to the following result:

Stock	Symbol	Amt.	Buy	Date Bought	Fees	Dividends	Close	Gain	Gain Perc	Gain per Day
MSCI ETF	SC0J	10	47.11	[2018-04-16 Mo]	12.35	0	49.290000	9.450	2.0059435	0.35552367
ProcterGamble	PG	5	65.014	[2015-10-01 Do]	10.61	72.03	61.342605	43.063	13.247301	0.045111962

In this example, Gain is first calculated by multiplying -$4 + $8, i.e. the difference between Buy and (today's) Close by the amount of stocks bought. Additionally, any dividends are added and any fees are subtracted, yielding a "net gain" for the stock. In column Gain per Day, this number is broken down per day since I bought the stock, highlighting my most efficient assets.

There is no limit to what types of functions one can use, and no limit on the sophistication of analysis. And all of this within a very light-weight, easy-to-use interface, without any unnecessary over-head. It is not only convenient, but also educational: while writing this article, I learned a lot about REST APIs and financial data.

Let me conclude this article by picking up a picture from one of the blog posts that got me into Emacs in the first place: Emacs is like a classical steel frame road bike, reliant, robust, nothing fancy but easy to repair. It is the ideal tool to explore the wilderness of the internet. And I can only invite everybody else to come along for the ride.

Don't Worry: Google Only Checks Your Location Every 10 Minutes

Sun, 15 Apr 2018 00:00:00 +0000

I have a personal Google account, complete with gmail, gdrive and everything else. I first opened it up as a sort of spam email for all kinds of logins, but started to it use more and more due to its convenience. I was always slightly worried about the magnitude of data collected by Google on me, yet I never found a way to pinpoint exactly the extent of my slight worrying.

Recently, I discovered Google Takeout. Everybody with a Google Account can simply click here, follow the instructions and Google Takeout will send all the data it (supposedly) has in a nice little zip folder. Within this zip-folder is a file called “locationhistory.json” (or “standortverlauf.json” for all you German users out there), with entries such as this:

{
    "timestampMs" : "1523378268382",
    "latitudeE7" : 494290669,
    "longitudeE7" : 86872541,
    "accuracy" : 34
}

Each of these entries encodes a location measurement taken by Google, with GPS coordinates (latitude/longitude) and a timestamp, which can be converted to a “normal date” by dividing the number by 1000 and using, e.g., this handy site.

The “location history” file is rather large and unwieldy (about 18 MB in my case). There is a very simple and free tool that visualizes your location history data in an interactive heatmap. That is the tool I used to create the intro picture to this entry. The heatmap allows you to gauge the precision with which Google matches your movements. For instance, my skiing trip in March last year to Serfaus-Fiss-Ladis shows up like this:

Don’t worry, I also down some slopes during the vacation…

There are some mistakes in this map, i.e., places that I have surely never visited. I was never in “Gasthaus zum weißen Lamm”, I know that for a fact. However, the detail is quite astonishing, leading me to the next question: How often does Google measure and store my location data? My “locationhistory.json” contains 59293 observation over the course of 465 days, so that, on average, there are more than 5 measurements per hour.

I decided to look a little closer at the distributions of the timestamps, using some wonderful ggplot magic (the R code can be found here):

Such a colorful mountain range.

The lines in the plot show the average number of location measurements taken by Google each hour, separated by weekdays. The dashed line indicates the aveerage over all weekdays. The plot highlights several information:

Between noon and 8 pm, Google takes on average more than one location measurement every 10 minutes
In the nighttime, the average number of measurements is only once every 20 minutes
Monday and Tuesday mornings are closely watched with many measurements, especially Tuesday mornings
Afternoons and evenings are always of interest, but especially on Fridday and Saturday.

The fact that Monday and Tuesday morning are such exceptions could be explained by my specific calendar in 2017: I worked as a consultant and usually left home on Tuesday morning to travel troughout Germany. I am not entirely sure why this should lead to more measurements, as this activity was rarely accompanied by Google services (I travel by Deutsche Bahn). However, my travel time back home, usually late on Thursday evening, can also be seen quite nicely in the plot.

In total, I am a bit shocked by the sheer magnitude of measurements Google has on me, even (and especially) at times at which I am positively certain that I have never used Google Location Services, (see, e.g., 4 am). I am glad that services like Google Maps exist and that they are so extremely convenient, but the drawback should also be made abundantly clear to anyone who uses these services: many machine-readable aspects of your life are available to a for-profit company.

Benchmarking AWS Instances with MNIST classification

Sat, 10 Mar 2018 00:00:00 +0000

In a previous post I have shown you how to setup an AWS instance running the newest RStudio, R, Python, Julia and so forth, where the configuration of the instance can be freely chosen. However, there is quite a lot of possibilities of instance configurations out there: There are different instance classes (General Purpose, Compute Optimized, RAM Optimized, … ) and different instance sizes within these classes. For General Purpose, or t2, there are, e.g. t2.nano, t2.micro, t2.small, t2.medium, t2.large, t2.xlarge and t2.2xlarge¹.

These instances differ in two dimensions: price and performance. Obviously, these dimensions are highly correlated, since higher price means (or should mean, at least) higher performance. Now, price is easily measured, yet performance is a bit trickier: For example, it is not entirely straightforward to assess the impact of higher RAM, CPU or even GPU directly across many different configurations. But we’re doing data science, right? So why not create a programmatic test in order to gauge the performance empirically? Well, let‘s do it!

The Test

For this benchmark test I chose a classical machine learning task: the classification of the MNIST dataset of handwritten digits, to be categorized as 0-9. This data set is very commonly used as an example set for machine learning algorithms.

For this benchmark test, I borrowed a nice skript by Kory Becker written here, which trains a Support Vector Machine (SVM) on the problem, using only the first 1000 observations of the dataset, each with 768 attributes. I altered the code ever so slightly to that each run of the script returns the following measurements:

Elapsed Time: The time elapsed since starting the script (excluding the time to install the libraries and download of the data),
Accuracy of model, i.e. the percentage of predictions that classified the digits correctly.

Additionally, I included the following information:

RAM in Gigabytes,
Number of CPUs in use, and finally
Price in Dollars per Hour.

The Candidates

AWS provides a large number of different configurations, and I will not discuss all of these in this post. Rather, let me focus on four different specifications of computing resource demands and chose a distinctive representative:

General Purpose: t2, m4
Compute Optimized: c4
Memory Optimized: r4

For each of these classes, I had planned to test the sizes small, medium, large, xlarge and 2xlarge. The sizes micro, small and medium are actually only available for t2 (oh, no!), so that I ended up only testing 14 configurations.

The Results

I started with the candidate t2.micro, which is free of charge. Unfortunately, the script never succesfully ran the training of the model, presumably because the dimension of merely 1 GB of RAM is not sufficient. Still, a “not possible” result is still a useful result for choosing the right infrastructure.

Let’s have a first look at the results, first in plain numbers:

instance_class	instance_size	ram	vcpus	ecu	price_per_hour	elapsed_time	accuracy
t2	micro	1.00	1.0	NA	0.0134	NA	NA
t2	small	2.00	1.0	NA	0.0268	68.624	0.917
t2	large	8.00	2.0	NA	0.1072	65.335	0.918
t2	xlarge	16.00	4.0	NA	0.2144	55.611	0.918
t2	2xlarge	32.00	8.0	NA	0.4288	56.284	0.919
t2	medium	4.00	2.0	NA	0.0536	63.961	0.919
m4	large	8.00	2.0	6.5	0.1200	82.823	0.933
m4	xlarge	15.00	4.0	13.0	0.2400	80.749	0.928
m4	2xlarge	32.00	8.0	26.0	0.4800	65.728	0.912
m4	4xlarge	64.00	16.0	53.5	0.9600	64.573	0.927
m4	16xlarge	256.00	64.0	188.0	3.8400	93.310	0.915
r4	large	15.25	2.0	7.0	0.1600	80.749	0.928
r4	xlarge	30.50	13.5	4.0	0.3200	68.372	0.920
c4	large	3.75	2.0	8.0	0.1140	121.004	0.915

At a quick glance, the accuracy of the models looks quite uniform. This is hardly surprising, as the algorithm itselg is unchanged by hardware limitation, and the apparent fluctuations can be explained by the stochastic nature of the train-test-data set sampling in the script.

A core assumption is that more computing power yields faster results. A second core assumption is that the higher the computing power, the higher the cost. Combining these assumptions leads us to assume that higher cost leads to a lower time elapsed. A quick visualization of the data demonstrates that the results support this notion:

The measurement of the instance “m4.16xlarge” doesn’t quite fit into the pattern, and I am frankly unsure of the reasons. The measurement was taken twice, so that circumstantial errors leading to this measurement can be rejected.

Let us look a little more precisely at the data, in order to establish the most influential factors determining the speed of the analysis. We use the wonderful ggpairs visualization of the GGally package and omit the observation of the instance “m4.16xlarge” in the analysis:

This plot contains a number of results at once. First off, and unsurprisingly, the price per hour correlates vey strongly with the number of virtual CPUs and the size of the RAM, indicating that “the higher the computing power, the higher the cost” was a correct core assumption.

Second, the correlation between Elapsed Eime and the numeric indicators of performance RAM, vCPUs and Price per Hour is clearly negative across the board, but the highest correlation is clearly attained by the dimension RAM. This provides yet another indication that the notion Performance of R hinges on RAM is true.

One last question to consider: which instance type is optimal for R purposes? Optimality will be defined by provide the quickest results for the least money. Compare the fits of a standard linear model:

instance_class	(Intercept)	price_per_hour
m4	83.66913	-22.66862
r4	93.12600	-77.35625
t2	66.86029	-29.47335

This shows that the entry price is cheapest for instances of the class “t2”, as the y-intercept is the lowest in this case. However, in cases of higher Price per Hour, i.e. higher necessary computing power, “r4” is the better choice: The time decreases quickest with the increase in power. Both lines meet at a price per hour of roughly 55 Cents per hour, corresponding to an instance r4.2xlarge with 61 GB RAM.

Takeaway Messages

To conclude this article, let me summarize the key findings:

The most important hardware feature for the increasing computing speed of “R” analysis is RAM
For analysis with a small to medium scope of performance (RAM less than 60 GB), the instance class “t2” is the best choice in AWS.
For larger scale projects, the instance class “r4”, optimzed for RAM usage, is the optimal choice.

…why is it ‚nano‘, ‚micro‘ but then ‚large‘ ‚extra large‘? Be consistent, dangit!↩

My Motivations for Starting a Blog

Sun, 28 Jan 2018 00:00:00 +0000

Hello world!

My name is Sebastian Schweer, and I am a Data Scientist. This job description is increasingly popular, but it is notoriously difficult to describe precisely, what that entails. Let me show you one of my favourite definitions:

Source.

My job requires me to spend a lot of time each day writing code in varying languages, mostly R but also Python and SAS. This inevitably leads me to spend a lot of time thinking about both code as well as the process of programming itself. The major question is, as always, “How do you ensure, that your product is of the best quality?”. Recently, I stumbled upon¹ a incredibly concise diagram:

The importance of collaboration

I believe this is an astute observations, and I find its reflections in many daily situations including (but not limited to) producing code or data analyses. More precisely, I identified these 3 consequences of writing code with the intent of publicizing:

Tested: Nobody wants to publish content that only works once or only works on a certain local machine. Thus, any project up for publication automatically gets tested and tried much more meticuously.
Modular: It is much easier to explain and distribute several single clear ideas than one larger, vague idea. Hence, publication leads to more modular code, creating a more flexible and adaptive code base.
Documented: It doesn’t suffice if you as the author understand what the function with non-descriptive names such as fn_011_v3 does, that should be apparent from the name or at least from the documentation. The onus of understanding the code is transferred from the mind of the author to the body of the code.

All of these characteristics increase the maturity and quality of the code. Since I am obviously interested in producing high quality work, I started this blog in order to have a public outlet for all my private little programming projects.

The scope of these projects will vary wildly, I am sure, since the inspiration are heterogeneous. For instance, the first three posts have three different “sponsors”:

I wrote Setting up an RStudio instance on AWS with the audience of my father in mind, since he showed such an interest in my explanations of the topic over the Christmas break,
I wrote (or rather ‘will write’) the post A benchmark for dplyr vs. dbplyr for my sister-in-law, since she asked me about the topic and I didn’t know anything about it at the time,
and for the present article, or rather statement, I had my former employer in mind, a great fan of simple but concise diagrams depicting deep thoughts.

I appreciate any remarks or comments on anything that I write, and I wish you lots of entertainment perusing my site.

I can’t find the original source anymore, I have spent a long time going through my Twitter feed. If anyone recognizes the slide and especially the author, I would be incredibly thankful for the information and would gladly update the source information here.↩

Setting up a Scalable RStudio Instance in AWS

Sat, 27 Jan 2018 00:00:00 +0000

Assume you want to start to write R code (a very good decision, in my opinion) and you want to be able to write and test code whereever you are. Wouldn’t it be awesome if one could set up an environment that can be used for R coding independent of any device? Where all you need is a decent browser, a working internet connection and you’re good to go?

Obviously, that is the case. In this post, I will show you the steps for setting up such an environment on Amazon Web Services (AWS). The main advantages of using such a set-up:

Runs on any infrastructure: All you need is a working internet connection, a decent browser and an AWS account, which is usually¹ free.
Runs everywhere: The AWS machine will be set up to automatically clone your GitHub repository (don’t worry if this doesn’t mean anything to you, this point is optional), so that you don’t even have to have your codes on the device.
Scalable: The AWS machine running your code can be chosen to suit any of your needs, in any session. Just playing around with a new package? Use the smallest size, doesn’t cost a dime. Trying to re-create state-of-the-art machine learning performance with a fancy DNN-classifier? Go all in with 500 GB of RAM; it’ll cost ya, but it’s fun.
Up-to-Date: Since the envirionment is freshly installed each time, your R version as well as the package versions in use are automatically up-to-date. In the latter case, that would also be easy to maintain on a local machine, the former, however, is a nice benefit.

Convinced? Awesome, let’s get started!

Overview of main steps

First a short overview of the main steps covered in this blog post:

Get an AWS account (duh!),
Configure your RStudio AMI,
1. Find the right RStudio AMI,
2. Configure Security Groups,
3. Automatically Change your RStudio Password,
4. Incorporate a clone of your GitHub repo,
Start your First RStudio instance (and bask in its glory),
Create a personal AMI for future convenience,
Shut down the Instance and all Resources.

Preconditions for this tutorial should be basically none, at least in terms of coding and/or understanding R itself. The main task will lie in clicking the right buttons.

Step 1: Get an AWS account.

Well, it isn’t really my place to tell you how to get an AWS account if Amazon itself did such a great job explaining it. Just use the link to set up your account, and I further suggest to follow this set of instructions, building your very first instance. Take your time going through these instructions, I’ll wait…

Ready? Alright, sweet. Then we continue with

Step 2: Configure your RStudio AMI.

In this step, I collected several steps, not all of which are necessary. Steps 2a and 2b are crucial, Step 2c is recommended. Step 2d can be skipped on the first set-up. The implementation of this step can always be re-assessed whenever it becomes necessary.

Let’s begin by starting an instance in the AWS Dashboard. Just open “Instances” on the side menu of your EC2 Dashboard and click on “Launch Instance”:

Here we go!

Step 2a: Find the current RStudio AMI.

The first task is to choose an Amazon Machine Image, or AMI, which is essentially an operating system container. More to the point, in an AMI a Linux distribution can be bundled with addtional software packages tailored to any type of need: web development, accounting (I’m guessing here, but … sure) and, most importantly, using RStudio. On Louis Anslett’s homepage you can find a wonderful storage of RStudio AMIs. We use the newest version for the correct geographical zone, in my case Frankfurt:

One AMI for each region. Neat

As you can see, thanks to Louis Anslett’s work, the AMI includes not only the newest version of RStudio but also of R itself as well as a handful of helpful additional software packages. For instance, Git comes pre-installed, which we will use later on; also Juliais installed for those looking to try out the possible future of data science languages. But I’m deviating… Let’s note the AMI-ID (in our case “ami-a80db3c7”), put this in the start-up options and let’s continue.

Step 2b: Configure the security groups for your RStudio instance

In AWS, security groups control the access to the machine over the internet (if you don‘t care about how exactly this works and only want to follow the instructions, just skip the next sentences). More precisely, they define which kind of protocols may use which ports on your machine from a given IP range. For example, you can set the access rights for a ssh protocol to be able to connect to your machine on port 22 only from your personal IP address at home.

In our case, we actually only need access via http protocol, since the RStudio instance will allow log-in via browser interface. Therefore, our security group can be kept quite simple:

The bottom option allows the whole world to see the instance. Golly.

The IP range can be limited to your own personal IP to ensure the safety of your instance. This precaution could be necessary since only the login page of RStudio stands between the internet and your instance (spooky, huh?). However, since the personal IP usually changes each day (roughly speaking), this becomes a personal question of “privacy vs. convenience”. In my case, as you can see, convenience won.

2c. Automatically Change your RStudio Password

In the documentation of the RStudio AMI we can find the following passage: “It is highly recommended you change the password immediately and an easy means of doing this is explained upon login in the script that is loaded there”. Alright, fine, but I’d rather to that programmatically, i.e. automatically. The weirdly named “User data” option provides just the framework: All commands placed here get executed at the beginning of the start-up. You can find this setting in the menu “Configure Instance Details” under “Advanced Details”.

In order to change the password of the user “RStudio” on start-up, we paste the following code:

#!/bin/bash
echo "rstudio:guest" | chpasswd

where you should replace the password “guest” with whatever you deem appropriate. We are almost done with the set-up now, there only remains

Step 2d (optional): Automatically Clone a GitHub repo

I write all my private code projects on my GitHub account (here: https://github.com/sebastianschweer. What a shameless self-plug!) and I also would like my code to be available for me each time I start up my RStudio instance. Fortunately, this is easily configured with “User data” again, by just adding the command

git clone https://github.com/sebastianschweer/sastibe.git /home/rstudio/sastibe
chmod -R 777 /home/rstudio/sastibe

to the “User data” of Step 2c. Now, when I start up the new RStudio instance, the repository sastibe gets cloned into the folder /home/rstudio/sastibe, which is automatically loaded in RStudio. The line with chmod ensures that any user (not just root, who is executing this command at startup) has the rights to alter content in that folder. This permission allows me to change code and pushing my changes to the repository and all that, which is just super convenient.

Step 3: Start your First RStudio instance (and bask in its glory),

The last and most exciting click is this one:

Hooray, all the hard work pays off!

We have now started the instance. This means that a virtual machine, configured according to our specifitcations is being run on one of Amazon’s bajillion² cloud computing servers. In the menu “Instances” we now see an active instance running. After we are done, we will use this menu to shut it down again (so that it doesn#t cost us), but not now: we are eager to test it out! Accessing the instance is quite easy in our case: Just copy the “IPv4 Public IP” adress and paste it in your browser:

Green lights indicate the instance runs harmoniously.

Hopefully, you haven’t forgotten your password (check Step 2c if you did), your username is “rstudio”. After succesful login, you’ll be greeted by this screen:

The login to a world of wonder.

Et voilà: Your very own scalable RStudio instance, accessible world-wide and ready to use at all times. In other words: Congratulations, you now have a state-of-the-art Data Science Machine at your command. Use it wisely. If you want to see what kind of wonders you can do with this setup, check out the upcoming blog post. Otherwise, let me just point you towards another wonderful introduction.

Step 4: Create a personal AMI for future convenience

Now, Step 3 consisted of 4 different steps, and it would be ratehr inconvenient to have to repeat these steps each time you need a new RStudio instance, right? Luckily, AWS has got you covered: You can create an “image” of any AWS instance: simply put, this saves your current configuration for later use. The creation of such an image is straightforward: Just go to “Instances” in your AWS Dashboard, right-click on the machine you want to base the image on and select “Create Image”:

Locate “Create Image” in the menu of “Instance Settings”

After this step, you will find the created image in the menu AMIs, ready to reuse. Before you go do crazy and wonderful Data Science in your wonderful new Environment, though, it is essential that you let me tell you about

The Last Step (After Each AWS Usage): Shutting Down

An AWS instance doesn’t shut down by itself, or go into hibernation or anything like that. It just keeps running unless otherwise specified, eventually costing lots of money (even the free tier services have their prices after some limit). So, let me show you how to shut down your brand new machine. It’s quite simple, just right-click on the running instance and set the “Instance State” to “Terminate”.

Show no mercy, terminate!

Since our instance also automatically loaded an EBS volume (like a hard disk to save data), we need to shut that down too. Choose the entry EBS volumes in the sidepane and Detach all volumes that are active. If your overview in the pane “Dashboard” looks similar to this :

“5 volumes”: make sure that they are not in-use, since storage may also cost after some initial period

There are no hidden services running racking up costs.

Summary

After configuring your AWS environment as decried above, your new ‚Data Science Workflow‘ can look like this:

Log in to AWS,
Choose your personal RStudio AMI,
Choose the Necessary Specifications of the Machine,
Log in to the Machine in the Browser,
Do Awesome Data Science,
Shut Down Machine and all Resources.

Have fun, and remember: Primere non nocere!

For a given value of usually. I personally try to test out lots of resources just because I can, yet even so, my total expenses for AWS result in 0.37€ (January 2018).↩
A rough estimate. Maybe only bajillions.↩

Posts on Sastibe's Data Science Blog

Everyman Proportion Test

TL;DR

Why?

How?

How? (a bit more detailed)

What is actually happening here?

Interpreting the p-value

Further reading

Take Screenshots Straight into Org Files in Emacs on Win10

Update, 2023-04-05:

The Final Result

The Step-by-Step Guide (there is only one step)

Alternative solution with subfolders

The (a little more) detailed description

Creating a Unique Name for the Image

Getting the Screenshot into the Clipboard

Saving the Contents of the Clipboard to Disk

A toy example for Reinforcement Learning, CLI via fire, CI/CD on gitlab in Python

Eager to Play TicTacToe?

How it’s done: Reinforcement Learning

How it’s done: Package set up

How it’s done: CLI with fire

How it’s done: CI/CD in gitlab

Don't forget to sharpen your axe

A wise man once said…

The rule of three

My own ungrinded axe

Writing DIN-Norm (German) Letters in Emacs

TL;DR

Prerequisites

Setting up the capture template

Hints for further customization

Usage

Umfragewerte oder Medienpräsenz: Was kommt zuerst?

Einleitung und Fragestellung

Datenquellen und erste Einblicke 1

Die Analysemethode

Das Ergebnis

Setting up Emacs as Mail Client

The Final Result

Setting up the framework

Mu

Encrypting passwords etc.

Setting up offlineimap

GMail

Config for .emacs

Including mu4e

Basic setup

Encryption for XOAUTH2

Mu4e Contexts

Debugging

Sources & Acknowledgements

De-Googling the Blog (and my life)

Supplement to 'Nonparametric estimation of the service time distribution in discrete-time queueing networks'

Calculating the Required Sample Size for a Binomial Test in R

A Standard Problem: Determining Sample Size

The test statistic

Existing functions in R

A new R function: calculate_binomial_samplesize

New Feature on the Blog: Publication List

Chester (sorry, Liverpool) is the Most Popular City in the World (relative to use as password per inhabitant)

A treasure trove of leaked passwords

City names as passwords

Update 2018-02-17: Chester has overtaken Liverpool

Tired of Waiting for your R Scripts to Finish? Let AWS do the Work, Get Notified by E-Mail

The Setting: Avoiding 4 Weeks of Runtime

The Step-by-Step Guide:

Setting up the AWS instance

Setting up E-Mail transmission from the instance

Getting R Code on the Instance

Start the Simulation and the E-Mail notification

Receive the Glorious Results

Is it Worth the Effort?

Registering my Blog at the VG Wort

Suddenly, Attention

VG Wort

A Step-by-Step Guide for registering your Blog at VG Wort

Registering an account

Registering a blog post

Datenquellen und erste Einblicke ¹