Rida Ayed

Patch private methods in python

Sun, 08 Feb 2026 12:44:00 +0100

I’ve encountered this rather profane yet tricky problem while testing a method:

class MyClass:

    def __method(self) -> str:
        pass

def test()
   with patch.object(MyClass,"__method") as _:
       pass

pytest

E           AttributeError: <class 'MyClass'> does not have the attribute '__method'

A llm pointed to the name mangling cause.
Changing ~~__method~~ to _MyClass__method fixes the error:

def test()
   with patch.object(MyClass,"_MyClass__method") as _:
       pass

The reason for this post, is that searching (ctrl+f) for ‘mangl’ or ‘priv’ yields nothing from the doc¹

https://docs.python.org/3/library/unittest.mock.html
↩︎

Display the wireless status on the emacs mode-line

Fri, 23 Jan 2026 16:23:00 +0100

I found this code¹ to display the wireless status in the modeline.

A first attempt led to nothing i.e. nothing was shown in my modeline.

I had to modify the regex in order to match the output of /proc/net/wireless:

diff --git a/wireless.el b/wireless.el
index 55ec7fc..e08f864 100644
--- a/wireless.el
+++ b/wireless.el
@@ -55,7 +55,7 @@
   :group 'wireless)

 (defcustom wireless-procfile-regexp
-  "^[ ]+\\(....\\):[ ]+\\([0-9]+\\)[ ]+\\([0-9]+\\)[ .]+\\([0-9]+\\)[ .]+\\([0-9]+\\)"
+  "\\(^.*\\):[ ]+\\([0-9]+\\)[ ]+\\([0-9]+\\)[ .]+\\(-?[0-9]+\\)[ .]+\\(-?[0-9]+\\)"
   "*Regular expression for extracting proc-file data."
   :type 'string
   :group 'wireless)

cat /proc/net/wireless

Inter-| sta-|   Quality        |   Discarded packets               | Missed | WE
 face | tus | link level noise |  nwid  crypt   frag  retry   misc | beacon | 22
wlp3s0: 0000   59.  -54.  -256        0      0      0      2    195        0

As as result, I now have the wireless card status displayed in my modeline:

https://mail.gnu.org/archive/html/emacs-devel/2007-11/msg01680.html ↩︎

Stop google chromium from hijacking MIMEs on arch linux

Sun, 02 Nov 2025 11:19:00 +0100

In addition to this blogpost – whom focuses on chrome but on chromium, I had to run the following command to get rid of the chromium tick:

sudo sed -i 's/chromium.desktop;//' /usr/share/applications/mimeinfo.cache

Writing in ancient greek with Emacs and Linux

Thu, 02 Oct 2025 14:00:00 +0200

For starters, you want to write brother in ancient greek.

Type adelfos to get an understanding of what’s happening.
Set the input method to greek-babel: M-x set-input-method greek-babel.¹
Type adelfos again.

The last typing of “adelfos” results in αδελφοσ. So far so good.

However, αδελφοσ is wrong. It should be ἀδελφός. To be specific, we’re missing diacritics on a (alpha) and o (omicron) and the final s (sigma) is wrongly written as well.

We solve the problem by using combinations of letters, such as:
Typing >a results in ἀ
Typing 'o results in ό
Typing c results in ς

For completeness, type >adelf'oc in order to eventually get the desired ἀδελφός.

Good 🙂

Go back to your previous input-method via M-x toggle-input-method

Documentation

Read the documentation M-x describe-input-method greek-babel

Input method: greek-babel (‘BG’ in mode line) for Greek
  The TeX Babel input method for Classical Greek using mule-unicode-0100-24ff.

-------------------------------------
character     capital	      small
-------------------------------------
alpha		A		a
beta		B		b
gamma		G		g
delta		D		d
epsilon		E		e
zeta		Z		z
eta		H		h
theta		J		j
iota		I		i
kappa		K		k
lambda		L		l
mu		M		m
nu		N		n
xi		X		x
omicron		O		o
pi		P		p
rho		R		r
sigma		S		s
final sigma			c
tau		T		t
upsilon		U		u
phi		F		f
chi		Q		q
psi		Y		y
omega		W		w
-------------------------------------
sampi				!
digamma		#
stigma				$
koppa		&		%
-------------------------------------

------------------------
mark		key
------------------------
ypogegrammeni	|
psili		>
dasia		<
oxia		\='
koronis		\='\='
varia		\=`
perispomeni	~
dialytika	"
ano teleia	;
erotimatiko	?
----------------------

Optional - Increase readability

To increase the readability –of the in particular tiny diacritics–, you can enlarge the greek characters by doing the following:
Install the Gentium font: $ yay ttf-gentium
Evaluate this line or add it appropriately to your init.el:

(set-fontset-font "fontset-default" 'greek (font-spec :family "Gentium" :size 30))

Sources

https://stackoverflow.com/a/49765665/1172907
https://youtu.be/Fl4eu_bv54c?t=266

Note how your modeline now may display BG.
↩︎

Understanding yield in python

Thu, 14 Aug 2025 21:21:00 +0200

I’m testing a script against a live database.
Accessing the db is slow and the test iterates trough all rows each time.
Since I’m gradually increasing rows on each test run, I need only some, but not all rows.

Problem: How can I avoid loading all rows but only fetch some?
Solution: Use yield

Takeaway: Memorize the word yield as alias for fetch some.

Why words matter is beautifully reasoned by Raymond Hettinger in this youtube video.

Beware of emacs pytest runner defaults

Thu, 07 Aug 2025 09:13:00 +0200

Emacs provides an incredible python developer experience.
One of it many features is the ability to run pytest.
However, I spent a reasonable amount of time, figuring out why M-x python-test-one was ignoring my pytest log level settings in pyproject.toml.
Whatever I set in pyproject.toml, M-x python-test-one was printing out any level e.g. logger.INFO("do not show in test")

pyproject.toml

[tool.pytest.ini_options]
log_cli_level = "Error"

The solution was to remove the -s flag from ~~pytest-cmd-flags “-x -s”~~.
Add this line to your init.el:

init.el

'(pytest-cmd-flags "-x")

Now your pytest buffer shows the log at it’s appropriate level.

A minimal example of using namedtuples in pytest parameterized tests

Wed, 30 Jul 2025 15:01:00 +0200

What means - in the test parameters below?
Nevermind I don’t get it either.
Only after counting the commas I could guess gender.

@pytest.mark.parametrize(
"lang, salutation, gender, title, expect",
[
    ("DE", "","m","","doe","Hallo"),
    ("DE", "","m","Dr.","doe","Hallo Dr. Doe"),
    ("EN", "","-","Dr.","doe","Dear Dr. Doe"),
    ("FR", "","f","","xyz","Chère Madame Xyz"),
    ("EN", "Dear Bob","m","","doe","Dear Bob"),
])

To put the problem in full context, I’ve simplified and highlighted the parameters in the complete example below:

import pytest

@pytest.mark.parametrize("gender,lang", (
    ('-', 'EN'),
))
def test(gender, lang):
    assert gender

Namedtuples are the salvation for this problem.
Notice now how easy the highlighted gender = '-' is understood.

import pytest
from typing import NamedTuple

class TC(NamedTuple): #TC means TestCase
    gender: str
    lang: str

@pytest.mark.parametrize(
    "tc", (
    TC(gender='-', lang='EN'),
))

def test(tc):
    assert tc.gender
    assert tc.lang

I have described another mitigation for incomprehensive structures here.
Another approach using NamedTuple from typing is shown here youtube.
This blogpost –with a bit more text to glance over– inspired me to write this post.
Finally I settled to the current version of this post upon discovering this article.

Patch the async json response of a mocked aiohttp clientsession in pytest

Sat, 26 Jul 2025 14:58:00 +0200

Haven’t found a short concise example covering pytest whilst focussing the patched json method of a mocked async session, hence this snippet.

First install pytest-asyncio.

pipenv install -d pytest-asyncio

Now use it as highlighted.

from unittest.mock import patch, AsyncMock
import aiohttp
import pytest


async def request_async():
    async with aiohttp.ClientSession(headers={}) as session:
        async with session.get("foo.bar") as response:
            jsn = await response.json()
    return jsn


@pytest.mark.asyncio
@patch('aiohttp.ClientSession.get')
async def test_get_request(mock_get):
    expected = {"a": "b"}
    mock_response = AsyncMock()
    mock_response.json.return_value = expected
    mock_get.return_value.__aenter__.return_value =\
         mock_response
    result = await request_async()
    assert result == expected

Django python IDE autocompletion in emacs

Sun, 04 May 2025 18:58:00 +0200

Presuming a working python layer, following settings will provide autocompletion.
e.g. typing Mymodel.ob will suggest Mymodel.objects for completion.

yay basedpyright

~/.spacemacs.d/init.el

(defun dotspacemacs/user-config ()
(use-package lsp-pyright
  :ensure t
  :custom (lsp-pyright-langserver-command "basedpyright")
  :hook (python-mode . (lambda ()
                         (require 'lsp-pyright)
                         (lsp))))
)
...
dotspacemacs-configuration-layers
'(
(auto-completion :variables
              auto-completion-minimum-prefix-length 1
              auto-completion-enable-snippets-in-popup t
              auto-completion-enable-help-tooltip t
              auto-completion-return-key-behavior 'complete
              auto-completion-tab-key-behavior 'cycle ;;'complete 'cycle
              auto-completion-enable-sort-by-usage t
              auto-completion-idle-delay 0.0
              :config
              add-to-list 'company-backends '(company-yasnippet company-capf))
)
...

Get these values with pipenv --venv:

djangoDirectory/pyproject.toml

[tool.basedpyright]
venvPath = "/home/ra/.local/share/virtualenvs"
venv = "django-xyz123"

djangoDirectory/Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django = "==4.2.16"

[dev-packages]
django-types = "*"
pytest-django = "*"
debugpy = "*"
importmagic = "*"
epc = "*"
autoflake = "*"

[requires]
python_version = "3.9"
python_full_version = "3.9.2"

Play a video from emacs org mode using lisp

Fri, 07 Mar 2025 11:29:00 +0100

Links in Emacs org mode allow arbitrary emacs-lisp execution.
As a result, we can run any shell command, such as your favorite movie player:

[[elisp:(async-shell-command "mplayer /tmp/lecture.mp4")][linktitle]]

Arguments work too. e.g. play the movie from a particular position.
This can come in handy when learning materials contain valuable information scattered throughout movies:

[[elisp:(async-shell-command "mplayer /tmp/lecture.mp4 -ss 00:05:10")][othertitle]]

Ignore regions when formatting buffers with three keys

Fri, 14 Feb 2025 12:05:00 +0100

I’m quite fond of keeping my code formatted.
For example I use yapf to format python code.
Sometimes I want particular regions not to be formatted.
In particular I want rows to stay rows but one line.

The solution is to indicate such regions for the formatter

Without indication, the buffer is formatted from this:

rows = [
    ('john', 'acme'),
    ('jane', 'acme')
]

to that:

rows = [('john', 'acme'), ('jane', 'acme')]

With added indication the formatter ignores such regions:

rows = [
    # yapf: disable
    ('john', 'acme'),
    ('jane', 'acme')
    # yapf: enable
]

Now manually typing these opening and closing indicators each time would be a bit cumbersome, wouldn’t it?
Luckily emacs’ evil-surround has you covered.

Add this to your init.el

(with-eval-after-load 'evil-surround
  (setq-default evil-surround-pairs-alist
                (push '(?y . ("# yapf: disable" . "# yapf: enable")) evil-surround-pairs-alist)))

Now visually selecting any desired region and hitting gSy surrounds as explained.

Get an email upon failed systemd services

Tue, 11 Feb 2025 14:31:00 +0100

Only by chance, I noticed a failed systemd service. It’s a rather crucial
service, namely my backup.
I’ve decided to get notified by email in such circumstances.
The solution consists of three parts:

The actual backup service unit
A unit status mail service
A python email script

On failure, the actual service triggers the email service which in turn triggers
the email script.

The %h in the highlighted line, refers to the User home directory as per systemd.unit

[Service]
Type=oneshot
ExecStart=/usr/bin/sh $h/bin/backup.sh

[Unit]
Description=Homefolder backup
OnFailure=unit-status-mail@%n.service

[Unit]
Description=Unit Status Mailer Service
After=network.target

[Service]
Type=simple
ExecStart=/home/ra/bin/unit-status-mail.py %I

#!/bin/python
import smtplib
import argparse
from subprocess import Popen, PIPE
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

parser = argparse.ArgumentParser()
parser.add_argument("unit")
args = parser.parse_args()
unit = args.unit

usr = 'foo@bar.com'
pwd = 'secretpassword'
host ='mail.host.com'
port = 465

msg = MIMEMultipart()
msg['From'] = usr
msg['To'] = usr
msg['Subject'] = f'Failed: {unit}'
cmd=f'systemctl status --user --full {unit}'
# Be aware of security issues when using shell=True
stdout = Popen(cmd, shell=True, stdout=PIPE)\
    .stdout.read().decode('utf-8')
msg.attach(MIMEText(stdout))

with smtplib.SMTP_SSL(host,port) as mail_server:
    mail_server.ehlo()
    mail_server.login(usr, pwd)
    mail_server.sendmail(usr, usr, msg.as_string())

I wrote this article while metaphorically standing on the shoulders of Laeffe
Furthermore I made slight changes according to this question

Labeling code blocks in ox-hugo

Wed, 15 Jan 2025 14:57:00 +0100

Table of Contents

By chance, I stumbled over an excellent article¹ about labeling code blocks in hugo.
Roger’s article answers a major part of a question² of mine:

How to visually distinguish between blocks of source code and result

Problems

There’s no not enough visual difference between the blocks of code and result

print("foo")

foo

The code’s filepath must be written out to be displayed to the reader.

Regardless if it’s already stored as tangle header argument³, the filepath must be written twice in order to be visible

That means alongside :tangle xyz.py, I must as well write as a comment # xyz.py to show what is meant.

#+begin_src python :tangle xyz.py
# xyz.py
print("foo")
#+end_src

Solution

We distinguish the code block by adding a label above it.

Before:

print("foo")

After:

print("foo")

What happens

Input org mode file:

#+begin_src python :tangle /tmp/foo.py
print("foo")
#+end_src

Output markdown file:

```python { tangle_filepath="/tmp/foo.py" }
print("foo")
```

Final rendered html:

print("foo")

If not necessary, we can easily disable⁴ the tangle function while still having access to the arguments value.

How it works

Instead of adding the filename verbatim as comment #xyz.py like in code block, we intercept the value of :tangle.
That intercepted value then is passed, parsed and rendered as an additional html element named <figcaption> right above the code block.

My contribution is in two parts:

Adding a custom markup renderer

By combining parts of Roger’s instructions⁵ and the example in the hugo documentation, I’ve come up whith his custom code block

{{- with .Attributes.tangle_filepath }}
<figcaption>
  <span>{{ . }}</span>
</figcaption>
{{- end }}
{{ $result := transform.HighlightCodeBlock . }}
{{ $result.Wrapped }}

Adding some new logic to ox-hugo.el

First I’ve added a variable named tangle_filepath to the mighty ox-hugo.el⁶.
Second I’ve added some logic to convert the :tangle parameter in the org mode file to an info-string⁷ in the rendered markdown.

Both additions are highlighted below

diff --git a/ox-hugo.el b/ox-hugo.el
index 469002a..0a66f6b 100644
--- a/ox-hugo.el
+++ b/ox-hugo.el
@@ -3471,6 +3471,11 @@ their Hugo site's config."
      ;; Regular src block.
      (t
       (let* (;; See `org-element-src-block-parser' for all SRC-BLOCK properties.
+             (tangle-filepath
+              (let ((tangle-value (cdr (assoc :tangle parameters))))
+                (if (or (null tangle-value) (string= tangle-value "no"))
+                    nil
+                  tangle-value)))
              (line-num-p (org-element-property :number-lines src-block)) ;Non-nil if -n or +n switch is used
              (linenos-style (or (cdr (assoc :linenos parameters))
                                 ;; If `org-hugo-src-block' is called from
@@ -3572,6 +3577,11 @@ their Hugo site's config."
               (setq code-attr-str (format "%s, hl_lines=%s" code-attr-str hl-lines))
             (setq code-attr-str (format "hl_lines=%s" hl-lines))))

+        (when tangle-filepath
+          (if (org-string-nw-p code-attr-str)
+              (setq code-attr-str (format "%s, tangle_filepath=\"%s\"" code-attr-str tangle-filepath))
+            (setq code-attr-str (format "tangle_filepath=\"%s\"" tangle-filepath))))
+
         (when code-refs
           (let* ((anchor-prefix (cdr code-refs-and-anchor))
                  (anchor-str (format "anchorlinenos=true, lineanchors=%s" anchor-prefix)))

CSS styling

Literally similar to Roger’s article¹: “I’ll leave the styling (of the figcaption html element) as an exercise for the reader”.

https://write.rog.gr/writing/labeling-code-blocks-in-hugo/
↩︎ ↩︎
https://github.com/kaushalmodi/ox-hugo/issues/385
↩︎
https://orgmode.org/manual/Using-Header-Arguments.html#Header-arguments-in-Org-mode-properties
↩︎

* org mode outline header
  :PROPERTIES:
  :header-args:    :tangle no
  :END:

↩︎

https://write.rog.gr/writing/labeling-code-blocks-in-hugo/#using-custom-render-hooks
↩︎
https://github.com/kaushalmodi/ox-hugo/blob/main/ox-hugo.el
↩︎
https://gohugo.io/render-hooks/code-blocks/
↩︎

Mark tagged files partially in image-dired

Sun, 28 Jul 2024 11:48:00 +0200

Given your image-dired tags have similar beginnings e.g

0.jpg;foo
1.jpg;foo:

Now all you enter – in order to mark – images tagged with foo but not foo:, is this:
M-x image-dired-mark-tagged-files foo$

Note the dollar sign suffix

Why is such profane regex noteworthy?
See how image-dired-dired.el preliminary establishes the tag collection:

"\\(^[^;\n]+\\);\\(.*\\)"

Create a public S3 bucket using the aws cli

Sat, 13 Jul 2024 10:38:00 +0200

Table of Contents

Given is an application requiring public network storage.
I’ve deciced to give AWS S3 a try.
My choices are:

aws-cdk aka Infrastructure as code
boto3
aws cli bash script

Due to the non-repeating nature of that job, I’m going with the aws cli bash script

Create an arbitrary bucket name

bucket=$(python -m uuid)
echo $bucket

cd643901-34ab-48ec-9435-5ffd23c0580b

Create the bucket

aws s3 mb \
s3://$bucket \
--region us-east-1

make_bucket: cd643901-34ab-48ec-9435-5ffd23c0580b

Delete public access block

The necessity of this step requires further investigation

aws s3api delete-public-access-block --bucket \
$bucket

Create a policy to allow public read access on the bucket

/tmp/policy.json

{
  "Statement":[
    {
      "Sid": "AddPerm",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::cd643901-34ab-48ec-9435-5ffd23c0580b"
    }
  ]
}

Apply the policy to the bucket

aws s3api put-bucket-policy --bucket \
$bucket \
--policy file:///tmp/policy.json

Create an example file

echo foobar >> /tmp/foo.txt

Upload the example file to our bucket

aws s3 cp /tmp/foo.txt \
s3://$bucket

[Assert] Download the file using wget

wget -O /tmp/downloaded_foo.txt \
"https://$bucket.s3.us-east-1.amazonaws.com/foo.txt"

Delete all files in the bucket

aws s3 rm \
s3://$bucket/*

Delete the bucket

aws s3api delete-bucket --bucket \
$bucket

Call a bash script with arguments in python

Wed, 10 Jul 2024 10:53:00 +0200

/tmp/foo.py

arguments=" ".join(["foo", "bar"])
cmd="foo.sh"
subprocess.run(['/bin/env', cmd]+arguments)

/tmp/foo.sh

#!/bin/sh
for arg in "$@"; do
    echo $arg
done

Now calling python foo.py calls foo.sh with arguments.

Mind possible offtopic tasks like chmod +x, permissions etc.

Source

Fix treemacs-add-project wrong type argument hash table p nil

Wed, 03 Jul 2024 09:53:00 +0200

After a recent cleanup of treemacs projects, I wanted to add again a known project:

~/src/django

Yet M-x treemacs-add-project returned

(wrong-type-argument hash-table-p nil)

Debugging with M-x debug-on-error gave away a suspicious gethash call:

Debugger entered--Lisp error:
  (wrong-type-argument hash-table-p nil)
  gethash("/home/ra/src/django/crm" nil nil)
  treemacs-add-project("/home/ra/src/django/")
...

gethash("/home/ra/src/django/crm" nil nil) contains crm. But why?

So I started digging in the treemacs options by calling M-x custom-options and found treemacs-persist-file

A short glance at that file gave away the problem:

cat ~/.emacs.default/.cache/treemacs-persist

...
* Default
** crm
 - path :: ~/src/django/crm
...

The project was already there but only it’s subdir (crm) was added to treemacs. Mind that only the parent dir is versioned (.git).

Long story short 💡:

Check if a project’s subdirectory already exists in treemacs e.g. ~/src/django/crm before attempting to add it’s versioned parent e.g. ~/src/django.

Set up to share ox-hugo blog posts on social media

Tue, 02 Jul 2024 09:00:00 +0200

While preparing to share this post on LinkedIn, I noticed something’s missing.
The preview didn’t show an image or description.
Apparently there’s a special so-called Open Graph protocol that solves this problem.

From the protocol’s introduction:

The Open Graph protocol enables any web page to become a
rich object in a social graph. For instance, this is used on Facebook to allow
any web page to have the same functionality as any other object on Facebook.

I got it working in ox-hugo by:

Adding opengraph = true in the [params] section of hugo’s config.toml
Adding this org property for the image: EXPORT_HUGO_CUSTOM_FRONT_MATTER
Adding this org block for description: begin_description

Long story short. Here’s the applied change for the beforementioned post. Shortened but still raw:

git show 2ce4aa9

...
     :ID:       d8a908ad-6672-4931-b3ea-0b595a53e272
+    :EXPORT_HUGO_CUSTOM_FRONT_MATTER: :images '(/ox-hugo/2021-08-07_19-11-35_screenshot.png)
     :END:
+#+begin_description
+Create an image in blender using python. In the most minimal way we'll add a sphere and render the image.
+#+end_description
+
...

Font awesome icons in hugo footer

Sun, 05 May 2024 11:19:00 +0200

Table of Contents

👋 I followed this concise blog post¹ about embedding font awesome in hugo. Of course, I expected everything to work flawlessly in my environment as well. And of course, this presumption was wrong again.

Neither simply adding {{< fa bath >}} to the footer key in config.toml nor as content into any org-mode file will work.

This post shall show how to circumvent that flaws, use fontawesome in hugo’s footer section and overall in your blog’s content using ox-hugo.

Directory path and completeness

In contradiction to the instructions², I put the fontawesome directory ~~not in the static~~ but in the assets directory. I had to create it first.

Furthermore I put the complete fontawesome directory, not only a particular file. Filepaths in fa.html³, pointing to subdirectories, made me skeptic about picking only particular files.

Thus, after following the instruction steps about downloading and adding the fa.html, the siteroot/assets directory tree should look like:

tree -d -L 2

.
└── fontawesome
    ├── css
    ├── js
    ├── less
    ├── metadata
    ├── scss
    ├── sprites
    ├── svgs
    └── webfonts

10 directories

config.toml

The js key is documented in my theme’s repository⁴. You might encounter other ways for your particular theme in order to customize css/js assets.

[params]
js = ["/fontawesome/js/all.js"]
footer = '''
<a href="https://instagram.com/rida.ayed"><div class="fab fa-fab fa-instagram"></div></a>
<a href="https://github.com/RidaAyed"><div class="fab fa-fab fa-github"></div></a>
'''

Now the footer contains links with icons. Namely and

Usage of the icons with ox-hugo shortcodes in content

Although being documented⁵, I rather show the shortcode usage in ox-hugo again and specifically.

For example:

Lorem @@hugo:{ {< fa bath >}}@@ ipsum

Only by chance, I noticed that surrounding the shortcode with the equal sign “=” works as well.

For example:

Lorem ={ {< fa bath >}}= ipsum

You must delete that first space between the curly brackets from my upper examples. So {{ instead of { {
Both examples result in: Lorem ipsum

Conclusion

I showed how to extend the icon usage into the footer section of a hugo website. Furthermore you should be able to use the icon shortcodes in a org file considering ox-hugo as blog platform.

Python project packaging with pipenv pytest and cli

Mon, 22 Apr 2024 11:41:00 +0200

Table of Contents

How can I start a python project the proper way? Extensible, with tests, as library and cli application, in a virtual environment and with an older python version (raspberry pi).

I’ve spent days conquering the same problems over and over again. Notorious import problems. Imports from parent modules not working. Tests work but the cli doesn’t. Pipenv not installing the right python version. You name it.

Each time I think I’m all set, I’m encountering another missed step in my endeavour for clean code.😶

This ’essay’ shall be the base for my future python projects.

A minimal python project

Our starting point is: https://packaging.python.org/en/latest/tutorials/packaging-projects/ I’ll use the absolute minimum. E.g. pyproject.toml contains no description nor licence etc. Furthermore I’ve deleted files e.g. readme.md and directories e.g. tests.

/home/ra/tmp/$ tree packaging_tutorial

packaging_tutorial
├── pyproject.toml
└── src
    └── example_package
        ├── example.py
        └── __init__.py

3 directories, 3 files

Let’s have a look at the content of pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "example_package"
version = "0.0.1"

What’s most counterintuitive is that the name key under the [project] section in pyproject.toml must match the name of the package (example_package) and not the parent’s directory name (packaging_tutorial). In my mind the parent directory is the project’s name. In fact, the parent directory name seems completely negligible in perspective of development.

Installing the package in a virtual environment using pipenv with python 3.8

We replicate the steps from¹ not from pypi but locally from within the project’s folder (thus the dot . after -e).

pipenv install -e . --python 3.8

The -e flag (editable) allows further development (hence editing) alongside the installation.

But am I really installing if using a virtual environment? At this moment I’ll leave the flag since I haven’t detected any downsides to it yet.

Ensure the virtual environment has been created

pipenv --venv

/home/ra/.local/share/virtualenvs/packaging_tutorial-okktpVF3

Activate this project’s virtualenv

pipenv shell

Ensure version 3.8 is used as virtual python environment

python -V

Check the new directory structure

tree packaging_tutorial

packaging_tutorial
├── Pipfile
├── Pipfile.lock
├── pyproject.toml
└── src
    └── example_package
        ├── example.py
        └── __init__.py

3 directories, 5 files

Let’s have a look at Pipfile’s content:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
example-package = {file = ".", editable = true}

[dev-packages]

[requires]
python_version = "3.8"
python_full_version = "3.8.12"

Activate the virtual environment

pipenv shell

python
# Python 3.8.12 (default, Dec  4 2021, 10:54:00)
# [GCC 11.1.0] on linux
# Type "help", "copyright", "credits" or "license" for more information.

Mind that I changed the name from ~~example_package_YOUR_USERNAME_HERE~~ to example_package

>>> from example_package import example
>>> example.add_one(2)

Review the installation in perspective of pipenv and python 3.8

At this stage we’ve successfully replicated the package installation locally with pipenv and another python version (3.8).

Run something from the cli

Great. We’re able to import the add_one function from the example.py module in our virtual environment.

For better depiction I’ll add a print statement to our example.py module:

def add_one(number):
    return number + 1


print(add_one(1))

Now let’s run it from the command line interface (cli)

python -m example

home/ra.local/share/virtualenvs/packaging_tutorial-okktpVF3/bin/python: No module named example

Let’s check what’s actually installed in our virtual environment

pip list

Package         Version Editable project location
--------------- ------- -------------------------------
example_package 0.0.1   /home/ra/tmp/packaging_tutorial
pip             24.0
setuptools      69.2.0
wheel           0.43.0

Apparently we’re all set. Let’s try calling the module differently:

python -m src.example_package.example

This works but somehow feels wrong. Why do I need to be that explicit? What about the claimed automatic discovery² for our src-layout³?

After further reading⁴, let’s try only the package name:

python -m example_package

No module named example_package.__main__; 'example_package' is a package and cannot be directly executed

At least, we got a new hint from the error, which moreover matches the setuptools instructions⁴. Thus, let’s make it so and add __main__.py

# __main__.py
from . import add_one

if __name__ == '__main__':
    add_one(1)

ImportError: cannot import name 'add_one' from 'example_package' (/home/ra/tmp/packaging_tutorial/src/example_package/__init__.py)

At this point it looks like __init__.py is consulted due to the highlighted line 2 in __main__.py.

Since we installed the package, we will import this package itself in our modules’s cli entrypoint __main__.py.

Let’s change __main__.py

from example_package import example
# Works as well:
# from . import example

if __name__ == '__main__':
    example.add_one(1)

Now running python -m example_package in terminal being in /packaging_tutorial works.

Review our current (cli) project’s state

At this point we’re able to

pipenv install a package using an older python version
import our package as library
run the packages’s module from the terminal/cli

tree packaging_tutorial

packaging_tutorial
├── Pipfile
├── Pipfile.lock
├── pyproject.toml
└── src
    └── example_package
        ├── example.py
        ├── __init__.py
        └── __main__.py

3 directories, 6 files

How to pipenv install a git pull request from github

Thu, 11 Jan 2024 10:46:00 +0100

Given is this pullrequest https://github.com/joeyespo/pytest-watch/pull/108

pipenv install -d \
git+https://github.com/joeyespo/pytest-watch.git@refs/pull/108/head#egg=pytest-watch

The -d flag installs the package in the Pipfile’s [dev-packages] section

The #egg=pytest-watch suffix changes the pytest-watch Pipfile from "*" to {ref = "refs/pull/108/head", git = "git+https://github.com/joeyespo/pytest-watch.git"}

[dev-packages]
debugpy = "*"
-pytest-watch = "*"
+pytest-watch = {ref = "refs/pull/108/head", git = "git+https://github.com/joeyespo/pytest-watch.git"}
epc = "*"

Automating Image Manipulation in Emacs Dired Mode

Tue, 18 Jul 2023 15:52:00 +0200

I manage my images using emacs dired, specifically image-dired. I found myself repeatedly marking images in order to paste their filepaths into various shell scripts for further manipulation.

This article presents a convenient way to automate image manipulation tasks using a Bash script and Emacs Dired mode. It presents an approach to creating a Bash script that resizes images, applies padding and aspect ratio, and removes EXIF information. The script is then bound to a C-c C-x key in Emacs Dired mode, allowing users to easily apply these operations to multiple images with a single keystroke. The article also covers making the script executable and monitoring the output of the script using journalctl.

Image manipulation script

#~/bin/image_manipulation.sh
outdir="/tmp/$(date +%Y%m%d_%H%M%S)"  # Create a directory with the current date and time
mkdir $outdir

for arg in "$@"; do
    source="$arg"  # Store the source file path
    filename="${source##*/}"  # Extract the filename from the source file path
    outfile="$outdir/$filename"  # Specify the output file path
    tmpfile="$outdir/tmp_$filename"  # Specify the temporary file path
    logger "Source: $arg, outfile: $outfile, tmpfile: $tmpfile"  # Log the source, output, and temporary file paths

    convert "$source" -resize 1350x1350 "$tmpfile"  # Resize the image
    ~/Downloads/aspectpad -a 1.25 -m al -p white "$tmpfile" "$outfile"  # Apply padding and aspect ratio

    # Remove all EXIF information from the image except the Orientation tag
    exiftool -overwrite_original -all= -tagsfromfile @ -Orientation "$outfile"

    rm $tmpfile  # Remove the temporary file
done

# Save the outdir path in the Emacs killring for easy access
echo "$outdir" > /tmp/kill

exit

Make the script executable

sudo chmod +x ~/bin/image_manipulation.sh

Bind script to key in emacs dired mode

(defun my-dired-send-to-image_manipulation ()
  "Send marked files in Dired to ~/bin/image_manipulation.sh
  assure ~/bin is in $PATH)"
  (interactive)
  (let ((files (dired-get-marked-files)))
    (shell-command (concat "image_manipulation.sh " (mapconcat #'shell-quote-argument files " ")))
    (message "Sent to image_manipulation.sh: %s" files)))
(define-key dired-mode-map (kbd "C-c C-x") 'my-dired-send-to-foo)

Check script output in log

journalctl -f

Blender python development workflow

Mon, 16 May 2022 14:06:00 +0200

Edit the python script
Keep a terminal open running the watch script e.g. $ sh watch.sh
See the rendered result

Each time you save the python script, the watch script gets notified and re-executes. I’m using feh (note the highlighted line in the watch script) to see the rendered result, since it refreshes automatically.

#/tmp/blender_mini_example.py
import bpy
# Delete the default cube
objs = bpy.data.objects
objs.remove(objs["Cube"], do_unlink=True)
# Add a sphere
bpy.ops.mesh.primitive_uv_sphere_add(location=(-2, 0, 0))

# /tmp/watch.sh
frame=1
filename=/tmp/foo_
mypython=blender_mini_example.py
myblender ()
{
    blender \
        --background \
        --python $mypython \
        --verbose 0 \
        --factory-startup \
        --use-extension 1 \
        --render-output //$filename \
        --render-frame $frame
}
# Run blender the first time
myblender
# Open the result
feh ${filename}"000$frame.png" &
# Re-run blender each time the script is saved
while inotifywait --event modify $mypython; do
    myblender
done

Ignore transactions in ledger-cli

Mon, 01 Nov 2021 00:00:00 +0100

Given is a ledger file containing imported bank transactions we want to ignore. Let’s take a pending transaction for example. It has a merely formal character but no mathematical impact. Thus we want transactions having the word pending in the payee line not taken into account from ledger.

2012-03-10 KFC **pending**
    Expenses:Food                $20.00
    Assets:Cash

2012-03-10 KFC
    Expenses:Food                $20.00
    Assets:Cash

Code Snippet 1: example.ledger

             $-40.00  Assets:Cash
              $40.00  Expenses:Food
--------------------
                   0

We circumvent the pending transaction using automated transactions and a complex expression

= @/pending/
    ($account)                  -1

Code Snippet 2: Complex expression in an automated transaction

By prepending 1 with 2, the pending transaction doesn’t get accounted anymore

= @/pending/
    ($account)                  -1

2012-03-10 KFC **pending**
    Expenses:Food                $20.00
    Assets:Cash

2012-03-10 KFC
    Expenses:Food                $20.00
    Assets:Cash

             $-20.00  Assets:Cash
              $20.00  Expenses:Food
--------------------
                   0

The reason lies in the register command. Note that the result from 1 derives from the balance command.

The plain example register on 1 returns:

12-Mar-10 KFC **pending**       Expenses:Food                $20.00       $20.00
                                Assets:Cash                 $-20.00            0
12-Mar-10 KFC                   Expenses:Food                $20.00       $20.00
                                Assets:Cash                 $-20.00            0

Whereas with 2 appended we get this result from the register:

12-Mar-10 KFC **pending**       Expenses:Food                $20.00       $20.00
                                Assets:Cash                 $-20.00            0
                                (Expenses:Food)             $-20.00      $-20.00
                                (Assets:Cash)                $20.00            0
12-Mar-10 KFC                   Expenses:Food                $20.00       $20.00
                                Assets:Cash                 $-20.00            0

Note that the virtual transactions are surrounded by braces.

Summary

By using ledger’s builtin options, we’re able to ignore transactions instead of e.g. deleting them. The applied method uses complex expressions and automated transactions. In ledger terminology, the latter are called virtual transactions. Virtual transactions are evaluated on runtime. They aren’t read from any ledger file. On the opposite, real transactions are transactions that are read from the ledger file. It is crucial to note that virtual transactions can be ignored by using ledger with the real option e.g. ledger -f example.ledger reg --real

Shark in a flower at sunrise in Venice

Thu, 14 Oct 2021 19:18:00 +0200

Get latest ledger transaction's date only

Wed, 13 Oct 2021 16:33:00 +0200

#Content of ~/tmp/file.dat
2099/02/07 * acme
    Expenses:Groceries                                 $ 1
    Assets:Cash

2089/11/07 * foobar
    Expenses:Groceries                                 $ 4
    Assets:Cash

I want – from the account Assets:Cash and from the latest transaction (acme) – only the date.

I need the date formatted as Year-Month-Day.

ledger -f ~/tmp/file.dat reg \
       Assets:Cash \
       --tail 1 \
       --format %D \
       --date-format %Y-%m-%d

2099-11-07

Parse contacts from email in python

Sun, 05 Sep 2021 00:00:00 +0200

Process all your email contacts using mu and python.

mu-cfind returns all contacts:

mu cfind

Excerpt from the returned results

...
Newsletter Museum Tinguely infos@tinguely.ch
...

Script

import subprocess
from collections import namedtuple

# fetch raw input
query = "mu cfind"
subprocess.check_output((query), shell=True).splitlines()
runQuery = subprocess.check_output((query), shell=True).splitlines()
raw_contacts = [line.decode("utf-8").split(" ") for line in runQuery]

# process input to output
Contact = namedtuple('Contact', ['email', 'name'])

Contacts = tuple(
# Get email including only the last element
    Contact(email=raw_contact[-1],
# Get name from all elements excluding the last element
            name=raw_contact[:-1])
    for raw_contact in raw_contacts)

# print to stdout tab separated
for Contact in Contacts:
    print(Contact.email," ".join(Contact.name),sep="\t")

</tmp/mu_contacts.tsv>

grep "Tinguely" /tmp/mu_contacts.tsv

infos@tinguely.ch	Newsletter Museum Tinguely

Update your financial ledger by email in python

Wed, 18 Aug 2021 00:00:00 +0200

Table of Contents

Abstract

Each time I buy something and pay it later, my expenses and liabilities increase.

In this blog post I’m using the example of the reservation of a tennis court.

Once a year, the club sends an invoice for every reservation made. In order to reflect this in my finances, I note each reservation as a transaction into my ledger. The final goal is to create such a transaction into my ledger file completely automatic.

Goal

 2021/3/27 Tennis
     Expenses    CHF 10
     Liabilities

Manual process briefly explained

In order to keep my ledger up, a couple of steps are necessary. For a better understanding of what’s happening in the python script below, I briefly go over the manual process:

Query mu for emails

 mu find from:no-reply@t.gotcourts.com subject:Reservation --fields 'd s l'

Read the email headers

Backslash and line-break added manually for better reading

 Sat 27 Mar 2021 06:54:07 PM CET \
 GotCourts - Reservation confirmation \
 /home/ra/Maildir/AnyOtherProvider/INBOX/new/1616867898.105333_1.x,U=3935:2,

Extract the html content if plain-text is not provided

 mu extract --parts=1 /home/ra/Maildir/AnyOtherProvider/INBOX/new/1616867898.105333_1.x,U=3935:2,

Read html content

 </p><p>Partner: </p><p>&nbsp;</p><p>Price: 10 CHF<p>

Write the ledger post

Parse and fetch all information. Write the post as shown in Goal.

Script - Step by step

We start by importing the necessary modules

 import subprocess
 from datetime import datetime
 import re

Query mu and return the fields (d)ate, (s)ubject and fi(l)e

 MU_FIND_QUERY = "mu find " + myemailAlias + " --fields 'd s l'"

Run the current mu query

 def mu_command(MU_FIND_QUERY):
     return subprocess.check_output((MU_FIND_QUERY), shell=True).splitlines()

Create a list of results returned by mu

 EMAILS = [mystr.decode("utf-8").split(" ") for mystr in mu_command(MU_FIND_QUERY)]

In our case, the sender’s unable to provide plain-text. We tackle this obstacle by extracting the html part.

 MU_EXTRACT_QUERY = "mu extract --target-dir=/tmp/ --overwrite --parts=1 "

We subsequently store it temporarily /tmp/1.msgpart. Finally we read and return it’s content.

 def html_extract(file):
     mu_command(MU_EXTRACT_QUERY + file)
     with open("/tmp/1.msgpart") as f:
         return f.read()

We parse the content for the price. A match contains at least one positive integer followed by another zero based integer followed by CHF. Thus e.g. 10 CHF is a match.

 PRICE_REG = r"[1-9]+[0-9]*(?= CHF)"

If found, we return the price match

 def search_price(file):
     text = html_extract(file)
     if text:
         match = re.search(PRICE_REG, text)
         if match:
             return match.group(0)

mu returns the date’s month name abbreviated in English e.g. JAN. Thankfully python’s strptime method cover’s this format.

 def month(text):
     return str(datetime.strptime(text, "%b").month)

Now we’re all set regarding the necessary information to create a proper ledger posting.

For each parsed email we write/print a posting to our ledger file. But only if the email contains a price.

{email[3]}/{month(email[2])}/{email[1]} prints the year/month/day e.g. 2021/12/31.

email[-1] prints the emails file path, we pass it as argument to get the price by {search_price(email[-1])}.

 def parse(query_file):
     with open(query_file, "w") as f:
         [
             print(f'''\
 {email[3]}/{month(email[2])}/{email[1]} Tennis
     Expenses:Tennis    CHF {search_price(email[-1])}
     Liabilities:Tennis
             ''',
                   file=f) for email in EMAILS if search_price(email[-1])
         ]

Catch and print any exception

 try:
     parse(query_file)
 except BaseException as e:
     print(e)

Complete script

 import subprocess
 from datetime import datetime
 import re
 MU_FIND_QUERY = "mu find " + myemailAlias + " --fields 'd s l'"
 def mu_command(MU_FIND_QUERY):
     return subprocess.check_output((MU_FIND_QUERY), shell=True).splitlines()
 EMAILS = [mystr.decode("utf-8").split(" ") for mystr in mu_command(MU_FIND_QUERY)]
 MU_EXTRACT_QUERY = "mu extract --target-dir=/tmp/ --overwrite --parts=1 "
 def html_extract(file):
     mu_command(MU_EXTRACT_QUERY + file)
     with open("/tmp/1.msgpart") as f:
         return f.read()
 PRICE_REG = r"[1-9]+[0-9]*(?= CHF)"
 def search_price(file):
     text = html_extract(file)
     if text:
         match = re.search(PRICE_REG, text)
         if match:
             return match.group(0)
 def month(text):
     return str(datetime.strptime(text, "%b").month)
 def parse(query_file):
     with open(query_file, "w") as f:
         [
             print(f'''\
 {email[3]}/{month(email[2])}/{email[1]} Tennis
     Expenses:Tennis    CHF {search_price(email[-1])}
     Liabilities:Tennis
             ''',
                   file=f) for email in EMAILS if search_price(email[-1])
         ]
 try:
     parse(query_file)
 except BaseException as e:
     print(e)

Parse ledger-cli using python

Thu, 15 Jul 2021 00:00:00 +0200

#Content of file.dat
2099/02/07 * foo
    ; MD5Sum: 4d2965fa2879f3c528006f2e9c3bca8d
    Expenses:Groceries                                 CHF 10
    Assets:Cash

2099/11/07 * bar
    ; MD5Sum: 4d2965fa2879f3c528006f2e9c3bca8e
    Expenses:Groceries                                 CHF 20
    Assets:Cash

import subprocess
from datetime import datetime

format = "'%(cleared)\t%D\t%P\t%(quantity(scrub(display_amount)))\t%(join(note | xact.note))\n'"
query = "ledger -f file.dat csv Assets:Cash --csv-format " + format
runQuery = subprocess.check_output((query), shell=True).splitlines()
trx = [x.decode("utf-8").split("\t") for x in runQuery]
return datetime.strptime(trx[0][1],"%Y/%m/%d").month

Month of first transaction is 11

Extract and convert vcf contacts from emails in python

Thu, 27 May 2021 00:00:00 +0200

I wasn’t able to export my contacts from my iphone (iOS 11). Thus I mailed each contact to myself using an alias e.g. john.doe+contact@example.com. I’m using mu (mail indexer/searcher) version 1.6.4 for my emails. Everything else happens below:

Extract vcf file from email

contacts=~/tmp/iPhoneContacts
rm -r $contacts
mkdir $contacts
cd $contacts

foundEmails=$(mu find to:$myemail --fields "l")

for email in $foundEmails
do
  mu extract $email '.*.vcf'
done

Export vcf to tsv file

import fnmatch
import os
import re
import phonenumbers

ROOTPATH = '/home/ra/tmp/iPhoneContacts/'
PATTERN = '*.vcf'
TSV = open('/home/ra/tmp/iPhoneContacts/iPhoneContacts.tsv', 'w')


def contacts():
    l = set()
    for root, dirs, files in os.walk(ROOTPATH):
        for filename in fnmatch.filter(files, PATTERN):
            l.add(os.path.join(root, filename))
    return (l)


contactsList = contacts()
print("Total %s files:" % (PATTERN), len(contactsList))
for contactFile in contactsList:
    with open(contactFile, "r") as f:
        #Gets Peter Mueller Meier.contact from /home/ra/tmp/iPhoneContacts/Peter Mueller Meier.contact
        contactName = contactFile[(contactFile.rfind('/')) + 1::]
        #Gets Peter Mueller Meier from Peter Mueller Meier.contact
        contactName = contactName.split(".")[0]
        #Gets remaining words after space e.g. Mueller Meier from Peter Mueller Meier
        contactName = contactName.split(" ")
        #Gets Peter from Peter Mueller Meier
        contactFirstName = contactName[0]
        #Gets Mueller Meier from Peter Mueller Meier
        contactLastName = " ".join(
            contactName[1:]) if (len(contactName)) > 1 else ""

        numbrs = [
            phonenumbers.format_number(match.number,
                                       phonenumbers.PhoneNumberFormat.E164)
            for match in phonenumbers.PhoneNumberMatcher(f.read(), "CH")
        ]

        print(contactFirstName,
              contactLastName,
              "\t".join(numbrs),
              sep="\t",
              file=TSV)
TSV.close()

Python unit testing in emacs org mode

Thu, 08 Apr 2021 00:00:00 +0200

Using emacs org and babel¹, I want to quickly note ideas in python. Now usually I start scripting up to a certain level of complexity / exhaustion which inevitably leads me to setup a complete python project folder in order to get proper unit testing.

Following setup allows me to stay in org mode and unit test. All together.

(defun ndk/org-babel-evaluate-test-block-from-code-block ()
  (interactive)
  (save-excursion
    (org-babel-goto-named-src-block "python_code")
    (org-babel-tangle '(4))
    (org-babel-goto-named-src-block "pytest_python_code")
    (org-babel-execute-src-block)))

(define-key evil-normal-state-map (kbd ", SPC") #'ndk/org-babel-evaluate-test-block-from-code-block)

Evaluating the upper elisp code block binds itself to , SPC .

Now hitting , SPC in block python_code, exports it and subsequently runs the unit test from block pytest_python_code

All while I’m staying in the code, at the same cursor position. No need to jump between code blocks. Beautiful quietness.

#+NAME: python_code
#+BEGIN_SRC python :tangle /tmp/hello.py
 import pytest

 def test_func():
     assert 1 == 1
#+END_SRC

#+NAME: pytest_python_code
#+BEGIN_SRC sh  :results output
  pytest /tmp/hello.py
  rm -f /tmp/hello.py
#+END_SRC

============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /tmp
plugins: cov-2.12.1, pycharm-0.6.0, vcr-1.0.2, hypothesis-6.15.0
collected 1 item

../../../tmp/hello.py .                                                  [100%]

============================== 1 passed in 0.01s ===============================

https://orgmode.org/worg/org-contrib/babel/ ↩︎

Create diagrams in emacs org mode

Wed, 10 Mar 2021 00:00:00 +0100

Table of Contents

Being able to explain something with your words only prevails. For those other cases, here’s a way to do it with diagrams in emacs org mode.

Figure 1: Gantt Diagram

Figure 1 was generated from this text:

#+name: gantt_diagram
#+BEGIN_SRC mermaid :file /tmp/gantt.png
gantt
    dateFormat  YYYY-MM-DD
    title  Acme Project Plan

    section Period 3
        COMP112358            :done,  des1, 2099-01-29, 2099-03-07

    section Period 4
        Supervisory Meetings (term) :active, sup1, 2099-03-08, 2099-05-04
        POP Submission :sub1, 2099-05-10, 2099-05-11

    section Exams
        Exam Period :exams, 2099-05-06, 2099-06-06
        Semester End : 2099-06-08, 2099-06-09

    section Summer
        Supervisory Meetings : sup2, 2099-06-09, 2099-09-07
        Dissertation Submission :sub2, 2099-09-06, 2099-09-07
#+END_SRC

Mermaid-cli¹

yay mermaid-cli

Mermaid for org mode²

Clone layer into ~/.emacs.d/private:

git clone git@github.com:arnm/mermaid-layer.git ~/.emacs.d/private/mermaid

Add mermaid layer to dotspacemacs-configuration-layers

;; ~/.spacemacs.d/init.el
   dotspacemacs-configuration-layers
   '(..
     mermaid
     ..)

Get mmdc executable path:

which mmdc

/usr/bin/mmdc

Set mmdc executable path:

;; ~/.spacemacs.d/init.el
(setq ob-mermaid-cli-path "/usr/bin/mmdc")

Usage

Open an org-mode buffer, create an org-babel source block and run org-babel-execute-src-block.

#+begin_src mermaid :file /temp/test.png
sequenceDiagram
A-->B: Works!
#+end_src

Export the org-mode document or invoke the org-babel-execute-src-block function to generate a diagram.

Problem solving

Check your Messages buffer

Test if mermaid-cli works independently from emacs

printf "graph TD;\nA-->B;" > /tmp/input.mmd
mmdc -i /tmp/input.mmd -o /tmp/output.png
echo "file://"/tmp/output.png

Create a blender scene in python

Sun, 28 Feb 2021 00:00:00 +0100

Using blender, I need to test multiple settings for my painting references e.g. lighting, material or the complete location. It all started with that minimal working example to render the sphere above.

# /tmp/blender/blender_mini_example.py
import bpy
import time
import os
# Initialize variables used to print the rendered picture
dir_path = os.path.dirname(os.path.realpath(__file__))
filename = str(int(time.time())) + '.render' + '.png'
# Delete the default cube
objs = bpy.data.objects
objs.remove(objs["Cube"], do_unlink=True)
# Add a sphere
bpy.ops.mesh.primitive_uv_sphere_add(location=(-3, 0, 0))
# Render
bpy.context.scene.render.filepath = filename
bpy.ops.render.render(write_still=True)
# Print directory path and filename
print(dir_path+"/"+filename)

I’ve added line 7 in order to get rid of unneeded addons-on and their accompanying exceptions.

cd /tmp/blender
rm *.png
blender \
    --background \
    --python blender_mini_example.py \
    --verbose 0 \
    --factory-startup

Blender 2.93.6 (hash c842a90e2fa1 built 2021-11-17 16:06:39)
/run/user/1000/gvfs/ non-existent directory
Fra:1 Mem:88.84M (Peak 89.51M) | Time:00:00.14 | Syncing Light
Fra:1 Mem:88.84M (Peak 89.51M) | Time:00:00.14 | Syncing Camera
Fra:1 Mem:88.84M (Peak 89.51M) | Time:00:00.14 | Syncing Sphere
Fra:1 Mem:89.25M (Peak 89.54M) | Time:00:00.20 | Rendering 1 / 64 samples
Fra:1 Mem:89.20M (Peak 89.54M) | Time:00:01.24 | Rendering 26 / 64 samples
Fra:1 Mem:89.20M (Peak 89.54M) | Time:00:02.02 | Rendering 51 / 64 samples
Fra:1 Mem:89.20M (Peak 89.54M) | Time:00:02.43 | Rendering 64 / 64 samples
Saved: '1639292187.render.png'
 Time: 00:03.13 (Saving: 00:00.61)

/tmp/blender/1639292187.render.png

Blender quit

Test automation using GNU Make in ox-hugo

Sun, 10 Jan 2021 00:00:00 +0100

Amongst others, I run this script as part of my tests after a system update.

#Initialize variables
input=test/site/content-org/screenshot-subtree-export-example.org
output=test/site/content/writing-hugo-blog-in-org-subtree-export.md
cd /tmp
# Cleanup
rm -r ox-hugo
# Clone latest repository
git clone git@github.com:kaushalmodi/ox-hugo.git
cd ox-hugo
# Make change in org file
sed -i "s/\(Kaushal\)/The awesome \1/" $input
# Simulate the export from org to markdown
make md1 ORG_FILE=$input
# Check if the changes appears in the
# created markdown (named after the title property in org)
grep Kaushal $output


test/site/content-org/screenshot-subtree-export-example.org ::
author = ["The awesome Kaushal Modi"]

Annotate pictures non-invasively in latex/pdf using org mode

Thu, 10 Dec 2020 00:00:00 +0100

Goal of this method is the non-invasive annotation of pictures in LaTeX/pdf. Thus all annotations are written in LaTeX. The source picture 1 remains untouched.

Figure 1: Source picture

Figure 2: Help grid enabled

   #+EXPORT_LATEX_HEADER: \usepackage[background=black,text=white,arrow=red]{callouts}

  \begin{annotate}
  {\includegraphics[width=0.8\textwidth]{/home/ra/Pictures/IMG_1340.jpg}}{1.0}
  %\helpgrid %uncomment to enable helpgrid
  \callout {-3.0,5.0}{Hip}{0.0,5.0}
  \draw [thick ,\arcol] (2.0 ,-2.0)  rectangle  (4.0 ,-4.0);
  \draw [thick ,\arcol] (-0.3 ,-5.8)  rectangle  (0 ,-6.2);
  \draw [red,thick,dashed] (1,5) circle (1.0);
  \callout {-3,0.7}{Knee}{-1.1,0.7}
  \callout {-3,-3.0}{Ankle}{2.0,-3.0}
  \arrow{ 2.0,-4.0}{0.0,-5.8} %Display the joint on the complete animal
  \note {-2.5,-8.0}{Some note without arrow}
  \end{annotate}
  \newpage

Things to remember:

First define the x (horizontal) coordinates. Then define the y (vertical) coordinates.
First comes the note. Then comes the arrow.
A note’s anchor is centered (not left, nor right)
In order to enable the help grid as shown on 2 uncomment line 3 in 1

Code expansion in org mode prose text

Tue, 20 Oct 2020 00:00:00 +0200

Say I want to write about the count of my system’s processors at the time of my writing. Or my current location’s weather or stock shares or …use your imagination. Now I do it like that:

> At this very moment I’m writing this text on a machine with 4 processors.

Verbatim:

At this very moment I'm writing this text on a machine with src_bash{cat /proc/cpuinfo | grep processor | wc -l} processors.

Practically I expand a snippet, write any bash command into the curly braces and instantly have it’s evaluated result written into my prose.

Python example

Inline

Two plus two equals 4

Verbatim: > Two plus two equals src_python{return(2+2)} {{{results(=4=)}}}

Code block

Expand any code block’s result as well: 12

Verbatim: > Expand any [[python_example][code block]]'s result as well: call_python_example()

#+name: python_example
#+begin_src python
def func():
    return 1+1+2+3+5

return func()
#+end_src

https://orgmode.org/manual/Structure-of-Code-Blocks.html

Fix [Pyright] Import could not be resolved in emacs

Wed, 30 Sep 2020 16:57:00 +0200

Say you open any python file containing an import. Your IDE is spacemacs and pyright. You might encounter this error:

[Pyright] Import "anymodule" could not be resolved

In order to get rid of this error, hit:

, F a

This launches lsp-workspaces-folders-add which requires you to enter the project’s path e.g. ~/tmp/mypython/ as workspace.

I had only ~/tmp/ in the lsp-workspaces and thought it is enough. Deleted this folder with: , F d .

Why do I need unit tests?

Tue, 07 Jul 2020 12:36:00 +0200

So I want to compare if two dates are equal. I therefore input two dates and in return expect the comparison’s outcome as a bool.

Usually I would:

Write the function

Print it’s result until I’m happy with it

#somecode.py
from datetime import datetime

def compare_two_dates(date,otherdate):
    return date==otherdate

date = datetime.now().date()
otherdate = datetime(1900,12,12)

true_expected=compare_two_dates(date,date)
false_expected=compare_two_dates(date,otherdate)
print("true_expected:",true_expected)
print("false_expected:",false_expected)

Result:

true_expected: True
false_expected: False

Why am I not happy with that approach?

I’m required to see the output. My focus constantly switches between looking what is printed and the code itself. I’m required to repeat it over and over since I can’t remember later if I already printed the functions outcome.

What do I want?

Write down the result I want. Write the necessary code. Done.

This translates into two steps for most of the coding:

Write what I want
Write the code to get what I want

Result

No more print statements cluttering my commits. No more insecurities about memorizing if I ever tested this before.

# my_unit_test.py
import pytest
from datetime import datetime


def compare_two_dates(date, otherdate):
    return date == otherdate


def test_compare_two_dates():
    date = datetime.now().date()
    otherdate = datetime(1900, 12, 12)

    assert compare_two_dates(date, date) == True
    assert compare_two_dates(date, otherdate) == False

pytest my_unit_test.py
echo

============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/ra/tmp/83820272
plugins: cov-2.12.1, hypothesis-6.14.5, pycharm-0.6.0, vcr-1.0.2
collected 1 item

my_unit_test.py .                                                        [100%]

============================== 1 passed in 0.01s ===============================

Conclusion:

Unit tests free me from printing over and over the same stuff. They free me from being needed to overview in person what I’ve already written down as an expected result. I can agree with third parties on tests similar to a contract.

How to use the ox-hugo Makefile

Mon, 04 May 2020 19:31:00 +0200

Amongst others, I run this script as part of my tests after a system update.

cd /tmp
#Cleanup
rm -r ox-hugo
#Clone latest repository
git clone git@github.com:kaushalmodi/ox-hugo.git
cd ox-hugo
#Make change in org file
sed -i "s/\(Kaushal\)/The awesome \1/"  test/site/content-org/screenshot-subtree-export-example.org
#Simulate the export from org to markdown
make md1 ORG_FILE=test/site/content-org/screenshot-subtree-export-example.org
#Check if the changes appear in the created markdown (named after the title property in org)
grep Kaushal test/site/content/writing-hugo-blog-in-org-subtree-export.md


test/site/content-org/screenshot-subtree-export-example.org ::
author = ["The awesome Kaushal Modi"]

Scraping Migros M-Budget with python

Tue, 07 Apr 2020 18:50:00 +0200

Table of Contents

Motivation

I recently caught myself repeatedly checking my internet consumption balance.

A perfect case for a script and an accompanying tutorial.

For the sake of this tutorial’s scope and the script’s brevity I’ll skip various code paradigms like exception handling, testing, continuous integration, virtual environments etc.

In this particular case the concerning ISP (Internet Service Provider) is Migros, their mobile brand is M-Budget and the actual provider is Swisscom.

How to understand this post

Each section contains a Manual and Automated section. Manual shall depict how the obstacle is taken by hand such as typing and clicking the mouse. Automated shows how we’re doing it with a code snippet.

The Complete script contains all snippets merged and ready for copy/paste as described in Usage

Script ingredients

Modules

import requests
from lxml import html
from lxml import etree

Urls

Our working URLs are built upon one URL which serves as base URL in order to build up all endpoints.

BASE_URL = 'https://selfcare.m-budget.migros.ch/eCare'
LOGIN_URL = BASE_URL+'/de/users/sign_in'
LOGOUT_URL = BASE_URL+'/de/users/sign_out'
CONSUMPTION_URL = BASE_URL+'/prepaid/de/my_consumption/index'

Process overview

Manual

Launch your browser, open login and enable the browser’s developer console by pressing F12 on your keyboard.

Automated

1
2

session_requests = requests.session()
result = session_requests.get(LOGIN_URL)

Manual

Enter your username and password and hit return.

After submitting your credentials, deliberate what you just submitted.

Superficially, our login seems to contain only our credentials, thus:

Username (here it’s a phone number beginning with 077)
Password

However, the developer console (F12 on your keyboard) and it’s Network tab reveal more.

The data is sent as POST request, therefore we look for POST methods and therein for the Form data (on the bottom right in figure 1) Alongside the mentioned credentials, a couple of hidden data was submitted as well.

In particular the authenticity_token is interesting. It’s part of a security remedy against a malicious hack. Also known as ¹ Cross-site request-forgery or in abbreviated form csrf.

Figure 1: Post form data

But where does that cryptic csrf token value come from?

Finding the csrf token

Remember the login page? https://selfcare.m-budget.migros.ch/eCare/de/users/sign%5Fin

Logout, visit it again and look at the source code (Mouse Right-Click: View page source).
```
<meta name="csrf-token" content="mX12K..." />
```

Automated

Fetch csrf token

Now we know where to seek what element:
1. Where: In the html source text at line 1
2. What: A meta tag named csrf-token contains the token as it’s content. At line 2
1 2

tree = html.fromstring(result.text) authenticity_token = tree.xpath("//meta[@name='csrf-token']/@content")[0]

Credentials

PASSWORD = 'changeMe'
USERNAME = 'changeMe'

Prepare our form

We pass the csrf token along with the remaining data as dictionary.

Compare the similarity to the form in figure 1:

data = {
    'utf8': '\u2713',
    'authenticity_token': authenticity_token,
    'user[id]': USERNAME,
    'user[password]': PASSWORD,
    'user[reseller]': '33',
    'button': ''
}

Sign-in

We’re all set and can finally sign-in passing our form

1
2
3

result = session_requests.post(LOGIN_URL,
                               data=data,
                               headers=dict(referer=LOGIN_URL))

Get the consumption values

Manual

There are multiple ways of finding the credit values

By searching for a value (here 500) in the browsers developer console as shown below in figure 2
By mouse hovering over the element (in the browser, not the dev console) and right-click Inspect element (q)

Figure 2: Searching for values and displaying css properties

We’re interested in all credits (10,0,500) in order to sum them up as our balance 510.

Automated

Here’s how we’re scripting the balance part starting at line 3:

First we set our focus on the parenting div in the class named circle__inside in order to extract the text from it’s first [1] span element. Then we loop through each child (circles) and append each circle’s credit value to our balance list. Finally we print the balance.

result = session_requests.get(CONSUMPTION_URL)
tree = etree.HTML(result.text)
circles = tree.xpath('.//div[@class="circle__inside"]//span[1]/text()')
balance = []
for credit in circles:
    balance.append(int(credit))
print("Your current balance is:", sum(balance), "MB")

Logout

We’re done and therefore properly logout.

Manual

Click on the profile icon in the upper right corner
Click sign-out (Abmelden)

Automated

`1`	`response = session_requests.get(LOGOUT_URL)`

Complete script

import requests
from lxml import html
from lxml import etree
BASE_URL = 'https://selfcare.m-budget.migros.ch/eCare'
LOGIN_URL = BASE_URL+'/de/users/sign_in'
LOGOUT_URL = BASE_URL+'/de/users/sign_out'
CONSUMPTION_URL = BASE_URL+'/prepaid/de/my_consumption/index'
PASSWORD = 'changeMe'
USERNAME = 'changeMe'


def main():
    session_requests = requests.session()
    result = session_requests.get(LOGIN_URL)
    tree = html.fromstring(result.text)
    authenticity_token = tree.xpath("//meta[@name='csrf-token']/@content")[0]
    data = {
        'utf8': '\u2713',
        'authenticity_token': authenticity_token,
        'user[id]': USERNAME,
        'user[password]': PASSWORD,
        'user[reseller]': '33',
        'button': ''
    }
    result = session_requests.post(LOGIN_URL,
                                   data=data,
                                   headers=dict(referer=LOGIN_URL))
    result = session_requests.get(CONSUMPTION_URL)
    tree = etree.HTML(result.text)
    circles = tree.xpath('.//div[@class="circle__inside"]//span[1]/text()')
    balance = []
    for credit in circles:
        balance.append(int(credit))
    print("Your current balance is:", sum(balance), "MB")
    response = session_requests.get(LOGOUT_URL)

if __name__ == '__main__':
    main()

Your current balance is: 510 MB

Usage

Store the complete script above as e.g. ~/tmp/mbudget.py
Change the Credentials in ~/tmp/mbudget.py
Run it:

python3 ~/tmp/mbudget.py

Your current balance is: 510 MB

Conclusion

Scripting our manual steps now allows us to store and process the returned data further. We could schedule it using a systemd timer or cron. Or send a notification by email or sms if a certain limit is reached.

And of course: No more clicking, calculating and repeating.

https://en.wikipedia.org/wiki/Cross-site%5Frequest%5Fforgery ↩︎

Testing APIs offline and fast with vcrpy

Sat, 04 Apr 2020 11:03:00 +0200

Table of Contents

Motivation

While testing code against online sources like websites or APIs I often find myself waiting for their responses. Furthermore it feels strange to knock on anyone’s door every couple of seconds just to test what I’m processing locally anyway.

A solution for this issue comes in the form of vcrpy.

Vcrpy frees you from the burden of waiting or even to be online at all. Responses are stored locally in the form of yaml files called cassettes.

In the following post I’ll test against Migros’ mobile plan named M-Budget. Migros is one of Switzerland’s largest wholesale retailers.

The test consists of asserting a word’s appearance on Migros’ website.

Requirements

Python

python --version

Python 3.8.1

Module

pip install --user pytest
pip install --user pytest-vcr

Create the test

# content of ~/tmp/vhs/test_mbudget.py
import os
import urllib.request
import pytest


@pytest.fixture(scope='module')
def vcr_cassette_dir(request):
    # Put all cassettes in ~/tmp/vhs/{module}/{test}.yaml
    return os.path.join('/home/ra/tmp/vhs', request.module.__name__)

@pytest.mark.vcr()
def test_mbudget():
    response = urllib.request.urlopen('https://selfcare.m-budget.migros.ch/eCare/de/users/sign_in').read()
    assert b'csrf-token' in response

Line 10 says we want our cassetes saved in home/ra/tmp/vhs/test_mbudget

Line 15 asserts the appearance of the word ‘csrf-token’ in the requested website’s source

Run the test

pytest ~/tmp/vhs/test_mbudget.py

============================= test session starts ==============================
platform linux -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/ra
plugins: vcr-1.0.2, pycharm-0.5.0
collected 1 item

../tmp/vhs/test_mbudget.py .                                             [100%]

============================== 1 passed in 11.40s ==============================

Note the time the test needed to pass: 11.40s

Check folders and files

Check the newly created cassettes folder

tree ~/tmp/vhs/

/home/ra/tmp/vhs/
├── __pycache__
│   └── test_mbudget.cpython-38-pytest-5.3.5.pyc
├── test_mbudget
│   └── test_mbudget.yaml
└── test_mbudget.py

2 directories, 3 files

Check the created file

cat ~/tmp/vhs/test_mbudget/test_mbudget.yaml | grep 'csrf-token'

/>\n<meta name=\"csrf-token\" content=\"wtwue7j8wcWPZEw2iI3aJtohGxfN9iSMVlGdbf4kUyT31iupUO2EyJ2i/aZA8z+LHclUsYBbH+HTrScrIaxGyg==\"

Go offline

Let’s disable our network connection

sudo ip link set wlp3s0 down

Run the test again - offline

pytest ~/tmp/vhs/test_mbudget.py

============================= test session starts ==============================
platform linux -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/ra
plugins: vcr-1.0.2, pycharm-0.5.0
collected 1 item

../tmp/vhs/test_mbudget.py .                                             [100%]

============================== 1 passed in 0.05s ===============================

Note how much faster (0.05s) the test ran and we’re offline.

Go back online

Mind to re-enable your network connection

sudo ip link set wlp3s0 up

Conclusion

We were able to reduce waiting from 11 seconds to 0.5 seconds (you’re already doing the ratio math in your head, right?). Furthermore we’re able to continue our development offline and aren’t bothering anyone with countless requests.

Or simply quoting vcrpy’s unique selling propositions:

The ability to work offline

Completely deterministic tests

Increased test execution speed

Take website screenshots with headless Firefox

Mon, 30 Mar 2020 15:50:00 +0200

This script takes a screenshot of a website. It does so headless with Firefox. A backup is taken before doing anything. It deletes traces left during the process.

Backup

     exec 2>&1
     orig=/home/ra/.mozilla/firefox
     backup=/home/ra/tmp/firefox_backup
     rm -rf $backup
     mkdir -p $backup
     cp -r $orig $backup
     :

Create the test profile

   which firefox >/dev/null 2>&1 && firefox --version
   which firefox >/dev/null 2>&1 && firefox -CreateProfile headlessTest

Take the screenshot

     exec 2>&1
     start_time="$(date -u +%s)"
     dt=$(date '+%Y-%m-%d-%H%M%S')
     filename=/tmp/$dt-headlessTest.jpg
     url=https://www.gnu.org/software/emacs/
     which firefox >/dev/null 2>&1 && firefox \
                                          -P headlessTest \
                                          --screenshot $filename $url \
                                          --window-size=1024,768
     end_time="$(date -u +%s)"
     elapsed="$(($end_time-$start_time))"
     echo "Total of $elapsed seconds elapsed for screenshot: $filename"
     echo "Size: $(du --block-size K $filename)"
     :

Cleanup and restore

     exec 2>&1
     orig=/home/ra/.mozilla/firefox
     backup=/home/ra/tmp/firefox
     rm -rf $orig/*
     cp -r $backup/* $orig
     rm -rf $backup #Uncomment to delete the backup
     echo "Done"
     :

Fullscript

          exec 2>&1
          orig=/home/ra/.mozilla/firefox
          backup=/home/ra/tmp/firefox_backup
          rm -rf $backup
          mkdir -p $backup
          cp -r $orig $backup
          :
        which firefox >/dev/null 2>&1 && firefox --version
        which firefox >/dev/null 2>&1 && firefox -CreateProfile headlessTest
          exec 2>&1
          start_time="$(date -u +%s)"
          dt=$(date '+%Y-%m-%d-%H%M%S')
          filename=/tmp/$dt-headlessTest.jpg
          url=https://www.gnu.org/software/emacs/
          which firefox >/dev/null 2>&1 && firefox \
                                               -P headlessTest \
                                               --screenshot $filename $url \
                                               --window-size=1024,768
          end_time="$(date -u +%s)"
          elapsed="$(($end_time-$start_time))"
          echo "Total of $elapsed seconds elapsed for screenshot: $filename"
          echo "Size: $(du --block-size K $filename)"
          :
          exec 2>&1
          orig=/home/ra/.mozilla/firefox
          backup=/home/ra/tmp/firefox
          rm -rf $orig/*
          cp -r $backup/* $orig
          rm -rf $backup #Uncomment to delete the backup
          echo "Done"
          :

4.0K /home/ra/.mozilla/firefox
4.0K /home/ra/tmp/firefox
*** You are running in headless mode.
Total of 7 seconds elapsed for screenshot: /tmp/2020-03-30-154448-headlessTest.jpg
Size: 104K	/tmp/2020-03-30-154448-headlessTest.jpg
4.0K /home/ra/.mozilla/firefox
4.0K /home/ra/tmp/firefox

Mount a USB drive persistently for all users

Sat, 28 Mar 2020 12:50:00 +0100

Table of Contents

I need to persistently permit all users read/write access to a USB drive

Create a mount point

mkdir /mnt/usb500

List devices

lsblk -o NAME,FSTYPE,LABEL

NAME        FSTYPE LABEL
sda
└─sda1      ext4   usb500gb
mmcblk0
├─mmcblk0p1 vfat   boot
└─mmcblk0p2 ext4   rootfs

Backup & print current fstab

cp /etc/fstab ~/fstab.bak
cat /etc/fstab

proc            /proc           proc    defaults          0       0
PARTUUID=45eaad39-01  /boot           vfat    defaults          0       2
PARTUUID=45eaad39-02  /               ext4    defaults,noatime  0       1

Add the device to fstab

Hint: nofail do not report errors for this device if it does not exist. You may boot without the drive.

echo "\n/dev/sda1    /mnt/usb500  ext4   user,rw,nofail" >> /etc/fstab

Change device’s permission

mount -a
sudo chmod 777 /mnt/usb500/

Test

Reboot

sudo reboot

Write into a temporary file

cd /mnt/usb500/
echo 'foo' > bar.txt
cat bar.txt
rm bar.txt

foo

Cleanup

After you thoroughly tested everything remove leftovers

rm ~/fstab.bak

Troubleshooting

You attached the drive after booting

sudo mount -a

Backup an iPhone with Linux

Sat, 21 Mar 2020 20:35:00 +0100

Table of Contents

This is a follow-up on Connect an iPhone to a Raspi.

Install the necessary packages

Find the event `iPhone connected`

Connect your iPhone:
Run this script:

lsusb | grep iPhone

Bus 001 Device 007: ID 05ac:12a8 Apple, Inc. iPhone5/5C/5S/6

Catch the event `iPhone connected`

# Content of /etc/udev/rules.d/99-iphone.rules
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ENV{PRODUCT}=="5ac/12[9a][0-9a-f]/*", ACTION=="add", ENV{SYSTEMD_WANTS}="iphone-connect.service"

Create the trigger `iPhone connected`

#Content of /etc/systemd/system/iphone-connect.service
[Service]
Type=oneshot
ExecStart=/bin/sh /home/pi/scripts/backup_iphone.sh

Create the action `Backup iPhone`

# Content of ~/scripts/backup_iphone.sh
src=/mnt/iphone/ #Must exist in prior
dst=/media/pi/usb500gb #Must exist in prior
log=/home/pi/backup_iphone.log
sleep 5
ifuse $src &&
    rsync --verbose --archive --compress --update --log-file=$log $src $dst

Restart and reload all actors (not the iPhone though)

sudo udevadm control --reload-rules && udevadm trigger
service udev restart
systemctl daemon-reload #THIS one is crucial

Monitor the backup process

journalctl -f --unit=iphone-connect.service --output=cat

Change Firefox's zoom by script

Sat, 14 Mar 2020 16:55:00 +0100

Each time I change the monitor I have to zoom in or out firefox. Here’s how to set it via script:

Figure 1: Large

Figure 2: Small

Locate the appropriate file

ls /home/*/.mozilla/firefox/*/prefs.js

Locate the appropriate parameter

grep layout.css.devPixelsPerPx /home/*/.mozilla/firefox/*/prefs.js

Large screen: Set the zoom

#!/usr/bin/env bash
factor=2.5
if grep 'layout.css.devPixelsPerPx' /home/*/.mozilla/firefox/*/prefs.js
then
    sed -i -e "s/^user_pref(\"layout.css.devPixelsPerPx.*$/user_pref(\"layout.css.devPixelsPerPx\", \"$factor\");/" /home/*/.mozilla/firefox/*/prefs.js
else
    echo "user_pref(\"layout.css.devPixelsPerPx\", \"$factor\");" >> /home/*/.mozilla/firefox/*/prefs.js
fi

Small screen: Set the zoom

#!/usr/bin/env bash
factor=1.0
if grep 'layout.css.devPixelsPerPx' /home/*/.mozilla/firefox/*/prefs.js
then
    sed -i -e "s/^user_pref(\"layout.css.devPixelsPerPx.*$/user_pref(\"layout.css.devPixelsPerPx\", \"$factor\");/" /home/*/.mozilla/firefox/*/prefs.js
else
    echo "user_pref(\"layout.css.devPixelsPerPx\", \"$factor\");" >> /home/*/.mozilla/firefox/*/prefs.js
fi

Make the scripts executable

sudo chmod --verbose +x ~/src/scripts/FirefoxZoomIn.sh
sudo chmod --verbose +x ~/src/scripts/FirefoxZoomOut.sh

~/src/scripts/FirefoxZoomIn.sh

~/src/scripts/FirefoxZoomOut.sh

Docking/undocking triggers those scripts, which I’m handling with Autorandr.

Connect an iPhone to a Raspi

Tue, 21 Jan 2020 13:10:00 +0100

Table of Contents

Environment:

iPhone 5SE iOS 11.0.1 (15A402)
Raspberry Pi 3 (4.19.66-v7+)

Most of the steps below are a gist of this blog post¹

Unlock your iPhone

Unlock/Enter the pass-code (in case you have one) on your iPhone in order to avoid errors.

Existence of an entry ‘Host raspi’ in ~/.ssh/config is assumed:

ssh raspi

Update - Retrieve new lists of packages

sudo apt-get update

Upgrade - Perform an upgrade

sudo apt-get upgrade

Install the necessary packages

sudo apt-get install libimobiledevice-utils

sudo apt-get install usbmuxd

sudo apt-get install ifuse

Connect the iPhone with the raspi via USB

Accept the trust dialog on the iPhone

Pair the phone

idevicepair pair

SUCCESS: Paired with device 3b9...

ERROR: Please accept the trust dialog on the screen of device 3b9...
, then attempt to pair again.

Accept another trust dialog the on the iPhone

Create a mount folder

sudo mkdir /mnt/iphone/ --verbose

mkdir: mkdir: created directory '/mnt/iphone/'

Mount the iPhone

Consider storing this command as a shortcut to your desktop for convenience:

ifuse /mnt/iphone/

Approve success

tree -d -L 1 /mnt/iphone/

/mnt/iphone/
├── Books
├── CloudAssets
├── DCIM
├── Downloads
├── LoFiCloudAssets
├── MediaAnalysis
├── PhotoData
├── PhotoStreamsData
├── Photos
├── PublicStaging
├── Purchases
├── Radio
├── Recordings
└── iTunes_Control

14 directories

Summary of all commands to copy/paste

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install libimobiledevice-utils
sudo apt-get install usbmuxd
sudo apt-get install ifuse
idevicepair pair
sudo mkdir /mnt/iphone/ --verbose
ifuse /mnt/iphone/
tree -d -L 1 /mnt/iphone/

https://github.com/justinpearson/Raspberry-Pi-for-iPhone-Backup/blob/master/README.md#4-configure-udev-to-run-the-backup-script-upon-the-phone-plug-in-event ↩︎

Setup a Fujitsu ix500 scanner on Linux

Fri, 17 Jan 2020 18:27:00 +0100

Table of Contents

Install package

sudo apt install sane-utils -y

Approve the connection

Enable the scanner by opening the scanner’s lid

sudo sane-find-scanner -q

found USB scanner (vendor=0x04c5 [Fujitsu], product=0x132b [ScanSnap iX500]) at libusb:001:007
found USB scanner (vendor=0x0424, product=0x7800) at libusb:001:005

Get a list of devices:

scanimage --list-devices

device `fujitsu:ScanSnap iX500:10443' is a FUJITSU ScanSnap iX500 scanner

Get a list of all device specific options:

scanimage --device 'fujitsu:ScanSnap iX500:10443' --all-options

Scan a test file

Scan with default settings to a temporary testfile

scanimage --format=tiff > /tmp/image.tiff
ls --size --block-size=M /tmp/image.tiff

5M /tmp/image.tiff

Scan productively

Mind making this script executable using chmod +x.

#!/bin/bash
#Content of ~/scripts/scan.sh
if [ $1 -eq 0 ]
then
    OUT_DIR=/tmp
else
    OUT_DIR=$1
fi
fileNAME=scan_`date +%Y-%m-%d-%H%M%S`_

echo 'scanning...'
scanimage --verbose\
          --page-width 221.121\
          --page-height 876.695\
          -l 0\
          -t 0\
          -x 221.121\
          -y 876.695 \
          --ald=yes \
          --overscan On \
          --prepick=On \
          --mode Gray \
          --resolution 300 \
          --source 'ADF Duplex' \
          --swcrop=yes \
          --buffermode On \
          --swdespeck 2 \
          --swdeskew=yes \
          --swskip 20% \
--device 'fujitsu:ScanSnap iX500:10443' \
         --batch="$OUT_DIR/$fileNAME%03d.tiff"
printf "Scanned document(s) to:\n\
           $(ls --size --block-size=M $OUT_DIR/$fileNAME*)"

Code Snippet 1: Scan documents in bash with scanimage

scanning...
Scanned document(s) to:
           9M /scan_2020-03-22-205607_001.tiff
9M /scan_2020-03-22-205607_002.tiff

Skipping empty pages

Pay attention at the highlighted swskip in the script above. Lower values e.g. 5% almost never skip empty pages. You get two files (double sided document). Higher values e.g. 20% skip empty pages. You get one file. Case-specific testing is recommended.

scanimage --help | grep swskip --after-context=1

    --swskip 0..100% (in steps of 0.100006) [0]
        Request driver to discard pages with low percentage of dark pixels

Alias the command

Add this command to your .profile

alias scan="ssh raspi '/home/pi/scripts/scan.sh /media/pi/usb500gb/scan'"

Take screenshots in emacs org mode with flameshot

Mon, 29 Apr 2019 15:40:00 +0200

Figure 1: Screencast showing flameshot in action

Setup

Enable the org layer in your .spacemacs dotspacemacs-configuration-layers

    dotspacemacs-configuration-layers
    '(org)

Install flameshot

 sudo pacman -S flameshot

Add the line below to your .spacemacs custom-set-variables

 (custom-set-variables
  '(org-download-screenshot-method "flameshot gui --raw > %s"))

Usage

While being in your org buffer:

Take your screenshot hitting , i D s or calling M-x org-download-screenshot
Edit your screenshot on the fly or.. don’t
Hit RET to insert your screenshot into your org file

Figure 2: A screenshot taken with flameshot in org mode

Write a letter in org mode

Mon, 22 Apr 2019 20:22:00 +0200

Table of Contents

Goals

Figure 1: Printed sample.pdf

In order to get the same printout as in figure 1 above, both includepdf lines starting at line 37 here must be uncommented.

Use emacs ¹org mode
Compose a new letter with a few keystrokes
Manage letters like e-mails

Setup

Change letter meta variables through a lco file

The ²koma manual at p.484 displays a nice overview of position possibilities:

Figure 2: Schematic of the pseudo-lengths for a letter

Custom LCO files must be placed in a directory where LaTeX will find them. On GNU/Linux, this defaults to ~/texmf/tex/latex.

Create the folder ~/texmf/tex

mkdir -p ~/texmf/tex

Create a new file ~/texmf/tex/mySN.lco containing these lines:

\ProvidesFile{mySN.lco}[2019/04/22 my_letter-class-option]
\LoadLetterOption{SN}
\let\raggedsignature=\raggedright
\setplength{toaddrvpos}{60mm}
\setplength{backaddrheight}{-3mm}
\setplength{toaddrheight}{25.5mm}
\usepackage[german]{babel}
\endinput

You can test whether foo.lco is recognized by TeX Live by running the command

kpsewhich mySN.lco

After adding a file to the TeX Live path you may have to run

mktexlsr

Create the yasnippets template

Create home/ra.emacs.d/private/snippets/text-mode/exampleLetter containing lines below:

# -*- mode: snippet -*-
# name:myExampleLetter
# key:,ltr
# --
  #+LCO: mySN
  #+OPTIONS: backaddress:t email:nil
  #+OPTIONS: after-closing-order:(after_closing ps encl cc)
  #+OPTIONS: after-letter-order:(after_letter)
  #+OPTIONS: foldmarks:nil phone:nil place:t special-headings:nil subject:t
  #+OPTIONS: title-subject:nil url:nil location:t title:nil

  #+LATEX_CLASS: default-koma-letter
  #+LATEX_HEADER: \setkomavar{signature}{\includegraphics[height=1cm]{/home/ra/Downloads/signature.pdf}\\\\\usekomavar{fromname}}
  #+LaTeX_HEADER: \usepackage{pdfpages}

  #+AUTHOR: Sender Name
  #+FROM_ADDRESS: Acmestreet 1, 5555 Footown
  #+PHONE_NUMBER: 0123456789
  #+URL: example.org
  #+EMAIL: foo@bar.com
  #+TO_ADDRESS: ${1:ReceiverInc}\\\ ${2:JohnDoe}\\\ ${3:SomeStreet}\\\ ${4:ZipTown}\\\Mittelerde
  #+PLACE: MyTown
  #+SUBJECT: ${5:MySubject}
  #+OPENING: ${6:Dear John}
  #+CLOSING: With best regards
  ${0:MyText}
*** PS                                                                   :ps:noexport:
    PS: I must remove the :noexport: tag to display this line in the exported pdf
*** CC                                                                   :cc:
    Jane & Alice
*** ENCL                                                               :encl:
    - A map
    - Another map
*** Included files                                             :after_letter:
    #+BEGIN_EXPORT latex
    % Uncomment line below to include other pdf files
    %\includepdf[pages=-]{/home/ra/Downloads/Map.pdf} %must use absolute path
    %\includepdf[pages=-]{/home/ra/Downloads/OtherMap.pdf} %must use absolute path
    #+END_EXPORT

Workflow (Spacemacs)

Create a new file hitting SPC f f
Enter the filename e.g. sample.org RET
Expand the template we created before hitting ,ltr TAB
Toggle through the fields hitting TAB
Export the pdf hitting , e e k o

Blog in org mode with ox-hugo and github pages

Tue, 16 Apr 2019 19:52:00 +0200

Table of Contents

Setup ox-hugo

https://ox-hugo.scripter.co/doc/usage/

Add these lines to .spacemacs

dotspacemacs-additional-packages '(
                                   ox-hugo)

(defun dotspacemacs/user-config ()
  (use-package ox-hugo
    :ensure t          ;Auto-install the package from Melpa (optional)
    :after ox))

Create a new github repo via web front-end

Login to https://github.com/new
Enter RidaAyed.github.io
Leave other fields

Clone the new repository

m-x magit-clone
git@github.com:RidaAyed/RidaAyed.github.io.git
RET

Setup Hugo in our new repository

Create the new hugo site

https://www.kengrimes.com/ox-hugo-tutorial/

hugo new site --force /home/ra/RidaAyed.github.io

Download a theme and add it as git sub-module

https://gohugo.io/getting-started/quick-start/#step-3-add-a-theme

cd /home/ra/RidaAyed.github.io
git submodule add https://github.com/budparr/gohugo-theme-ananke.git themes/ananke

Check our current config.toml

cat /home/ra/src/RidaAyed.github.io/config.toml

Add the Ananke theme to our config.toml

cd /home/ra/RidaAyed.github.io
echo 'theme = "ananke"' >> config.toml

Recheck our current config.toml

cat /home/ra/src/RidaAyed.github.io/config.toml

Create the org content folder

cd /home/ra/RidaAyed.github.io
mkdir content-org

Create a minimal ¹ org file / web page

#+hugo_base_dir: must be followed by two dots .. since our file resides in the subfolder org-content

#+hugo_base_dir: ..

* My post
:PROPERTIES:
:EXPORT_FILE_NAME: my-post
:END:
This gets created in ~<HUGO_BASE_DIR>/content/posts/~.

Start the hugo server

cd /home/ra/RidaAyed.github.io
hugo server -D --navigateToChanged

Figure 1: Screenshot - $ hugo

Browse to our local site

Visit http://localhost:1313/

Figure 2: Screenshot - localhost

Export our org file to hugo

M-x org-export-dispatch

Figure 3: Screenshot - org-export-dispatch

Note how hugo immediately rebuilds the site

Figure 4: Screenshot - Hugo instant rebuilding changes

Remark: The second change was due to adding the second dot as mentioned here

Check the browser

Figure 5: Screenshot - Revisiting localhost

Recheck what has been created

tree /home/ra/RidaAyed.github.io -D -L 2

/home/ra/RidaAyed.github.io [error opening dir]

0 directories, 0 files

Publish to Github pages

In case you followed https://gohugo.io/hosting-and-deployment/hosting-on-github/#step-by-step-instructions. I missed the /public folder from step 4.

The public folder will not be created, if $ hugo is executed with the server option. Simply run $ hugo as shown at line 2:

1
2

cd /home/ra/RidaAyed.github.io
hugo

Following the next step from the step by step instruction

cd /home/ra/RidaAyed.github.io
git submodule add -b master git@github.com:RidaAyed/RidaAyed.github.io.git public

By not following the step-by-step instructions whom propose creating two repositories results in

Cloning into '/home/ra/src/RidaAyed.github.io/public'...
warning: You appear to have cloned an empty repository.
fatal: 'origin/master' is not a commit and a branch 'master' cannot be created from it

Let’s create another git branch

cd /home/ra/RidaAyed.github.io
git checkout -b source

Trying again

cd /home/ra/RidaAyed.github.io
git submodule add -b master git@github.com:RidaAyed/RidaAyed.github.io.git public

Better

Adding existing repo at 'RidaAyed.github.io/public' to the index

Create a script deploy.sh to deploy and publish to github

echo -e "\033[0;32mDeploying updates to GitHub...\033[0m"

# Build the project.
hugo # if using a theme, replace with `hugo -t <YOURTHEME>`

# Go To Public folder
cd public
# Add changes to git.
git add .

# Commit changes.
msg="rebuilding site `date`"
if [ $# -eq 1 ]
then msg="$1"
fi
git commit -S --message "$msg"

# Push source and build repos.
git push origin master

# Come Back up to the Project Root
cd ..

sudo chmod +x /home/ra/src/RidaAyed.github.io/deploy.sh

cd /home/ra/RidaAyed.github.io
./deploy.sh

Visit https://ridaayed.github.io

Figure 6: Screenshot - https://ridaayed.github.io

What about speed?

Figure 7: Screenshot - of PageSpeedInsights

Using prodigy for hugo auto build on saving the org file

Since you’re lazy and event don’t want to bother to Start the hugo server , you’ll delegate this to prodigy:

Add the prodigy ²layer
Add this section at the end of .spacemacs:

(prodigy-define-service
 :name "Hugo Personal Blog"
 :command "/usr/bin/hugo"
 :args '("server" "-D" "--navigateToChanged" "-t" "hugo-coder")
 :cwd "~/RidaAyed.github.io"
 :tags '(personal)
 :stop-signal 'sigkill
 :kill-process-buffer-on-stop t)

Start prodigy to take over hugo SPC a t p and hit r on Hugo Personal Blog