I’m flabbergasted how good this tracker-neuralnet plugin for #OpenTrack works. It does the #headTracking with just a webcam without any clips, reflectors or LED stripes. I kinda expected this to not work really well in a dark room, that I prefer for gaming, but I was wrong. Even with a tiny light in one corner of the room only it kept tracking flawless.

…can even scratch my nose and it keeps tracking.

To get this neuralnet tracker input in the first place I had to download the ONNX runtime package onnxruntime-linux-x64-1.18.1.tgz from github.com/microsoft/onnxrunti… (My Fedora offered 1.15.1 from it’s repo but this was at the time of writing not sufficient and having it installed resulted in failure due to an API mismatch). I didn’t even install it somewhere, just unpacked it in my Downloads folder.

Back in my OpenTracks folder I run cmake the way I’ve done it before several times but this time I added the unpacked onnxruntime folder to the config.

Configure did it’s magic (note how it picked up module tracker-neuralnet once the ONNXRuntime_DIR was set) and here we are one make later. This is rather impressive 🤓

YMMV

This content is licensed under a Creative Commons Attribution 4.0 International license.

beko.famkos.net/2024/08/01/com…

#gamingonlinux #Headtracking #linuxgaming #opentrack

Given: There’s a CI system that automatically builds docker images from your VCS (e.g. git), we use self-hosted gitlab.
Goal: Both initial and subsequent automated deployments to different environments (staging and production).

Rejected Approaches

Most existing blog articles and howtos for this use case, specifically in the context of gitlab, tend to be relatively simple, relatively easy, and very very wrong. The biggest issue is with root access to the production server. I believe that developers (and the CI/CD system) should not have full root access to the production system(s), to retain semblance of separation in case of breaches. Yes, sure, a malicious developer could still check-in bad code which might eventually get deployed to production, but there is (should be) a review process before that, and traces in the VCS.

And yet, most recommendations on how to do deployment with gitlab circle around one of two approaches:

1. Install a gitlab “runner” on your production server. That is, an agent which gets commands from the gitlab server and executes them. This runner needs full root access (or, equivalently, docker daemon access), thus giving the gitlab server (and anyone who has/gains control over it) full root access to the production system(s).
   This approach also needs meticulous management of the different runners, since they are now being used not just for build purposes but also have a second, distinct, duty for deployment.
2. Use your normal gitlab runners that are running somewhere else, but explicitly give them root access to the target servers, e.g. with a remote SSH login.
   Again, this gives everybody in control of the gitlab server full production access, as well as anybody in control of one of the affected runners. Usually this is made less obvious by “only” giving docker daemon access, but that’s still equivalent to full root access.

There’s variants on this theme, like using Ansible for some abstraction, but it always boils down to somehow making it so that the gitlab server is capable of executing arbitrary commands as root on the production system.

Our Approach

For container management we’re going to use docker compose, the new one, not docker-compose. A compose.yaml file (with extensions, see below) is going to fully describe the deployment, and compose will take care of container management for updates.

Ideally we want to divide the task into two parts:

1. Initial setup
2. Continuous delivery

For the initial setup there’s not a pressing reason for full automation. We’re not setting up new environments all the time. There’s still some best practices and room for automation, see below, but in general it’s a one-time process executed with high privileges.

The continuous updates on the other hand should be fast, automated, and, above all, restricted. An update to a deployed docker application does exactly one thing: pull new image(s) then restart container(s).

Restricted SSH keys for update deployment

Wouldn’t it be great if we had an agent on the production server that could do that, and only that? Turns out, we have! Using additional configuration on ~/.ssh/authorized_keys we can configure a public key authenticated login that will only execute a (set of) predefined command(s), and nothing else1. And since sshd is already running and exposed to the internet anyway, we don’t get any new attack surface.

The options we need are:

• restrict to disable, roughly, all other functionality
• command="cd ...; docker compose pull && docker compose up -d" which will make any login with that key execute only this command (you’ll need to fill in the path to cd into).

Using docker compose

In order for this to seamlessly work, there’s some best practices to follow when creating the environment:

• All container configuration is handled by the docker compose framework
  
  • Specifically: docker compose up just works.
    No weird docker compose -f compose.foo.yaml -f compose.bar.yaml -e WTF_AM_I_DOING=dunno up incantations.

  
  

• The docker compose configuration should itself be version controlled
• The containers come up by themselves in a usable configuration, and can handle container updates gracefully
  
  • For example in Django, the django-admin migrate call must be part of the container startup
  • In general it’s not allowed to need to manually execute commands in the containers or the compose environment for updates. You’re allowed to require one necessary initialization command on first setup, under extenuating circumstances only.

  
  

• There’s also good container design (topic of a different blog post) with regards to separation of code and data

Good docker compose setup

There’s two ways to handle the main compose configuration of a project: As part of the git repository of one of the components, or as a separate git repository by itself.

• The first approach applies if it’s a very simple project, maybe just one component. If it contains only the code you wrote, and possibly some ancillary containers like the database, then you’ll put the compose.yaml into the root directory of the main git repository. This also applies if your project consists of multiple components maintained by you, but it’s obvious which one is the main one (usually the most complex one).
  Like if you have a backend container (e.g. Python wsgi), a frontend container (statically compiled HTML/JS, hosted by an nginx), a db container (standard PostgreSQL), maybe a cache, and some helper daemon (another Python project). Three of these are maintained by you, but the main one is the backend, so that’s where the compose.yaml lives.
• For complex projects it makes sense to create a dedicated git repository that only hosts the compose file and associated files. This specifically applies if the compose file needs to be accompanied by additional configuration files to be mounted into the containers. These usually do not belong in your application’s git repository.

The idea here is that the main compose.yaml file (using includes is allowed) handles all the basic configuration and setup of the project, independent of the environment. Doing a docker compose up -d should bring up the project in some default state configured for a default environment (e.g. staging). Additional environment specific configuration should be placed in a compose.override.yaml file, which is not checked into git and which contains all the modifications necessary for a specific environment. Usually this will only set environment variables such as URL paths and API keys.

Additional points of note:

• All containers should be configured read_only: true, possibly assisted by tmpfs: ["/run","/var/run/someapp"] or similar. If that’s not possible, go yell at the container image creator2.
• All configuration that is mounted from the outside should be mounted read-only
• Data paths are handled by volumes
• The directory name is the compose project name. That’s how you get the ability to deploy more than one instance of a project on the same host. The directory name should be short and to the point (e.g. frobnicator or maybe frobnicator-staging).
• Ports in the main compose.yaml file are a problem, since port numbers are a global resource. A useful pattern is to not specify a port binding in the compose.yaml file and instead rely on compose.override.yaml for each deployment to specify a unique port for this deployment. That’s one of the few cases where it’s acceptable to absolutely require a compose.override.yaml for correct operation, and it must be noted in the README.

Putting It All Together

This example shows how to set up deployment for project transmogrify/frobnicator, hosted on gitlab at git.example.com, with the registry accessible as registry.example.com, to host deploy-host, using non-root (but still docker daemon capable) user deploy-user.

Preliminaries

On deploy-host, we’ll create a SSH public/private key to be used as deploy key for the git repository containing the main compose.yaml and configure docker pull access. This probably only needs to be done once for each target host.
ssh-keygen -t ed25519
Just hit enter for default filename (~/.ssh/id_ed25519) and no passphrase. Take the resulting public key (in ~/.ssh/id_ed25519.pub) and configure it in gitlab as a read-only deploy key for the project containing the compose.yaml (under https://git.example.com/transmogrify/frobnicator/-/settings/repository).

We’ll also need a deploy token for docker registry access. This should be scoped to access all necessary projects. In general this means you’ll want to keep all related projects in a group and create a group access token under https://git.example.com/groups/transmogrify/-/settings/access_tokens. Create the access token with a name of deploy-user@deploy-host, role Reporter and Scope read_registry.

Caveat 1: docker can only manage one set of login credentials per registry host. Either use non-privileged/user-space docker daemons separated by project (e.g. with different users on the deploy host, each one only managing one project), which is a topic of a different blog post. Or use a “Personal” Access Token for a global technical user which has access to all the necessary projects instead.
(There’s a third option: Create multiple .docker/config.json files and set the DOCKER_CONFIG environment variable accordingly. This violates the “docker compose up should just work” requirement.)

Caveat 2: docker really doesn’t want to store login credentials at all. There’s a couple of layers of stupidity here. Just do the following (note: this will overwrite all previously saved docker logins on this host, but you shouldn’t have any):
mkdir -p ~/.dockerecho '{"auths": {"registry.example.com": {"auth": ""}}}' > ~/.docker/config.json
Then you can do a normal login with the deploy token and it’ll work:
docker login registry.example.com

First deployment / Setup

Clone the repository and configure any overrides necessary. Then start the application.
git clone ssh://git@git.example.com/transmogrify/frobnicator.gitcd frobnicatorvi compose.override.yaml # Or whatever is necessarydocker compose up -d
Your project should now be running. Finish any remaining steps (set up reverse proxy etc.) and debug whatever mistakes you made.

Set up for autodeploy

1. On deploy-host (or a developer laptop) generate another SSH key. We’re not going to keep it for very long, so do it like this:

ssh-keygen -t ed25519 -f pull-key # Hit enter a couple of times for no passphrase

1. Also retrieve the SSH host keys from deploy host (possibly through another way):

ssh-keyscan deploy-host

1. On deploy-host, add the following line to ~deploy-user/.ssh/authorized_keys (where ssh-ed25519 AAA... is the contents of pull-key.pub from step 1):

restrict,command="cd /home/deploy-user/frobnicator; docker compose pull && docker compose up -d" ssh-ed25519 AAA....

1. In gitlab, on group level, configure variables (git.example.com/groups/transmo…):

You may use the environments feature of gitlab here, which will generally mean a different set of values per environment (and then choosing the environment in the job in the next step). Afterwards, delete the temporary files pull-key and pull-key.pub from step 1.

1. In your project’s or projects’ .gitlab-ci.yaml file (this is in the code projects, not necessarily the project containing compose.yaml), add this (the publish docker job is outside of the scope of this post):

stages: - build - deploy# ...deploy docker: image: docker:git stage: deploy cache: [] needs: - publish docker before_script: | mkdir -p ~/.ssh echo "${SSH_KNOWN_HOSTS}" > ~/.ssh/known_hosts chmod -R go= ~/.ssh "${SSH_AUTH}" script: ssh -i "${SSH_AUTH}" -o StrictHostKeyChecking=no "${SSH_DEPLOY_TARGET}"
In order to handle multiple environments, you can also add
deploy docker: # ... rules: - if: $CI_COMMIT_BRANCH == "dev" variables: ENVIRONMENT: staging - if: "$CI_COMMIT_BRANCH =~ /^master|main$/" variables: ENVIRONMENT: production environment: "${ENVIRONMENT}"
Voila. Every time after a docker image has been built, a gitlab runner will now trigger a docker compose pull/up, with minimal security impact since that’s the only thing it can do.

Addenda

The preliminaries and initial setup can be automated with Ansible.

You can put the gitlab-ci configuration into a common template file that can be referenced from all projects. For example, we have a common tools/ci repository, so the only thing necessary to get auto deployment is to put
stages: - publish - deployinclude: - project: tools/ci file: docker-ssh.yml
into a project and do the deploy-host setup and variable definitions (well, and add the other include files that handle the actual docker image building).

1. man authorized_keys, section “AUTHORIZED_KEYS FILE FORMAT” ↩︎
2. Their image is bad and they should feel bad. ↩︎

blog.ploetzli.ch/2024/docker-d…

#ciCd #deployment #Docker #security #ssh

Do you use Flipboard?

I’ve been using Flipboard for many years, since its early days – and, I love it. It has been a brilliant social magazine and discovery platform for me. I’ve been curating a number of “magazines” there around my interests, over the same period of time. Up until recently, these were largely general / significant world news, tech news, and products I thought looked cool; this year I also added gaming and social web magazines as well. They are a nice way to aggregate stories in a visual manner, and share them with anyone interested in similar topics. I follow a lot of other magazines in the same way – Flipboard is an app that lives on my phone’s home screen and is often what I’ll open to check what’s new and happening in the world.

I was immensely grateful to have the opportunity to spend a bunch of time with Flipboard co-founder and CEO Mike McCue and members of the team during a recent trip to California. It was a delight for me personally to get to know Mike a little bit – although he was on the board of Twitter for a time that overlapped with when I was working there, it was not a situation that would have caused us to interact; his background in and knowledge of the tech industry is phenomenal (he was head of tech at Netscape, c’mon! amazing), and he is really excited about the opportunity that the Fediverse offers to keep the web open. I’m fully on-board with that!

One of the things that the Flipboard team has been working on is pivoting the platform into a fully ActivityPub-compatible experience. It has been exciting to follow their progress as these efforts have expanded. Last week, Flipboard announced that a range of local news publications around the US were joining the fediverse through this integration. This sort of thing really speaks to the power and opportunity that the open social web provides, both to publishers to reach the federated audience, and for readers to follow more topics and feeds that interest them, in platforms and apps of their own choosing.

It has been possible to follow my Mastodon account inside of the Flipboard app for a while, but I’ve been excitedly waiting for the opportunity for my Flipboard account to be federated in the other direction – and that happened this week!
My Flipboard profile page, showing that it is now federated to other ActivityPub servers
What does this mean in practice?

• You can now choose to follow my main Flipboard profile without using the Flipboard app directly (if you want). If you follow my main account, federated address @[url=https://andypiper.co.uk/author/andypiper/]andypiper[/url]@flipboard.com, that will mean you get to see flips to all of my magazines.
• You can also follow the individual magazines if you’re only interested in one or two of them. That’s where things get a bit less straightforward to directly discover from your Fediverse client app – the addresses are easy to find if you click through to one of them from my Flipboard profile, but you might need to know the complete address to search in your Mastodon client.
  
  • My “Fediverse & The Social Web” magazine is here on the web, and the federated address is @[url=https://flipboard.com/magazines/zPfyChp-R7uaEvFMKiLszQ:m:3626430]fediverse-the-social-web-andypiper@flipboard.com[/url] – a little bit longer and maybe less easy to find from Mastodon.

  
  

BTW I’m not sure when others can get their Flipboard accounts federated – it’s a beta feature for a limited set of curators at the moment and I am one of the lucky ones, partly since I am working with my friends at Flipboard to build community around the fediverse.

I’ve got a toy app that I’ve been noodling with for a while (based on something that Les Orchard made) that is intended to act as a discovery page for my various fediverse profiles; it doesn’t quite work the way I’d like, since WebFinger responses are sometimes not consistent / to spec, some platforms require authentication, so several are missing right now – I’m tinkering occasionally. I should probably add the Flipboard bits to that, at some point… oh and this reminds me to update it to point at my new PeerTube account as well.

Another quick plug for my t-shirt design for Fediverse fans, while I’m at it! Flipboard not mentioned there, though…

Final recommendation – regardless of your interest in either Flipboard, or my own magazines and profile there – you need to check out Dot Social, Mike’s podcast about the future of the open social web. It has been really varied and the episodes have been full of deep and interesting thinking. Worth a listen, if you’re following the evolution of the web today.

andypiper.co.uk/2024/07/31/fed…

#100DaysToOffload #activitypub #discovery #federated #fediverse #flipboard #magazines #News #openWeb #Reading #socialWeb

Andy Piper

2024-07-25 14:51:54

I’m all-in on the Fediverse as the best way to own my own data and network. It’s based on an open standard (ActivityPub), loosely-coupled and open source services, and everyday people (and, it isn’t subject to the whims of lying, narcissistic billionaires). You can think of it as the next iteration of the open web, with social features baked in.

I’ve got a number of accounts that correspond with content that the different networks are good at – posts on Mastodon, photos on Pixelfed, reading habits on Bookwyrm. If I post something on one that I think my followers on a different network may like, I can boost it directly for them to see; or, folks can follow me on whatever platform they choose. My WordPress blog has federation switched on, too, so you can if you like follow @andypiper@andypiper.co.uk, and read my blog posts directly in your Fediverse platform of choice. (of course, RSS remains another excellent way to follow my blog).

By the way, if you’re curious what the buzz about the Fediverse is all about, I recommend Elena Rossini’s newsletter The Future is Federated – in the edition that was published today, she did a great job of explaining some of the interoperability between federated networks, from the perspective of, you know, just actually, using them, rather than from a deep technical angle. Worth following!

One of the other services I’ve been using is PeerTube, a federated alternative to YouTube. Up until now I’ve been on Diode Zone. However, that instance recently started to run into some storage issues, and also switched off the live broadcast feature that I’d occasionally used to stream some 3D printing and pen plotter content (this is straightforward to configure in OBS, by the way). I really appreciated my time on Diode Zone, but I’ve chosen to move across to MakerTube, a relatively newer instance dedicated to “makers, musicians, artists and DIY content creators”.

One of the core elements of the Fediverse is data ownership, and some form of portability. When I first joined Mastodon I started out at mastodon.social/@andypiper – if you visit that profile page now, you’ll find my posts starting in November 2016 and ending in November 2022, when I moved over to my current home, macaw.social – there’s a large message that points you at my current location if you look at my original profile. When I switched instances, my whole follower network went with me, seamlessly – unlike, for example, Twitter, where I lost everything when I deleted all my accounts; or Facebook, which heavily relies on its lock-in – read Cory Doctorow‘s excellent book The Internet Con to understand what’s happening there.

The current state of portability is absolutely not perfect – in the case of Mastodon, there’s a process which enables to you to migrate from one server to another, and that automatically resubscribes you to your network, and your followers to your new account, but it’s currently not technically possible to take the past posts with you (there is a W3C Social Web Incubator Community Group taskforce that is working on a more complete set of specifications for data portability that may help to improve this in the future).

Here is how I moved from Diode Zone to MakerTube:

• Requested an account on MakerTube. They want to know who is part of the community, their content intentions, etc. I explained that I’m a maker and I also may plan to use the streaming feature.
• Requested an export of data from Diode Zone.
  
  • this was technically a little bit frustrating, as it got stuck the first time (likely because I had requested it when there were earlier storage issues), but the instance owner was really kind and helped to clear the stuck process.
  • it was also a bit annoying because of the size of my export including the videos, which was a lot of gigabytes; so I ended up having to run a script that kept running wget with a resume flag to get the data in chunks.

  
  

• Setup the basics of my new account on MakerTube.
• Imported the export from the other instance.
• Modified a few places that were pointing to Diode Zone, such as my links page, and also updated a few embeds that were loading videos from my previous account, such as some of the Fedidevs.org meeting recordings.
  
  • … while I was doing that, I thought I’d take advantage of PeerTube’s multi-channel capability, and created a specific channel for Fedidevs content, which you can now follow @fedidevs from other Fediverse platforms.

  
  

There were a couple of slightly rough edges, but nothing very significant:

• Unlike Mastodon, PeerTube does not run a process to tell your followers that you have moved, and to resubscribe them to your new account. In my case I didn’t have a huge number of followers, but I will be posting a video there to say that I moved, and I also updated my profile information to point to the new instance. It did re-follow the accounts I had followed, but didn’t do the other side of the process.
• The new instance imported my playlists, which was great – but some of them were playlists of my own videos, which I tend to create for curation and organisation, and of course, they still pointed at the videos on the previous instance. This was fairly straightforward to fix, just removed and re-added the videos on the new instance.

So there you are. You can follow my entire MakerTube account, my main channel, or the Fedidevs channel, if you like. I also have it set up to import future content that I may choose to post to YouTube, so that it has a free and open backup that Google can’t delete if I ever lose my account there.

I’m still frustrated that WordPress doesn’t seem to have a good integration for PeerTube content yet – I can paste a YouTube link here and get an embedded video, that’s less easy for PeerTube – but, I’m hopeful that will improve in the future.

andypiper.co.uk/2024/07/25/swi…

#100DaysToOffload #activitypub #coryDoctorow #data #diodeZone #fedidevs #fediverse #makertube #migration #portability #streaming #video #YouTube

The Future of Social is Here: a Show and Tell (part 1: Mastodon ↔️ Pixelfed)

The secret ingredient of the Fediverse is the ActivityPub protocol and how it allows networks to talk to each other. See the magic in action!

^{Elena Rossini}