Celery Autoscaling: No touchy

Celery Autoscaling: No touchy

Despite casually using Celery for the last few years, it's only been the last 6 months where I've had to cope with using it at serious scale. This is the point you start looking at best practices to help with the pain of it all. And you might come across the idea of using autoscaling along with different queues. For example, you could use autoscaling with a default queue and a high priority queue: (venv) $ > celery -A proj worker -l info -Q default --autoscale 4,2 (venv) $ > celery -A proj worker -l info -Q high_priority --autoscale 8,2 It…

Read More

Git Hooks: Autoformat before commit

Git Hooks: Autoformat before commit

Autoformatters can be great, keeping diffs small and a code base readable across teams and engineers. Black formatting has been enforced in Lobit builds since day one and I recently added it to the repos at Downstream as well. But what isn't great is forgetting to run the formatter, having the CI build fail, and having to add a "Formatting" commit. That's where Git Hooks come in. Git hooks are stored in .git/hooks folder of any given Git project. There should be some samples pre-populated. For our purposes, pre-commit is the hook of interest. Lets focus on…

Read More

How to automate Ghost/Medium cross-posting via Zapier

How to automate Ghost/Medium cross-posting via Zapier

With this blog, I'm eager to automate a lot of the tedium that comes with cross-posting to other platforms. Medium was first on the docket due to its popularity - and I quickly came across this fantastic article by Christoph Michel on how he automates the process. At the time of writing, the typenil blog uses Ghost for publishing content, so I had to do things a little differently than Christoph. I also wanted to sidestep the need to host the automation code, myself, so using Ghost's built-in Zapier integration made the most sense. (NOTE: you can use the repo…

Read More

SQLAlchemy + FactoryBoy: Passing arbitrary sessions to factories

SQLAlchemy + FactoryBoy: Passing arbitrary sessions to factories

In work projects in the past, my team would try to avoid dealing with the complexity of SQLAlchemy database sessions by making one global session that every module referenced. It made things easy and straightforward and it played nice with FactoryBoy - whose factories seem to work quite well under those conditions. Going into building lobit.io, I tried to do things the same way, but as the code base and the number of unit tests grew, I kept facing what seemed to be an unquashable avalanche of OperationalErrors griping about "Too many connections". I implemented an overhaul…

Read More