Search Notes Fast: With Taskwarrior, jq & ripgrep
Search through Taskwarrior annotations fast.
If you've installed Taskwarrior, jq and ripgrep, you're already good to go! The searching/filtering command is actually a single line:
task export | jq -r '.[].annotations[]?.description' | rg -ni
I prefer to alias the full command (I like tt because of how short it is):
alias tt="(above)"
See the breakdown below!
As a lawyer, I write notes all the time.
And I do it with perhaps more vigour than most people, given the weight of documentation and the professional care taken in my line of work.
I've therefore a bit of a ritual:
When events happen, I scribble them down to ensure that they are at least recorded. There will be a date (and sometimes a time), and a single line describing a thought or event. No more. As to why:
Simple lines are quick to read and write.
Annotations should not be memoirs. All I need is to jog my memory, to help me to decide what to do now/next or in the future.
Details are nice, but unnecessary. I expand into details when I have time, but if something is truly significant, reference documents often already exist.
Where important, I make detailed notes of events or observations. For such notes I ensure historical, solicitor-like precision.
For instance, my legal annotations record my compliance with a particular law/rule/procedure, the laws that were raised/touched, what a judge/registrar directed, persons involved, communications with clients/third parties, instructions taken with exact dates and times, and so on.
I log most things I've read, watched or listened to - e.g. books, articles, shows, videos, podcasts. I notice that it's almost impossible for me to understand something with a single pass; so my annotations are how I help myself and my future self learn. My annotations bear thoughts, marginalia, page/location numbers, timestamps of interest - which let me return to any particular logged item so that I may review and re-appreciate them again.
I record insightful, useful or whimsical quotes/comments. I enjoy moments of epiphany and quick chuckles. So I often copy hot takes and funny retorts (e.g. from Reddit/YouTube/Hacker News) and sprinkle them around arbitrarily. These comments, even where not entirely relevant, often become serendipitous triggers of ideas or leaping points for exploration.
Given how important my notes are to me, two points arise: How fast can I make them? And how do I retrieve them at speed?
I previously used Todoist. It was painful for annotations, and terrible at searching notes. I've stopped using it personally, and no longer recommend it.
These days? I use Taskwarrior. I love just how fast it lets me make and annotate my tasks. (The default key bindings - a to add tasks, and A to annotate - work great.) It blazes when searching, but generates clutter when sometimes all I want are just my notes.
That led me to think: what if I pair Taskwarrior with jq and ripgrep? Will it work?
I'm pleased to report: yes! Frankly, this is the fastest system I've ever used, which counts for much if you're a prolific note-taker/note-searcher like me.
Here's an example, if I were to look for annotations on 'vim', 'rust' or 'zombie' in my tasklist:
If you've never used jq and ripgrep:
jq is a data processor of sorts. It works with JSON, and lets you handle compatible data structures from the command line in an almost object-oriented-like fashion.
ripgrep is a text-searching command line tool. It lets you search through files/directories for specific text patterns. You can also feed it lines of text.
The command, when broken-down, has three parts. Let's go step by step:
First, export all tasks with Taskwarrior. On the commandline:
> task export
The command task export gets you JSON data/dicts, line by line.
Process with jq. Each line contains a key-value structure titled annotations, containing two keys - entry (date-time) and description (the annotation-text). You only want the latter, so pipe the exports through jq:
> ... | jq -r '.[].annotations[]?.description'
Notice the ? symbol above. Sometimes jq can't read Taskwarrior's exported lines (which gets you Cannot iterate over null errors). The ? is a validity/exception check; it lets processing continue past erroneous lines.
And here's a more detailed variant with UUIDs, via string interpolation:
> ... | jq -r '.[] | \"\(.uuid) / \(.annotations[]?.description)\"'
Search with ripgrep. Once jq returns with all your annotations, hit it with ripgrep to filter for lines matching the phrase. Add as much salt as you wish; I prefer just a pinch of the basics - case insensitivity, line numbers:
> ... | rg -ni
The above is the perfect minimalist solution for me, as I'm only interested in the content of my notes. But with a bit of smarts, you could expand upon this into other cool ideas - e.g. annotations with entry dates, UUIDs, and so on.
One more consideration: I do not pre-process or filter tasks. I prefer retrieving all tasks outright (via task export).
Am I worried about performance? Not at all! First, I see no need to optimize prematurely:
... programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.
- Donald Knuth
Taskwarrior is also performant - it can go through thousands of tasks on the order of microseconds. Given how insanely fast modern/future CPUs are:
If you have no scalability issues, there's little to worry about.
Even with the overhead of jq + ripgrep, Taskwarrior slowdowns won't be felt for single users.
Even if I had 10x the amount of notes I now have (which will take years or decades to amass), CPU improvements would have scaled alongside it in the meantime.