Tools! What are they good for?

If you aren’t using tools to think, you’re not really thinking.

I use four tools for thinking:

• notes: external memory, minutes, lists, reminders, drafts, analysis
• [excalidraw: spatial reasoning, conceptual relationships, mental models
• excel: fast math & analysis, modeling, fermi analysis
• websequencediagrams: reasoning about sequences of events over time

Three of these I love, but I struggle with notes.

There’s an endless stream of note-taking and rough-drafting that very rarely morphs into something more durable. I don’t think I’m alone in the way I use my notes app as a sort of rolling buffer of whatever is important in the moment.

I’m always taking notes. Adding to lists, organizing ideas, writing draft essays, exploring ideas, taking minutes, creating reminders. It’s all just flotsam in the river. I’ve tried a few different tools over the years. Pen and paper, onenote, evernote, apple notes. I keep coming back to apple notes for the synchronization across devices, because a notes tool is even worse when you have to go out of your way to read a note.

The sorry state of notes apps is inexcusible, considering Tim Berners Lee’s paper on information management was written more than 35 years ago. In it he identifies the two principal ways that every notes app fails:

1. hierarchies / trees / folders - are arbitrary and are inadequate to describe reality
2. keywords / search - are challenged because language has synonyms and over time we (individually) forget the words we used to describe things

Tim is right. Links between data are at least as important as the data itself. They make data retrievable, and without a reliable and effective means of retrieving data, an information system is a failure. My notes app is filled with withered folder hierarchies, the logic of which is lost to time. And I can’t tell you how many times I’ve searched Apple notes and come up empty handed because I was using the wrong synonym, or had someone’s name wrong, or forgot when something had happened. Or worse, when my search includes a common term and the results are useless unrelated content. A keyword search requires exact specificity of recall, and because we’re all human, a significant portion of searches will fail.

But when an information system permits traversal using links, the recall system can use the same associations as our memory! I may not remember who I worked with, but I remember the project, the technology, or something else. That something else, when coupled with links, means a successful search is more likely.

Most notes apps do have a mechanism of linking between notes. But they’re not built around that concept. Obsidian is.

Obsidian

Obsidian is Yet Another Notes Tool. It’s also qualitatively different - let’s explore

• All notes are markdown. This is a text-first interface.
• The syntax for linking to another note is dead simple. Type [[ and the app starts searching note titles.
• Links between notes are bidirectional. I can search based on inbound links, rather than require an index be made only of outbound links.
• There are tools for exploring the interconnectedness (or, ’the graph’) of information in your notes.

The killer feature here is the links between notes. Links are such a powerful concept for relating bits of information together - they’re the foundation of the whole internet! And while the internet has a problem with bad actors poisoning the dataset, my own personal notes will not. Every other notes app gets this wrong. They require right-clicking and entering context menus to get fully qualified note URLs, they open up modal popups that have to be closed, they take three minutes traversing a database to rewrite links when you move a document. Obsidian is completely frictionless in the most important way.

It also has two other features that are awesome for me specifically.

• Create excalidraws in the app: great! (i can’t even)
• Link from excalidraw to notes: great!
• Plop the notes into a shared folder (like my iCloud drive) and they’ll synchronize everywhere automatically: great!
• Back notes up with git: great!

It does a lot and it’s worth a shot. It’s a bit of a challenge coming into this cold, with a large corpus of existing notes in a different tool without any links. But it’s possible to pull data out of Apple notes and insert it into Obsidian. Because Obsidian stores all notes as markdown on disk, all you need is to get your existing notes to markdown on disk

Migrating from Apple Notes

This was “easy” except for how the obsidian app doesn’t support rendering tiff files.

1. Back up notes

   I used Exporter for this and it was really convenient. Drop the markdown headers.

2. Convert all the tiffs to pngs

   What wasn’t convenient was discovering that I had 300 tiff files embedded in my notes for some reason.
   I wrote a script to convert them. It appended .png to the original filename without changing the extension because there was one (1) name collision.

3. Rewrite all the links in bash

   There was a great stackoverflow answer for this. When editing the files it was important to me that I preserve the modified time. So this was more complicated than it needed to be.

   find iCloud -name '*.md' -exec bash -c 'cp -p "$0" tmp; sed -i "" "s/\\.tiff/\\.tiff\\.png/g" "$0"; touch -r tmp "$0" ' {} \;

4. Move all the files to the obsidian vault

   They just show up automatically.

First Impressions

Once I got my existing 2000 notes into the tool I was able to play around a little more.

• Tabs and panels in a notes interface is smart.
• The excalidraw plugin gives me heart palpitations.
• Links are really easy, so is rearranging stuff.
• iphone, ipad apps are great
• my own ambition around linking existing stuff together is misplaced, migrating existing data is hard
• Would love it if mouseover events showed keyboard shortcuts
• Would also love it if I could focus on the directory tree and preview existing notes quickly with a keyboard
• Not much value out of the graph yet. This is the part that will take time

Next Impression

I’m hopeful that making an investment each day over time will yield a reasonably well-connected graph of knowledge. As of right now it’s looking a little sad. Call it a humble beginning.

My plan is to incrementally add links whenever I add a new note, and to opportunistically link existing stuff together whenever I go through the trouble of searching my old notes for some information on a topic. Do this for a couple of months, and I should start to experience moments where the fact that notes are linked together allows me to retrieve a more complete set of context and information on a topic faster.

I have a feeling this is going to be a lasting tool. The dev team is great, the design principles are solid, and the resulting tool is excellent.

Tiff Converter Code & Notes

Had to install libtiff, edit a library to fix a type error, and run my command w/ cgo. But it worked.

brew instal libtiff
CGO_CFLAGS="-I/opt/homebrew/include" CGO_LDFLAGS="-L/opt/homebrew/lib" go run -v tiffconvert.go

And here’s the source. Not going to bother putting this one on github.

package main

import (
	"fmt"
	"image/png"
	"io/fs"
	"log"
	"os"
	"path/filepath"
	"strings"

	"github.com/andviro/go-libtiff/libtiff"
)

type filedata struct {
	Count int
	Bytes int64
}

func main() {

	wd, err := os.Getwd()
	if err != nil {
		log.Fatal(err)
	}

	log.Printf("searching for tiffs in %s", wd)

	i := 0
	tiffs := []string{}
	extensions := map[string]*filedata{}

	filepath.WalkDir(wd, func(path string, d fs.DirEntry, err error) error {
		i += 1
		ext := filepath.Ext(path)
		if _, ok := extensions[ext]; !ok {
			extensions[ext] = &filedata{}
		}
		data := extensions[ext]
		data.Count += 1
		s, _ := d.Info()
		data.Bytes += s.Size()

		if strings.HasSuffix(path, ".tiff.png") {
			log.Printf("removing %s", path)
			os.Remove(path)
		}

		if ext == ".tiff" {
			tiffs = append(tiffs, path)
		}

		return nil
	})

	for ext, data := range extensions {
		log.Printf("%s: %d files, %d bytes", ext, data.Count, data.Bytes)
	}
	log.Printf("tiffs: %d", len(tiffs))

	successes := 0
	errs := []error{}
	for i, tiff := range tiffs {
		if err := validate(tiff); err != nil {
			log.Printf("error validating #%d. %s: %s", i, tiff, err)
			errs = append(errs, err)
			continue
		}

		if err := libtiffConvert(tiff); err != nil {
			log.Printf("error converting #%d. %s: %s", i, tiff, err)
			errs = append(errs, err)
			continue
		}

		successes += 1
		if successes%10 == 0 {
			log.Printf("successes: %d. errors: %d", successes, len(errs))
		}
	}

	for _, err := range errs {
		log.Println(err.Error())
	}

	log.Printf("successes: %d. errors: %d", successes, len(errs))
}

func libtiffConvert(path string) error {
	var outErr error
	newPath := renameTiff(path)

	defer func() {
		if outErr != nil {
			os.Remove(newPath)
		}
	}()

	tiff, err := libtiff.Open(path)
	if err != nil {
		outErr = fmt.Errorf("could not open file: %s. error: %v", path, err)
		return outErr
	}
	defer tiff.Close()

	_ = tiff.Iter(func(n int) {
		if outErr != nil {
			return
		}
		if n > 0 {
			outErr = fmt.Errorf("tiff has more than one image: %s", path)
			return
		}

		img, err := tiff.GetRGBA()
		if err != nil {
			outErr = fmt.Errorf("could not get rgba: %s. error: %v", path, err)
			return
		}
		_ = img

		out, err := os.Create(newPath)
		if err != nil {
			outErr = fmt.Errorf("could not create file: %s. error: %v", newPath, err)
			return
		}
		defer out.Close()

		if err := png.Encode(out, &img); err != nil {
			outErr = fmt.Errorf("could not encode png: %s. error: %v", newPath, err)
			return
		}

	})

	return outErr
}

func validate(path string) error {
	newPath := renameTiff(path)
	if _, err := os.Stat(newPath); err == nil {
		return fmt.Errorf("file already exists: %s", newPath)
	}
	return nil
}

// every file ends in .tiff
// replacing the last 4 characters with png
func renameTiff(path string) string {
	dir, filename := filepath.Split(path)
	newFilename := filename + ".png"
	return filepath.Join(dir, newFilename)
}