Jun 23 2023 TIL: converting JSON using git-history

git-history is a tool created by Simon Willison, that makes it easy to work with data collected via git scraping.

It expects data in a particular format - a flat JSON array. If your data isn't already there, you can use the --convert option to run any python code to make it the right shape.

For example, if your JSON looks like this:

{
    "incidents": [
        {
            "id": "552",
            "name": "Hawthorne Fire",
            "engines": 3
        },
        {
            "id": "556",
            "name": "Merlin Fire",
            "engines": 1
        }
    ]
}

You might ingest it via a command like this:

git-history file incidents.db incidents.json --id "id" --convert "json.load(content)['incidents']"

If you're using git scraping to track any data, it's useful for working with the data efficiently, so try it out!