Handling JSON on command line

JSON is one of my favorite formats for data as it balances human readability/writeability with ease of parsing programmatically. In the modern software development, at least on the web, we deal with JSON all the time, whether it’s thanks to calling APIs, dealing with configuration, using JSON data when building static sites or any of the other use cases.

One downside of JSON is that it’s verbose and even relatively small amount of data becomes difficult to read and figure out on command line. Thankfully, people have built tools that can help you become even more productive with JSON.

jq

First tool I want to introduce is jq, a lightweight and flexible command-line JSON processor. It’s a tool that uses its own language syntax that allows the user to write compact one-liners that select, filter and manipulate the data provided to it.

jq is written as a series of filters separated with pipe character |.

Let’s look at how it works through an example. Here we have a snippet of localization data in three languages.

{
  "languages": {
    "fi": "Suomi",
    "en": "English",
    "sv": "Svenska"
  },
  "send": {
    "en": "Send",
    "fi": "Lähetä",
    "sv": "Sänd"
  },
  "ordinals": {
    "1": {
      "en": "first",
      "fi": "Ensimmäinen",
      "sv": "första"
    },
    "2": {
      "en": "second",
      "fi": "toinen",
      "sv": "andra"
    }
  },
  "age": {
    "en": "Age",
    "fi": "Ikä",
    "sv": "Ålder"
  }
}

Let’s say we want to extract only the Finnish language values from it and transform a three-language sub-object into a string value.

jq 'walk(if type == "object" and has("fi") then .fi else . end)' < input.json

We use jq’s walk function that recursively goes through the entire JSON structure. For each item, we check if it is an object and has a key “fi”. If it does, we replace that item with the value of key “fi” with .fi. In jq, we refer to keys with syntax that starts with a dot and follows with the key. If it is not an object or it does not have the key “fi”, we “replace the item with itself” or in other words, do nothing.

This gives us the end result of

{
  "languages": "Suomi",
  "send": "Lähetä",
  "ordinals": {
    "1": "Ensimmäinen",
    "2": "toinen"
  },
  "age": "Ikä"
}

As you can see, jq’s syntax is very powerful and compact but the learning curve is quite steep and takes a lot to learn.

Since it uses a query based approach, the huge benefit is that it’s possible to share these snippets in chats, forums, blog posts and such so they can be reused. I have previously written how that is one of the reasons I love command-line interfaces.

jless

The second tool in my toolbox is jless which is an interactive JSON viewer for the command line. It uses vim keybindings to traverse the data which can be a blessing or a curse depending on how comfortable you are with using vim. For me, it made a big difference as I was right at home with jless from the very beginning.

Piping a JSON object into jless launches it into an interactive session where you can move around, collapse and expand objects, copy nodes, search with regular expressions and a few other lovely things.

As vim has a quite a notoriously rough learning curve, I imagine getting productive with jless can have similar learning curve for those not familiar with vim.

fx

The third tool offers tools similar to both jq and jless and it’s called fx. Similar to jless, when you pipe JSON data into it, it starts an interactive thing. But similar to jq, you can also add arguments to the CLI tool call and it’ll run those arguments sequentially to the data.

Where jq uses its own, sometimes bit cryptic looking syntax, fx supports Javascript.

For example, if we’d get a list of objects with attribute name and we want to find all the unique ones, we can call fx with two functions: first one maps through all objects to pick only the name to return and the second uses fx’s built-in uniq function to filter out duplicates.

fx 'map(item => item.name)' uniq

I’m still new to fx and one thing I haven’t managed to figure out yet is how to do recursive functions like I used in the earlier example with jq. Despite spending half of an afternoon trying to come up with a solution, I couldn’t come up with anything and at one point I started to notice that the possible outcome, if I came up with one, would have been overly complicated compared to the jq one.

Googling for fx related things is very hard since fx (or f(x)) is a mathematical concept and either there’s not a lot of content to be found or it’s hard to find them from amidst all the math content.

So if you’re familiar with Javascript, using fx might be more productive and easier to get start with for you compared to jq. For the interactive mode, it supports similar vim-style keybindings as jless.

Wrap up

I keep all these three in my toolbox to speed up my processes. Sometimes I find something easier and faster to write with jq’s processing and syntax style and sometimes it’s the case with fx.

edit 2023-12-03: I discovered there's also jaq and jql. I haven't tried them out yet myself.

Any other great JSON tools that I have missed in my life? Share them in the Mastodon post below and let’s build our collective knowledge!