Using jq and bash for JSON processing

Given by Henry J Schmale on 2020 November 06

This is a lightning talk I gave to the CPLUG November 2020 meeting on how to use jq with bash for dealing with JSON in modern tooling. NOTE: use the space bar to scroll.

The Problem

  • More and more programs are using JSON as a data interchange format.
    • See docker inspect
    • Github API
  • JSON is not easily parsed using grep, sed, and awk in shell pipelines.
    • JSON is easily consumed via many programming languages, and this makes it very popular as an interchange format.
  • Shell pipelines are great for explorations of certain APIs and running programs.
    • Composable, Fast, Parallel by default.

The Solution: jq

jq can be thought of as a stream editor (sed) for JSON data. It can slice and filter, map, and transform this data very easily.

Slice
Extract elements out of an array.
Filter
Extract things matching a boolean expression
Map
Apply some operation to the resulting value. Ex. add, subtract, concatenate.
Transform
Take one JSON input, and move the fields around to become a different JSON object.

Real World Usages

  1. Parsing out health check information. Many services have a health check endpoint.
    • Not every business has setup proper monitoring for these end points.
  2. Extracting fields from your security IDS.
    • You find your company had a data breach over the VPN. You have a JSON log of where everyone logged in from. You need to find the anonomylous login.
  3. Working with object stores on the command line.
    • Ex. Pumping information out of MongoDB, and working on it with the shell.

I initially learned about it through hacker news, and learned it to compete in a capture the flag event.

Usages of jq

Direct file input:

jq $OPERATION_EXPRESSION test-input.json

From a pipe:

cat test-input.json | jq $OPERATION_EXPRESSION

Sending the output to another operation:

cat test-input.json | jq $OPERATION_EXPRESSIOn | less

I know this is a useless use of cat, but it’s for example purposes.

Examples of jq operations

The Identity Transformation

An identity transformation just shunts the input to the output. jq will pretty print to help the human read the output.

jq '.' array.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Get the value of a key

We want to extract a value from some path of the json object’s keys.

jq '.name' array.json

Input

{
  "name": "Bob",
  "age": 20,
  "pets": {
    "cats": 1
  },
  "hobbies": [
    "bowling"
  ]
}

Output

"Bob"

Get the value of multiple keys

Sometimes we want to get more than 1 key out at a time. A comma can separate the multiple keys to pull.

jq '.name,.age' object.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

"Bob"
20

Extracting a key from an array

In order to access the objects in a given array, they must first be unwraped with .[], we can the pipe them to the key expressions.

jq '.[] | .name,age' array.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

"Bob"
20
"Joe"
25
"John"
37
"Ron"
51

Indexing an Array

jq uses the standard array syntax to pull out single elements. In this case, the last object is grabbed.

jq '.[3]' array.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

{
  "name": "Ron",
  "age": 51,
  "hobbies": [
    "eating steak",
    "woodworking"
  ]
}

Slicing a range of elements from an array

It might be neccessary to pull a contigious series of elements. That is a slicing operation. A range is in the pattern of start:end_exclusive. In this example we get the middle 2 elements.

jq '.[1:3]' array.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

[
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  }
]

Slicing a specific elements from an array

In this example, the first and last elements in our example are pulled by index.

jq '.[0,3]' array.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

{
  "name": "Bob",
  "age": 20,
  "pets": {
    "cats": 1
  },
  "hobbies": [
    "bowling"
  ]
}
{
  "name": "Ron",
  "age": 51,
  "hobbies": [
    "eating steak",
    "woodworking"
  ]
}

Transforming a series of objects

Sometimes we want to change how a JSON object is formatted. Such as creating an array where the first is the name, and the remaining indexes are their hobbies. The first operation unpacks the array. Then generates an array by taking the name, and splatting the hobbies array.

jq '.[] | [.name, .hobbies[]]' array.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

[
  "Bob",
  "bowling"
]
[
  "Joe",
  "fishing",
  "football"
]
[
  "John",
  "cooking",
  "woodworking"
]
[
  "Ron",
  "eating steak",
  "woodworking"
]

Mapping to a new value

Arithmatic and basic string operations can be applied using jq. It prints out the modified object. In this case, we are doubling every person’s age.

jq 'map(.age * 2)' array.json

Input

[
  {
    "name": "Bob",
    "age": 20,
    "pets": {
      "cats": 1
    },
    "hobbies": [
      "bowling"
    ]
  },
  {
    "name": "Joe",
    "age": 25,
    "pets": {
      "dogs": 1
    },
    "hobbies": [
      "fishing",
      "football"
    ]
  },
  {
    "name": "John",
    "age": 37,
    "pets": {
      "dogs": 1,
      "cats": 2
    },
    "hobbies": [
      "cooking",
      "woodworking"
    ]
  },
  {
    "name": "Ron",
    "age": 51,
    "hobbies": [
      "eating steak",
      "woodworking"
    ]
  }
]

Output

[
  40,
  50,
  74,
  102
]

Filtering JSON

Sometimes we want to filter things. We can use the select function in jq. This allows us to apply a boolean expression to the input array.

jq '.[] | select(.id == "second")'

Input

[
   {"id": "first", "val": 1}, 
   {"id": "second", "val": 2},
   {"id": "third", "val": 3}
]

Output

{
  "id": "second",
  "val": 2
}

Building a histogram

A histogram is a very useful tool for determining frequency of values. jq doesn’t have a reduce, but we can use other shell commands to get us there. In this example we use the last 5 commits from the jq repository to determine who commited the most in that time period. Awk could be used as an alternative for sort uniq pattern.

jq '.[].commit.author.name' jq_repo_commits.json | sort | uniq -c

Input

[
  {
    "sha": "a17dd3248a666d01be75f6b16be37e80e20b0954",
    "node_id": "MDY6Q29tbWl0NTEwMTE0MTphMTdkZDMyNDhhNjY2ZDAxYmU3NWY2YjE2YmUzN2U4MGUyMGIwOTU0",
    "commit": {
      "author": {
        "name": "Maximilian Roos",
        "email": "5635139+max-sixty@users.noreply.github.com",
        "date": "2020-03-29T21:40:42Z"
      },
      "committer": {
        "name": "William Langford",
        "email": "wlangfor@gmail.com",
        "date": "2020-06-08T16:35:13Z"
      },
      "message": "Add some missing code quoting to the manual",
      "tree": {
        "sha": "53eb4bc72594be6cd3847450cd3d3e06246199cd",
        "url": "https://api.github.com/repos/stedolan/jq/git/trees/53eb4bc72594be6cd3847450cd3d3e06246199cd"
      },
      "url": "https://api.github.com/repos/stedolan/jq/git/commits/a17dd3248a666d01be75f6b16be37e80e20b0954",
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null
      }
    },
    "url": "https://api.github.com/repos/stedolan/jq/commits/a17dd3248a666d01be75f6b16be37e80e20b0954",
    "html_url": "https://github.com/stedolan/jq/commit/a17dd3248a666d01be75f6b16be37e80e20b0954",
    "comments_url": "https://api.github.com/repos/stedolan/jq/commits/a17dd3248a666d01be75f6b16be37e80e20b0954/comments",
    "author": {
      "login": "max-sixty",
      "id": 5635139,
      "node_id": "MDQ6VXNlcjU2MzUxMzk=",
      "avatar_url": "https://avatars0.githubusercontent.com/u/5635139?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/max-sixty",
      "html_url": "https://github.com/max-sixty",
      "followers_url": "https://api.github.com/users/max-sixty/followers",
      "following_url": "https://api.github.com/users/max-sixty/following{/other_user}",
      "gists_url": "https://api.github.com/users/max-sixty/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/max-sixty/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/max-sixty/subscriptions",
      "organizations_url": "https://api.github.com/users/max-sixty/orgs",
      "repos_url": "https://api.github.com/users/max-sixty/repos",
      "events_url": "https://api.github.com/users/max-sixty/events{/privacy}",
      "received_events_url": "https://api.github.com/users/max-sixty/received_events",
      "type": "User",
      "site_admin": false
    },
    "committer": {
      "login": "wtlangford",
      "id": 3422295,
      "node_id": "MDQ6VXNlcjM0MjIyOTU=",
      "avatar_url": "https://avatars2.githubusercontent.com/u/3422295?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/wtlangford",
      "html_url": "https://github.com/wtlangford",
      "followers_url": "https://api.github.com/users/wtlangford/followers",
      "following_url": "https://api.github.com/users/wtlangford/following{/other_user}",
      "gists_url": "https://api.github.com/users/wtlangford/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/wtlangford/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/wtlangford/subscriptions",
      "organizations_url": "https://api.github.com/users/wtlangford/orgs",
      "repos_url": "https://api.github.com/users/wtlangford/repos",
      "events_url": "https://api.github.com/users/wtlangford/events{/privacy}",
      "received_events_url": "https://api.github.com/users/wtlangford/received_events",
      "type": "User",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "6306ac89667cf35f47ddc40aa0630546c57e387f",
        "url": "https://api.github.com/repos/stedolan/jq/commits/6306ac89667cf35f47ddc40aa0630546c57e387f",
        "html_url": "https://github.com/stedolan/jq/commit/6306ac89667cf35f47ddc40aa0630546c57e387f"
      }
    ]
  },
  {
    "sha": "6306ac89667cf35f47ddc40aa0630546c57e387f",
    "node_id": "MDY6Q29tbWl0NTEwMTE0MTo2MzA2YWM4OTY2N2NmMzVmNDdkZGM0MGFhMDYzMDU0NmM1N2UzODdm",
    "commit": {
      "author": {
        "name": "itchyny",
        "email": "itchyny@hatena.ne.jp",
        "date": "2020-05-09T01:39:38Z"
      },
      "committer": {
        "name": "William Langford",
        "email": "wlangfor@gmail.com",
        "date": "2020-05-26T16:30:27Z"
      },
      "message": "Reduce allocation on string multiplication",
      "tree": {
        "sha": "6bbbd60758a976421e85d3a7053ce1be9c27d19b",
        "url": "https://api.github.com/repos/stedolan/jq/git/trees/6bbbd60758a976421e85d3a7053ce1be9c27d19b"
      },
      "url": "https://api.github.com/repos/stedolan/jq/git/commits/6306ac89667cf35f47ddc40aa0630546c57e387f",
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null
      }
    },
    "url": "https://api.github.com/repos/stedolan/jq/commits/6306ac89667cf35f47ddc40aa0630546c57e387f",
    "html_url": "https://github.com/stedolan/jq/commit/6306ac89667cf35f47ddc40aa0630546c57e387f",
    "comments_url": "https://api.github.com/repos/stedolan/jq/commits/6306ac89667cf35f47ddc40aa0630546c57e387f/comments",
    "author": {
      "login": "itchyny",
      "id": 375258,
      "node_id": "MDQ6VXNlcjM3NTI1OA==",
      "avatar_url": "https://avatars2.githubusercontent.com/u/375258?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/itchyny",
      "html_url": "https://github.com/itchyny",
      "followers_url": "https://api.github.com/users/itchyny/followers",
      "following_url": "https://api.github.com/users/itchyny/following{/other_user}",
      "gists_url": "https://api.github.com/users/itchyny/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/itchyny/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/itchyny/subscriptions",
      "organizations_url": "https://api.github.com/users/itchyny/orgs",
      "repos_url": "https://api.github.com/users/itchyny/repos",
      "events_url": "https://api.github.com/users/itchyny/events{/privacy}",
      "received_events_url": "https://api.github.com/users/itchyny/received_events",
      "type": "User",
      "site_admin": false
    },
    "committer": {
      "login": "wtlangford",
      "id": 3422295,
      "node_id": "MDQ6VXNlcjM0MjIyOTU=",
      "avatar_url": "https://avatars2.githubusercontent.com/u/3422295?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/wtlangford",
      "html_url": "https://github.com/wtlangford",
      "followers_url": "https://api.github.com/users/wtlangford/followers",
      "following_url": "https://api.github.com/users/wtlangford/following{/other_user}",
      "gists_url": "https://api.github.com/users/wtlangford/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/wtlangford/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/wtlangford/subscriptions",
      "organizations_url": "https://api.github.com/users/wtlangford/orgs",
      "repos_url": "https://api.github.com/users/wtlangford/repos",
      "events_url": "https://api.github.com/users/wtlangford/events{/privacy}",
      "received_events_url": "https://api.github.com/users/wtlangford/received_events",
      "type": "User",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "9163e09605383a88f6e953d6cb5cc2aebe18c84f",
        "url": "https://api.github.com/repos/stedolan/jq/commits/9163e09605383a88f6e953d6cb5cc2aebe18c84f",
        "html_url": "https://github.com/stedolan/jq/commit/9163e09605383a88f6e953d6cb5cc2aebe18c84f"
      }
    ]
  },
  {
    "sha": "9163e09605383a88f6e953d6cb5cc2aebe18c84f",
    "node_id": "MDY6Q29tbWl0NTEwMTE0MTo5MTYzZTA5NjA1MzgzYTg4ZjZlOTUzZDZjYjVjYzJhZWJlMThjODRm",
    "commit": {
      "author": {
        "name": "itchyny",
        "email": "itchyny@hatena.ne.jp",
        "date": "2020-05-08T05:22:25Z"
      },
      "committer": {
        "name": "William Langford",
        "email": "wlangfor@gmail.com",
        "date": "2020-05-26T16:30:27Z"
      },
      "message": "Fix multiple string multiplication",
      "tree": {
        "sha": "d580cd0089241528805e6ce7dbb89675a66a8ded",
        "url": "https://api.github.com/repos/stedolan/jq/git/trees/d580cd0089241528805e6ce7dbb89675a66a8ded"
      },
      "url": "https://api.github.com/repos/stedolan/jq/git/commits/9163e09605383a88f6e953d6cb5cc2aebe18c84f",
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null
      }
    },
    "url": "https://api.github.com/repos/stedolan/jq/commits/9163e09605383a88f6e953d6cb5cc2aebe18c84f",
    "html_url": "https://github.com/stedolan/jq/commit/9163e09605383a88f6e953d6cb5cc2aebe18c84f",
    "comments_url": "https://api.github.com/repos/stedolan/jq/commits/9163e09605383a88f6e953d6cb5cc2aebe18c84f/comments",
    "author": {
      "login": "itchyny",
      "id": 375258,
      "node_id": "MDQ6VXNlcjM3NTI1OA==",
      "avatar_url": "https://avatars2.githubusercontent.com/u/375258?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/itchyny",
      "html_url": "https://github.com/itchyny",
      "followers_url": "https://api.github.com/users/itchyny/followers",
      "following_url": "https://api.github.com/users/itchyny/following{/other_user}",
      "gists_url": "https://api.github.com/users/itchyny/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/itchyny/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/itchyny/subscriptions",
      "organizations_url": "https://api.github.com/users/itchyny/orgs",
      "repos_url": "https://api.github.com/users/itchyny/repos",
      "events_url": "https://api.github.com/users/itchyny/events{/privacy}",
      "received_events_url": "https://api.github.com/users/itchyny/received_events",
      "type": "User",
      "site_admin": false
    },
    "committer": {
      "login": "wtlangford",
      "id": 3422295,
      "node_id": "MDQ6VXNlcjM0MjIyOTU=",
      "avatar_url": "https://avatars2.githubusercontent.com/u/3422295?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/wtlangford",
      "html_url": "https://github.com/wtlangford",
      "followers_url": "https://api.github.com/users/wtlangford/followers",
      "following_url": "https://api.github.com/users/wtlangford/following{/other_user}",
      "gists_url": "https://api.github.com/users/wtlangford/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/wtlangford/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/wtlangford/subscriptions",
      "organizations_url": "https://api.github.com/users/wtlangford/orgs",
      "repos_url": "https://api.github.com/users/wtlangford/repos",
      "events_url": "https://api.github.com/users/wtlangford/events{/privacy}",
      "received_events_url": "https://api.github.com/users/wtlangford/received_events",
      "type": "User",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83",
        "url": "https://api.github.com/repos/stedolan/jq/commits/15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83",
        "html_url": "https://github.com/stedolan/jq/commit/15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83"
      }
    ]
  },
  {
    "sha": "15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83",
    "node_id": "MDY6Q29tbWl0NTEwMTE0MToxNWZhMWRlYzk5ZjJhNzA5ZGQ2YzNlZTk3OTFhZjRhOWY3MmI0YTgz",
    "commit": {
      "author": {
        "name": "itchyny",
        "email": "itchyny@hatena.ne.jp",
        "date": "2020-05-13T03:09:53Z"
      },
      "committer": {
        "name": "William Langford",
        "email": "wlangfor@gmail.com",
        "date": "2020-05-26T16:29:34Z"
      },
      "message": "Fix error handling in strftime",
      "tree": {
        "sha": "83070d1a9ac4fdcb5648a888d7f0e77362b6ec48",
        "url": "https://api.github.com/repos/stedolan/jq/git/trees/83070d1a9ac4fdcb5648a888d7f0e77362b6ec48"
      },
      "url": "https://api.github.com/repos/stedolan/jq/git/commits/15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83",
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null
      }
    },
    "url": "https://api.github.com/repos/stedolan/jq/commits/15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83",
    "html_url": "https://github.com/stedolan/jq/commit/15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83",
    "comments_url": "https://api.github.com/repos/stedolan/jq/commits/15fa1dec99f2a709dd6c3ee9791af4a9f72b4a83/comments",
    "author": {
      "login": "itchyny",
      "id": 375258,
      "node_id": "MDQ6VXNlcjM3NTI1OA==",
      "avatar_url": "https://avatars2.githubusercontent.com/u/375258?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/itchyny",
      "html_url": "https://github.com/itchyny",
      "followers_url": "https://api.github.com/users/itchyny/followers",
      "following_url": "https://api.github.com/users/itchyny/following{/other_user}",
      "gists_url": "https://api.github.com/users/itchyny/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/itchyny/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/itchyny/subscriptions",
      "organizations_url": "https://api.github.com/users/itchyny/orgs",
      "repos_url": "https://api.github.com/users/itchyny/repos",
      "events_url": "https://api.github.com/users/itchyny/events{/privacy}",
      "received_events_url": "https://api.github.com/users/itchyny/received_events",
      "type": "User",
      "site_admin": false
    },
    "committer": {
      "login": "wtlangford",
      "id": 3422295,
      "node_id": "MDQ6VXNlcjM0MjIyOTU=",
      "avatar_url": "https://avatars2.githubusercontent.com/u/3422295?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/wtlangford",
      "html_url": "https://github.com/wtlangford",
      "followers_url": "https://api.github.com/users/wtlangford/followers",
      "following_url": "https://api.github.com/users/wtlangford/following{/other_user}",
      "gists_url": "https://api.github.com/users/wtlangford/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/wtlangford/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/wtlangford/subscriptions",
      "organizations_url": "https://api.github.com/users/wtlangford/orgs",
      "repos_url": "https://api.github.com/users/wtlangford/repos",
      "events_url": "https://api.github.com/users/wtlangford/events{/privacy}",
      "received_events_url": "https://api.github.com/users/wtlangford/received_events",
      "type": "User",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "ccc79e592cfe1172db5f2def5a24c2f7cfd418bf",
        "url": "https://api.github.com/repos/stedolan/jq/commits/ccc79e592cfe1172db5f2def5a24c2f7cfd418bf",
        "html_url": "https://github.com/stedolan/jq/commit/ccc79e592cfe1172db5f2def5a24c2f7cfd418bf"
      }
    ]
  },
  {
    "sha": "ccc79e592cfe1172db5f2def5a24c2f7cfd418bf",
    "node_id": "MDY6Q29tbWl0NTEwMTE0MTpjY2M3OWU1OTJjZmUxMTcyZGI1ZjJkZWY1YTI0YzJmN2NmZDQxOGJm",
    "commit": {
      "author": {
        "name": "Christopher Degawa",
        "email": "ccom@randomderp.com",
        "date": "2020-03-03T16:23:45Z"
      },
      "committer": {
        "name": "William Langford",
        "email": "wlangfor@gmail.com",
        "date": "2020-03-03T16:50:04Z"
      },
      "message": "Makefile: prepend srcdir to jq.1.prebuilt to fix out of source compilation",
      "tree": {
        "sha": "aee535a3870e6990f8ec6d7e0b9909ea2e7070f8",
        "url": "https://api.github.com/repos/stedolan/jq/git/trees/aee535a3870e6990f8ec6d7e0b9909ea2e7070f8"
      },
      "url": "https://api.github.com/repos/stedolan/jq/git/commits/ccc79e592cfe1172db5f2def5a24c2f7cfd418bf",
      "comment_count": 0,
      "verification": {
        "verified": false,
        "reason": "unsigned",
        "signature": null,
        "payload": null
      }
    },
    "url": "https://api.github.com/repos/stedolan/jq/commits/ccc79e592cfe1172db5f2def5a24c2f7cfd418bf",
    "html_url": "https://github.com/stedolan/jq/commit/ccc79e592cfe1172db5f2def5a24c2f7cfd418bf",
    "comments_url": "https://api.github.com/repos/stedolan/jq/commits/ccc79e592cfe1172db5f2def5a24c2f7cfd418bf/comments",
    "author": {
      "login": "1480c1",
      "id": 8345542,
      "node_id": "MDQ6VXNlcjgzNDU1NDI=",
      "avatar_url": "https://avatars0.githubusercontent.com/u/8345542?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/1480c1",
      "html_url": "https://github.com/1480c1",
      "followers_url": "https://api.github.com/users/1480c1/followers",
      "following_url": "https://api.github.com/users/1480c1/following{/other_user}",
      "gists_url": "https://api.github.com/users/1480c1/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/1480c1/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/1480c1/subscriptions",
      "organizations_url": "https://api.github.com/users/1480c1/orgs",
      "repos_url": "https://api.github.com/users/1480c1/repos",
      "events_url": "https://api.github.com/users/1480c1/events{/privacy}",
      "received_events_url": "https://api.github.com/users/1480c1/received_events",
      "type": "User",
      "site_admin": false
    },
    "committer": {
      "login": "wtlangford",
      "id": 3422295,
      "node_id": "MDQ6VXNlcjM0MjIyOTU=",
      "avatar_url": "https://avatars2.githubusercontent.com/u/3422295?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/wtlangford",
      "html_url": "https://github.com/wtlangford",
      "followers_url": "https://api.github.com/users/wtlangford/followers",
      "following_url": "https://api.github.com/users/wtlangford/following{/other_user}",
      "gists_url": "https://api.github.com/users/wtlangford/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/wtlangford/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/wtlangford/subscriptions",
      "organizations_url": "https://api.github.com/users/wtlangford/orgs",
      "repos_url": "https://api.github.com/users/wtlangford/repos",
      "events_url": "https://api.github.com/users/wtlangford/events{/privacy}",
      "received_events_url": "https://api.github.com/users/wtlangford/received_events",
      "type": "User",
      "site_admin": false
    },
    "parents": [
      {
        "sha": "50a7022ea68fb37faefdcd7a35661df72fbfd655",
        "url": "https://api.github.com/repos/stedolan/jq/commits/50a7022ea68fb37faefdcd7a35661df72fbfd655",
        "html_url": "https://github.com/stedolan/jq/commit/50a7022ea68fb37faefdcd7a35661df72fbfd655"
      }
    ]
  }
]

Output

      1 "Christopher Degawa"
      3 "itchyny"
      1 "Maximilian Roos"

Adding a Downloaded Field to a JSON Object

It can be useful to timestamp a downloaded JSON object. For example, we are pulling from a random endpoint that needs to be cached locally on the web server, and the end user needs to be told when the file was last updated. In this example, the updated_at field is added to an object stored in a file.

TZ=America/New_York
jq --arg updated_at "$(date)" '. + {updated_at: $updated_at}' req.json

Input

{
    "foo_count": 10,
    "next_update": "24 hours"
}

Output

{
  "foo_count": 10,
  "next_update": "24 hours",
  "updated_at": "Fri Jan 22 03:38:25 PM EST 2021"
}

Useful jq Command Flags

--compact-output/-c
Cut down on the white space and the pretty printing. Pretty printing is some what expensive with the excess of bytes it produces. Useful when chaining jq calls.
--unbuffered
Are you reading from a slow source? This sends stuff as soon as it’s ready to the next pipe or output.
--arg name value
jq can use variables in your expression. If you call it with jq --arg foo 123, the value "123" will be bound to $foo in your expressions.
--sort-keys/-S
Sort the fields in each output object by the keys.
--monochrome-output/-M
Do NOT output any color. By default jq will colorize output.

Sources

Appendix

How the jq commits were retrieved?

I used curl and the GitHub api to retrieve them.

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' > jq_repo_commits.json

How To Install

Ubuntu/Debian

sudo apt install jq

Fedora

sudo dnf install jq

Arch

sudo pacman -S jq

This is the end. Thank you for reading or attending.

This post is tagged:

Read more by exploring the posts in the above tags. Otherwise go home.

Hit Counter