Jq Usage: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(11 intermediate revisions by the same user not shown)
Line 127: Line 127:
=Updating a File=
=Updating a File=


test.json has to exist and be a valid JSON file.
<code>test.json</code> has to exist and be a valid JSON file.


jq '.something="something else"' test.json
<syntaxhighlight lang='bash'>
jq '.something="something else"' test.json
</syntaxhighlight>
 
Note that the file is not updated "in-place", <code>jq</code> sends the updated result to stdout.
 
<font color=darkkhaki>The <code>|=</code> operator is used to update a value. What is the difference?</font>


=Array Filters=
=Array Filters=
Line 180: Line 186:


===Last Elements===
===Last Elements===
Last element:
<syntaxhighlight lang='bash'>
jq '.[-1]'
</syntaxhighlight>
Second element from the end:
<syntaxhighlight lang='bash'>
jq '.[-2]'
</syntaxhighlight>


Last 2 elements:
Last 2 elements:
Line 218: Line 234:
The filter does not work on primitives:
The filter does not work on primitives:


echo \"blah\" | jq '.[]'
<syntaxhighlight lang='bash'>
jq: error (at <stdin>:1): Cannot iterate over string ("blah")
echo \"blah\" | jq '.[]'
jq: error (at <stdin>:1): Cannot iterate over string ("blah")
</syntaxhighlight>
 
===Array-to-Iterator Filter applied on a Object value===
 
If an Object value is an Array, then the elements of that Array can be turned into a stream with the Array-to-iterator filter. The key the Array is associated with should be specified:
 
For:
<syntaxhighlight lang='json'>
{
  "items": [
    {
      "color": "red"
    },
    {
      "color": "blue"
    }
  ]
}
</syntaxhighlight>
 
the following expression:
<syntaxhighlight lang='bash'>
jq ".items[]"
</syntaxhighlight>
produces the following stream:
<syntaxhighlight lang='json'>
{
  "color": "red"
}
{
  "color": "blue"
}
</syntaxhighlight>


==Stream to Array Filter==
==Stream to Array Filter==
Line 232: Line 282:
<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
... | jq '[.[]]'
... | jq '[.[]]'
</syntaxhighlight>
==Array Length==
<syntaxhighlight lang='bash'>
cat ... | jq -r '.items | length'
</syntaxhighlight>
</syntaxhighlight>


Line 256: Line 311:


Note that ${select_value} is enclosed in double quotes, and the resulted value is "introduced" in jq enclosed in single quotes.
Note that ${select_value} is enclosed in double quotes, and the resulted value is "introduced" in jq enclosed in single quotes.
=Generating JSON with jq=
{{External|https://spin.atomicobject.com/2021/06/08/jq-creating-updating-json/}}
<syntaxhighlight lang='bash'>
jq --null-input --arg arg1 "A" --arg arg2 "B" '{"a": $arg1, "b": $arg2}'
</syntaxhighlight>
produces:
<syntaxhighlight lang='json'>
{
  "a": "A",
  "b": "B"
}
</syntaxhighlight>
<font color=darkkhaki>How to use jq to filter a larger JSON and only select a few fields to be emitted as a smaller JSON, in a pipeline?</font>

Latest revision as of 18:05, 28 April 2022

External

Internal

Command Line Options

-r

Output raw strings, not JSON content. This is very useful when jq is used as part of shell pipelines and the results feed other shell commands.

Universal Filters

Identity Filter .

select() Filter

select(boolean-expression)

The select() filter lets its input pass through unchanged if the boolean expression evaluates to true, and produces no output (swallows the input) otherwise. The result consists of the same kind of pipeline items that are the input of select().

For more details about expressions, see:

jq Concepts - Expressions

Example:

jq '... | select(.Name == "something")'

Spaces are optional:

jq '... | select(.Name=="something")'

.Name extracts the value of a map key value pair whose key is "Name" and the filter lets pass only those maps for which the "Name" key is equals to "something". Note that "==" should be used, it is semantically different from "=".

Also see Double Quotes in select() Expressions scripting tips.

select() Expressions

Equality

select(.name == "something")

String Matching

select(.name | contains("something"))
select(.name | startswith("somet"))
select(.name | endswith("ething"))

Functions

https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions

Map Filters

Object Identifier-Index Filter .key

.key

The filter expects a map and produces the value associated with the key given as argument of the filter, or null if there is no such object. The value, if exists, can be a primitive, a map or an array.

The key is case sensitive.

cat example.json | jq '.color'

This syntax only works for "identifier-like" keys: keys that are all made of alphanumeric characters and underscore, and which do not start with a digit.

The .key syntax is actually an alias for the more generic syntax:

.["key"]

If the key contains special characters, the .key alias cannot be used, and the complete .["key"] syntax should be used:

jq '.["complex::key"]'

For:

{
  "complex::color": "something"
}
jq '.complex::color'

will fail:

jq: error: syntax error, unexpected ':', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.complex::color

while

jq '.["complex::color"]' 

will work.

This syntax is particularly useful if the key contains dashes:

.["my-color"]

Display Two or More Map Keys

jq '... | .name, .color'

Concatenate Two or More Map Keys

With a dot:

jq '... | (.name + "." + .color)'

With a space:

jq '... | (.name + " " + .color)'

Updating a File

test.json has to exist and be a valid JSON file.

jq '.something="something else"' test.json

Note that the file is not updated "in-place", jq sends the updated result to stdout.

The |= operator is used to update a value. What is the difference?

Array Filters

Array Index Filter .[index]

.[zero-based-index]

selects the corresponding array element, or null if the index does not designate an element that exists.

For:

[
  "blue",
  "red",
  "green"
]
jq '.[0]'

return "blue"

while

jq '.[5]'

returns null.

If the input is not an array, the filter fails:

echo \"blah\" | jq '.[]'
jq: error (at <stdin>:1): Cannot iterate over string ("blah")

Array Slices Filters

Index Range Filter .[index:index]

jq '.[4:6]'

First Elements

First 4 elements:

jq '.[:4]'

Last Elements

Last element:

jq '.[-1]'

Second element from the end:

jq '.[-2]'

Last 2 elements:

jq '.[-2:]'

Before Last Element

jq '.[-2]'

Array-to-Iterator Filter []

The filter expects and array and produces the array's elements as an iteration.

echo "[\"red\", \"blue\"]" | jq '.[]'

has the same result as:

echo "\"red\"" "\"blue\"" | jq '.'

Interestingly enough, and somewhat counterintuitively, the filter also works on maps. The effect is equivalent with Java's Map#values().

echo "{\"a\": 1, \"b\": 2}" | jq '.[]'

produces:

1 
2

The filter does not work on primitives:

echo \"blah\" | jq '.[]'
jq: error (at <stdin>:1): Cannot iterate over string ("blah")

Array-to-Iterator Filter applied on a Object value

If an Object value is an Array, then the elements of that Array can be turned into a stream with the Array-to-iterator filter. The key the Array is associated with should be specified:

For:

{
  "items": [
    {
      "color": "red"
    },
    {
      "color": "blue"
    }
  ]
}

the following expression:

jq ".items[]"

produces the following stream:

{
  "color": "red"
}
{
   "color": "blue"
}

Stream to Array Filter

A stream can collected into a single array by wrapping the underlying filter in square brackets.

... | jq '[ .[] | select(.color=="red") ]'

A simple no-op transformation from array to stream and back to array is:

... | jq '[.[]]'

Array Length

cat ... | jq -r '.items | length'

Use Cases

Select the Array Element which is a Map and Contains a Specific Key

... | jq -r '.clusters[] | select(.name | contains("docker")) | .cluster.server'

Selects the array element whose "name" key contains "docker" and displays the .cluster.server path of the selected map.

jq Scripting Tips

Double Quotes in select() Expressions

Use a shell variable to hold the value we want to select against. Node that the value may contain spaces.

local select_value="some value that may or may not contain spaces"
local jq_result
jq_result=$(${command_that_generates_json} | jq 'select(.keyA=="'"${select_value}"'"')

Note that ${select_value} is enclosed in double quotes, and the resulted value is "introduced" in jq enclosed in single quotes.

Generating JSON with jq

https://spin.atomicobject.com/2021/06/08/jq-creating-updating-json/
jq --null-input --arg arg1 "A" --arg arg2 "B" '{"a": $arg1, "b": $arg2}'

produces:

{
  "a": "A",
  "b": "B"
}

How to use jq to filter a larger JSON and only select a few fields to be emitted as a smaller JSON, in a pipeline?