from ZqPosts

examples use zq version 1.16.0.

A subset of zq operators are Implied Operators, which means “Zed allows certain operator names to be optionally omitted when they can be inferred from context.” The operators are evaluated in this order:

search expressionsearchfoosearch foo
boolean expressionwherea >= 1where a >= 1
field assignmentputa:=x+1,b:=y-1put a:=x+1,b:=y-1
aggregationsummarizecount()summarize count()
expressionyield{a:x+1,b:y-1}yield {a:x+1,b:y-1}

Note: The -C flag can be passed to zq to output the parsed query with explicit operators.

In many contexts this is really helpful, but as I've been learning zq, it's been confusing at times.

I was experimenting with the example data found in this Brim Data article and put this together this query:

❯ zq 'over docs
| has(author_name)
| grep(/tuta/, author_name)
| yield author_key' openlibrary.json

Browsing the docs for some functions, I decided to try out lower function, and as is my wont given my jq experience, piped our previous output to lower. But this didn't work, and I got no output:

❯ zq 'over docs
| has(author_name)
| grep(/tuta/, author_name)
| yield author_key | over this | lower' openlibrary.json

Remembering what I learned in ZqPipeCharacter, I realized I've made an error in how to call the function, and fixed it accordingly:

❯ zq 'over docs
| has(author_name)
| grep(/tuta/, author_name)
| yield author_key | over this | lower(this)' openlibrary.json

I'm aware -- heh -- that I wrote has and grep correctly. But, lower didn't seem to need any arguments, and I just fell into a habit.

But I am still curious. Why didn't I get an error or something letting me know I wasn't using lower properly? If I make a mistake by passing a non-string into it, I'll get an error:

❯ zq 'lower(1)'
error({message:"lower: string arg required",on:1})

So, why no error here?

❯ zq 'yield "HEY" | lower'

Our friend the -C flag has the answer:

❯ zq -C 'yield "HEY" | lower'
yield "HEY"
| search lower

Ahhhhhh. The real reason this returns nothing, not even an error, is search is now the implied operator for the term ‘lower’! It's not parsed as a built-in function.

While this example is obviously contrived, it does highlight a frequent experience for me while learning zq. Remember, if you make a change, and get no output, an Implied Operator is probably in play.

from ZqPosts

examples use zq version 1.16.0.

The pipe character appears quite frequently in both jq and zq. Here's an example of an
equivalent query in both:

jq -c '[.docs[]
| {title, author_name: .author_name[0], publish_year: .publish_year[0]}
| select(.author_name!=null and .publish_year!=null)
| group_by(.author_name)
| [.[] | {author_name: .[0].author_name, count: . | length}]
| sort_by(.count) | reverse | limit(3;.[])' openlibrary.json

zq -j 'over docs
| {title, author_name: author_name[0], publish_year: publish_year[0]}
| has(author_name) and has(publish_year)
| count() by author_name | sort -r count | head 3' openlibrary.json

(Examples taken from this Brim Data article comparing zq and jq).

In jq every program is a series of filters separated by the pipe operator. There are a few places the pipe operator cannot be used, but it's fairly ubiquitous.

zq has a more elaborate and structured syntax. At its highest level, one or more dataflow operators are joined together in a sequence with the pipe character (sequence overview).
operator | operator | ...
But within operators, syntax varies and the pipe character isn't used. Field references and expressions are common, and most functions receive arguments passed in parentheses. Function outputs can only be passed to other functions as nested calls, not via the pipe character.

In this example, the who value is replaced with “me” whenever it's “chrismo”:

> zq -j '[{who:"bob"}, {who:"chrismo"}]
| over this
| put who:=replace(who, "chrismo", "me")'

put is an operation that sets a field to an expression. replace is a function taking three string arguments.

If not all records have a who field, we can use the coalesce function to return an empty string if who is missing, but we cannot use the pipe character like we could in jq between two functions:

❯ zq -j '[{}, {who:"bob"}, {who:"chrismo"}]
| over this
| put who:=((coalesce(who, "") | replace(who, "chrismo", "me"))'

zq: error parsing Zed at line 3, column 39:
| put who:=((coalesce(who, "") | replace(who, "chrismo", "me"))
=== ^ ===

The syntax for put is
[put] <field>:=<expr>
and the pipe character is not valid in an expression. This can be accomplished by nesting the function calls:

❯ zq -j '[{}, {who:"bob"}, {who:"chrismo"}]
| over this
| put who:=replace(coalesce(who, ""), "chrismo", "me")'

Over the past couple of years in my ops work, I've built up a fair amount of jq code. Recently, I've been checking out zq and evaluating it as a jq replacement. The Brim folks have a wonderful zq tutorial that helps introduce zq to folks familiar with jq. Big shout-out to Phil Rzewski from the Brim team, who's been incredibly helpful, both in their support Slack and in the GitHub repo, fielding my questions patiently and thoughtfully.

As I've been learning zq in more detail, while it looks similar on the surface, it's pretty different under the hood and I've been tripped up by a few things that I've written up here. Hopefully they'll be helpful to someone else, too.

ZqPipeCharacterYou can't use | everywhere, esp. not in or between Expressions. Only in-between Dataflow Operators.
ZqImpliedOperatorsDid you make a change and now there's no output? It's probably the search implied operator hiding the fact that you tried to pipe something to a function like you would in jq, but that's not how it works. See previous post.

(So, with these hurdles, why bother? There's lots to love about zq: simple search; easier querying than jq (despite some learning curve); more performant; multiple input & output formats: from csv to Parquet and more; flexibly discover, massage, and query unstructured data alongside typed records with the power of relational joins in a file system lake; and more. Stay tuned for additional posts, but until then, take some time to go read over their excellent zq tutorial.)

Inspired by LawsOfUX and this tweet by @viktorcessan, a collection of Laws all managers should know. Some are focused on technology work, but some apply broadly. (As I continued to assemble the list, the categorization of it all is a bit sprawly, but, let's live a little).


These are great. Some of my favs:

“The value of a company is the sum of all problems solved.”

The last season of Startup, Gimlet's first podcast, discusses their acquisition by Spotify, and it has some interesting behind-the-scenes stuff (maybe not as much as I'd like, but ...)

One of the ongoing themes is the tension between Alex and Matt, the co-founders of Gimlet. Alex cares more about the quality of the product and Matt cares more about the business needs, and Gimlet was struggling, even in the quarter preceding the acquisition. Neither of them were handling the conflict very well.

In this episode, Thanksgiving in Stockholm, Alex is talking with Daniel Ek, CEO of Spotify, after the acquisition has happened to reflect on prior conversations they'd had before.

[36:28] Daniel Ek, Spotify co-founder: It's actually my co-founder's [Martin Lorentzon] saying. He said this thing, I'm not even sure he was aware that he coined what I think is an iconic quote. He said, “the value of a company is the sum of all problems solved.” Even to this day, it's one of those things that I think about. You may think about all the things you guys went through as all the issues that you went through, but you solved them, one by one. And I think the most important thing that you got right is the integrity of the programming and the shows that you make. At the end of the day, that is the value that you're bringing to that and bringing to consumers and it really served you well in the end.

Then Alex sums up his thoughts in a way I don't quite agree with.

[37:20] Alex Blumberg: In other words, Matt's and mine constant fighting had produced something valuable, the fighting itself in fact was the thing that made it better. If I hadn't cared about what I cared about and he hadn't cared about what he cared about and we hadn't each cared enough to fight with each other, the company we built, it wouldn't have worked as well.

It's the caring that matters, not the fighting. It's possible to combine caring with good conflict resolution skills and minimize the fighting. Then things are even better.
In his book, Slack, Tom DeMarco writes:

Slack is a prescription for building a capacity to change into the modern enterprise. It looks into the heart of the efficiency-flexibility quandary: The more efficient you get, the harder it is to change. The book shows managers how to make their organizations slightly less efficient but enormously more effective. It coaches them on the introduction of slack, the missing ingredient required for all change. It counsels a thoughtful use of slack instead of the mindless obsession with elimination of all slack in the interests of efficiency.

The other day on the job one of the devs on my team posted a PR for code review with a comment like: “This removes X from the codebase. It's not terribly important, but it's been bugging for a long time and I'm glad to be rid of it.”

I realized that the presence of such PRs is an interesting metric to measure the amount of Slack of my team.

If they're too efficient in knocking out cards from stakeholders, I probably won't see any work like this.

Willem van Bergen, in the devproductivity.io Slack, mentioned a talk given at the @Scale conference of September 2017 that shared an internal study done by Google around the productivity costs of their build speeds. Taking aggregate data from tracks of how individual engineers are spending their time, they arrived at this graph:

The speaker Collin Winter had this to say about it (starting 27:44 into the video).
This is a graph from a report that we just published internally a few weeks ago that looks at, for builds of a different duration, how much time are people spending in each of these different cognitive modes. What we can see here is slow builds, builds lasting more than an hour, actually aren't taking that much time in aggregate. Likewise, builds that are really fast, well, they’re really fast, and they’re also not taking that much time in aggregate.

But we can see here that builds in the 2 to 5 minute range and the 5 to 10 minute range, those are actually pretty expensive. … That red bar is time where, as far as we can tell, people were just sitting there twiddling their thumbs staring at the terminal. … We can see time where we simply don’t know what they were doing [yellow].

It looks like the 2 to 10 minutes range is where we should spend our optimization effort and we can use that to focus our efforts more intelligently than a scattershot approach.

I'm of two minds here. Part of me is really interested in this data, but I'm also doubtful. They must be making some possibly large presumptions when interpreting the data of what counts as “untracked” vs. what's “waiting”. I'm also curious if this data is collected from their entire engineer population, or just volunteers? Do they know they're being tracked? How do they deal with the Heisenberg effect?

If the data is trustworthy, and I have a 15 minute build, will I make things worse by getting it into the 2-10 minute range? And how large does my engineering team need to be before any of this matters?


Read at medium.com

I've had the privilege of working remote for almost 5 years here at LivingSocial. Remote work still gets big time plugs, but it seems many places, including young, hip startups don't buy it, that co-location is an important attribute for a team.

And I get it. Co-location was a big deal in the Agile movement to help foster communication. But having worked remotely for 5, managing 3 teams of almost 100% remote employees, why don't I care?

I think the co-location agile pushes has much more to do with communication than proximity. Proximity can greatly drive down communication costs, but these days with Slack and Google Hangouts (to name but a couple), communication can be had without proximity.

Want to build a great team? Google studied this for a long time, and on a recent Freakonomics podcast (How to Be More Productive), Stephen Dubner interviewed Laszlo Bock, SVP of People Operations at Google about their Project Aristotle.

After analyzing data from 200 teams across Google of all varieties, they derived five “norms” the best teams at Google had. Things that weren't on the list?

... in the academic research it says consensus-driven decision-making is often better than top-down direction. And academic research says workload matters a lot. Having teams in the same location. We actually found none of those things were in the top five of what mattered in terms of effectiveness for teams. (around the 24 minute mark)

What did top the list?
the most important attribute of a high-performing team is not who leads it or who’s on it or how many people or where it is. It's psychological safety.

Charles Duhigg, author of The Power of Habit elaborates:
Which means that everyone at the table feels like they have the opportunity to speak up, and they all feel like each other is actually listening to them, as demonstrated by the fact that their teammates are sensitive to nonverbal cues.

Bock continues:
We ask if the team members feel that they can fail openly or do they feel that they are going to be shunned by failing? We ask, do they feel as if other team members are supporting or undermining them?

Or, as I would say it, they're on a team that doesn't shame them. See BeyondImpostorSyndrome for more.

The four other norms Bock identifies are Dependability, Structure and Clarity, Meaning, and Impact. He also tacks on the importance of actually doing 1:1s, and making sure everyone feels included, don't let the dominators dominate.

Which I love.

But I love umpteen times more the first point.

Low shame teams beat out other factors. So does low shame anything.

If you're outsourcing your soft skills to the fringe of your organization (or congregation, or marriage, or friendship ...), you're prolly not doing it right. You're probably also bored and/or frustrated out of your mind.
Read at medium.com

From my little corner of the internet, Impostor Syndrome seems to be getting more attention these days, which is cool. But there’s a bigger world underlying it. Camp out in the mines of Impostor Syndrome and you may miss out on what lies beneath and the resources we have to combat it.

“It” … is shame.

Shame is about my identity. We typically talk of shame in false views of self (“I should be better than I am/I’m unlovable”), though an accurate view of self I’d label as “healthy shame” or humility (“We all make mistakes/I need help”).

Impostor Syndrome is rooted in shame. It speaks to specific instances of identity struggle, usually involving a sense of deception or fraud about our work and accomplishments.

But the shame underneath in each of us can play out in many other ways.

The refusal to pour out my heart’s frustration over making a grocery list with my significant other because it’s just an f’ing grocery list and I don’t want to feel like I’m being disrespected over something so insignificant.

The apology I interrupt myself with to make sure no one has an adverse reaction to my opinions.

The verbal shove to make my friend think twice about asking me that question again.

The give-up on taking care of myself because there’s nothing worth saving here anymore.

The king-of-the-hill race to drop knowledge so that no possible conflict can occur because I allowed someone else to muddy the playing field with something confusing or imprecise.

The faux-patient submission to an unyielding relationship I tuck deep down in my purse of resentments.

The refusal to speak up to an act I’ve been guilty of myself because who am I to out myself as a hypocrite.

The spoken snub to try and silence the proof that I’m not really comfortable with how others express themselves.

The interruption of my spouse’s story to make sure they get it right so others won’t think poorly of me because of my ignorant other.

The justification where I draw the line so help me god you can have everything else, but not this.

The turned over phone, the ignored text, the deleted email.

Years ago I knew of a man in his fifties or sixties, a successful business man, who had a secret: he couldn’t read. He’d worked around it his whole career, had figured out how to have others read for him so he would never have to.

As part of his recovery he decided to remedy this. The tutor started with some simple reading samples to get their bearings and by the time they were done, it was determined he could actually read at about a 7th grade level.

His shame around reading had skewed his beliefs so much over the years, that he really didn’t think he could read at all.

The topic of depression is pertinent as well here. While I believe everyone copes with shame, those who receive their helping of shame swimming in a gravy of depression have it rougher.

The physiological effects of depression are akin to circus mirror glasses on top of the blinders of shame. I’m a bit (more) out of my depth to talk about depression, other than to say my spouse and I have struggled through hers our entire marriage. Successfully, I might add, on the whole. But it’s no small beast.

How, then, do we cope?

For many, anger is the defender of our secret shame. Look behind it, what’s it so desperate to guard?

For others, withdrawal and fear are aiming to protect. Peel it back, what’s beneath it?

Look for hurt. For loneliness. And look for false beliefs. Toxic shame arises from lies. Tie yourself to something right.

Stay connected. Shame removes options, isolates. Do not hesitate in getting help from someone trained in dealing with it. And when you hesitate and the shame wins for a day, get some help again.

Shame runs deep. But we can learn to recognize it, in all its myriad ways. If Impostor Syndrome has been your introduction, keep digging.


Brené Brown has some wonderful resources on shame, see her site here: http://brenebrown.com. These TED talks of hers are good introductions: Listening to Shame and The Power of Vulnerability.

Greg Bauges has a great blog on the topic of Depression with a focus on the Software Developer community, and runs a support site at http://devpressed.com/

A couple of recent tweet conversations prompted me to get this out of my head and into words. Thx to Glenn Vanderburg, @gregvaughn, kerri miller and others for, at the very least, allowing me to invade on their tweet-turf. Thanks also to folks who additionally reviewed this article for me: Tara, Adam Keys, Morgan Whaley, @erniemiller, and Tony Pitale.

I gave a little lightning talk on TechnicalIntimidation at RailsConf 2013 which touches on these topics, video and lots o’ other links there. If yer a techie, give it a spin.


Tanmay Vora sketchnotes an article from Harvard Business Review called “Why Organizations Don't Learn”: