Grab bag of random thoughts. @ Irrational Exuberance

Hi folks,

This is the weekly digest for my blog, Irrational Exuberance. Reach out with thoughts on Twitter at @lethain, or reply to this email.

Posts from this week:

- Grab bag of random thoughts.
- Interviewing engineering executives.
- Poking around OpenAI.

Grab bag of random thoughts.

A bit over a week from now, I’ll be joining a company to start a new role, and I wanted to ramble a bit to braindump the numerous loose threads in my head as I transitioned from Calm to the past month of full-time writing, and then into this new role. This isn’t really a job announcement post per se, as I won’t share any details about the job itself until I’ve officially started. Instead, this is a snapshot of what’s top of mind for me, particularly driven by the dozens of discussions with friends and colleagues as I thought about what’s next for me over the past few months.

Job search

My last day at Calm was in early March, and I was planning to take three to six months off before starting my next job search. My initial goal was to specifically avoid talking to anyone about open roles until I was ready to start, because there is a certain momentum to executive searches that is hard to avoid: if you’re an eligible candidate who interviews reasonably well, once you start talking to VCs and recruiters, you’re going to end up placed somewhere. If you fight against that momentum for too long, you’ll eventually annoy the folks trying to place you, turning supporters into neutral parties at best.

If you disagree that there’s some risk to resisting executive search momentum, a quick aside for you. At this point in my career, I am selected as a backchannel reference for a number of folks. If you’ve worked with me, and you interview for an executive role at companies backed by a certain handful of venture firms or staffed with executives whose network overlaps with mine, it’s fairly likely I will get a call about you. I try extremely hard to center the positive for everyone I’ve worked with, even in the rare case that I didn’t love working together with them, but it highlights something important: even if you’re trying to spare me from performing too many references on your behalf, if you run a long executive search and we’re worked together, then I am getting a lot of pings. It’s not that I’m special, this is how the executive recruiting ecosystem works. If you run a long search–even if you don’t personally pull many people in–you will tire out your network. The VCs and recruiters will get tired too.

Back to my own search: Talking to friends, a recurring theme was the lack of exceptionally good executive openings in 2023 relative to searches in prior years, especially relative to the 2020-2021 era. There were still many open executives roles, but many were in deeply challenged businesses or very early businesses (e.g. Series A). There were still some executive roles in thriving businesses, but there simply weren’t very many of them. This made me rethink my planned three to six months break before talking with recruiters.

I ended up deciding to talk with companies about roles that I felt confident I would accept, barring uncovering major flags in the interview process, and where I could accept an offer without experiencing FOMO about the other companies I didn’t speak with. In other words, set a high bar,, and let it take as long as it takes to find an opportunity. This worked a bit faster than anticipated, and I’m quite excited about the role and company I’m starting next Monday. I’m confident that I wouldn’t have found a better opportunity for me, even if I’d spent the next six months talking with companies, and equally confident I would hae regretted saying no.

Leaving Calm & how I thought about my next role

When I was thinking through my decision to leave Stripe (the role I left to join Calm), I wrote A forty-year career, which describes each role as a mix of profit, people, prestige, learning and pace. This framework continues to resonate with me. The nuance I’d add to it, as I’ve gotten better at managing my own energy, is that pace is often more of a ratio between energizing and draining work rather than an absolute speed.

My time at Calm was very rewarding, and the hardest part was leaving the leaders, peers and team that I got to work. What solidified my decision to leave was my belief that my rate of contribution to the business and my rate of learning were both tapering off. If I could go back in time to 2020 and pick any available job, I’d pick Calm again, because I have learned so much over the past three years, but it also felt like the right moment for me to move on to something new.

As I thought about what I would do next, finding an opportunity with a significant rate of both contribution and learning was the foremost criteria, and I believe that four years from now I’ll have learned just as much as I learned over the past four. This isn’t a precise science, but if I keep learning enough to write a new book every four years, that will be good evidence that my learning at work remains at the right pace.

Sabbatical

As mentioned above, I left Calm planning to take a three to six month sabbatical, aiming to finish writing my next book. I’d say that I’m about 60% done writing the initial draft, and will slow down a bit as I start my new job. I remain optimistic that I’ll finish it over the next four months or so, which was the timeline I established with O’Reilly, admittedly under the assumption that I wouldn’t be returning to work quite so early.

The book’s first chapters will be up for early release very soon, at which point I’ll write more about that project. It just feels a wee bit premature to write about something I can’t link to. (For the record, this upcoming book is not going to be Infrastructure Engineer, which has fallen a bit down my priority list. My hope at this point is to pick it back up in late 2024, I don’t think I’ll see much progress on it until then.)

A number of people have asked me for sabbatical advice, and I’ve established absolutely no credibility in terms of doing the sabbatical I intended to, but I’ll still share what was important to me when thinking about the sabbatical and then implementing it:

• Particularly in 2023, you should have a financial position to support a six month job search. That is in addition to the resources to pay for the sabbatical itself. The job market out there is just very strange right now. I’m seeing some folks find new jobs very quickly, and others struggle to find new jobs for months.

• Figure out whether others in your family will or won’t have time to do it with you, and adjust for that. My wife was continuing to work, so this meant I had more time to myself, but also that I needed to keep to my existing child care and family schedules. Life is long and complex, don’t try to make others take time when you’re taking time, let people live!

• Give yourself a few weeks, maybe two, to do absolutely nothing productive. Don’t have a schedule, don’t have meetings, don’t take calls, don’t catch up with friends. Just relax. Maybe go somewhere else.

• Don’t get anxious and start scheduling lots of meetings. You can easily spend all of your time taking meetings. You could just keep working and get paid to take meetings if that’s what you want.

• Have a clear set of goals to focus on. I kept a strict writing schedule during my time, which helped me feel productive. I occasionally “cheated” on the schedule sometimes to take care of my son, work on a fun project, meet up with a friend or whatever, but I mostly kept to the schedule. It was not a brutal schedule by any means. It was peaceful, but focused.

  I also had a running goal, to get back up to an 8-mile weekly “long run”, which I hit last week as the deadline started to get uncomfortably close. As someone who’d fallen into three mile maintenance runs for the past decade, it was a good to remember that running further is mostly about not stopping when it gets uncomfortable.

• Remember that rest is about resting. There have been days when I didn’t get stuff done, or even get stuff started, and I just gave myself permission to rest. When working, I would have pushed through and gotten it all done, but my goal for this time was essentially to heal and recuperate from the last ~15 years of working full-time, so I didn’t ever push. If it felt like I needed to push productivity harder, I just relaxed the constraints instead. I wanted to be ready to utilize my higher gears when I returned to work, rather than to exhaust them during my break.

If I could do it again, I think I’d do it the same way! Life comes at you fast, and rarely according to plan, but I don’t think I could have planned it more with better results.

Who’s successful anyway?

Recently when I chatted with a friend about our careers, we got onto what I think is an interesting topic, which is the inscrutability of success. There are people in the industry who appear extremely successful, but who are not. There’s two different dimensions to consider here.

1. First, there are individuals who you’d assume are financially very successful but who are working out of necessity despite working very high prestige roles (e.g. head of engineering at a trendy company for years, but still working out of necessity 25-30 years into their career).
2. Second, there are individuals who are widely considered very successful by folks without detailed context of their work, but who are considered very unsuccessful by those with concrete awareness of their work (e.g. well-known on conference circuit but considered a poor performer in role by former colleagues).

As I’ve gotten deeper into the industry and a bit better networked, it’s less common that I’m shocked by someone’s performance–usually I know someone who knows someone who worked with them–but I remain constantly surprised at the industry’s inconsistent financial outcomes. I don’t have any concrete lesson to share, but I think it’s interesting to note that I’m still frequently surprised at how little financial success is extended to some nominally extremely successful people in the industry. Conversely, there are also many, many folks who are financially successful, without a clear correlation between that success and their various contributions. Downturns make this phenomenon even more extreme, with many excellent folks failing to “cash out” of their work, largely due to timing constraints out of their control.

Culturally, there’s a strong pull towards using financial success as a sort of moral compass, but it’s my experience that luck plays far too high a role. Even if you think you’re appropriately discounting the role of luck, I suspect you are still underestimating its impact.

Executives without much range

There are a number of executives out there who are very good at some things, but lack the flexibility to operate in varied environments. Sometimes this is because they are stubborn, and have a specific working style that they insist on following independently of the company. Another major contributor, in my experience, is executives who lack experience working in middle management.

Middle management is, of course, something that people often view as fake work, but it’s the critical work of translating an executive’s stated plan into a series of real plans that the company can actually implement. Executives who don’t understand this are doomed to create systems and processes that impede organizational execution, often screwing things up while claiming to improve organizational execution. Based on my experience, I don’t think you can be an excellent executive at a scaled (or scaling) company without middle management experience.

There are lots of details here, but the two biggest ones I’d mention are: I find that executives without middle management experience often rely on trust rather than inspection, and then generally a lack of understanding of how to design useful processes (usually viewing process as a one-off thing rather than an ongoing evolution).

LLMs

I had fun playing around with the OpenAI API this past week. I summarized my view of this technology shift as, “LLMs are showing significant promise at mediocre solutions to very general problems.” This captures my perspective pretty fully, but there’s one other nuance I’ll throw in for the moment: chat hasn’t yet proven itself as a durable paradigm for serious interactions. It’s a good paradigm for broad discussions that can tolerate some loss of meaning, e.g. initial communication about intent to purchase a SaaS product or customer support for a user who’s ordered two specific items, but not a good interface for areas where meaning is fundamental, e.g. here is how you should write a legal document. This is similar to the issues Alexa and Siri as meaningful product lines, when they’re largely stuck playing music, setting timers, and checking the weather.

The issues with chat absolutely don’t mean that LLMs won’t find a sweet spot, I’m just a bit skeptical that chat’s the sweet spot. Most valuable applications are trying to do valuable work where correctness matters, and I don’t see this technology as likely to upend those sorts of interactions. I’m not in the rooms making these decisions, but my personal hypothesis is that chat is a very smart use case where you can show significant value while minimizing costs (e.g. imagine the price of evaluating LLM responses for each of your ten million users every night as opposed to only incurring those costs when a human directly asks for one).

That said, it’s clear that LLMs are going to absolutely break the current paradigms of writing, editing and evaluating written work. Publishers are already getting overwhelmed with generated submissions, and it will be instructive to see where that ends up. I’m personally a bit concerned about it, not because I think it’ll impact me in a major way, but because I think it has the potential to make it much more difficult for new writers to distinguish themselves. That said, it’s exciting for the world to keep changing, and it was never going to statically remain the same to prioritize my comfort over technology’s drive to innovate.

That’s all folks

OK, I’ve gotten my random thoughts out of my head for now. I’ve been publishing on pretty much a weekly cadence for the past few months, and suspect my cadence will slip a bit as I ramp up in my new role, but hopefully not too much!

Interviewing engineering executives.

Earlier I wrote about getting hired as an Engineering executive, and it’s perhaps even more important to discuss the opposite question: how should you interview and evaluate Engineering executives? As an Engineering executive, you may not directly run one of these searches, but you’ll likely be asked for advice about how to run them, and may be asked to design the process to hire your successor.

The key topics I want to explore are:

• Avoiding the unicorn search
• How interviewing executives goes wrong
• Structuring your evaluation process
• Focusing on four areas to evaluate engineering executives

These topics will prepare you to conduct an engineering executive search that culminates in hiring a leader that can support your company today, and will help you avoid bogging the executive team down in a multi-month process that is ambivalent about potential candidates.

Avoiding the unicorn search

While most companies struggle to evaluate incoming executives effectively, there’s a second category that are effective at evaluating executives but nonetheless fail to hire effectively because they want a rare intersection of skills. For example, I once saw an engineering executive search that wanted someone with experience leading a large Engineering function, with deep go-to-market and Product experience, deep domain exposure to a narrow infrastructure engineering domain, cultural alignment with consensus-based decision making, and a sufficiently strong motor to skip consensus-making to accelerate company processes.

There is always some candidate who fits any mold you define, but hiring them gets very challenging. Identifying and hiring them is even harder once you acknowledge the breadth of the error bars inherent to this process. Worse, you usually can’t go back to reactivate a candidate after you’ve passed on them, so even if you later realize an earlier candidate was excellent, that prospect will have already passed you by. If you run a narrow search for too long, by the time you open the search up, you may have already rejected your best potential candidates.

My biggest advice for avoiding the unicorn search is to get the search’s sponsor, generally the CEO for an Engineering executive, to spend time before the formal search talking to seasoned Engineering executives to assess the profile. These don’t need to be folks you could hire, and your goal isn’t really to hire them, rather it’s to listen to their feedback on the profile you want to hire and what could make your opportunity sufficiently compelling that a qualified candidate would accept it. This will take a few weeks, but will save you months of time in the long run.

How interviewing executives goes wrong

Typically, the hiring loop for a software engineering role starts out messy and is slowly refined into an effective hiring process as you make more successful hires. This post-hire calibration process is particularly important to reduce the number of false positives and false negatives in your interviewers feedback. Executive searches only make one hire, while evaluating for a very broad role, which makes these loops even harder to calibrate.

These loops are further made challenging because there’s rarely someone wholly qualified to assess the potential engineering executives, who are being hired to be the most senior technical leader at the company, but despite that gap you’ll have many folks with strong opinions about who should be hired. Combining these messy incentives and challenges, most companies bouncing between two interview formats:

• Vibes and backchannel: hiring is heavily weighted on a small number of discussions along with backchannel references who provide input on candidates previous work. These processes generate very little direct signal, which means that internal colleagues often don’t feel bought into the hires. Even the candidates themselves may not feel particularly evaluated either, which may cause them to decline the offer.
• Broadchurch: hiring incorporates a wide range of internal interviewers. This might include an interview with the CEO, Product, Design, People, Finance, Marketing, Sales, along with four or five folks from Engineering. Introducing this many interviewers, many who will be unpracticed at interviewing for this role and may rely on an entirely informal interview, will generate numerous false negatives, often anchoring evaluation on the perspectives of folks without clear evaluation criteria and limited exposure to the role you’re hiring.

My experience is that neither of these are particularly effective at evaluating candidates, with the former accepting too many candidates and the latter rejecting candidates randomly. Further, these experiences leave the candidates themselves skeptical of the company’s decision making.

Structure for evaluating executives

Fortunately, it’s straightforward to design a reasonable process that’s comparable to most engineering executive evaluations and avoids some of the common missteps. That’s not to say that this process is perfect, but rather than most obvious changes introduce at least as many problems as they solve. I recommend starting with:

• Recruiter screening: generally executive searches are run through an executive recruiter, an executive recruiting firm, or a VC recruiting firm. They should lightly filter for candidates’ interest in the role and their plausibility for getting an offer. My experience is that executive recruiters are excellent at this filtering as long as you listen to them!
• CEO chat(s): make sure the CEO and candidate could work together well, with a focus on the candidate’s understanding of the opportunity and the core challenges. Don’t lean out of the challenges: the best candidates know the challenges exist and will be skeptical if you try to avoid or downplay them. Have as many of these as necessary to build conviction that the candidate is plausible and engaged.
• 2-3 interviews with executive peers: have two to three executive peers interview. These interviews should explicitly cover the topics discussed in the next section, and should have a documented rubric for assessing candidates. A written rubric will particularly reduce the risk of false negatives and false positives, which unstructured executive interviews often introduce.
• 30 minute presentation with a 30 minute Q&A: a short presentation given by the candidate that’s focused on their understanding of what they’d need to do in their new role is an excellent way to assess whether the candidate is listening throughout the process, and whether they have the executive acumen to operate within your organization. Avoid the temptation to expand attendees, and instead reuse the executives and CEO who have already met the candidate.
  
  Introducing more attendees will randomize evaluation rather than improve evaluation. For example, bringing in the Chief Marketing Officer (CMO) for the first time will often cause the CMO to observe that the presentation missed several key marketing needs, which is somewhat expected if the candidate hasn’t met anyone yet from Marketing. That’s a randomizing signal. If the CMO is indeed a key stakeholder then they should be one of the executive peers included in the previous step rather than added to the presentation.
• Perform rigorous backchannel references: find three to four individuals who have worked directly with the individual in question for an extended period of time. I’m generally ambivalent about backchannels, but in the case of executives I believe the risk of hiring a poor executive is high enough that it’s an essential step. Candidate supplied references are not very high signal at this level, because the candidate will prepare their reference with talking points, including how to answer questions around gaps.
• 2-3 interviews with members of Engineering: assuming the other steps have gone well, then end with several interviews with engineers and engineering managers. Your primary goal here is to build commitment to the candidate from within the Engineering team, but you also want to listen for any major concerns from Engineering. Your interviewers should be running a structured interview with explicit areas of evaluation. It’s not ideal, but acceptable, to get some lukewarm signals at this stage, as long as you don’t get any major concerns. It’s very rare to hire any new manager whose team doesn’t have some concerns.

At this point in the process, either go to offer or decide not to extend an offer. Resist the temptation to hedge. Delaying sends candidates a bad message, and there’s rarely additional information out there that will change your mind for the better.

Four areas of evaluation

There are an unlimited number of skills to assess executives on, and ultimately there are more skills than you can viably assess. I recommend drilling in on these areas:

• Executive skills: are they an effective listener and communicator? Do they have the fundamental skills expected of an executive at this level, such as operating to a financial plan, supporting a single or multi-business unit organization, running a hiring or performance process, etc? You’ll get a signal on this from the presentation, sessions with peer executives, and from backchannel references.
• Role and company specific skills: every executive role you’re hiring for is aimed to solve a handful of specific problems at your company, and you should assess on those dimensions. In some cases this is improving partnership between Sales and Engineering, in other cases it’s improving Engineering velocity, and in others it’s partnering more effectively with peer executives. Identify whatever these are, and ensure that you explicitly cover them in either the CEO or peer executive sessions.
• Engineering functional expertise: depending on how you’ve scoped your engineering executive role, you’re going to want some sort of functional expertise. In some companies this is deep on running a scaled organization, guiding new product development, partnering between Engineering and commercial functions, technical architecture, or even infrastructure. Whatever it is for the role you’re hiring for, you should ensure that either the engineering interviews or the peer executive interviews cover these points.
• Historical performance and behavior: use your backchannel references to get an accurate understanding of the candidate’s true performance and behavior over time. There are effective executives who leave a trail of angry peers behind them. Similarly, there are very ineffective but beloved executives who remain far too long at companies that they serve poorly. You can assess self-awareness by asking about these directly, but you can only assess actual performance by talking to folks who were there. Good executives can spin even the worst performance into something positive, which means you simply cannot rely on them to self-evaluate.

You’ll note that I’ve not provided a checklist of skills to evaluate against. This is deliberate, because the role of a strong engineering executive is exceptionally broad, and reducing it to a list of skills will distract you from evaluating what is particularly valuable to your search. Most current CTOs and VPs of Engineering out there are the wrong fit for your role at your company. Evaluate on the specifics, not the universal.

Summary

You now know how to avoid the unicorn search, avoid bringing too many interviewers into the process, and how to evaluate the particulars of what you need rather than anchoring too heavily on vibes. Even with all of that in mind, these are still difficult searches. Don’t get discouraged if it takes you five or six candidates before you find someone you’re excited about, this is a natural part of learning how to hire a new executive role. Conversely, do get worried if you’re not excited after talking to ten-plus candidates; that probably means your search is going a bit off the rails.

Poking around OpenAI.

I haven’t spent much time playing around with the latest LLMs, and decided to spend some time doing so. I was particularly curious about the usecase of using embeddings to supplement user prompts with additional, relevant data (e.g. supply the current status of their recent tickets into the prompt where they might inquire about progress on said tickets). This usecase is interesting because it’s very attainable for existing companies and products to take advantage of, and I imagine it’s roughly how e.g. Stripe’s GPT4 integration with their documentation works.

To play around with that, I created a script that converts all of my writing into embeddings, tokenizes the user-supplied prompt to identify relevant sections of my content to inject into an expanded prompt, and sent that expanded prompt to OpenAI AI’s API.

You can see the code on Github, and read my notes on this project below.

References

This exploration is inspired by the recent work by Eugene Yan and Simon Willison. I owe particular thanks to Eugene Yan for his suggestions to improve the quality of the responses.

The code I’m sharing below is scrapped together from a number of sources:

• OpenAI Cookbook on Question Answering using Embeddings
• OpenAI Cookbook on preparing data for use in embeddings
• OpenAI Cookbook on creating embeddings

I found none of the examples quite worked as documented, but ultimately I was able to get them working with some poking around, relearning Pandas, and so on.

Project

My project was to make the OpenAI API answer questions with awareness of all of my personal writing from this blog, StaffEng and Infrastructure Engineering. Specifically this means creating embeddings from Hugo blog posts in Markdown to use with OpenAI.

You can read the code on Github. I’ve done absolutely nothing to make it easy to read, but it is a complete example, and you could use it with your own writing by changing Line 112 to point at your blog’s content directories. (Oh, and changing the prompts on Line 260.

You can see a screenshot of what this looks like below.

This project is pretty neat, in the sense that it works. It did take me a bit longer than expected, probably about three hours to get it working given some interruptions, mostly because the documentation’s examples were all subtly broken or didn’t actually connect together into working code. After it was working, I inevitably spent a few more hours fiddling around as well. My repo is terrible code, but is a full working code if anyone else had similar issues getting the question answering using embeddings stuff working!

The other comment on this project is that I don’t really view this as a particularly effective solution to the problem I wanted to solve, as it’s performing a fairly basic k-means algorithm to match tokenized versions of my blog posts against the query, and then injecting the best matches into the GPT query as context. Going into this, I expected, I dunno, something more sophisticated than this. It’s a very reasonable solution, and a cost efficient solution because it avoids any model (re)training, but feels a bit more basic than I imagined.

Also worth noting, the total cost to developing this app and running it a few dozen times: $0.50.

Thoughts

This was a fun project, in part because it was a detour away from what I’ve spent most of my time on the last few months, which is writing my next book. Writing and editing a book is very valuable work, but it lacks the freeform joy of hacking around a small project with zero users. Without overthinking or overstructuring things too much, here are some bullet points thoughts about this project and expansion of AI in the industry at large:

• As someone who’s been working in the industry for a while now, it’s easy to get jaded about new things. My first reaction to the recent AI hype is very similar to my first reaction to the crypto hype: we’ve seen hype before, and initial hype is rarely correlated with long-term impact on the industry or on society. In other words, I wasn’t convinced.
• Conversely, I think part of long-term engineering leadership is remaining open to new things. The industry has radically changed from twenty years ago, with mobile development as the most obvious proof point. Most things won’t change the industry much, but some things will completely transform it, and we owe cautious interest to these potentially transformational projects.
• My personal bet is that the new AI wave is moderately transformative but not massively so. Expanding on my thinking a bit, LLMs are showing significant promise at mediocre solutions to very general problems. A very common, often unstated, Silicon Valley model is to hire engineers, pretend the engineers are solving a problem, hire a huge number of non-engineers to actually solve the problem “until the technology automates it”, grow the business rapidly, and hope automation solves the margins in some later year. LLM adoption should be a valuable tool in improving margins in this kind of business, which in theory should enable new businesses to be created by improving the potential margin. However, we’ve been in a decade of zero-interest-rate policy which has meant that current-year margins haven’t mattered much to folks, which implies that most of these ideas that should be enabled by improved margins should have already been attempted in the preceeding margin-agnostic decade. This means that LLMs will make those businesses better, but the businesses themselves should have already been tried, and many of them have failed ultimately due to market size preventing required returns moreso than margin of operating their large internal teams to mask over missing margin-enhancing technology.
• If you ignore the margin-enhancement opporunties represented by LLMs, which I’ve argued shouldn’t generate new business ideas but improve existing business ideas already tried over the last decade, then it’s interesting to ponder what the sweet spot is for these tools. My take is that they’re very good at supporting domain experts, where the potential damaged caused by inaccuracies is constrained, e.g. Github Copilot is a very plausible way to empower a proficient programmer, and a very risky way to train a novice in a setting where the code has access to sensitive resources or data. However, to the extent that we’re pushing experts from authors to editors, I’m not sure that’s an actual speed improvement for our current generation of experts, who already have mastery in authorship and (often) a lesser skill in editing. Maybe there is a new generation of experts who are exceptional editors first, and authors second, which these tools will foster. If that’s true, then likely the current generation of leaders is unable to assess these tools appropriately, but… I think that most folks make this argument about most new technologies, and it’s only true sometimes. (Again, crypto is a clear example of something that has not overtaken existing technologies in the real world with significant regulatory overhead.)

Anyway, it was a fun project, and I have a much better intuitive sense of what’s possible in this space after spending some time here, which was my goal. I’ll remain very curious to see what comes together here as the timeline progresses.

That's all for now! Hope to hear your thoughts on Twitter at @lethain!

This email was sent to you why did I get this?    unsubscribe from this list    update subscription preferences Will Larson · 77 Geary St · co Calm 3rd Floor · San Francisco, CA 94108-5723 · USA