Igor47's Blog https://igor.moomers.org/ The feed for Igor's personal writing Thu, 29 Jan 2026 05:24:14 GMT https://validator.w3.org/feed/docs/rss2.html NextJs + Feed en Igor47's Blog https://igor.moomers.org/images/myhead.jpg https://igor.moomers.org/ All rights reserved 2023, Igor Serebryany Technology Philosophy Humanity <![CDATA[Making Medical Chocolate]]> https://igor.moomers.org/posts/medical-chocolate https://igor.moomers.org/posts/medical-chocolate Wed, 28 Jan 2026 00:00:00 GMT This works out to 2.44ml of oil added to 65 grams of chocolate. However -- this is a problem. I'm using 72% chocolate, which contains about 30% fat (cocoa butter) -- about 20 grams per 65g bar. Adding 2.44ml of oil would increase the fat content by over 10%, likely resulting in a soft, greasy-feeling bar. To make sure I ended up with a nicely tempered bar, I ended up halving the oil, making the dose 2 squares instead of 1 square. To make the medicated chocolate, I heated enough chocolate for 4 bars to around 120°F, and let it cool to 90°F. Then, I added my cannabis tincture and stirred gently but thoroughly. Finally, I poured the chocolate into the mold and allowed it to set at room temperature for quite a while. ![The chocolate is poured into molds](/images/chocolate-poured.jpg "Chocolate is poured into molds") After letting it sit for a while, it solidified nicely and got a very pretty swirly texture. I wonder if this was the result of the tincture I added? ![The chocolate has set](/images/chocolate-solidified.jpg "Chocolate after it's solidified") ## Upshot I wrapped the chocolate I made in aluminum foil and proudly delivered it to my mom's memory care community. The next day, I learned that the staff cannot legally administer this chocolate to my mom. Apparently, the medicine cart cannot include an obviously home-made product. They're only allowed to administer cannabis products that are "commercially produced". So in the end, this project was, technically, a complete waste of time. Still, I had fun learning about chocolate! Plus, now I have a bunch of CBD chocolate for home consumption. When you're on a caregiving journey for a loved one with dementia, you quickly learn to take the small wins and look at the bright side. Stay tuned for my next post -- producing professional-looking cannabis product wrappers. ]]> <![CDATA[Preventing Political Fundraising Spam]]> https://igor.moomers.org/posts/political-fundraising-spam https://igor.moomers.org/posts/political-fundraising-spam Fri, 23 Jan 2026 00:00:00 GMT <![CDATA[My WLED Christmas Lights]]> https://igor.moomers.org/posts/wled-christmas-lights https://igor.moomers.org/posts/wled-christmas-lights Thu, 01 Jan 2026 00:00:00 GMT <![CDATA[Data Boundary Layer for HTMX with Zod]]> https://igor.moomers.org/posts/zod-schemas-htmx https://igor.moomers.org/posts/zod-schemas-htmx Fri, 14 Nov 2025 00:00:00 GMT ` in your form, and the user only selects `apple`. You get `fruit=apple` in your request body. Ruby's [rack](https://github.com/rack/rack) will turn this into an object like `params.fruit = "apple"`. What if the user select both `apple` and `orange`? Well, in that case you'll get `params.fruit = ["apple", "orange"]`! You can solve this problem by using `name="fruit[]"` instead, which `rack` uses as a hint to remove the `[]` and always turn `fruit` into an array. This is pure convention -- the `[]` have no meaning outside the request parsing logic in that stack. Your favorite stack might also do this, but you don't know unless you look it up. In Hono, the situation is actually even more annoying. You would typically use [`parseBody`](https://hono.dev/docs/api/request#parsebody), which returns `Record`. If your user selects `apple` and `orange`, you'll get `params.fruit = "orange"`, totally dropping the `apple`. You should invoke `parseBody({ all: true })` instead, in which case you'll get a `Record`. You don't need to add `[]` to your input `name`s -- Hono ignores this convention, and since it doesn't strip the `[]` you'll end up with inconvenient field names in your objects. Then there's also `{ dot: true }`, which treats `.` as special in your field names and turns dot-separated fields into nested objects. ## Zod and Service Representation In your services, you will probably want to use objects with sensible types. For instance, you might want something like: ```ts const dietSchema = z.object({ fruit: z.array(z.enum(["apple" | "banana" | "orange"])).min(1), perDay: z.number().int().positive() }) ``` This will produce a type like: ```ts type DietSchemaT = { fruit: ("apple" | "banana" | "orange")[], perDay: Number, } ``` To get from your `parseBody` representation to this typed representation, you have to both parse and validate. For example, `perDay` might be any of: * (happy path) a string containing a number, like `"5"` * an empty string `""` * some random string `"bob"` * something else totally unexpected I took several stabs at this problem in CSheet. Initially, I split up my validation and submission logic. I had my validation run on the unparsed schema, which is a `Record` type. This meant doing a bunch of parsing directly in validation. For example, I might do something like: ```ts if (body.perDay) { const perDay = parseInt(body.perDay, 10) if (isNaN(perDay)) { errors.perDay = "Invalid number" ``` Then in my submission logic, I would attempt to parse using the `zod` schema, turning any parsing errors into form errors. And my business logic operated on the parsed service schema. This is obviously a lot of duplication. In version 2, I unified validation and submission. I added an `is_check` parameter to all my requests, representing either a request to validate or perform the submission. Both began with attempting to parse the request body using the `zod` schema. This meant I could lean on `zod` to perform the parsing: ```ts type DietSchemaT = { perDay: z.preprocess((val) => { if (val === "" || val === null || val === undefined) { return null } if (typeof val === "string") { const num = Number(val) return isNaN(num) ? val : num; } return val }, z.number().int().positive()) } ``` There's a lot of common patterns here, so I eventually factored these out into what I call [`formSchemas`](https://github.com/igor47/csheet/blob/main/src/lib/formSchemas.ts). When validating, an empty form field is not an error -- it just means the user hasn't gotten to the form field yet. To deal with this, my version-2 schema definitions/service logic had two features: * I actually parsed twice -- for validation, using the [`partial`](https://zod.dev/api#partial) version of the schema * Since `zod` doesn't well-represent `z.preprocess(method, schema).optional()`, I had to embed optionality into my `schema` This eventually let me to V3: just define the schema the way it *should* be. Arguably, I should have done this from the beginning. I still wanted to avoid displaying errors on missing fields after validation. But I realized I could do this when rendering the form. If the field has an error, I now check if (a) we were validating (vs submitting) the form and (b) the field is blank. If both of those are true, I [ignore the error](https://github.com/igor47/csheet/blob/06df22002c79c2fad32388c98f3762e6d5871e07/src/lib/formErrors.ts#L67-L85). ## Re-rendering the Form After validation, we re-render the form for `htmx`, possibly containing new fields or errors. We want to pass the user's answers back to the component, so that the form can be rendered with the old answers populated. We have at least 3 choices for which representation to pass to the form component: 1. generate a `FormData` from the string encoding 2. the version we get from `parseBody` 3. the parsed version we get inside the service I went back and forth several times on which representation to use. It's tempting to use (3), the typed service representation. It's strongly typed, and it seems somehow more correct to pass around a `{ perDay: Number }` type than a `Record` type. However, using the typed representation in form rendering meant converting values back to their `FormData` types. I would have to do `value={String(values.perDay)}`. Also, if parsing fails, I don't actually have a typed representation to use! I eventually settled on using (2) as the least of all evils. It's an object, so for forms with nested fields or arrays I can at least do sensible iteration to render the form. It also has the benefit of containing exactly what the user input, avoiding the unexpected UX of your values changing for you. ## LLM Representation Since I have an LLM assistant with tool calling embedded into the system, my incoming request data might also be generated by the LLM instead of a form. I use [Vercel AI](https://ai-sdk.dev/) to interface with LLMs, and it accepts `zod` schemas to create descriptions for LLM tool-calling. Making the schemas strict/non-optional helped tighten up what the LLM generates. Now, a field is marked optional only if it's actually optional in the tool call -- not because it might be optional during validation. For `zod`, there's a distinction between input and output schemas. My output schema is the strongly typed representation the service operates on. When I give this schema to the LLM, it generates strongly-typed (JSON) tool calls. However, my *input* schema is implicitly written to consider the output of browser forms. As a result, since I perform schema parsing *inside* the service, I unfortunately have to convert the LLM's strongly-typed tool call into a "stringly"-typed input for the service. The strings are then immediately parsed back into the strongly-typed representation. There might be a good way to handle this, probably by parsing the form data outside the main service function, but I'm not sure if the added complexity is worth it. ## Takeaways After three iterations, I landed on a pattern that works well: - Define schemas as they **should be** for your services (strict, typed, non-optional) - Use `zod.preprocess` to handle the string→type conversions from `FormData` - Handle validation-vs-submission differences at render time, not in the schema The same schemas work for both browser forms and LLM tool calls, which has been a fortunate convergence. I do find myself missing `pydantic`, which feels more ergonomic for parsing data thanks to its default coercion However, `zod`'s `preprocess` works, and is definitely more explicit. I've factored the common patterns into [`formSchemas.ts`](https://github.com/igor47/csheet/blob/main/src/lib/formSchemas.ts). If there's interest in packaging this as a standalone library, ping me on [the issue](https://github.com/igor47/csheet/issues/57). ]]> <![CDATA[Notes from SF Climate Week 2025]]> https://igor.moomers.org/posts/climate-week-2025 https://igor.moomers.org/posts/climate-week-2025 Fri, 25 Apr 2025 00:00:00 GMT If you're doing market rate of return, you're participating in the existing extractive economy This is somewhat of a hard sell. I think the folks at Carbon Collective would like to argue that you can have your cake and eat it, too -- investing in values-aligned organizations while preserving your portfolio for continued activism, or maybe even building wealth. Other sources agree:

Our investment conviction is that sustainability-integrated portfolios can provide better risk-adjusted returns to investors.

I lean more on Larry Fink than Dan Chus side here. I think climate investing has the potential to both make a difference, while providing *better* returns. I'm down to put some money behind this conviction, and would like to encourage my friends and family to do the same. ## Summing Up In an unprecedented dark moment for climate change, I found my day at SF Climate week inspiring and full of actionable insights. An important next step for me is to continue to participate in this community. This also feels like a good moment to step up my climate philanthropy, and encourage my fellow philanthropists to do the same. We can't close the gaps left by the retreat of federal funding, but we should strive to preserve what we can. Additionally, I have a ton of next steps on the impact investing front: * Look into shareholder activism using iconik * Look into shifting my assets, perhaps using something like [Values Advisor](https://valuesadvisor.org/) * Continue doing more early-stage climate investing If you're looking for angel investors in your climate tech company -- get in touch! ]]> <![CDATA[A Solar-powered EV at Burning Man 2024]]> https://igor.moomers.org/posts/solar-ev-at-burning-man-2024 https://igor.moomers.org/posts/solar-ev-at-burning-man-2024 Sat, 28 Sep 2024 00:00:00 GMT 2, and less readily available for purchase. ## Calculating Power Needs I estimated that, if I wanted to run the AC in the Rivian for a few hours a day, that would be my biggest power draw, far dwarfing any other loads I might need (e.g. lights, charging personal batteries, etc...). How much power did I actually need? This is not an easy question to answer. Rivian studiously avoids any mentions of watts, volts, or amps in the in-car UI. All battery displays are in %, with the total capacity of the battery (in kWh) nowhere to be found, or in miles, which are a meaningless unit only peripherally correlated to actual road miles given the wildly differing efficiencies given terrain, tires, driving style, and a thousand other factors. So, how much power would I use keeping the AC on? I did a bunch of hacky tests, keeping the AC on in my driveway and checking the differences in % with my eyeballs every period of time. But I was only really able to answer the question when I learned about a [secret RiDE menu](https://www.rivianforums.com/forum/threads/latest-ride-menu-code.21755/). Using this menu, I was able to learn that when awake, my Rivian R1S uses about 500W of power *baseline*, just sitting there. With the AC on, the usage goes to about 2500W. That was more than double the amount of solar I planned for, meaning that for every hour of running the AC, I would need about 2.5 hours of charging to keep the same SoC. It's actually slightly worse because, when charging at L1/120V, the Rivian seems to put only about 60% of the power into the HV battery. Presumably, the rest is going into some combo of staying awake (500W!) and keeping a built-in inverter running. ## Inverters and Batteries I spent a LOT of time researching solar inverters. The first annoying part is that I had to get a battery. This is frustrating, because my Rivian is already basically a giant battery on wheels. Why did I need *another* battery? One reason is that the R1S is a 400V architecture, and I didn't really see any inverters running at those voltages. Another is that, while [Rivian continues promising V2G and bi-directional charging](https://enteligent.com/products/enteligent%E2%84%A2-tlcev-t1-trusted-charging-presale), this so far remains vaporware. Without bidirectional charging, there's no way to way to use power when the sun is not shining. So I would need a battery if we wanted to run any loads at night. But also, most inverters need either a battery or grid power to function. There do seem to be *some* off-grid solar EV chargers that don't require an intermediate battery, but these are rare. I found [this one](https://enteligent.com/products/enteligent%E2%84%A2-tlcev-t1-trusted-charging-presale), but it's in pre-order only and not yet generally available. In the end, I went for [this solar inverter](https://richsolar.com/collections/inverters/products/nova-3k-3000-watt-48-volt-off-grid-hybrid-inverter), which was temporarily available from [ShopSolarKits](https://shopsolarkits.com/collections/off-grid-solar-inverters/products/3000-watt-48v-all-in-one-inverter) for only $400. I also snagged [this battery](https://www.amazon.com/gp/product/B0CP7FZC1P) for about $500. Finally, it seemed like the inverter didn't have a good way of showing the power in/out of the battery, so I last-minute purchased [this little power meter](https://www.amazon.com/dp/B013PKYILS) to install on the battery. This last purchase was incredibly clutch, allowing me to track power consumption at a glance. ## Wiring and Dust The inverter I got was not rated for dust, and I didn't want it failing half-way through the burn. I decided to build an enclosure for it, with air filters on top and bottom and an auxiliary fan so it could shed heat while not getting too dusty. I began by mounting all the components on the back -- a friend helped with this. ![Initial system being wired](/images/bmsolar/system-with-friend.jpg) This allowed me to do the full system test. Nothing exploded! ![First system test](/images/bmsolar/first-system-test.jpg) I used three breaks -- one as a battery disconnect, one for the HVDC solar, and another for the inter AC output. There are not a lot of devices that run on 48V, so I added a [12V step-down regulator](https://www.amazon.com/gp/product/B07GPZWG1S) meant for golf carts. In hindsight, I wished I had gotten a larger one so I could power more USB-C ports in parallel, but this one was okay. Next, I built a plywood box around the back plate. I used a pocket hole jig to join all the sides. The molding you see on the bottom in this photo is where the bottom air filter is meant to rest. ![Initial three-sided enclosure](/images/bmsolar/initial-enclosure.jpg) On the front of the enclosure, I wanted a door so I could turn switches on/off and see inverter status. I routed an opening out of the front panel, and added molding so the door would have something to sit against as a dust barrier. I used velcro to keep the door closed. ![Opening routed out of the front panel](/images/bmsolar/front-panel-opening.jpg) I also routed out openings for the 120V outlet, and for handles on the sides of the box. I used bolts through the back panel to bring out battery power, as well as a ground connection which I didn't end up using. I brought PV in and 12VDC out using [Anderson power pole panel mounts](https://www.amazon.com/dp/B097QG383J). Here's a final walkthrough of the system I made for the camp:
## So ... How Did It All Work Out? On the whole, the system worked well. Mounting the panels on the car was easy, and they felt secure. Nothing died, and my air filters kept the dust out of the enclosure (although it wasn't a very dusty burn). I got surprisingly good power production, more than 1kW at peak. It wasn't a very hot year, and we didn't end up needing the Rivian's AC. However, a lot of folks in camp had swamp coolers, and we ran a swamp cooler grid off the solar. It's always convenient when your biggest loads run when the sun is shining! We also had several morning when we made "solar waffles" using electric waffle irons powered by the solar system. I found this incredibly satisfying, though opinions differed on whether this made the waffles taste any better. On the other hand, some things didn't work well, mostly having to do with the Rivian. First -- we pulled a pretty heavy trailer, which really affected my range. ![Rivian with the trailer](/images/bmsolar/rivian-with-trailer.png) As a result, we arrived to camp with about 49% range -- clearly not enough to get back to a charger in Fernley. All week long, I kept trying to charge up the Rivian off solar, but all I managed to do was keep it at 47%. One reason was that, although I set the Rivian to `stay off` mode, I failed to turn off an AC schedule that was set up around my partner's daily commute, and which cooled the car down for her in the mornings and afternoons. I discovered this on Thursday, when I went into the Rivian to grab something and found it refreshingly cool. Chalk another one up to the Rivian's counter-intuitive UI! The Rivian's portable charger, in 120V mode, allowed me to set the charge rate between 8A (960W) and 12A (1500W). Even at 8A, I would be draining the battery somewhat. Whenever I plugged the Rivian in, I had to remember to unplug it, which I sometimes forgot to do until too late in the day to fully re-charge the 48V battery. As a result, on two morning I found the 48V battery totally dead in BMS undervolt protection mode. This is is when I learned that my inverter wouldn't turn on without battery power. I ended up having to "jump-start" the inverter. This, I did by wiring two 20V DeWalt batteries in series (40V total, sufficient to meet the inverters 36V min-voltage threshold) and connecting them to the battery terminals long enough for the inverter to boot, and then disconnecting the jump starter once the solar kicked in. My conclusion was that 1kW of solar is enough to either run camp, or to charge my EV, but not both. On the final day of the burn, I ended up plugging the Rivian into the Silicon Village grid, where a few plugs had freed up once a few of their RVs had left. In hindsight, I would have had a better time with the system if I hadn't attempted to charge the Rivian off the solar at all. ## What's Next I think the basics of the system are solid, but I definitely need MOAR SOLAR. With another 5 panels, I could have enough power to charge the EV at max 120V speed, and still keep the camp fully powered. The big question for me then is, how to mount 8 solar panels, ideally in a way that also makes use of the shade they cast. I honestly think that building the mounting structure on playa is the biggest obstacle to adoption. Looking at a few big solar camps, systems varied wildly, mostly using steel tubing with some kind of panel mount clamps. My experience locating the correct hardware even just for my unistrut helps me realize just how much work is involved in the hardware selection. I wonder if there's a market for easy plug-and-play systems for camps, including all the components -- panels, structural members, mounting hardware, inverter, and battery. This is like a [black rock hardware](https://formandreform.com/blackrock-hardware/) for in-camp solar. If you're interested, let me know -- maybe I can put this together? Overall, it was pretty encouraging to see *much* more solar at the burn this year, and commensurately fewer loud, smelly generators. However, there's still a lot of work to do. Also, bringing an EV to the burn continues to be fairly challenging. I'm excited to keep iterating on the problem with my fellow burners! ]]> <![CDATA[Now: March 2024]]> https://igor.moomers.org/posts/now-march-2024 https://igor.moomers.org/posts/now-march-2024 Sun, 17 Mar 2024 00:00:00 GMT <![CDATA[Light Up Boat Parade 2023]]> https://igor.moomers.org/posts/light-up-boats https://igor.moomers.org/posts/light-up-boats Mon, 18 Dec 2023 00:00:00 GMT

## Rigging The Mainsail A friend had a giant cache of [twinkly strings](https://twinkly.com/en-us/products/strings-multicolor). Our plan was to hang them up in the mainsail triangle, and display some cool mapped patterns in 2D. Here's our rigging plan: ![Rigging Plan](/images/light-up-boat-rigging.svg) I was really worried about losing the main halyard, so we used a serious line with some extra-redundant knots to secure it at the mainsail clip and on the deck. For the rigging, we used paracord with some alpine butterflies tied into it: ![Alpine Butterfly](/images/alpine-butterfly.jpg) The first few twinkly strings, next to the mast, used the entire length of the string. For the later strings, we could start at the top, go back down to the boom, then go back up to the top again. This required hosting the rigging, figuring out where the string would end up, and securing it on the rigging that ran along the boom. Then we could lower the whole rigging, and secure the other end of the string in the loop that ran along the topping lift. This was a huge pain. Christmas lights **want** to get tangled, and hosting them up and down gives them just the chance they're looking for. It took us until the very end to figure out that we could just put extra paracord through the loops, and use it to hoist one string at a time. Next time, we'll definitely just install extra paracord in every loop from the very beginning. We ended up using 2 strings of 400 lights, and 1 string of 600 lights -- 1400 LEDs total on just the mainsail triangle. ## Mapping We hoped to use the Twinkly app to map the lights to a 2D image. This **absolutely** did not work. First, the mast is really tall -- it was difficult to even get the whole set of lights in the frame. To do that, we had to stand pretty far back, which made each LED hard to distinguish in the picture. Finally, the strings swayed in the wind, and the whole boat rocked in the water -- no way to get a still image. Thankfully, with just the default unmapped patterns, the Twinkly lights look pretty good. However, I am now kind of obsessed with the idea of mapping the lights to a 2D image. I think using a hybrid automatic-manual approach should work well. I should be able to take a single photo as the base of the scene. Then, I should be able to turn on sections of the lights, and then indicate their position in the base image. It doesn't have to be perfect -- just good enough to get the general shape of the boat. I would like to try writing some software for this in time for next year's parade. ## Other Lights I bought some of [these lights](https://amzn.to/3RSKvv1) to put on the lifelines. My plan was to control them with [WLED](https://kno.wled.ge/). But I discovered that, though they are individually addressable, they have some bug in the implementation of the protocol that causes them to flicker unpredictably. I did bring some of [these strings](https://amzn.to/3NI0LfP). Unlike the previous lights, these don't explicitly advertise WS2812. However, they do work well with WLED. Their implementation of WS2812 is kind of [odd](https://todbot.com/blog/2021/01/01/ws2812-compatible-fairy-light-leds-that-know-their-address/). Rather than shifting out the bits for the next light, they allow the controller to wiggle the entire common data line, and each LED knows its own address. This means you cannot link multiple strands together serially -- the second string will just mirror the first. I didn't have time to do anything fancy with these lights. We used them with their default controllers using some built-in patterns. ## Future Work For next time, I would dispense with the fancy Twinkly lights (which, wow, are pretty expensive!). Instead, I would use the cheap strings off Amazon and write my own software to control them over WS2812. Let me know if you have ideas for how to create a 2D mapping UI for this! ## The Parade Oh yeah, there was a parade! That part was super-fun. We got to see all the other boats up-close, and listening to the marine radio chatter with the stressed-out parade organizers trying to keep everything going was pretty entertaining. Next time I would like some other crew to feel confident driving -- turns out operating a 40' boat in close quarters with a bunch of other boats, in the dark, in a shallow marina, is stressful! We didn't win any prizes, but we had a great time. ]]> <![CDATA[Docker Compose Secrets Manager]]> https://igor.moomers.org/posts/secrets-in-docker-compose https://igor.moomers.org/posts/secrets-in-docker-compose Fri, 15 Dec 2023 00:00:00 GMT <![CDATA[Soviet KGB Stories II]]> https://igor.moomers.org/posts/soviet-kgb-stories-pt2 https://igor.moomers.org/posts/soviet-kgb-stories-pt2 Thu, 21 Sep 2023 00:00:00 GMT **Note**: There is a [previous post](/posts/soviet-kgb-stories-pt1), about my uncle. I never met my grandfather Lev -- he died (of stubbornness) just about a year before I was born. But I heard lots of stories. In the land of [блат (blat)](https://en.wikipedia.org/wiki/Blat_(favors)), the person in charge of the pharmacy containing scarce but potentially life-saving medication is a good person to have on your side. And my grandfather, by all accounts, knew how to play the game. In my home city of Nikolaev, everyone knew him, and everyone owed him a favor -- therefore, or possibly also, everyone respected him. My grandfather could possibly have aspired to a greater status than a two-bedroom apartment in a [хрущёвка](https://en.wikipedia.org/wiki/Khrushchevka) and the once-yearly government-sponsored vacation to the Black Sea. In his later years, he was offered an opportunity to join [The Party](https://en.wikipedia.org/wiki/Communist_Party_of_the_Soviet_Union) -- a key opportunity for advancement in the USSR. But he turned it down, and this is the story of why. ## Some Flavor Before we get into the meat of my story, an anecdote. There's a common expression among Soviet Jews, which goes "Бьют не по паспорту, а по морде." Transliterated -- "biut-ne-po-pasportu-a-po-morde". Translated -- "they punch you, not in the passport, but in the face". This is a reference to the ["Пятая графа"](https://ru.wikipedia.org/wiki/%D0%9F%D1%8F%D1%82%D0%B0%D1%8F_%D0%B3%D1%80%D0%B0%D1%84%D0%B0) -- the "fifth line" of the Soviet passport, which described the passport-holder's "Nationality". For Soviet Jews, this line was filled in, "Jewish", and it served to close many doors. But -- they don't punch you in the passport. Anti-semites are keenly attuned to stereotypical ethnic features, and it was enough to simply look Jewish to get into trouble. Lev, for what it was worth, didn't look particularly Jewish. So, when he showed up at my grandmother's house to ask her father for her hand, he was at first confused for a government inspector coming for a surprise visit to their family pharmacy. While they figured out what to do with him -- the chaos abated somewhat when my grandmother's precocious kid brother announced, "That's not an inspector, that's Sofia's boyfriend!" -- they bustled him into the kitchen. My grandmother's grandmother was tasked with feeding Lev, which she went about grudgingly. Because he didn't look Jewish, she felt free to mutter her complaints in Yiddish while she served him food. "They put this goy in my kitchen, and I have to feed him? What's next, he's gonna ask for an (alcoholic) drink?" My grandfather, however, spoke fluent Yiddish. He learned it at a yeshiva in Kherson, which he attended at the same time as my grandfather Semyon -- they actually knew each other in school, decades before my dad met my mom in Nikolaev. So, hearing the muttering, he responded, "Actually, a shot of vodka wouldn't go amiss." When the old woman realized that the suitor was actually Jewish after all, her reluctance instantly disappeared. Moments later the table was covered with the best in the house. ## The War Years On June 22nd, 1941, Germany declared war on the Soviet Union. My grandfather Lev had just finished his first year of medical school, and he was immediately drafted. He ended up serving as a medic on the front lines through the entire war. After Germany surrendered in 1945, Lev was not immediately released from the military. He ended up serving in the occupation for another year, helping rebuild medical facilities in a Germany that had been reduced to rubble by the Allied campaign. When Lev finally returned to Odessa in October of 1946, his medical school had already finished more than a month of classes. He was told to come back the following year to restart his education. Lev was already way behind -- he would be four or five years older than his classmates -- and he didn't want to put life off for another year. Wandering around Odessa, he saw an announcement that the pharmacy school was still recruiting students. My grandmother had actually seen the same announcement when she showed up in Odessa to start school, a little late due to illness. And that's why I'm here to write this story today! ## After School My grandparents were married in 1949, and my uncle from [the previous story](/posts/soviet-kgb-stories-pt1) was born in 1950. They moved to Nikolaev, where my grandfather immediately became the head of the pharmacological supply warehouse. I tried to figure out why a kid right out of school was immediately in charge of a medical warehouse. It seems to boil down to two reasons. The first is a massive shortage of men in the aftermath of the war. While Soviet casualties [are disputed](https://en.wikipedia.org/wiki/World_War_II_casualties_of_the_Soviet_Union), they total at least 9 million soldiers, all men. The disputes all purport the official figures to be much too low, and there are estimates as high as 40 million military and civilian deaths -- twice as much as official figures. The second reason is that the USSR had what, by modern US standards, unreasonably generous medical leave. There was a baby boom in the USSR, just as there was in the US, and women were going into декрет, making them unreliable employees. ## Arrest, Exile, and Release As Lev was running the pharmaceutical warehouse, the USSR was in the grips of a paranoid Stalin. As a result of the [Doctors' plot](https://en.wikipedia.org/wiki/Doctors%27_plot), many Jewish doctors and medical workers were "dismissed from their jobs, arrested, and tortured to produce admissions". In 1951, Lev was using petty cash from his warehouse to purchase newspapers, which he hung up on the walls for public reading. Because of this, he was accused by the local KGB of "misappropriation of public funds". In a swift trial, he was pronounced guilty. At this time, the USSR was in a Nuclear arms race with the USA. Uranium ore was discovered in [Жёлтые Воды (Yellow Waters)](https://en.wikipedia.org/wiki/Zhovti_Vody), and the USSR needed people to mine the ore. I cannot tell if the Yellow River was actually yellow because of yellowcake uranium, or if the color was a coincidence. In any case, Lev was sent to this region of Ukraine to work in the mines. Because of his medical training and experience, Lev was spared hard labor in the mines. Instead, he provided medical services to the miners and surrounding community. Nevertheless, he was not free to leave, and my grandmother could not visit him. Instead, my grandmother expanded much effort trying to free him. She spoke to many lawyers, and visited many government officials. She was finally told by a well-respected advocate in Kiev, -- "Child, you just have to wait. There is no case against him, and they will let him out soon." ## Aftermath Stalin died in 1953. It was not until 1956 that his successor, Khrushchev, [denounced Stalin](https://www.britannica.com/event/Khrushchevs-secret-speech) and began the "de-Stalinization" of the USSR. However, almost immediately after Stalin's death, many of the political prisoners arrested during his rule were released. Among them, my grandfather Lev. After his release, Lev was restored to his former rights and returned to his previous job. He went on to make many advances in his field. Among them, building one the first pharmacies located inside a hospital, an innovation copied from the USA. For his service to the USSR in both military and civilian roles, he was awarded several medals. My mom likes to recount how, when he wanted to call someone in Moscow, he would connect to the operator and just say "This is Lev, connect me with so and so" -- an ordeal that was much more difficult for other people. However, though he was liked and respected, he never forgave the USSR for sending him to prison for more than 2 years. He did not join the party, and refused to get involved in any sort of politics. ]]> <![CDATA[A self-hosted URL shortener in Rust]]> https://igor.moomers.org/posts/self-hosted-link-shortener https://igor.moomers.org/posts/self-hosted-link-shortener Wed, 30 Aug 2023 00:00:00 GMT You are comfortable using the Rust Programming Language, and have written, run, and debugged Rust applications on a desktop environment. I was like, "Nope, I am not", and proceeded to the [general-purpose Rust book](https://rust-book.cs.brown.edu/) (I've been reading the Brown-hosted version because I want to take the little embedded quizzes). The book is *really* good, but there's too much reading in between the practical sections, and [I learn by doing](https://psycnet.apa.org/fulltext/2014-55719-001.pdf). So I resuscitated a previous idea of hosting my own link shortener, and here we are. ## Choices This project has a really odd mix of new and old technology. The modern approach is to run the project as a single binary which includes an embedded web server. The binary is responsible for serving any static files, and also for generating the dynamic responses. This seemed like too much work, and using an existing web server crate seemed like **too little** work. So, I am running the service behind Apache and using good-old [CGI](https://en.wikipedia.org/wiki/Common_Gateway_Interface) for the dynamic content. Remember CGI? Instead of your application code running as a long-lived process, it's invoked by the web server for each request. The details of the request are placed in the environment, and whatever is written to `stdout` is sent back to the client as the HTTP response. It's probably slower than having the process already running (`fork` and `exec` are slow!), but it lets the program be really simple. For storage, I looked at a bunch of options (like `sled` or `leveldb`) but ended up with good-old `sqlite`. Also, I'm probably doing a non-conventional thing with sessions. I didn't want to implement logins, plus probably for lots of things you don't even care about user-level persistence. But I figured it might be nice, and instead just exposed the user's session ID for them to see. If they want to return to the site later and see their short links, they can just "log in" with their session ID. For the front-end, I wanted it to be as lightweight as possible. I immediately discovered that I need to either have multiple pages, or write an SPA. But for multiple pages, how do I share common elements like a header/footer? This is where I learned about Apache's [SSI](https://httpd.apache.org/docs/2.4/howto/ssi.html) (server-side includes). I factored my header out into a separate file, and was able to include it in each page with a simple ``. I still had some javascript to write, and I attempted to just use plain JS with no libraries. But then I discovered [Alpine.js](https://alpinejs.dev/), which is just **so** lightweight and easy to use. For CSS, I used [Skeleton](http://getskeleton.com/) but I'm not super-excited about it. This is because it doesn't have a responsive grid system -- once something is, say, 6 columns, it's **always** 6 columns. I'm probably going to migrate this to [Bluma](https://bulma.io/) at some point to fix appearance on intermediate-size screens. ## Deploying I run my self-hosted services via `docker-compose` on my server, and it was really easy to add this one. I set up [my `Dockerfile`](https://github.com/igor47/smrs/blob/master/Dockerfile) for multi-stage builds -- my `dev` target just contains configured Apache. I bind-mount my `htdocs` into the container, and iterating on the rust code means building it locally and copying into the container filesystem. For production, my final stage builds the release version of the code and generates a container with the `htdocs` baked in. I have this container built and tagged via [Github Actions](https://github.com/igor47/smrs/blob/master/.github/workflows/publish.yaml). Then, in my `docker-compose`, I can just pull the image from `ghcr.io`. I ran into trouble because I'm hosting this behind a Cloudflare proxy, and so `traefik` couldn't get an SSL certificate for it. I had to add a custom `traefik` config for validating SSL via modifying Cloudflare DNS: ```yaml # for zones behind cloudflare proxied DNS, we use the cloudflare dns provider # see: # https://www.techaddressed.com/tutorials/certbot-cloudflare-reverse-proxy/ # this relies on credentials in a file on purr. see the `env_file` directive in # docker-compose for traefik cf: acme: email: admins@example.org storage: /acme/acme-cf.json dnsChallenge: provider: "cloudflare" ``` I then had to generate a Cloudflare API token with the `Zone.DNS` permission. I stored that in a file on my server, and then added it to my `docker-compose` via the `env_file` directive: ```yaml env_file: - ${STORAGE}/traefik/cloudflare.env ``` ## Link Shortener Criticism There's a whole section on [Awesome Self-Hosted](https://awesome-selfhosted.net/index.html) for [URL shorteners](https://awesome-selfhosted.net/tags/url-shorteners.html). This is where I came across [this Wikipedia section](https://en.wikipedia.org/wiki/URL_shortening#Shortcomings) which lists a bunch of criticisms of URL shorteners. In fact, a bunch of the shorteners in the Awesome list have a publically-hosted instance, but every one of those seems dead. Others explicitly say they had to put their public version behind a password or login because of abuse -- for instance, [liteshort](https://git.ikl.sh/132ikl/liteshort) says about their live demo: > (Unfortunately, I had to restrict creation of shortlinks due to malicious use by bad actors, but you can still see the interface.) I put mine behind Cloudflare, but I'm still seeing a bunch of nonsense (like wordpress hack attempts) in the request logs. So, I'm keeping the link a "secret" for now. I might add a password system to create new links if I see a bunch of abuse. ]]> <![CDATA[My Rivian R1S: Initial Thoughts]]> https://igor.moomers.org/posts/my-rivian-r1s-initial-thoughts https://igor.moomers.org/posts/my-rivian-r1s-initial-thoughts Fri, 19 May 2023 00:00:00 GMT <![CDATA[Purchase Allocation via Graph]]> https://igor.moomers.org/posts/purchase-allocation-via-graph https://igor.moomers.org/posts/purchase-allocation-via-graph Thu, 11 May 2023 00:00:00 GMT **Note**: This is cross-posted to the [Recoolit blog](https://www.recoolit.com/post/purchase-allocations-eng). When you [buy carbon credits from Recoolit](https://www.recoolit.com/buy), you are buying a specific amount of prevented atmospheric warming. We prevent warming by preventing the release of refrigerants into the atmosphere. (Note: if you want to learn more about how refrigerants cause warming, check out [our post on refrigerants](/posts/refrigerants-what-are-they).) Our purchase receipts are designed to be transparent -- you see exactly how much warming you are preventing, and exactly how. This post explains the technical details of how we create this transparency. ## Brief Overview of Recoolit When refrigerators, air conditioners, or other kinds of heat pumps reach end-of-life or need maintenance, the refrigerants inside need to be removed from the system. This is because refrigerant is at high pressure inside the system, and needs to be depressurized before the system can be serviced safely. It is common for refrigerants to be vented into the atmosphere during this process. Recoolit provides technicians with the tools, know-how, and incentives to capture these waste gasses instead of releasing them. Then, we destroy the captured gasses in a high-temperature incinerator, permanently preventing their release into the atmosphere. Our main mechanism for incentivizing refrigerant capture is to pay technicians for the refrigerants they capture. We also have to pay for the cost of destroying the refrigerants, including getting the destruction facility inspected and certified to international standards. Finally, we have to cover our operational expenses. This includes buying and maintaining the pumps and cylinders that we use to capture and store refrigerants. We maintain several depots, where technicians can borrow the equipment needed for recovery. Finally, we pay for a warehouse where we store refrigerants before they are destroyed. We cover all of these costs by selling carbon credits to individuals and organizations that want to offset their carbon emissions. ## Why Transparency? The world has almost no way to pay for pollution remediation. Sometimes, governments will force polluters to pay for the cost of cleaning up their pollution. Other times, governments will directly pay for cleanup using tax dollars, especially when the pollution is a public health hazard and there is no obvious responsible party who can be forced to tackle the cost. So, even with the growing awareness that climate change is a big, looming problem, there are too few ways to get money to organizations that are tackling the issue. The voluntary carbon credit markets -- where individuals and companies pay for carbon credits out of a sense of social responsibility, and not for any regulatory reason -- are one of the few ways to pay for climate change remediation. However, the voluntary carbon credit markets have been a bit of a mess. There are no international agreements or standards on how to quantify the impact of carbon credits. A few private organizations have stepped in to fill this gap, but their standards are not universally accepted. Finally, some of the largest private organizations -- known as registries -- have faced controversy. For instance, Verra, one of the largest registries, [has been criticized](https://www.theguardian.com/environment/2023/jan/18/revealed-forest-carbon-offsets-biggest-provider-worthless-verra-aoe) for allowing carbon credits to be sold for projects that were already underway, and for projects that were not actually preventing warming. For these reasons, Recoolit's founder Louis was determined for the company to be as transparent as possible from day one. We collect detailed data on every step of our operational process. When you make a purchase, we show you all of the data we've collected that pertains to the molecules that went into your purchase. Finally, we show as much data as possible on [our public registry](https://registry.recoolit.com/registry), so that anyone can see exactly what we are doing. ## What Is Our Data? Every Recoolit carbon credit begins with a recovery. This is where a technician captures refrigerants from a refrigerator, air conditioner, or other heat pump. At this step, we collect photos of the equipment that's being serviced, the reason for the refrigerant recovery, the amount of gas recovered, and the type of gas recovered (though we don't always know the exact type of gas at this point). Next, technicians return the cylinder used for the recovery to one of our depots. We verify the amount of gas recovered, and we also test the gas using a gas analyzer. We pay the technician for the amount of gas they collected. This payment compensates the technician for the time and effort they spent performing the recovery. Because we want to return the smaller recovery cylinders back into the field, we will often consolidate the gas from multiple recovery cylinders into a larger storage cylinder. We cannot mix different types of refrigerants, so we have to keep track of the type of gas in each cylinder. Every time gas is transferred, we keep detailed records on the source and destination cylinders and weights. Some small amount of gas is always lost during transfers, and the losses are not included in the carbon credits we sell. Next, we transport the gas to our destruction facility. Every time we transport gas, we weigh and test the cylinders at both ends. We maintain a chain of custody for the cylinders at all times, using signed transport manifests. Some of the refrigerants we transport are becoming expensive, because they are no longer allowed to be produced under internal agreements. Our procedures protect against loss or theft of refrigerants during transport. Finally, our refrigerants go through the destruction process. First, we take a sample of each cylinder that has arrived at the destruction facility. The sample is lab-tested under rigorous standards, to confirm the exact makeup of the contents of the cylinder. Next, the cylinder is hooked up to a high-temperature incinerator. In Indonesia, we partner with cement kiln operators, because their facilities reach the high temperatures needed to destroy refrigerants and need only minimal modifications to do so. After destruction, the cylinders are shipped back to us. We vacuum-test the cylinders to make sure they're not leaking, and use them for future recoveries or consolidations. ## Building the transfer graph When you buy a carbon credit from Recoolit, you are buying a specific quantity of a specific gas that was destroyed. In order to show you all of the data that went into your destruction, we need to trace the path of the gas from the recovery to the destruction. We call this data structure the "transfer graph", and it is the core of our transparency system. The transfer graph is a directed acyclic graph (DAG). In this graph, the edges are transfers from a source to a destination node. Each edge has a weight, which is the amount of gas transferred, as well as a gas type. Because cylinders are reused, the nodes are actually a specific cylinder ID during a specific time interval. A node is created when gas is first transferred into it, and "closes" when all of the gas is transferred out and we vacuum out the cylinder. A subsequent transfer into the same cylinder ID would create a new node. Each node can have multiple incoming and also outgoing edges. This is because we sometimes do partial transfers. Finally, each node can have a series of "events" associated with it. Events include things like "the gas was tested", "the cylinder was transported", or "the cylinder was weighed". ## An example This might be easier with an example, so let's use some real data from our public registry. I bought some credits from Recoolit, and [here is my receipt for that purchase](https://registry.recoolit.com/purchases/f436f5ca-a6fe-49c4-b10c-f4e46cafcb8d). This is a view of the same data in our internal system: ![Igor's purchase graph](/images/igor-purchase-graph.png) To allocate this purchase, we first look through all of our destructions to find one that has enough gas to cover the purchase. We might need to combine multiple destructions to cover the purchase. In this case, we find that we destroyed about 50kg of R-22 from cylinder `berkabut-panas-bebek`. R-22 is such a high-GWP refrigerant that only 511 grams of it were needed to cover my purchase of 1 tonne of CO2e. Next, we do a depth-first search through the graph, starting at the destruction node, and looking for a path that has enough gas to cover the purchase. We see that about 10 kg of R-22 arrived in `berkabut-panas-bebek` from `dingin-kering-harimau`, so we allocate 511 grams of that 10kg. Finally, we see that 5.8 kg of R-22 was recovered directly into `dingin-kering-harimau`, so we allocate 511 grams of that 5.8kg. This was a fairly easy case, as it only involved a single trajectory through the graph. Here's an example of a more complicated sale, which involved multiple kinds of gas allocated through multiple destructions: ![A more complicated purchase graph](/images/complicated-purchase-graph.png) In this case, the purchase of 30 tonnes of CO2e was covered by 3 different destructions. We destroyed 13235 grams of R-410a and 193 grams of R-32 to cover this purchase, and these gasses were recovered by two technicians on 3 different occasions. Three of the five recoveries were on the same day into three different cylinders, indicating a large recovery job that filled up multiple cylinders! ## Allocating purchases We allocate purchases using a depth-first search through the graph, starting at the destruction node. Every time we find a node that has enough gas to cover the purchase, we recursively look through the source nodes. The recursion terminates at a recovery node. We then propagate the list back up through the stack, allocating the purchase to each node in the path. Allocating the purchase means creating edges. For each purchase, we create a `sale` node in the graph. For each node that contributes gas to the purchase, we create a `sale` edge, from the source node to the `sale` node. To find all the nodes involved in allocating a sale, we look for all nodes connected to the `sale` node through a `sale` edge. Each `sale` edge includes the amount of gas that was allocated to the sale. To figure out if a node still has enough gas to cover the sale, we subtract the sum of all `sale` edges from the weight of the node's outgoing transfer edge. ## Formatting for display We've already discussed all of the data we collect during operations to enable this kind of transparency. You can now also know how our system allocates your purchase to destructions and recoveries. The final piece is presenting this data in a way that is easy to understand. In your receipt, we show you a path-decomposed version of the transfer graph. A path is a linked list of edges and nodes, starting at a recovery and ending at a destruction. However, in your purchase subgraph, a single node or edge might be involved in multiple paths. When we do the decomposition, we clone the shared nodes and edges, so that each path has its own copy. Here's an example of a non-decomposed graph: ![Non-path decomposed graph](/images/transfer-path.png) You can see that this includes two recoveries -- one in the blue box, and one in the red box. The green box includes a consolidation and destruction nodes that are shared between the two paths. When we display it, it would look more like this: ![Decomposed paths](/images/transfer-path-decomposed.png) There are two paths in this graph -- the one on the left, and the one on the right -- and the nodes in green are duplicated between the two paths. Here's how the same data might look in your purchase receipt: ![Paths in the purchase receipts](/images/transfer-path-final.png) Doing this is surprisingly non-trivial because it's not clear, just from the subgraph, how many paths through a particular node there are. To make it easier, we actually keep track of the paths when we construct the graph. Each time we begin trying to allocate gas from a destruction, we generate a path identifier. When we find a path that works, we store the path identifier in the `sale` edges that track the allocated purchase. This means that a node might actually have multiple edges connecting it to a `sale` node, each with a different path identifier. ## Wrapping up As you can see, we've put a lot of thought into how to make our data as transparent as possible. Our goal is to create the highest-quality carbon credits, with the greatest possible assurance to buyers that their purchase is actually making a difference. If you like what we're doing here, and want to support us, we urge you again to [buy some carbon credits](https://registry.recoolit.com/buy)! ]]> <![CDATA[Refrigerants: What Are They?]]> https://igor.moomers.org/posts/refrigerants-what-are-they https://igor.moomers.org/posts/refrigerants-what-are-they Mon, 08 May 2023 00:00:00 GMT **Note**: This is cross-posted to the [Recoolit blog](https://www.recoolit.com/post/refrigerants-what-are-they). At [Recoolit](https://www.recoolit.com), our mission is to reduce climate impact by collecting and destroying waste refrigerants. This mission is often unclear to people who aren't familiar with the refrigeration industry. We often hear questions like, "What are refrigerants?" or "Why are refrigerants a problem?" We've skipped over such questions on our [explainer page](https://www.recoolit.com/our-work) because they're quite technical, and we don't want to overwhelm our customers. This post is for those special nerdy few who want to know all the gory details! ## Heat pumps -- How do they work? Refrigerators and air conditioners are all a type of heat pump -- a device that moves heat from one location to another. They do this by means of a fluid that is pumped through a closed loop. First, the fluid is compressed, which causes it to heat up, turn into a gas, and become much warmer than the surrounding air. Next, the gas is pumped through a heat exchanger -- like a radiator on a car -- which results in the gas becoming cooler while the surrounding air becomes warmer. In a heat pump on "warming" mode, this heat exchanger is located inside a building or room, and this warming is the desired effect. The gas is then pumped to a second heat exchanger, where it is allowed to expand, turning it back into a liquid. The expansion cools the fluid. It's then pumped through a second heat exchanger, where it becomes warmer, while the air around it becomes cooler. In a heat pump on "cooling" mode, this second heat exchanger is inside the refrigerator, freezer, or too-warm room or building. The fluid is then pumped back to the compressor, and the cycle begins again. Mechanically, all heat pumps rely on just these few components -- a compressor, pumps, heat exchangers, and the fluid -- sometimes a gas, sometimes a liquid -- that runs between them. Physically, this process relies on just a few basic principles of nature -- the ideal gas law, plus the laws of thermodynamics. ## What are refrigerants? Refrigerants are the fluid used inside heat pumps to move heat from one place to another. There are many considerations when picking a substance to use as a refrigerant. For instance, a common early refrigerant was ammonia (R-717). Ammonia boils at a low temperature, which makes it easy to turn into a gas. It does not freeze until a very low temperature, so there's no risk of it freezing inside the heat pump during the expansion phase. And it has a fairly high specific heat, which means that it can absorb and release a lot of heat. Unfortunately, ammonia is highly toxic and corrosive. Other early refrigerants, such as sulfur dioxide (R-764) and methyl chloride (R-40), were also toxic. To avoid the safety issues associated with early refrigerants, scientists began experimenting with other substances. In 1928, Thomas Midgley, Jr. invented the first chlorofluorocarbon (CFC) refrigerant, which was marketed by DuPont as Freon-12 (R-12). R-12 was cheap to produce, non-toxic, non-flammable, non-corrosive, and it had a low boiling point and a high specific heat. These properties led to R-12 becoming the most popular refrigerant in the world, with production peaking at over 1 billion tons per year in the 1980s. ## The Montreal Protocol When searching for safer replacements to early refrigerants, scientists looked for substances which were highly stable, since stable substances pose less of a health risk to humans. This made R-12 an attractive choice, since it was highly stable and non-toxic. However, this stability also proved to be a problem. In the 1970s, scientists studying the ultimate fate of CFCs like R-12 discovered that these gasses could make their way up to the stratosphere. There, they could be broken down by ultraviolet radiation, releasing chlorine atoms (the first C in CFC and HCFC). The chlorine would then react with, and destroy, atmospheric ozone. In response to these findings, in 1987 the international community adopted the Montreal Protocol on Substances that Deplete the Ozone Layer. The agreement was initially signed by 46 countries, and has since been ratified by 198 parties, including all member states of the United Nations. It sets out a schedule for the phase-out of the production and consumption of ozone-depleting substances (ODSs), including CFCs like R-12 and HCFCs like R-22. The phase-out schedule differed for different substances and in different places. For instance, R-12 was phased out in developed countries in 1996, and in developing countries in 2010. Meanwhile, R-22, an HCFC common in home and commercial heat pumps, was phased out in developed countries in 2010, but it continues to be in-use in developing countries until 2030. ## GWP and the Kigali Amendment If you recall, one of the properties of a good refrigerant is its ability to absorb and release heat. Refrigerants continue to play this role even after they've been released into the atmosphere. In particular, they absorb and release heat in the form of infrared radiation, which is the same type of radiation that is absorbed and released by greenhouse gasses like carbon dioxide and methane. The heat-trapping ability of a greenhouse gas is measured by its global warming potential (GWP). The standard basis of comparison is the most common greenhouse gas, carbon dioxide (CO2), which has a GWP of 1. The GWP of Freon, or R-12, is 10,900. This means that, released into the atmosphere, R-12 traps 10,900 times as much heat as an equivalent amount of CO2. CFCs and HCFCs phased out under the Montreal protocol were often replaced by hydrofluorocarbons, which have many of the desirable properties of a refrigerant, but without the ozone-destroying chlorine. For example, an HFC called R-134a is a common replacement for R-12 in automotive air conditioners. However, R-134a has a GWP of 1,430 -- much less than R-12, but still substantially heat-trapping. When the Montreal protocol was adopted in 1987, the focus was on the ozone-depleting properties of CFCs and HCFCs. However, as the world began to phase-out these substances, it became clear that their global warming potential was also a problem. In response, in 2016 the international community adopted the Kigali Amendment to the Montreal Protocol. The Kigali amendment is meant to prevent additional global warming from refrigeration by phasing-out the production and consumption of hydrofluorocarbons (HFCs), which are the most common replacements for CFCs and HCFCs. The amendment has, so far, been ratified by [148 parties](https://ozone.unep.org/all-ratifications). Like the Montreal Protocol, the Kigali Amendment sets out a schedule for the phase-out of HFCs, but this schedule differs from place to place. For instance, for R-3 In the developed world, the phase-out began in 2019, and is scheduled to complete by 2036. In the developing world, however, the phase-out is not scheduled to end until 2045, with some countries receiving additional extensions to 2047 or later. ## How Recoolit Fits In Refrigerant pollution is a major global problem, and the world is working hard to solve it. However, as you can see, even the phase-out of ozone-depleting substances like R-22 will not be complete until 2030. With high-GWP HFCs like R-134a, the process has barely begun, and will last for decades. What do we do in the meantime? This is where Recoolit comes in. While the world works to phase out harmful refrigerants, we can work to reduce the amount of refrigerant pollution that is released into the atmosphere. Certainly, some of this release is accidental or the result of equipment malfunction. However, a significant portion of refrigerant pollution is the result of intentional venting. This is because, without Recoolit, technicians simply have no other way to dispose of refrigerant properly. Even well-intentioned technicians, who are concerned about pollution and environmental impacts, have no other choice when a system must be drained for maintenance or repair. We've seen technicians vent refrigerant through a hose inserted into a bucket of water, in the (mistaken) belief that this will somehow "scrub" the refrigerant. Additionally, the process of recovering refrigerant is time-consuming and expensive. The international protocols do not provide any funding for the transition to cleaner refrigerants, or to mitigate the damage from refrigerant pollution. This is where you come in. Now that you know about the problem, you can help us solve it. Fund our work by [buying our carbon credits](https://registry.recoolit.com/buy). Tell your friends and colleagues about the problem, and about our solution. ]]> <![CDATA[An Open Letter to GPE]]> https://igor.moomers.org/posts/open-letter-gpe https://igor.moomers.org/posts/open-letter-gpe Fri, 09 Sep 2022 00:00:00 GMT [my compulsion] was Beating The Crowd, finding the path of least resistance, filling the gaps, guessing the short queue, dodging the traffic, changing lanes with a whisper to spare -- moving with precision and grace and, above all, _expedience_. > I spied a queue ... that was slightly longer than the others, but I joined it and ticced nervously as I watched my progress relative to the other spots I could've chosen. > I was borne out, a positive omen for a wait-free World, and I was sauntering down Main Street, USA long before my ferrymates. Reading this passage, I realized that I was not alone in my weird obsession. I hate lines, the waiting in of, and like Jules I will obsessively attempt to pick the shortest queue to optimize my wait time. As you can imagine, Burning Man, with it's notorious entry and exodus lines, is a kind of special torture for me. I struggle constantly to remain patient and silence my overeager mind as it attempts to compute my rate of progress relative to my line-mates and my destination. I become short and distracted with my car mates, and annoyed at the perceived inefficiencies in the system. In order to channel my frustration into a productive direction, back in 2012 I joined the Gate, Perimeter, and Exodus (GPE) department. In the decade since, I've worked just about every position in the department -- apex, airport, lanes, perimeter, and this year I did a stint in the Traffic Operations Center (TOC). I worked enough shifts in 2019 for a staff-priced ticket, and enough this year for a staff credential for my next burn. To some extent, joining the department (as well as simply becoming a decade older and a little bit more patient) has helped me deal entry/exit crawl. When the line totally stops moving at 6 pm, I know that there's a shift change, and we'll get moving soon, and I can reassure people around me about this. What I hoped for, however, was a sense that there's some giant master plan that's helping to move me out of a 10-hour traffic jam as quickly as possible. This sense continues to elude me, and is the reason I'm writing this open letter. I do not mean to throw shade on any people in the department. I understand first-hand how difficult these shifts are, standing out in a white-out in 100+ degree heat, breathing dust and exhaust, dealing with cranky and cantankerous burners while trying not to get crushed by traffic. I also understand the difficulty that the leaders of the department have, doing this thankless job in exchange for meal pogs and t-shirts, trying to take care of their volunteers while balancing the demands of the org, various state agencies, and the forces of pure chaos. But I cannot help but get the sense that something is not working quite right. I am 200% open to the possibility that, actually, everything is going as well as possible, and it's just that my spidey sense is off here. I know that when other department members express concerns on the gate list, they sometimes get (vehemently) dismissed as "armchair quarterbacks" who, because they weren't working that specific shift or that specific role, have no right to an opinion. But in this case, I think it's a communication problem. If even long-time department volunteers (including this one) are frustrated and confused by the operation of the department, what of the participants? These are the people who, after being stuck in a 10-hour traffic jam for inscrutable reasons, unload their anger on the front-line Exodus volunteers. Everyone here deserves better -- and even if this is *just* a communication problem, that's still a real problem that, I believe, demands attention. ## Entry My entry into Burning Man was more than 3 hours long -- on Wednesday pre-event! As others have noted on the gate list, the second and third lines from the left on the way in merged right before apex, meaning those two lines moved half as fast as the other lines. I attempted to reassure my car mate for almost an hour that this must be an illusion, and that no line moves faster through Apex than any other line, but my poor choice of lanes cost me a lot of extra wait time, and anxiety that we would miss our camp's placers for the night and would have to sleep in our truck instead of setting up camp. The department doesn't want participants changing lanes, because it messes up the cone placement. The best way to prevent lane-jumping is to make sure the lanes are actually fair! Next, I had to pick up my staff credential from the box office. On the way out of the Will-Call lot, we again had a massive line stretching back into the lot. Two volunteers at the front were letting cars out of Will-Call, only into the two rightmost lanes, which Apex was also using. This resulted in an additional 30 to 45 minutes of wait out of the lot. It was incredibly frustrating to have waited at the Apex line already, only to have cars that arrived hours after us hit the lanes ahead of us. Apex can hold traffic and empty out the Will-Call line. Why not do this? ## TOC I worked a TOC shift on Sunday of Temple Burn. For 6 hours, I had control of [this twitter account](https://twitter.com/bmantraffic) -- the first time I've ever posted anything on Twitter! This was my chance to understand exodus, and to help tens of thousands of people to leave the Burn in the easiest way possible. I do not believe that, through any of my tweets, I helped anyone make better decisions. Some of my tweets were of the inane variety, like "don't break down in the lanes". I had a campmate whose car broke down on the way in, and he was just going to leave when it was time to leave -- the tweets had no bearing on his decision. The most useful tweets could have been wait time estimates, and I believe that some of those were purely made up. I tweeted that the wait was [6 hours](https://twitter.com/bmantraffic/status/1566535630493913089), and tweeted again that the wait was [1 hour](https://twitter.com/bmantraffic/status/1566539015972474881) just 20 minutes later! The amount of information coming to the TOC from folks on the ground is miniscule -- reporting wait times is not a huge priority, even though this is the one number everyone in the city wants to know. We communicated this number to BMIR *maybe* twice during my six-hour shift, and once it was only because we had [totally stopped all traffic](https://twitter.com/bmantraffic/status/1566571695158153216) and asked them to report it. I didn't have a radio in my truck, but campmates reported that BMIR was more interested in their programming than in keeping folks informed about exodus -- just was well, since I don't believe they have any special information. At least on Sunday afternoon, *nobody* seemed to have any accurate information about the wait time in Exodus. My TOC shift lead was warm, personable, efficient on the radio, and a great person. However, I was sometimes shocked at their claims and decisions. At the beginning of my shift, someone radioed into TOC, asking, "BMIR is reporting a 3.5 hour wait time, do you know where they got that number"? My shift lead said, "Yeah, that makes sense, because gate road is 4 miles long and the speed limit is 5 miles an hour." After some back-and-forth, the person on the radio signed off, unconvinced by this obviously faulty math, but clearly unwilling to continue taking up airtime. My shift lead then told me to [tweet that number](https://twitter.com/bmantraffic/status/1566522222767812608)! ## Exodus I left on Sunday night, shortly after my TOC shift ended. Because I sent [this tweet](https://twitter.com/bmantraffic/status/1566539015972474881) about using all the lanes, I was able to satisfy my compulsion and skip a ton of traffic by using the empty lanes. The pulses themselves, however, drove me nuts. Specifically, in all my pulses, I observed the final portion of gate road completely draining of all cars -- nobody else merging onto the highway. Exodus would then continue to hold traffic for anywhere from 20 to 40 minutes, for no reason I could discern. I sort of assumed this was because traffic was getting stuck on 447, but my brothers in black running Exodus that night assured me this was not happening. It was sad to watch participants get into their vehicles and start them as they watched the road drain, then shut them down again 5 minutes later when the line showed no signs of movement. Keeping these people informed would help everyone. ## Wrap-Up My exodus took 9 hours (though it would've been 7 if my janky borrowed truck hadn't refused to start at the beginning of my last pulse!) To everyone who helped out, including volunteers who brought me a jump box -- THANK YOU!) I know people who spent 11 hours in the full heat of Monday afternoon. These anecdotes are, sadly, the best data we have! Where are the charts of wait time by departure time over the years of exodus, the charts that would help level out traffic flows out of the city? If your answer is "leave Tuesday morning", then you clearly don't understand the reality of participants who have to get back to their lives and jobs, but first must drive a theme camp home, unload it, and de-dust it. I can anticipate the reaction to this essay among the GPE crew. At worst, I'll be dismissed as ignorant, not someone who trully understands the process and the constraints. Guity! To these people, I would say that, if a 10-year department veteran is confused and frustrated by gate operations, then there's a real problem here. At best, some GPE folks will tell me that, if I think I can do better, I should sign up -- as though there's a slot in the shift system for "Run everything as you see fit". My goal is not to blindly criticize, but to contribute constructively. Ultimately, I would love to just relax and enjoy the Gate process. I think a little bit of information would help me and the other participants to do just that. ]]> <![CDATA[Peter Eckersley]]> https://igor.moomers.org/posts/peter-eckersley https://igor.moomers.org/posts/peter-eckersley Thu, 08 Sep 2022 00:00:00 GMT <![CDATA[The War in Ukraine]]> https://igor.moomers.org/posts/war-in-ukraine https://igor.moomers.org/posts/war-in-ukraine Mon, 07 Mar 2022 00:00:00 GMT crude oil accounted for $110.2 billion, oil products for $68.7 billion, pipeline natural gas for $54.2 billion and liquefied natural gas $7.6 billion ([source](https://www.reuters.com/markets/europe/russias-oil-gas-revenue-windfall-2022-01-21/)). So, even if we assume all pipelines stop flowing, Russia will still find a ready export market for the remaining 80% of it's fossil fuels. While the conflict might accelerate renewables transition in the EU, it also has a bunch of negative knock-on effects. Most especially, it makes burning coal a better deal -- something [the EU is already struggling with](https://www.newyorker.com/magazine/2022/02/07/can-germany-show-us-how-to-leave-coal-behind). In addition, fracking becomes more profitable again, as does refining poor-quality petroleum such as tar sands. The US administration was already walking a fine line between advocating a renewables shift while also protecting low gas prices at all costs. The calculus there has become harder, and the administration will face tough choices. Approve more drilling and face the anger of the base? Deny new oil and gas leases, and face the political consequences? ## Political Opportunity ## This war is very prominent in public conciousness, and people want a frame in which to think about it. I recommend using the following frame whenever talking to anyone about this conflict. __This War Is a Climate War__. Putin is just one more character in the cast of fossil-fuel-powered dictators. He's been wreaking havoc on the world stage for 20 years, and the world rolled over and took it because he supplied the gas. Every bullet shot at a Ukrainian citizen, every missile hitting a residential building, every tank and warplane was paid for when Westerners pumped their gas and turned up their heat. __Ukrainian refugees are climate refugees__. They were displaced because of fossil fuels. They join migrants from the Syrian civil war, and the people coming to the US from South and Central America -- all people displaced by our addiction to fossil fuels. __The climate crisis is a refugee crisis__. Millions of Ukrainians fleeing to the EU is a foreshadowing of what's to come. We are reacting today in the best possible climate -- a very unpopular war, a very definite and very unpopular aggressor, very sympathetic (because white, Christian) victims. How we treat these folks is the best-case scenario, and it [could be better](https://www.reuters.com/world/europe/britain-may-ease-immigration-rules-ukrainian-refugees-sun-2022-03-07/?taid=62260ffc18c5730001d4f729). ]]> <![CDATA[Words I Avoid]]> https://igor.moomers.org/posts/words-i-avoid https://igor.moomers.org/posts/words-i-avoid Fri, 24 Sep 2021 00:00:00 GMT <![CDATA[Kitchen Table Decisions]]> https://igor.moomers.org/posts/kitchen-table-decisions https://igor.moomers.org/posts/kitchen-table-decisions Mon, 09 Aug 2021 00:00:00 GMT <![CDATA[x86_64 on an Apple M1 MacBook]]> https://igor.moomers.org/posts/navigating-arch-on-osx https://igor.moomers.org/posts/navigating-arch-on-osx Tue, 11 May 2021 00:00:00 GMT BUILD FAILED (OS X 11.2.3 using python-build 1.2.27-29-gfd3c891d) <-- snip --> configure: error: Unexpected output of 'arch' on OSX ``` This still does not work if you explicitly specify an arch: ```bash $ arch -x86_64 asdf install python 3.8.9 python-build 3.8.9 /Users/igor47/.asdf/installs/python/3.8.9 python-build: use openssl@1.1 from homebrew python-build: use readline from homebrew <-- snip --> Installing Python-3.8.9... python-build: use readline from homebrew python-build: use zlib from xcode sdk WARNING: The Python readline extension was not compiled. Missing the GNU readline lib? ERROR: The Python ssl extension was not compiled. Missing the OpenSSL lib? BUILD FAILED (OS X 11.2.3 using python-build 1.2.27-29-gfd3c891d) ``` You'll need to install the `x86_64` versions of readline and openssl to make progress! The second install of brew to the rescue (notice, I'm using my alias, `brow` and not `brew`, below): ``` $ brow install readline $ brow install openssl $ brow install xz ``` Now, you need those libraries in your linker and compiler. I figured I might need to do this again, so I am using my [direnv](https://github.com/asdf-community/asdf-direnv) setup to make this easier. I created a directory -- `mkdir ~/x86_64` -- and added a `.envrc` that looks like this: ```bash use asdf export BROW="/usr/local/homebrew" export LDFLAGS="-L${BROW}/opt/openssl@1.1/lib -L${BROW}/opt/readline/lib -L${BROW}/opt/xz/lib" export CPPFLAGS="-I${BROW}/opt/openssl@1.1/include -I${BROW}/opt/readline/include -I${BROW}/opt/xz/include" export PATH="${BROW}/bin:$PATH" export ARCHPREFERENCE="x86_64" ``` A quick `direnv allow`, and now, when I `cd ~/x86_64`, I am in a good place to do Rosetta-type things: ```bash $ uname -a Darwin planetarium.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 arm64 $ cd ~/x86_64 direnv: loading ~/x86_64/.envrc direnv: using asdf direnv: loading ~/.asdf/installs/direnv/2.28.0/env/3601927912-4079855243-2014015008-304321844 direnv: using asdf rust 1.52.1 direnv: using asdf python 3.8.9 direnv: using asdf direnv 2.28.0 direnv: export +ARCHPREFERENCE +CARGO_HOME +CPPFLAGS +LDFLAGS +RUSTUP_HOME ~PATH $ arch uname -a Darwin planetarium.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 x86_64 ``` Now, from this directory, I can easily install my Python. I didn't have to specify `-x86_64` to `arch`, because this is already set by my `ARCHPREFERENCE`. ```bash $ arch asdf install python 3.8.9 <-- snip --> Installed Python-3.8.9 to /Users/igor47/.asdf/installs/python/3.8.9 ``` ]]> <![CDATA[Arch Linux Configuration]]> https://igor.moomers.org/posts/arch-linux-config https://igor.moomers.org/posts/arch-linux-config Wed, 27 Jan 2021 00:00:00 GMT <![CDATA[Minimum Viable Air Quality Monitoring]]> https://igor.moomers.org/posts/minimal-viable-air-quality https://igor.moomers.org/posts/minimal-viable-air-quality Sun, 13 Sep 2020 00:00:00 GMT <![CDATA[Kubernetes for ETL]]> https://igor.moomers.org/posts/building-etl-kubernetes https://igor.moomers.org/posts/building-etl-kubernetes Thu, 10 Sep 2020 00:00:00 GMT None: pass class JobType(Protocol): perform: PerformType ``` We then created a standard `main.py` entrypoint which would accept parameters like the job name and options, and dispatch them to the correct job class. The Kubernetes work is a little more complicated. The jobs look basically the same, but have different arguments -- in K8s-speak, the pod spec container args -- are different. The solution is to template the K8s manifests, but templating a data structure (in this case, `yaml`) like a string (e.g., with [jinja](https://jinja.palletsprojects.com/en/2.11.x/)) is a recipe for disaster, and popular tools like [jsonnet](https://jsonnet.org/) seem quite heavyweight. We managed to find [json-e](https://json-e.js.org/), which hit just the right note. This allows you to template and render a pod manifest: ```yaml spec: containers: - name: {$eval: 'container_name'} image: {$eval: 'image'} args: {$eval: 'args'} ``` with something like: ```python pod_manifest = jsone.render( YAML(typ="safe").load('pod_manifest.yaml'), { 'container_name': job_name, 'image': 'latest', 'args': [job_name, start_time, end_time, job_config] } ) ``` You end up with a valid pod manifest as a data structure. You can then render it into your `CronJob` manifest: ```yaml apiVersion: batch/v1beta1 kind: CronJob spec: schedule: {$eval: schedule} jobTemplate: spec: template: {$eval: pod_manifest} ``` in the same way: ```python cron_manifest = jsone.render( YAML(typ="safe").load('cron_manifest.yaml'), { 'schedule': '0 2 * * *', 'pod_manifest': pod_manifest, } ) ``` The resulting `cron_manifest` can be serialized to YAML again, and passed to `kubectl apply` to update your resources: ```python with NamedTemporaryFile() as ntf: ntf.write(YAML(typ="safe").dump(cron_manifest)) ntf.flush() os.system(f"kubectl apply -f {ntf.name}") ``` Of course, we updated our `inv` tooling to support gathering input from users about which jobs they wanted to deploy, and with which arguments. We also provided helpers for the `main.py` dispatcher, so you could specify arguments like `yesterday` to a job and have it run over the previous day. ## Inter-job Dependencies ## Soon enough, we had dozens of jobs, and they had inter-job dependencies. You can only go so far by saying "job A runs at 2 am, and usually finishes in an hour, so we'll run job B at 3:30 am". This was time, again, to re-examine existing popular open-source tooling, and we seriously considered [Argo](https://argoproj.github.io/argo/). But again, some functionality was missing -- specifically, we got bit by [this bug](https://github.com/argoproj/argo/issues/703#issuecomment-494183536) preventing the templating of resources, which was a step back from being able to template any part of the manifest using `json-e`. Instead, we ended up creating two new concepts. The first, a `workflow`, listed all the jobs that depended on each other, along with their dependency relationships, encoding a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph). It's easy to parse a YAML list, and verify that it is indeed a DAG, with [networkx](https://networkx.github.io/documentation/stable/reference/algorithms/dag.html). Creating a workflow specification also gave us a place to keep track of standard job parameters, such as command-line options or resource requirements for a job. (As an aside, resource management for jobs was a constant chore, especially as the product was scaled up and data volumes increased.) The second concept was a `dispatcher` job. Instead of actually doing ETL work, this type of job, when dispatched using a `CronJob` resource, would accept a workflow as an argument, and then dispatch all the jobs in that workflow definition. A job would not get dispatched until the jobs it depended on completed successfully. This allowed us to schedule a workflow to run daily instead of scheduling a pile of jobs individually. Most of the code was re-cycled from the code already used by users locally to schedule their jobs -- the `dispatcher` was doing the same thing, just from *inside* Kubernetes. ## Monitoring via Web UI ## As the complexity of the ETL pipeline grew, we began encountering workflow failures. We needed an audit log of which jobs ran for which days, so we could confirm that we were delivering the data we processed. We also needed tooling to reliably re-process certain days of data, and keep track of those re-processing runs. By this point, the ETL system we were building had run for a year without requiring any UI beyond the one provided by GKE and `kubectl`. However, to manage the complexity, it seemed like a UI was needed. I ended up building one using [firestore](https://cloud.google.com/firestore/), React, and [ag-grid](https://www.ag-grid.com/). Previously, the `dispatcher` job that ran workflows was end-to-end responsible for all the jobs in the workflow. It ran for as long as any job in the workflow was running. If any of the jobs failed the `dispatcher` would exit, and the subsequent jobs would not run. Likewise, if `dispatcher` itself failed, remaining workflow jobs would be left orphaned. Instead, we turned the workflow `dispatcher` into something that merely manipulated the state in `firestore`. The job would parse the workflow `.yaml` file into a DAG, and then create `firestore` entries for each job in the graph. The `firestore` entry would include job parameters like command-line arguments and resource requests, as well as job dependency information. Jobs at the head of the DAG would be placed into a `SCHEDULED` state, while jobs that were dependent on other jobs were placed into a `BLOCKED` state. An always-running K8s service called `scheduler` would subscribe to `firestore` updates and take action when jobs were created or changed state. For instance, if a job was in the `SCHEDULED` status, the `scheduler` would create pods for those jobs via the K8s API, and then mark them as `RUNNING`. If a task finished (by marking itself as `COMPLETED`), the `scheduler` would notice, and clean up the completed pods. It would also check if any jobs were `BLOCKED` waiting on the completed job, make sure all their dependencies had completed, and placed them into the `SCHEDULED` state. If a job marked itself as `FAILED` (via a catch-all exception handler), we had the option to track retries and re-schedule the job. Because the `scheduler` became the only thing that interacted with the K8s API, it made building user-facing tooling easier. Those tools merely had to manipulate the DB state in `firestore`. This enabled less technical users, without `gcloud` or `kubectl` permissions, to create, terminate, restart, and monitor job and workflow progress. From what I can tell, components like the `scheduler` are often built using K8s [operators](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) and [custom resources](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/). However, we found that just running a service with permissions to manage pods is sufficient. This avoids having to dive too deep into K8s internals, beyond the basic API calls necessary to create and remove pods and check on their status. ## Parallelism ## Some jobs in our system were trivially parallelise-able, e.g. because they processed a single sensor's data in isolation. The system of using the `scheduler` to run additional jobs unlocked infinite parallelism inside jobs. For instance, we could schedule a job like `ProcessAllSensors`. This job would first list all sensors active during the `start`/`end` interval, and then could create a child job in `firestore` for each sensor. Creating child jobs was as simple as writing a job entry into `firestore`, with the ID of the sensor to process. Parallelism was limited only by the auto-scaling constraints on the K8s cluster. I created an abstraction called `TaskPoolExecutor`, based on the existing [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor). Each job submitted to the `executor` would run in a different K8s pod. Not only did this make data processing much faster, but more resilient, too. Previously, if a particular sensor failed in it's processing, restarting that processing was complicated. In the new system, the existing retry system baked into the `scheduler` could retry individual sensors, without the parent job even knowing about it. ## Takeaway ## My focus at Aclima was on business objectives -- making sure data is delivered, data scientists and other technicians are productive, and we can manage our fleets of vehicles and devices. In 18 months, I was able to build an infinitely-auto-scalable, reliable parallel job scheduling system and UI accessible to non-technical users. I was able to build this one step at a time, from "how do I regularly run one cron job" all the way to "how to I parallelize and monitor a run of a job graph over a year of high-precision sensor data". I think this is a tribute to the power of the abstractions that Kubernetes provides. It's an infinite pile of compute, and it's pretty easy to utilize it. This experience has sold me on the premise, and I would definitely use Kubernetes for other projects again. ## Do Differentlys ## The scheduler and all the code that interacts with K8s was written in Python, mostly because the data processing code was also written in Python. However, as soon as I wanted to add a UI, I had to begin writing Javascript. This means I have to share at least data structures -- for instance, the structure of a `job` in firestore -- between the two languages. In the future, if I'm writing a UI, even a CLI, I will consider strongly whether I should just write it in JS to begin with. ]]> <![CDATA[AWS MFA on the CLI with `direnv`]]> https://igor.moomers.org/posts/aws-mfa-cli-direnv https://igor.moomers.org/posts/aws-mfa-cli-direnv Fri, 28 Aug 2020 00:00:00 GMT export MASTER_AWS_SECRET_ACCESS_KEY= export AWS_MFA_ARN=arn:aws:iam:::mfa/igor export AWS_SESSION_FILE="${HOME}/.config/aws/session-${MASTER_AWS_ACCESS_KEY_ID}" watch_file $AWS_SESSION_FILE direnv_load ~/bin/aws_load_session ``` This file causes 4 environment variables to be exported into my environment whenever I `cd` into this repo's directory. The `MASTER_AWS_ACCESS_KEY_ID` and `MASTER_AWS_SECRET_ACCESS_KEY` are just the access key ID and key that I created for my account via IAM. I've prefixed their usual environment variable names with `MASTER` to distinguish them from the session-specific keys created by authenticating with MFA. The `AWS_MFA_ARN` variable contains the ID of my MFA token. You can get this from [your security credentials page](https://console.aws.amazon.com/iam/home#/security_credentials), under the `Multi-factor authentication (MFA)` section. Finally, the `AWS_SESSION_FILE` variable will keep track of where my MFA session is stored in my filesystem. The next two lines handle reloading the MFA session. I've told `direnv` to reload my local environment whenever the contents of the file at `$AWS_SESSION_FILE` change. Next, we use `direnv_load` (from the [direnv stdlib](https://direnv.net/man/direnv-stdlib.1.html)) to load the environment exported by my `aws_load_session` script. ## Session-management Scripts ## I have two custom scripts to manage the MFA session. The first is `aws_get_session`, and it's responsible for prompting me for my MFA token, creating an MFA session, and storing it into the `AWS_SESSION_FILE`. I run this script whenever my MFA session expires. Here's the script: ```bash #!/bin/bash TOKEN=$1 shift if [[ -z $TOKEN ]]; then echo "Usage: aws_get_session " exit 1 fi set -u mkdir -p `dirname ${AWS_SESSION_FILE}` unset AWS_SESSION_TOKEN AWS_ACCESS_KEY_ID=${MASTER_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${MASTER_AWS_SECRET_ACCESS_KEY} aws sts get-session-token --serial-number $AWS_MFA_ARN --token-code ${TOKEN} > ~/.config/aws/session-${MASTER_AWS_ACCESS_KEY_ID} > ${AWS_SESSION_FILE} echo "saved session info to ${AWS_SESSION_FILE}" ``` The other script, `aws_load_session`, loads the MFA session into my environment. It's run by `direnv`, whenever the `AWS_SESSION_FILE` changes. Here's the script: ```bash #!/bin/bash set -u if [[ ! -f ${AWS_SESSION_FILE} ]]; then echo "No session found; did you run `aws_get_session ` ?" fi export AWS_ACCESS_KEY_ID=`cat ${AWS_SESSION_FILE} | jq --raw-output .Credentials.AccessKeyId` export AWS_SECRET_ACCESS_KEY=`cat ${AWS_SESSION_FILE} | jq --raw-output .Credentials.SecretAccessKey` export AWS_SESSION_TOKEN=`cat ${AWS_SESSION_FILE} | jq --raw-output .Credentials.SessionToken` direnv dump ``` Both of these scripts depend on having `aws` and `jq` installed and in your `PATH`. ## Example Session ## Here's how this looks in real use, with a terraform repo that stores it's state in AWS S3. ```bash igor47@fortress:~/repos/terraform/roots/prod {master} $ terraform plan Error: error using credentials to get account ID: error calling sts:GetCallerIdentity: ExpiredToken: The security token included in the request is expired status code: 403, request id: abfd729b-4dad-41f4-857f-2539170f68a9 igor47@fortress:~/repos/terraform/roots/prod {master} $ aws_get_session 123456 saved session info to /home/igor47/.config/aws/session-AKIA direnv: loading ~/repos/terraform/.envrc direnv: using asdf direnv: loading ~/.asdf/installs/direnv/2.21.2/env/733966593-20565860-1008169379-2914714444 direnv: using asdf python 2.7.18 direnv: using asdf python 3.8.3 direnv: using asdf nodejs 12.13.1 direnv: using asdf ruby 2.7.1 direnv: using asdf direnv 2.21.2 direnv: using asdf terraform 0.12.29 direnv: export +AWS_ACCESS_KEY_ID +AWS_MFA_ARN +AWS_SECRET_ACCESS_KEY +AWS_SESSION_FILE +DD_API_KEY +DD_APP_KEY +MASTER_AWS_ACCESS_KEY_ID +MASTER_AWS_SECRET_ACCESS_KEY +NPM_CONFIG_PREFIX +RUBYLIB ~AWS_SESSION_TOKEN ~PATH igor47@fortress:~/repos/terraform/roots/prod {master} $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. ``` Here, `terraform plan` fails because my MFA session has expired. I re-run `aws_get_session` to update my `AWS_SESSION_FILE`. `direnv` notices that the file has been updated, and reloads the environment. I can then continue using `terraform` as normal. As a bonus, in any other shell, the session will also be re-loaded automatically whenever I get a new prompt. ]]> <![CDATA[Website Scalability]]> https://igor.moomers.org/posts/website-scalability https://igor.moomers.org/posts/website-scalability Tue, 04 Aug 2020 00:00:00 GMT Hello, World!` and sends it back to your browser, which displays it. How does this app scale? ### Caching ### This initial version of the app is quite static, and you could utilize caching to help you scale. Caching here would involve routing requests to your app through a CDN, like Akamai or Cloudflare or Cloudfront. The CDN would keep a copy of your “Hello World”, and when visitors ask for it, it would come from the CDN’s servers, not your server. Suppose your app, instead of printing “Hello World”, printed `“Hello World, today is ”`. This is still almost entirely static – the text only changes once a day. You will need to carefully configure cache-control headers, so that the CDN mostly serves the file from it’s servers, but will occasionally come back to your servers to retrieve an updated version of the page. ### Horizontal Scaling ### Suppose your app prints out `“Hello, World! It's :: where I’m at.”.` Also, assume you have to render this server-side (as a client-side app, you could do the rendering in JS, and then this app just become the perfectly-cached one from the section above). This app would be perfectly horizontally scalable. You do this by putting a load balancer in front of your server, and then adding more, identical web servers as needed. If you run a single-threaded web server, it can only serve one visitor at a time. When a second visitor visitor shows up while your server is busy rendering “Hello, World” for the first visitor, the second visitor has to wait – to get into a queue. If you have a lot of visitors showing up at the same time, the queue will get quite long – and then some of those visitors will give up before seeing “Hello, World”. Some of them give up subjectively, because the page isn’t doing anything, and for the really patient ones their browser will give up for them, eventually. If you were running your single-threaded web server on a 16-CPU machine, or if the reason it took 200ms to serve “Hello, World” is because you called `sleep(0.196)` in your web server code, your server would not *look* overloaded. You wouldn’t see excessive CPU or memory or IO usage. The only way to tell that visitors are giving up is by looking at metrics, such as queue size, timed-out connections, number of requests at the load balancer vs. your web app, etc. If you looked at these metrics, and discovered that you were in fact failing to greet a bunch of visitors, then you might engage in performance engineering. You would say, “why does it take 200ms to serve this page?" If it took less time, we'd be able to serve more visitors total. This might lead you to finding the `sleep(0.196)`, or switching to a more performant programming language or architecture. Alternatively, you might realize that you have a bunch of under-utilized resources on your server – say, 15 additional CPUs – and then you might use a threaded or forked web server to serve more visitors concurrently. OR – you might not do any of that. Instead, you might just decide, since your app is so horizontally scalable, to launch a bunch more web servers behind your load balancer. The end result would be the same – more satisfied visitors to your site. But you might end up paying more money for all those extra servers. However, you would save all the time and money you spend combing through the code looking for sleep, or tuning your server software for concurrency. This is a relevant trade-off, and should be considered when you want to scale your web app. ### State and Vertical Scaling ### Suppose that when a visitor came to your page, you would log their IP address and the time of their visit. Then, you would display either “Hello, visitor, for the first time!” or “Hello, visitor, welcome back!”, depending on whether you had or had not previously seen their IP address. This version of the web app is stateful – it retains state between requests -- and thus is no longer purely horizontally scalable. To realize this, think about where you would store the state. You could store it directly on the server which renders “Hello”. But, if scale by launching additional servers, then a visitor might get different servers on different requests, and would get the wrong “Hello” message and be sad. Typically, looking up some data in a database is much faster than rendering a complicated web page, and so you end up with a two-tier web infrastructure. The first tier is a bunch of horizontally scalable, stateless web servers. These may have been optimized to some extent, or else just scaled as needed, according to the necessary trade-off (discussed above). The second tier is the database, which is heavily optimized to be as fast as possible. Web optimizations are harder, because each website is different, while “retrieve some data” is pretty generic and can be iteratively optimized over time. However, how the database server can no longer be easily scaled horizontally. Once you are using all the memory and all the CPUs on your database, you would be up against a wall. In thise case, ou might decide to scale vertically – by getting a bigger, badder, beefier database server. In this scenario, your goal is to squeeze the most possible out of the resources on your database server. You’ll want to watch for high CPU usage, running out of memory, or saturating your disk IO or network buses. These would alert you that you need to either scale your database server, or reduce the load from your application. ### Scaling through load shedding ### Suppose that your simple site now has two pages. The first page is that same one that says “Hello, visitor, for the first time!” or “Hello, visitor, welcome back!”, while the second page shows a cute random kitten. The kitten page scales horizontally (each server has it’s own kitten repository), but the “Hello” page still requires a DB read/write before you can render it. Suppose that your DB server is having trouble keeping up with all the demand for the “Hello” page. As the DB server slows to a crawl, the “Hello” page takes longer and longer to render. The kitten page would keep working quickly but, alas, visitors to the kitten-page are stuck in line behind the “Hello”-page visitors, and nobody is getting either “Hello”ed or kittened. Kind of how an escalator that fails becomes stairs, your kitten-and-hello site, when it fails, should become just a kitten-showing site, instead of no site at all. You can do that if you quickly turn away the “Hello” visitors when you realize that the DB server is having problems. This can be accomplished through automation, called “circuit breaking”, in the DB layer; as a bonus, it might even help the DB layer automatically recover from transient spikes. This pattern is often useful with dependencies that scale neither horizontally nor vertically. For instance, suppose that a page on your site will offer visitors the ability to enter their phone number, and then recieve a “Hello, World” as a text message through Twillio. Twillio becomes a dependency, but you can neither add more horizontal Twillio capacity, nor increase the size of Twillio’s machines then they’re overloaded. In fact, the only way you know that Twillio is overloaded is when your own site is totally down – visitors can’t get a “Hello” page and they can’t get a kitten, because there are too many people waiting for Twillio API calls that never complete. If you used a circuit breaker around calls to Twillio, then you might be able to quickly show those visitors an error message, while the other visitors continue getting “Hello” and kittens. Coming full circle, caching can also be a form of load shedding. You may be able to serve visitors a cached version of the page which is not strictly correct (for instance, the time on the "Hello" page is stale), but still more useful than an error page. ### Dependencies as bottlenecks ### Life was easy for your awesome “Hello, World” site, so long as you could just horizontally scale it. But, as soon as you began introducing dependencies – DB servers, Twillio APIs – things got a whole lot more annoying. What’s clear is that any dependency ought to be treated with suspicion – these are what’ll getcha on big launch night. In fact, only three things will break your site: * You yourself, by accidentally breaking it. This usually happens through a deploy. * Some malicious actors, by breaking it on purpose * A non-horizontally-scalable dependency of your site, which breaks suddenly in response to increased traffic If you want your site to stay up, you have to engineer around all three failure causes. ]]> <![CDATA[Syncthing]]> https://igor.moomers.org/posts/my-syncthing-setup https://igor.moomers.org/posts/my-syncthing-setup Sat, 25 Jul 2020 00:00:00 GMT <![CDATA[Reliable SSH Tunnel for Raspberry Pi]]> https://igor.moomers.org/posts/embedded-system-ssh-tunnel https://igor.moomers.org/posts/embedded-system-ssh-tunnel Sat, 03 Aug 2019 00:00:00 GMT <![CDATA[Different Approaches to Environmentalism]]> https://igor.moomers.org/posts/differences-in-environmentalism https://igor.moomers.org/posts/differences-in-environmentalism Mon, 13 May 2019 00:00:00 GMT Fossouo was born in Cameroon and went to school in Paris, but his real education seems to have come in the seven summers he spent in the United States selling books for Southwestern Publishing, a Nashville-based titan of door-to-door marketing. (Rick Perry is another alum; ditto Ken Starr.) “I did Los Angeles for years,” he said. “‘Hi, my name is Max. I’m a crazy college student from France, and I’m helping families with their kids’ education. I’ve been talking to your neighbors A, B, and C, and I’d like to talk to you. Do you have a place where I can come in and sit down?’” > > All selling, he insists, is the same: “It starts with a person understanding they have a problem. Someone might live in the dark but not understand it’s a problem. So, you have to show them. And then you have to create a sense of urgency to spend the money to solve the problem now.” > > … > This prospect is a farmer and a schoolteacher, and we settle down in his classroom, which has a few low desks with slates—literal shards of slate—resting on top. Max quickly figures out that the man has two wives, and he starts sprinkling their names liberally through the conversation. “There’s no pressure. It’s okay. I don’t want to sell you anything,” he says, as they move through the steps familiar to anyone who’s seen an infomercial. > > … > The customer is resistant, but Max tries angle after angle. “You have to think big here. When I talked to your chief, he said, ‘Don’t think small.’ If your kid could see the news on TV, he might say, ‘I, too, could be president.’” > “This is great,” the man says. “I know you’re trying to help us. I just don’t have the money. Life is hard, things are expensive, sometimes we’re hungry.” > Max nods, helpful. “What if I gave you a way to pay for it, so the dollar wouldn’t even come from your pocket. If you get a system, people will pay you to charge their phones. Or, if you had a TV, you could charge people to come watch the football games.” > “I couldn’t charge a person for coming in to watch a game,” the man says. “We’re all one big family. If someone is wealthy enough to have a TV, everyone is welcome to it.” It was super-interesting to read this section after the following section, from *Climate: A New Story*. I think comparing them will give you a sense for the different perspectives: > Economic growth means the growth in goods and services exchanged for money. Therefore, a remote village in India or a traditional tribal area in Brazil presents a big growth opportunity, because the people there barely pay for anything. They grow or forage their own food. They build their own houses. They use traditional healing methods to treat their sick. They make their own music and drama. Imagine the development expert goes there and says, “What a tremendous market opportunity! These backward people grow their own food—they could buy it instead. They cook their own food too—restaurants and supermarket delis could do it for them much more efficiently. The air is full of song—they could buy entertainment instead. The children play with each other for free—they could enroll in day care. They accompany adults learning traditional skills—this society could pay for schooling. When a house burns down, the community gets together to rebuild it—if we can unravel those ties of mutual aid, there’s a big market for insurance. Everyone has a strong sense of social identity, a strong sense of belonging—they could buy brand name products instead. Everyone is joyful and content—they could be buying a semblance of that through legal and illegal drugs and other forms of consumption.” > > Okay, I’m getting a bit dizzy with visions of riches, but you get the idea. The question is: how are these people going to pay for all that? Easy. They earn money by converting local natural resources and their own labor into commodities. The rainforest becomes a palm oil plantation. The mountain becomes a strip mine. The river becomes a hydroelectric plant. The population abandons their traditional ways and goes to work in the money economy. A few become doctors, lawyers, and engineers. The rest migrate to the slums. > > In a nutshell, this is the process called “development.” It is what development loans have funded for more than half a century. It accompanies an ideology that says that money equates with well-being, that development along the model of the West is a good thing (or an inevitable thing), that a high-tech life is superior to a life close to nature. These assumptions are difficult to refute using logical arguments. Usually, shedding them requires spending time in less developed cultures, witnessing the joy and depth of aliveness there, and seeing their beauty erode as they modernize. There's definitely more to say about this, but I wanted to note this down while it's in my mind. ]]> <![CDATA[Individual Action and Climate Change]]> https://igor.moomers.org/posts/individual-action-and-climate https://igor.moomers.org/posts/individual-action-and-climate Sat, 11 May 2019 00:00:00 GMT There are numerous and varied ways to address key waste points. > In lower-income countries, improving infrastructure for storage, processing, and transportation is essential. > In higher-income regions, major interventions are needed at the retail and consumer levels. > National food-waste targets and policies can encourage widespread change. It seems that to help, you should either be a grocery magnate or a politician. Failing that, you might try to lobby or influence your local politicians or [greengrocers](http://openproduce.org/#contact). This view is well-expressed by David Wallace-Wells in his best seller "Uninhabitable Earth": > …the climate calculus is such that individual lifestyle choices do not add up to much, unless they are scaled by politics. Later on: > accusations of individual irresponsibility were a kind of weaponized red herring, as they often are in communities reckoning with the onset of climate pain. > We frequently choose to obsess over personal consumption, in part because it is within our control and in part as a very contemporary form of virtue signaling. > But ultimately those choices are, in almost all cases, trivial contributors, ones that blind us to the more important forces. What are the "more important forces"? Politics: > Eating organic is nice, in other words, but if your goal is to save the climate your vote is much more important. Wallace-Wells is particularly dismissive of individual action, even calling it "virtue signaling". Bill McKibben, founder of 350.org, expressed a more charitable version of this view on a [recent episode of KQED Forum](https://www.kqed.org/forum/2010101870787/bill-mckibben-warns-of-dire-consequences-of-unchecked-climate-change), right here in the Bay Area. Several locals called in to the show to ask how they can help. A typical caller is Jenny from Petaluma (around 31:30 in the show): > I'm willing to upend my life to be a part of really solving this problem in my own small way. > I'd love to hear how to do that sensibly. McKibben responds in [a typical way](https://www.youtube.com/watch?v=DYLWZPFEWTw): > There are a number of individual actions that are useful. > Eating lower on the food chain. > Putting solar panels all over your own roof. > Figuring out how to get around on public transit. > Not jumping on an airplane just because you want to get to some place that's a little warmer than the place you are now. > But, let me add this caution. > Climate change is a math problem, and at this late stage in the game you can't make the math work anymore one Tesla at a time, one vegan meal at a time. > My house is covered with solar panels, I'm prod of them, I don't try to fool myself that this is how we're going to stop climate change. > The most important thing an individual can do is be a little less of an individual and join together with others in movements of a size enough to make a difference. If you take McKibben's advice and visit the [Bay Area 350.org website](https://350bayarea.org/), their actions -- writing letters, organizing rallies, and convincing local governments to declare a state of emergency -- are all oriented around politics. The biggest call to action on their page is the Donate button. ## What About Money? ## I live in the Bay Area, where [one out of 11,000 people is a billionaire](https://www.vox.com/recode/2019/5/9/18537122/billionaire-study-wealthx-san-francisco) and [everyone is really busy all the time](https://thebolditalic.com/why-are-san-franciscans-so-goddamn-busy-all-the-time-the-bold-italic-san-francisco-2e15a498d750). Maybe one way that these rich, busy people can help is by throwing a few bucks towards a cause? I saw an extreme version of this approach [articulated on the SSC subreddit recently](https://www.reddit.com/r/slatestarcodex/comments/bm3j6x/how_to_effectively_buy_carbon_offsets/emtimk7?utm_source=share&utm_medium=web2x): The OP is asking about carbon offsets, and another poster expresses skepticism that those are effective. The OP replies: > I agree it's not the systemic solution, but for now I'm just looking for a personal moral offset. In today's world (`/me shakes fist at the kids on their escooters`), where most life concerns are outsourced, this kind of thinking makes sense. What if i just go about my life as normal, but I pay someone else to clean up the mess? This is a diametric opposite of Jenny from Petaluma, who was willing to "upend [her] life" -- u/BistanderEffect is unwilling to change in almost any way, but is willing to spend a little money. In the Bay Area, an even more pragmatic approach is available. For instance, Malcolm Handley, founder of [Strong Atomics](https://strong-atomics.com/), once told me that he realized he personally knows several of the Bay's many billionaires. Perhaps the most effective way to help with climate change, he told me once, is to convince some of them to spend a bit of their money to fund, say, fusion reactors. In a more cliche version of Bay Area thinking -- maybe we don't even have to spend any money at all? After all, brilliant people like Elon Musk are working on climate solutions, like a fancy electric car, that actually *make* money. All we have to do is wait a little while, and The Market and technology will be our salvation. ## I Disagree ## I've outlined some typical memes in our culture around climate change which I find fundamentally unsatisfying. Politics is, of course, necessary, but it's also exhausting and it's very difficult to keep focused and motivated. I could write a separate blog post specifically on my views around politics as an infinite game of tug-of-rope, but everyone pulling as hard as they can in their direction has given us the current state of stalemate. The several versions of "someone else will do something" -- be it greengrocers, farmers, refrigerant technicians, or Elon Musk -- are quite disempowering. Surely, if you believe the climate change is a big deal, just sitting back and doing nothing won't be the right course of action for you. The most complicated argument to counter is the one around around outsourcing the problem. Shouldn't it be enough to just donate a few bucks to some organizations, and maybe buy some carbon offsets? ## Living the Difference ## Two facts: 1. To stop the worst effects of climate change, we have to leave fossil fuels in the ground. 2. There is no electric passenger air service Taken together, these two facts imply that passenger air service is impossible to do in an ecologically friendly way right now. We need to both plant a bunch of trees AND stop flying -- one does not excuse the other. In the same way that you cannot "pay" for flights with trees planted in the ground, you cannot "pay" for climate change with money. We definitely have to spend money to ameliorate climate change, for instance by building solar and wind farms. But we cannot just pay the universe to take out our CO2 for us, the way we pay a plumber to fix a leak. There are too many complicated ideas here for me to unpack all of them. For instance, humans compare themselves to others, and evaluate their own status based on their relative status in their social circle. When we argue for carbon taxes, we might suspect that they would make certain activities -- like flying -- more expensive and therefore less commonly-practiced. But maybe if *everyone* had to fly less, it wouldn't be as big a deal? You might still fly more than your friend Bob, and that's good enough. This then starts involving complicated ideas of climate justice. What if we pass [high enough carbon taxes](https://www.jbs.cam.ac.uk/fileadmin/user_upload/research/workingpapers/wp1109.pdf) to double the cost of flights. Do the rich people still get to fly as much as they want, while the poor and middle class folks get even less access to flights? Carbon taxes would also raise the price of food -- do the rich still get to eat as much as they want while the poor starve in greater numbers? Advocating for solutions like carbon taxes -- the means -- lets you avoid thinking about their consequences -- the actual ends we're pursuing, and what those look like. I think we should reverse our thinking and start with the end. What does a world where climate catastrophe has been averted look like? Maybe people fly less. Maybe we eat less meat and more plants. Maybe our energy is produced through renewable sources. We don't have to wait for governments to force us to make these changes. We can just start living as through the future was already here. ## Enter the Spaceship ## I don't want you to keep doing whatever you want in the hopes that someone else will solve climate change. But I also don't want you to decide that, so long as you're pure and holy, you've done your part. Bill McKibben is not wrong to say that we need collective action, but there might be kinds of collective action that he hasn't anticipated. This is where the idea of [Spaceship Earth](https://spaceshipearth.org) comes from. You should be living as though we've already decided to stop pumping petroleum from the ground and instituted carbon taxes. But you need to get everyone you know to start living like that, too. The carbon taxes are one way to get them to start living like there already exist carbon taxes. But Spaceship Earth is another way, and unlike passing carbon taxes, you can start playing Spaceship Earth *right now* (well, once we're done building it). ]]> <![CDATA[Platform for Issue Journalism]]> https://igor.moomers.org/posts/issue-journalism-platform https://igor.moomers.org/posts/issue-journalism-platform Wed, 01 May 2019 00:00:00 GMT <![CDATA[Mailman Behind HTTPS]]> https://igor.moomers.org/posts/mailman-behind-https https://igor.moomers.org/posts/mailman-behind-https Fri, 23 Feb 2018 00:00:00 GMT /tmp/newurl ``` I then ran the following little bash script to fix all my lists: ```bash for i in $(list_lists -b); do config_list -i /tmp/newurl -v $i; done ``` Hopefully, if you're having the same problem as me (mailman's admin page still submits to HTTP instead of HTTPS), you might come across this page and save yourself some trouble! ]]> <![CDATA[The Belden Spa Summer 2017]]> https://igor.moomers.org/posts/my-projects-belden-spa https://igor.moomers.org/posts/my-projects-belden-spa Sat, 20 Jan 2018 00:00:00 GMT <![CDATA[Reflections on Leaving Airbnb]]> https://igor.moomers.org/posts/thoughts-on-leaving-airbnb https://igor.moomers.org/posts/thoughts-on-leaving-airbnb Wed, 26 Apr 2017 00:00:00 GMT .dyno` took you to an instance of that service. Dyno eventually added an authentication mechanism, so internal services didn't have to write their own authentication code. While the `.dyno` instances were initially manually entered into DNS, once Themis become available we allowed it to handle DNS registration for those boxes as well. Today, Airbnb engineers regularly interact with dozens of dyno services. ### Monitoring ### In the beginning of 2014, our systems monitoring was pretty spotty. At that time, we were using [Scout](http://server-monitor.pingdom.com/) for instance monitoring, but monitoring was inconsistently available. Also, Scout was a dead-end for metrics -- it only supported system-level stats that it's agent was able to collect. At the time, several cool monitoring SaaS companies were getting started, and I embarked on a project to evaluate our options. In the end I ended up choosing [DataDog](https://www.datadoghq.com/). A strong reason was DataDog's very good `haproxy` integration. This integration allowed us to have metrics for how much traffic each service was getting, where this traffic was coming from, and the distribution of response sizes, result codes, and other interesting statistics. Another reason was that the [DataDog agent](https://github.com/DataDog/dd-agent) accepted [StatsD metrics](http://docs.datadoghq.com/guides/dogstatsd/), so we could monitor instance statistics like CPU, memory, and other resource utilization alongside our own custom metrics. Furthermore, DataDog had [server-side CloudWatch scraping](http://docs.datadoghq.com/integrations/aws/), which meant we could see CloudWatch specific information like RDS utilization stats alongside all other metrics (and avoid asking engineers to log into the AWS console just to see CloudWatch metrics). Finally, DataDog had a [very comprehensive API](http://docs.datadoghq.com/api/), so it seemed possible to do more automation as time went on. In early 2014, I rolled out DataDog and removed Scout from all of our systems. I would be a primary point of contact for managing Airbnb's relationship with DataDog until my departure from the company. As I was leaving Airbnb, there were teams contemplating what it would look like for us to run at least some of our own monitoring tools. However, on the whole the decision to use DataDog worked out. Despite some bumps, the product scaled well with Airbnb, and the company was very responsive in managing problems and rolling out new features. I strongly recommend DataDog. I also became a primary point of contact for anything monitoring-related at Airbnb. After rolling out the `dd-agent` to systems, I added [DatDog's StatsD client](https://github.com/DataDog/dogstatsd-ruby) to many of our applications. I encouraged other developers to liberally instrument their code with `statsd` calls. Additionally, I ended up writing lots of internal documentation on monitoring best practices. I also developed monitoring curriculum, originally for the SysOps group but later as a bootcamp class for all new hires. I encouraged engineers to formulate hypothesises about what could be causing issues, and then to use the available monitoring tools to test these hypothesises. Since it is impossible to formulate such hypothesises without at least some understanding the overall infrastructure, my bootcamp class became a primer on both the Airbnb infrastructure as a whole as well as the monitoring tools that illuminate that infrastructure. ### Alerting ### After migrating metric monitoring to DataDog, I was able to tackle alerting. I had several strong requirements. I wanted alerts to be defined automatically, so new hosts get alerts as they're spun up and alerts are cleaned up when hosts go away. I wanted the notifications to be automatically routed to the right people. Finally, I wanted alerts to be configuration-as-code, not created via manual clicking in the UI. First, I had to take a stab at the ownership issue. This was a constant problem during my tenure at Airbnb, and we never really solved it. As people moved around and teams formed and dissolved, systems would become orphaned, and the maintenance burden would fall on "whoever cares most". However, I at least made an initial system for assigning ownership, even if the data in that system was not always consistent. I stored the information in the Chef repo, inside the role files which defined instances, and made it available for querying via `optica`. Next, I created [Interferon](https://github.com/airbnb/interferon). This project uses a Ruby DSL to programmatically create alerts. I chose a DSL because I wanted complicated filtering logic, and found myself inventing mini-programming languages inside pure-data formats like JSON or YAML. As an example for how Interferon was used, lets take CPU utilization. I was able to write a single alert file which specified a CPU utilization threshold. Whenever Interferon ran, it would pull a list of all hosts from inventory and make sure that each host had a CPU alert, cleaning up stale alerts for any terminated hosts. The alerts would be routed to the owners for each host, and the alert file could be modified to explicitly filter out any systems where high CPU usage is expected. Because all changes are code changes, the usual review process applies, so there are no surprises for new alerts or alerts suddenly going missing. Also, writing alerts in a file encourages developers to write longer, more informative alert messages, and the DSL allows information about hosts to be encoded in the alert, making alerts more actionable. I gave [a talk about this work](https://www.usenix.org/conference/srecon15/program/presentation/serebryany) at SREConf 2015, which includes more details if you're interested. There's also a [blog post about Interferon/the Alerts Framework](https://medium.com/airbnb-engineering/alerting-framework-at-airbnb-35ba48df894f) on the Airbnb blog. ### Product Work ### In the middle of 2014, I was becoming burned out on SRE work. Also, although I had been running systems at Airbnb, I hadn't done any work on the product -- I didn't even know how to work with Rails. To change things up, I transitioned to a product team which was building an experimental cleaning integration for Airbnb. I worked with a front-end engineer for six months to build this product, and we launched it in several markets. However, in the end it wasn't viable and was shut down. ### Developer Happiness ### In early 2015, it was clear that the cleaning product I had been working on wasn't going to ship, and the team would dissolve. I was looking around for new problems to focus on, and they weren't hard to spot. Shipping code to the Airbnb Rails monolith was becoming increasingly difficult. Build and test times were increasing rapidly. Spurious test failure was a constant problem and required regular re-builds, further delaying shipping. The build-test-deploy pipeline was unowned, meaning that whenever it broke it was up to whoever was most frustrated to fix it. Overall, the experience of being a product engineer was quite frustrating because of gaps in tooling, documentation, ownership, and communication. So, when my product was finally terminated, [Topher Lin](https://github.com/clizzin) and I started the Airbnb Developer Happiness team. Our broad mandate was to work on whatever was causing the most frustration. Our initial top target was build times, but we envisioned tackling a wide range of issues around internal communication and tooling. To understand our problem space and to get buy-in for our projects, we began conducting the Airbnb developer survey. I collected and analyzed the data, which showed widespread frustration with our tooling and infrastructure. I spent most of my remaining time at Airbnb working in this problem space. The team Topher and I started ended up expanding to more than 20 engineers on at least 4 sub-teams. Although we never got time to work on many of the broader problems we initially envisioned tackling (like internal communication practices), the intersection of people and tooling is the area I remain most passionate about. ### Build System ### The Developer Happiness Team's initial target was slow build times, at that time creeping into 30-minute-plus territory. At the time, we were using [Solano](https://www.solanolabs.com/), a third-party ruby testing platform, to run any commit-time tasks including builds. We had hacked building an artifact into this system as a fake test. We were also using Solano to build non-ruby projects, including all fat JARs from our Java monorepo. Solano was running on AMIs provided by the company, and we didn't understand the build environment, how to debug any problems or build failures, or how to control system dependencies for builds. We decided that we would start by moving builds to our own hardware, where we could optimize the environment. Since we would end up with multiple systems performing build and test tasks, we decided to create a unified UI where all such tasks could be collected and visualized. I also began evaluating multiple build systems to replace Solano, with an eye towards a system which supported arbitrary pipelines to support optimizations to the Java builds as well as Ruby builds. A build system is just an executor which performs tasks in response to events, usually commit events. We already had a system that fed all [webhook events](https://developer.github.com/webhooks/) from Github Enterprise into RabbitMQ, providing a convenient trigger. We were already very familiar, too, with [Resque](https://github.com/resque/resque), a Ruby task executor for delayed or long-running tasks which we used throughout our production infrastructure. Finally, we were tired of writing build tasks as shell scripts (which can't be tested) and which integrated with the build system by making API calls via `curl`. We envisioned instead a small library of common tasks, written as Ruby functions with good test coverage, and which could report their status, progress, and results directly into log systems and databases. These design considerations lead us to decide to roll our own build system. We built it into Deployboard, the tool we were already using to deploy the builds. Instead of learning about new deployable builds via API calls, Deployboard would now generate them using Ruby executed in response to RabbitMQ events. It would display any progress and error logs. The end result was the Deployboard Build System -- built in less than 4 months by just three engineers, who were also supporting frequently-failing CI for an engineering team of 400+. We migrated the Airbnb Rails monolith to this system in summer of 2015. This system immediately improved the speed and reliability of builds by an order of magnitude. In November of 2015, I wrote a Ruby test splitter which allowed arbitrary parallelism on our monolith's RSpec suite, and migrated the tests from Solano to Deployboard as well. This improved test times from 30+ minutes to around 10 minutes, as well as reduced spuriousness and made test result tracking easier. By March of 2016, we completely terminated Solano, migrated all builds to Deployboard, and introduced and integrated Travis CI for testing most projects except the Rails monolith and the Java monorepo (which were tested in Deployboard for performance reasons). The combination of Deployboard Build System and Travis CI remains in use for all projects at Airbnb today. There are a few talks about Deployboard online. One is [a talk Topher and I gave at Github Universe 2015](https://www.youtube.com/watch?v=4etQ8s74aHg). There is also [a talk I gave at FutureStack 2015](https://blog.newrelic.com/2015/12/15/airbnb-democratic-deploys-futurestack15-video/). However, Deployboard was also unfortunately never open-sourced, mostly due to lots of Airbnb-specific code and it's use of O2, an internal Bootstrap-style CSS framework. ### Ruby Migration ### In mid-2016, after a brief break to focus on load balancing and Themis, I embarked on what became my final project at Airbnb. At the time, we were still using Ruby 1.9.3 on Ubuntu 12.04 for all of our projects. In general, we had no story around how to upgrade system dependencies of any kind for our projects. My goal was to create such a mechanism, and then to use it to upgrade the Ruby version for our Rails monolith. Our build artifacts were generated directly on build workers, using system versions of any dependencies. They are deployed as tarballs to instances which are required to have matching versions of these system dependencies. Upgrading such dependencies had to happen in concert between the build and production systems -- a difficult operation that would be more difficult still to roll back in case of trouble. We had no way of even tracking what system dependencies a given artifact required. I began by tagging builds with system dependencies used to create the build -- things like ruby version, NodeJS version, and Ubuntu version (as a shorthand for any dynamic library dependencies). Next, I rebuilt our deploy system UI, which previously asked engineers to pick a specific build artifact to deploy. The new UI asked engineers to pick a specific version (SHA) of the code, which may be associated with any number of build artifacts. Finally, I modified the system which actually performed deploys on instances. Previously, that system would receive a specific artifact (e.g. a tarball of Ruby code along with it's `bundle install`ed dependencies) and then go through the steps (untar, link, restart) to deploy that artifact. In the new version, the system would receive a list of possible artifacts, along with their tags. It would then compare local dependency versions with the tags on artifacts, and pick an artifact that matched the system (or error out if, for instance, the system Ruby version didn't match the Ruby version tag on any of the available artifact). This system allowed me to build the Rails monolith concurrently for Ruby 1.9.3 and Ruby 2.1.10. It also allowed me to have web workers for both versions of Ruby in production -- each worker, upon receiving a deploy, would pick a correct artifact. I also began running tests for the monolith under both versions of Ruby, fixing any spec failures that were version-dependent. By February of 2017, this preliminary work was completed. I began running some upgraded Ruby workers to watch for unexpected errors, and also to compare performance between the populations. The vanilla Ruby 2.1.10 build actually had *worse* performance than the [Brightbox PPA](https://www.brightbox.com/docs/ruby/ubuntu/) build of Ruby 1.9.3 we had been using. In the end, I created a custom build of Ruby 2.1.10 with several performance patches. In March 2017, I performed the Ruby upgrade for the monolith. Also, the system dependency upgrade system I created was used to upgrade our Ubuntu version from 12.04 to 14.04. Other engineers were beginning to use the system to upgrade other system dependencies, including NodeJS versions for Node projects and Ruby versions of other services in our SOA. After completing the migration and documenting the work, I announced my departure. ### Non-Technical Projects ### Besides the big chunks of code I wrote while at Airbnb, I was also involved in lots of non-technical (or at least, non-coding) projects. In hindsight, some of those projects were arguably more important than any of the strictly technical work that I did; see the section below on a post-mortem around those thoughts. It seems worthwhile to document those here, too, while I still remember them. One big area of focus was SysOps. I spent a lot of time on-call, especially during the hectic years in 2013 and 2014 when we were growing rapidly and our infrastructure was in flux. I eventually transitioned into a leadership role of the SysOps group. This involved organizing training for new members, planning the on-call schedule, and running the weekly postmortem meetings. The SysOps group was incredibly successful, and I frequently hear astonishment from my peers when I tell them that Airbnb has a strictly volunteer on-call rotation. The group was so successful that we eventually had more people who wanted to be in the on-call rotation than we could fit into slots during a 6-month period. We ended up reducing on-call shift duration from a week to just two days. In 2015 several long-tenured members, including me, began stepping back from the group to allow newer engineers to take the lead. Another big focus was our overall technical vision. I was a member of Tech Leads, an initial stab at such a vision, in 2013. However, when [Mike Curtis](https://www.linkedin.com/in/curtismike/) became VP of Engineering, he dissolved the group as part of his efforts to abolish any hierarchy among engineers. However, we still needed a way to collectively decide how to evolve our infrastructure. I took lead on an initial stab at such a system, called Tech Decisions, in late 2013, but that system was too bureaucratic and never had much adoption. In 2014, a crisis around whether and how we run an SOA precipitated another attempt, called the Infrastructure Working Group. We held a series of meetings to come up with a shared set of principles for our infrastructure, which formed the basis of any future decision-making. I drafted several of the principles we eventually settled on. We also created a structure called the Infrastructure Working Group, which worked to influence individual teams and engineers to make technical decisions in accordance with the principles. I was heavily involved with the group, at least until the Developer Happiness Team began taking all my time. I was very involved with our Bootcamp efforts for new hires. I participated in the meetings that created the Airbnb Bootcamp. Afterwards, I ended up regularly teaching two of the sessions. The first was on monitoring our infrastructure. The second was titled "Contributing and Deploying", and covered the developer workflow from committing code (including how to write good commit messages -- a personal quest) to getting that code successfully out in production. This was the only mandatory session of the bootcamp. Finally, as a technical leader and senior engineer, I spent a lot of time on mentorship and code review. During our Chef roll-out, I ended up reviewing almost every pull request to our Chef repo in an effort to broadly seed Chef best practices. Later, as I transitioned on working primarily on Deployboard, I reviewed most PRs to that repo, trying to ensure consistent architecture, style, and test coverage. As the Developer Happiness/Infrastructure group of teams grew and hired many new engineers, I worked to get them up to speed on the codebase and to become productive and self-sufficient contributors. Finally, I spent a large amount of time maintaining what I call "situational awareness". This meant engaging with the firehouse of stuff that the Airbnb engineering team was doing, from project proposals and infrastructure decisions down to individual pull requests, email threads, and even Slack conversations. I attempted to inject vision and guidance wherever I could, connect the dots between disparate projects, and in general to be helpful. For instance, I could tell an engineer that a project they were trying to accomplish would become easier when another engineer on a different team completed a different project. I could catch PRs that were likely to cause problems, or connect outages to specific changes. This connector role was performed by several people in the engineering organization, and these people never got the credit they deserved for this thankless and never-ending task. ## What Didn't Work? ## While I knew that I had been incredibly productive at Airbnb, writing everything that I worked on really created some perspective. Looking back over that I worked on, I see that my technical projects -- Chef, SmartStack, Deployboard, the work on Monitoring -- were very successful. They made life easier for other engineers, and have survived the test of time to continue providing value. However, I always had more grand ambitions than to just accomplish a specific project. I had a vision for how I wanted our infrastructure and our engineering team to function, and I did not succeed, in most cases, in making that vision a reality. A great example of this is our original vision for the Developer Happiness team. We did not set out to become the CI team, although that's where we eventually ended up. We wanted to improve documentation, communication, and make being an Airbnb engineer easier and more fun. I ran the engineering team survey and collected pain points, but never had enough bandwidth to address more than a few of these pain points. Why didn't I have enough bandwidth? I think it's because I failed to navigate the transition from individual contributor to technical leader. The easiest way for me to get things done at Airbnb was to just do the things I thought needed doing. When our builds were slow, I jumped in with both feet and made them faster -- by writing a build system and a test splitter and a Javascript UI for result visualization, etc... In the meantime, I let things fall to the floor. I ignored the structural problems that lead to the situation, or merely kvetched about them and left others to try to solve them. I spent too much time writing code, and not enough time influencing or guiding other contributors -- which would have allowed me to focus on a broader range of problems. I focused on my ability, as an IC, to definitively solve smaller (though usually still very important) problems. This took away from my ability to address the larger ones. This wasn't *entirely* my fault. Airbnb could have done better to support my transition. The engineering team is completely flat -- even though we have engineering levels, they're supposed to be secret. Expectations around what engineers should be focusing on at each level are vague at best, and there's no consensus among managers about what makes a more senior engineer. Mike Curtis once told me, in a one-on-one, that it should be possible to reach the highest engineering levels by focusing on deep technical work, but I think that assertion would come as a surprise to most of his engineering managers. I had always wanted to make technical leadership an explicit role, not something a few engineers did in their spare time. I didn't succeed in making this vision a reality. I don't think I really even tried. If I could change anything about my time at Airbnb, it would be a change in focus. I wish I was more focused on building relationships, and less on accomplishing objectives. In the end, when I was leaving, it was the relationships that I did manage to build that persisted in my life -- the technical stuff is all someone else's problem now. ## Why I Quit ## It took about 6 months for the Developer Happiness team to go from three to 4 engineers. This was an incredibly stressful time. Between broken builds, broken tests, and bugs in the code, it could take an entire day to ship a single changeset. We were concerned about a code backlog so deep we couldn't clear it in one day. We were performing heroics to keep the system running, at the same time trying to build and roll out a replacement, all with minimal headcount. Yet, we didn't seem to get the recognition and support we deserved from engineering management. From the founding of the team, our manager was part-time, also managing a separate product team. In the summer of 2015, he approached me about potentially taking over as the full-time manager. However, I was in the middle of several deep technical projects, which I felt couldn't lose me as an IC. Also, I felt I was a role model for the parallel IC career path at Airbnb. I wanted to continue being an example of a successful IC who got things done without transitioning. As a result, I declined the offer. Instead, another recently-hired team member took on the management role, and we also added a newly-hired project manager. Together, the two of them came up with broad roadmap for the team. They organized a process to plan specific projects to meet that roadmap. This process turned out to be incredibly frustrating for me. I was spending all my time putting out fires and shipping high-impact code, but the new roadmap had no room for several of the projects I thought were most important, including some that I was in the middle of working on. More, this situation epitomized the disconnect between the Airbnb engineering team's vision for technical leadership, and the reality. It seemed to me that senior engineers should have a lot of input over our planning process. Instead, I faced what I now recognize as the structural disadvantage of remaining an IC. The new manager and PM had all their time to influence, plan, and decide. As an IC, I had to split my time between those activities and actually participating in the tech process -- coding, code review, maintenance, mentoring. To live up to our goals, the management team would need to put a lot of effort into active engagement and deference. This just didn't seem likely. The combination of high-stress firefighting, loss of control in planning, and disillusionment with our ideals lead me to utterly burn out. I ended up taking a two-month leave of absence in March 2016. By the time I returned from leave, I was even further sidelined in my own team. Instead of continuing to fight for influence, I decided to step aside. The Observability team I joined consisted of some of the most senior engineers working on some of the deepest technical problems of the infrastructure. I focused on a single project, the Ruby upgrade, and lead it through to completion almost single-handedly. However, it was clear to me that this wasn't the kind of work I wanted to be doing. I enjoyed the social aspects of my job as much or more as the deep technical ones. I am most passionate about the interface of people and technology, and I wanted to be a technical leader. It felt like the right moment to move on from Airbnb. I announced the completion of the Ruby upgrade, and my departure, in the same all-hands meeting. ]]> <![CDATA[Setting up IMAP]]> https://igor.moomers.org/posts/setting-up-imap https://igor.moomers.org/posts/setting-up-imap Mon, 20 Mar 2017 00:00:00 GMT /var/mail/igor47 ``` ## Procmail to local Maildir Next, I wanted to ensure that new mail would continue to be delivered to my Maildir folder. I was already using [Procmail](https://wiki.archlinux.org/index.php/Procmail) to filter spam and other kinds of messages, but I didn't have a final fallback rule. This meant that any mail not delivered to a specific location by Procmail would come back to Postfix, which delivered it to the `mbox` inbox. To resolve the situation, I added a new catch-all rule to my `procmailrc`: ```bash $ echo INCLUDERC=${PMDIR}/rc.final > ~/procmail/procmailrc ``` `rc.final` looks like so (my `procmailrc` sets `$MAILDIR` to `~/Maildir`): ``` :0: $MAILDIR/new ``` As always, [this reference](http://www.zer0.org/procmail/quickref.html) is inestimably helpful to write these obscure Procmail filter rules. ## `mutt` uses new inbox Now, mutt should be told where my mail is coming in. I set the following variable in my .muttrc: ``` mailboxes ~/Maildir ``` Note that I am continuing to get an error (`/var/mail/igor47 is not a mailbox`) when I first open mutt, but it seems to cause no trouble after I open the correct inbox. ## SSL certs for mail I used [Let's Encrypt](https://letsencrypt.org/) to get SSL certs for my mail server. Because let's encrypt uses HTTP to authenticate that you really own the domain, I first needed my mail server (`mail.moomers.org`) to be accessible on an HTTP port. I did this in Apache by making `mail.moomers.org` a [`ServerAlias`](https://httpd.apache.org/docs/2.4/mod/core.html#serveralias) for the [www.moomers.org](https://www.moomers.org) virtual host. That done, I invoked Let's Encrypt like so: ```bash $ letsencrypt certonly -a webroot -d mail.moomers.org -w /var/www/moomers.org/htdocs ``` Once the cert was acquired, I double-checked that automatic renewal works, too: ```bash $ letsencrypt renew --dry-run ``` [This article was very helpful with helping to configure Dovecot/Postfix for SSL](https://ubuntu101.co.za/ssl/postfix-and-dovecot-on-ubuntu-with-a-lets-encrypt-ssl-certificate/). ## Configure Dovecot I was ready to [install Dovecot](https://help.ubuntu.com/community/Dovecot): ```bash $ aptitude install dovecot-imapd ``` I want system users to also be Dovecot users, but I didn't want passwords to be transmitted unencrypted over the web. I modified `10-auth.conf` (all of the config files here are relative to `/etc/dovecot/conf.d`) like so: ``` disable_plaintext_auth = yes auth_mechanisms = plain login ``` To enable SSL, I set these in `10-ssl.conf`: ``` ssl = yes ssl_cert = <![CDATA[Backups to another server]]> https://igor.moomers.org/posts/backups-to-another-server https://igor.moomers.org/posts/backups-to-another-server Thu, 22 Dec 2016 00:00:00 GMT " | sudo -u purrbackups tee /home/purrbackups/.ssh/authorized_keys This will allow the `moobacker` user from the local server to log into the `purrbackups` account on the remote server. Copy-pasta the actual SSH public key in place of ``. Also, replace `` with the actual IP address of the source server. The `from=` option in `authorized_keys` only allows this user to log in from that IP address for greater security, even if the SSH private key leaks out. At this point, you should test the setup so far by trying to SSH from the local server to the remote server as the two new users we set up: igor47@local:~$ sudo -u moobacker ssh purrbackups@ After accepting the host keys, you should connect and then immediately get a `Connection closed` error. Congratulations -- we got the two machines talking to each other! We should still lock down the `purrbackups` user so it can only be used for sftp purposes. To do this, lets edit the `/etc/ssh/sshd_config` file. Find a line containing `Subsystem sftp` and make sure it is uncommented (no `#` at the beginning) and looks like this: Subsystem sftp internal-sftp Then, at the end of the file, add a section like so: ``` Match User purrbackups ForceCommand internal-sftp ChrootDirectory /home/purrbackups AllowAgentForwarding no AllowTCPForwarding no X11Forwarding no ``` This will only allow the `purrbackups` user to use SFTP (for file-copying purposes), and will further only allow it access to it's own home directory. Finally, set permissions on the `Chroot` directory, and add another directory for backup purposes: igor47@remote:~$ sudo chown root:root /home/purrbackups igor47@remote:~$ sudo chmod 755 /home/purrbackups igor47@remote:~$ sudo mkdir /home/purrbackups/backups igor47@remote:~$ sudo chown purbackups /home/purrbackups/backups Now, test this config again via `sftp`: igor47@local:~$ sudo -u moobacker sftp purrbackups@ You should get an SFTP prompt. Hooray! ## Allow local user to run backups We want the backup user to be able to access files owned by other users on the system. This means the backup user will need to run the backup program as root. Lets set up a backup script for this purpose. ```bash #!/bin/bash whoami ``` Save this as `/home/moobacker/backup.sh`, and then set permissions appropriately: igor47@local:~$ sudo chown root:root /home/moobacker/backup.sh igor47@local:~$ sudo chmod 755 /home/moobacker/backup.sh Next, lets allow the backup user to run this as root. Use `visudo` to edit `/etc/sudoers.d/backups`: igor47@local:~$ sudo visudo /etc/sudoers.d/backups The contents should be: ``` moobacker ALL = (root) NOPASSWD: /home/moobacker/backup.sh ``` This will allow the `moobacker` user to invoke the script as root. Test it like so: igor47@local:~$ sudo -u moobacker sudo /home/moobacker/backup.sh The output should be the word `root` ## Set up GPG key for `duplicity` We will use this key to encrypt the backups. Since the key will need to be distributed to other systems (so you can decrypt your backups in case you need them), set a passphrase on the private key. You can pass this passphrase to the backup script when it runs. igor47@local:~$ sudo -H -u moobacker gpg --gen-key Accept all the defaults. You can pick some values for the name and email addresses. The command will take a while to generate the entropy required for the keys -- you can run some random tasks in the meantime (like `aptitude update`). Note the `-H` option to `sudo` -- we need this, otherwise `gpg` won't know where the home directory is and won't save the resulting keys. Normally, private keys should remain on the system where they were generated. In this case, you'll want to copy the private key to another system so you can decrypt your backups if necessary. I used the `ccrypt` program to encrypt the backup with a symmetric key: igor47@local:~$ sudo -u moobacker tar -czf - /home/moobacker/.gnupg | ccencrypt > /tmp/moobacker.gnupg.tgz.cc You can then distribute the file freely. You'll need the passphrase to the `.cc` file containing the GPG private key, as well as the GPG key passphrase, to recover your backups. If you don't have the `ccencrypt` binary, it can be had on most distros by installing the `ccrypt` package. ## Set up backups Let's put all of the pieces together. Remember the script we created earlier, in `/home/moobacker/backup.sh`? Here's what the final version of mine looks like: ```bash #!/bin/sh set -o errexit set -o nounset encryption_key_id="DC3EEE04" duplicity="duplicity --verbosity error --no-print-statistics --encrypt-key $encryption_key_id" remote="sftp://purrbackups@remote/backups" # back up /etc $duplicity /etc ${remote}/etc # back up /var $duplicity /var ${remote}/var # back up /home $duplicity /mnt/raid/home ${remote}/home ``` The `encryption_key_id` can be gotten like so: igor47@local:~$ sudo -H -u moobacker gpg --list-keys I picked the ID of the `sub` key, which is typically used for encryption. For initial runs of the script, you might wish to use a more verbose output model, so you can see any errors: ```bash duplicity="duplicity --encrypt-key $encryption_key_id" ``` Perform an initial run (or two) of the script: igor47@local:~$ sudo -H -u moobacker /home/moobacker/backup.sh If this succeeds, it's time to add the script to a cron tab to run regularly. igor47@local:~$ sudo -H -u moobacker crontab -e My `moobacker` user's crontab looks like this: ```cron MAILTO="admins@example.org" # m h dom mon dow command 23 23 * * 0 sudo /home/moobacker/backup.sh ``` This will run backups at 23:23 every Sunday. I expect the backup script to produce no output -- any output indicates errors. The `MAILTO` setting will send any error output to me via email. ]]> <![CDATA[What is to be done?]]> https://igor.moomers.org/posts/what-is-to-be-done https://igor.moomers.org/posts/what-is-to-be-done Wed, 28 Sep 2016 00:00:00 GMT <![CDATA[The 12V Music Manifesto]]> https://igor.moomers.org/posts/the-12v-music-manifesto https://igor.moomers.org/posts/the-12v-music-manifesto Mon, 19 Sep 2016 00:00:00 GMT <![CDATA[Congenital Heart Disease in the USSR]]> https://igor.moomers.org/posts/heart-disease-in-the-ussr https://igor.moomers.org/posts/heart-disease-in-the-ussr Mon, 12 Sep 2016 00:00:00 GMT <![CDATA[Recovering OpenID]]> https://igor.moomers.org/posts/recovering-openid https://igor.moomers.org/posts/recovering-openid Sat, 23 Apr 2016 00:00:00 GMT ` section: ```apacheconf ProxyPass / http://localhost:4567/ ProxyPassReverse / http://localhost:4567/ RewriteEngine on RewriteCond %{HTTP:Authorization} !^$ RewriteCond %{QUERY_STRING} openid.mode=authorize RewriteCond %{QUERY_STRING} !auth= RewriteCond %{REQUEST_METHOD} =GET RewriteRule (.*) %{REQUEST_URI}?%{QUERY_STRING}&auth=%{HTTP:Authorization} [L] ``` I made sure that this was the only ProxyPass directive that was uncommented, and then ran `apache2ctl restart`. 4. Attempt to log in via my OpenID at StackOverflow. This caused some output like so to be printed out from the running server: ``` localhost - - [23/Apr/2016:19:59:48 CDT] "GET /xrds HTTP/1.1" 200 567 - -> /xrds Not allowed: 172.5.245.84 You need to put this IP in the 'allowed_ips' array in: /home/igor47/.local-openid/config.yml ``` 5. Edit my `~/.local-openid/config.yml` file. This file had an `allowed_ips` section into which I added the IP address that was making requests. I also saw a section like so: ```yaml https://stackoverflow.com/users/authenticate/: assoc_handle: updated: 2016-04-24 01:01:39.873137626 Z expires: 1970-01-01 00:00:00.000000000 Z session_id: 1461459588.9306.0.1628395514528984 expires1m: 2016-04-24 01:02:39.873137626 Z ``` I removed the `expires` key and renamed the `expires1m` key to `expires`. 6. Reload the auth page in the browser. Viola! I was logged in! Hopefully this helps you if you also lost access to some old OpenId account and would like to regain access. ]]> <![CDATA[SmartStack vs. Consul]]> https://igor.moomers.org/posts/smartstack-vs-consul https://igor.moomers.org/posts/smartstack-vs-consul Thu, 01 May 2014 00:00:00 GMT Consul uses an integrated [gossip protocol](http://www.consul.io/docs/internals/gossip.html) to track all nodes and perform server discovery. > This means that server addresses do not need to be hardcoded and updated fleet wide on changes, unlike SmartStack. This is a fair criticism of SmartStack -- the addresses of the [Zookeeper](https://zookeeper.apache.org/) machines must be statically configured. Of course, [Serf must also be bootstrapped](http://www.serfdom.io/intro/getting-started/join.html) with at least one existing node to join the cluster. If all of the bootstrapped nodes you have hard-coded into your configuration management system (like [Chef](http://www.getchef.com/chef/) or [Puppet](http://puppetlabs.com/)) die, new nodes will not be able to join the cluster. Really, there are two choices here. The first is statically hard-coding a list of Zookeeper instances and relying on Zookeeper. The second is static configuration of bootstrapping information for Serf and relying on Serf's [gossip protocol](http://www.serfdom.io/docs/internals/gossip.html). The gossip protocol is a modified version of [SWIM](http://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf). Consul uses this not just for bootstrapping but for propagate ALL information, including the availability information you're trying to discover. I have many unanswered questions about the gossip protocol. For instance, in the case of a network partition, it seems like a partitioned-off node will be alternatively marked suspected-down and then back up by different group members. This may result in a partitioned-off node never leaving the cluster. In the end, replacing ZooKeeper with Serf may be a viable option for SmartStack. I would welcome pull requests to [synapse](https://github.com/airbnb/synapse) that use Serf, or maybe Consul, as a [service watcher](https://github.com/airbnb/synapse/tree/master/lib/synapse/service_watcher) instead of [Zookeeper](https://github.com/airbnb/synapse/blob/master/lib/synapse/service_watcher/zookeeper.rb). ## Service Discovery ## > For discovery, SmartStack clients must use HAProxy, requiring that Synapse be configured with all desired endpoints in advance. > Consul clients instead use the DNS or HTTP APIs without any configuration needed in advance. > Consul also provides a "tag" abstraction, allowing services to provide metadata such as versions, primary/secondary designations, or opaque labels that can be used for filtering. > Clients can then request only the service providers which have matching tags. The first sentence here doesn't really even make sense. Sure, Synapse must be configured to discover the services you are going to want to talk to, but you could just as easily configure it to discover ALL of your services. On the other hand, explicitly specifying which services you are going to want to talk to from which box is extremely useful, because it allows you to build [a dependency graph of your infrastructure](/images/airbnb-infrastructure-oct13.png). I view this as a benefit, not a drawback. Another benefit is using [HAProxy](http://haproxy.1wt.eu/#desc) to actually route between services. Whenever a service inside Airbnb talks to a dependency SmartStack, that service knows nothing about the underlying implementation. The ability to avoid writing a client (even a simple, HTTP client) for service discovery into each application was a fundamental design goal for us. If you want a third-party application you didn't write to run on your network and consume Consul information, you must use DNS. However, DNS is even worse -- when, how, and for how long will DNS resolutions be cached by your underlying libraries or applications? Instead of insisting on a simple HTTP API, Consul provides you with the ability to do complex tag-based discovery. It is almost certainly a mistake to utilize these features. Your infrastructure should aim to be as simple and flat as possible. A service instance is a service instance, and if it's different then it is a different service! If you find yourself 6 months in, only talking to instances of service Y which provide property X from some unknown number of clients which have requirement X hardcoded into an HTTP request buried in their codebase, you are going to wish that you hadn't done that. Finally, HAProxy is an extremely stable, popular, well-tested, well-utilized, fundamental component of the internet which provides amazing introspection. That we use HAProxy means that synapse and zookeeper can just go away, and your service will keep on working (although it won't get updates about new or down instances). Using connectivity checks in HAProxy means that we can survive network partitions -- services which remain registered will be taken out of rotation by HAProxy. Using HAProxy's [built-in load balancing algorithms](http://docs.neo4j.org/chunked/stable/ha-haproxy.html) meant that we didn't have to write them. Using HAProxy's [built-in status page](http://haproxy.1wt.eu/img/haproxy-stats.png) means we can easily see what's happening on a particular box with that box's service dependencies. Using HAProxy's logging, we can see a detailed history of communications between services. And using monitoring tools that scrape and aggregate HAProxy's stats, we can get instant insight into what kinds of load services are seeing, from which kinds of other services. Many of these advantages can again be gained by configuring synapse to use Consul as a discovery source. But I strongly feel that synapse/HAProxy combo is better in many ways than Consul, and urge you to consider the benefits I've outlined above. ## Health Checking ## > Consul generally provides a much richer health checking system. > Consul supports Nagios style plugins, enabling a vast catalog of checks to be used. > It also allows for service and host-level checks. The current list of health checks in nerve is minimal at best, although it's been sufficient for our needs here at Airbnb. I like the simple model, of nerve doing a direct check on a service from the machine it's running on. Conceptually, it's easier to wrap your head around. Why is this box deregistered? Because it failed it's nerve health check? Or because Nagios is down or overloaded, or because the application pinged your service to ask to be deregistered and then kept running, or for what other unseen reasons? Although I would discourage the use of complex health checks, I can see the advantages, and I would welcome PRs to nerve to add better health checking. ## Multi-DC ## > While it may be possible to configure SmartStack for multiple datacenters, the central ZooKeeper cluster would be a serious impediment to a fault tolerant deployment. I am not certain that I would want to run a UDP-based gossip protocol across the public internet. Running a Zookeeper cluster across the public internet is also not an ideal situation. I think that the correct approach is to provide mostly-local service clusters per datacenter. A single, global Zookeeper cluster will contain only the list of services that are truly cross-DC (like the front-end load balancers), while most services only talk to services inside their local DC. Assuming a flat cross-DC topography is setting yourself up for much higher than necessary latency. Of course, with Consul you could probably configure your services to discover only dependencies tagged with your local datacenter. But this reaches into the realm of configuration management, and at that point both Consul and SmartStack become equivalent -- a Chef change is a Chef change. ## Summing up ## I love Hashicorp, and I think Serf is a great idea, implemented well. I think that from an operations perspective, SmartStack has a bit of an edge on Consul. I am happy to have the opportunity to engage in dialog like this, and I'm excited about how much easier it's getting all of the time to operate internet infrastructure. If you have comments, or corrections, please do get in touch! ]]> <![CDATA[Third-party domain delegation considered harmful]]> https://igor.moomers.org/posts/third-party-domains https://igor.moomers.org/posts/third-party-domains Wed, 26 Mar 2014 00:00:00 GMT <![CDATA[Recovering an Android Phone With a Broken Screen]]> https://igor.moomers.org/posts/recovering-an-android-phone https://igor.moomers.org/posts/recovering-an-android-phone Sat, 04 Jan 2014 00:00:00 GMT > build.prop ``` If you edit `build.prop` via `adb pull` and `adb push`, remember to `chmod 644 build.prop` or your phone won't boot. ## Accessing the phone ## Once I did this, my phone booted with USB debugging turned on. Now, I wanted access to my phone. There are many options for this -- screencasting, VNC server, etc... -- but the easiest solution is a bluetooth mouse. Once you've got one paired with your phone, it's as though you've go your touchscreen back. You will need to access your settings menu and pair the mouse. From `adb shell`, you can use the `input tap` command to send screen tap events. `input tap X Y` is a tap at the corresponding X and Y coordinate, with `0 0` being the upper left-hand corner. I had an icon for the settings menu on my home screen, so I got to it by running `input tap 250 800`. Once there, from `adb shell` run: ```bash input tap 600 400 # for bluetooth input tap 100 1100 # search for devices input tap 100 1000 # to pick the first device that you've found ``` At the end of this process, you will have a pointer on-screen which works just like your finger. ## Moving to a new phone ## I used titanium backup (I paid for the Pro version). I used the mouse to do a backup of all of my apps and their data. I also backed up my SMS/MMS and call log. To do that, click `Menu` -> `Backup data to XML` and then pick a local location to save the XML files. I copied the `TitaniumBackup` dir using `adb pull` and pushed it to the new phone with `adb push`. I had to do the restore of the XML data separately, using `Menu` -> `Restore data from XML`. I had to enable the advanced view in the file picker to find the location of my XML files on KitKat 4.4.2. Afterwards, I had Google Play update all of those apps to newer versions. This happened without problems. ]]> <![CDATA[Tech Interview Questions]]> https://igor.moomers.org/posts/interview-questions https://igor.moomers.org/posts/interview-questions Sat, 09 Nov 2013 00:00:00 GMT <![CDATA[Soviet KGB Stories Pt. 1]]> https://igor.moomers.org/posts/soviet-kgb-stories-pt1 https://igor.moomers.org/posts/soviet-kgb-stories-pt1 Sat, 14 Sep 2013 00:00:00 GMT <![CDATA[Github pages: proxies and redirects]]> https://igor.moomers.org/posts/github-pages-proxying-and-redirects https://igor.moomers.org/posts/github-pages-proxying-and-redirects Sun, 14 Jul 2013 00:00:00 GMT .github.com`. These days, they service it on `.github.io`, probably for security reasons (session cookies?). Anyway, when I wanted my page to be at `igor.moomers.org`, my own domain name. I also wanted the same thing under `igor.monksofcool.org`, which is another domain name that is also my OpenID. So, I configured apache to just proxy my domains to the github page. Here is my config: ```nginx ProxyPass / http://igor47.github.com/ ProxyPassReverse / http://igor47.github.com/ ``` This change made my openid inaccessible. I [had my openid settings in my head](https://github.com/igor47/igor47.github.com/commit/1b7b28605aa5b74e7f150e1a1a5e67f93b8d6138). However, when I would sign in to my openid, [stackoverflow](http://stackoverflow.com/) would ask me if I wanted to create a new account for `igor47.github.io`. In fact, I realized that even though I was proxying, I would get redirected to `github.io` in my browser. Looks like the github web server only accepts requests on one hostname. You can tell it's just one because they say so [here](https://help.github.com/articles/my-custom-domain-isn-t-working#multiple-domains-in-cname-file). So, I figured I could fix my problem by setting a [custom cname page](https://help.github.com/articles/setting-up-a-custom-domain-with-pages). I tried that, [here](https://github.com/igor47/igor47.github.com/commit/2d3ce308de32dd734d35633f32442db6759cec68). This resulted in strange behavior. Even going to `igor.monksofcool.org` in my browser resulted in an infinite 301 redirect loop to `igor.monksofcool.org`! This finally made me realize what was happening. I was proxying all requests to `github.com` and should have been using `github.io`. Even requests to `igor.monksofcool.org` would go to `igor47.github.com`, which is NOT the domain name in my CNAME file. Github would redirect away, and hitting that domain would again issue an improper sub-request. To fix the problem, I removed my CNAME file and fixed my apache config to go to proxy to `github.io`. Suddenly, everything was magically working again! ]]> <![CDATA[Now: February 2025]]> https://igor.moomers.org/posts/now-feb-2025 https://igor.moomers.org/posts/now-feb-2025 Tue, 20 Feb 2001 00:00:00 GMT