Python PSA: Pitfalls of Pivoting from requests to urllib3

Let me start this quick Python PSA with a major disclaimer: I have no idea what I’m doing or talking about and the entire reason for any issues I’ve encountered was my desire to avoid diving too deep into documentation while porting from one Python HTTP package to another, but…

I’ve been using Habitica for managing my recurring tasks and to dos for a couple of years have had success (and mild amounts of fun) battling mundane tasks and monsters alongside my wife’s digital avatar. I’ve also used Trello for way longer, but have found that it’s imposing blank slate design is equally a blessing and a curse and that bottomless Trello lists are often where I send ideas and to dos to die. But, because Trello has tons of integrations, you can turn it into an awesome productivity engine with a little careful design. As a guy who needs structure, I’ve been using planyway to drag my to dos onto the calendar and force them to become real.

But living life in more than one digital realm is complicated and complexity is the enemy. I want to vanquish dragons, build good habits AND use the helpful features of Trello + plugins, so I built a thing. It’s janky, but I built a little python script on AWS Lambda’s Free Tier that links Trello and Habitica webhooks and APIs so that when I make a new to do in Habitica, it crams it into my Trello Inbox where I can decide if it’s destined for Next Actions, Do It Soon or either of the nearly synonymous Do It Later or Do It At All? lists. By using webhooks to watch for changes in either of the systems, I can keep the cards linked and updated. It’s not an elegant integration, but it works for me. Or it did…

This isn’t a post about my hacky productivity system, it’s a post about the AWS Lamba Python SDK removing requests from their built in packages. For those of you who haven’t used requests, it’s a great python package that makes HTTP requests really easy and has powered an awful lot of my scraping and integration projects. AWS botocore (the Lambda standard library) had a version of requests ready to go, but since it’s disappearing I had to figure out how to either install and package requests or figure out how to use urllib3, the python standard library HTTP tool. Out of a desire to avoid learning anything new about AWS or packaging Lambda functions, I decided to figure out urllib3.

urllib3 is the underpinning of requests and does all the same things, and yet people find value in requests and I was about to figure out why. Some of my POSTs transitioned seamlessly, requiring only mild syntax changes:

But then I started running into gremlins, complexities that requests had protected me from for all these years. Where requests had happily accepted a dict of field information that included lists of values for a specific key (like say, when you need to have more than one label on a new Trello card) urllib3 threw byte encoding errors that took a bit to unwind. I faffed around with lists of tuples and json encoding, but nothing seemed to work, so I ended up building a comma separated string in my field dict:

Apparently I’d also made extensive use of requests’s builtin JSON parsing, so you’re going to have to think a tiny bit more about your response parsing:

Also, every time I needed to use a new request type, I found slightly different behavior. POSTing to the Habitica API required an additional ‘Content-Type’: ‘application/json’ header entry, but GETting did not. For some request types such as PUT or DELETE, I ended up sticking most of the variables in manually constructed URL strings instead of headers or fields because I got sick of debugging the inputs for each request. I learned to appreciate the consistency in requests’s requests.

Lastly, next time I’m going to learn how to write proper AWS Lambda code using a local AWS SDK, because while there are some nifty features in their online editor environment, I spent way more time hunting down tab/space indentation errors than I care to admit, a frustrating time sink that would have been easily avoided with a proper editor/IDE.

This is obviously not an exhaustive list of pitfalls nor an authoritative account, but an anecdote from a non-pro programmer who tried really hard to do something quickly and ended up reading more of that docs than desired. Also, I’m not knocking either of the packages, urllib3 is the infrastructure for lots of great HTTP stuff including requests, which is still the easiest way to quickly snag web content for fun projects.

Speaking of fun projects, at some point I’ll get off my butt and post my Habitica/Trello Lambda integration to github and write a more thorough expainer post here. At least I can add that to my to do list with the knowledge that my digital domains are all still in sync 🙂