Reading this recent post got me thinking: what are my current AI Workflows? Structuring some thoughts around this topic took me also down the intersection path of AI & technologies I used to work with often in a, not so distant, prior life.

A. MY AI WORKFLOWS

MY AI WORKFLOWS are similar to what Paul describes with a few differences and additions - their description and my learnings follow:

1. Take some time to chew on important design decisions

When I am working on an Epic (think of it as large chunk of AI Coding work that can span over multiple hours or even a day or 2 - split across multiple implementation milestones), Codex replies with a response that can have quite a bit of background info, design decisions, tech choices to make (e.g. what NPM library to use for a particular task at hand) and questions.

I read that carefully and thoroughly - it can be up to a 2-3 pages worth of content - which is a lot of long-form reading these days 😊. This is in response to my own long initial prompt. Lately, I seldomly reply right away to this initial Codex response: instead I sit on it overnight or think about it during a run or do some additional related research/learning first. I want to be well-informed and thorough in setting the future direction for the epic and I found that chewing on this for a bit, leads to better outcomes.

2. Let the agents to do the talking amongst themselves

The kind of large chunks of work described above often get me into working with more than one agent at time. It could be up to 3 or 4 doing concurrent work in different IDEs on/or different machines: a frontend agent, a backend agent, an Infrastructure (IaC) agent or an agent doing work with Browser Use.

I will probably have some Coordinator/Orchestrator Agent in the future managing the work for these agents at times, but I am not ready for that yet. Main reason why I prefer to still be in the loop: there is frequently a need for mid-course adjustments/corrections when implementing milestones - based on new findings during implementation or due to things we (me and the planning agent) have not thought about at the beginning or, just simple realizations that some other adjusted approach can work better. And I prefer then to be involved with the course adjustments. Maybe a coordinator agent can take care of some of these on-the-fly adjustments. But, honestly, I enjoy the experience of doing this team coordination + I want to learn more naturally and gradually how the agents work together.

These multiple agents workflows require frequent message exchanges between the agents through handoff requirements/instructions from one to the other. It is important to find a shareable location of where these handoff docs can be passed between agents. I found that picking-up the git repo for one of the key agents as the location for dropping handoff docs is a good place to store these docs, for sharing and for posteriority, when organized well in a docs folder - by epics and using good naming conventions.

Do I read these inter-agents handoff docs? Not really. Once the execution plan and the requirements are well understood by all agents + a master plan is produced with all milestones across all agents and upstream dependencies are spelled out, I purposely have the agents generate these docs to exchange between themselves, as they do a better and faster job then me at producing them: they are more verbose and more precise because, amongst other things, they can communicate with each other at a much higher bit rate than I can.

3. A hidden reason for AI over-architecting and over-coding

AI Agents diligently try to take your data and codebase through migrations and transitions so things don’t break and are backwards compatible as you move through development, and sometimes that is not needed. Case in point: the shape of data we work with during development frequently changes and you might have some older rows sitting in the database or in a log store in an older format that we can safely left behind. So just wipe off the old data and assume the code can just deal with the latest formats. Don’t let the agents do pretzels for data you really don’t care about. The obsolete data will go away but the unneeded code will likely not.

4. I still need to understand well the mechanics, but less and less of the code

When I am not doing work with Codex, I have a couple of work-related readings that I like to do:

a. Tutorials

When I am introducing a new technology, concept or integration I have not worked with before, today is much easier to get very good quality tutorials, for free. I just use Gemini: I describe the topic I want to learn about and the goals of what I am trying to build and it does a very good job at preparing a tutorial tailored to my needs. As a free model it might not be as precise or accurate as a frontier model but that is not so important. I don’t actually need to implement the tutorial -> the tutorial markdown is handed off to Codex anyway, where it goes through more refinements. I just need to understand “what it takes” to do what I am trying to do - and reading a step-by-step instructions gives me a very good idea.

b. Human Developer Docs

While I find myself less and less reading code (I tend to do it more for key / framework / shareable pieces of the codebase) I still want to understand the design of the tech implementation. So I frequently have Codex produce in parallel with the code base it writes, human developer docs - thinking about that bit rate I mentioned about above 😊 - I specifically ask it now for docs that are better suited for the development team, than for an agent. These docs include a variety of content: from Mermaid architecture / system / flow / class diagrams and sample code a developer would need to follow to use the framework package covered by the doc, to ways to configure it or extend it. Related thought exercise: what if the AI tool is not available to help me? How can I quickly get up to speed to still be able to do my work? So I always have a good backlog of such docs to read to better understand my codebase.

So what kind of systems do you apply these learnings to?

I am starting to realize that there are very likely good and bad ways to structure and build a system: in terms of making it easier or harder for an AI to work with it. And that we can move faster now even if we have to take on large rewrites of existing functionality - given the speed of development with AI tools. So maybe it is better to take parts of existing/legacy systems that might carry a lot of baggage, and rebuild them piece by piece with proper architecture and design vs trying to constantly patch.

Example of a large chunk of work I completed over the weekend: backend applications can produce good logs and we can get real-time view into them in production, through centralized logging systems. However I have rarely seen front end applications produce good logs/telemetry available in centralized logging.

The idea is: say many of the users of a system get repeated errors for a particular part of the application - what if all these error log events written to the browser’s console could be shipped to a centralized logging system with all enriched data that can be attached to them: user information, session information, user browser/OS info, etc.

This is an example of a large epic I implemented with AI tools and describe it more below.

B. AI TOOLS & CENTRAL LOGGING

A couple of the reasons why I think centralized logging for client-side logs is not done that frequently:

  • Shipping client-side logs can generate large volumes of data if not managed and configured properly: with potential negative impacts both on the client-side and the backend ingesting the logs (every user active in the system generates this telemetry data). And we can’t burden backends handling main application workloads with taking care of this task.

  • So frequently this client-side log data has to be sent to a separate target/sink - which is frequently a SAAS logs management destination. And given the sensitivity of application logs (not to speak of developer mistakes that can leak confidential data in log messages/fields) the transmission and management of the logs has to be done securely (as securely I would say as the actual application data). At Alphaledger, we used Mezmo (formerly known as LogDNA) as the centralized logging store and I often thought about using its browser library to ship these kind of logs to it. To ingest logs to it, an ingestion/API key is needed: but browsers cannot handle securely such non-transient credentials. So I could not really see a good way to achieve this goal. Until now.

1. Centralized logging starting from the local environment

a. DIY Centralized Logging

First, do I still really need a 3rd party centralized logging store? When Elasticsearch has been used for this many years, when it is easier and faster than ever to stand up a well configured and secure Elasticsearch + Kibana + Filebeat (and possibly Logstash) stack to ingest, manage, monitor and act on logs data? And when I also have some good experience with these tools?

b. Codex and Pulumi to the rescue

So in a few hours, using Pulumi for Infrastructure as Code + Codex I put together a Kubernetes cluster (K3D is very lightweight, great for local env. execution) + all services needed to run the ELK stack I mentioned above + NGINX + Let’s Encrypt: secured with SSL, Kibana authentication and Elasticsearch user ids for Filebeat ingestion, client-side logs ingestion and AI agents introspection and administration - with narrowly scoped roles defined for each. And then properly configured with ILM policies, data streams, index mappings and ingestion pipelines. Certainly good enough for local development and not too far from what I would want to run in production.

c. No need for logs ingestion API keys in the browser

How about shipping data securely to the Elasticsearch central data store? My Rust Actix-web API servers use a common framework for middleware components (authentication, authorization, rate limiting, common error handling, request and user context management, etc). So all I had to do is have Codex create a lighter version of such an Actix-Web server that shared a subset of the common middlewares whose job is very specialized and narrow: just to pick up ingestion log events from the browser, enhance them using data available on the backend, sanitize it and send it to Elasticsearch. AND do so securely, as this new logging proxy server can share the same Authentication Middleware as the API servers and thus piggyback on the same Auth strategy (Browser Cookie + CSRF) as what the main front-end code uses after the user signs-in. So the client-side code can just send logs events to the same NGINX proxies w/ the same authentication credentials as the main app flow, and NGINX takes care of routing it to Elasticsearch through my Actix Web logging proxy.

d. AI w/ Browser Use - A lot of potential

After I had all this in place next thought was: how about some Kibana dashboards/visualizations/filters/search etc so I can easily get to the logs I need to see?

First attempt to go about that: why not have Codex generate the Dashboards/Visualization resources generated as .json and I just have them setup, configured and deploy with Pulumi to the Kibana server? This did not go far: Kibana resources (like Grafana resources too and many other tools with specialized UIs) can’t be generated by hand-crafting JSON/YAML files even through the super powerful AI tools. You need to actually setup the dashboards/visualization through the Kibana UI/Console and then export them as .json for import into other clusters.

So after I described my requirements, I had Codex generate for me step-by-step instructions on how to setup these UI components through the Kibana Console. That soon broke down on my side: “I ain’t spending 2-3 hours in a UI configuring these dashboards 😊.” - the AI tools just got us too lazy.

Why not just have Codex be instructed to collaborate with ChatGPT’s browser plugin to leverage its Browser Use capability and have it just build these dashboards for me? Sure enough after regenerating a much more details handoff for ChatGPT, I logged in to the Kibana console with my credentials and let it take it over. To my amazement, peeking at it every few mins do the work in the browser just like a human would, it built the whole dashboard in about 30 mins - about 15 visualizations/panels/tables in it. It got it 70-80% right on the first shot. 2 more turns of refinements handoffs from Codex and this part was ready too. Interestingly enough the token usage was really not any faster when doing this Browser Use work vs regular coding work -> which got me thinking: how do AI tools meter token usage when they have to work a lot more with images / visuals?

2. Why is central logging useful starting from a local environment?

Aside from my desire to have client-side logs present in a centralized logging system, there was another impetus for this weekend project: I have been relying for too long on IDE debuggers to troubleshoot my code, knowing well that does not scale well.

Good tracing and logs in an application are a much better approach -> especially in Rust applications /w a lot of async code like Actix-web API servers. As an example a well optimized build of ours for Rust with a debugging profile turned on produced 135Gb of binaries. Without debugging -> 5Gb. Plus considerably slower application servers startup when using debugging, etc. So much better to have the API server logs shipped to a central location and do analysis there.

The benefits I am looking forward to with my new logging / troubleshooting setup:

  • it is much better to troubleshoot in your local env, the same way as you would troubleshoot an application in production; builds that muscle early
  • forces me to think about any system component’s logging aspects earlier
  • with the Elasticsearch query and inspection AI Skills I put in place with Codex, it will become a lot easier to just say to an AI agent: take the last session correlation id (or tracing id) and analyze logs across all components in Elasticsearch. No more need to go through the back and forth of me needing to copy and paste errors or screenshots into the prompt window.
  • the AI tools are taught good practices from the early dev life cycles -> just go add more tracing / debug statement if you need more info/context to troubleshoot a problem
  • my feeling is that having the AI tools consistently use well structured data from a logging system will lead over time to more economical token usage vs ad-hoc / diverse sources of troubleshooting context fed to them.

They say: “laziness is the first great virtue of a programmer”. AI tools put that on steroids through all the automation they offer us. We just have to be careful to not completely fall prey to it and lose touch of what we are building.