November of 2022 is a month that will go down in history as the one in which the world was all looking at the same shiny new thing: ChatGPT. This revelation captivated the imaginations of countless developers - including myself, as I immediately started building proofs of concept on top of it. In hindsight, I realize the ideas I came up with were the same ideas that most individuals and companies were trying at the time. Previously expensive products were now achievable at a fraction of the cost, making them much more accessible. Here, I dive into the invaluable experience and lessons I learned during my time at Codelitt.

Proofs of concept

RailsCodeCare

My very first POC was nothing short of bold; it was a company. As I work in a software as a service company, creating a new service to maintain Ruby On Rails applications would be a great idea. I already had this idea before, but just thinking about the marketing side made me dizzy. That was when I realized I could use this new technology to help me with marketing. The flow of my application was the following:

1. I would send a topic as the title of an email to my bot. It would be something like “How to build data pipelines with Ruby on Rails 7.0”
2. The bot would receive the email and transform the title into one that could be better marketing-wise
3. Use the new title to create a blog post
4. Upload the well-formatted blog post to my website

I understood that the likelihood of Search engines ranking my content would be small, but I also knew that I could learn a ton from building it. Fortunately, what I was suspicious about became a reality. A while ago, Google said that it wouldn’t rank AI-generated content. That said, I believe anyone can use AI-generated content as a draft and create high-quality content from it.

Tasketeer

A long-standing problem for big companies is knowledge sharing and retention. Over the years, I lost count of the applications I had built around this topic. The goal was simple - the tool was meant to democratize access to vital company knowledge. Documents and company information should be able to be stored efficiently, and those repositories of documents and information were turned into searchable databases, thus eliminating the everlasting issue of knowledge being forever lost with personnel changes. I brought it up with Codelitt’s CEO, and we decided to build a tool to make this dream a reality. We even presented it at MongoDB.Local in New York.

The user flow was the following:

1. The user creates an account
2. The user uploads any file they want to make searchable. For instance, I uploaded all of our HR documentation
3. The user can now ask questions on Slack or Tasketeer’s chat about the content and get valuable answers

We first used it internally, and it was amazing to see our HR knowledge available to our team. At some point, we started seeing many big players solving this same problem, like Google’s notebook project, and we decided to open our code for anyone curious about how we built it. You can find it here.

Distressed property appraiser

This project was by far the most complex one. The goal was to extract information for property APIs, process it, and use it to evaluate the cost of properties in a specific area. To achieve this, we had to pull information from many different resources, and use the latest options from Google Cloud Provider to process this data and turn it into valuable information.

Plain text to API JSON filters

This project was the most significant example of the power of automation using AI. A customer reached out to Codelitt with a simple problem: Their application had over 250 filters from which users had to choose manually. Their goal was to allow their users to specify what they were looking for through a single text input and have the application present the results by way of structured API requests. That means that the application had to:

• Get the text input value
• Send the text value to a “plain text to JSON API”
• Get the API result and send it to the filter API

This was a straightforward problem, and we solved it using LangChain and Python. It is currently in production, resulting in a better user experience for our customers.

StoryForge

This one was more of an “I’ve been repeating this way too many times” situation. After building the first projects, I realized they all had something in common. They all would:

• Receive an input in plain text through an HTTP request
• Match this input with content stored in a vector database
• Send the result asynchronously to another API

To save time for future cases, I built StoryForge, which is an open-source application that does precisely those steps, but in a configurable way. Once the server is up, the developer can:

• Send any text document supported by Box
• Define the context ID, allowing it to have multiple “libraries” or “data sources,” where each library/data source can consist of multiple files
• Send a task in plain text, i.e., “Tell me what is our Company’s HR policy” passing the ID of the data source and what kind of prompt it should use
• Set the webhook with an identifier, this way the receiver application can identify the request

Lessons learned

1. Adding AI to products is easy

Dealing with AI through APIs has never been easier, but solving hard, real-world problems is still hard. With OpenAI (and now many competitors), processing data with AI is as easy as making an HTTP request, but it is only that simple for easy problems. For instance, building the entire RailsCodeCare flow was easy. It meant using the Ruby library Zapier to read the emails and send the content to my Ruby/Sinatra API, and that was it. Because it wasn’t data-heavy, it was simple. However, for the Distressed property appraiser, it was more complicated. Training an AI model is still costly and time-consuming. Many tools are available that make it easier, but it still took us a couple of months to get this last project done, evin with a highly specialized AI engineer supporting us. The main challenges around building a custom LLM come from two sources: specialized technical resources and the lack of data available for many mundane tasks like appraisals.

2. It will only get cheaper to build AI products

Although the goal for any technology is for it to become more accessible and advanced over time, I wasn’t expecting it to happen at this velocity with AI. In 2023 alone, OpenAI created a new model (GPT 4.0) and doubled the amount of tokens available, going up to 128,000. Other companies, such as Anthropic, offer models like Claude 3 that support 200,000 token context windows. While building the “Plain text to API JSON filters,” the context window in OpenAI had 32,000 tokens, which was a challenge for us, as we had to send a description for each of the 250 filters. A month after we released the first version, the limit went to 128,000 tokens, and the price dropped.

I can only expect the context window to become close to unlimited in one or two years, and the costs to decrease by at least one order of magnitude.

3. Healthy data sources are important

It is cheaper to have a healthy database than to expect AI to deal with wrong data input. Since AI hit the scene, I’ve seen many companies try to put all their data into AI models, expecting them to return valuable data. Unfortunately, I haven’t seen them find success in any of those cases. AI models can only perform properly when given the minimum data quality level, which will vary depending on your expectations. The essential requirement for any AI-related project is to prepare the data uniformly and meaningfully. For instance, just throwing in an entire SQL database to a model and expecting it to give you insights is the same as doing it for an engineer who doesn’t understand the data. You’ll waste resources, valuable time and get nothing out of it.

4. Profession replacement isn’t as simple as plugging in an AI model (for most cases)

I first built an integration with Intercon for a chatbot in 2015. The problem at the time was that the company’s CEO couldn’t pay someone to answer customer’s questions, so he had to answer them himself. Nowadays, that is almost no longer a problem. Many companies offer “The best and only chatbot your company needs, using your documents to answer any customer question”. This looks great, until you realize that you need more documentation, or that it isn’t updated as often as you need. I don’t see myself talking to chatbots frequently, but I can think of two cases this year that made me give up on using a product due to my poor customer experience; on both cases I tried to talk to customer service, and not only was I not given the option to speak to a human, the bot would only return “I don’t have an answer, please try asking differently.” The money I would spend on these products wouldn’t pay for a customer rep, but I’m sure that the cost of many users bailing on the product because of the lack of one could make a dent.

In another similar situation, I’ve seen this replacement being successful. Self-checkout in markets with the option of paying a person gets the best of both worlds. If I am knowledgeable enough, I can use the self-checkout; if not, I can have someone help me.

This includes replacing engineers with AI. If my only job as an engineer had been writing code, my life would’ve been much easier over the years. But the reality is that besides writing code, I’m also expected to talk to clients, understand their requirements, implement the product, deploy it, monitor it in production, solve bugs, and so on. AI can support me and help me be more efficient with most of these steps, but I don’t see it replacing me - at least not in the next five years.