Latest From Our World

How to Automate and Optimize Your Data Pipeline with AI Agents

Vid Bevčar, Lead Python engineer

Introduction

Data pipelines are essential for any data-driven business. However, they can be challenging from maintenance perspective. Manual work, ongoing supervision, and frequent adjustments might be needed to keep up with the changes in data sources, formats, and needs. What if there was a way to automate and optimize your data pipeline with minimal human input?

In this article, we will show you how to use AI agents, which are autonomous software programs that can interact with their environment and other agents to achieve specific goals, to improve your data pipeline automation and flexibility. This blog post will cover what AI agents are, how they work, and how they can help you save time, money, and resources while delivering high-quality and reliable data.

The AI Agent Revolution in Data Management

Our exploration into AI agents for data pipeline management revealed their potential to not only take over routine tasks from data engineers but also to elevate the adaptability of pipelines to new heights. Despite the emergent stage of technologies like Large Language Models (LLMs) and AI agents, their capability to significantly improve operational outcomes was clear, along with some unexpected challenges.

What are AI agents?

AI agents are autonomous implementations of artificial intelligence that can perceive, decide, and act to achieve specific goals. They can leverage large language models, like Mistral, Llama and GPT-4 for natural language and text data (Json, code, …) understanding and reasoning. When coding is required, they can combine the power of LLM’s with code interpreters. They can leverage a multitude of tools to observe, remember and take actions in the outside world, such as Web browsing, DBs, and APIs.

The Problem

Current data pipelines are still quite rigid in a way that every change in input or output data format requires human intervention. Can we leverage the power of AI agents to “auto heal” pipelines when a change in data format is detected?
Desired workflow:

1.            User requests additional column in output dataset
2.             AI agents research available data sources to see where to get the new data from
3.             AI agents update pipeline
4.             AI agents perform test
5.             Pipeline backfills the data

Question: ChatGPT can write a code for a data pipeline. Why would I need agents?
Answer: Yes, but human interaction is required. Someone needs to write a prompt, then take the code, run it, test and deploy.

Tools

To interact with the outside world, AI agents require “tools”. Tools can be implemented as methods with extensive documentation, to allow LLMs to choose the correct one and use it properly. For the PoC implemented these tools:

Problem solving

In an ideal world, this problem solving would look straightforward.

In reality, things can go wrong, for example:

• Selected data source does not contain desired field
• Extraction code has bugs
• This is where AI Agents shine; when faced with a problem, they can adapt the strategy and reason about appropriate solutions.

• End results and findings:
  AI agents are way smarter than expected. When faced with bugs in the code they composed, they were able to recover. They also did very well on data source selection - when information was not available in the one, they quickly looked in another.
• LLMs are great at searching unstructured data. Searching through multiple data sources for correct information took near seconds. And that is without prior knowledge about datasets.
• Writing correct prompts took longer than expected. Key takeaway is that you need to be super exact about intentions and tool documentation. Leave no room for interpretation.

Conclusion

This blog post showed the power of AI agents in making data pipelines smarter and more adaptable. We explored their ability to keep up with changing data sources, formats, and needs, and how they work together with other agents and tools to tackle complicated tasks. We also walked through an example of creating a self-updating data pipeline that rises to any data handling challenge. By incorporating AI agents into their workflows, data engineers can harness advanced technology to drive innovation and improve efficiency in their fields. This sets the stage for significant progress in the domain of data management and analysis, shaping the future direction of these disciplines.

Stay tuned for our upcoming blog post, where we'll delve even deeper into the latest advancements using this cutting-edge technology!

AI – Can It Replace Your Software Developers?

It would be impossible to avoid news on the rise of AI over the last twelve months. Whether it’s ChatGPT, Midjourney, DALL-E or the more subtle applications such as Grammarly, ClickUp or Otter.ai, the range of tools that suddenly have appeared on the market has captured the imagination of the world. The hype surrounding the technology has given rise to many questions including whether some traditional roles, currently occupied by humans, are now in danger due to this emerging technology.

Are we really facing down the next industrial revolution that will result in mass layoffs as the machines finally take all the jobs? Are we likely to see a rise in opposition, a twenty first century Luddite movement if you will? While the future might hold such changes, the present tells a different story.

Rise of the Machines?

The greatest challenge we experience with any technology that transitions from the world of research into the public consciousness is the number of experts that appear out of nowhere. There quickly follows vast quantities of information and opinion, that in turn gets re-interpreted leading, inevitably, to more confusion than clarity. This contribution aims to thoughtfully add to the ongoing discourse, offering insights while recognizing the subject's complexity.

AI's emergence is not a sudden phenomenon; it has been evolving for decades, advancing quietly behind the scenes of the technology industry. A pivotal moment was in 1997, when chess grandmaster Garry Kasparov was defeated by IBM's Deep Blue, but AI's history stretches back even further.

In the last decade, many strategy games have consistently been defeated by an AI powered machine. Go, which was often considered to be the most difficult of the strategy board games started to fall as early as 2015.

It hasn’t been easy though.

In more recent history, some of the failures of AI quietly passed without garnering too much attention. One of the more notable of which was Microsoft’s Tay project, an early AI powered chatbot (like ChatGPT) that quickly had to be shut down as it started expressing Nazi sympathies and a range of either lewd or racist messages. Similar challenges have been faced by Google and Meta (formerly Facebook) in the past, with both companies forced to shut down these technologies in short order.

Peeling Back the Layers of AI

To better understand what we are seeing, we will need to get a basic understanding of the technology. This exploration won't cover every detail, as more comprehensive resources exist for deeper study. Instead, we’ll focus on AI's general challenges, avoiding complex technical details such as coding or server specifications, which are not central to this broader overview.

First, let’s remind ourselves that AI is an umbrella term that covers a lot of areas including:

• machine learning, which has become prominent in all the discussions on Big Data,
• natural language processing, which is at the heart of ChatGPT,
• computer vision, which many of us have seen in image recognition systems (it’s how your favourite photo app gathers similar pictures into collections),
• and of course - increasingly intelligent games.

Next, and this is very general, In the current landscape of mainstream AI, a fundamental approach combines learning models and reinforcement learning. The learning model, also known as the training set, represents a dataset that mirrors the real-world context that the AI is intended to engage with.

For example, a motor insurance company might build a training set that includes information on the policies that it has issued such as driver age, vehicle age, vehicle power and so on. It would also record whether ultimately there was a claim made on that policy. This representation of the world allows the AI to ‘predict’ the risk on a new policy by considering what it ‘knows’.

Here we hit the first challenge. The AI is only as good as its training set. Imagine if you will that of all the policies issued, only one was issued to a 34-year-old male, and that policy had three claims against it. The AI’s world view is that 34-year-old males are an extremely high risk. If you want to improve the quality of the results from the AI, you need to provide a sufficiently large training set. Training sets can have a build in bias, and where that bias exists, the AI will follow it, and as we have seen over time, it will often amplify it.

Reinforcement learning is probably a phrase we’re more familiar with. We humans gain our experience in this way – try something, if it works do more of that, if it fails try something else, repeat as necessary. AI systems, and especially those that play games ‘learn’ in this way. Those of you old enough to remember the early Mathew Broderick movie ‘War Games’ will recognise this as the way that World War 3 was averted.

Here we hit the second challenge. This approach, from a computing perspective is only viable as an approach where there are a finite number of options, or moves, at any given step in a process. Hence it is particularly appropriate for games, but also for natural language processing, where there are a finite number of words and syntax rules for structure. It may be a large quantity of data, but it is finite, and the range of material available for training is huge.

If you want AI to solve more general problems, you need to give it enough scenarios to cover the problems and enough data for it to ‘learn’. If we can ignore the simple problems of storage, network traffic to move the data around, the amount of time required for the AI to learn, and to verify that we haven’t created a ‘monster‘(see Microsoft above), is for now an intractable problem.

The Sentience Gap

These challenges all point us to a clearer understanding of what AI is, and perhaps more importantly, what it isn’t. AI forces us to redefine our concept of intelligence. AI as a technology demonstrates an ability to make decisions, often more quickly to humans, to process vast quantities of data in a short period of time, and to learn. All of this can lull us into believing that AI is perhaps sentient.

This is of course not the case, not yet anyway.

This lack of sentience leads us to the ultimate understanding of what AI is, and by extension what it can do. AI is superb at distilling data and combining that data in clever, sometimes even novel ways to draw some conclusions. Not all of the conclusions are correct, but with the clever application of reinforcement learning techniques, it can be trained not to draw that conclusion again.

But AI won’t innovate in the way a human can. AI, given a finite set of data may combine and recombine, but it won’t make leaps of innovation; it will not create something completely new.

This can be hard to discern, and particularly when you look at something like Midjourney, for example, where with a few simple prompts it will generate a completely new image based on the information provided. It’s extraordinarily impressive, but we must remind ourselves that it is combining what it already has data on with what is being requested to produce a result. In the end, even the amazing Midjourney is a follower, not a leader. If you want it to do more, it must be trained with additional material. Combination is a form of creation, but it is always a ‘tribute’ to something that has gone before.

So, Can I Do Without my Developers?

Unsurprisingly, there’s a connection to the past, which is perhaps more relevant in the context of AI than in any other technology.

The Luddites were a labour group in the 19th century that resisted the rise of mass production factories who used unskilled labour with mechanised looms to produce their products. The skilled textile workers were losing their influence, and as a result their income. This is so well recognised as a protest against technology that the word ‘Luddite’ has entered the language as a synonym for technophobe.

But there’s an aspect of the Luddite argument that resonates today, particularly in the context of AI. As history has shown, the skill of weaving has declined, the apprenticeships that allowed the industry to thrive and grow faded into obscurity as they had warned at the time.

We now face a similar risk with AI.

On one side, much like the mechanical weaving machines, AI can rapidly execute a process based on a known pattern. The patterns in this case are the body of knowledge used as a training set for AI. But where is this knowledge coming from? Currently it is the development community who feed the many web sites with answers to questions from other developers. But these are the very people who are at risk of being replaced. The ultimate outcome of this approach will be a steady decline of experienced developers resulting in a similar decline in the development of innovative solutions.

It would be reasonable to think that the reinforcement approach would work. Surely the AI can learn to be better?

This leads us to the other side. Reinforcement learning works on the basis of an exhaustive search and feedback on solutions to narrow down the results to the correct outcome. We must consider two questions: first, can we ever say in the realm of software development that there is a single ‘correct’ outcome? Second, who will be responsible for deciding what is correct?

Each of these questions require the same resource to resolve the questions – an expert developer. Expert developers don’t just spring into existence, like every other profession they emerge from many years of work, gaining valuable experience through a process of trial and error, and of course mentoring. But if AI as a replacement for developers, the result will be a gradual erosion of expertise to the point where there will be, once again, a stagnation of progress due to a lack of expertise.

Conclusion

AI has moved from research labs to a prominent role in public consciousness, showing immense potential to affect various aspects of our lives. The concerns that have been expressed by actors, screen writers amongst others should also give us all pause for thought on whether AI is a tool to augment human creativity, or a tool to replace people.

As an aid, AI has proven its value in boosting productivity and analytics, and in reducing monotonous tasks. Its growth in these areas will likely permeate our daily routines, potentially bringing significant benefits if implemented thoughtfully.

However, when seen as a replacement for humans, AI's capabilities are limited to executing repetitive tasks more efficiently and cost-effectively. It falls short in areas demanding innovation and creativity, fields where human problem-solving skills are irreplaceable. Although AI may appear capable, relying on it too heavily could lead to a gradual decline in innovation, quality, and creativity, particularly due to short-term cost savings.

In creative sectors like scriptwriting, this reliance on AI might lead to uniform storytelling, recycling old narratives without introducing fresh ideas. A similar risk looms in the technology sector, where overdependence on AI could stifle product innovation.

Remote Working Riddle: Is it Really Working?

We’re experiencing one of the most significant shifts in how the world thinks about the working environment arguably since the 1900s and the emergence of Taylorism and scientific management theory.

The COVID virus and associated global lock down forced much of the working population, where possible, to relocate to their homes, or at the very least away from the office environment. The subsequent scramble resulted in a worldwide shortage of laptops, web cameras and other equipment deemed necessary for remote work.

As these new circumstances emerged, companies quickly became aware that far from being the cataclysmic drop in productivity that they feared, working from home or remote working in general could result in equal, or even increased productivity from their people.

Rapid rise of digital nomadism

Equally people started to embrace the notion of becoming digital nomads, where the ‘office’ was anywhere that could support a laptop with a decent connection to the internet. They started questioning the need for the constraints of an office if they could now work equally well elsewhere. Lifestyle choices and greater consideration of work / life balance gained prominence and importance.

We still haven’t fully returned to the pre-COVID world, and many companies continue to operate a hybrid remote / in-office approach to work. The question remains though, does remote working work?

For part one of this two-part blog post, we’ll look at some of the factors that need to be considered, and the many perspectives that relate to remote working. For our purposes, remote working distinguishes itself from near-shore or offshore development teams by considering the individual operating in an environment separate from the traditional office environment. It may mean working from home but should also include some of those digital nomad aspirations.

Companies

More and more businesses are urging their employees to make a comeback to the office. Undoubtedly, a myriad of reasons are driving this push. From a pure commercial standpoint, numerous companies are bound by leases on office spaces that they can't easily renegotiate. This leaves them shouldering the costs for underutilized spaces filled with equipment that’s being replicated for remote operations. Such expenditures offer little to no value in a fully remote setup yet are unavoidable. While many companies might not readily admit it, these financial considerations are possibly the main driving forces behind their push.

The productivity debate is another narrative that's often put forward. Interestingly, research on productivity during lockdowns is polarized. Some studies point towards a surge in efficiency, while others indicate a decline. What's crucial to consider is whether there was any transformation in management approaches to better accommodate the subtleties of remote work.In essence, our management tactics witnessed only minor tweaks to accommodate the "temporary" challenges of widespread remote teams. There was no deep-seated evolution in our people-management strategies. Most viewed the situation as a transient hurdle rather than a pivotal change in our work paradigm.

These are some of the seeds of the current conflict between companies and their staff that we are now experiencing.

Staff

The staff in these companies have experienced a range of positive benefits from remote working, particularly if they traditionally worked from offices in large cities. The time they can reclaim from simply avoiding the daily commute to and from work is material and has been clearly demonstrated to have a positive impact on their mental health.

They are getting to spend more time with their family and especially young children, time which was often missed by parents before remote working. They also have had the time to become more connected with the communities which has brought benefits to local businesses leading to commuter towns becoming much more than places where people simply sleep before trudging to their chosen mode of transport to shuttle to their workplace.

All of this culminated in a wave of optimism in relation to the environment. Ironically, in the depts of lockdown, the levels of pollution dropped measurably as a direct result of less cars being on the roads.

For those less constrained by family circumstances, the opportunity to take up residence in a more favourable location, perhaps less expensive or a more appealing climate became a genuine consideration. If work could be successfully conducted remotely, then why be limited in the choice of location? Couldn’t the young, single worker simply work from anywhere?

Government

While debates on remote working usually centre on the differing needs and wishes of businesses and their employees, the government's influence in this discourse often goes unnoticed.

Legislation remains woefully inadequate to deal with the emerging demands of remote working. Whilst individual member states are pursuing various adjustments to their local laws in relation to remote work, the overall posture of the EU largely relies on the existing teleworking agreements which are now under significant review.

For those people who attempted to embrace a more nomadic lifestyle, they will have quickly run up against legal and taxation challenges. If we pause for a moment, the issues faced by legislators and tax authorities is obvious. Remote staff consume the resources of the jurisdiction in which they are operating; that jurisdiction, often a country, has a reasonable right to demand some recompense for that consumption, which is normally captured in the form of taxation.
For instance, Portugal had a tax incentive program to entice newcomers, including digital nomads. Yet, this contributed to escalating property prices and rents​1​. To curb this, the government will cease this program for new entrants by 2024, although existing participants will retain their benefits​ 2 ​​3​. Now, these tax incentives are narrowed down to individuals in higher education or scientific research, indicating a shift towards managing the economic implications while still attracting specific skilled individuals​4​.

The rules are simple enough, if you’re going to operate in a jurisdiction, an entity must exist for the purposes of taxation. The implication then is that remote workers would need to be associated with a company registered in every jurisdiction where they want to work.

Social Societies

For ages, humans have gathered together – be it in early tribal communities, as wandering tribes, or within villages and cities. The underlying truth? We are inherently social beings who crave interaction to flourish.

The COVID pandemic altered our interaction landscape, shifting our attention from coworkers to loved ones and neighbors. While we didn't become isolated, our circle of interaction changed, filling some gaps left by the workplace but also revealing new ones.

In a work setting, the pandemic has made evident that we we gave up more than we gained. This is notably evident with newcomers to the workplace. These individuals, in their formative stages with a company, need ample interaction to assimilate. In remote setups, many felt detached, leading to diminished engagement and productivity compared to those onboarded in conventional settings.

It's hardly surprising. While we might resonate with TV characters, there's an inherent disconnect. Similarly, interacting solely through platforms like Zoom or Microsoft Teams can feel impersonal. This detachment grows with blurred backgrounds, tech glitches, and other virtual meeting interruptions. This diminished sense of connection in virtual meetings probably explains why we quickly returned to jumping on planes for meetings at the earliest opportunity.

Finding the Value

Companies assemble individuals into cohesive units with the intent of fostering collaboration towards shared goals. The rhythm of these units stems both from structured operations and the innate social needs of its members.

The key lies in understanding this rhythm to strike a balance between in-person and remote work dynamics. Communication is pivotal at the onset of any project. Whether it's brainstorming, discussing alternatives, or clearly stating decisions, no tool has yet replicated the efficiency of face-to-face interaction. While there are countless digital tools designed for virtual brainstorming or ideation, the magic of human interaction in a shared physical space remains unmatched.

So, should everyone rush back to the office?

Not exactly.

Each project phase demands its own dynamic. There's a time for collaboration and a time for execution. The latter demands undistracted focus. Pre-COVID, finding such focused environments was already a topic of debate. How can one foster concentration in bustling offices? Remote work provided a simple solution—remove potential interruptions. By working remotely, unwanted distractions were minimized. If one needed undisturbed focus, they could simply mute notifications and delve into their tasks.

Looking to the Future

The benefits of remote work on people’s wellbeing are too significant to be ignored. The need for social interaction and collaboration are so inherent in human beings that we need to be together. Equally, we need space, peace and quiet to focus on work. Reconciling these apparent contradictory needs requires a shift in how we are thinking about the engagement between people and their employers.

Companies will need to adopt more flexible frameworks that support both the demands of collaboration, bringing people together, and the desire for remote focused work. In some cases, this will necessitate a change in thinking for some managers that are desperately clinging onto a more Taylorist view of the world, where staff can’t be trusted to be working if you’re not looking at them – we’re well beyond that.

Similarly, people will need to adapt their thinking on what constitutes reasonable remote working. The lifestyle of the digital nomad, working from the beautiful beaches of the world doesn’t leave much room for alignment, collaboration, education and upskilling, or simple engagement with the organisation that is paying for the lifestyle you’re enjoying. Furthermore, until legislation catches up with the current zeitgeist, freedom to work from anywhere remains more fiction than fact.

There are periods in any undertaking, generally, where the location of the work has no material impact on the outcome, and there are periods where people need to be present in body and mind to collaborate with their co-workers.
The future of office work is apparently leaning towards a hybrid model. As we transition, there's an imperative to reshape management ideologies and methodologies. Following this, we can anticipate the evolution of novel structures and infrastructures to support this new work paradigm.

What are your thoughts? Is the hybrid model the definitive future, or are we yet to uncover the ideal workplace structure? Let's discuss below!