An answer to a Q&A

I have an open form that my engineering team can anonymously drop in questions. I do my best to answer them as directly and honestly as I can. I thought this question was a good one to share.

Question

What does an ideal future of AI look like in our industry and space in Data Engineering as this tech continues to evolve and advance? Conversely, what would a bad future look like with AI?

Will AI continue to serve as development tools for us as Data Engineers/Software Engineers or do we see a near future where our work is slowly offloaded to AI and jobs will be consolidated into fewer positions which can lead to DEs/SWEs to manage or oversee AI and review code vs building it from the ground up? How can we best prepare for an unknown future? I believe smart people will always succeed and strong SWE fundamentals will keep us at the forefront of the AI movement. And continuing to hone and improve our craft in Data Engineering will continue to be valuable, but I worry about inevitable job replacement by AI and what this will mean for humans to do "other work".

My Response

Phew, this is a big question. I will try to address it step by step.

To me an ideal future of AI in our space helps us be more efficient and reduces some of the tasks that get left behind. With the current technology we still need “engineering”, which I think of as discernment in the application of theory to practice. The AI tools of today have lots of theory and thoughts, and they have seen it in practice, but in my experience with them, they lack the ability to properly apply it unless guided correctly. I believe this also plays into why we are seeing vibe coded apps with a lot of security vulnerabilities; the AI doesn’t know what is good nor bad, it just knows this was applied.

A bad future is where too many people in leadership do not understand this distinction and believe it can replace the human worker. We are seeing news reports of that, and my hope is that these companies are really covering up different issues and using AI as a crutch to reduce their workforce.

AI will continue to serve as development tools. I take the glass half full approach and say no; it will not consolidate into fewer positions. It will enable us to do more with each hour/day and that will be a boost to society. To look at another industry that went through rapid transformation, look to transportation. Horse and buggy was basically replaced in 1908 by the Model T. The first commercial flight took off in 1914 with a transatlantic flight following in 1919. That is a ton of change to how people get around in just 10 years and this was all in a push to do something quicker and more efficiently.

I also see it enabling us to fail more often. We can try things quickly that we might not otherwise venture into due to the time constraint of doing that thing. I see DEs/SWEs as becoming more overseeing the AI workers, again though to get more done in the same amount of time. The key will be how we train new engineers to catch pitfalls and prevent the AI from stumbling into them. It will probably retain the form that teams do now with a tech lead that will employ AI to help them oversee the products being built. The best way to prepare is to continue to invest in yourself. Make sure that you can exercise discernment and do not ignore AI in hopes it will go away. Get a license, try it out on personal projects.