Current issues with gen ai

I’ve recently started working in the Gen AI space and am really loving the optimism. But to some degree, I find the hype a little overwhelming. I attended several Google Developer Network events, AWS events, startup events, and a host of other conference calls around Asia. It was surprising to me that AI brings out very polarizing opinions from several well-known speakers. Some call it the end of the world, while others see it as a productivity boon that will usher in an age of prosperity and sufficiency.

I took a closer look at the large language models (LLMs) of today and noted some key facts and issues that organizations and their engineering teams often overlook while talking about the much-touted productivity:

• LLMs are just tools to predict the next word and require significant manual effort to engineer into usable applications. There is no superhuman intelligence underneath.

• LLMs are generally embedded into very data-intensive and sensitive applications. These applications need careful tuning with a huge amount of clean and relevant data, which is always a challenge to procure and maintain.

• AI applications need a lot of GPUs to function, and they are very costly to purchase and run. Not a lot of companies can afford them, so they survive on APIs provided by cloud providers or OpenAI.

• The recurring infrastructure (servers) cost of running Gen AI applications is often more than the users’ subscription fees. Even when companies want to subsidize Gen AI feature development, it often ends up being unaffordable for users.

• Despite being around for several years, LLMs still critically suffer from hallucinations and accuracy issues.

• Any specialized Gen AI use case needs grounding with business data, usually powered by vector or embedding search over the data stored somewhere (preferably a database). Vector search operations are more compute-intensive than traditional word-based search.

• Agentic workflows are slow and unreliable. Often, key issues remain within discussions among agents. Developers feel like fixing one last issue will get the tool working end-to-end, but new users often break the entire flow.

• As of today, AI tools need AI engineers to function.