I was in the midst of writing a snarky reply and then realized my actual issue with Kafka is that people reach for it way too often and use it in ways that don't really make sense.
Kind of like how people use docker for evrything, when what you really should be doing is learn how to package software.
Agree on the Kafka thing though. I've seen so many devs trip over Kafka topics, partitions and offsets when their throughput is low enough that RabbitMQ would do fine.
The people distributing software should shut them damn up about how the rest of the system it runs in is configured. (But not you, your job is packaging full systems.)
That said, it seems to me that this is becoming less of a problem.
Nothing inherently wrong with the core product IMHO. The issue is more with Confluent, who have been constantly swinging from hot buzzword to hot buzzword for the last few years in search of growth. Confluent cloud is very expensive, and you still have to deal with a surprising amount of scaling headaches. I have people I consider friends that work there, so I don't want to go too deep into their various missteps, but the Kafka ecosystem has been largely stagnant outside of getting rid of Zookeeper and simplifying operations/deployment. There have been some decent quality of life fixes, but the platform is very expensive, yet if you are really all-in on Kafka, you would be insane to not get support from Confluent- it can break in surprising ways.
So you are stuck with some really terrible tradeoffs- Go with Confluent Cloud, pay a fortune, and still likely have some issues to deal with. Or you could go with Confluent Platform, still have to pay people to operate it, while Confluent the company focuses most of their attention on Cloud and still charges you a fortune. Or you could just go completely OS and forgo anything Confluent and risk being really up the river when something inevitably breaks, or you have to learn the hard way that librdkafka has poor support for a lot of the shiny features discussed in the release notes.
Redpanda has surpassed them from a technical quality perspective, but Kafka has them beat on the ecosystem and the sheer inertia of moving from one platform to another. Kafka for example was built in a time of spinning rust hard disks, and expects to be run on general purpose compute nodes, where Redpanda will actually look at your hardware and optimize the number of threads its spawns for the box it is on- assuming it is going to be the only real app running there, which is true for anything but a toy deployment.
This is my experience from running platform teams and being head of messaging at multiple companies.
So much that we presume in the modern cloud wasn't a given when Apache Kafka was first released in 2011.
kevstev wrote just above about Kafka being written to run on spinning disks (HDDs), while Redpanda was written to take advantage of the latest hardware (local NVMe SSDs). He has some great insights.
As well, Apache Kafka was written in Java, back in an era when you were weren't quite sure what operating system you might be running on. For example, when Azure first launched they had a Windows NT-based system called Windows Azure. Most everyone else had already decided to roll Linux. Microsoft refused to budge on Linux until 2014, and didn't release its own Azure Linux until 2020.
Once everyone decided to roll Linux, the "write once run everywhere" promise of Java was obviated. But because you were still locked into a Java Virtual Machine (JVM) your application couldn't optimize itself to the underlying hardware and operating system you were running on.
Redpanda, for example, is written in C++ on top of the Seastar framework (seastar.io). The same framework at the heart of ScyllaDB. This engine is a thread-per-core shared-nothing architecture that allows Redpanda to optimize performance for hardware utilization in ways that a Java app can only dream of. CPU utilization, memory usage, IO throughput. It's all just better performance on Redpanda.
It means that you're actually getting better utility out of the servers you deploy. Less wasted / fallow CPU cycles — so better price-performance. Faster writes. Lower p99 latencies. It's just... better.
Now, I am biased. I work at Redpanda now. But I've been a big fan of Kafka since 2015. I am still bullish on data streaming. I just think that Apache Kafka, as a Java-based platform, needs some serious rearchitecture,
Even Confluent doesn't use vanilla Kafka. They rewrote their own engine, Kora. They claim it is 10x faster. Or 30x faster. Depending on what you're measuring.
the idea is to use AI to build super productive farms and greenhouses, improve the capability to do that in urban areas, automated and super efficient transportation.
but it's not just AI doing all that, it's someone who wants to start a business using AI himself to figure out how to best start up a greenhouse in his community and setup the tech infra needed including the API for people to be able to view available produce, estimates on availability, initiate trades, etc. (this greenhouse thing is just one example).
another example could be someone wants to build an ecosystem monitoring station to monitor the nearby ravine (pollution levels with rainfall and other events etc.) and air quality over time. this is just a small datapoint but if people all over the place build their own ecosystem/weather monitoring things using basic electronics ordered from the internet and all plug them in to a standard observability software system then that could provide some pretty awesome outcomes including figuring the best way to clean polluted water (because some of the places will surely have implemented varying methods of sanitizing their own water).
Okay this is even more pie-in-the-sky. You can build productive greenhouses today, we don’t need AI to summarize the internet to figure out how to do it. There are no secrets hidden in the generative token tea-leaves that reveals better greenhouses. Urban farming will never be profitable or sustainable in a dense urban center. It’s been done.
“Standard observability software” whatever that is also does not require AI to build. We need 10GW to calculate rainfall for who? What benefit over how we currently calculate rainfall? This rainfall is hallucinated through summarization?
> the idea is to use AI to build super productive farms and greenhouses
How? What are the mechanisms in which AI will lead to farms and greenhouses being more productive? How will AI improve the existing automation that already exists for the farming sector, and has existed for a hundred years?
fully automated with robots. the AI designs thousands of experiments and deploys them at scale. idk, i'm not an agriculture expert. it was just one example. what other possibilities are there?
electronics recycling, disassembling old computers to get the raw materials into a form that can be used again. we'll need programs to automate the production and testing and analysis of the robots that will recycle the components.
That much is obvious. The fact that you’re straining so hard to come up with these bongcloud “ideas” should clue you in that maybe this isn’t the revolutionary tech that the suits are selling it as
To their peers, i.e. their golf billionaire buddies from Fortune-500. They talk with each other and I strongly suspect propagate a whole set of alternative reality ideas among themselves. Like this obsession on the voice activated and controlled everything. Billionaire CEOs probably find it very convenient to pretend to multitask constantly and make voice recordings and commands while doing other CEO tasks or during endless meetings. After all their human secretary can later verify information without taking his time. Meanwhile almost no one from my peer group or relatives uses voice activated anything really, no voice mails, no voice controls, no voice assistants. And I never see people on the streets doing that too.
> Meanwhile almost no one from my peer group or relatives uses voice activated anything really, no voice mails, no voice controls, no voice assistants. And I never see people on the streets doing that too.
Could also be that however your peer group uses things, isn't the only way that thing gets used?
For example, voice messages seems more popular than texting around me right now, at least in Europe and Asia, where people even respond to my texts over Whatsapp and Telegram with voice messages instead. I constantly see people on the street listening and sending voice messages too, in all age ranges.
I don't think any of those people would need an AI assistant to recite cooking recipes though, but "voice as interface" seems to be getting more popular as far as I can tell.
Why you wouldn't just transcribe your message (which most keyboards and messengers support) instead of sending minutes worth of meandering audio full of "uhm" is beyond me. I use voice all the time (assistants, LLM, etc.) but voice messages can die in a fire.
So, the obvious answer to me is that voice communications accurately include tone and inflection. But other than that, there are "edge cases" (I mean, they're more like "people") that make it more appealing, especially after Google made their keyboard transcription worse for the people who get the most use out if it (aforementioned "edge cases").
My dyslexic friend's experience with software transcriptions has changed recently. No longer can they say, "What time do I need to pick you up, question mark, I'm just leaving now, comma, so I might be a little late, period." and have it use the punctuation as specified. Now, it's LLM-powered and converts the speech without really letting the user choose the punctuation, except manually after it's been written out, which is difficult to impossible for both dyslexics and blind people.
(As a side note, if a person is an "edge case", it's actually that person's every-time case.)
They don’t want to spend 30 min explaining domain knowledge required to understand a certain super specific case.
Instead they show tech’s quality on a basic highest common denominator use case and allow people to extrapolate to their cases.
Similarly car ads show people going from home to a store (or to mountains). You’re not asking there “but what if I want to go to a cinema with the car”. If it can go to a store, it can go to a cinema, or any other obscure place, as long as there is a similar road getting there.
But those are things cars make sense for. When would I stand in my kitchen with a bunch of random ingredients strewn about the counter wondering what to make with them and conclude that an LLM would have a good answer? And what am I supposed to extrapolate from that example? I guess they were showing off that the system had good vision capabilities? Okay, but generative AIs are notoriously unreliable, unlike cars. Even if the demo had worked, it would tell me nothing about whether it would help me solve some random problem I could think up.
A better analogy would be the first cars being advertised as being usable as ballast for airships. Irrelevant and non-representative of a car's actual usefulness.
The sociopaths pushing this kinda crap don't live the same lives you or I do. They have people they pay to make decisions for them, or they pay people to do shit like buy their weekly groceries for them or whatever other stupid crap they're trying to sell as a usecase for these useless AI tools. That's why all these demos are stupid shit like "Buy me plane tickets for my trip", despite the fact that 99.9% of people need very specific criteria out of their plane tickets and it's more easily done with currently available tools anyways.
They literally think "What does a regular Joe need in their day-to-day?" and their out of touch answer is "I have all these ingredients but don't know what to cook" or whatever. It's obvious these people haven't spoken to anyone who isn't an ass-licking yesman in a looooong time.
reply