The guy got arrested, lost his job and had to hire a lawyer. Almost got charged with a felony assault, but the jury decided that he shouldn't be charged. Instead, later charged with a misdemeanor. I'd be super stressed about all of that, plus the trial and then having to wait 7 hours to find out I'm not guilty, but also incredibly happy after. However, all that and his name is forever associated with this incident, so despite no criminal history he might find future employment more challenging. He has probably been through quite a bit despite not being found guilty of a crime. I think that any person who is somehow inspired by doing what he did because he got away with it was always willing to do it anyway. Surely any reasonable person would realize how much he already had to go through which is enough of a deterrent. It was some degree of luck as well, even with similar or same circumstances it might up that another individual is charged and found guilty.
He's fairly lucky he doesn't have a criminal record, but it didn't come without consequences. I think the fact that the sandwich was still wrapped on the ground, hit the officer's shoulder, that the other police at the time were visibly amused during the incident, and clearly joking about it for several days after as well with the officer who had it happen to him, showed that the incident wasn't serious enough to ruin anyone's life over. A formal criminal conviction in the US would've made it hard for him to get employment for some time, if not the rest of his life.
> The guy got arrested, lost his job and had to hire a lawyer. Almost got charged with a felony assault, but the jury decided that he shouldn't be charged. Instead, later charged with a misdemeanor. I'd be super stressed about all of that, plus the trial and then having to wait 7 hours to find out I'm not guilty, but also incredibly happy after. However, all that and his name is forever associated with this incident, so despite no criminal history he might find future employment more challenging.
While it probably won't be with DoJ again (at least under this Administration), I don't think he's going to have much problem finding a job. Being associated with "this incident" I don't think is the kind of universal black mark you seem to think it is.
You are buried in the details. The guy is a hero who opposed a fascist takeover of the US government. He's not going to have any difficulty finding employment.
Would really be great to move windows or tab groups between different profiles; Edge offers this. Sometimes you don't realize what profile you're in and you start doing stuff in your personal profile that doesn't fit there. With firefox right now there is no way to select and move any tabs, tab groups or windows to the profile that is best suited for those links.
It could be a privacy risk, though, because by directly moving a URL (and all its parameters) to another profile, it might be easy to link them together on the server side. It’s not necessarily a problem, but it’s definitely important to be aware of, since one of the main uses of profiles is precisely isolation for privacy and security.
There is inherent non-determinism in all machine learning models unless you explicitly configure pytorch or other frameworks to do determinism (https://docs.pytorch.org/docs/stable/notes/randomness.html). However, this is very unlikely to be done in models that are being run in production due to performance and other issues.
Hey Simon, do you have any posts diving into how one might be able to deal with evaluating LLMs or Machine Learning models in general when reproducibility is so difficult given non-determinism? Pytorch has an article on it https://docs.pytorch.org/docs/stable/notes/randomness.html but then doesn't really go into how one would then take this deterministic result, and evaluate a model that is in production (which would very likely need for performance reasons the non-determinism features enabled).
While this affects all models it seems, I think the case gets worse for in particular LLMs because I would imagine all backends, including proprietary ones, are batching users prompts. Other concurrent requests seem to change the output of your request, and then if there is even a one token change to the input or output token, especially on large inputs or outputs, the divergence can compound. Also vLLM's documentation mentions this:
https://docs.vllm.ai/en/latest/usage/faq.html
So how does one do benchmarking of AI/ML models and LLMs reliably (lets ignore arguing over the flaws of the metrics themselves, and just the fact that the output for any particular input can diverge given the above). You'd also want to redo evals as soon as any hardware or software stack changes are made to the production environment.
Seems like one needs to setup a highly deterministic backend, by forcing non-deterministic behavior in pytorch and using a backend which doesn't do batching for an initial eval that would allow for troubleshooting and non-variation in output to get a better sense of how consistent the model without the noise of batching and non-deterministic GPU calculations/kernels etc.
However then, for production, when determinism isn't guaranteed because you'd need batching and non-determism for performance, I would think that one would want to do multiple runs in various real-world situations (such as multiple users doing all sorts of different queries at the same time) and do some sort of averaging of the results. But I'm not entirely sure, because I would imagine the types of queries other users are making would then change the results fairly significantly. I'm not sure how much the batching that vLLM does would change the results of the output; but vLLM does say that batching does influence changes in the outputs.
This is so hard! I don't yet have a great solution for this myself, but I've been collecting notes about this on my "evals" tag for a while: https://simonwillison.net/tags/evals/
Nearly everything controversial about them can be simplified to most entirely being before the existence of the Church committee.
The reforms put in place since then has prevented any rouge sort of activity which might not have been in the interests of the US to peruse, be legal, or otherwise explicitly asked for by the president or congress for them do
> The reforms put in place since then has prevented any rouge sort of activity which might not have been in the interests of the US to peruse, be legal, or otherwise explicitly asked for by the president or congress for them do
Whew! What a relief! Now all the nasty rogue intel agency things are only done at the behest of elected officials!
(And if you believe that I have a number of bridges to sell you. And if you believe that's a good state of affairs, well, elected officials come and go, and someday there will be one you really don't like, if there hasn't been one already.)
There's two main claims that you see when it comes to the ACLU losing its way. The first one is that they don't really do second amendment cases. Some of the motivation is genuinely the conscience of the people who work there, but the much much bigger factor, because the ACLU takes on lots of conscience testing cases, is that the NRA is extremely well funded and will fight the case anyway. There's already an organization laser focused on that issue and is legislatively successful, they don't need the ACLU lawyers' help. More broadly people see the ACLU taking on more "left wing" cases, and this is largely for the same reason. If you have limited resources then you end up having to focus on the cases where there aren't other organizations with legal expertise to step in. And despite what conservative media will tell you, there aren't actually all that many people defending, say, the lgbt. Lots of organizations will show their disapproval of these laws and make a lot of unproductive noise about them, but when it comes time to actually put up money and lawyers to fight them, all the hands stay down.
The second claim is their handling of covid. And the thing is this is an interesting case study for the ACLU. One of the things that people expect from organizations is devotion to a set of principles, and that the organization will die on any hill involving them. And Covid was a situation in which they had to ask themselves whether or not dying on their hill was the right thing to do even if they were wrong. Not wrong in sense that they are defending the "bad people," they do that all the time, but that this is a situation that is genuinely exceptional and that the health and safety of the nation is more important than people's individual civil liberties. And if you turn up the severity of the pandemic I personally think it is fairly obvious that this would be the right choice. It's a hard thing to look at ideals that you have dedicated your life and career to and weigh them against people's lives. Where the controversy lies is how severe you think covid was and whether that severity crossed the threshold. And if you didn't the ACLU's actions look like an arbitrary betrayal of their core principles.
